mirror of
https://github.com/Tencent/WeKnora.git
synced 2026-06-04 13:30:32 +08:00
Lets users stop an in-flight document parse to free up LLM / worker
resources without losing the chunks and index already written. The
core insight is that the previous parse_status=completed flipped as
soon as primary chunks landed, while the most expensive subtasks
(graph extract = N LLM calls per chunk, plus summary, question
generation) were still running in the background — so "completed"
wasn't actually terminal from a resource standpoint.
State machine
pending -> processing -> finalizing -> completed
|
+-> cancelled (any of the three
in-flight states)
+-> failed
+-> deleting
`finalizing` is the new post-process fan-out window. parse_status
only promotes to `completed` once pending_subtasks_count (a new
column tracking summary + question + per-chunk graph extract)
drains to zero via atomic FinalizeSubtask. Wiki ingest is
intentionally excluded from the counter — it's a KB-scoped
debounced batch and would otherwise pin parse_status in
`finalizing` for the wiki batch window.
Backend
- New ParseStatusFinalizing + pending_subtasks_count column with
migration 000056.
- knowledgeRepository.SetFinalizing transitions processing -> finalizing
conditionally so a racing cancel cannot be clobbered.
- knowledgeRepository.FinalizeSubtask atomically decrements the
counter and self-promotes the row to completed when it hits zero.
- KnowledgePostProcess restructured to compute expected subtask
count up front, flip to finalizing (or completed when no
enrichment is enabled), and only then fan out subtasks. Subtask
handlers (summary, question, graph extract) defer-decrement on
terminal exit using the existing isFinalAsynqAttempt convention.
- New POST /api/v1/knowledge/{id}/cancel-parse handler accepting
pending / processing / finalizing. Marks the row cancelled,
zeroes the counter, best-effort dequeues asynq tasks via a new
TaskInspector abstraction (asynq-mode walks pending/scheduled/
retry queues; Lite-mode noop), and scrubs wiki ingest pending op.
- SpanTracker.AbortAttempt flat-sweeps every still-running span
for the attempt via a new repo.CancelAllOpenSpans helper so the
trace viewer's striped bars all flip to cancelled, even leaf
generations whose parent stage already EndSpan'd (multimodal
fan-out pattern). knowledge_post_process closes its postSpan
via SkipSpan on the cancel/deleting entry guard so a worker
that opens a span AFTER the cancel sweep doesn't leak it.
- Housekeeping and resetPendingTasks sweep finalizing rows
identically to processing so a crash/restart can't strand them.
- DeleteKnowledge/DeleteKnowledgeList proactively dequeue
downstream tasks via the same TaskInspector path.
- ChunkExtractService gets a cancel entry guard so the most
expensive enrichment (graph extract) bails immediately when the
parent knowledge is aborted.
Frontend
- New cancelKnowledgeParse API client + "Stop parsing" entry in
both list view and card view more menus, gated on
pending/processing/finalizing.
- Polling predicate refactored to a shared isParseInFlight helper
that recognises `finalizing` (previously the doc list silently
stopped polling once parse_status flipped from processing).
- Knowledge processing timeline: isPolling includes finalizing,
new isHardTerminal short-circuits LIVE for cancelled/failed/
completed so stranded child spans cannot pin LIVE on.
- DocumentListView.computeStatus distinguishes finalizing
("增强中") from completed and shows the previous "生成摘要中"
copy when summary_status is still pending under finalizing.
Added cancelled badge as well.
- i18n: statusFinalizing / statusCancelled / cancelParse* keys
across zh-CN, en-US, ko-KR, ru-RU.
Docs / SDK
- docs/api/knowledge.md: documents the new finalizing state,
cancel-parse semantics, and which statuses accept cancel.
- client (Go SDK): CancelKnowledgeParse with docstring listing
the cancellable statuses.
WeKnora HTTP Client
This package provides a client library for interacting with WeKnora services, supporting all HTTP-based interface calls, making it easier for other modules to integrate with WeKnora services without having to write HTTP request code directly.
Main Features
The client includes the following main functional modules:
- Session Management: Create, retrieve, update, and delete sessions
- Knowledge Base Management: Create, retrieve, update, and delete knowledge bases
- Knowledge Management: Add, retrieve, and delete knowledge content
- Tenant Management: CRUD operations for tenants
- Knowledge Q&A: Supports regular Q&A and streaming Q&A
- Chunk Management: Query, update, and delete knowledge chunks
- Message Management: Retrieve and delete session messages
- Model Management: Create, retrieve, update, and delete models
- Evaluation Function: Start evaluation tasks and get evaluation results
Usage
Creating Client Instance
import (
"context"
"github.com/Tencent/WeKnora/client"
"time"
)
// Create client instance
apiClient := client.NewClient(
"http://api.example.com",
client.WithToken("your-auth-token"),
client.WithTimeout(30*time.Second),
)
Tenant Configuration
You can set a default tenant with WithTenantID; the client will automatically send the X-Tenant-ID header:
tenantID := uint64(10000)
apiClient := client.NewClient(
"http://api.example.com",
client.WithToken("your-auth-token"),
client.WithTenantID(tenantID),
)
If a single request needs a different tenant, set TenantID in the request context. The value can be a uint64, *uint64, or a numeric string, and it will take precedence over the client default:
ctx := context.WithValue(context.Background(), "TenantID", uint64(10000))
// Pass ctx into any client method to switch to tenant 10000 for that request
Example: Create Knowledge Base and Upload File
// Create knowledge base
kb := &client.KnowledgeBase{
Name: "Test Knowledge Base",
Description: "This is a test knowledge base",
ChunkingConfig: client.ChunkingConfig{
ChunkSize: 500,
ChunkOverlap: 50,
Separators: []string{"\n\n", "\n", ". ", "? ", "! "},
},
ImageProcessingConfig: client.ImageProcessingConfig{
ModelID: "image_model_id",
},
EmbeddingModelID: "embedding_model_id",
SummaryModelID: "summary_model_id",
}
kb, err := apiClient.CreateKnowledgeBase(context.Background(), kb)
if err != nil {
// Handle error
}
// Upload knowledge file with metadata
metadata := map[string]string{
"source": "local",
"type": "document",
}
knowledge, err := apiClient.CreateKnowledgeFromFile(context.Background(), kb.ID, "path/to/file.pdf", metadata)
if err != nil {
// Handle error
}
Example: Create Session and Chat
// Create session
sessionRequest := &client.CreateSessionRequest{
KnowledgeBaseID: knowledgeBaseID,
SessionStrategy: &client.SessionStrategy{
MaxRounds: 10,
EnableRewrite: true,
FallbackStrategy: "fixed_answer",
FallbackResponse: "Sorry, I cannot answer this question",
EmbeddingTopK: 5,
KeywordThreshold: 0.5,
VectorThreshold: 0.7,
RerankModelID: "rerank_model_id",
RerankTopK: 3,
RerankThreshold: 0.8,
SummaryModelID: "summary_model_id",
},
}
session, err := apiClient.CreateSession(context.Background(), sessionRequest)
if err != nil {
// Handle error
}
// Regular Q&A
answer, err := apiClient.KnowledgeQA(context.Background(), session.ID, &client.KnowledgeQARequest{
Query: "What is artificial intelligence?",
})
if err != nil {
// Handle error
}
// Streaming Q&A
err = apiClient.KnowledgeQAStream(context.Background(), session.ID, "What is machine learning?", func(response *client.StreamResponse) error {
// Handle each response chunk
fmt.Print(response.Content)
return nil
})
if err != nil {
// Handle error
}
Example: Managing Models
// Create model
modelRequest := &client.CreateModelRequest{
Name: "Test Model",
Type: client.ModelTypeChat,
Source: client.ModelSourceInternal,
Description: "This is a test model",
Parameters: client.ModelParameters{
"temperature": 0.7,
"top_p": 0.9,
},
IsDefault: true,
}
model, err := apiClient.CreateModel(context.Background(), modelRequest)
if err != nil {
// Handle error
}
// List all models
models, err := apiClient.ListModels(context.Background())
if err != nil {
// Handle error
}
Example: Managing Knowledge Chunks
// List knowledge chunks
chunks, total, err := apiClient.ListKnowledgeChunks(context.Background(), knowledgeID, 1, 10)
if err != nil {
// Handle error
}
// Update chunk
updateRequest := &client.UpdateChunkRequest{
Content: "Updated chunk content",
IsEnabled: true,
}
updatedChunk, err := apiClient.UpdateChunk(context.Background(), knowledgeID, chunkID, updateRequest)
if err != nil {
// Handle error
}
Example: Getting Session Messages
// Get recent messages
messages, err := apiClient.GetRecentMessages(context.Background(), sessionID, 10)
if err != nil {
// Handle error
}
// Get messages before a specific time
beforeTime := time.Now().Add(-24 * time.Hour)
olderMessages, err := apiClient.GetMessagesBefore(context.Background(), sessionID, beforeTime, 10)
if err != nil {
// Handle error
}
Complete Example
Please refer to the ExampleUsage function in the example.go file, which demonstrates the complete usage flow of the client.