mirror of
https://github.com/Tencent/WeKnora.git
synced 2026-06-04 13:30:32 +08:00
40b74e2efa
Phase 3 (#1440) gate flip. PR 1 (#1445) + PR 2a (#1481) + PR 2b (#1482) laid the type prep + driver skeleton + read/write paths as gated dead code; this PR wires every activation surface so opensearch becomes a registerable VectorStore engine. Activation wiring - internal/types: validEngineTypes / GetVectorStoreTypes (with HNSW bounds + knn_engine enum + Immutable hints) / retrieverEngineMapping / buildEnvStoreForDriver — every gated surface now recognises "opensearch". IndexConfig grows four omitempty HNSW fields (HNSWM / HNSWEFConstruction / HNSWEFSearch / KNNEngine), keeping other engines' serialised config byte-identical. - internal/container: createOpenSearchEngine + the switch case in createEngineServiceFromStore; the RETRIEVE_DRIVER=opensearch env path in initRetrieveEngineRegistry; NewEngineFactory now closes over the AuditLogService (the EngineFactory type itself is unchanged). - internal/application/service/vectorstore_healthcheck.go: a testOpenSearchConnection case so CreateStore's connectivity probe accepts opensearch instead of returning 400. - internal/application/repository/retriever/opensearch/transport.go: NewOpenSearchClient is exported so the factory and env path can build the TLS-hardened client; healthcheck.go reuses the unexported probeVersion / probeKNNPlugin for the service-layer probe. Service-layer validation - validateOpenSearchIndexConfig validates the HNSW caps (m 2-100, ef_construction 2-4096, ef_search 1-10000, knn_engine ∈ lucene|faiss). Shards/replicas continue to be enforced by the flat ValidateIndexConfig. Create-only: UpdateStore mutates the name only. - validateConnectionConfig requires addr for opensearch. Sync implementations (stubs.go shrinks) - CopyIndices (copy.go) mirrors the Elasticsearch / Qdrant pattern — search → BatchSave with the source_id remap for generated questions — so dim/keyword routing and the source_id contract come from BatchSave for free. embeddingMap is keyed by the *target* SourceID because OpenSearch's BatchSave looks up embeddings by SourceID (lookupEmbedding), not by chunk_id (the ES driver's convention). Pagination is from/size; copies larger than max_result_window (default 10000) need the scroll-based async path that lands later. - BatchUpdateChunkEnabledStatus / BatchUpdateChunkTagID (bulk_update.go) group the input by target value and issue one _update_by_query per group over the cross-dim <base>_* pattern. Caller values flow through bound script params only — never string-interpolated into the Painless source — closing the script-injection surface. - inspectByQueryResponse (byquery.go) mirrors inspectBulkResponse: the full failure reason goes to the debug log only; the returned error carries the bounded id + type. - UpdateByQueryParams.Refresh is *bool in opensearch-go v4.6.0 (the same shape as DeleteByQuery's quirk), so refresh=wait_for is not expressible; we use refresh=true. Driver-owned audit (DIP) - A new opensearch.AuditSink interface (with nopSink + WithAuditSink functional option) lets the driver emit opensearch.index_created and opensearch.reindex_executed events without importing any service package — the service layer implements the interface. NewRepository takes opts, so existing 4-arg test call sites keep compiling unchanged. - internal/container/audit_sink.go bridges AuditSink to AuditLogService. When the context carries no tenant (the env-path registration ctx during boot, for example) the adapter skips the emit with a warning rather than silently writing tenant_id=0, which would collide with the system-scope sentinel. Frontend + polish - FieldSchema (frontend/src/api/vector-store.ts) gains min/max/enum/ immutable. VectorStoreSettings.vue is now schema-driven: a closed `enum` renders a t-select; number inputs use the schema's `:min`/`:max` and fall back to the legacy replica-vs-shard heuristic only when the schema does not pin them; a danger-coloured warning fires when insecure_skip_verify is toggled on (the switch and warning are wrapped in a vertical stack so the warning sits on its own row below the switch). - i18n: labels for hnsw_m / hnsw_ef_construction / hnsw_ef_search / knn_engine / insecure_skip_verify plus the warning copy in en-US, ko-KR, zh-CN, ru-RU. - docker-compose.dev.yml: an opensearch profile (single-node 3.3.2 with security plugin disabled for dev only). OpenSearch Dashboards lives in a separate, opt-in opensearch-ui profile so the heavy UI container is not forced up alongside the cluster (the driver e2e is fully curl-verifiable against :9200). The new docs/dev/opensearch-integration-test.md covers the end-to-end exercise and the single-node guidance (set replicas=0 to keep the cluster Green). Gating-guard tests flipped - The "OpenSearch is NOT in validEngineTypes / mapping / types list / env builder / stubs" guard tests from PR 1 / PR 2 are replaced by their positive counterparts in this PR. The test suite was the activation checklist; the activation flip is its diff. Backward compatibility - Additive everywhere. IndexConfig's new HNSW fields are omitempty so other engines' serialised config is byte-identical. Existing Elasticsearch / Qdrant / Milvus / Weaviate / Doris / TencentVectorDB stores are untouched. No migrations. Test plan - go build ./... clean - go vet ./... clean - gofmt -l clean on touched files - go test ./... — only TestOssEnsureBucket_CreateFails (Aliyun OSS endpoint), the docreader gRPC tests, and the doris SQL-shape tests fail; all three are pre-existing on upstream/main and untouched by this PR. - New tests across internal/types, opensearch, service and container — including a full end-to-end env-path test that exercises initRetrieveEngineRegistry with RETRIEVE_DRIVER=opensearch against an httptest cluster.
75 lines
2.2 KiB
Go
75 lines
2.2 KiB
Go
package container
|
|
|
|
import (
|
|
"context"
|
|
"testing"
|
|
|
|
"github.com/gin-gonic/gin"
|
|
|
|
"github.com/Tencent/WeKnora/internal/types"
|
|
"github.com/Tencent/WeKnora/internal/types/interfaces"
|
|
)
|
|
|
|
// fakeAuditSvc records Log calls.
|
|
type fakeAuditSvc struct {
|
|
logged []*types.AuditLog
|
|
err error
|
|
}
|
|
|
|
func (f *fakeAuditSvc) Log(_ context.Context, e *types.AuditLog) error {
|
|
f.logged = append(f.logged, e)
|
|
return f.err
|
|
}
|
|
func (f *fakeAuditSvc) LogDenied(context.Context, *gin.Context, uint64, string, string, types.TenantRole) error {
|
|
return nil
|
|
}
|
|
func (f *fakeAuditSvc) List(context.Context, uint64, *interfaces.AuditLogQuery) ([]*types.AuditLog, error) {
|
|
return nil, nil
|
|
}
|
|
func (f *fakeAuditSvc) Purge(context.Context, int) (int64, error) { return 0, nil }
|
|
|
|
func ctxWithTenant(id uint64) context.Context {
|
|
return context.WithValue(context.Background(), types.TenantIDContextKey, id)
|
|
}
|
|
|
|
func TestAuditSinkAdapter_EmitsWithTenant(t *testing.T) {
|
|
f := &fakeAuditSvc{}
|
|
sink := newAuditSinkAdapter(f)
|
|
|
|
sink.EmitIndexCreated(ctxWithTenant(42), "weknora_768", 768)
|
|
if len(f.logged) != 1 {
|
|
t.Fatalf("want 1 audit entry, got %d", len(f.logged))
|
|
}
|
|
e := f.logged[0]
|
|
if e.TenantID != 42 {
|
|
t.Errorf("tenant: want 42, got %d", e.TenantID)
|
|
}
|
|
if e.Action != types.AuditActionOpenSearchIndexCreated {
|
|
t.Errorf("action: want %s, got %s", types.AuditActionOpenSearchIndexCreated, e.Action)
|
|
}
|
|
if e.Details == nil {
|
|
t.Error("details should be populated")
|
|
}
|
|
|
|
sink.EmitReindexExecuted(ctxWithTenant(7), "src", "dst", 9)
|
|
if len(f.logged) != 2 || f.logged[1].Action != types.AuditActionOpenSearchReindexExecuted {
|
|
t.Errorf("reindex audit not recorded: %+v", f.logged)
|
|
}
|
|
}
|
|
|
|
func TestAuditSinkAdapter_SkipsWithoutTenant(t *testing.T) {
|
|
f := &fakeAuditSvc{}
|
|
sink := newAuditSinkAdapter(f)
|
|
// background ctx carries no tenant → adapter must skip (never write tenant=0)
|
|
sink.EmitIndexCreated(context.Background(), "weknora_768", 768)
|
|
sink.EmitReindexExecuted(context.Background(), "a", "b", 1)
|
|
if len(f.logged) != 0 {
|
|
t.Errorf("want 0 audit entries without tenant, got %d", len(f.logged))
|
|
}
|
|
}
|
|
|
|
func TestAuditSinkAdapter_NilServiceNoPanic(t *testing.T) {
|
|
sink := newAuditSinkAdapter(nil)
|
|
sink.EmitIndexCreated(ctxWithTenant(1), "x", 1) // must not panic
|
|
}
|