Two test surfaces, picked for cost/value:
internal/types/audit_log_test.go — extend the existing invariant suite
to include the system namespace:
- DotNamespaceConvention now covers system.setting_changed,
system.admin_promoted, system.admin_revoked.
- NoCollisionsAcrossNamespaces guards against duplicates across all
three new constants.
- New SystemNamespacePrefix test pins the shared "system." prefix —
this is the contract by which GET /system/admin/audit-log filters
out per-tenant rbac.* rows. Drift here would either leak per-tenant
events into the platform feed or hide platform events from
SystemAdmin.
- New SystemWireValues test pins the exact wire strings consumed by
the new frontend audit drawer, Langfuse exporters, and future SIEM
integrations; changes to these are a breaking change.
internal/handler/system_admin_audit_test.go — direct unit tests for
SystemHandler.emitAdminAudit, the helper that promote/revoke /
ApplyDefaultStorageQuotaToAllTenants all delegate to. Uses a
capturingAuditService stub (interface-embedded so any other method
call surfaces drift loudly) and a minimal SystemHandler with only
auditSvc wired — the helper deliberately doesn't touch other deps.
Coverage:
- NilServiceIsNoop: degraded-mode contract — a handler built without
an audit service must not panic on the audit hook.
- PopulatesCanonicalFields: every responsibility of the helper —
TenantID=0 (system scope), actor from ctx, role hard-pinned to
"system_admin", action passed through, outcome=success,
TargetType="user", TargetID/TargetUserID echoing user.ID, details
round-tripping through JSON.
- NilDetailsLeavesEmptyPayload: nil details map must NOT fabricate a
payload; the DB column defaults to '{}' and emitting an explicit
null would muddle "no extra context" filters.
- NilTargetStillEmitsRow: guards the nil-target defensive branch —
promote/revoke always supply one today, but the row still goes out
with empty target ids rather than crashing.
- IdempotentBranchSurvivesMarshal: pins the two boolean discriminator
flags (promote.idempotent, revoke.changed) so the audit reader can
distinguish a real grant from a probe and a real revoke from a noop.
Regression guard against accidentally swapping the payload to
stringly-typed shapes.
- LogErrorIsSwallowed: best-effort contract — a failing audit write
must NOT propagate, because the underlying privilege change has
already succeeded and bubbling the error would force the caller to
retry or roll back, both strictly worse than log-and-continue.
Mirrors the existing TestAuditLogHandler_* suite for the new
GET /system/admin/audit-log endpoint:
- AlwaysQueriesTenantZero: the defining contract — handler must call
AuditLogService.List with tenant_id=0 unconditionally, regardless of
any URL/header input. A regression here would leak per-tenant rbac.*
rows into the platform feed (or hide system.* rows from SystemAdmin).
- PassesQueryFiltersThrough: every advertised query key (after_id,
limit, action, outcome, actor) propagates exactly. Catches typos in
the param-key list.
- EmptyResultProducesZeroCursor: an empty service response must
collapse next_cursor to 0 so the drawer's infinite-scroll watcher
stops paginating.
- GarbageCursorAndLimitTolerated: malformed after_id / non-positive
limit fall back to defaults (matches ListTenantAuditLog) instead of
hard-failing, so stale URL params never blank-screen the drawer.
- ServiceErrorReturns500: List() errors surface as 500 via
errors.NewInternalServerError + ErrorHandler middleware, with a
non-empty body so the drawer alert has something to render.
The system_settings update, system admin promote/revoke, and
apply-default-storage-quota routes have been writing audit rows since
the prior commits, but with TenantID=0 (system-scope). The per-tenant
GET /tenants/:id/audit-log endpoint filters by tenant_id and never
returns them, so until now those rows existed only in the DB with no UI
surface. This commit closes the loop:
Backend:
- GET /api/v1/system/admin/audit-log: new SystemAdmin-gated endpoint
reusing AuditLogService.List with tenant_id=0. Same cursor-paged
shape and query params (after_id / limit / action / outcome / actor)
as the per-tenant feed, so the frontend reuses the same client logic.
- Wired through RegisterSystemAdminRoutes (mounted on the existing
adminRoutes group so it inherits the SystemAdmin() guard). The
handler dependency is optional: nil auditLogHandler skips the route,
mirroring RegisterTenantRoutes' /audit-log handling.
Frontend — new platform audit drawer in SystemSettings.vue:
- "审计日志" entry button in the section header opens a side drawer
(880px) listing system-scope events. Lazy-loaded on first open;
refresh is explicit via a button inside the drawer.
- Table columns: stacked date/time (so 50 events in the same minute
remain distinguishable), stacked actor/role, action tag, structured
target (subject key + diff line), outcome. The dead request column
(system actions don't go through middleware path capture) is dropped
in favour of richer target rendering.
- Per-action target formatters:
* system.setting_changed: subject = registry key, diff = `old → new`
(JSON-encoded, 80-char truncation). Reset shows `old → (空)`.
* tenant_storage_quota bulk apply: subject = "批量同步", diff =
"applied to N tenants (X GB)".
* system.admin_promoted / revoked: subject = "name (email)", diff
annotates idempotent / noop branches so an audit reader can tell
a real grant from a probe.
- Click-to-expand row reveals the full audit context: actor UUID,
target_user_id / target_type / target_id, and raw details JSON in
monospaced scroll-capped block. No psql round-trip needed for
forensic spot checks.
- Sticky thead pinned to the scroll container so column labels survive
long scrolls. Cells vertical-aligned middle to keep single-line tag
cells visually balanced against multi-line target cells. No zebra
stripes — the stacked content already provides row separation, and
stripes on top read as noise.
Frontend — same polish back-ported to TenantMembers.vue audit drawer:
- Same drawer width, stacked time / actor cells, structured target +
diff layout, expandable raw-details row, sticky thead, vertical-
align middle, no stripes. Refresh button reformulated as a text
button with label (was an outlined square icon-only).
- request_path column kept (rbac.access_denied carries meaningful
paths) but empty values render as a placeholder dash so they don't
read as broken.
- Diff line now covers rbac.invitation_sent / invitation_revoked role
in addition to the existing role_changed / access_denied details.
API:
- frontend/src/api/system/index.ts: listSystemAuditLog() reuses the
AuditLog / ListAuditLogParams types from @/api/tenant/audit-log
(re-exported) so consumers don't need to cross-import.
i18n (zh-CN / en-US / ko-KR / ru-RU):
- system.globalSettings.audit.*: full drawer copy + per-action labels
(system.setting_changed / admin_promoted / admin_revoked) + target
diff templates + expanded-row labels.
- tenantMember.audit.expanded.*: expanded-row labels added so the
shared drawer treatment renders cleanly under tenant scope.
Replace the standalone /platform/system/* routes with a single
"系统设置" section inside the canonical Settings modal. The previous
SystemLayout.vue / SystemAdmins.vue surfaces are removed and their
functionality (system admin roster, global settings) is hosted directly
in SystemSettings.vue under the standard `.section-header` /
`.settings-group` skeleton. Legacy URLs redirect to the modal section
so external bookmarks don't 404.
Backend:
- SystemSettingService.Reset + repo.Delete: drop the DB override for a
key so the 3-tier resolver falls back to ENV / built-in default.
Idempotent (resetting a never-persisted key returns nil); emits an
audit row only on real deletions, invalidates the local cache, and
publishes to peers via the existing pubsub channel.
- TenantService.BulkSetStorageQuota: overwrite every tenant's
storage_quota in one statement. Powers POST /system/admin/tenants/
apply-default-storage-quota; bypasses the per-tenant whitelist on
PUT /tenants/:id which intentionally forbids storage_quota edits.
- AuditAction{SystemAdminPromoted,SystemAdminRevoked} constants and
emitAdminAudit() in SystemHandler — promote / revoke now leave an
audit trail with TenantID=0 and {target_email,target_username,
idempotent|changed} in details.
- SystemSetting.LastModifiedByName: derived per-request display label
(username, email fallback) so the UI shows "wizardchen" instead of
a UUID prefix without storing a denormalised column.
Frontend:
- SystemSettings.vue rewritten against the Settings modal skeleton with
auto-persisting controls (switch/select @change, input/inputnumber
@blur, tag-input + per-delta popconfirm for SSRF whitelist and admin
roster). auth.registration_mode change goes through an inline
popconfirm; cancel rolls back. Reset / bulk-apply also inline
popconfirms — no dialog modal for these per-row affordances.
- Priority hint panel surfaces the DB > ENV > default resolver order
so operators can reason about "I set the env but it doesn't show up".
- Router: /platform/system, /platform/system/settings, /platform/system/
admins are now compatibility redirects to /platform/settings?section=
system-global.
- Settings modal sized 900x700 → 1080x780 and content-wrapper 600 → 760
so the wider tables (members, system settings) breathe; <1100px
viewport still flexes to the screen.
- i18n: system.globalSettings.* keys for title / description / loading /
empty / badges / priority hint / per-key labels / reset / bulkApply /
admins (label, placeholder, save messages, popconfirm copy) across
zh-CN, en-US, ko-KR, ru-RU.
Misc:
- internal/utils/filesize.go doc: clarify MAX_FILE_SIZE_MB is a
deploy-time-only knob (nginx + docreader + frontend each cache the
env at startup); a SystemAdmin UI override would mislead operators
because nginx would still 413. Until all four layers can hot-reload
the limit in lockstep, this stays env-only.
- internal/utils/security.go: SSRF whitelist parser/runtime now drives
off SystemSettingService for live updates; ENV remains the fallback
for never-overridden deployments.
- Added RevokeSystemAdmin functionality to the user service and repository, ensuring atomic checks for self-revoke and last admin scenarios.
- Updated the system handler to utilize the new revocation method, improving error handling for various edge cases.
- Enhanced the bootstrap process to prevent unintended promotions when system admins already exist.
- Refactored related comments and documentation for clarity on the new behavior and safeguards in place.
- Added WEKNORA_BOOTSTRAP_SYSTEM_ADMIN_EMAIL environment variable to promote a specified user to system admin on startup.
- Introduced a new bootstrap process in `bootstrap.go` to handle the promotion logic.
- Updated `.env.example` to document the new environment variable and its behavior.
- Created new views for managing system administrators and system settings, including listing, promoting, and revoking admin privileges.
- Enhanced the frontend to reflect the new system admin features, including UI elements for admin management and settings configuration.
- Updated API interfaces to support system admin functionalities, ensuring proper data handling and user management.
The intent prompt override logic in query_understand applied
strings.TrimSpace to the value before assigning it as the system prompt
override, which silently stripped trailing newlines and intentional
formatting from agent-supplied prompts. Use TrimSpace only to detect
emptiness (so whitespace-only strings still fall back to the global
default) while passing the raw string through verbatim.
Extract the resolution into applyIntentPromptOverride and add unit tests
covering agent-wins, whitespace preservation, blank fallback, no-config,
and global-only paths.
First half of the gated OpenSearch k-NN driver introduced in PR 1
(#1445) by way of #1440. PR 2a ships a hollow, interface-compliant
shell of the `internal/application/repository/retriever/opensearch/`
package — every behavioural method (Save / BatchSave / DeleteBy* /
Retrieve, plus the previously-stubbed CopyIndices / BatchUpdate* /
EstimateStorageSize / swapToVersion) returns `ErrFeatureNotEnabled`
or a conservative sentinel value. PR 2b lands the real read/write
implementations in dedicated files (`query.go` + `retrieve.go` +
`crud.go`) and replaces the stubs accordingly.
Strict feature-gate (unchanged from PR 1): no entry is added to
validEngineTypes / GetVectorStoreTypes / retrieverEngineMapping /
BuildEnvVectorStores / container env path / engine factory switch,
so the driver remains unreachable. Attempting to register an
`engine_type=opensearch` VectorStore continues to fail with the
existing "not a valid engine type" error.
What lands in PR 2a
-------------------
Driver skeleton (6 production files + 2 test files, ~1170 + ~1115 LoC):
- `repository.go` — Repository struct + NewRepository constructor
that validates cluster reachability + OS version (2.4+ / 3.x;
primary tested 3.3.2) + k-NN plugin presence on every cluster
node. sync.Once-guarded ensureReady(ctx, dim) for lazy per-
dimension index creation, with transient errors not cached so a
momentary cluster blip does not permanently poison a dim.
sanitizeIndexName enforces a strict OS-compatible name spec.
probeVersion uses robust strings.Split/Atoi parsing for
pre-release suffixes and missing-patch versions. EngineType
returns the PR 1 constant; Support returns [keywords, vector].
- `transport.go` — newOpenSearchClient ships TLS posture
(MinVersion TLS 1.2, opt-in InsecureSkipVerify, forward-secrecy-
only cipher list) and transport tuning for the driver. Caller
exists only in PR 3 (container.go + engine_factory.go); PR 2a
remains gated dead code.
- `mapping.go` — buildIndexMapping(cfg, dim) produces the full
knn_vector + HNSW + content-analyzer mapping with every *_id
field as an explicit keyword and source_type as integer.
buildKeywordsMapping ships the dim-less keyword-only index
mapping used by the no-embedding save path. createIndexAndAlias
creates <alias>_v1 and aliases <alias> to it, with best-effort
orphan cleanup and mapping-drift detection.
- `config.go` — internalCfg (value type) applying OpenSearch
defaults (hnsw_m=16, ef_construction=100, ef_search=100,
shards=4, replicas=1, engine=lucene).
- `errors.go` — nine sentinels (ErrIndexNotFound,
ErrDimensionMismatch, ErrAuth, ErrTransport,
ErrVersionUnsupported, ErrConfigInvalid, ErrFeatureNotEnabled,
ErrBatchTooLarge, ErrCircuitBreaker). Repository never imports
apperrors; PR 3's engine factory wraps these to typed AppError
2200/2201.
- `stubs.go` — every behavioural method returns
ErrFeatureNotEnabled. EstimateStorageSize returns a conservative
HNSW lower-bound estimate (not 0) so the Phase 2 KB-delete guard
fails-closed for non-empty KBs.
Tests (~1115 LoC, 50 cases):
- `repository_test.go` — interface satisfaction, sentinel mapping,
sanitizeIndexName positive/negative matrix, semver parsing
(pre-release / missing-patch), buildIndexMapping JSON shape pin
(Lucene + Faiss + Keywords), probeVersion matrix (OS 1.x / 2.2 /
2.5 / 2.11 / 3.x / 3.0.0-rc1 / ES rejection), probeKNNPlugin
multi-node coverage, ensureReady concurrency + per-dim isolation
+ transient retry, NewRepository storeID validation, all 11
stubs (CopyIndices, BatchUpdate*, EstimateStorageSize,
SwapToVersion + Save / BatchSave / Retrieve / DeleteBy*),
wrapTransport sentinel mapping + leak guard, isNotFound /
isAlreadyExistsError, drainAndClose / limitedDecode helpers.
- `transport_test.go` — TLS defaults / opt-in InsecureSkipVerify /
TLS 1.2 pinning / cipher list / transport tuning.
Single dependency addition: github.com/opensearch-project/
opensearch-go/v4 v4.6.0 in go.mod/go.sum.
SDK quirks discovered (opensearch-go v4.6.0)
--------------------------------------------
PR 2a includes the workarounds for two of three SDK limitations
that landed during full implementation (the third, Refresh:*bool,
only affects the delete path that ships in PR 2b):
- AliasExists method passes dataPointer=nil to its internal do(),
which means non-2xx responses come back as a plain
*errors.errorString ("status: 404 Not Found") rather than as
*opensearch.StructError. aliasExists therefore inspects
resp.StatusCode directly (resp is returned even when err is
non-nil) and only falls back to wrapTransport for the "no
response at all" case.
- sync.OnceReset is not in the standard library; the keyword-only
index uses a mutex + ready/err flag pattern so transient failures
can be retried by the next caller. The per-dimension path uses
the `once map[int]*sync.Once` delete-and-recreate trick.
Test fixes folded in
--------------------
While doing a full `go test ./...` against PR 1-merged main, two
deterministic regressions surfaced that block a clean run-everything
signal. Both are unrelated to the driver and are folded into PR 2a
so the PR's own CI run is green:
(1) Follow-up to #1445 — fanout test missed the new normalizer policy
(internal/application/service/knowledgebase_search_fanout_test.go,
+46 / -6). #1445 changed EngineAwareNormalizer for ES /
ElasticFaiss / OpenSearch / Weaviate / Postgres / SQLite /
Qdrant / TencentVectorDB / Doris from (score+1)/2 to clamp01
passthrough (those engines surface non-negative cosine to the
normalizer per Lucene script_score non-negative invariant for
ES, k-NN plugin SpaceType.COSINESIMIL.scoreTranslation for
OpenSearch, engine-internal or IR-normalized conversions for
the rest). Milvus is now the only engine that still surfaces
raw signed cosine in [-1, 1].
TestRetrieveFromStores_MixedEngine_Normalizes still asserted
the old cosine-shift behaviour for ES (raw -0.4 → expected 0.3)
which under passthrough now becomes clamp01(-0.4) = 0. The
normalizer's own _test.go was updated at #1445 time, but this
fan-out integration test was not.
Fix: rewrite the godoc to spell out the two engine groups;
restate sub-case 2 as ES passthrough on a production-possible
mid-range cosine (0.3 → 0.3, PG out-ranks ES); add sub-case 3
pinning the cosine-shift branch via Milvus -0.4 → 0.3.
(2) Pre-existing — SSRF whitelist singleton race surfaced by this run
(internal/utils/security.go + internal/utils/security_test.go +
internal/infrastructure/web_search/searxng_test.go,
+33 / -9). loadSSRFWhitelist in internal/utils/security.go is
cached via sync.Once on first call. The internal reset helper
resetSSRFWhitelistForTest was unexported, so tests in other
packages could not reset and saw whatever whitelist was cached
by the first sync.Once.Do() in the same test binary. In
internal/infrastructure/web_search/, TestValidateProxyURL runs
before TestValidateSearxngBaseURL alphabetically and exercises
ValidateURLForSSRF with no SSRF_WHITELIST set, caching an empty
whitelist; the later setenv in searxng_test then has no effect
and 127.0.0.1 is rejected with "hostname 127.0.0.1 is restricted".
Pre-existing on main; surfaced now because this PR was the
first to do a full `go test ./...` run on top of #1445.
Fix: capitalize the helper to ResetSSRFWhitelistForTest (the
ForTest suffix is the test-only contract); update in-package
callers; in web_search/searxng_test.go import internal/utils
and call ResetSSRFWhitelistForTest around the env mutation in
both TestValidateSearxngBaseURL and TestSearxngProvider_Search.
No production code path changes.
Roadmap
-------
- PR 2b (next, depends on this PR) — read/write implementations:
query.go + retrieve.go + crud.go land their real bodies; stubs
for Save / BatchSave / DeleteBy* / Retrieve in stubs.go are
removed; corresponding CRUD/retrieve/filter test cases (~430
LoC) join repository_test.go.
- PR 3 — activation switch + async paths (CopyIndices,
BatchUpdate*, large-batch async deletes) + i18n + docker-compose
dev profile. After PR 3 merges, the OpenSearch driver becomes
reachable via either `engine_type=opensearch` VectorStore or
`RETRIEVE_DRIVER=opensearch` env.
Backward compatibility
----------------------
- New package — additive only. No existing file modified except
go.mod / go.sum, the two test files in (1)/(2), and the
test-only export rename in utils/security.go.
- Driver is unreachable: no registry path activates it.
- No SQL migration.
- The PR 1 normalizer case for OpenSearch remains unreachable
here (no driver instance produces a result yet).
Test plan
---------
- [x] go build ./... clean
- [x] go vet ./... clean
- [x] go test -race -count=1 ./internal/application/repository/retriever/opensearch/... passes
- [x] grep -r "case types.OpenSearchRetrieverEngineType" internal/
shows only PR 1's normalizer case + this driver's EngineType()
and tests — no activation path.
- [x] grep -r "case \"opensearch\"" internal/ shows no hits.
- Add Type field to v2User struct to distinguish personal/team tokens
- Route team tokens directly to ListGroupRepos (skip ListUserGroups)
- Gracefully handle ListUserGroups 404 for personal tokens without teams
- Add rate-limiting delay between GetDocDetail calls to avoid API throttling
When the rerank model returns an error (e.g. 401 Unauthorized), the
pipeline previously discarded all retrieved candidates and returned
empty results to the caller.
Now p.rerank returns ([]RankResult, error) to distinguish API failure
from threshold-filtered empty results. On API error, the pipeline
falls back to the original retrieval candidates (directLoad +
candidatesToRerank) and continues to CHUNK_MERGE/FILTER_TOP_K,
so users still get results even when the rerank model is misconfigured.
Problem:
Doris does not support ? placeholders for LIMIT and OFFSET clauses, causing
VectorRetrieve, KeywordsRetrieve, and CopyIndices queries to fail at runtime
with SQL parse errors.
Root cause:
1. Query builders passed TopK/pageSize/offset as ? bind parameters (valid in
MySQL/PostgreSQL but not in Doris's SQL dialect)
2. Doris MySQL driver was not configured to interpolate parameters, preventing
automatic client-side placeholder replacement
Fix:
1. Enable parameter interpolation in Doris connection DSN:
- Add &interpolateParams=true to mysql.Open() DSN in container.go
- This enables driver-level parameter substitution at query time
2. Inline LIMIT/OFFSET as string literals in Doris queries:
- VectorRetrieve: LIMIT ? → LIMIT %d (params.TopK)
- KeywordsRetrieve: LIMIT ? → LIMIT %d (params.TopK)
- CopyIndices: LIMIT ? OFFSET ? → LIMIT %d OFFSET %d (pageSize, offset)
- Remove inlined values from args slice
Result:
Queries are now built with literal LIMIT/OFFSET values instead of placeholders,
compatible with Doris's SQL parser.
Three correctness fixes that the lifecycle PR deliberately deferred:
1. ID length / struct-tag drift
- models.id is varchar(64) on both PG and SQLite (per the init
migrations), but Model.ID's GORM tag said varchar(36) — a remnant
from when the field only held UUIDs. The mismatch is harmless under
golang-migrate (struct tag is ignored), but misleading on AutoMigrate
paths and in IDE tooltips. Tag now matches the real column width.
- New ModelIDMaxLen constant (=64) is the single source of truth for
anyone accepting user-provided ids. The YAML loader uses it to
reject too-long ids up front with a clear message instead of letting
the INSERT explode with a generic "value too long for type" error.
2. Field validation in the YAML loader
- Type, Source, and Status are typed strings but YAML can supply any
value. Misspellings (e.g. `type: knowledgeqa` lowercase, `type: LLM`)
were previously persisted as-is and produced rows that looked fine
in the table but failed at provider-factory lookup time, which is
hard to debug.
- validateBuiltinModelEntry now checks: empty id, id length, empty
type, type ∈ {KnowledgeQA, Embedding, Rerank, VLLM, ASR}, and
status ∈ {active, downloading, download_failed, empty}. Source is
intentionally NOT validated because the provider matrix in
internal/models/* keeps growing and a strict allow-list here would
force changes in two places per new provider.
- Invalid entries are warned + skipped (not aborting the whole load),
and excluded from the keep-set so the drift sweep does not delete
existing matching rows on the strength of a typo'd YAML retry.
3. Magic number cleanup
- DefaultBuiltinModelTenantID (=10000) replaces the hard-coded `10000`
literal in toModel(). The invariant lives in three places already
(PG migration, SQLite migration, this constant); naming it makes
the cross-reference explicit and grep-able.
Tests:
- New TestLoadBuiltinModelsConfig_RejectsInvalidEntries with five
sub-cases (id-too-long, missing-type, lowercase-type, unknown-type,
unknown-status) asserts the table stays empty after each.
- All 11 existing tests still pass.
Original PR #1453 used fmt.Printf, which lands as unstructured noise in
release/JSON log pipelines. The natural fix is logger.Infof/Warnf, but
internal/logger itself imports internal/types — using it from here
would create an import cycle (this is the same constraint that forces
model.go's crypto error logs onto stdlib log).
Switch all loader output to log.Printf with a stable "[builtin-models]"
prefix (mirroring the "[crypto]" convention already established in
model.go). The prefix gives operators a grep handle even though the
lines stay unstructured.
Levels are encoded inline as "WARN:" for warnings so log shippers and
humans can still discriminate.
Extend the builtin_models.yaml loader so the YAML file becomes a complete
source of truth for the rows it owns. Builds on the previous commit's
managed_by column.
Lifecycle contract:
- Every UPSERTed entry is tagged managed_by="yaml".
- The DoUpdates list now includes deleted_at, so an entry that was
soft-deleted (e.g. via UI/API) is automatically resurrected when it
reappears in the file. Closes the "ghost row that exists but is
invisible" failure mode.
- After all UPSERTs, the loader soft-deletes rows where
managed_by='yaml' AND id NOT IN (current YAML id set). Removing an
entry from YAML is now the supported way to retire a built-in model —
no manual SQL needed.
- Rows tagged managed_by='' (UI/API/SQL-seeded built-ins) are invisible
to the reconcile path and never touched.
- When a YAML entry sets is_default=true, the loader first clears
is_default on any other rows in the same (tenant_id, type) bucket,
mirroring the invariant enforced by the API path
(repository.UnsetDefaultModel).
Failure handling stays defensive:
- File missing / not a regular file / parse error: warn and skip; the
drift sweep is NOT executed so a malformed file cannot wipe rows.
- Per-entry UPSERT error: warn, drop the id from the keep-set so the
sweep also leaves the existing row alone ("leave alone on failure").
Tests cover: file-missing, parse-error, basic upsert + defaults,
idempotency, ${ENV} interpolation (set vs unset), drift sweep removing
YAML rows, drift sweep ignoring manual rows, soft-delete resurrection,
is_default cleanup across tenant+type, explicit empty list sweeping all
yaml-managed rows, and a regression guard ensuring BeforeCreate does not
overwrite YAML-supplied stable ids.
Docs are rewritten so operators see "delete from YAML and restart" as
the supported removal path; SQL is retained only for the legacy
managed_by='' slice.
Introduce a `managed_by` varchar column on `models` so future declarative
loaders can claim ownership of a subset of rows without disturbing entries
created via the UI/API or seeded by hand-written SQL.
- versioned/000052_models_managed_by.{up,down}.sql add the column with a
default of '' and a partial index on non-empty values to keep startup
reconciliation cheap.
- sqlite/000000_init.up.sql is updated in place (the Lite init migration
is a single file per project convention) so fresh SQLite databases get
the column from the start.
- Model.ManagedBy mirrors the column. Existing rows default to '' which
the YAML loader treats as "manually managed, never touch".
Schema half of the YAML-driven built-in-model lifecycle work that follows
up on #1453; the reconciler that uses the column lands in the next commit.
Allow built-in models to be declared in config/builtin_models.yaml
instead of inserting rows via SQL. On every startup the file is read
and each entry is UPSERT-ed into the models table (is_builtin=true)
by stable id.
Any string field may reference an environment variable with ${NAME}.
Unset variables are left as the literal placeholder so
misconfiguration surfaces clearly in provider calls rather than
failing silently with an empty token.
The file is optional: missing file, parse errors, and per-entry
upsert failures all log a warning without aborting startup.
docker-compose.yml adds env_file (.env, required:false) so
deployment-specific variables are passed through automatically.
When a custom agent has `MultiTurnEnabled=false`, `applyAgentOverridesToChatManage`
sets `chatManage.MaxRounds = 0` to signal "no history". Two pipeline plugins
mistreated this zero value as "use the global default" and silently re-loaded
session history into the LLM context:
- `PluginLoadHistory` fell back to `Conversation.MaxRounds` when
`chatManage.MaxRounds == 0`.
- `PluginQueryUnderstand.loadHistory` had the same fallback, and even when
`LOAD_HISTORY` was skipped it would re-populate `chatManage.History`,
leaking previous turns into rewrite, image analysis, and the final answer.
The RAG branch in `session_knowledge_qa` also added `LOAD_HISTORY`
unconditionally, unlike the pure-chat branch which guarded it with `hasHistory`.
This change:
- Treats `chatManage.MaxRounds <= 0` as an explicit disable in both plugins;
no fallback to global config.
- Makes the RAG pipeline consistent with the pure-chat path by gating
`LOAD_HISTORY` on `hasHistory`.
- Removes the duplicated `Current Time: {{current_time}}` line from
`agent_system_prompt.yaml`. The agent already receives a fresh
`<runtime_context><current_time>` block with each turn from
`observe.buildRuntimeContextBlock`, so the static placeholder was
redundant.
The ReAct agent path (`session_agent_qa`) already checked `MultiTurnEnabled`
directly and is not affected.
Closes#1479
- Refactored checkStorageEngineConfigured to be a method of knowledgeService, enhancing encapsulation and readability.
- Updated logic to allow fallback to global file service when no storage provider is configured at the KB or tenant level, improving error handling.
- Added detailed comments to clarify the method's behavior and internal logic, ensuring better understanding for future maintenance.
- Removed the VideoInfo field from the Chunk struct in chunk.go, streamlining the data model.
- This change reflects a shift in focus away from video information storage within the Chunk type.
Reuse enqueueKnowledgeListDelete inside DeleteKnowledge so that single-item
delete shares the same hardening as BatchDeleteKnowledge / ClearKnowledgeBase:
asynq retries, business-aware queue routing, and marking-as-deleting inside
the worker.
The endpoint now returns 200 once the delete task has been enqueued; the
response body carries the asynq task_id and the message is updated to
"Delete task submitted". Swagger annotations, generated docs and the Go
client SDK comment are updated to reflect the new asynchronous semantics.
Note: this is a behavior change. Callers that previously assumed the
knowledge was already gone on a 200 response should poll the task status
or accept eventual consistency, matching the existing BatchDeleteKnowledge
contract.
`cached_tokens` is reported by every OpenAI-compatible provider that
supports prompt caching, but how it becomes non-zero differs by mode:
- Implicit caching (OpenAI, Azure OpenAI, DeepSeek, …) populates the
field automatically whenever a prompt prefix matches a previous
request within the provider's cache TTL. No client-side opt-in.
- Explicit caching (Qwen on Aliyun, Anthropic Claude, …) only
populates the field after the caller attaches `cache_control:
{"type": "ephemeral"}` to the relevant message / content block.
Until that opt-in is applied upstream of the request, the field
stays zero even when the prefix is otherwise byte-stable.
Without this distinction documented, the previous commit reads as if
`TokenUsage.CachedTokens` will show non-zero values for Qwen / Claude
once this PR lands — which is not the case. The plumbing here is a
prerequisite (stable prefix via sorted tools) and a meter (visibility
of the field), but the explicit-cache opt-in itself is out of scope
and lives elsewhere.
Document this on `TokenUsage.CachedTokens` and the `cachedTokens`
helper so callers do not mistake observability for activation.
OpenAI-compatible providers (Qwen, DeepSeek, OpenAI, Azure, etc.) report
prompt-cache hits in `usage.prompt_tokens_details.cached_tokens`. This
value was being read by go-openai but dropped at the WeKnora boundary,
so there was no way to tell whether prompt caching was actually working.
This change plumbs the field end-to-end:
- `types.TokenUsage.CachedTokens` (json:"cached_tokens,omitempty") —
zero-values are omitted so payloads stay quiet for providers that
never report cache hits.
- `cachedTokens` helper in remote_api.go guards against nil
`PromptTokensDetails` (Ollama and older OpenAI-compat backends omit
the details block entirely).
- All three response-parsing paths populate it:
* `parseCompletionResponse` (non-streaming)
* `processStream` (SDK streaming)
* `processRawHTTPStream` (raw-HTTP streaming, used when callers
need to inject custom fields like `cache_control`)
- The five `[LLM Usage]` log lines now print `cached_tokens=%d` so
cache hit rate is visible in `journalctl` / log tail without going
through metrics.
Together with the deterministic tool ordering from the previous commit,
this makes Qwen explicit caching observable: a warmed prefix should
show `cached_tokens` ≈ system + tools token count (typically several
thousand) on subsequent requests within the 5-minute TTL.
Tests:
- `TestCachedTokensHelper` — nil safety + round-trip
- `TestParseCompletionResponse_CachedTokens` — populated + missing
details paths through `parseCompletionResponse`
- `TestTokenUsage_CachedTokensJSONOmitempty` — zero is omitted,
non-zero is emitted
`ToolRegistry.GetFunctionDefinitions` and `ListTools` previously ranged
over the internal map directly. Go map iteration is intentionally
randomized, so the resulting `tools` array reshuffled on every request.
That reshuffling silently breaks provider-side prompt caches that key on
a byte-level prefix match — most visibly Qwen explicit caching, which
requires the messages (system + tools + history) to be byte-identical up
to the `cache_control` marker. With random ordering the serialized tools
block changes every call, so the cache prefix never matches and the
hit rate stays at 0%.
Sort by tool name in both functions. Output is now byte-stable across
calls and `cache_control: ephemeral` can actually take effect.
Tests in registry_test.go cover:
- Deterministic ordering across 50 iterations
- JSON byte-stability across 20 iterations (the real motivation)
- Field projection (Name / Description / Parameters)
- Empty registry returns `[]` not `null`
- ListTools sorting
- First-wins duplicate registration policy (GHSA-67q9-58vj-32qx)
Lays type-system groundwork for the upcoming OpenSearch k-NN driver
(Phase 3 PR 2, see Tencent/WeKnora#1440), with strict feature-gate:
this PR ships only inert constants, schema extensions, and an
unreachable normalizer case. No path activates an OpenSearch
VectorStore yet — creation continues to fail with "not a valid engine
type" until the activation switch lands in Phase 3 PR 3.
Changes:
- OpenSearchRetrieverEngineType constant ("opensearch") in
internal/types/retriever.go. Not added to validEngineTypes /
GetVectorStoreTypes / retrieverEngineMapping yet (gated).
- ConnectionConfig.InsecureSkipVerify (bool, default false) in
internal/types/vectorstore.go, placed inside the // Common section
because it is a cross-driver TLS option. Distinct from the
Qdrant-specific UseTLS, which enables TLS on gRPC — InsecureSkipVerify
only controls verification of an already-TLS HTTPS connection.
AES-GCM Value/Scan round-trips the field as plaintext (it travels
alongside the encrypted Password / APIKey but is not sensitive itself).
- VectorStoreFieldInfo gains four optional fields: Immutable (bool),
Min/Max (*float64), Enum ([]string). All omitempty so existing UI
schema entries serialize identically. The fields will drive the UI
in Phase 3 PR 3 (read-only on Edit + range/enum constraints).
- Six new AuditAction constants under vector_store.* and opensearch.*
namespaces. Definitions only — emission lands in Phase 3 PR 3.
- EngineAwareNormalizer.Normalize is restructured to group engines by
the effective score range observed by the normalizer (not the
theoretical raw cosine range):
Range [-1, 1] (raw cosine, mapped via (score + 1) / 2):
- Milvus (COSINE metric mode). Milvus docs explicitly state the
COSINE metric range is [-1, 1].
Range [0, 1] (passthrough — already on the target scale):
- Elasticsearch v8 / ElasticFaiss. The driver issues a
cosineSimilarity(...) script_score script, and Lucene rejects
negative final scores ("Final relevance scores from the
script_score query cannot be negative" — ES docs); the
effective range observed by the normalizer is therefore [0, 1]
for IR-normalized embeddings. ES was previously grouped with
Milvus and over-corrected via (score + 1) / 2, which inflated
every ES result by 50% in mixed-engine RRF fusion.
- OpenSearch. The k-NN plugin's
SpaceType.COSINESIMIL.scoreTranslation maps the underlying
Lucene/Faiss distance (1 - cosine) to (1 + cosine) / 2 ∈ [0, 1]
before the score reaches us. Source:
github.com/opensearch-project/k-NN at
src/main/java/org/opensearch/knn/index/SpaceType.java
(COSINESIMIL enum, scoreTranslation method).
- Weaviate. The driver requests `certainty`, defined by Weaviate
as (2 - distance) / 2 = (1 + cosine) / 2, intrinsically [0, 1].
- Postgres pgvector / SQLite sqlite-vec / Qdrant /
TencentVectorDB / Doris. The driver computes (1 - cosine_distance)
or normalized inner_product, whose theoretical range is [-1, 1].
The IR-normalized positive-component unit vectors WeKnora
targets (BGE / OpenAI text-embedding-3 / Cohere /
sentence-transformers) keep the observed range in [0, 1];
negative-cosine embedding models would silently clamp to 0
downstream — explicitly documented as the IR-normalization
caveat in the struct godoc.
Dead enum references (ElasticFaiss, Infinity) are flagged in the
godoc with a pointer to internal/types/vectorstore.go's existing
"legacy/experimental, no standalone deployable instance" annotation.
Their case labels are kept for switch exhaustiveness.
Test coverage:
- OpenSearch constant wire value + collision check against existing
10 engines + gated invariant (NOT in validEngineTypes).
- ConnectionConfig.InsecureSkipVerify backward-compat (missing JSON
field deserializes as false) + round-trip + AES-GCM coexistence
with encrypted Password / APIKey.
- VectorStoreFieldInfo omitempty preservation + new-field serialization
+ *float64 pointer distinction (min=0 vs nil).
- AuditAction dot-namespace convention enforcement, prefix invariants
for vector_store.* and opensearch.*, no wire-string collisions, exact
wire-value pins for the six new constants.
- EngineAwareNormalizer:
- CosineRange retains its (score + 1) / 2 coverage but now only for
Milvus.
- UnitInterval now covers the full passthrough group (ES /
ElasticFaiss / OpenSearch / Weaviate / Postgres / SQLite / Qdrant /
Infinity / TencentVectorDB / Doris).
- New TestEngineAwareNormalizer_ElasticsearchCosinePassthrough is an
explicit regression guard for the score-range correction: cos=0.5
maps to 0.5 (not (0.5 + 1) / 2 = 0.75 as an earlier draft assumed).
- OpenSearch passthrough across (0 / 0.5 / 0.75 / 1) + engine drift
(1.0001) + defensive negative + ±Inf / NaN edges + keyword
passthrough + nil-ctx safety.
Backward compatibility is preserved at every layer:
- All new struct fields are omitempty / pointer-tagged so existing
rows and existing wire formats remain unchanged.
- Normalizer's new OpenSearch case is unreachable until the driver
lands. The ES regrouping changes the post-normalization value for
every ES vector search result (a correctness fix, not a feature) —
ES vector retrieval is currently the only production path affected.
- AuditAction constants emit no audit_log rows in this PR.
- engine_type=opensearch VectorStore creation still rejected (gated).
- Add CustomFormatter.Template driven by LOG_FORMAT env var with
placeholders %d/%level/%thread/%logger/%traceId/%msg; default
format unchanged for backward compatibility.
- Replace chained ReplaceAll with strings.NewReplacer for single-pass
substitution, avoiding accidental re-substitution when a field value
contains a literal placeholder string.
- Inject ANSI color at the %level substitution step; removes the old
whole-line ReplaceAll(line, "INFO", ...) which would mis-color
literal level tokens appearing inside messages.
- Cache threadNeeded on the formatter so runtime.Stack only runs when
the template references %thread.
Backend
-------
ListKnowledgeBases now enriches each row with the resolved vector_store
metadata (name / source / engine_type / status) via a new
buildKBListResponse helper. Store views are batch-resolved once per
request through BatchResolveStoreView so an N-KB list costs one
vector-store service call rather than N — closing the N+1 limitation
flagged in #1372's known-limitations section. Cross-tenant shared KBs
continue to render via SharedStoreDisplay so the owning tenant's store
inventory cannot be correlated across rows; the underlying vector_store_id
UUID is stripped from those responses.
Resolver failures degrade gracefully: bound KBs render as unavailable
instead of breaking the list. Test coverage pins the env / bound /
shared distinction, the batch-call-count invariant, and the
graceful-failure path.
Frontend
--------
KB editor modal gains a new "Vector Store" section. Create mode shows a
dropdown that combines the system default (env store) and the tenant's
configured user stores, fetched once at mount via the existing
listVectorStores API. Edit mode shows the bound store read-only via a
new VectorStoreBadge component with an explicit immutability hint —
matching the backend's `<-:create` GORM tag and the service-layer
UpdateKnowledgeBaseRequest DTO that already omit the field.
KB list cards surface a small engine-type badge for own-tenant bound
KBs, and a warning badge when the bound store is unavailable. Env-bound
and shared KBs render no badge (visual noise control). KB detail header
shows the bound store via the same VectorStoreBadge component; shared
KBs fall through to the badge's internal "shared" branch with no name /
engine / id rendered.
The KB editor's create-time error handler translates the typed
ErrVectorStoreBindingInvalid (2200) and ErrVectorStoreUnavailable
(2201) into localized messages and jumps the user back to the
VectorStore section so they can pick a different store or fall back to
the system default.
The KB row type gains five optional fields (vector_store_id / name /
engine_type / source / status). i18n: 18 new keys added to en-US,
ko-KR, zh-CN; ru-RU receives English placeholders pending translation
(consistent with prior PRs in this locale).
Part of #993 (Phase 2: Per-KB VectorStore Binding).
Phase 2 roadmap item: PR 5 (KB binding UI + list-response enrichment).
Depends on #994, #1310, #1372, #1386 (all backend in the Phase 2 series).
Documents whose only payload is an embedded image (e.g. a docx with a
single picture) intermittently produced the refusal line "No textual
content was extractable from this document." even though the vision
model had successfully extracted a caption.
Three coordinated fixes:
- Clarify the summary prompt that text inside `<image_caption>` and
`<image_ocr>` is first-class extracted content, not an image
reference, so the model only triggers the empty-content branch when
the body is genuinely textless.
- For image-dominated documents (real text < 200 runes after stripping
image markup) include OCR alongside captions so screenshots and
scanned figures contribute their actual content; text-heavy
documents continue to use caption-only enrichment to avoid OCR
noise from incidental figures.
- Add `EnrichContentCaptionAndOCR` which embeds caption + OCR text
inline next to the original Markdown image link, deliberately
omitting the `<image url=...>` and `<image_original>` wrapper
blocks. Those wrappers carry only opaque export hashes that consume
tokens and have been observed to retrigger the LLM's "image
reference with no extracted text" heuristic.
Tenant RBAC headline release: 4-tier role matrix (Owner/Admin/
Contributor/Viewer), per-KB resource ownership, per-tenant audit
log, tenant member management, self-service workspaces.
Also: CLI v0.3/v0.4 GA, KB retrieval fan-out across vector stores,
AES-256-GCM credential at-rest, docreader gRPC TLS+Token, Zhipu
embedding, Huawei OBS, vLLM URL for MinerU, Apache Doris compat
modes, server-side user preferences, Go 1.26.0.
See CHANGELOG.md for the full list.
docs(rbac): wire RBAC screenshots into READMEs and RBAC guide
- README.md / README_CN.md / README_JA.md / README_KO.md: replace the
single member-management thumbnail under the v0.6.0 RBAC highlight
with a 2×2 showcase (member management, workspace switcher,
self-service workspace creation, pending invitations).
- docs/RBAC说明.md: add the member-management screenshot to the
existing 前端实际界面 showcase so the guide is self-contained
and no longer cross-references README for it.
feat(rbac-ui): link tenant member page to RBAC guide
Add an inline doc-link in the Tenant Members settings page that
opens docs/RBAC说明.md on GitHub in a new tab, complementing the
existing in-app role-matrix popover. New i18n key
tenantMember.learnRbacGuide covered for zh-CN / en-US / ko-KR /
ru-RU.
Replace the English docs/rbac.md with a comprehensive Chinese
docs/RBAC说明.md and a wiki-style summary under docs/wiki/安全认证/.
Explain how tenant RBAC relates to the shared space feature (they
are orthogonal: tenant RBAC is the vertical defense, shared space
is the horizontal collaboration channel) and cross-link the two
docs in both the flat and wiki trees. Update inbound references in
.env.example, docker-compose.yml, and the auth legacy env test to
point at the new file name.
Three follow-up fixes on top of the MinerU markdown preservation work:
- Stop applying normalizeMinerUMarkdown inside ResolveAndStore. The
helper is already called by MinerUReader.Read, and ResolveAndStore is
shared by every parser (docreader, session attachments, ...). Running
the heading/image unescape regexes globally would silently rewrite
content (including inside fenced code blocks) for non-MinerU sources.
- Recognize MinerU image references whose path contains spaces, e.g.
"images/第 1 页.jpg". The previous regex used in
extractImageRefsFromContent disallowed whitespace in the URL group,
so such images were never matched and never persisted. Use a
whitespace-tolerant pattern aligned with ResolveAndStore's own
imgPattern.
- Deduplicate uploads when the same MinerU image is referenced under
multiple path forms (e.g. "images/foo.png" vs "./images/foo.png").
saveReferencedImage now caches by ref.Filename in addition to the
raw ref path, so the second variant reuses the previously stored
ServingURL instead of writing the same bytes to object storage
again.
Tests added:
- TestProcessImagesMatchesPathsWithSpaces
- TestResolveAndStoreDedupsSameImageRefVariants
MinerU already returns markdown with embedded HTML blocks, but the current\nreader runs the whole document back through html-to-markdown. That\nsecond conversion escapes valid headings and image syntax, so chunk\nprofiling sees plain text instead of markdown structure and relative\nimage references stop matching the storage pipeline.\n\nKeep MinerU output in its original markdown form and only apply narrow\ncompatibility normalization for the specific over-escaped patterns we\nactually need to recover. The converter now matches image refs by the\npaths that are really present in markdown or embedded HTML instead of\nassuming a single images/<name> form.\n\nExtend ImageResolver so relative HTML <img src=...> references share the\nsame storage rewrite path as markdown images, deduplicate repeated saves,\nand keep the frontend sanitizer compatible with MinerU's details/summary\nblocks. Add focused docparser tests that cover escaped markdown repair,\nvariant image path matching, and relative HTML image persistence.
Multi-KB hybrid search now groups KBs by their bound VectorStore (partition
key (storeID, owner_tenant_id)), retrieves in parallel via errgroup with a
SetLimit(4) cap and a per-group timeout (MULTI_STORE_RETRIEVE_TIMEOUT_SEC,
default 30s), and merges results. When the collected results span more than
one engine type, an EngineAwareNormalizer rescales vector scores to [0, 1];
keyword (BM25) scores pass through to the existing RRF fusion. Single-group
calls take the fast path with zero fan-out overhead, preserving today's
behavior for deployments where every KB has vector_store_id = NULL.
Embedding-model consistency is now enforced explicitly via
ResolveEmbeddingModelKeys. Multi-KB searches across KBs whose resolved
model identities differ return BadRequest instead of silently producing
incomparable scores. Cross-tenant Organization-shared KBs are preserved by
partitioning on KB.TenantID so the factory's ownership lookup runs against
the source tenant. Foreign-tenant KB UUIDs injected via the request body
are rejected via kbShareService.HasTenantKBPermission (Plan 3 of #1303,
3-D capped) before any retrieval; rejected scopes surface as 404 to avoid
leaking foreign KB existence.
Service-layer typed AppErrors (ErrVectorStoreBindingInvalid 2200 /
ErrVectorStoreUnavailable 2201) are mapped from PR2 sentinel hierarchy and
preserved end-to-end: the iterative FAQ path returns them rather than
swallowing, and the HybridSearch handler routes typed AppErrors to the
client unchanged instead of downgrading to 500.
Part of #993 (Phase 2: Per-KB VectorStore Binding).
Phase 2 roadmap item: PR 4 (Multi-store fan-out search).
Depends on #994, #1310, #1372.
The grep_chunks tool previously accepted an array of regex queries (1-5)
and an optional knowledge_base_ids filter and limit. In practice the LLM
either fired multiple near-duplicate calls or split synonyms across
entries instead of using POSIX alternation, and KB scoping plus result
limit are server-side concerns the model should not control.
Reshape the contract to match `grep -E -i` semantics:
- Schema accepts a single required `query` string. Combine concepts with
`|` alternation in one regex instead of multiple calls.
- Drop `knowledge_base_ids` and `limit` from the schema; the tool now
always searches the full agent scope and uses a fixed internal cap.
- Legacy `pattern`, `queries`, `patterns`, `max_results` keys are still
accepted and joined into a single alternation regex so older callers
and in-flight model outputs keep working.
- Update the agent system prompt template to document the new single
`query` field.
- Frontend tool title now reads `query`/`queries`/`pattern`/`patterns`
in that order so the search text is shown again under the new schema.
- Add a dedicated `grepSearch` / `grepSearchFailed` tool status (zh-CN,
en-US, ko-KR, ru-RU) and rename the zh-CN tool label to "搜索关键词"
so the UI no longer prefixes the call with a generic "调用 ..." label.
Add support for configuring vLLM server URL when using vlm-http-client or hybrid-http-client backend in MinerU.
Changes:
- internal/types/tenant.go: Add MinerUVLMServerURL field to ParserEngineConfig
- internal/infrastructure/docparser/mineru_converter.go: Pass server_url to MinerU API when backend is vlm-http-client or hybrid-http-client
- frontend/src/api/system/index.ts: Add TypeScript type definition
- frontend/src/views/settings/ParserEngineSettings.vue: Add vLLM server URL input field
- frontend/src/i18n/locales/*.ts: Add translations (zh-CN, en-US, ko-KR)
Problem:
The hard-coded Doris vector function implementation (cosine_distance_approximate with
UNIQUE KEY ANN tables) fails on SelectDB 4.0.2-rc01 and other Doris builds lacking that
specific function support. Users had no way to adapt without code changes.
Root cause:
There is assumption all Doris deployments support the same vector function API, but different
builds (Doris OSS, SelectDB, Doris Cloud) ship with different function variants and table
key constraints. No capability detection or user configuration existed.
Solution:
Implement DORIS_COMPAT_MODE environment variable with three modes:
* auto (default/recommended): probe Doris server on first use to detect available vector
functions; prefer inner_product_duplicate (modern Doris 4.0+), fall back to legacy
(older builds lacking inner_product_approximate)
* legacy: hard-set to cosine_distance_approximate + UNIQUE KEY (for older Doris/SelectDB
builds without inner_product_approximate support)
* inner_product_duplicate: hard-set to inner_product_approximate + DUPLICATE KEY
(for modern Doris 4.0+ and current SelectDB with normalized embeddings)
Implementation details:
- add compat.go with one-time mode resolution (sync.Once) and capability probing
- inspect existing weknora_embeddings_* table DDL via SHOW CREATE TABLE to detect and
enforce schema compatibility; prevents silent mismatches
- fail fast with clear error message when configured mode does not match existing tables,
with explicit remediation steps (recreate tables or change env var)
- branch all query paths (inner_product_approximate vs cosine_distance_approximate),
DDL generation (DUPLICATE KEY vs UNIQUE KEY), write paths (embed normalization),
and chunk updates (Stream Load vs read-modify-write) by resolved compat mode
- add comprehensive repository tests for mode selection, auto-detection, and mismatch
scenarios; all tests pass
- expose DORIS_COMPAT_MODE in docker-compose.yml with auto as default
- document in .env.example with clear mode decision guidance
- log all mode decisions (requested, detected, probed, final) at INFO/WARN level
Key guarantee:
⚠️ DORIS_COMPAT_MODE is NOT interchangeable after embedding tables are created.
App will reject mode switches that conflict with existing table layout, preventing
silent data mismatches and query failures.
Sessions now record the input-bar state used for the most recent QA
request (agent, model, KB scope, web search). The chat UI hydrates
those settings on session reopen so users see the same configuration
they used last time, instead of the global default.
The state is stored in the existing sessions.agent_config JSONB column
to avoid a new migration. Frontend snapshots the user's global defaults
on session enter and restores them on session leave, so opening an old
session does not pollute new-chat defaults.
