Commit Graph

1867 Commits

Author SHA1 Message Date
wizardchen
7d1fe6f11b chore(deps): add opensearch-go v4.6.0 to go.mod and update go.sum
- Added the opensearch-go v4.6.0 dependency to the go.mod file.
- Removed the indirect requirement for opensearch-go from go.sum as it is now a direct dependency.
2026-05-26 21:13:56 +08:00
wizardchen
2a117262bf test(system): cover system.* audit constants + emitAdminAudit contract
Two test surfaces, picked for cost/value:

internal/types/audit_log_test.go — extend the existing invariant suite
to include the system namespace:
- DotNamespaceConvention now covers system.setting_changed,
  system.admin_promoted, system.admin_revoked.
- NoCollisionsAcrossNamespaces guards against duplicates across all
  three new constants.
- New SystemNamespacePrefix test pins the shared "system." prefix —
  this is the contract by which GET /system/admin/audit-log filters
  out per-tenant rbac.* rows. Drift here would either leak per-tenant
  events into the platform feed or hide platform events from
  SystemAdmin.
- New SystemWireValues test pins the exact wire strings consumed by
  the new frontend audit drawer, Langfuse exporters, and future SIEM
  integrations; changes to these are a breaking change.

internal/handler/system_admin_audit_test.go — direct unit tests for
SystemHandler.emitAdminAudit, the helper that promote/revoke /
ApplyDefaultStorageQuotaToAllTenants all delegate to. Uses a
capturingAuditService stub (interface-embedded so any other method
call surfaces drift loudly) and a minimal SystemHandler with only
auditSvc wired — the helper deliberately doesn't touch other deps.

Coverage:
- NilServiceIsNoop: degraded-mode contract — a handler built without
  an audit service must not panic on the audit hook.
- PopulatesCanonicalFields: every responsibility of the helper —
  TenantID=0 (system scope), actor from ctx, role hard-pinned to
  "system_admin", action passed through, outcome=success,
  TargetType="user", TargetID/TargetUserID echoing user.ID, details
  round-tripping through JSON.
- NilDetailsLeavesEmptyPayload: nil details map must NOT fabricate a
  payload; the DB column defaults to '{}' and emitting an explicit
  null would muddle "no extra context" filters.
- NilTargetStillEmitsRow: guards the nil-target defensive branch —
  promote/revoke always supply one today, but the row still goes out
  with empty target ids rather than crashing.
- IdempotentBranchSurvivesMarshal: pins the two boolean discriminator
  flags (promote.idempotent, revoke.changed) so the audit reader can
  distinguish a real grant from a probe and a real revoke from a noop.
  Regression guard against accidentally swapping the payload to
  stringly-typed shapes.
- LogErrorIsSwallowed: best-effort contract — a failing audit write
  must NOT propagate, because the underlying privilege change has
  already succeeded and bubbling the error would force the caller to
  retry or roll back, both strictly worse than log-and-continue.
2026-05-26 21:13:56 +08:00
wizardchen
5b9542ebe6 test(system): cover ListSystemAuditLog handler contract
Mirrors the existing TestAuditLogHandler_* suite for the new
GET /system/admin/audit-log endpoint:

- AlwaysQueriesTenantZero: the defining contract — handler must call
  AuditLogService.List with tenant_id=0 unconditionally, regardless of
  any URL/header input. A regression here would leak per-tenant rbac.*
  rows into the platform feed (or hide system.* rows from SystemAdmin).
- PassesQueryFiltersThrough: every advertised query key (after_id,
  limit, action, outcome, actor) propagates exactly. Catches typos in
  the param-key list.
- EmptyResultProducesZeroCursor: an empty service response must
  collapse next_cursor to 0 so the drawer's infinite-scroll watcher
  stops paginating.
- GarbageCursorAndLimitTolerated: malformed after_id / non-positive
  limit fall back to defaults (matches ListTenantAuditLog) instead of
  hard-failing, so stale URL params never blank-screen the drawer.
- ServiceErrorReturns500: List() errors surface as 500 via
  errors.NewInternalServerError + ErrorHandler middleware, with a
  non-empty body so the drawer alert has something to render.
2026-05-26 21:13:56 +08:00
wizardchen
acdc0a526f feat(system): expose platform audit log + polish audit drawers
The system_settings update, system admin promote/revoke, and
apply-default-storage-quota routes have been writing audit rows since
the prior commits, but with TenantID=0 (system-scope). The per-tenant
GET /tenants/:id/audit-log endpoint filters by tenant_id and never
returns them, so until now those rows existed only in the DB with no UI
surface. This commit closes the loop:

Backend:
- GET /api/v1/system/admin/audit-log: new SystemAdmin-gated endpoint
  reusing AuditLogService.List with tenant_id=0. Same cursor-paged
  shape and query params (after_id / limit / action / outcome / actor)
  as the per-tenant feed, so the frontend reuses the same client logic.
- Wired through RegisterSystemAdminRoutes (mounted on the existing
  adminRoutes group so it inherits the SystemAdmin() guard). The
  handler dependency is optional: nil auditLogHandler skips the route,
  mirroring RegisterTenantRoutes' /audit-log handling.

Frontend — new platform audit drawer in SystemSettings.vue:
- "审计日志" entry button in the section header opens a side drawer
  (880px) listing system-scope events. Lazy-loaded on first open;
  refresh is explicit via a button inside the drawer.
- Table columns: stacked date/time (so 50 events in the same minute
  remain distinguishable), stacked actor/role, action tag, structured
  target (subject key + diff line), outcome. The dead request column
  (system actions don't go through middleware path capture) is dropped
  in favour of richer target rendering.
- Per-action target formatters:
    * system.setting_changed: subject = registry key, diff = `old → new`
      (JSON-encoded, 80-char truncation). Reset shows `old → (空)`.
    * tenant_storage_quota bulk apply: subject = "批量同步", diff =
      "applied to N tenants (X GB)".
    * system.admin_promoted / revoked: subject = "name (email)", diff
      annotates idempotent / noop branches so an audit reader can tell
      a real grant from a probe.
- Click-to-expand row reveals the full audit context: actor UUID,
  target_user_id / target_type / target_id, and raw details JSON in
  monospaced scroll-capped block. No psql round-trip needed for
  forensic spot checks.
- Sticky thead pinned to the scroll container so column labels survive
  long scrolls. Cells vertical-aligned middle to keep single-line tag
  cells visually balanced against multi-line target cells. No zebra
  stripes — the stacked content already provides row separation, and
  stripes on top read as noise.

Frontend — same polish back-ported to TenantMembers.vue audit drawer:
- Same drawer width, stacked time / actor cells, structured target +
  diff layout, expandable raw-details row, sticky thead, vertical-
  align middle, no stripes. Refresh button reformulated as a text
  button with label (was an outlined square icon-only).
- request_path column kept (rbac.access_denied carries meaningful
  paths) but empty values render as a placeholder dash so they don't
  read as broken.
- Diff line now covers rbac.invitation_sent / invitation_revoked role
  in addition to the existing role_changed / access_denied details.

API:
- frontend/src/api/system/index.ts: listSystemAuditLog() reuses the
  AuditLog / ListAuditLogParams types from @/api/tenant/audit-log
  (re-exported) so consumers don't need to cross-import.

i18n (zh-CN / en-US / ko-KR / ru-RU):
- system.globalSettings.audit.*: full drawer copy + per-action labels
  (system.setting_changed / admin_promoted / admin_revoked) + target
  diff templates + expanded-row labels.
- tenantMember.audit.expanded.*: expanded-row labels added so the
  shared drawer treatment renders cleanly under tenant scope.
2026-05-26 21:13:56 +08:00
wizardchen
9cce0c8e5e feat(system): consolidate system admin and settings into one Settings panel
Replace the standalone /platform/system/* routes with a single
"系统设置" section inside the canonical Settings modal. The previous
SystemLayout.vue / SystemAdmins.vue surfaces are removed and their
functionality (system admin roster, global settings) is hosted directly
in SystemSettings.vue under the standard `.section-header` /
`.settings-group` skeleton. Legacy URLs redirect to the modal section
so external bookmarks don't 404.

Backend:
- SystemSettingService.Reset + repo.Delete: drop the DB override for a
  key so the 3-tier resolver falls back to ENV / built-in default.
  Idempotent (resetting a never-persisted key returns nil); emits an
  audit row only on real deletions, invalidates the local cache, and
  publishes to peers via the existing pubsub channel.
- TenantService.BulkSetStorageQuota: overwrite every tenant's
  storage_quota in one statement. Powers POST /system/admin/tenants/
  apply-default-storage-quota; bypasses the per-tenant whitelist on
  PUT /tenants/:id which intentionally forbids storage_quota edits.
- AuditAction{SystemAdminPromoted,SystemAdminRevoked} constants and
  emitAdminAudit() in SystemHandler — promote / revoke now leave an
  audit trail with TenantID=0 and {target_email,target_username,
  idempotent|changed} in details.
- SystemSetting.LastModifiedByName: derived per-request display label
  (username, email fallback) so the UI shows "wizardchen" instead of
  a UUID prefix without storing a denormalised column.

Frontend:
- SystemSettings.vue rewritten against the Settings modal skeleton with
  auto-persisting controls (switch/select @change, input/inputnumber
  @blur, tag-input + per-delta popconfirm for SSRF whitelist and admin
  roster). auth.registration_mode change goes through an inline
  popconfirm; cancel rolls back. Reset / bulk-apply also inline
  popconfirms — no dialog modal for these per-row affordances.
- Priority hint panel surfaces the DB > ENV > default resolver order
  so operators can reason about "I set the env but it doesn't show up".
- Router: /platform/system, /platform/system/settings, /platform/system/
  admins are now compatibility redirects to /platform/settings?section=
  system-global.
- Settings modal sized 900x700 → 1080x780 and content-wrapper 600 → 760
  so the wider tables (members, system settings) breathe; <1100px
  viewport still flexes to the screen.
- i18n: system.globalSettings.* keys for title / description / loading /
  empty / badges / priority hint / per-key labels / reset / bulkApply /
  admins (label, placeholder, save messages, popconfirm copy) across
  zh-CN, en-US, ko-KR, ru-RU.

Misc:
- internal/utils/filesize.go doc: clarify MAX_FILE_SIZE_MB is a
  deploy-time-only knob (nginx + docreader + frontend each cache the
  env at startup); a SystemAdmin UI override would mislead operators
  because nginx would still 413. Until all four layers can hot-reload
  the limit in lockstep, this stays env-only.
- internal/utils/security.go: SSRF whitelist parser/runtime now drives
  off SystemSettingService for live updates; ENV remains the fallback
  for never-overridden deployments.
2026-05-26 21:13:56 +08:00
wizardchen
311a033523 refactor(migrations): remove deprecated user system admin migration files and introduce new system settings migration
- Deleted the old migration files for user system admin (000052) as they are no longer needed.
- Added new migration files (000053) to implement system-level administrator flag and a new system settings table, consolidating related changes for improved management of platform-wide settings.
- Ensured that the new migrations include proper rollback procedures for both the system admin flag and the system settings table.
2026-05-26 21:13:56 +08:00
wizardchen
554ba9ef74 feat(settings): refactor system settings management and UI enhancements
- Consolidated global system settings into the standard Settings modal, improving accessibility and user experience.
- Updated routing to redirect system settings to the new modal structure, ensuring a seamless transition for users.
- Enhanced UI components for better responsiveness and visual clarity, including adjustments to layout and styling.
- Added visibility controls for system admin-specific settings, ensuring proper access management.
- Refactored related comments and documentation for improved clarity on the new settings structure.
2026-05-26 21:13:56 +08:00
wizardchen
d074dc067a feat(system-admin): implement revocation of system admin privileges with safeguards
- Added RevokeSystemAdmin functionality to the user service and repository, ensuring atomic checks for self-revoke and last admin scenarios.
- Updated the system handler to utilize the new revocation method, improving error handling for various edge cases.
- Enhanced the bootstrap process to prevent unintended promotions when system admins already exist.
- Refactored related comments and documentation for clarity on the new behavior and safeguards in place.
2026-05-26 21:13:56 +08:00
wizardchen
47a183aa65 feat(system-admin): implement bootstrap for system admin promotion and enhance system settings management
- Added WEKNORA_BOOTSTRAP_SYSTEM_ADMIN_EMAIL environment variable to promote a specified user to system admin on startup.
- Introduced a new bootstrap process in `bootstrap.go` to handle the promotion logic.
- Updated `.env.example` to document the new environment variable and its behavior.
- Created new views for managing system administrators and system settings, including listing, promoting, and revoking admin privileges.
- Enhanced the frontend to reflect the new system admin features, including UI elements for admin management and settings configuration.
- Updated API interfaces to support system admin functionalities, ensuring proper data handling and user management.
2026-05-26 21:13:56 +08:00
wizardchen
e6ee87759d fix(agent): preserve agent intent prompt whitespace and add tests
The intent prompt override logic in query_understand applied
strings.TrimSpace to the value before assigning it as the system prompt
override, which silently stripped trailing newlines and intentional
formatting from agent-supplied prompts. Use TrimSpace only to detect
emptiness (so whitespace-only strings still fall back to the global
default) while passing the raw string through verbatim.

Extract the resolution into applyIntentPromptOverride and add unit tests
covering agent-wins, whitespace preservation, blank fallback, no-config,
and global-only paths.
2026-05-26 21:05:40 +08:00
ochan.kwon
11c3236e52 feat(retriever): add OpenSearch driver skeleton + interface stubs (PR 2a of 3)
First half of the gated OpenSearch k-NN driver introduced in PR 1
(#1445) by way of #1440. PR 2a ships a hollow, interface-compliant
shell of the `internal/application/repository/retriever/opensearch/`
package — every behavioural method (Save / BatchSave / DeleteBy* /
Retrieve, plus the previously-stubbed CopyIndices / BatchUpdate* /
EstimateStorageSize / swapToVersion) returns `ErrFeatureNotEnabled`
or a conservative sentinel value. PR 2b lands the real read/write
implementations in dedicated files (`query.go` + `retrieve.go` +
`crud.go`) and replaces the stubs accordingly.

Strict feature-gate (unchanged from PR 1): no entry is added to
validEngineTypes / GetVectorStoreTypes / retrieverEngineMapping /
BuildEnvVectorStores / container env path / engine factory switch,
so the driver remains unreachable. Attempting to register an
`engine_type=opensearch` VectorStore continues to fail with the
existing "not a valid engine type" error.

What lands in PR 2a
-------------------

Driver skeleton (6 production files + 2 test files, ~1170 + ~1115 LoC):

- `repository.go` — Repository struct + NewRepository constructor
  that validates cluster reachability + OS version (2.4+ / 3.x;
  primary tested 3.3.2) + k-NN plugin presence on every cluster
  node. sync.Once-guarded ensureReady(ctx, dim) for lazy per-
  dimension index creation, with transient errors not cached so a
  momentary cluster blip does not permanently poison a dim.
  sanitizeIndexName enforces a strict OS-compatible name spec.
  probeVersion uses robust strings.Split/Atoi parsing for
  pre-release suffixes and missing-patch versions. EngineType
  returns the PR 1 constant; Support returns [keywords, vector].
- `transport.go` — newOpenSearchClient ships TLS posture
  (MinVersion TLS 1.2, opt-in InsecureSkipVerify, forward-secrecy-
  only cipher list) and transport tuning for the driver. Caller
  exists only in PR 3 (container.go + engine_factory.go); PR 2a
  remains gated dead code.
- `mapping.go` — buildIndexMapping(cfg, dim) produces the full
  knn_vector + HNSW + content-analyzer mapping with every *_id
  field as an explicit keyword and source_type as integer.
  buildKeywordsMapping ships the dim-less keyword-only index
  mapping used by the no-embedding save path. createIndexAndAlias
  creates <alias>_v1 and aliases <alias> to it, with best-effort
  orphan cleanup and mapping-drift detection.
- `config.go` — internalCfg (value type) applying OpenSearch
  defaults (hnsw_m=16, ef_construction=100, ef_search=100,
  shards=4, replicas=1, engine=lucene).
- `errors.go` — nine sentinels (ErrIndexNotFound,
  ErrDimensionMismatch, ErrAuth, ErrTransport,
  ErrVersionUnsupported, ErrConfigInvalid, ErrFeatureNotEnabled,
  ErrBatchTooLarge, ErrCircuitBreaker). Repository never imports
  apperrors; PR 3's engine factory wraps these to typed AppError
  2200/2201.
- `stubs.go` — every behavioural method returns
  ErrFeatureNotEnabled. EstimateStorageSize returns a conservative
  HNSW lower-bound estimate (not 0) so the Phase 2 KB-delete guard
  fails-closed for non-empty KBs.

Tests (~1115 LoC, 50 cases):

- `repository_test.go` — interface satisfaction, sentinel mapping,
  sanitizeIndexName positive/negative matrix, semver parsing
  (pre-release / missing-patch), buildIndexMapping JSON shape pin
  (Lucene + Faiss + Keywords), probeVersion matrix (OS 1.x / 2.2 /
  2.5 / 2.11 / 3.x / 3.0.0-rc1 / ES rejection), probeKNNPlugin
  multi-node coverage, ensureReady concurrency + per-dim isolation
  + transient retry, NewRepository storeID validation, all 11
  stubs (CopyIndices, BatchUpdate*, EstimateStorageSize,
  SwapToVersion + Save / BatchSave / Retrieve / DeleteBy*),
  wrapTransport sentinel mapping + leak guard, isNotFound /
  isAlreadyExistsError, drainAndClose / limitedDecode helpers.
- `transport_test.go` — TLS defaults / opt-in InsecureSkipVerify /
  TLS 1.2 pinning / cipher list / transport tuning.

Single dependency addition: github.com/opensearch-project/
opensearch-go/v4 v4.6.0 in go.mod/go.sum.

SDK quirks discovered (opensearch-go v4.6.0)
--------------------------------------------

PR 2a includes the workarounds for two of three SDK limitations
that landed during full implementation (the third, Refresh:*bool,
only affects the delete path that ships in PR 2b):

- AliasExists method passes dataPointer=nil to its internal do(),
  which means non-2xx responses come back as a plain
  *errors.errorString ("status: 404 Not Found") rather than as
  *opensearch.StructError. aliasExists therefore inspects
  resp.StatusCode directly (resp is returned even when err is
  non-nil) and only falls back to wrapTransport for the "no
  response at all" case.
- sync.OnceReset is not in the standard library; the keyword-only
  index uses a mutex + ready/err flag pattern so transient failures
  can be retried by the next caller. The per-dimension path uses
  the `once map[int]*sync.Once` delete-and-recreate trick.

Test fixes folded in
--------------------

While doing a full `go test ./...` against PR 1-merged main, two
deterministic regressions surfaced that block a clean run-everything
signal. Both are unrelated to the driver and are folded into PR 2a
so the PR's own CI run is green:

(1) Follow-up to #1445 — fanout test missed the new normalizer policy
    (internal/application/service/knowledgebase_search_fanout_test.go,
    +46 / -6). #1445 changed EngineAwareNormalizer for ES /
    ElasticFaiss / OpenSearch / Weaviate / Postgres / SQLite /
    Qdrant / TencentVectorDB / Doris from (score+1)/2 to clamp01
    passthrough (those engines surface non-negative cosine to the
    normalizer per Lucene script_score non-negative invariant for
    ES, k-NN plugin SpaceType.COSINESIMIL.scoreTranslation for
    OpenSearch, engine-internal or IR-normalized conversions for
    the rest). Milvus is now the only engine that still surfaces
    raw signed cosine in [-1, 1].

    TestRetrieveFromStores_MixedEngine_Normalizes still asserted
    the old cosine-shift behaviour for ES (raw -0.4 → expected 0.3)
    which under passthrough now becomes clamp01(-0.4) = 0. The
    normalizer's own _test.go was updated at #1445 time, but this
    fan-out integration test was not.

    Fix: rewrite the godoc to spell out the two engine groups;
    restate sub-case 2 as ES passthrough on a production-possible
    mid-range cosine (0.3 → 0.3, PG out-ranks ES); add sub-case 3
    pinning the cosine-shift branch via Milvus -0.4 → 0.3.

(2) Pre-existing — SSRF whitelist singleton race surfaced by this run
    (internal/utils/security.go + internal/utils/security_test.go +
    internal/infrastructure/web_search/searxng_test.go,
    +33 / -9). loadSSRFWhitelist in internal/utils/security.go is
    cached via sync.Once on first call. The internal reset helper
    resetSSRFWhitelistForTest was unexported, so tests in other
    packages could not reset and saw whatever whitelist was cached
    by the first sync.Once.Do() in the same test binary. In
    internal/infrastructure/web_search/, TestValidateProxyURL runs
    before TestValidateSearxngBaseURL alphabetically and exercises
    ValidateURLForSSRF with no SSRF_WHITELIST set, caching an empty
    whitelist; the later setenv in searxng_test then has no effect
    and 127.0.0.1 is rejected with "hostname 127.0.0.1 is restricted".
    Pre-existing on main; surfaced now because this PR was the
    first to do a full `go test ./...` run on top of #1445.

    Fix: capitalize the helper to ResetSSRFWhitelistForTest (the
    ForTest suffix is the test-only contract); update in-package
    callers; in web_search/searxng_test.go import internal/utils
    and call ResetSSRFWhitelistForTest around the env mutation in
    both TestValidateSearxngBaseURL and TestSearxngProvider_Search.
    No production code path changes.

Roadmap
-------

- PR 2b (next, depends on this PR) — read/write implementations:
  query.go + retrieve.go + crud.go land their real bodies; stubs
  for Save / BatchSave / DeleteBy* / Retrieve in stubs.go are
  removed; corresponding CRUD/retrieve/filter test cases (~430
  LoC) join repository_test.go.
- PR 3 — activation switch + async paths (CopyIndices,
  BatchUpdate*, large-batch async deletes) + i18n + docker-compose
  dev profile. After PR 3 merges, the OpenSearch driver becomes
  reachable via either `engine_type=opensearch` VectorStore or
  `RETRIEVE_DRIVER=opensearch` env.

Backward compatibility
----------------------

- New package — additive only. No existing file modified except
  go.mod / go.sum, the two test files in (1)/(2), and the
  test-only export rename in utils/security.go.
- Driver is unreachable: no registry path activates it.
- No SQL migration.
- The PR 1 normalizer case for OpenSearch remains unreachable
  here (no driver instance produces a result yet).

Test plan
---------

- [x] go build ./... clean
- [x] go vet ./... clean
- [x] go test -race -count=1 ./internal/application/repository/retriever/opensearch/... passes
- [x] grep -r "case types.OpenSearchRetrieverEngineType" internal/
      shows only PR 1's normalizer case + this driver's EngineType()
      and tests — no activation path.
- [x] grep -r "case \"opensearch\"" internal/ shows no hits.
2026-05-26 20:54:58 +08:00
liuwei435
e0c6599c54 fix(datasource): support Yuque team token in connector
- Add Type field to v2User struct to distinguish personal/team tokens
- Route team tokens directly to ListGroupRepos (skip ListUserGroups)
- Gracefully handle ListUserGroups 404 for personal tokens without teams
- Add rate-limiting delay between GetDocDetail calls to avoid API throttling
2026-05-26 20:46:22 +08:00
ChenRussell
80a69ae6f3 fix(pipeline): fallback to raw retrieval results when rerank API fails
When the rerank model returns an error (e.g. 401 Unauthorized), the
  pipeline previously discarded all retrieved candidates and returned
  empty results to the caller.

  Now p.rerank returns ([]RankResult, error) to distinguish API failure
  from threshold-filtered empty results. On API error, the pipeline
  falls back to the original retrieval candidates (directLoad +
  candidatesToRerank) and continues to CHUNK_MERGE/FILTER_TOP_K,
  so users still get results even when the rerank model is misconfigured.
2026-05-26 20:45:17 +08:00
helloandyzhang
e6a469631b feat(agent): add intent prompt customization in agent editor
Allow per-intent system prompt overrides for non-retrieval intents in normal mode.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-26 20:41:10 +08:00
Miles Lai
09892ef763 fix(doris): inline LIMIT/OFFSET as literals and enable parameter interpolation
Problem:
Doris does not support ? placeholders for LIMIT and OFFSET clauses, causing
VectorRetrieve, KeywordsRetrieve, and CopyIndices queries to fail at runtime
with SQL parse errors.

Root cause:
1. Query builders passed TopK/pageSize/offset as ? bind parameters (valid in
   MySQL/PostgreSQL but not in Doris's SQL dialect)
2. Doris MySQL driver was not configured to interpolate parameters, preventing
   automatic client-side placeholder replacement

Fix:
1. Enable parameter interpolation in Doris connection DSN:
   - Add &interpolateParams=true to mysql.Open() DSN in container.go
   - This enables driver-level parameter substitution at query time

2. Inline LIMIT/OFFSET as string literals in Doris queries:
   - VectorRetrieve: LIMIT ? → LIMIT %d (params.TopK)
   - KeywordsRetrieve: LIMIT ? → LIMIT %d (params.TopK)
   - CopyIndices: LIMIT ? OFFSET ? → LIMIT %d OFFSET %d (pageSize, offset)
   - Remove inlined values from args slice

Result:
Queries are now built with literal LIMIT/OFFSET values instead of placeholders,
compatible with Doris's SQL parser.
2026-05-26 20:32:49 +08:00
jackson.jia
dbdae65e9b fix(input-field): wrong toast when selecting built-in agents other than quick-answer / smart-reasoning
When selecting an agent from the chat page's agent selector dropdown,
built-in agents other than `builtin-quick-answer` and
`builtin-smart-reasoning` trigger the wrong toast. "Switched to
Intelligent Reasoning" pops up regardless of which agent was actually
picked — Wiki Questioner, Data Analyst, Wiki Fixer, etc.

Root cause: the toast logic checks only `agent.is_builtin`, then
branches on `agent_mode === 'smart-reasoning'` to pick between the
"Switched to Intelligent Reasoning" / "Switched to Quick Q&A" strings.
Those strings were designed for the two original built-ins
(`builtin-quick-answer` and `builtin-smart-reasoning`), which the
dropdown re-brands as "Normal Mode" / "Agent Mode". They don't make
sense for any other built-in.

Narrow the special-case branch to those two IDs. Other built-ins fall
back to the generic `agentSelected` toast (the path custom agents
already take), so they get a toast naming the agent that was actually
selected.
2026-05-26 20:26:31 +08:00
helloandyzhang
af2f55c29f fix(frontend): apply default context template when switching to quick-answer mode
Start new agents with an empty context_template so mode-switch logic can
populate the system default instead of keeping the smart-reasoning placeholder.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-26 20:13:31 +08:00
wizardchen
500c821817 feat(builtin-models): validate YAML entries and align ID length with schema
Three correctness fixes that the lifecycle PR deliberately deferred:

1. ID length / struct-tag drift
   - models.id is varchar(64) on both PG and SQLite (per the init
     migrations), but Model.ID's GORM tag said varchar(36) — a remnant
     from when the field only held UUIDs. The mismatch is harmless under
     golang-migrate (struct tag is ignored), but misleading on AutoMigrate
     paths and in IDE tooltips. Tag now matches the real column width.
   - New ModelIDMaxLen constant (=64) is the single source of truth for
     anyone accepting user-provided ids. The YAML loader uses it to
     reject too-long ids up front with a clear message instead of letting
     the INSERT explode with a generic "value too long for type" error.

2. Field validation in the YAML loader
   - Type, Source, and Status are typed strings but YAML can supply any
     value. Misspellings (e.g. `type: knowledgeqa` lowercase, `type: LLM`)
     were previously persisted as-is and produced rows that looked fine
     in the table but failed at provider-factory lookup time, which is
     hard to debug.
   - validateBuiltinModelEntry now checks: empty id, id length, empty
     type, type ∈ {KnowledgeQA, Embedding, Rerank, VLLM, ASR}, and
     status ∈ {active, downloading, download_failed, empty}. Source is
     intentionally NOT validated because the provider matrix in
     internal/models/* keeps growing and a strict allow-list here would
     force changes in two places per new provider.
   - Invalid entries are warned + skipped (not aborting the whole load),
     and excluded from the keep-set so the drift sweep does not delete
     existing matching rows on the strength of a typo'd YAML retry.

3. Magic number cleanup
   - DefaultBuiltinModelTenantID (=10000) replaces the hard-coded `10000`
     literal in toModel(). The invariant lives in three places already
     (PG migration, SQLite migration, this constant); naming it makes
     the cross-reference explicit and grep-able.

Tests:
- New TestLoadBuiltinModelsConfig_RejectsInvalidEntries with five
  sub-cases (id-too-long, missing-type, lowercase-type, unknown-type,
  unknown-status) asserts the table stays empty after each.
- All 11 existing tests still pass.
2026-05-26 11:37:03 +08:00
wizardchen
4bce10d1ea refactor(builtin-models): unify logging via stdlib log with stable prefix
Original PR #1453 used fmt.Printf, which lands as unstructured noise in
release/JSON log pipelines. The natural fix is logger.Infof/Warnf, but
internal/logger itself imports internal/types — using it from here
would create an import cycle (this is the same constraint that forces
model.go's crypto error logs onto stdlib log).

Switch all loader output to log.Printf with a stable "[builtin-models]"
prefix (mirroring the "[crypto]" convention already established in
model.go). The prefix gives operators a grep handle even though the
lines stay unstructured.

Levels are encoded inline as "WARN:" for warnings so log shippers and
humans can still discriminate.
2026-05-26 11:37:03 +08:00
wizardchen
ea47dce337 fix(compose): use env_file array form for builtin_models compatibility
PR #1453 introduced the map form of env_file:

  env_file:
    - path: .env
      required: false

which is only recognised by Docker Compose v2.24+ (Jan 2024). Older
deploys would refuse to parse docker-compose.yml even when builtin_models
YAML is not enabled — breaking the "no impact unless opted in" promise.

Switch to the array form (`env_file: [.env]`) which has been supported
since the earliest Compose schemas. The trade-off is that Compose now
errors when .env is absent, so:

- scripts/start_all.sh already calls check_env_file (cp .env.example .env
  if missing) before docker compose up; that path is unaffected.
- The bare `make docker-run` / `make docker-restart` targets in the
  Makefile are taught the same fallback: touch / copy .env.example before
  invoking docker-compose, so fresh clones keep working.

The docker-compose.yml comment block is updated to explain the version
trade-off so future maintainers don't re-introduce the map form.
2026-05-26 11:37:03 +08:00
wizardchen
7b192e546f feat(builtin-models): reconcile YAML lifecycle with drift sweep
Extend the builtin_models.yaml loader so the YAML file becomes a complete
source of truth for the rows it owns. Builds on the previous commit's
managed_by column.

Lifecycle contract:
- Every UPSERTed entry is tagged managed_by="yaml".
- The DoUpdates list now includes deleted_at, so an entry that was
  soft-deleted (e.g. via UI/API) is automatically resurrected when it
  reappears in the file. Closes the "ghost row that exists but is
  invisible" failure mode.
- After all UPSERTs, the loader soft-deletes rows where
  managed_by='yaml' AND id NOT IN (current YAML id set). Removing an
  entry from YAML is now the supported way to retire a built-in model —
  no manual SQL needed.
- Rows tagged managed_by='' (UI/API/SQL-seeded built-ins) are invisible
  to the reconcile path and never touched.
- When a YAML entry sets is_default=true, the loader first clears
  is_default on any other rows in the same (tenant_id, type) bucket,
  mirroring the invariant enforced by the API path
  (repository.UnsetDefaultModel).

Failure handling stays defensive:
- File missing / not a regular file / parse error: warn and skip; the
  drift sweep is NOT executed so a malformed file cannot wipe rows.
- Per-entry UPSERT error: warn, drop the id from the keep-set so the
  sweep also leaves the existing row alone ("leave alone on failure").

Tests cover: file-missing, parse-error, basic upsert + defaults,
idempotency, ${ENV} interpolation (set vs unset), drift sweep removing
YAML rows, drift sweep ignoring manual rows, soft-delete resurrection,
is_default cleanup across tenant+type, explicit empty list sweeping all
yaml-managed rows, and a regression guard ensuring BeforeCreate does not
overwrite YAML-supplied stable ids.

Docs are rewritten so operators see "delete from YAML and restart" as
the supported removal path; SQL is retained only for the legacy
managed_by='' slice.
2026-05-26 11:37:03 +08:00
wizardchen
fdc22fd7a5 feat(builtin-models): add managed_by column to models table
Introduce a `managed_by` varchar column on `models` so future declarative
loaders can claim ownership of a subset of rows without disturbing entries
created via the UI/API or seeded by hand-written SQL.

- versioned/000052_models_managed_by.{up,down}.sql add the column with a
  default of '' and a partial index on non-empty values to keep startup
  reconciliation cheap.
- sqlite/000000_init.up.sql is updated in place (the Lite init migration
  is a single file per project convention) so fresh SQLite databases get
  the column from the start.
- Model.ManagedBy mirrors the column. Existing rows default to '' which
  the YAML loader treats as "manually managed, never touch".

Schema half of the YAML-driven built-in-model lifecycle work that follows
up on #1453; the reconciler that uses the column lands in the next commit.
2026-05-26 11:37:03 +08:00
jackson.jia
d439b3ae07 feat(builtin-models): add YAML-based declarative config with ${ENV} interpolation
Allow built-in models to be declared in config/builtin_models.yaml
instead of inserting rows via SQL. On every startup the file is read
and each entry is UPSERT-ed into the models table (is_builtin=true)
by stable id.

Any string field may reference an environment variable with ${NAME}.
Unset variables are left as the literal placeholder so
misconfiguration surfaces clearly in provider calls rather than
failing silently with an empty token.

The file is optional: missing file, parse errors, and per-entry
upsert failures all log a warning without aborting startup.
docker-compose.yml adds env_file (.env, required:false) so
deployment-specific variables are passed through automatically.
2026-05-26 11:31:01 +08:00
wizardchen
e9be53e830 fix: respect multi-turn-disabled flag in KnowledgeQA pipeline
When a custom agent has `MultiTurnEnabled=false`, `applyAgentOverridesToChatManage`
sets `chatManage.MaxRounds = 0` to signal "no history". Two pipeline plugins
mistreated this zero value as "use the global default" and silently re-loaded
session history into the LLM context:

- `PluginLoadHistory` fell back to `Conversation.MaxRounds` when
  `chatManage.MaxRounds == 0`.
- `PluginQueryUnderstand.loadHistory` had the same fallback, and even when
  `LOAD_HISTORY` was skipped it would re-populate `chatManage.History`,
  leaking previous turns into rewrite, image analysis, and the final answer.

The RAG branch in `session_knowledge_qa` also added `LOAD_HISTORY`
unconditionally, unlike the pure-chat branch which guarded it with `hasHistory`.

This change:

- Treats `chatManage.MaxRounds <= 0` as an explicit disable in both plugins;
  no fallback to global config.
- Makes the RAG pipeline consistent with the pure-chat path by gating
  `LOAD_HISTORY` on `hasHistory`.
- Removes the duplicated `Current Time: {{current_time}}` line from
  `agent_system_prompt.yaml`. The agent already receives a fresh
  `<runtime_context><current_time>` block with each turn from
  `observe.buildRuntimeContextBlock`, so the static placeholder was
  redundant.

The ReAct agent path (`session_agent_qa`) already checked `MultiTurnEnabled`
directly and is not affected.

Closes #1479
2026-05-26 10:45:42 +08:00
wizardchen
91e4fe8b6b feat(frontend): introduce KBSwitcherDropdown component for improved knowledge base selection
Added a new KBSwitcherDropdown component to enhance the user experience when selecting knowledge bases. This component displays a list of knowledge bases with the current selection highlighted, allowing users to easily switch between them. Integrated the dropdown into both KnowledgeBase.vue and FAQEntryManager.vue, replacing the previous dropdown implementation for a more streamlined and consistent UI. The new component supports dynamic updates and maintains the order of knowledge bases while ensuring the current selection is always visible at the top.
2026-05-26 10:20:56 +08:00
wizardchen
257766545e style(frontend): refine KBInfoPopover button styles for improved visual hierarchy
Updated the styles for the info button in KBInfoPopover.vue to enhance its appearance and interaction. Adjusted dimensions, colors, and hover effects to create a clearer distinction between the info button and the settings button. Added a warning state for better visual feedback when necessary. These changes aim to improve user experience and maintain consistency in the UI design.
2026-05-26 10:20:56 +08:00
wizardchen
baae7dd806 fix(frontend): update KBInfoPopover to correctly display FAQ and document counts
Refactor the logic in KBInfoPopover.vue to differentiate between FAQ and document knowledge bases. The component now uses `chunk_count` for FAQ KBs and retains `knowledge_count` for document KBs, ensuring accurate representation of entry totals. This change aligns the display logic with the intended user-facing metrics for each KB type.
2026-05-26 10:20:56 +08:00
wizardchen
f97d588042 refactor(frontend): extract KBInfoPopover and reuse it on FAQ KBs
Move the info popover (intro'd in the previous commit) into a
dedicated `frontend/src/components/KBInfoPopover.vue` component so
both the document KB header and the FAQ KB header can share the
exact same metadata surface.

- KnowledgeBase.vue: replace the inline popover and the now-dead
  helper computeds (kbLastUpdated, kbHasDistinctUpdate,
  infoCardCapabilities, infoCardChunking, infoCardStats,
  chunkingStrategyLabel, supportedFileTypesList, accessRoleLabel,
  accessPermissionSummary) and their CSS with a single
  <KBInfoPopover> tag. The page still fetches parser engines and
  forwards the supported file types so the "Accepted formats" row
  renders for document KBs.
- FAQEntryManager.vue: drop the legacy inline meta strip
  (role tag · share source · last updated) along with the
  accessRoleLabel / accessPermissionSummary / kbLastUpdated /
  effectiveKBPermission computeds it only served, and wire the
  same <KBInfoPopover> into the actions cluster next to the
  settings button. FAQ KBs don't pass supportedFileTypes so the
  format row simply hides itself.
- KBInfoPopover.vue: self-contained — re-derives access role,
  shared-binding and chunking config from kbInfo + the org/auth
  stores. supported-file-types remains an opt-in prop so callers
  decide whether the parser-engine pipeline is relevant.

No user-visible regressions: both pages now render the same
Basic / Access / Capabilities / Chunking / Stats / Storage
Binding sections, with rows hiding when they don't apply.
2026-05-26 10:20:56 +08:00
wizardchen
de8ee5ae16 feat(frontend): consolidate KB metadata into an info popover
Replace the inline meta strip on the knowledge base detail header
(role tag · share source · last updated · vector store badge) and
the vector-store engine badge on the list cards with a single
structured "info" popover next to the settings button.

The popover groups Basic, Access, Capabilities, Chunking, Stats
and Storage Binding into separate sections, only rendering those
with data. A warning dot on the info button surfaces unreachable
vector-store bindings without occupying header real estate. The
popover caps at min(70vh, 560px) with an internal scrolling body
so it never overflows the viewport.

Hide the last-updated row when it equals created_at, since GORM
only bumps KB.updated_at on metadata edits — never on document
uploads — so surfacing two identical timestamps next to "created
at" was misleading. Once the KB metadata is actually edited the
row reappears naturally.

i18n entries added for zh-CN, en-US, ko-KR, ru-RU.
2026-05-26 10:20:56 +08:00
wizardchen
31471f5efa refactor(knowledgeService): update checkStorageEngineConfigured method for improved clarity and functionality
- Refactored checkStorageEngineConfigured to be a method of knowledgeService, enhancing encapsulation and readability.
- Updated logic to allow fallback to global file service when no storage provider is configured at the KB or tenant level, improving error handling.
- Added detailed comments to clarify the method's behavior and internal logic, ensuring better understanding for future maintenance.
2026-05-25 19:16:08 +08:00
wizardchen
caeb6a44d8 refactor(chunk): remove VideoInfo field from Chunk struct
- Removed the VideoInfo field from the Chunk struct in chunk.go, streamlining the data model.
- This change reflects a shift in focus away from video information storage within the Chunk type.
2026-05-25 19:15:39 +08:00
wizardchen
13301ca026 feat(parser): enhance web parser with improved image extension handling and XPath prioritization
- Added a regex pattern for image file extensions to the utils module for better image detection.
- Updated the BODY_XPATH in the xpaths module to prioritize matching specific content structures in web pages.
- These changes aim to improve the accuracy and efficiency of content extraction from web pages using the StdWebParser class.
2026-05-25 19:15:17 +08:00
begoniezhao
5c0243cd92 feat: Support filtering chunks by multiple chunk_type params 2026-05-25 19:14:46 +08:00
wizardchen
8dc50b3921 fix(frontend): close card popover before opening delete confirm dialog
The per-card "more actions" popover (edit / rebuild / move / delete ...)
stayed open when clicking "Delete document", so the popover floated above
the t-dialog and visually obscured the confirmation dialog.

Reset the card's isMore flag and moreIndex inside delCard before toggling
delDialog so the popover closes synchronously with the dialog opening.
2026-05-25 17:11:20 +08:00
wizardchen
15845f4e62 fix(frontend): poll knowledge list after single-item delete
DeleteKnowledge is now an async pipeline on the backend, so refreshing the
list immediately after a successful response can still show the deleted
entry. Mirror the polling loop already used for batch delete: re-list up to
~12s (30 polls * 400ms) until the deleted id disappears, then refresh tags.

Without this the UI shows "delete success" but the list visibly contains
the entry until the user manually refreshes.
2026-05-25 17:11:20 +08:00
wizardchen
d12fb42e60 refactor(knowledge): route single delete through async pipeline
Reuse enqueueKnowledgeListDelete inside DeleteKnowledge so that single-item
delete shares the same hardening as BatchDeleteKnowledge / ClearKnowledgeBase:
asynq retries, business-aware queue routing, and marking-as-deleting inside
the worker.

The endpoint now returns 200 once the delete task has been enqueued; the
response body carries the asynq task_id and the message is updated to
"Delete task submitted". Swagger annotations, generated docs and the Go
client SDK comment are updated to reflect the new asynchronous semantics.

Note: this is a behavior change. Callers that previously assumed the
knowledge was already gone on a 200 response should poll the task status
or accept eventual consistency, matching the existing BatchDeleteKnowledge
contract.
2026-05-25 17:11:20 +08:00
chenwenhui
af1f2b469a 增加sandbox对windows编译支持,现在默认是linux的实现,windows直接编译报错 2026-05-25 16:57:56 +08:00
wolfkill
6331cce23d fix: skip empty milvus enabled status groups 2026-05-25 16:54:56 +08:00
cn-kali-team
fdb53ff92a fix type 2026-05-25 16:50:17 +08:00
young1lin
29820e4cac docs(chat): clarify cached-token semantics for explicit-cache providers
`cached_tokens` is reported by every OpenAI-compatible provider that
supports prompt caching, but how it becomes non-zero differs by mode:

- Implicit caching (OpenAI, Azure OpenAI, DeepSeek, …) populates the
  field automatically whenever a prompt prefix matches a previous
  request within the provider's cache TTL. No client-side opt-in.

- Explicit caching (Qwen on Aliyun, Anthropic Claude, …) only
  populates the field after the caller attaches `cache_control:
  {"type": "ephemeral"}` to the relevant message / content block.
  Until that opt-in is applied upstream of the request, the field
  stays zero even when the prefix is otherwise byte-stable.

Without this distinction documented, the previous commit reads as if
`TokenUsage.CachedTokens` will show non-zero values for Qwen / Claude
once this PR lands — which is not the case. The plumbing here is a
prerequisite (stable prefix via sorted tools) and a meter (visibility
of the field), but the explicit-cache opt-in itself is out of scope
and lives elsewhere.

Document this on `TokenUsage.CachedTokens` and the `cachedTokens`
helper so callers do not mistake observability for activation.
2026-05-25 16:47:14 +08:00
young1lin
582f7b3056 feat(chat): surface cached prompt tokens from upstream usage
OpenAI-compatible providers (Qwen, DeepSeek, OpenAI, Azure, etc.) report
prompt-cache hits in `usage.prompt_tokens_details.cached_tokens`. This
value was being read by go-openai but dropped at the WeKnora boundary,
so there was no way to tell whether prompt caching was actually working.

This change plumbs the field end-to-end:

- `types.TokenUsage.CachedTokens` (json:"cached_tokens,omitempty") —
  zero-values are omitted so payloads stay quiet for providers that
  never report cache hits.
- `cachedTokens` helper in remote_api.go guards against nil
  `PromptTokensDetails` (Ollama and older OpenAI-compat backends omit
  the details block entirely).
- All three response-parsing paths populate it:
    * `parseCompletionResponse` (non-streaming)
    * `processStream` (SDK streaming)
    * `processRawHTTPStream` (raw-HTTP streaming, used when callers
      need to inject custom fields like `cache_control`)
- The five `[LLM Usage]` log lines now print `cached_tokens=%d` so
  cache hit rate is visible in `journalctl` / log tail without going
  through metrics.

Together with the deterministic tool ordering from the previous commit,
this makes Qwen explicit caching observable: a warmed prefix should
show `cached_tokens` ≈ system + tools token count (typically several
thousand) on subsequent requests within the 5-minute TTL.

Tests:
- `TestCachedTokensHelper` — nil safety + round-trip
- `TestParseCompletionResponse_CachedTokens` — populated + missing
  details paths through `parseCompletionResponse`
- `TestTokenUsage_CachedTokensJSONOmitempty` — zero is omitted,
  non-zero is emitted
2026-05-25 16:47:14 +08:00
young1lin
29bec4204a fix(agent/tools): sort function definitions for deterministic ordering
`ToolRegistry.GetFunctionDefinitions` and `ListTools` previously ranged
over the internal map directly. Go map iteration is intentionally
randomized, so the resulting `tools` array reshuffled on every request.

That reshuffling silently breaks provider-side prompt caches that key on
a byte-level prefix match — most visibly Qwen explicit caching, which
requires the messages (system + tools + history) to be byte-identical up
to the `cache_control` marker. With random ordering the serialized tools
block changes every call, so the cache prefix never matches and the
hit rate stays at 0%.

Sort by tool name in both functions. Output is now byte-stable across
calls and `cache_control: ephemeral` can actually take effect.

Tests in registry_test.go cover:
- Deterministic ordering across 50 iterations
- JSON byte-stability across 20 iterations (the real motivation)
- Field projection (Name / Description / Parameters)
- Empty registry returns `[]` not `null`
- ListTools sorting
- First-wins duplicate registration policy (GHSA-67q9-58vj-32qx)
2026-05-25 16:47:14 +08:00
wizardchen
30332cca99 fix(frontend): correct floating UI positioning under root zoom
The user font-size preference applies CSS `zoom` on `<html>`
(see `composables/useFont.ts`). This breaks every popover that
anchors itself with `getBoundingClientRect()` + `position: fixed/absolute`,
because the rect is in visual pixels while CSS lengths are then
multiplied by zoom again — shifting popovers toward the lower-right
and miscomputing maxHeight / width / viewport-edge clamping.

PR #1459 patched only `AgentSelector.vue` (and left its width/maxHeight
unfixed). This PR introduces a shared helper `frontend/src/utils/zoom.ts`
(`getRootZoom`, `rectToCssPx`, `cssViewportSize`) and applies it to every
affected popover:

- `AgentSelector.vue` — refactored to use the helper; also normalizes
  `width` and `maxHeight` (previously still in visual px).
- `Input-field.vue` — model dropdown, mention popup (typed @ and button-
  triggered), and agent-mode dropdown.
- `KnowledgeBaseSelector.vue` — primary anchor path and fallback path.
- `UserMenu.vue` — tenant and IM floating submenus + viewport clamp.
- `FAQTagTooltip.vue` — fixed tooltip positioning + edge clamp.
- `AgentEditorModal.vue` — placeholder popups for system_prompt,
  context_template, and rewrite/fallback prompts.
- `AgentStreamDisplay.vue` — `kb-float-popup` (absolute under body, also
  under the zoom containing block).

Verified with `npm --prefix frontend run build`.
2026-05-24 23:03:26 +08:00
shendong
a02d6d06b9 fix: correct agent selector position under zoom 2026-05-24 22:44:18 +08:00
wizardchen
a469ee6a36 feat(wiki): add "View in Graph" entry on wiki page
Add a link in the wiki page header that navigates to the graph tab
with the page slug pre-selected, so users can jump from a wiki page
to its corresponding graph node in one click.
2026-05-22 21:02:01 +08:00
ochan.kwon
744a367f16 feat(retriever): add OpenSearch type prep ahead of Phase 3 driver
Lays type-system groundwork for the upcoming OpenSearch k-NN driver
(Phase 3 PR 2, see Tencent/WeKnora#1440), with strict feature-gate:
this PR ships only inert constants, schema extensions, and an
unreachable normalizer case. No path activates an OpenSearch
VectorStore yet — creation continues to fail with "not a valid engine
type" until the activation switch lands in Phase 3 PR 3.

Changes:

- OpenSearchRetrieverEngineType constant ("opensearch") in
  internal/types/retriever.go. Not added to validEngineTypes /
  GetVectorStoreTypes / retrieverEngineMapping yet (gated).

- ConnectionConfig.InsecureSkipVerify (bool, default false) in
  internal/types/vectorstore.go, placed inside the // Common section
  because it is a cross-driver TLS option. Distinct from the
  Qdrant-specific UseTLS, which enables TLS on gRPC — InsecureSkipVerify
  only controls verification of an already-TLS HTTPS connection.
  AES-GCM Value/Scan round-trips the field as plaintext (it travels
  alongside the encrypted Password / APIKey but is not sensitive itself).

- VectorStoreFieldInfo gains four optional fields: Immutable (bool),
  Min/Max (*float64), Enum ([]string). All omitempty so existing UI
  schema entries serialize identically. The fields will drive the UI
  in Phase 3 PR 3 (read-only on Edit + range/enum constraints).

- Six new AuditAction constants under vector_store.* and opensearch.*
  namespaces. Definitions only — emission lands in Phase 3 PR 3.

- EngineAwareNormalizer.Normalize is restructured to group engines by
  the effective score range observed by the normalizer (not the
  theoretical raw cosine range):

    Range [-1, 1] (raw cosine, mapped via (score + 1) / 2):
      - Milvus (COSINE metric mode). Milvus docs explicitly state the
        COSINE metric range is [-1, 1].

    Range [0, 1] (passthrough — already on the target scale):
      - Elasticsearch v8 / ElasticFaiss. The driver issues a
        cosineSimilarity(...) script_score script, and Lucene rejects
        negative final scores ("Final relevance scores from the
        script_score query cannot be negative" — ES docs); the
        effective range observed by the normalizer is therefore [0, 1]
        for IR-normalized embeddings. ES was previously grouped with
        Milvus and over-corrected via (score + 1) / 2, which inflated
        every ES result by 50% in mixed-engine RRF fusion.
      - OpenSearch. The k-NN plugin's
        SpaceType.COSINESIMIL.scoreTranslation maps the underlying
        Lucene/Faiss distance (1 - cosine) to (1 + cosine) / 2 ∈ [0, 1]
        before the score reaches us. Source:
        github.com/opensearch-project/k-NN at
        src/main/java/org/opensearch/knn/index/SpaceType.java
        (COSINESIMIL enum, scoreTranslation method).
      - Weaviate. The driver requests `certainty`, defined by Weaviate
        as (2 - distance) / 2 = (1 + cosine) / 2, intrinsically [0, 1].
      - Postgres pgvector / SQLite sqlite-vec / Qdrant /
        TencentVectorDB / Doris. The driver computes (1 - cosine_distance)
        or normalized inner_product, whose theoretical range is [-1, 1].
        The IR-normalized positive-component unit vectors WeKnora
        targets (BGE / OpenAI text-embedding-3 / Cohere /
        sentence-transformers) keep the observed range in [0, 1];
        negative-cosine embedding models would silently clamp to 0
        downstream — explicitly documented as the IR-normalization
        caveat in the struct godoc.

  Dead enum references (ElasticFaiss, Infinity) are flagged in the
  godoc with a pointer to internal/types/vectorstore.go's existing
  "legacy/experimental, no standalone deployable instance" annotation.
  Their case labels are kept for switch exhaustiveness.

Test coverage:

- OpenSearch constant wire value + collision check against existing
  10 engines + gated invariant (NOT in validEngineTypes).
- ConnectionConfig.InsecureSkipVerify backward-compat (missing JSON
  field deserializes as false) + round-trip + AES-GCM coexistence
  with encrypted Password / APIKey.
- VectorStoreFieldInfo omitempty preservation + new-field serialization
  + *float64 pointer distinction (min=0 vs nil).
- AuditAction dot-namespace convention enforcement, prefix invariants
  for vector_store.* and opensearch.*, no wire-string collisions, exact
  wire-value pins for the six new constants.
- EngineAwareNormalizer:
  - CosineRange retains its (score + 1) / 2 coverage but now only for
    Milvus.
  - UnitInterval now covers the full passthrough group (ES /
    ElasticFaiss / OpenSearch / Weaviate / Postgres / SQLite / Qdrant /
    Infinity / TencentVectorDB / Doris).
  - New TestEngineAwareNormalizer_ElasticsearchCosinePassthrough is an
    explicit regression guard for the score-range correction: cos=0.5
    maps to 0.5 (not (0.5 + 1) / 2 = 0.75 as an earlier draft assumed).
  - OpenSearch passthrough across (0 / 0.5 / 0.75 / 1) + engine drift
    (1.0001) + defensive negative + ±Inf / NaN edges + keyword
    passthrough + nil-ctx safety.

Backward compatibility is preserved at every layer:

- All new struct fields are omitempty / pointer-tagged so existing
  rows and existing wire formats remain unchanged.
- Normalizer's new OpenSearch case is unreachable until the driver
  lands. The ES regrouping changes the post-normalization value for
  every ES vector search result (a correctness fix, not a feature) —
  ES vector retrieval is currently the only production path affected.
- AuditAction constants emit no audit_log rows in this PR.
- engine_type=opensearch VectorStore creation still rejected (gated).
2026-05-22 20:43:50 +08:00
yuheng.huang
3d5b4c16fe refactor(logger): support LOG_FORMAT template and harden level coloring 2026-05-22 20:31:54 +08:00
yuheng.huang
39c9985c3b refactor(logger): support LOG_FORMAT template and harden level coloring
- Add CustomFormatter.Template driven by LOG_FORMAT env var with
  placeholders %d/%level/%thread/%logger/%traceId/%msg; default
  format unchanged for backward compatibility.
- Replace chained ReplaceAll with strings.NewReplacer for single-pass
  substitution, avoiding accidental re-substitution when a field value
  contains a literal placeholder string.
- Inject ANSI color at the %level substitution step; removes the old
  whole-line ReplaceAll(line, "INFO", ...) which would mis-color
  literal level tokens appearing inside messages.
- Cache threadNeeded on the formatter so runtime.Stack only runs when
  the template references %thread.
2026-05-22 20:31:54 +08:00
wizardchen
bccc27b162 fix(frontend): keep X-Tenant-ID override when switching back to home
After re-login, the JWT is minted with the user's last-active tenant
(see userService.resolveLoginTenantID). Both TenantSelector and
UserMenu used to clear weknora_selected_tenant_id when "switching to
home", which made request.ts stop attaching X-Tenant-ID. Without the
header, the auth middleware fell back to the JWT-encoded tenant id —
i.e. the peer tenant the user just tried to leave — so the switch was
silently a no-op until the user manually wiped localStorage and
re-logged in.

Always write the active tenant id into selectedTenantId so the
"always attach X-Tenant-ID" invariant request.ts relies on stays
true. The server-side last_active_tenant_id preference is still
cleared when switching to home, so a clean re-login still lands the
user on home.

Also fix TenantSelector.defaultTenantId to read user.tenant_id (the
immutable home id) instead of authStore.tenant.id (the active tenant,
which is overwritten by /auth/me). This matches the contract spelled
out in useHomeTenant() and prevents switchingToHome from misfiring
when the active tenant differs from home.
2026-05-22 18:20:23 +08:00
ochan.kwon
8b7d00b260 feat: expose KB ↔ vector store binding in list, editor, and detail UI
Backend
-------
ListKnowledgeBases now enriches each row with the resolved vector_store
metadata (name / source / engine_type / status) via a new
buildKBListResponse helper. Store views are batch-resolved once per
request through BatchResolveStoreView so an N-KB list costs one
vector-store service call rather than N — closing the N+1 limitation
flagged in #1372's known-limitations section. Cross-tenant shared KBs
continue to render via SharedStoreDisplay so the owning tenant's store
inventory cannot be correlated across rows; the underlying vector_store_id
UUID is stripped from those responses.

Resolver failures degrade gracefully: bound KBs render as unavailable
instead of breaking the list. Test coverage pins the env / bound /
shared distinction, the batch-call-count invariant, and the
graceful-failure path.

Frontend
--------
KB editor modal gains a new "Vector Store" section. Create mode shows a
dropdown that combines the system default (env store) and the tenant's
configured user stores, fetched once at mount via the existing
listVectorStores API. Edit mode shows the bound store read-only via a
new VectorStoreBadge component with an explicit immutability hint —
matching the backend's `<-:create` GORM tag and the service-layer
UpdateKnowledgeBaseRequest DTO that already omit the field.

KB list cards surface a small engine-type badge for own-tenant bound
KBs, and a warning badge when the bound store is unavailable. Env-bound
and shared KBs render no badge (visual noise control). KB detail header
shows the bound store via the same VectorStoreBadge component; shared
KBs fall through to the badge's internal "shared" branch with no name /
engine / id rendered.

The KB editor's create-time error handler translates the typed
ErrVectorStoreBindingInvalid (2200) and ErrVectorStoreUnavailable
(2201) into localized messages and jumps the user back to the
VectorStore section so they can pick a different store or fall back to
the system default.

The KB row type gains five optional fields (vector_store_id / name /
engine_type / source / status). i18n: 18 new keys added to en-US,
ko-KR, zh-CN; ru-RU receives English placeholders pending translation
(consistent with prior PRs in this locale).

Part of #993 (Phase 2: Per-KB VectorStore Binding).
Phase 2 roadmap item: PR 5 (KB binding UI + list-response enrichment).
Depends on #994, #1310, #1372, #1386 (all backend in the Phase 2 series).
2026-05-22 17:40:10 +08:00