diff --git a/docs/references/data/database-construction.md b/docs/references/data/database-construction.md index e6d85615140..3b1114f5f57 100644 --- a/docs/references/data/database-construction.md +++ b/docs/references/data/database-construction.md @@ -100,7 +100,7 @@ A table rebuild (drizzle's `INSERT…SELECT` drops the implicit rowid) **and `VA ### Knowledge `search_text_fts` follows the same rule -`src/main/features/knowledge/vectorstore/indexStore/schema.ts` keys `search_text_fts` on a stable `fts_rowid` column too (assigned by the `search_text_ai` trigger; `content_rowid='fts_rowid'`). It is a **separate per-base `index.sqlite`** (not the main DB, not drizzle-managed, not in `CUSTOM_SQL_STATEMENTS`), but the same hazard applies: its `reclaim()` path runs `VACUUM` to return freed pages to the OS after a large delete, which renumbers the implicit rowid — keying on `fts_rowid` keeps the external-content index aligned by construction. The regression guard is `KnowledgeIndexStore.test.ts` → "keeps search_text_fts aligned after a rowid-reshuffling rebuild". +`src/main/features/knowledge/pipeline/vectorstore/indexStore/schema.ts` keys `search_text_fts` on a stable `fts_rowid` column too (assigned by the `search_text_ai` trigger; `content_rowid='fts_rowid'`). It is a **separate per-base `index.sqlite`** (not the main DB, not drizzle-managed, not in `CUSTOM_SQL_STATEMENTS`), but the same hazard applies: its `reclaim()` path runs `VACUUM` to return freed pages to the OS after a large delete, which renumbers the implicit rowid — keying on `fts_rowid` keeps the external-content index aligned by construction. The regression guard is `KnowledgeIndexStore.test.ts` → "keeps search_text_fts aligned after a rowid-reshuffling rebuild". ## 5. Testing the build diff --git a/docs/references/knowledge/experiment/knowledge-technical-design.md b/docs/references/knowledge/experiment/knowledge-technical-design.md index 46c505e6090..ae31fb0c15b 100644 --- a/docs/references/knowledge/experiment/knowledge-technical-design.md +++ b/docs/references/knowledge/experiment/knowledge-technical-design.md @@ -101,7 +101,7 @@ Unique on `(target_type, target_id, kind)`; both FTS and vectors enter through ` ### 4.7 search_text_fts -External-content FTS5 (trigram). **FTS hits must join back through `search_text.rowid = search_text_fts.rowid`** — `search_text_id` is a TEXT business key, not the FTS rowid. +External-content FTS5 (trigram). **FTS hits must join back through `search_text.fts_rowid = search_text_fts.rowid`** — `fts_rowid` is a stable integer surrogate column (not SQLite's implicit rowid), because VACUUM and any `INSERT...SELECT` rebuild renumber the implicit rowid and would silently desync an external-content index keyed on it. `search_text_id` is a separate TEXT business key, not the FTS rowid. ## 5. Index interface and implementation notes @@ -149,7 +149,7 @@ A vector base's `knowledge_base.embeddingModelId` / `dimensions` must be valid t 1. Relational tables use generic SQLite DDL only; FTS5 is built into both engines; CJK handling lives in the application layer. 2. **Decision A1**: the canonical vector storage is a plain `BLOB` column holding little-endian float32 bytes (a generic SQLite BLOB, not any engine-proprietary vector type); it is the source of truth and both engines read the same bytes. 3. First-version vector retrieval is a brute-force scan over the canonical BLOBs (sqlite-vec's `vec_distance_cosine(col, ?)`, binding the query vector as raw little-endian float32 BLOB bytes — no `vector32()` wrapper), exposed through the `VectorIndex` adapter; **no** vec0 / ANN derived index (left as a purely additive change after performance evaluation). -4. A thin `SqliteDriver` port (execute / transaction / close) so the store is written once; better-sqlite3 keeps one synchronous, persistent connection — PRAGMAs (e.g. WAL) are set once and persist — and per-base writes are serialized by `KnowledgeLockManager.withBaseMutationLock(baseId)`, so the driver carries no write mutex or busy-retry of its own. +4. A thin `SqliteDriver` port (execute / transaction / close) so the store is written once; better-sqlite3 keeps one synchronous, persistent connection — PRAGMAs (e.g. WAL) are set once and persist — and per-base writes are serialized by the per-base mutation lock (`KeyedMutex.runExclusive`), so the driver carries no write mutex or busy-retry of its own. ## 6. Retrieval @@ -184,6 +184,6 @@ A vector base's `knowledge_base.embeddingModelId` / `dimensions` must be valid t - An explicit `maxParallelCalls` (plus token-aware batching) for `AiService.embedMany`, so one large document cannot fan out unbounded batches, exceed provider per-request token limits, and discard embeddings already paid for in a failed attempt. - Startup-recovery cross-cancellation: a crash-recovered delete-subtree job and the `recoverDeletingItems` re-enqueue get different idempotency keys and cancel each other via roots-intersection (`jobTouchesSubtree`); cancel only jobs whose roots are fully covered by the current job's roots. - Hybrid search runs its two lanes as independent read snapshots; a rebuild committing between them can transiently return both copies of a chunk — close with a shared read transaction or a second dedupe by material id + unit index. - - The per-base index driver's `close()` no longer needs to take a write mutex: better-sqlite3 uses one synchronous, persistent connection and per-base writes are serialized by `KnowledgeLockManager.withBaseMutationLock(baseId)`, so there is no driver-level async write mutex to race — shutdown safety rests on JobManager draining before the store service stops. + - The per-base index driver's `close()` no longer needs to take a write mutex: better-sqlite3 uses one synchronous, persistent connection and per-base writes are serialized by the per-base mutation lock (`KeyedMutex.runExclusive`), so there is no driver-level async write mutex to race — shutdown safety rests on JobManager draining before the store service stops. - Retrieval-surface follow-ups (PR C): the `searchMode` `default`→`vector` rename is externally visible through the gateway's pass-through base entity, and a permanent open failure (legacy layout) currently maps to a retryable 503. - PR A's full test matrix and risk notes live in this repo's test suites (`src/main/features/knowledge/**/__tests__`) and the PR #15973 description. diff --git a/docs/references/knowledge/knowledge-service.md b/docs/references/knowledge/knowledge-service.md index eebb9e2c396..64d8be02b6d 100644 --- a/docs/references/knowledge/knowledge-service.md +++ b/docs/references/knowledge/knowledge-service.md @@ -2,7 +2,7 @@ This document records the current v2 knowledge backend shape in the main process. -It covers the `src/main/services/knowledge` workflow path and the SQLite-backed data services. It does not describe the legacy `src/main/knowledge` service or the old `knowledge-base:*` IPC channels. +It covers the `src/main/features/knowledge` workflow path and the SQLite-backed data services. It does not describe the legacy `src/main/knowledge` service or the old `knowledge-base:*` IPC channels. For workflow guard details, see [Knowledge Operation Guards](./operation-guards.md). For the workflow architecture overview, see [Knowledge Workflow Architecture](./workflow-architecture.md). @@ -21,15 +21,14 @@ The current implementation is split into four responsibility areas: - Expose database-backed list/get operations and base metadata/config patch. - Do not perform vector-store mutations. 3. `KnowledgeService` - - Owns caller-facing runtime IPC workflow. - - Creates/deletes/restores bases through data services and vector store services. - - Registers Knowledge JobManager handlers. - - Holds the `KnowledgeWorkflowService` and `KnowledgeLockManager`. - - Collapses delete/reindex item inputs to top-level roots and enforces runtime guards. + - Thin lifecycle facade: registers Knowledge JobManager handlers, runs boot recovery, and delegates every public method to `base/`, `ingestion/`, and `query/`. + - `base/` (`KnowledgeBaseAdminService`) creates/deletes/restores bases through data services and vector store services. The per-base mutation lock (a core `KeyedMutex`) is created by the `KnowledgeService` facade and shared with `base/`, `ingestion/`, and the job handlers. + - `ingestion/` (`KnowledgeIngestionService`) collapses delete/reindex item inputs to top-level roots, enforces runtime guards, and schedules the next workflow step. + - `query/` (`KnowledgeQueryService` / `KnowledgeConceptService`) serves search and the Concept ID tool surface. 4. Knowledge job handlers - Execute durable workflow stages through JobManager. - - Use `KnowledgeWorkflowService` for next-step scheduling. - - Use `KnowledgeLockManager` for same-base mutations and vector cleanup. + - Use `KnowledgeIngestionService` for next-step scheduling. + - Use the per-base mutation lock (`KeyedMutex.runExclusive`) for same-base mutations and vector cleanup. ```text caller @@ -39,13 +38,13 @@ caller caller -> preload knowledge IPC -> KnowledgeService - -> KnowledgeWorkflowService + -> KnowledgeIngestionService -> JobManager -> knowledge.prepare-root / knowledge.index-documents -> knowledge.delete-subtree / knowledge.reindex-subtree - -> KnowledgeLockManager + -> KeyedMutex.runExclusive -> KnowledgeBaseService / KnowledgeItemService - -> KnowledgeVectorStoreService / FileManager + -> KnowledgeVectorStoreService ``` There is no current `KnowledgeRuntimeService` and no in-memory Knowledge queue. Durable work is owned by `JobManager`. @@ -104,21 +103,19 @@ reindex-items(baseId, itemIds) ## IPC Surface -`KnowledgeService` currently owns these public IPC entrypoints: +`KnowledgeService` currently owns these public IpcApi routes, defined in `src/shared/ipc/schemas/knowledge.ts` and handled in `src/main/ipc/handlers/knowledge.ts`: -- `knowledge:create-base` -- `knowledge:restore-base` -- `knowledge:delete-base` -- `knowledge:add-items` -- `knowledge:delete-items` -- `knowledge:reindex-items` -- `knowledge:search` -- `knowledge:list-item-chunks` +- `knowledge.create_base` +- `knowledge.restore_base` +- `knowledge.delete_base` +- `knowledge.add_items` +- `knowledge.delete_items` +- `knowledge.reindex_items` +- `knowledge.search` +- `knowledge.list_item_chunks` These IPC handlers are workflow-oriented. They validate payloads, call data services, and enqueue or execute runtime work internally. (The former `knowledge:delete-item-chunk` entrypoint was removed with the per-base index store cutover — chunks are derived index rows, replaced wholesale by reindexing.) -`KnowledgeService` also owns one orphaned v1 bridge entrypoint, `knowledge-base:delete`. Its only caller was the legacy Redux `store/knowledge` slice, which has now been removed, so this entrypoint is dead and pending cleanup. It routes to the same `delete-base` path. - The chunk IPC entrypoint is a runtime inspection helper: - `list-item-chunks` rejects failed bases. @@ -134,11 +131,10 @@ Knowledge runtime work is persisted in JobManager. `KnowledgeService.onInit` reg - `knowledge.delete-subtree` - `knowledge.reindex-subtree` -Each base uses queue `base.${baseId}`. JobManager owns queue persistence, dispatch, retry, cancellation, timeout, and startup recovery. Knowledge code uses `KnowledgeLockManager` to serialize same-base vector and item mutations inside the current process. +Each base uses queue `base.${baseId}`. JobManager owns queue persistence, dispatch, retry, cancellation, timeout, and startup recovery. Knowledge code uses the per-base mutation lock (`KeyedMutex.runExclusive`) to serialize same-base vector and item mutations inside the current process. Current item statuses are: -- `idle` - `preparing` - `processing` - `reading` diff --git a/docs/references/knowledge/operation-guards.md b/docs/references/knowledge/operation-guards.md index c238ed26933..4e7c00b6006 100644 --- a/docs/references/knowledge/operation-guards.md +++ b/docs/references/knowledge/operation-guards.md @@ -59,7 +59,7 @@ Used only by `reindexItems`. - Runs after selected item ids have been collapsed to top-level roots. - Loads each selected root subtree with roots included. - Allows reindex only when every item in every selected subtree is terminal: `completed` or `failed`. -- Rejects active or deleting subtree state: `idle`, `preparing`, `processing`, `reading`, `embedding`, or `deleting`. +- Rejects active or deleting subtree state: `preparing`, `processing`, `reading`, `embedding`, or `deleting`. This is the backend authority for user-triggered reindex. UI may hide the reindex action for non-terminal rows, but the service guard must still reject stale or direct calls. @@ -262,7 +262,7 @@ The child scheduling compensation mirrors `addItems`: once a child job was accep ## Shutdown -`KnowledgeService` does not cancel knowledge jobs during service shutdown. Knowledge job handlers use JobManager `recovery: 'retry'`, so unfinished pending, delayed, or running rows are left for JobManager startup recovery instead of being terminal-cancelled while their knowledge items still show active statuses. +`KnowledgeService` does not cancel knowledge jobs during service shutdown. The indexing handlers (`prepare-root`, `index-documents`, `check-file-processing-result`) and `reindex-subtree` use JobManager `recovery: 'abandon'` — an app restart never silently resumes them, which would otherwise re-spend the paid embedding API. `KnowledgeIngestionService.recoverInterruptedItems()` runs on startup and parks any item left in an active status by an interrupted job as `failed`. Only `delete-subtree` uses `recovery: 'retry'`, so unfinished pending, delayed, or running delete jobs are left for JobManager startup recovery instead of being terminal-cancelled. ## Review Checklist diff --git a/docs/references/knowledge/workflow-architecture.md b/docs/references/knowledge/workflow-architecture.md index d17893efc92..01a0320e872 100644 --- a/docs/references/knowledge/workflow-architecture.md +++ b/docs/references/knowledge/workflow-architecture.md @@ -8,20 +8,20 @@ Knowledge operations are modelled as a lightweight workflow rather than a single ```text API / user action - -> KnowledgeWorkflowService + -> KnowledgeIngestionService -> JobManager -> Knowledge job handlers - -> KnowledgeLockManager - -> SQLite / vector store / FileManager + -> KeyedMutex.runExclusive + -> SQLite / index store / knowledge-owned files (raw/) ``` The design keeps three owners: -- `KnowledgeWorkflowService` decides the next workflow step. -- `KnowledgeLockManager` serializes same-base mutations and cleanup. +- `KnowledgeIngestionService` decides the next workflow step. +- The per-base mutation lock (`KeyedMutex.runExclusive`) serializes same-base mutations and cleanup. - Knowledge job handlers execute one durable stage and call the workflow service for the next step. -Helpers may own source planning, lifecycle writes, artifact refs, and FileProcessing adaptation. They should stay as modules until they need lifecycle-managed resources, IPC, timers, or long-lived state. +Helpers may own source planning, lifecycle writes, knowledge-owned raw files, and FileProcessing adaptation. They should stay as modules until they need lifecycle-managed resources, IPC, timers, or long-lived state. ## Workflow Entry Points @@ -78,10 +78,10 @@ Round 1 job types: ## Mutation And Crash Semantics -Same-base Knowledge mutations must go through `KnowledgeLockManager`. Main SQLite writes still use a synchronous write transaction (`DbService.withWriteTx`, or an equivalent `db.transaction()`); the lock manager is not a replacement for that write transaction. +Same-base Knowledge mutations must go through the per-base mutation lock (`KeyedMutex.runExclusive`). Main SQLite writes still use a synchronous write transaction (`DbService.withWriteTx`, or an equivalent `db.transaction()`); the mutation lock is not a replacement for that write transaction. Crash safety comes from durable jobs, durable item states, JobManager recovery, and idempotent cleanup. The in-memory mutation lock only serializes concurrent work in the current process. Delete and reindex span two stores: the main SQLite database and the per-base vector store. They cannot be one cross-store transaction. Consistency relies on durable re-entry and idempotent vector/artifact/row cleanup. -User-triggered reindex is not a cancellation primitive. The service admits reindex only when the entire selected subtree is already `completed` or `failed`. Active states (`idle`, `preparing`, `processing`, `reading`, `embedding`) and `deleting` are rejected; delete remains the operation that can be requested at any time. +User-triggered reindex is not a cancellation primitive. The service admits reindex only when the entire selected subtree is already `completed` or `failed`. Active states (`preparing`, `processing`, `reading`, `embedding`) and `deleting` are rejected; delete remains the operation that can be requested at any time. diff --git a/migrations/sqlite-drizzle/0018_easy_gertrude_yorkes.sql b/migrations/sqlite-drizzle/0018_easy_gertrude_yorkes.sql new file mode 100644 index 00000000000..560a4cc0750 --- /dev/null +++ b/migrations/sqlite-drizzle/0018_easy_gertrude_yorkes.sql @@ -0,0 +1,39 @@ +PRAGMA foreign_keys=OFF;--> statement-breakpoint +CREATE TABLE `__new_knowledge_item` ( + `id` text PRIMARY KEY NOT NULL, + `base_id` text NOT NULL, + `group_id` text, + `type` text NOT NULL, + `data` text NOT NULL, + `status` text NOT NULL, + `error` text, + `created_at` integer NOT NULL, + `updated_at` integer NOT NULL, + FOREIGN KEY (`base_id`) REFERENCES `knowledge_base`(`id`) ON UPDATE no action ON DELETE cascade, + FOREIGN KEY (`base_id`,`group_id`) REFERENCES `knowledge_item`(`base_id`,`id`) ON UPDATE no action ON DELETE cascade, + CONSTRAINT "knowledge_item_type_check" CHECK("__new_knowledge_item"."type" IN ('file', 'url', 'note', 'directory')), + CONSTRAINT "knowledge_item_status_check" CHECK("__new_knowledge_item"."status" IN ('preparing', 'processing', 'reading', 'embedding', 'completed', 'failed', 'deleting')), + CONSTRAINT "knowledge_item_type_status_check" CHECK( + ("__new_knowledge_item"."type" IN ('file', 'url', 'note') AND "__new_knowledge_item"."status" IN ('processing', 'reading', 'embedding', 'completed', 'failed', 'deleting')) + OR ("__new_knowledge_item"."type" = 'directory' AND "__new_knowledge_item"."status" IN ('preparing', 'processing', 'completed', 'failed', 'deleting')) + ), + CONSTRAINT "knowledge_item_status_error_check" CHECK( + ( + "__new_knowledge_item"."status" IN ('preparing', 'processing', 'reading', 'embedding', 'completed', 'deleting') + AND "__new_knowledge_item"."error" IS NULL + ) + OR ( + "__new_knowledge_item"."status" = 'failed' + AND "__new_knowledge_item"."error" IS NOT NULL + AND length(trim("__new_knowledge_item"."error")) > 0 + ) + ) +); +--> statement-breakpoint +INSERT INTO `__new_knowledge_item`("id", "base_id", "group_id", "type", "data", "status", "error", "created_at", "updated_at") SELECT "id", "base_id", "group_id", "type", "data", "status", "error", "created_at", "updated_at" FROM `knowledge_item`;--> statement-breakpoint +DROP TABLE `knowledge_item`;--> statement-breakpoint +ALTER TABLE `__new_knowledge_item` RENAME TO `knowledge_item`;--> statement-breakpoint +PRAGMA foreign_keys=ON;--> statement-breakpoint +CREATE INDEX `knowledge_item_base_type_created_idx` ON `knowledge_item` (`base_id`,`type`,`created_at`);--> statement-breakpoint +CREATE INDEX `knowledge_item_base_group_created_idx` ON `knowledge_item` (`base_id`,`group_id`,`created_at`);--> statement-breakpoint +CREATE UNIQUE INDEX `knowledge_item_baseId_id_unique` ON `knowledge_item` (`base_id`,`id`); \ No newline at end of file diff --git a/migrations/sqlite-drizzle/meta/0018_snapshot.json b/migrations/sqlite-drizzle/meta/0018_snapshot.json new file mode 100644 index 00000000000..315ee23507c --- /dev/null +++ b/migrations/sqlite-drizzle/meta/0018_snapshot.json @@ -0,0 +1,3866 @@ +{ + "version": "6", + "dialect": "sqlite", + "id": "c7a83c70-3fe5-4b08-b56a-db4a64463c93", + "prevId": "53ba65df-7a61-4a0c-a563-3e32dfba69fa", + "tables": { + "agent": { + "name": "agent", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "type": { + "name": "type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "description": { + "name": "description", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "''" + }, + "instructions": { + "name": "instructions", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "model": { + "name": "model", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "plan_model": { + "name": "plan_model", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "small_model": { + "name": "small_model", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "disabled_tools": { + "name": "disabled_tools", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'[]'" + }, + "configuration": { + "name": "configuration", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'{}'" + }, + "order_key": { + "name": "order_key", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "deleted_at": { + "name": "deleted_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": { + "agent_name_idx": { + "name": "agent_name_idx", + "columns": ["name"], + "isUnique": false + }, + "agent_type_idx": { + "name": "agent_type_idx", + "columns": ["type"], + "isUnique": false + }, + "agent_order_key_idx": { + "name": "agent_order_key_idx", + "columns": ["order_key"], + "isUnique": false + } + }, + "foreignKeys": { + "agent_model_user_model_id_fk": { + "name": "agent_model_user_model_id_fk", + "tableFrom": "agent", + "tableTo": "user_model", + "columnsFrom": ["model"], + "columnsTo": ["id"], + "onDelete": "set null", + "onUpdate": "no action" + }, + "agent_plan_model_user_model_id_fk": { + "name": "agent_plan_model_user_model_id_fk", + "tableFrom": "agent", + "tableTo": "user_model", + "columnsFrom": ["plan_model"], + "columnsTo": ["id"], + "onDelete": "set null", + "onUpdate": "no action" + }, + "agent_small_model_user_model_id_fk": { + "name": "agent_small_model_user_model_id_fk", + "tableFrom": "agent", + "tableTo": "user_model", + "columnsFrom": ["small_model"], + "columnsTo": ["id"], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "agent_channel": { + "name": "agent_channel", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "type": { + "name": "type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "agent_id": { + "name": "agent_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "session_id": { + "name": "session_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "workspace": { + "name": "workspace", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "config": { + "name": "config", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "is_active": { + "name": "is_active", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": true + }, + "active_chat_ids": { + "name": "active_chat_ids", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'[]'" + }, + "permission_mode": { + "name": "permission_mode", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "agent_channel_agent_id_idx": { + "name": "agent_channel_agent_id_idx", + "columns": ["agent_id"], + "isUnique": false + }, + "agent_channel_type_idx": { + "name": "agent_channel_type_idx", + "columns": ["type"], + "isUnique": false + }, + "agent_channel_session_id_idx": { + "name": "agent_channel_session_id_idx", + "columns": ["session_id"], + "isUnique": false + } + }, + "foreignKeys": { + "agent_channel_agent_id_agent_id_fk": { + "name": "agent_channel_agent_id_agent_id_fk", + "tableFrom": "agent_channel", + "tableTo": "agent", + "columnsFrom": ["agent_id"], + "columnsTo": ["id"], + "onDelete": "set null", + "onUpdate": "no action" + }, + "agent_channel_session_id_agent_session_id_fk": { + "name": "agent_channel_session_id_agent_session_id_fk", + "tableFrom": "agent_channel", + "tableTo": "agent_session", + "columnsFrom": ["session_id"], + "columnsTo": ["id"], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": { + "agent_channel_type_check": { + "name": "agent_channel_type_check", + "value": "\"agent_channel\".\"type\" IN ('telegram', 'feishu', 'qq', 'wechat', 'discord', 'slack')" + }, + "agent_channel_permission_mode_check": { + "name": "agent_channel_permission_mode_check", + "value": "\"agent_channel\".\"permission_mode\" IS NULL OR \"agent_channel\".\"permission_mode\" IN ('default', 'acceptEdits', 'bypassPermissions', 'plan')" + } + } + }, + "agent_channel_task": { + "name": "agent_channel_task", + "columns": { + "channel_id": { + "name": "channel_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "task_id": { + "name": "task_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "agent_channel_task_channel_id_idx": { + "name": "agent_channel_task_channel_id_idx", + "columns": ["channel_id"], + "isUnique": false + }, + "agent_channel_task_task_id_idx": { + "name": "agent_channel_task_task_id_idx", + "columns": ["task_id"], + "isUnique": false + } + }, + "foreignKeys": { + "agent_channel_task_channel_id_agent_channel_id_fk": { + "name": "agent_channel_task_channel_id_agent_channel_id_fk", + "tableFrom": "agent_channel_task", + "tableTo": "agent_channel", + "columnsFrom": ["channel_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "agent_channel_task_task_id_job_schedule_id_fk": { + "name": "agent_channel_task_task_id_job_schedule_id_fk", + "tableFrom": "agent_channel_task", + "tableTo": "job_schedule", + "columnsFrom": ["task_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": { + "agent_channel_task_channel_id_task_id_pk": { + "columns": ["channel_id", "task_id"], + "name": "agent_channel_task_channel_id_task_id_pk" + } + }, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "agent_global_skill": { + "name": "agent_global_skill", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "description": { + "name": "description", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "folder_name": { + "name": "folder_name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "source": { + "name": "source", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "source_url": { + "name": "source_url", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "namespace": { + "name": "namespace", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "author": { + "name": "author", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "tags": { + "name": "tags", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'[]'" + }, + "content_hash": { + "name": "content_hash", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "is_enabled": { + "name": "is_enabled", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "agent_global_skill_folder_name_unique": { + "name": "agent_global_skill_folder_name_unique", + "columns": ["folder_name"], + "isUnique": true + }, + "agent_global_skill_source_idx": { + "name": "agent_global_skill_source_idx", + "columns": ["source"], + "isUnique": false + }, + "agent_global_skill_is_enabled_idx": { + "name": "agent_global_skill_is_enabled_idx", + "columns": ["is_enabled"], + "isUnique": false + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "agent_session": { + "name": "agent_session", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "agent_id": { + "name": "agent_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "is_name_manually_edited": { + "name": "is_name_manually_edited", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "description": { + "name": "description", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "''" + }, + "workspace_id": { + "name": "workspace_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "trace_id": { + "name": "trace_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "order_key": { + "name": "order_key", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "agent_session_order_key_idx": { + "name": "agent_session_order_key_idx", + "columns": ["order_key"], + "isUnique": false + } + }, + "foreignKeys": { + "agent_session_agent_id_agent_id_fk": { + "name": "agent_session_agent_id_agent_id_fk", + "tableFrom": "agent_session", + "tableTo": "agent", + "columnsFrom": ["agent_id"], + "columnsTo": ["id"], + "onDelete": "set null", + "onUpdate": "no action" + }, + "agent_session_workspace_id_agent_workspace_id_fk": { + "name": "agent_session_workspace_id_agent_workspace_id_fk", + "tableFrom": "agent_session", + "tableTo": "agent_workspace", + "columnsFrom": ["workspace_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "agent_session_message": { + "name": "agent_session_message", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "session_id": { + "name": "session_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "role": { + "name": "role", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "data": { + "name": "data", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "searchable_text": { + "name": "searchable_text", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "''" + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "model_id": { + "name": "model_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "model_snapshot": { + "name": "model_snapshot", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "stats": { + "name": "stats", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "runtime_resume_token": { + "name": "runtime_resume_token", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "fts_rowid": { + "name": "fts_rowid", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "agent_session_message_session_created_id_idx": { + "name": "agent_session_message_session_created_id_idx", + "columns": ["session_id", "created_at", "id"], + "isUnique": false + }, + "agent_session_message_status_idx": { + "name": "agent_session_message_status_idx", + "columns": ["status"], + "isUnique": false + }, + "agent_session_message_fts_rowid_uniq": { + "name": "agent_session_message_fts_rowid_uniq", + "columns": ["fts_rowid"], + "isUnique": true + } + }, + "foreignKeys": { + "agent_session_message_session_id_agent_session_id_fk": { + "name": "agent_session_message_session_id_agent_session_id_fk", + "tableFrom": "agent_session_message", + "tableTo": "agent_session", + "columnsFrom": ["session_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "agent_session_message_model_id_user_model_id_fk": { + "name": "agent_session_message_model_id_user_model_id_fk", + "tableFrom": "agent_session_message", + "tableTo": "user_model", + "columnsFrom": ["model_id"], + "columnsTo": ["id"], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": { + "agent_session_message_role_check": { + "name": "agent_session_message_role_check", + "value": "\"agent_session_message\".\"role\" IN ('user', 'assistant', 'system')" + }, + "agent_session_message_status_check": { + "name": "agent_session_message_status_check", + "value": "\"agent_session_message\".\"status\" IN ('pending', 'success', 'error', 'paused')" + } + } + }, + "agent_skill": { + "name": "agent_skill", + "columns": { + "agent_id": { + "name": "agent_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "skill_id": { + "name": "skill_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "is_enabled": { + "name": "is_enabled", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "agent_skill_agent_id_idx": { + "name": "agent_skill_agent_id_idx", + "columns": ["agent_id"], + "isUnique": false + }, + "agent_skill_skill_id_idx": { + "name": "agent_skill_skill_id_idx", + "columns": ["skill_id"], + "isUnique": false + } + }, + "foreignKeys": { + "agent_skill_agent_id_agent_id_fk": { + "name": "agent_skill_agent_id_agent_id_fk", + "tableFrom": "agent_skill", + "tableTo": "agent", + "columnsFrom": ["agent_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "agent_skill_skill_id_agent_global_skill_id_fk": { + "name": "agent_skill_skill_id_agent_global_skill_id_fk", + "tableFrom": "agent_skill", + "tableTo": "agent_global_skill", + "columnsFrom": ["skill_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": { + "agent_skill_agent_id_skill_id_pk": { + "columns": ["agent_id", "skill_id"], + "name": "agent_skill_agent_id_skill_id_pk" + } + }, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "agent_workspace": { + "name": "agent_workspace", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "path": { + "name": "path", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "type": { + "name": "type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'user'" + }, + "order_key": { + "name": "order_key", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "agent_workspace_path_unique_idx": { + "name": "agent_workspace_path_unique_idx", + "columns": ["path"], + "isUnique": true + }, + "agent_workspace_order_key_idx": { + "name": "agent_workspace_order_key_idx", + "columns": ["order_key"], + "isUnique": false + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": { + "agent_workspace_type_check": { + "name": "agent_workspace_type_check", + "value": "\"agent_workspace\".\"type\" IN ('user', 'system')" + } + } + }, + "app_state": { + "name": "app_state", + "columns": { + "key": { + "name": "key", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "value": { + "name": "value", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "description": { + "name": "description", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": {}, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "assistant": { + "name": "assistant", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "prompt": { + "name": "prompt", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "''" + }, + "emoji": { + "name": "emoji", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "description": { + "name": "description", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "''" + }, + "model_id": { + "name": "model_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "settings": { + "name": "settings", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "order_key": { + "name": "order_key", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "deleted_at": { + "name": "deleted_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": { + "assistant_created_at_idx": { + "name": "assistant_created_at_idx", + "columns": ["created_at"], + "isUnique": false + }, + "assistant_order_key_idx": { + "name": "assistant_order_key_idx", + "columns": ["order_key"], + "isUnique": false + } + }, + "foreignKeys": { + "assistant_model_id_user_model_id_fk": { + "name": "assistant_model_id_user_model_id_fk", + "tableFrom": "assistant", + "tableTo": "user_model", + "columnsFrom": ["model_id"], + "columnsTo": ["id"], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "agent_mcp_server": { + "name": "agent_mcp_server", + "columns": { + "agent_id": { + "name": "agent_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "mcp_server_id": { + "name": "mcp_server_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": {}, + "foreignKeys": { + "agent_mcp_server_agent_id_agent_id_fk": { + "name": "agent_mcp_server_agent_id_agent_id_fk", + "tableFrom": "agent_mcp_server", + "tableTo": "agent", + "columnsFrom": ["agent_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "agent_mcp_server_mcp_server_id_mcp_server_id_fk": { + "name": "agent_mcp_server_mcp_server_id_mcp_server_id_fk", + "tableFrom": "agent_mcp_server", + "tableTo": "mcp_server", + "columnsFrom": ["mcp_server_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": { + "agent_mcp_server_agent_id_mcp_server_id_pk": { + "columns": ["agent_id", "mcp_server_id"], + "name": "agent_mcp_server_agent_id_mcp_server_id_pk" + } + }, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "assistant_knowledge_base": { + "name": "assistant_knowledge_base", + "columns": { + "assistant_id": { + "name": "assistant_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "knowledge_base_id": { + "name": "knowledge_base_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": {}, + "foreignKeys": { + "assistant_knowledge_base_assistant_id_assistant_id_fk": { + "name": "assistant_knowledge_base_assistant_id_assistant_id_fk", + "tableFrom": "assistant_knowledge_base", + "tableTo": "assistant", + "columnsFrom": ["assistant_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "assistant_knowledge_base_knowledge_base_id_knowledge_base_id_fk": { + "name": "assistant_knowledge_base_knowledge_base_id_knowledge_base_id_fk", + "tableFrom": "assistant_knowledge_base", + "tableTo": "knowledge_base", + "columnsFrom": ["knowledge_base_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": { + "assistant_knowledge_base_assistant_id_knowledge_base_id_pk": { + "columns": ["assistant_id", "knowledge_base_id"], + "name": "assistant_knowledge_base_assistant_id_knowledge_base_id_pk" + } + }, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "assistant_mcp_server": { + "name": "assistant_mcp_server", + "columns": { + "assistant_id": { + "name": "assistant_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "mcp_server_id": { + "name": "mcp_server_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": {}, + "foreignKeys": { + "assistant_mcp_server_assistant_id_assistant_id_fk": { + "name": "assistant_mcp_server_assistant_id_assistant_id_fk", + "tableFrom": "assistant_mcp_server", + "tableTo": "assistant", + "columnsFrom": ["assistant_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "assistant_mcp_server_mcp_server_id_mcp_server_id_fk": { + "name": "assistant_mcp_server_mcp_server_id_mcp_server_id_fk", + "tableFrom": "assistant_mcp_server", + "tableTo": "mcp_server", + "columnsFrom": ["mcp_server_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": { + "assistant_mcp_server_assistant_id_mcp_server_id_pk": { + "columns": ["assistant_id", "mcp_server_id"], + "name": "assistant_mcp_server_assistant_id_mcp_server_id_pk" + } + }, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "file_entry": { + "name": "file_entry", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "origin": { + "name": "origin", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "ext": { + "name": "ext", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "size": { + "name": "size", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "external_path": { + "name": "external_path", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "deleted_at": { + "name": "deleted_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": { + "fe_deleted_at_idx": { + "name": "fe_deleted_at_idx", + "columns": ["deleted_at"], + "isUnique": false + }, + "fe_created_at_idx": { + "name": "fe_created_at_idx", + "columns": ["created_at"], + "isUnique": false + }, + "fe_external_path_lower_unique_idx": { + "name": "fe_external_path_lower_unique_idx", + "columns": ["lower(\"external_path\")"], + "isUnique": true + }, + "fe_external_path_idx": { + "name": "fe_external_path_idx", + "columns": ["external_path"], + "isUnique": false + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": { + "fe_origin_check": { + "name": "fe_origin_check", + "value": "\"file_entry\".\"origin\" IN ('internal', 'external')" + }, + "fe_origin_consistency": { + "name": "fe_origin_consistency", + "value": "(\"file_entry\".\"origin\" = 'internal' AND \"file_entry\".\"external_path\" IS NULL) OR (\"file_entry\".\"origin\" = 'external' AND \"file_entry\".\"external_path\" IS NOT NULL)" + }, + "fe_external_no_delete": { + "name": "fe_external_no_delete", + "value": "\"file_entry\".\"origin\" != 'external' OR \"file_entry\".\"deleted_at\" IS NULL" + }, + "fe_size_internal_only": { + "name": "fe_size_internal_only", + "value": "(\"file_entry\".\"origin\" = 'internal' AND \"file_entry\".\"size\" IS NOT NULL AND \"file_entry\".\"size\" >= 0) OR (\"file_entry\".\"origin\" = 'external' AND \"file_entry\".\"size\" IS NULL)" + } + } + }, + "chat_message_file_ref": { + "name": "chat_message_file_ref", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "file_entry_id": { + "name": "file_entry_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "source_id": { + "name": "source_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "role": { + "name": "role", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "cmfr_entry_id_idx": { + "name": "cmfr_entry_id_idx", + "columns": ["file_entry_id"], + "isUnique": false + }, + "cmfr_source_id_idx": { + "name": "cmfr_source_id_idx", + "columns": ["source_id"], + "isUnique": false + }, + "cmfr_unique_idx": { + "name": "cmfr_unique_idx", + "columns": ["file_entry_id", "source_id", "role"], + "isUnique": true + } + }, + "foreignKeys": { + "chat_message_file_ref_file_entry_id_file_entry_id_fk": { + "name": "chat_message_file_ref_file_entry_id_file_entry_id_fk", + "tableFrom": "chat_message_file_ref", + "tableTo": "file_entry", + "columnsFrom": ["file_entry_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "chat_message_file_ref_source_id_message_id_fk": { + "name": "chat_message_file_ref_source_id_message_id_fk", + "tableFrom": "chat_message_file_ref", + "tableTo": "message", + "columnsFrom": ["source_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": { + "cmfr_role_check": { + "name": "cmfr_role_check", + "value": "\"chat_message_file_ref\".\"role\" IN ('attachment')" + } + } + }, + "painting_file_ref": { + "name": "painting_file_ref", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "file_entry_id": { + "name": "file_entry_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "source_id": { + "name": "source_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "role": { + "name": "role", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "pfr_entry_id_idx": { + "name": "pfr_entry_id_idx", + "columns": ["file_entry_id"], + "isUnique": false + }, + "pfr_source_id_idx": { + "name": "pfr_source_id_idx", + "columns": ["source_id"], + "isUnique": false + }, + "pfr_unique_idx": { + "name": "pfr_unique_idx", + "columns": ["file_entry_id", "source_id", "role"], + "isUnique": true + } + }, + "foreignKeys": { + "painting_file_ref_file_entry_id_file_entry_id_fk": { + "name": "painting_file_ref_file_entry_id_file_entry_id_fk", + "tableFrom": "painting_file_ref", + "tableTo": "file_entry", + "columnsFrom": ["file_entry_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "painting_file_ref_source_id_painting_id_fk": { + "name": "painting_file_ref_source_id_painting_id_fk", + "tableFrom": "painting_file_ref", + "tableTo": "painting", + "columnsFrom": ["source_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": { + "pfr_role_check": { + "name": "pfr_role_check", + "value": "\"painting_file_ref\".\"role\" IN ('output', 'input')" + } + } + }, + "group": { + "name": "group", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "entity_type": { + "name": "entity_type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "order_key": { + "name": "order_key", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "group_entity_type_order_key_idx": { + "name": "group_entity_type_order_key_idx", + "columns": ["entity_type", "order_key"], + "isUnique": false + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "job_schedule": { + "name": "job_schedule", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "type": { + "name": "type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "''" + }, + "trigger": { + "name": "trigger", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "job_input_template": { + "name": "job_input_template", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "enabled": { + "name": "enabled", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": true + }, + "next_run": { + "name": "next_run", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "last_run": { + "name": "last_run", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "catch_up_policy": { + "name": "catch_up_policy", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "metadata": { + "name": "metadata", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'{}'" + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "job_schedule_type_name_uq": { + "name": "job_schedule_type_name_uq", + "columns": ["type", "name"], + "isUnique": true + }, + "job_schedule_enabled_next_run_idx": { + "name": "job_schedule_enabled_next_run_idx", + "columns": ["enabled", "next_run"], + "isUnique": false + }, + "job_schedule_type_idx": { + "name": "job_schedule_type_idx", + "columns": ["type"], + "isUnique": false + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "job": { + "name": "job", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "type": { + "name": "type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "priority": { + "name": "priority", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "queue": { + "name": "queue", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "idempotency_key": { + "name": "idempotency_key", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "schedule_id": { + "name": "schedule_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "scheduled_at": { + "name": "scheduled_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "started_at": { + "name": "started_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "finished_at": { + "name": "finished_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "attempt": { + "name": "attempt", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "max_attempts": { + "name": "max_attempts", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 3 + }, + "input": { + "name": "input", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "output": { + "name": "output", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "error": { + "name": "error", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "parent_id": { + "name": "parent_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "cancel_requested": { + "name": "cancel_requested", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "metadata": { + "name": "metadata", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'{}'" + }, + "timeout_ms": { + "name": "timeout_ms", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "job_queue_status_scheduled_at_idx": { + "name": "job_queue_status_scheduled_at_idx", + "columns": ["queue", "status", "scheduled_at"], + "isUnique": false + }, + "job_status_idx": { + "name": "job_status_idx", + "columns": ["status"], + "isUnique": false + }, + "job_schedule_id_finished_at_idx": { + "name": "job_schedule_id_finished_at_idx", + "columns": ["schedule_id", "finished_at"], + "isUnique": false + }, + "job_parent_id_idx": { + "name": "job_parent_id_idx", + "columns": ["parent_id"], + "isUnique": false + }, + "job_idempotency_key_partial_uq": { + "name": "job_idempotency_key_partial_uq", + "columns": ["idempotency_key"], + "isUnique": true, + "where": "\"job\".\"idempotency_key\" IS NOT NULL AND \"job\".\"status\" NOT IN ('completed','failed','cancelled')" + } + }, + "foreignKeys": { + "job_schedule_id_job_schedule_id_fk": { + "name": "job_schedule_id_job_schedule_id_fk", + "tableFrom": "job", + "tableTo": "job_schedule", + "columnsFrom": ["schedule_id"], + "columnsTo": ["id"], + "onDelete": "set null", + "onUpdate": "no action" + }, + "job_parent_id_job_id_fk": { + "name": "job_parent_id_job_id_fk", + "tableFrom": "job", + "tableTo": "job", + "columnsFrom": ["parent_id"], + "columnsTo": ["id"], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": { + "job_status_check": { + "name": "job_status_check", + "value": "\"job\".\"status\" IN ('pending','delayed','running','completed','failed','cancelled')" + } + } + }, + "knowledge_base": { + "name": "knowledge_base", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "group_id": { + "name": "group_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "dimensions": { + "name": "dimensions", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "embedding_model_id": { + "name": "embedding_model_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "error": { + "name": "error", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "rerank_model_id": { + "name": "rerank_model_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "file_processor_id": { + "name": "file_processor_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "chunk_size": { + "name": "chunk_size", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "chunk_overlap": { + "name": "chunk_overlap", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "chunk_strategy": { + "name": "chunk_strategy", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'structured'" + }, + "chunk_separator": { + "name": "chunk_separator", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'\\n\\n'" + }, + "threshold": { + "name": "threshold", + "type": "real", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "document_count": { + "name": "document_count", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "search_mode": { + "name": "search_mode", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "hybrid_alpha": { + "name": "hybrid_alpha", + "type": "real", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": {}, + "foreignKeys": { + "knowledge_base_group_id_group_id_fk": { + "name": "knowledge_base_group_id_group_id_fk", + "tableFrom": "knowledge_base", + "tableTo": "group", + "columnsFrom": ["group_id"], + "columnsTo": ["id"], + "onDelete": "set null", + "onUpdate": "no action" + }, + "knowledge_base_embedding_model_id_user_model_id_fk": { + "name": "knowledge_base_embedding_model_id_user_model_id_fk", + "tableFrom": "knowledge_base", + "tableTo": "user_model", + "columnsFrom": ["embedding_model_id"], + "columnsTo": ["id"], + "onDelete": "no action", + "onUpdate": "no action" + }, + "knowledge_base_rerank_model_id_user_model_id_fk": { + "name": "knowledge_base_rerank_model_id_user_model_id_fk", + "tableFrom": "knowledge_base", + "tableTo": "user_model", + "columnsFrom": ["rerank_model_id"], + "columnsTo": ["id"], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": { + "knowledge_base_search_mode_check": { + "name": "knowledge_base_search_mode_check", + "value": "\"knowledge_base\".\"search_mode\" IN ('vector', 'bm25', 'hybrid')" + }, + "knowledge_base_chunk_strategy_check": { + "name": "knowledge_base_chunk_strategy_check", + "value": "\"knowledge_base\".\"chunk_strategy\" IN ('structured', 'delimiter')" + }, + "knowledge_base_status_check": { + "name": "knowledge_base_status_check", + "value": "\"knowledge_base\".\"status\" IN ('completed', 'failed')" + }, + "knowledge_base_status_error_check": { + "name": "knowledge_base_status_error_check", + "value": "\n (\n \"knowledge_base\".\"status\" = 'completed'\n AND \"knowledge_base\".\"error\" IS NULL\n AND (\n (\n \"knowledge_base\".\"embedding_model_id\" IS NOT NULL\n AND \"knowledge_base\".\"dimensions\" IS NOT NULL\n AND \"knowledge_base\".\"dimensions\" > 0\n )\n OR (\n \"knowledge_base\".\"embedding_model_id\" IS NULL\n AND \"knowledge_base\".\"dimensions\" IS NULL\n AND \"knowledge_base\".\"search_mode\" = 'bm25'\n )\n )\n )\n OR (\n \"knowledge_base\".\"status\" = 'failed'\n AND \"knowledge_base\".\"error\" IS NOT NULL\n AND length(trim(\"knowledge_base\".\"error\")) > 0\n )\n " + } + } + }, + "knowledge_item": { + "name": "knowledge_item", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "base_id": { + "name": "base_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "group_id": { + "name": "group_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "type": { + "name": "type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "data": { + "name": "data", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "error": { + "name": "error", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "knowledge_item_base_type_created_idx": { + "name": "knowledge_item_base_type_created_idx", + "columns": ["base_id", "type", "created_at"], + "isUnique": false + }, + "knowledge_item_base_group_created_idx": { + "name": "knowledge_item_base_group_created_idx", + "columns": ["base_id", "group_id", "created_at"], + "isUnique": false + }, + "knowledge_item_baseId_id_unique": { + "name": "knowledge_item_baseId_id_unique", + "columns": ["base_id", "id"], + "isUnique": true + } + }, + "foreignKeys": { + "knowledge_item_base_id_knowledge_base_id_fk": { + "name": "knowledge_item_base_id_knowledge_base_id_fk", + "tableFrom": "knowledge_item", + "tableTo": "knowledge_base", + "columnsFrom": ["base_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "knowledge_item_base_id_group_id_knowledge_item_base_id_id_fk": { + "name": "knowledge_item_base_id_group_id_knowledge_item_base_id_id_fk", + "tableFrom": "knowledge_item", + "tableTo": "knowledge_item", + "columnsFrom": ["base_id", "group_id"], + "columnsTo": ["base_id", "id"], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": { + "knowledge_item_type_check": { + "name": "knowledge_item_type_check", + "value": "\"knowledge_item\".\"type\" IN ('file', 'url', 'note', 'directory')" + }, + "knowledge_item_status_check": { + "name": "knowledge_item_status_check", + "value": "\"knowledge_item\".\"status\" IN ('preparing', 'processing', 'reading', 'embedding', 'completed', 'failed', 'deleting')" + }, + "knowledge_item_type_status_check": { + "name": "knowledge_item_type_status_check", + "value": "\n (\"knowledge_item\".\"type\" IN ('file', 'url', 'note') AND \"knowledge_item\".\"status\" IN ('processing', 'reading', 'embedding', 'completed', 'failed', 'deleting'))\n OR (\"knowledge_item\".\"type\" = 'directory' AND \"knowledge_item\".\"status\" IN ('preparing', 'processing', 'completed', 'failed', 'deleting'))\n " + }, + "knowledge_item_status_error_check": { + "name": "knowledge_item_status_error_check", + "value": "\n (\n \"knowledge_item\".\"status\" IN ('preparing', 'processing', 'reading', 'embedding', 'completed', 'deleting')\n AND \"knowledge_item\".\"error\" IS NULL\n )\n OR (\n \"knowledge_item\".\"status\" = 'failed'\n AND \"knowledge_item\".\"error\" IS NOT NULL\n AND length(trim(\"knowledge_item\".\"error\")) > 0\n )\n " + } + } + }, + "mcp_server": { + "name": "mcp_server", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "type": { + "name": "type", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "description": { + "name": "description", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "base_url": { + "name": "base_url", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "command": { + "name": "command", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "registry_url": { + "name": "registry_url", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "args": { + "name": "args", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "env": { + "name": "env", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "headers": { + "name": "headers", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "provider": { + "name": "provider", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "provider_url": { + "name": "provider_url", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "logo_url": { + "name": "logo_url", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "tags": { + "name": "tags", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "long_running": { + "name": "long_running", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "timeout": { + "name": "timeout", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "dxt_version": { + "name": "dxt_version", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "dxt_path": { + "name": "dxt_path", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "reference": { + "name": "reference", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "search_key": { + "name": "search_key", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "config_sample": { + "name": "config_sample", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "disabled_tools": { + "name": "disabled_tools", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "disabled_auto_approve_tools": { + "name": "disabled_auto_approve_tools", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "should_config": { + "name": "should_config", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "sort_order": { + "name": "sort_order", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false, + "default": 0 + }, + "is_active": { + "name": "is_active", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "install_source": { + "name": "install_source", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "is_trusted": { + "name": "is_trusted", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "trusted_at": { + "name": "trusted_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "installed_at": { + "name": "installed_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "mcp_server_name_idx": { + "name": "mcp_server_name_idx", + "columns": ["name"], + "isUnique": false + }, + "mcp_server_is_active_idx": { + "name": "mcp_server_is_active_idx", + "columns": ["is_active"], + "isUnique": false + }, + "mcp_server_sort_order_idx": { + "name": "mcp_server_sort_order_idx", + "columns": ["sort_order"], + "isUnique": false + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": { + "mcp_server_type_check": { + "name": "mcp_server_type_check", + "value": "\"mcp_server\".\"type\" IS NULL OR \"mcp_server\".\"type\" IN ('stdio', 'sse', 'streamableHttp', 'inMemory')" + }, + "mcp_server_install_source_check": { + "name": "mcp_server_install_source_check", + "value": "\"mcp_server\".\"install_source\" IS NULL OR \"mcp_server\".\"install_source\" IN ('builtin', 'manual', 'protocol', 'unknown')" + } + } + }, + "message": { + "name": "message", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "parent_id": { + "name": "parent_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "topic_id": { + "name": "topic_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "role": { + "name": "role", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "data": { + "name": "data", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "searchable_text": { + "name": "searchable_text", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "''" + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "siblings_group_id": { + "name": "siblings_group_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "model_id": { + "name": "model_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "model_snapshot": { + "name": "model_snapshot", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "stats": { + "name": "stats", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "fts_rowid": { + "name": "fts_rowid", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "deleted_at": { + "name": "deleted_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": { + "message_parent_id_idx": { + "name": "message_parent_id_idx", + "columns": ["parent_id"], + "isUnique": false + }, + "message_topic_created_idx": { + "name": "message_topic_created_idx", + "columns": ["topic_id", "created_at"], + "isUnique": false + }, + "message_status_idx": { + "name": "message_status_idx", + "columns": ["status"], + "isUnique": false + }, + "message_topic_root_uniq": { + "name": "message_topic_root_uniq", + "columns": ["topic_id"], + "isUnique": true, + "where": "\"message\".\"parent_id\" is null and \"message\".\"deleted_at\" is null" + }, + "message_fts_rowid_uniq": { + "name": "message_fts_rowid_uniq", + "columns": ["fts_rowid"], + "isUnique": true + } + }, + "foreignKeys": { + "message_topic_id_topic_id_fk": { + "name": "message_topic_id_topic_id_fk", + "tableFrom": "message", + "tableTo": "topic", + "columnsFrom": ["topic_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "message_model_id_user_model_id_fk": { + "name": "message_model_id_user_model_id_fk", + "tableFrom": "message", + "tableTo": "user_model", + "columnsFrom": ["model_id"], + "columnsTo": ["id"], + "onDelete": "set null", + "onUpdate": "no action" + }, + "message_parent_id_message_id_fk": { + "name": "message_parent_id_message_id_fk", + "tableFrom": "message", + "tableTo": "message", + "columnsFrom": ["parent_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": { + "message_role_check": { + "name": "message_role_check", + "value": "\"message\".\"role\" IN ('user', 'assistant', 'system', 'root')" + }, + "message_status_check": { + "name": "message_status_check", + "value": "\"message\".\"status\" IN ('pending', 'success', 'error', 'paused')" + }, + "message_root_parent_check": { + "name": "message_root_parent_check", + "value": "(\"message\".\"role\" = 'root') = (\"message\".\"parent_id\" is null)" + } + } + }, + "mini_app": { + "name": "mini_app", + "columns": { + "app_id": { + "name": "app_id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "preset_mini_app_id": { + "name": "preset_mini_app_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "url": { + "name": "url", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "logo": { + "name": "logo", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'enabled'" + }, + "order_key": { + "name": "order_key", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "bordered": { + "name": "bordered", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": true + }, + "background": { + "name": "background", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "supported_regions": { + "name": "supported_regions", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "configuration": { + "name": "configuration", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "name_key": { + "name": "name_key", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "mini_app_status_order_key_idx": { + "name": "mini_app_status_order_key_idx", + "columns": ["status", "order_key"], + "isUnique": false + }, + "mini_app_preset_mini_app_id_idx": { + "name": "mini_app_preset_mini_app_id_idx", + "columns": ["preset_mini_app_id"], + "isUnique": false + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": { + "mini_app_status_check": { + "name": "mini_app_status_check", + "value": "\"mini_app\".\"status\" IN ('enabled', 'disabled', 'pinned')" + } + } + }, + "note": { + "name": "note", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "root_path": { + "name": "root_path", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "path": { + "name": "path", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "is_starred": { + "name": "is_starred", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "is_expanded": { + "name": "is_expanded", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "note_root_path_path_unique_idx": { + "name": "note_root_path_path_unique_idx", + "columns": ["root_path", "path"], + "isUnique": true + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": { + "note_has_state_check": { + "name": "note_has_state_check", + "value": "\"note\".\"is_starred\" = 1 OR \"note\".\"is_expanded\" = 1" + } + } + }, + "painting": { + "name": "painting", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "provider_id": { + "name": "provider_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "model_id": { + "name": "model_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "prompt": { + "name": "prompt", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "order_key": { + "name": "order_key", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "painting_order_key_idx": { + "name": "painting_order_key_idx", + "columns": ["order_key"], + "isUnique": false + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "pin": { + "name": "pin", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "entity_type": { + "name": "entity_type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "entity_id": { + "name": "entity_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "order_key": { + "name": "order_key", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "pin_entity_type_entity_id_unique_idx": { + "name": "pin_entity_type_entity_id_unique_idx", + "columns": ["entity_type", "entity_id"], + "isUnique": true + }, + "pin_entity_type_order_key_idx": { + "name": "pin_entity_type_order_key_idx", + "columns": ["entity_type", "order_key"], + "isUnique": false + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "preference": { + "name": "preference", + "columns": { + "scope": { + "name": "scope", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'default'" + }, + "key": { + "name": "key", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "value": { + "name": "value", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": {}, + "foreignKeys": {}, + "compositePrimaryKeys": { + "preference_scope_key_pk": { + "columns": ["scope", "key"], + "name": "preference_scope_key_pk" + } + }, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "prompt": { + "name": "prompt", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "title": { + "name": "title", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "content": { + "name": "content", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "order_key": { + "name": "order_key", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "prompt_order_key_idx": { + "name": "prompt_order_key_idx", + "columns": ["order_key"], + "isUnique": false + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "entity_tag": { + "name": "entity_tag", + "columns": { + "entity_type": { + "name": "entity_type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "entity_id": { + "name": "entity_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "tag_id": { + "name": "tag_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "entity_tag_tag_id_idx": { + "name": "entity_tag_tag_id_idx", + "columns": ["tag_id"], + "isUnique": false + } + }, + "foreignKeys": { + "entity_tag_tag_id_tag_id_fk": { + "name": "entity_tag_tag_id_tag_id_fk", + "tableFrom": "entity_tag", + "tableTo": "tag", + "columnsFrom": ["tag_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": { + "entity_tag_entity_type_entity_id_tag_id_pk": { + "columns": ["entity_type", "entity_id", "tag_id"], + "name": "entity_tag_entity_type_entity_id_tag_id_pk" + } + }, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "tag": { + "name": "tag", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "color": { + "name": "color", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "tag_name_unique": { + "name": "tag_name_unique", + "columns": ["name"], + "isUnique": true + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "topic": { + "name": "topic", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "''" + }, + "is_name_manually_edited": { + "name": "is_name_manually_edited", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "assistant_id": { + "name": "assistant_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "active_node_id": { + "name": "active_node_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "group_id": { + "name": "group_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "trace_id": { + "name": "trace_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "order_key": { + "name": "order_key", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "deleted_at": { + "name": "deleted_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": { + "topic_group_updated_idx": { + "name": "topic_group_updated_idx", + "columns": ["group_id", "updated_at"], + "isUnique": false + }, + "topic_updated_at_idx": { + "name": "topic_updated_at_idx", + "columns": ["updated_at"], + "isUnique": false + }, + "topic_order_key_idx": { + "name": "topic_order_key_idx", + "columns": ["order_key"], + "isUnique": false + }, + "topic_assistant_id_idx": { + "name": "topic_assistant_id_idx", + "columns": ["assistant_id"], + "isUnique": false + } + }, + "foreignKeys": { + "topic_assistant_id_assistant_id_fk": { + "name": "topic_assistant_id_assistant_id_fk", + "tableFrom": "topic", + "tableTo": "assistant", + "columnsFrom": ["assistant_id"], + "columnsTo": ["id"], + "onDelete": "set null", + "onUpdate": "no action" + }, + "topic_group_id_group_id_fk": { + "name": "topic_group_id_group_id_fk", + "tableFrom": "topic", + "tableTo": "group", + "columnsFrom": ["group_id"], + "columnsTo": ["id"], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "translate_history": { + "name": "translate_history", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "source_text": { + "name": "source_text", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "target_text": { + "name": "target_text", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "source_language": { + "name": "source_language", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "target_language": { + "name": "target_language", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "star": { + "name": "star", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "translate_history_created_at_idx": { + "name": "translate_history_created_at_idx", + "columns": ["created_at"], + "isUnique": false + }, + "translate_history_star_created_at_idx": { + "name": "translate_history_star_created_at_idx", + "columns": ["star", "created_at"], + "isUnique": false + } + }, + "foreignKeys": { + "translate_history_source_language_translate_language_lang_code_fk": { + "name": "translate_history_source_language_translate_language_lang_code_fk", + "tableFrom": "translate_history", + "tableTo": "translate_language", + "columnsFrom": ["source_language"], + "columnsTo": ["lang_code"], + "onDelete": "set null", + "onUpdate": "no action" + }, + "translate_history_target_language_translate_language_lang_code_fk": { + "name": "translate_history_target_language_translate_language_lang_code_fk", + "tableFrom": "translate_history", + "tableTo": "translate_language", + "columnsFrom": ["target_language"], + "columnsTo": ["lang_code"], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "translate_language": { + "name": "translate_language", + "columns": { + "lang_code": { + "name": "lang_code", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "value": { + "name": "value", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "emoji": { + "name": "emoji", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": {}, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "user_model": { + "name": "user_model", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "provider_id": { + "name": "provider_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "model_id": { + "name": "model_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "preset_model_id": { + "name": "preset_model_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "description": { + "name": "description", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "group": { + "name": "group", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "capabilities": { + "name": "capabilities", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "input_modalities": { + "name": "input_modalities", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "output_modalities": { + "name": "output_modalities", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "endpoint_types": { + "name": "endpoint_types", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "custom_endpoint_url": { + "name": "custom_endpoint_url", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "context_window": { + "name": "context_window", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "max_input_tokens": { + "name": "max_input_tokens", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "max_output_tokens": { + "name": "max_output_tokens", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "supports_streaming": { + "name": "supports_streaming", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": true + }, + "reasoning": { + "name": "reasoning", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "parameters": { + "name": "parameters", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "pricing": { + "name": "pricing", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "is_enabled": { + "name": "is_enabled", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": true + }, + "is_hidden": { + "name": "is_hidden", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "is_deprecated": { + "name": "is_deprecated", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "order_key": { + "name": "order_key", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "notes": { + "name": "notes", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "user_overrides": { + "name": "user_overrides", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "user_model_preset_idx": { + "name": "user_model_preset_idx", + "columns": ["preset_model_id"], + "isUnique": false + }, + "user_model_provider_enabled_idx": { + "name": "user_model_provider_enabled_idx", + "columns": ["provider_id", "is_enabled"], + "isUnique": false + }, + "user_model_provider_id_order_key_idx": { + "name": "user_model_provider_id_order_key_idx", + "columns": ["provider_id", "order_key"], + "isUnique": false + }, + "user_model_provider_model_unique": { + "name": "user_model_provider_model_unique", + "columns": ["provider_id", "model_id"], + "isUnique": true + } + }, + "foreignKeys": { + "user_model_provider_id_user_provider_provider_id_fk": { + "name": "user_model_provider_id_user_provider_provider_id_fk", + "tableFrom": "user_model", + "tableTo": "user_provider", + "columnsFrom": ["provider_id"], + "columnsTo": ["provider_id"], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "user_provider": { + "name": "user_provider", + "columns": { + "provider_id": { + "name": "provider_id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "preset_provider_id": { + "name": "preset_provider_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "endpoint_configs": { + "name": "endpoint_configs", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "default_chat_endpoint": { + "name": "default_chat_endpoint", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "api_keys": { + "name": "api_keys", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false, + "default": "'[]'" + }, + "auth_config": { + "name": "auth_config", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "api_features": { + "name": "api_features", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "provider_settings": { + "name": "provider_settings", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "is_enabled": { + "name": "is_enabled", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "order_key": { + "name": "order_key", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "user_provider_preset_idx": { + "name": "user_provider_preset_idx", + "columns": ["preset_provider_id"], + "isUnique": false + }, + "user_provider_enabled_idx": { + "name": "user_provider_enabled_idx", + "columns": ["is_enabled"], + "isUnique": false + }, + "user_provider_order_key_idx": { + "name": "user_provider_order_key_idx", + "columns": ["order_key"], + "isUnique": false + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + } + }, + "views": {}, + "enums": {}, + "_meta": { + "schemas": {}, + "tables": {}, + "columns": {} + }, + "internal": { + "indexes": { + "fe_external_path_lower_unique_idx": { + "columns": { + "lower(\"external_path\")": { + "isExpression": true + } + } + } + } + } +} diff --git a/migrations/sqlite-drizzle/meta/_journal.json b/migrations/sqlite-drizzle/meta/_journal.json index 5bd210be24e..9da89e45795 100644 --- a/migrations/sqlite-drizzle/meta/_journal.json +++ b/migrations/sqlite-drizzle/meta/_journal.json @@ -126,6 +126,13 @@ "when": 1782817365873, "tag": "0017_minor_echo", "breakpoints": true + }, + { + "idx": 18, + "version": "6", + "when": 1783173408643, + "tag": "0018_easy_gertrude_yorkes", + "breakpoints": true } ], "version": "7" diff --git a/src/main/core/application/serviceRegistry.ts b/src/main/core/application/serviceRegistry.ts index 7647eb0661d..8ef611b906d 100644 --- a/src/main/core/application/serviceRegistry.ts +++ b/src/main/core/application/serviceRegistry.ts @@ -21,7 +21,7 @@ import { WindowManager } from '@main/core/window/WindowManager' import { ApiGatewayService } from '@main/features/apiGateway/ApiGatewayService' import { FileProcessingService, TesseractRuntimeService } from '@main/features/fileProcessing' import { KnowledgeService } from '@main/features/knowledge' -import { KnowledgeVectorStoreService } from '@main/features/knowledge/vectorstore/KnowledgeVectorStoreService' +import { KnowledgeVectorStoreService } from '@main/features/knowledge/pipeline/vectorstore/KnowledgeVectorStoreService' import { IpcApiService } from '@main/ipc/IpcApiService' import { AnalyticsService } from '@main/services/AnalyticsService' import { AppMenuService } from '@main/services/AppMenuService' diff --git a/src/main/data/db/schemas/knowledge.ts b/src/main/data/db/schemas/knowledge.ts index d27fad0e2fe..7ffeaf9147b 100644 --- a/src/main/data/db/schemas/knowledge.ts +++ b/src/main/data/db/schemas/knowledge.ts @@ -102,20 +102,20 @@ export const knowledgeItemTable = sqliteTable( check('knowledge_item_type_check', sql`${t.type} IN ('file', 'url', 'note', 'directory')`), check( 'knowledge_item_status_check', - sql`${t.status} IN ('idle', 'preparing', 'processing', 'reading', 'embedding', 'completed', 'failed', 'deleting')` + sql`${t.status} IN ('preparing', 'processing', 'reading', 'embedding', 'completed', 'failed', 'deleting')` ), check( 'knowledge_item_type_status_check', sql` - (${t.type} IN ('file', 'url', 'note') AND ${t.status} IN ('idle', 'processing', 'reading', 'embedding', 'completed', 'failed', 'deleting')) - OR (${t.type} = 'directory' AND ${t.status} IN ('idle', 'preparing', 'processing', 'completed', 'failed', 'deleting')) + (${t.type} IN ('file', 'url', 'note') AND ${t.status} IN ('processing', 'reading', 'embedding', 'completed', 'failed', 'deleting')) + OR (${t.type} = 'directory' AND ${t.status} IN ('preparing', 'processing', 'completed', 'failed', 'deleting')) ` ), check( 'knowledge_item_status_error_check', sql` ( - ${t.status} IN ('idle', 'preparing', 'processing', 'reading', 'embedding', 'completed', 'deleting') + ${t.status} IN ('preparing', 'processing', 'reading', 'embedding', 'completed', 'deleting') AND ${t.error} IS NULL ) OR ( diff --git a/src/main/data/migration/v2/migrators/KnowledgeMigrator.ts b/src/main/data/migration/v2/migrators/KnowledgeMigrator.ts index 1f763f012f3..d45657180b4 100644 --- a/src/main/data/migration/v2/migrators/KnowledgeMigrator.ts +++ b/src/main/data/migration/v2/migrators/KnowledgeMigrator.ts @@ -12,7 +12,7 @@ import { loggerService } from '@logger' import { needsProcessedArtifactReservation, reserveImportedFileRelativePath -} from '@main/features/knowledge/utils/storage/pathStorage' +} from '@main/features/knowledge/pathStorage' import { sanitizeFilename } from '@main/utils/file' import { copy, ensureDir } from '@main/utils/file/fs' import type { ExecuteResult, PrepareResult, ValidateResult, ValidationError } from '@shared/data/migration/v2/types' @@ -609,7 +609,7 @@ export class KnowledgeMigrator extends BaseMigrator { // A resolved embedding model whose per-base legacy vector store is missing/empty/locked // yields dimensions===null. We must NOT drop the base (that loses the library with no // recoverable row): keep it as a `failed` row, like the dangling-model branch below, so - // the name/model/config/idle items survive and the UI offers a restore/re-index entry. + // the name/model/config/unindexed items survive and the UI offers a restore/re-index entry. // `vectorsWillMigrate` also gates directory expansion: a base whose vectors will not // migrate must not expand folders into `completed` children that would be empty shells. const vectorStoreUnresolved = embeddingResolution.kind === 'resolved' && resolvedDimensions.dimensions === null diff --git a/src/main/data/migration/v2/migrators/KnowledgeVectorMigrator.ts b/src/main/data/migration/v2/migrators/KnowledgeVectorMigrator.ts index 0ce8306a726..03b65af9b5e 100644 --- a/src/main/data/migration/v2/migrators/KnowledgeVectorMigrator.ts +++ b/src/main/data/migration/v2/migrators/KnowledgeVectorMigrator.ts @@ -3,29 +3,26 @@ import path from 'node:path' import { knowledgeBaseTable, knowledgeItemTable } from '@data/db/schemas/knowledge' import { loggerService } from '@logger' -import { DOCUMENT_SEPARATOR } from '@main/features/knowledge/utils/indexing/chunk' -import { - type MaterialFieldSource, - toMaterialRelativePath -} from '@main/features/knowledge/utils/indexing/materialFields' -import { deriveNoteSnapshotSlug } from '@main/features/knowledge/utils/sources/noteSnapshot' -import { serializeOkfFrontmatter } from '@main/features/knowledge/utils/sources/okfFrontmatter' -import { deriveUrlSnapshotSlug, deriveUrlSnapshotTitle } from '@main/features/knowledge/utils/sources/urlSnapshot' +import { type MaterialFieldSource, toMaterialRelativePath } from '@main/features/knowledge/items' import { assertSafeKnowledgeRelativePath, collectKnowledgeReservedRelativePaths, reserveImportedFileRelativePath -} from '@main/features/knowledge/utils/storage/pathStorage' +} from '@main/features/knowledge/pathStorage' +import { DOCUMENT_SEPARATOR } from '@main/features/knowledge/pipeline/indexing/chunk' +import { deriveNoteSnapshotSlug } from '@main/features/knowledge/pipeline/sources/noteSnapshot' +import { serializeOkfFrontmatter } from '@main/features/knowledge/pipeline/sources/okfFrontmatter' +import { deriveUrlSnapshotSlug, deriveUrlSnapshotTitle } from '@main/features/knowledge/pipeline/sources/urlSnapshot' import { type BetterSqlite3Driver, openBetterSqlite3IndexDriver -} from '@main/features/knowledge/vectorstore/indexStore/BetterSqlite3Driver' -import { betterSqlite3VectorIndex } from '@main/features/knowledge/vectorstore/indexStore/BetterSqlite3VectorIndex' -import { hashEmbeddingText } from '@main/features/knowledge/vectorstore/indexStore/hashing' -import { ensureIndexMeta } from '@main/features/knowledge/vectorstore/indexStore/indexMeta' -import { KnowledgeIndexStore } from '@main/features/knowledge/vectorstore/indexStore/KnowledgeIndexStore' -import type { RebuildMaterialInput } from '@main/features/knowledge/vectorstore/indexStore/model' -import { createKnowledgeIndexSchema } from '@main/features/knowledge/vectorstore/indexStore/schema' +} from '@main/features/knowledge/pipeline/vectorstore/indexStore/BetterSqlite3Driver' +import { betterSqlite3VectorIndex } from '@main/features/knowledge/pipeline/vectorstore/indexStore/BetterSqlite3VectorIndex' +import { hashEmbeddingText } from '@main/features/knowledge/pipeline/vectorstore/indexStore/hashing' +import { ensureIndexMeta } from '@main/features/knowledge/pipeline/vectorstore/indexStore/indexMeta' +import { KnowledgeIndexStore } from '@main/features/knowledge/pipeline/vectorstore/indexStore/KnowledgeIndexStore' +import type { RebuildMaterialInput } from '@main/features/knowledge/pipeline/vectorstore/indexStore/model' +import { createKnowledgeIndexSchema } from '@main/features/knowledge/pipeline/vectorstore/indexStore/schema' import type { ExecuteResult, PrepareResult, ValidateResult, ValidationError } from '@shared/data/migration/v2/types' import { KNOWLEDGE_BASE_ERROR_MISSING_EMBEDDING_MODEL, @@ -48,7 +45,7 @@ import { const logger = loggerService.withContext('KnowledgeVectorMigrator') // Runtime vector store + material layout — source of truth: -// src/main/features/knowledge/utils/storage/pathStorage.ts +// src/main/features/knowledge/pathStorage.ts // (CHERRY_META_DIR / VECTOR_STORE_FILE / MATERIAL_ROOT_DIR). Runtime opens // {knowledgeBaseDir}/{baseId}/.cherry/index.sqlite by the migrated (new) base id, and resolves // every material's bytes at {knowledgeBaseDir}/{baseId}/raw/{relativePath}, so the migrator must @@ -991,7 +988,7 @@ export class KnowledgeVectorMigrator extends BaseMigrator { const store = new KnowledgeIndexStore(driver, betterSqlite3VectorIndex) for (const material of plan.materials) { - await store.rebuildMaterial(material.itemId, material.input) + store.rebuildMaterial(material.itemId, material.input) processedWork += 1 this.reportRebuildProgress(processedWork, totalWork) await yieldToEventLoop() diff --git a/src/main/data/migration/v2/migrators/README-KnowledgeMigrator.md b/src/main/data/migration/v2/migrators/README-KnowledgeMigrator.md index f54e4b7b228..9f5dc7ad28b 100644 --- a/src/main/data/migration/v2/migrators/README-KnowledgeMigrator.md +++ b/src/main/data/migration/v2/migrators/README-KnowledgeMigrator.md @@ -52,7 +52,7 @@ - Legacy `processingStatus` is treated as runtime-only and not trusted for migration. - Item status is inferred from `uniqueId`: - `uniqueId` present and non-empty -> `completed` - - otherwise -> `idle` + - otherwise -> `failed` (`never_indexed`) 6. Vector dimension dependency - Completed bases require a resolved positive `knowledge_base.dimensions` value. @@ -91,7 +91,7 @@ | _no legacy grouping field_ | `groupId` | V1 exports are flat; migrated items are inserted without grouping metadata (`null`) | | `type` | `type` | Supported target types: file/url/note/directory. Legacy sitemap maps to url. | | `content` + Dexie lookups | `data` | Type-specific transform | -| `uniqueId` | `status` | `uniqueId` non-empty => `completed`, otherwise `idle` | +| `uniqueId` | `status` | `uniqueId` non-empty => `completed`, otherwise `failed` (`never_indexed`) | | `processingError` | `error` | Direct copy | | `created_at` | `createdAt` | Timestamp conversion | | `updated_at` | `updatedAt` | Timestamp conversion | diff --git a/src/main/data/migration/v2/migrators/__tests__/KnowledgeMigrator.test.ts b/src/main/data/migration/v2/migrators/__tests__/KnowledgeMigrator.test.ts index e41d3c2fc63..6f44de1d104 100644 --- a/src/main/data/migration/v2/migrators/__tests__/KnowledgeMigrator.test.ts +++ b/src/main/data/migration/v2/migrators/__tests__/KnowledgeMigrator.test.ts @@ -3,7 +3,8 @@ import fs from 'node:fs' import { KNOWLEDGE_BASE_ERROR_MISSING_EMBEDDING_MODEL, KNOWLEDGE_BASE_ERROR_MISSING_VECTOR_STORE, - KNOWLEDGE_ITEM_ERROR_DIRECTORY_NOT_MIGRATED + KNOWLEDGE_ITEM_ERROR_DIRECTORY_NOT_MIGRATED, + KNOWLEDGE_ITEM_ERROR_NEVER_INDEXED } from '@shared/data/types/knowledge' import Database from 'better-sqlite3' import { beforeEach, describe, expect, it, vi } from 'vitest' @@ -856,9 +857,9 @@ describe('KnowledgeMigrator dimensions resolution', () => { ) expect(result.success).toBe(true) - expect(statusByLegacyId.get('i-no-unique-id')).toBe('idle') + expect(statusByLegacyId.get('i-no-unique-id')).toBe('failed') expect(statusByLegacyId.get('i-with-unique-id')).toBe('completed') - expect(statusByLegacyId.get('i-with-empty-unique-id')).toBe('idle') + expect(statusByLegacyId.get('i-with-empty-unique-id')).toBe('failed') expect(statusByLegacyId.get('i-processing-but-no-unique-id')).toBe('failed') expect(statusByLegacyId.get('i-failed-with-unique-id')).toBe('failed') }) @@ -1463,10 +1464,10 @@ describe('KnowledgeMigrator dimensions resolution', () => { ) }) - it('keeps an idle directory (loader ids but no completed marker) as idle without expanding it', async () => { + it('keeps a never-indexed directory (loader ids but no completed marker) as failed without expanding it', async () => { // Expansion keys off the `completed` marker (singular `uniqueId`), not the mere presence of - // child loader ids (plural `uniqueIds`). A folder with loader ids but no completed marker is - // `idle`, so the gate must NOT expand it even when the loader sources resolve. + // child loader ids (plural `uniqueIds`). A folder with loader ids but no completed marker never + // started indexing, so the gate must NOT expand it even when the loader sources resolve. const migrator = new KnowledgeMigrator() as any vi.spyOn(migrator, 'resolveDimensionsForBase').mockResolvedValue({ dimensions: 1024, reason: 'ok' }) vi.spyOn(migrator, 'loadLoaderSourceMap').mockResolvedValue({ @@ -1507,12 +1508,12 @@ describe('KnowledgeMigrator dimensions resolution', () => { expect(result.success).toBe(true) const migratedId = migrator.legacyItemIdRemap.get('item-directory') - // Not expanded: a single idle directory item, no synthesized children, no loader remap. + // Not expanded: a single never-indexed directory item, no synthesized children, no loader remap. expect(migrator.preparedItems.filter((i: any) => i.groupId === migratedId)).toHaveLength(0) expect(migrator.preparedItems.find((i: any) => i.id === migratedId)).toMatchObject({ type: 'directory', - status: 'idle', - error: null + status: 'failed', + error: KNOWLEDGE_ITEM_ERROR_NEVER_INDEXED }) expect(migrator.directoryChildLoaderRemap.size).toBe(0) }) @@ -1932,7 +1933,7 @@ describe('KnowledgeMigrator execute/validate paths', () => { groupId: null, type: 'note', data: { content: 'n1' }, - status: 'idle' + status: 'processing' }, { id: 'item-2', @@ -1940,7 +1941,7 @@ describe('KnowledgeMigrator execute/validate paths', () => { groupId: null, type: 'note', data: { content: 'n2' }, - status: 'idle' + status: 'processing' } ] @@ -2049,7 +2050,7 @@ describe('KnowledgeMigrator execute/validate paths', () => { groupId: null, type: 'note', data: { source: 'n1', content: 'n1' }, - status: 'idle', + status: 'processing', error: null } ] @@ -2140,7 +2141,7 @@ describe('KnowledgeMigrator execute/validate paths', () => { groupId: null, type: 'note', data: { source: 'note', content: 'note' }, - status: 'idle', + status: 'processing', error: null, createdAt: 1775114958369, updatedAt: 1775114958369 @@ -2175,7 +2176,7 @@ describe('KnowledgeMigrator execute/validate paths', () => { expect.objectContaining({ id: 'item-1', baseId: 'kb-missing-model', - status: 'idle' + status: 'processing' }) ] ]) @@ -2204,7 +2205,7 @@ describe('KnowledgeMigrator execute/validate paths', () => { groupId: null, type: 'note', data: { content: 'n1' }, - status: 'idle' + status: 'processing' }, { id: 'item-2', @@ -2212,7 +2213,7 @@ describe('KnowledgeMigrator execute/validate paths', () => { groupId: null, type: 'note', data: { content: 'n2' }, - status: 'idle' + status: 'processing' } ] @@ -2368,7 +2369,7 @@ describe('KnowledgeMigrator file item path storage', () => { groupId: null, type: 'file', data: { source: '/tmp/a.pdf', relativePath: 'a.pdf' }, - status: 'idle' + status: 'processing' }, { id: 'item-b', @@ -2376,7 +2377,7 @@ describe('KnowledgeMigrator file item path storage', () => { groupId: null, type: 'file', data: { source: '/tmp/b.pdf', relativePath: 'b.pdf', indexedRelativePath: 'b.md' }, - status: 'idle' + status: 'processing' }, { id: 'item-note', @@ -2384,7 +2385,7 @@ describe('KnowledgeMigrator file item path storage', () => { groupId: null, type: 'note', data: { source: 'some note', content: 'some note' }, - status: 'idle' + status: 'processing' } ] @@ -2419,7 +2420,7 @@ describe('KnowledgeMigrator file item path storage', () => { groupId: null, type: 'file', data: { source: '/tmp/ok.pdf', relativePath: 'ok.pdf' }, - status: 'idle' + status: 'processing' }, { id: 'item-skipped-file-entry', @@ -2427,7 +2428,7 @@ describe('KnowledgeMigrator file item path storage', () => { groupId: null, type: 'file', data: { source: '/tmp/bad.xyz', relativePath: 'bad.xyz' }, - status: 'idle' + status: 'processing' } ] migrator.legacyItemIdRemap = new Map([ diff --git a/src/main/data/migration/v2/migrators/__tests__/KnowledgeVectorMigrator.test.ts b/src/main/data/migration/v2/migrators/__tests__/KnowledgeVectorMigrator.test.ts index 5212862cfe8..3d1df3fed24 100644 --- a/src/main/data/migration/v2/migrators/__tests__/KnowledgeVectorMigrator.test.ts +++ b/src/main/data/migration/v2/migrators/__tests__/KnowledgeVectorMigrator.test.ts @@ -2,10 +2,10 @@ import * as fs from 'node:fs' import * as os from 'node:os' import path from 'node:path' -import { stripOkfFrontmatter } from '@main/features/knowledge/utils/sources/okfFrontmatter' -import { hashEmbeddingText } from '@main/features/knowledge/vectorstore/indexStore/hashing' -import { KnowledgeIndexStore } from '@main/features/knowledge/vectorstore/indexStore/KnowledgeIndexStore' -import { encodeVectorBlob } from '@main/features/knowledge/vectorstore/indexStore/vectorBlob' +import { stripOkfFrontmatter } from '@main/features/knowledge/pipeline/sources/okfFrontmatter' +import { hashEmbeddingText } from '@main/features/knowledge/pipeline/vectorstore/indexStore/hashing' +import { KnowledgeIndexStore } from '@main/features/knowledge/pipeline/vectorstore/indexStore/KnowledgeIndexStore' +import { encodeVectorBlob } from '@main/features/knowledge/pipeline/vectorstore/indexStore/vectorBlob' import { KNOWLEDGE_BASE_ERROR_MISSING_EMBEDDING_MODEL, KNOWLEDGE_BASE_ERROR_MISSING_VECTOR_STORE, @@ -73,7 +73,7 @@ function createTempRoot() { } // Mirrors the runtime vector store layout in -// src/main/features/knowledge/utils/storage/pathStorage.ts: {root}/{baseId}/.cherry/index.sqlite. +// src/main/features/knowledge/pathStorage.ts: {root}/{baseId}/.cherry/index.sqlite. // Read-back assertions use this so they fail if the migrator ever writes to a path the runtime // would not open — the exact bug this regression guards against. function runtimeVectorStorePath(baseId: string): string { @@ -1513,7 +1513,9 @@ describe('KnowledgeVectorMigrator', () => { expect([...migrator.directoryItemsToDegrade]).toEqual([]) expect(materialItemIds(migrator)).toEqual([CHILD_A]) - vi.spyOn(KnowledgeIndexStore.prototype, 'rebuildMaterial').mockRejectedValueOnce(new Error('rebuild failed')) + vi.spyOn(KnowledgeIndexStore.prototype, 'rebuildMaterial').mockImplementationOnce(() => { + throw new Error('rebuild failed') + }) const executeResult = await migrator.execute(migrationCtx as any) expect(executeResult.success).toBe(true) @@ -1883,7 +1885,9 @@ describe('KnowledgeVectorMigrator', () => { const migrator = new KnowledgeVectorMigrator() as any expect((await migrator.prepare(migrationCtx as any)).success).toBe(true) - vi.spyOn(KnowledgeIndexStore.prototype, 'rebuildMaterial').mockRejectedValueOnce(new Error('rebuild failed')) + vi.spyOn(KnowledgeIndexStore.prototype, 'rebuildMaterial').mockImplementationOnce(() => { + throw new Error('rebuild failed') + }) // A per-base failure is non-fatal (P1-6): execute succeeds overall, the failed base is left // out of successfulBaseIds (so validate never checks it), and its error surfaces as a warning @@ -1944,7 +1948,9 @@ describe('KnowledgeVectorMigrator', () => { expect((await migrator.prepare(migrationCtx as any)).success).toBe(true) // The rebuild fails (the real error), sending execute into its catch block... - vi.spyOn(KnowledgeIndexStore.prototype, 'rebuildMaterial').mockRejectedValueOnce(new Error('rebuild failed')) + vi.spyOn(KnowledgeIndexStore.prototype, 'rebuildMaterial').mockImplementationOnce(() => { + throw new Error('rebuild failed') + }) // ...where the partial-store cleanup itself also throws (e.g. a Windows-locked index.sqlite). // The first call (pre-build target clear) must still succeed so the rebuild is reached. migrator.removeIndexStoreFiles = vi @@ -2046,7 +2052,9 @@ describe('KnowledgeVectorMigrator', () => { const migrator = new KnowledgeVectorMigrator() as any expect((await migrator.prepare(migrationCtx as any)).success).toBe(true) - vi.spyOn(KnowledgeIndexStore.prototype, 'rebuildMaterial').mockRejectedValueOnce(new Error('rebuild failed')) + vi.spyOn(KnowledgeIndexStore.prototype, 'rebuildMaterial').mockImplementationOnce(() => { + throw new Error('rebuild failed') + }) const executeResult = await migrator.execute(migrationCtx as any) expect(executeResult.success).toBe(true) @@ -2094,7 +2102,9 @@ describe('KnowledgeVectorMigrator', () => { // Fail the rebuild itself: the per-base catch marks the base failed when its store never // finished building (storePromoted stays false). - vi.spyOn(KnowledgeIndexStore.prototype, 'rebuildMaterial').mockRejectedValueOnce(new Error('rebuild failed')) + vi.spyOn(KnowledgeIndexStore.prototype, 'rebuildMaterial').mockImplementationOnce(() => { + throw new Error('rebuild failed') + }) const executeResult = await migrator.execute(migrationCtx as any) expect(executeResult.success).toBe(true) @@ -2182,7 +2192,9 @@ describe('KnowledgeVectorMigrator', () => { // real rebuild so its store is written for real. const realRebuild = KnowledgeIndexStore.prototype.rebuildMaterial vi.spyOn(KnowledgeIndexStore.prototype, 'rebuildMaterial') - .mockRejectedValueOnce(new Error('base a rebuild failed')) + .mockImplementationOnce(() => { + throw new Error('base a rebuild failed') + }) .mockImplementation(function (this: KnowledgeIndexStore, ...args: Parameters) { return realRebuild.apply(this, args) }) diff --git a/src/main/data/migration/v2/migrators/mappings/KnowledgeMappings.ts b/src/main/data/migration/v2/migrators/mappings/KnowledgeMappings.ts index f3240862b28..6685c728f65 100644 --- a/src/main/data/migration/v2/migrators/mappings/KnowledgeMappings.ts +++ b/src/main/data/migration/v2/migrators/mappings/KnowledgeMappings.ts @@ -8,6 +8,7 @@ import { DEFAULT_KNOWLEDGE_SEARCH_MODE, KNOWLEDGE_BASE_ERROR_MISSING_EMBEDDING_MODEL, KNOWLEDGE_ITEM_ERROR_DIRECTORY_NOT_MIGRATED, + KNOWLEDGE_ITEM_ERROR_NEVER_INDEXED, KNOWLEDGE_NOTE_CONTENT_MAX, type KnowledgeItemData, type KnowledgeItemStatus @@ -143,7 +144,7 @@ export const inferKnowledgeItemStatus = ( return 'failed' } - return typeof item.uniqueId === 'string' && item.uniqueId.trim() !== '' ? 'completed' : 'idle' + return typeof item.uniqueId === 'string' && item.uniqueId.trim() !== '' ? 'completed' : 'failed' } const normalizeKnowledgeItemError = ( @@ -421,9 +422,14 @@ export const transformKnowledgeItem = ( // home), so letting the directory claim `completed` would leave an empty shell // that never re-indexes. Mark it `failed` with a code the UI renders as a // delete-and-re-upload prompt (it migrated as a record but its vectors were dropped). - // Interrupted (failed) and never-indexed (idle) directories keep their inferred status - // (only a `completed` directory is overridden to `failed`). + // Interrupted and never-indexed items keep their inferred `failed` status (only a + // `completed` directory is overridden to `failed`). const directoryIndexDropped = type === 'directory' && inferredStatus === 'completed' + const wasInterrupted = + item.processingStatus === 'failed' || item.processingStatus === 'processing' || item.processingStatus === 'pending' + // A v1 item that never started indexing (no uniqueId, not interrupted) has nothing to + // resume from; mark it with a code the UI renders as a re-index prompt. + const neverIndexed = !directoryIndexDropped && inferredStatus === 'failed' && !wasInterrupted const status = directoryIndexDropped ? 'failed' : inferredStatus return { @@ -439,7 +445,9 @@ export const transformKnowledgeItem = ( status, error: directoryIndexDropped ? KNOWLEDGE_ITEM_ERROR_DIRECTORY_NOT_MIGRATED - : normalizeKnowledgeItemError(status, item.processingStatus, item.processingError), + : neverIndexed + ? KNOWLEDGE_ITEM_ERROR_NEVER_INDEXED + : normalizeKnowledgeItemError(status, item.processingStatus, item.processingError), createdAt: toTimestamp(item.created_at), updatedAt: toTimestamp(item.updated_at) }, diff --git a/src/main/data/migration/v2/migrators/mappings/__tests__/KnowledgeMappings.test.ts b/src/main/data/migration/v2/migrators/mappings/__tests__/KnowledgeMappings.test.ts index e87dd7e7f53..27d498e31b2 100644 --- a/src/main/data/migration/v2/migrators/mappings/__tests__/KnowledgeMappings.test.ts +++ b/src/main/data/migration/v2/migrators/mappings/__tests__/KnowledgeMappings.test.ts @@ -2,6 +2,7 @@ import { FILE_TYPE } from '@shared/data/types/file' import { KNOWLEDGE_BASE_ERROR_MISSING_EMBEDDING_MODEL, KNOWLEDGE_ITEM_ERROR_DIRECTORY_NOT_MIGRATED, + KNOWLEDGE_ITEM_ERROR_NEVER_INDEXED, KNOWLEDGE_NOTE_CONTENT_MAX } from '@shared/data/types/knowledge' import { describe, expect, it } from 'vitest' @@ -42,13 +43,13 @@ describe('KnowledgeMappings', () => { expect(legacyModelToUniqueId({ id: 'silicon::BAAI/bge-m3', provider: 'silicon' })).toBe('silicon::BAAI/bge-m3') }) - it('inferKnowledgeItemStatus maps legacy transient states to failed', () => { + it('inferKnowledgeItemStatus maps legacy transient and never-indexed states to failed', () => { expect(inferKnowledgeItemStatus({ uniqueId: 'loader-1' } as any)).toBe('completed') - expect(inferKnowledgeItemStatus({ uniqueId: ' ' } as any)).toBe('idle') + expect(inferKnowledgeItemStatus({ uniqueId: ' ' } as any)).toBe('failed') expect(inferKnowledgeItemStatus({ processingStatus: 'pending' } as any)).toBe('failed') expect(inferKnowledgeItemStatus({ processingStatus: 'processing' } as any)).toBe('failed') expect(inferKnowledgeItemStatus({ processingStatus: 'failed', uniqueId: 'loader-1' } as any)).toBe('failed') - expect(inferKnowledgeItemStatus({} as any)).toBe('idle') + expect(inferKnowledgeItemStatus({} as any)).toBe('failed') }) it('transformKnowledgeBase marks knowledge bases without an embedding model as failed', () => { @@ -281,8 +282,8 @@ describe('KnowledgeMappings', () => { source: 'https://dexie.example.com', content: 'dexie-content' }, - status: 'idle', - error: null, + status: 'failed', + error: KNOWLEDGE_ITEM_ERROR_NEVER_INDEXED, createdAt: expect.any(Number), updatedAt: expect.any(Number) } @@ -514,13 +515,13 @@ describe('KnowledgeMappings', () => { expect(warnings[0]).toContain('blank v1 filename') }) - it('transformKnowledgeItem clears blank legacy processing errors for idle and completed items', () => { - const idleResult = transformKnowledgeItem( + it('transformKnowledgeItem marks a never-started item never_indexed (ignoring any stale processing error) and clears blank legacy processing errors for completed items', () => { + const neverIndexedResult = transformKnowledgeItem( 'kb-1', { - id: 'idle-note', + id: 'never-indexed-note', type: 'note', - content: 'idle note', + content: 'never indexed note', processingError: '' }, { @@ -543,11 +544,11 @@ describe('KnowledgeMappings', () => { } ) - expect(idleResult).toStrictEqual({ + expect(neverIndexedResult).toStrictEqual({ ok: true, value: expect.objectContaining({ - status: 'idle', - error: null + status: 'failed', + error: KNOWLEDGE_ITEM_ERROR_NEVER_INDEXED }) }) expect(completedResult).toStrictEqual({ @@ -651,7 +652,7 @@ describe('KnowledgeMappings', () => { }) }) - it('transformKnowledgeItem maps directory items to v2 directory node data', () => { + it('transformKnowledgeItem maps a never-indexed directory to v2 directory node data, marked never_indexed', () => { const result = transformKnowledgeItem( 'kb-1', { @@ -675,8 +676,8 @@ describe('KnowledgeMappings', () => { data: { source: '/tmp/docs' }, - status: 'idle', - error: null, + status: 'failed', + error: KNOWLEDGE_ITEM_ERROR_NEVER_INDEXED, createdAt: expect.any(Number), updatedAt: expect.any(Number) } diff --git a/src/main/data/services/KnowledgeBaseService.ts b/src/main/data/services/KnowledgeBaseService.ts index 5529eded56d..857e35591a5 100644 --- a/src/main/data/services/KnowledgeBaseService.ts +++ b/src/main/data/services/KnowledgeBaseService.ts @@ -7,7 +7,7 @@ import { application } from '@application' import { knowledgeBaseTable, knowledgeItemTable } from '@data/db/schemas/knowledge' import { loggerService } from '@logger' -import { DataApiErrorFactory } from '@shared/data/api' +import { DataApiErrorFactory, toDataApiError } from '@shared/data/api' import type { OffsetPaginationResponse } from '@shared/data/api/apiTypes' import type { KnowledgeBaseListItem, @@ -24,7 +24,8 @@ import { DEFAULT_KNOWLEDGE_CHUNK_STRATEGY, DEFAULT_KNOWLEDGE_SEARCH_MODE, type KnowledgeBase, - KnowledgeBaseSchema + KnowledgeBaseSchema, + KnowledgeBaseWriteSchema } from '@shared/data/types/knowledge' import { and, asc, count as sqlCount, desc, eq, gte, ne, type SQL, sql } from 'drizzle-orm' @@ -35,63 +36,6 @@ const logger = loggerService.withContext('DataApi:KnowledgeBaseService') type KnowledgeBaseRow = typeof knowledgeBaseTable.$inferSelect type KnowledgeBaseEntitySearchItem = Extract -function validateKnowledgeBaseConfig(config: { - chunkSize: number - chunkOverlap: number - chunkStrategy?: string | null - chunkSeparator?: string | null - searchMode?: string | null - hybridAlpha?: number | null -}): Record { - const fieldErrors: Record = {} - - if (config.chunkOverlap >= config.chunkSize) { - fieldErrors.chunkOverlap = ['Chunk overlap must be smaller than chunk size'] - } - - if (config.chunkStrategy === 'delimiter' && !config.chunkSeparator) { - fieldErrors.chunkSeparator = ['Separator is required when chunk strategy is delimiter'] - } - - if (config.hybridAlpha != null && config.searchMode !== 'hybrid') { - fieldErrors.hybridAlpha = ['Hybrid alpha requires hybrid search mode'] - } - - return fieldErrors -} - -// Vector and hybrid retrieval need an embedding model; without one a base is -// BM25-only and cannot run a non-bm25 search mode. Mirrors the `completed`-only -// gate in `KnowledgeBaseSchema.superRefine`: a failed base's leftover searchMode -// isn't governed by this invariant until it goes through restore, so callers -// must only apply it to a base that is (or will become) completed. Only -// update() calls this: create() always coerces searchMode to 'bm25' up front -// when there is no model, so the invariant already holds by construction there. -function validateSearchModeNeedsEmbedding( - embeddingModelId: string | null, - searchMode: string | null | undefined -): Record { - if (embeddingModelId == null && searchMode != null && searchMode !== 'bm25') { - return { searchMode: ['A knowledge base without an embedding model can only use bm25 search'] } - } - return {} -} - -// The vector arm of the DB CHECK requires a positive dimensions alongside the model; -// a no-model base always persists a null dimensions regardless of what is passed. The -// IPC boundary already rejects a model without dimensions via CreateKnowledgeBaseSchema's -// refine, so this guards internal callers (e.g. restoreBase) that build a DTO directly, -// before the write reaches the DB CHECK as an untranslated constraint violation. -function validateDimensionsForEmbeddingModel( - embeddingModelId: string | null, - dimensions: number | null | undefined -): Record { - if (embeddingModelId != null && !(typeof dimensions === 'number' && Number.isInteger(dimensions) && dimensions > 0)) { - return { dimensions: ['A knowledge base with an embedding model requires positive dimensions'] } - } - return {} -} - function rowToKnowledgeBase(row: KnowledgeBaseRow): KnowledgeBase { const clean = nullsToUndefined(row) return KnowledgeBaseSchema.parse({ @@ -223,17 +167,6 @@ export class KnowledgeBaseService { searchMode: usesEmbeddings ? (dto.searchMode ?? DEFAULT_KNOWLEDGE_SEARCH_MODE) : 'bm25', hybridAlpha: usesEmbeddings ? dto.hybridAlpha : undefined } - const createFieldErrors = { - // Validated against the raw dto.hybridAlpha, not the coerced createConfig value - // below, so an explicit hybridAlpha on a no-model base is rejected instead of - // silently discarded — create() and update() reject the same input shape. - ...validateKnowledgeBaseConfig({ ...createConfig, hybridAlpha: dto.hybridAlpha }), - ...validateDimensionsForEmbeddingModel(embeddingModelId, dto.dimensions) - } - if (Object.keys(createFieldErrors).length > 0) { - throw DataApiErrorFactory.validation(createFieldErrors) - } - const createValues: Omit = { name: dto.name.trim(), groupId: dto.groupId ?? null, @@ -253,6 +186,20 @@ export class KnowledgeBaseService { hybridAlpha: createConfig.hybridAlpha ?? null } + // Validated against the raw dto.hybridAlpha, not the coerced createValues value, + // so an explicit hybridAlpha on a no-model base is rejected instead of silently + // discarded — create() and update() reject the same input shape. + const createCandidate = { + ...createValues, + hybridAlpha: dto.hybridAlpha ?? undefined, + threshold: createValues.threshold ?? undefined, + documentCount: createValues.documentCount ?? undefined + } + const createValidation = KnowledgeBaseWriteSchema.safeParse(createCandidate) + if (!createValidation.success) { + throw toDataApiError(createValidation.error, 'create knowledge base') + } + const db = application.get('DbService').getDb() const [row] = db.insert(knowledgeBaseTable).values(createValues).returning().all() @@ -307,19 +254,30 @@ export class KnowledgeBaseService { nextConfig.hybridAlpha = null } - // Only a completed base is governed by the no-model=>bm25 invariant (mirrors - // KnowledgeBaseSchema.superRefine's own completed-only gate): a failed base - // may carry a leftover incompatible searchMode from before it failed/migrated, - // and metadata-only updates (rename, move group) must not be blocked by it. - const updateFieldErrors = { - ...validateKnowledgeBaseConfig(nextConfig), - ...validateDimensionsForEmbeddingModel(nextEmbeddingModelId, nextDimensions), - ...(existing.status === 'completed' - ? validateSearchModeNeedsEmbedding(nextEmbeddingModelId, nextConfig.searchMode) - : {}) + // Validate the merged next-state (existing row + this PATCH) against the same + // invariants as the read schema — a failed base's leftover-incompatible pairing + // or searchMode isn't governed by these invariants until it goes through + // restore, so metadata-only updates (rename, move group) must not be blocked by + // them; that gating lives inside `refineKnowledgeBaseInvariants` itself (only + // enforced when `status === 'completed'`). + const { id: _id, createdAt: _createdAt, updatedAt: _updatedAt, ...existingConfig } = existing + void _id // Intentionally unused - excluding id/createdAt/updatedAt from the write candidate + void _createdAt + void _updatedAt + const updateCandidate = { + ...existingConfig, + embeddingModelId: nextEmbeddingModelId, + dimensions: nextDimensions, + chunkSize: nextConfig.chunkSize, + chunkOverlap: nextConfig.chunkOverlap, + chunkStrategy: nextConfig.chunkStrategy, + chunkSeparator: nextConfig.chunkSeparator, + searchMode: nextConfig.searchMode, + hybridAlpha: nextConfig.hybridAlpha ?? undefined } - if (Object.keys(updateFieldErrors).length > 0) { - throw DataApiErrorFactory.validation(updateFieldErrors) + const updateValidation = KnowledgeBaseWriteSchema.safeParse(updateCandidate) + if (!updateValidation.success) { + throw toDataApiError(updateValidation.error, 'update knowledge base') } const updates: Partial = {} diff --git a/src/main/data/services/KnowledgeItemService.ts b/src/main/data/services/KnowledgeItemService.ts index f25af098da7..cea463b2b58 100644 --- a/src/main/data/services/KnowledgeItemService.ts +++ b/src/main/data/services/KnowledgeItemService.ts @@ -7,7 +7,7 @@ import { application } from '@application' import { knowledgeItemTable } from '@data/db/schemas/knowledge' import { type SqliteErrorHandlers, withSqliteErrors } from '@data/db/sqliteErrors' -import type { DbType } from '@data/db/types' +import type { DbOrTx, DbType } from '@data/db/types' import { loggerService } from '@logger' import { DataApiErrorFactory } from '@shared/data/api' import type { KnowledgeItemListResponse, ListKnowledgeItemsQuery } from '@shared/data/api/schemas/knowledges' @@ -29,8 +29,8 @@ const logger = loggerService.withContext('DataApi:KnowledgeItemService') const CONTAINER_CHILD_FAILURE_ERROR = 'One or more child items failed' /** * Item statuses that mean "indexing is in progress" — the item is being driven - * by a job and is neither finished (completed/failed), idle (never started), nor - * being deleted. Used to find items stranded by an interrupted job on boot. + * by a job and is neither finished (completed/failed) nor being deleted. Used to + * find items stranded by an interrupted job on boot. */ const KNOWLEDGE_ITEM_IN_FLIGHT_STATUSES = [ 'preparing', @@ -218,7 +218,7 @@ export class KnowledgeItemService { * A single UPDATE covers leaves and their ancestor containers together: a * container with an in-flight descendant is itself in-flight * (`reconcileContainers` keeps it `processing` while any child is active), so - * failing every in-flight row leaves no completed/idle container pointing at a + * failing every in-flight row leaves no completed container pointing at a * failed child — the rollup invariant holds without a separate reconcile pass. * * @returns the number of items marked failed. @@ -245,7 +245,9 @@ export class KnowledgeItemService { return updatedRows.length } - create(baseId: string, item: CreateKnowledgeItemDto): KnowledgeItem { + /** Create an item already in its active status (`preparing` for a directory container, `processing` for a leaf) and reconcile its parent container so an add into an already-completed directory pulls it back into an active status. */ + createActive(baseId: string, item: CreateKnowledgeItemDto): KnowledgeItem { + const status: KnowledgeItemStatus = item.type === 'directory' ? 'preparing' : 'processing' const dbService = application.get('DbService') const row = dbService.withWriteTx((tx) => { this.validateGroupOwnerTx(tx, baseId, item.groupId) @@ -259,7 +261,7 @@ export class KnowledgeItemService { groupId: item.groupId ?? null, type: item.type, data: item.data, - status: 'idle', + status, error: null }) .returning() @@ -289,7 +291,8 @@ export class KnowledgeItemService { return insertedRow }) - logger.info('Created knowledge item', { baseId, id: row.id, type: row.type }) + this.reconcileContainers(baseId, [row.id, row.groupId]) + logger.info('Created active knowledge item', { baseId, id: row.id, type: row.type, status }) return rowToKnowledgeItem(row) } @@ -355,6 +358,27 @@ export class KnowledgeItemService { rootIds: string[], status: 'deleting' | 'failed', update: FailedKnowledgeItemStatusUpdate | undefined = undefined + ): string[] { + if (status === 'failed') { + return this.setSubtreeStatusTx(this.db, baseId, rootIds, status, update as FailedKnowledgeItemStatusUpdate) + } + return this.setSubtreeStatusTx(this.db, baseId, rootIds, status) + } + + setSubtreeStatusTx(tx: DbOrTx, baseId: string, rootIds: string[], status: 'deleting', update?: never): string[] + setSubtreeStatusTx( + tx: DbOrTx, + baseId: string, + rootIds: string[], + status: 'failed', + update: FailedKnowledgeItemStatusUpdate + ): string[] + setSubtreeStatusTx( + tx: DbOrTx, + baseId: string, + rootIds: string[], + status: 'deleting' | 'failed', + update: FailedKnowledgeItemStatusUpdate | undefined = undefined ): string[] { const error = status === 'failed' ? update?.error.trim() : null @@ -369,8 +393,7 @@ export class KnowledgeItemService { return [] } - const db = application.get('DbService').getDb() - const updatedRows = db.all<{ id: string; groupId: string | null }>(sql` + const updatedRows = tx.all<{ id: string; groupId: string | null }>(sql` WITH RECURSIVE subtree AS ( SELECT id FROM knowledge_item @@ -508,7 +531,7 @@ export class KnowledgeItemService { } const dbService = application.get('DbService') - const { item, startContainerIds } = dbService.withWriteTx((tx) => { + const { item, startContainerIds, blocked } = dbService.withWriteTx((tx) => { const [existingRow] = tx.select().from(knowledgeItemTable).where(eq(knowledgeItemTable.id, id)).limit(1).all() if (!existingRow) { @@ -518,7 +541,8 @@ export class KnowledgeItemService { if (existingRow.status === 'deleting' && status !== 'deleting') { return { item: rowToKnowledgeItem(existingRow), - startContainerIds: [] + startContainerIds: [], + blocked: true } } @@ -541,12 +565,17 @@ export class KnowledgeItemService { startContainerIds: status === 'failed' && updatedRow.type === 'directory' ? [existingRow.groupId] - : [updatedRow.id, existingRow.groupId] + : [updatedRow.id, existingRow.groupId], + blocked: false } }) - this.reconcileContainers(item.baseId, startContainerIds) - logger.info('Updated knowledge item status', { id, status }) + if (blocked) { + logger.warn('Skipped status update for deleting item', { id, attemptedStatus: status }) + } else { + this.reconcileContainers(item.baseId, startContainerIds) + logger.info('Updated knowledge item status', { id, status }) + } return item } diff --git a/src/main/data/services/__tests__/KnowledgeBaseService.test.ts b/src/main/data/services/__tests__/KnowledgeBaseService.test.ts index c0bbdba5932..9998eea3cdb 100644 --- a/src/main/data/services/__tests__/KnowledgeBaseService.test.ts +++ b/src/main/data/services/__tests__/KnowledgeBaseService.test.ts @@ -345,6 +345,12 @@ describe('KnowledgeBaseService', () => { expect(() => service.getById(KNOWLEDGE_BASE_ID)).toThrow('Chunk overlap must be smaller than chunk size') }) + + it('should reject an invalid persisted separator at the read boundary', async () => { + await seedKnowledgeBase({ chunkStrategy: 'delimiter', chunkSeparator: '' }) + + expect(() => service.getById(KNOWLEDGE_BASE_ID)).toThrow('Separator is required when chunk strategy is delimiter') + }) }) describe('create', () => { @@ -453,17 +459,17 @@ describe('KnowledgeBaseService', () => { code: ErrorCode.VALIDATION_ERROR, details: { fieldErrors: { - dimensions: ['A knowledge base with an embedding model requires positive dimensions'] + dimensions: ['Embedding model and dimensions must be set together'] } } }) }) it.each([ - ['zero', 0], - ['negative', -1], - ['non-integer', 1.5] - ])('rejects an embedding model with %s dimensions', (_label, dimensions) => { + ['zero', 0, 'Too small: expected number to be >0'], + ['negative', -1, 'Too small: expected number to be >0'], + ['non-integer', 1.5, 'Invalid input: expected int, received number'] + ])('rejects an embedding model with %s dimensions', (_label, dimensions, message) => { let err: unknown try { service.create({ @@ -478,7 +484,7 @@ describe('KnowledgeBaseService', () => { code: ErrorCode.VALIDATION_ERROR, details: { fieldErrors: { - dimensions: ['A knowledge base with an embedding model requires positive dimensions'] + dimensions: [message] } } }) @@ -769,7 +775,7 @@ describe('KnowledgeBaseService', () => { code: ErrorCode.VALIDATION_ERROR, details: { fieldErrors: { - searchMode: ['A knowledge base without an embedding model can only use bm25 search'] + searchMode: ['A knowledge base without an embedding model must use bm25 search mode'] } } }) @@ -798,6 +804,24 @@ describe('KnowledgeBaseService', () => { expect(result.searchMode).toBe('hybrid') }) + it('allows renaming or moving a recoverable failed base despite a leftover mismatched embedding/dimensions pair', async () => { + // A migration-failed base can carry a leftover embeddingModelId with null + // dimensions (or vice versa); the DB CHECK only constrains this pairing for + // completed bases, so a plain metadata PATCH must not resurrect that check + // (the pairing invariant is still enforced for completed bases — see "rejects + // an embedding model without positive dimensions" above). + await seedKnowledgeBase({ + embeddingModelId: createUniqueModelId('openai', 'embed-model'), + dimensions: null, + status: 'failed', + error: KNOWLEDGE_BASE_ERROR_MISSING_EMBEDDING_MODEL + }) + + const result = service.update(KNOWLEDGE_BASE_ID, { name: 'Renamed while failed' }) + expect(result.name).toBe('Renamed while failed') + expect(result.dimensions).toBeNull() + }) + it('should reject shrinking chunkSize when the existing chunkOverlap no longer fits', async () => { await seedKnowledgeBase({ chunkSize: 256, chunkOverlap: 120 }) diff --git a/src/main/data/services/__tests__/KnowledgeItemService.test.ts b/src/main/data/services/__tests__/KnowledgeItemService.test.ts index c7e2b8edeec..57c6b1519e6 100644 --- a/src/main/data/services/__tests__/KnowledgeItemService.test.ts +++ b/src/main/data/services/__tests__/KnowledgeItemService.test.ts @@ -7,6 +7,7 @@ import { ErrorCode } from '@shared/data/api' import type { CreateKnowledgeItemDto } from '@shared/data/types/knowledge' import { createUniqueModelId } from '@shared/data/types/model' import { setupTestDatabase } from '@test-helpers/db' +import { mockMainLoggerService } from '@test-mocks/MainLoggerService' import { eq } from 'drizzle-orm' import { beforeEach, describe, expect, it, vi } from 'vitest' @@ -79,7 +80,7 @@ describe('KnowledgeItemService', () => { groupId: null, type: 'note', data: { source: 'seed-note', content: 'hello world' }, - status: 'idle', + status: 'processing', error: null, ...overrides } @@ -229,7 +230,7 @@ describe('KnowledgeItemService', () => { groupId: null, type: 'note', data: { source: ITEM_1_ID, content: 'item 1' }, - status: 'idle', + status: 'processing', error: null }) }) @@ -383,12 +384,11 @@ describe('KnowledgeItemService', () => { const READING_LEAF = itemId('7e02') const EMBEDDING_LEAF = itemId('7e03') const PROCESSING_LEAF = itemId('7e04') - const IDLE_LEAF = itemId('7e05') const COMPLETED_LEAF = itemId('7e06') const FAILED_LEAF = itemId('7e07') const DELETING_LEAF = itemId('7e08') - it('marks every in-flight item (leaves and containers) failed, leaving terminal/idle/deleting untouched', async () => { + it('marks every in-flight item (leaves and containers) failed, leaving terminal/deleting untouched', async () => { // In-flight: a container mid-expansion, a container with an in-flight child, and standalone leaves. await seedItem({ id: PREPARING_DIR, type: 'directory', data: { source: '/p' }, status: 'preparing' }) await seedItem({ id: PROCESSING_DIR, type: 'directory', data: { source: '/q' }, status: 'processing' }) @@ -401,8 +401,7 @@ describe('KnowledgeItemService', () => { await seedItem({ id: EMBEDDING_LEAF, data: { source: 'e', content: 'e' }, status: 'embedding' }) await seedItem({ id: PROCESSING_LEAF, data: { source: 'p2', content: 'p2' }, status: 'processing' }) - // Untouched: not started, already terminal, or being deleted. - await seedItem({ id: IDLE_LEAF, data: { source: 'i', content: 'i' }, status: 'idle' }) + // Untouched: already terminal, or being deleted. await seedItem({ id: COMPLETED_LEAF, data: { source: 'c', content: 'c' }, status: 'completed' }) await seedItem({ id: FAILED_LEAF, data: { source: 'f', content: 'f' }, status: 'failed', error: 'real failure' }) await seedItem({ id: DELETING_LEAF, data: { source: 'd', content: 'd' }, status: 'deleting' }) @@ -416,7 +415,6 @@ describe('KnowledgeItemService', () => { expect(item.error).toBe(INTERRUPTED) } - expect(service.getById(IDLE_LEAF).status).toBe('idle') expect(service.getById(COMPLETED_LEAF).status).toBe('completed') const realFailure = service.getById(FAILED_LEAF) expect(realFailure.status).toBe('failed') @@ -437,19 +435,37 @@ describe('KnowledgeItemService', () => { }) describe('create', () => { - it('creates one knowledge item as idle', async () => { + it('creates a directory item as preparing', async () => { const item: CreateKnowledgeItemDto = { type: 'directory', data: { source: '/tmp/files' } } - const result = service.create(KNOWLEDGE_BASE_ID, item) + const result = service.createActive(KNOWLEDGE_BASE_ID, item) expect(result).toMatchObject({ baseId: KNOWLEDGE_BASE_ID, groupId: null, type: 'directory', - status: 'idle', + status: 'preparing', + error: null, + data: item.data + }) + }) + + it('creates a leaf item as processing', async () => { + const item: CreateKnowledgeItemDto = { + type: 'note', + data: { source: 'note', content: 'note' } + } + + const result = service.createActive(KNOWLEDGE_BASE_ID, item) + + expect(result).toMatchObject({ + baseId: KNOWLEDGE_BASE_ID, + groupId: null, + type: 'note', + status: 'processing', error: null, data: item.data }) @@ -458,7 +474,7 @@ describe('KnowledgeItemService', () => { it('accepts a group owner in the same base', async () => { await seedItem({ id: DIR_A_ID, type: 'directory', data: { source: '/a' } }) - const result = service.create(KNOWLEDGE_BASE_ID, { + const result = service.createActive(KNOWLEDGE_BASE_ID, { groupId: DIR_A_ID, type: 'note', data: { source: 'new grouped note', content: 'new grouped note' } @@ -481,7 +497,7 @@ describe('KnowledgeItemService', () => { let err: unknown try { - service.create(KNOWLEDGE_BASE_ID, { + service.createActive(KNOWLEDGE_BASE_ID, { groupId: DIR_A_ID, type: 'note', data: { source: 'child note', content: 'child note' } @@ -504,7 +520,7 @@ describe('KnowledgeItemService', () => { let err: unknown try { - service.create(KNOWLEDGE_BASE_ID, { + service.createActive(KNOWLEDGE_BASE_ID, { groupId: NOTE_OWNER_ID, type: 'note', data: { source: 'child note', content: 'child note' } @@ -525,7 +541,7 @@ describe('KnowledgeItemService', () => { it('rejects blank group owner ids before hitting foreign key constraints', async () => { let err: unknown try { - service.create(KNOWLEDGE_BASE_ID, { + service.createActive(KNOWLEDGE_BASE_ID, { groupId: ' ', type: 'note', data: { source: 'child note', content: 'child note' } @@ -546,7 +562,7 @@ describe('KnowledgeItemService', () => { it('translates missing base and missing group owner constraints', () => { let missingBaseErr: unknown try { - service.create('missing-base', { type: 'note', data: { source: 'note', content: 'note' } }) + service.createActive('missing-base', { type: 'note', data: { source: 'note', content: 'note' } }) } catch (e) { missingBaseErr = e } @@ -557,7 +573,7 @@ describe('KnowledgeItemService', () => { let missingOwnerErr: unknown try { - service.create(KNOWLEDGE_BASE_ID, { + service.createActive(KNOWLEDGE_BASE_ID, { groupId: 'missing-owner', type: 'note', data: { source: 'child note', content: 'child note' } @@ -624,7 +640,7 @@ describe('KnowledgeItemService', () => { }) it('creates a file knowledge item with a copied relative path', async () => { - const result = service.create(KNOWLEDGE_BASE_ID, { + const result = service.createActive(KNOWLEDGE_BASE_ID, { type: 'file', data: { source: '/docs/a.md', @@ -880,6 +896,20 @@ describe('KnowledgeItemService', () => { error: 'One or more child items failed' }) }) + + it('marks a subtree deleting', async () => { + await seedItem({ id: NOTE_1_ID, data: { source: 'note', content: 'note' }, status: 'processing' }) + + expect(service.setSubtreeStatus(KNOWLEDGE_BASE_ID, [NOTE_1_ID], 'deleting')).toEqual([NOTE_1_ID]) + await expect(getItemRow(NOTE_1_ID)).resolves.toMatchObject({ status: 'deleting', error: null }) + }) + + it('is a no-op to mark an already-deleting subtree deleting again', async () => { + await seedItem({ id: NOTE_1_ID, data: { source: 'note', content: 'note' }, status: 'deleting' }) + + expect(service.setSubtreeStatus(KNOWLEDGE_BASE_ID, [NOTE_1_ID], 'deleting')).toEqual([NOTE_1_ID]) + await expect(getItemRow(NOTE_1_ID)).resolves.toMatchObject({ status: 'deleting', error: null }) + }) }) describe('updateStatus', () => { @@ -1016,6 +1046,8 @@ describe('KnowledgeItemService', () => { const seeded = await seedItem({ status: 'deleting' }) + mockMainLoggerService.warn.mockClear() + mockMainLoggerService.info.mockClear() const result = service.updateStatus(seeded.id, 'completed') @@ -1028,12 +1060,19 @@ describe('KnowledgeItemService', () => { status: 'deleting', error: null }) + expect(mockMainLoggerService.warn).toHaveBeenCalledWith( + 'Skipped status update for deleting item', + expect.objectContaining({ id: seeded.id, attemptedStatus: 'completed' }) + ) + expect(mockMainLoggerService.info).not.toHaveBeenCalledWith('Updated knowledge item status', expect.anything()) }) it('does not overwrite deleting items with failed status from settled jobs', async () => { const seeded = await seedItem({ status: 'deleting' }) + mockMainLoggerService.warn.mockClear() + mockMainLoggerService.info.mockClear() const result = service.updateStatus(seeded.id, 'failed', { error: 'cancelled' }) @@ -1046,6 +1085,11 @@ describe('KnowledgeItemService', () => { status: 'deleting', error: null }) + expect(mockMainLoggerService.warn).toHaveBeenCalledWith( + 'Skipped status update for deleting item', + expect.objectContaining({ id: seeded.id, attemptedStatus: 'failed' }) + ) + expect(mockMainLoggerService.info).not.toHaveBeenCalledWith('Updated knowledge item status', expect.anything()) }) }) @@ -1608,5 +1652,22 @@ describe('KnowledgeItemService', () => { await expect(getItemRow(DIR_A_ID)).resolves.toMatchObject({ status: 'completed', error: null }) }) + + it('pulls a completed container back to processing when createActive adds a new child', async () => { + await seedItem({ + id: DIR_ROOT_ID, + type: 'directory', + data: { source: '/docs' }, + status: 'completed' + }) + + service.createActive(KNOWLEDGE_BASE_ID, { + groupId: DIR_ROOT_ID, + type: 'note', + data: { source: 'note', content: 'note' } + }) + + await expect(getItemRow(DIR_ROOT_ID)).resolves.toMatchObject({ status: 'processing', error: null }) + }) }) }) diff --git a/src/main/features/fileProcessing/tasks/__tests__/jobExecution.test.ts b/src/main/features/fileProcessing/tasks/__tests__/jobExecution.test.ts index ef773c11c11..a0201a5fc56 100644 --- a/src/main/features/fileProcessing/tasks/__tests__/jobExecution.test.ts +++ b/src/main/features/fileProcessing/tasks/__tests__/jobExecution.test.ts @@ -163,7 +163,7 @@ describe('prepareFileProcessingJob', () => { }) // The `{kind:'path'}` branch is what the knowledge workflow actually uses -// (KnowledgeWorkflowService passes `{kind:'path', path}`), bypassing FileManager. +// (KnowledgeIngestionService passes `{kind:'path', path}`), bypassing FileManager. describe('resolveFileProcessingFileInfo — kind:path', () => { beforeEach(() => { fsStatMock.mockReset() diff --git a/src/main/features/knowledge/KnowledgeLockManager.ts b/src/main/features/knowledge/KnowledgeLockManager.ts deleted file mode 100644 index e247d3914fb..00000000000 --- a/src/main/features/knowledge/KnowledgeLockManager.ts +++ /dev/null @@ -1,34 +0,0 @@ -import { Mutex } from 'async-mutex' - -export class KnowledgeLockManager { - private readonly baseMutexes = new Map() - - async withBaseMutationLock(baseId: string, task: () => T | Promise): Promise { - const mutex = this.getBaseMutex(baseId) - const release = await mutex.acquire() - - try { - return await task() - } finally { - release() - this.deleteIdleBaseMutex(baseId, mutex) - } - } - - private getBaseMutex(baseId: string): Mutex { - let mutex = this.baseMutexes.get(baseId) - if (!mutex) { - mutex = new Mutex() - this.baseMutexes.set(baseId, mutex) - } - return mutex - } - - private deleteIdleBaseMutex(baseId: string, mutex: Mutex): void { - // Only delete the exact mutex we released; a queued waiter may have already - // created a replacement for the same base after this task released. - if (!mutex.isLocked() && this.baseMutexes.get(baseId) === mutex) { - this.baseMutexes.delete(baseId) - } - } -} diff --git a/src/main/features/knowledge/KnowledgeService.ts b/src/main/features/knowledge/KnowledgeService.ts index b7c997848a2..f94bd43b403 100644 --- a/src/main/features/knowledge/KnowledgeService.ts +++ b/src/main/features/knowledge/KnowledgeService.ts @@ -1,448 +1,93 @@ import { application } from '@application' -import { knowledgeBaseService } from '@data/services/KnowledgeBaseService' -import { knowledgeItemService } from '@data/services/KnowledgeItemService' -import { loggerService } from '@logger' +import { KeyedMutex } from '@main/core/concurrency/KeyedMutex' import { BaseService, DependsOn, Injectable, Phase, ServicePhase } from '@main/core/lifecycle' -import { TraceMethod } from '@mcp-trace/trace-core' -import { DataApiErrorFactory, ErrorCode, isDataApiError } from '@shared/data/api' -import { KNOWLEDGE_BASES_MAX_LIMIT } from '@shared/data/api/schemas/knowledges' -import { - type CreateKnowledgeBaseDto, - getKnowledgeItemDisplayTitle, - KNOWLEDGE_ITEM_ERROR_INDEXING_INTERRUPTED, - type KnowledgeAddConflictStrategy, - type KnowledgeAddItemInput, - KnowledgeAddItemInputSchema, - type KnowledgeAddItemsResult, - type KnowledgeBase, - type KnowledgeItem, - type KnowledgeItemChunk, - type KnowledgeItemStatus, - type KnowledgeItemType, - type KnowledgeSearchResult, - type RestoreKnowledgeBaseDto, - type RestoreKnowledgeBaseResult +import type { + CreateKnowledgeBaseDto, + KnowledgeAddConflictStrategy, + KnowledgeAddItemInput, + KnowledgeAddItemsResult, + KnowledgeBase, + KnowledgeItem, + KnowledgeItemChunk, + KnowledgeSearchResult, + RestoreKnowledgeBaseDto, + RestoreKnowledgeBaseResult } from '@shared/data/types/knowledge' -import { isCompletedVectorKnowledgeBase } from '@shared/data/types/knowledge' -import { estimateTokenCount } from 'tokenx' -import { KnowledgeLockManager } from './KnowledgeLockManager' -import { KnowledgeWorkflowService } from './KnowledgeWorkflowService' +import { KnowledgeBaseAdminService } from './base/KnowledgeBaseAdminService' +import { KnowledgeIngestionService } from './ingestion/KnowledgeIngestionService' +import type { + KnowledgeConceptContent, + KnowledgeConceptGrep, + KnowledgeConceptMutationResult, + KnowledgeOrganizationTree +} from './query/KnowledgeConceptService' +import { KnowledgeConceptService } from './query/KnowledgeConceptService' +import { KnowledgeQueryService } from './query/KnowledgeQueryService' import { createCheckFileProcessingResultJobHandler } from './tasks/checkFileProcessingResultJobHandler' import { createDeleteSubtreeJobHandler } from './tasks/deleteSubtreeJobHandler' import { createIndexDocumentsJobHandler } from './tasks/indexDocumentsJobHandler' import { createPrepareRootJobHandler } from './tasks/prepareRootJobHandler' import { createReindexSubtreeJobHandler } from './tasks/reindexSubtreeJobHandler' -import { narrowKnowledgeJobInput } from './tasks/utils/jobInput' -import { - KNOWLEDGE_ACTIVE_JOB_LIMIT, - KNOWLEDGE_ACTIVE_JOB_STATUSES, - KNOWLEDGE_JOB_TYPES, - knowledgeDeleteSubtreeIdempotencyKey, - knowledgeQueueName, - toKnowledgeBaseId, - toKnowledgeItemId, - toKnowledgeItemIds -} from './types' -import { embedKnowledgeQuery } from './utils/indexing/embed' -import { toMaterialRelativePath } from './utils/indexing/materialFields' -import { rerankKnowledgeSearchResults } from './utils/indexing/rerank' -import { classifyKnowledgeItemSource } from './utils/items' -import { applyRelevanceThreshold, getInitialSearchScoreKind, withSearchRanks } from './utils/search' -import { getKnowledgeBaseFilePath } from './utils/storage/pathStorage' -import type { KnowledgeIndexStore } from './vectorstore/indexStore/KnowledgeIndexStore' -import type { KnowledgeIndexSearchMatch } from './vectorstore/indexStore/model' -const logger = loggerService.withContext('KnowledgeService') -const SEARCH_TOKEN_PATTERN = /[\p{L}\p{N}_]+/u -const DELETE_RECOVERY_ROOT_CHUNK_SIZE = 500 /** - * Fetch this many × the requested result count as index candidates. The index - * store only filters by material state; the item-visibility filter (missing / - * other-base / not-completed) runs afterwards in the caller and can drop matches, - * so over-fetching keeps the final set from shrinking below topK. + * Facade of the knowledge feature: registers the job handlers, runs boot-time + * recovery, and delegates every public operation to the module that owns it — + * base lifecycle (KnowledgeBaseAdminService), write-side orchestration (ingestion/), and the read + * side (query/). Holds no domain logic of its own. */ -const KNOWLEDGE_SEARCH_OVERFETCH_FACTOR = 5 -/** Hard ceiling on fetched candidates, bounding the brute-force vector scan and rerank cost regardless of topK. */ -const KNOWLEDGE_SEARCH_CANDIDATE_CAP = 200 -const REINDEX_ALLOWED_STATUSES = new Set(['completed', 'failed']) -const KNOWLEDGE_JOB_TYPE_SET = new Set(KNOWLEDGE_JOB_TYPES) - -/** Max characters {@link KnowledgeService.readConcept} returns in one slice, so a large document can't flood the agent's context. */ -const CONCEPT_READ_MAX_CHARS = 20_000 -/** Default returned-match cap for {@link KnowledgeService.grepConcept} (the agent may raise it up to {@link CONCEPT_GREP_MAX_MATCHES}). */ -const CONCEPT_GREP_DEFAULT_MAX_MATCHES = 50 -/** Hard ceiling on returned grep matches, bounding the response size. */ -const CONCEPT_GREP_MAX_MATCHES = 200 -/** Characters of context kept on each side of a grep match in its snippet. */ -const CONCEPT_GREP_SNIPPET_PAD = 60 -/** - * Max characters of any single line {@link KnowledgeService.grepConcept} runs the pattern over. Matching one bounded - * line at a time keeps a catastrophic-backtracking pattern (e.g. `(a+)+$`) from freezing the main-process event loop - * by spanning the whole document; matches past this point on an over-long line are not scanned. This only removes the - * whole-document blow-up — a pathological pattern can still backtrack exponentially within one 2000-char line, so it - * is not RE2/ripgrep-grade linear-time matching. Reuses the same 2000-char value as the filesystem grep tool (there - * it truncates displayed lines; here it bounds the text the pattern actually runs over). - */ -const CONCEPT_GREP_MAX_LINE_CHARS = 2000 -/** Hard ceiling on nodes {@link KnowledgeService.getOrganizationTree} returns, bounding the response for a huge base. */ -export const KNOWLEDGE_TREE_MAX_NODES = 1000 - -/** Verbatim slice of a knowledge concept's indexed text, addressed by Concept ID (the material's relative path, OKF §2). */ -export interface KnowledgeConceptContent { - conceptId: string - title: string - itemType: KnowledgeItemType - /** Length of the full document text, so the caller can tell when a slice is partial and page through it. */ - totalChars: number - charStart: number - charEnd: number - content: string - /** True when the returned slice was capped at {@link CONCEPT_READ_MAX_CHARS} and the document continues past `charEnd`. */ - truncated: boolean -} - -/** One exact-pattern match within a concept's indexed text. */ -export interface KnowledgeConceptGrepMatch { - /** 1-based line number of the match start within the document text. */ - line: number - charStart: number - charEnd: number - /** The matched text with a little surrounding context (see {@link CONCEPT_GREP_SNIPPET_PAD}). */ - snippet: string -} - -/** Result of grepping a knowledge concept's indexed text for a regular expression. */ -export interface KnowledgeConceptGrep { - conceptId: string - title: string - itemType: KnowledgeItemType - /** Total matches found in the document (may exceed `matches.length` when the cap was hit). */ - totalMatches: number - matches: KnowledgeConceptGrepMatch[] -} - -/** - * One node of a knowledge base's organization tree, emitted in pre-order DFS so - * the flat list reads top-down like an outline. `depth` (0 at the base root) - * carries the hierarchy without a recursive shape. `conceptId` is set only for a - * readable leaf (a completed file/url/note) so it can be passed to kb_read - * (read or grep mode); directories and not-yet-indexed leaves have none. - */ -export interface KnowledgeTreeNode { - depth: number - title: string - itemType: KnowledgeItemType - status: KnowledgeItemStatus - conceptId?: string -} - -/** - * A knowledge base's organization tree — the logical groupId hierarchy of - * knowledge_item (directories as folders, file/url/note as leaves), NOT the flat - * physical `raw/` layout. The synthesized view OKF surfaces as an `index.md`. - */ -export interface KnowledgeOrganizationTree { - baseId: string - /** - * Count of non-deleting items in the base. May exceed `nodes.length` for two reasons: the node - * list hit the {@link KNOWLEDGE_TREE_MAX_NODES} cap, or a `maxDepth` filter dropped deeper items - * (which are still counted here but not emitted as nodes). - */ - totalItems: number - /** - * True only when the emitted node list was capped at {@link KNOWLEDGE_TREE_MAX_NODES} — it does - * NOT flag `maxDepth` filtering. A reliable "whole tree returned" check is therefore - * `truncated === false` AND no `maxDepth` was passed. - */ - truncated: boolean - nodes: KnowledgeTreeNode[] -} - -/** - * Result of a Concept ID-addressed mutation ({@link KnowledgeService.deleteConcepts} / - * {@link KnowledgeService.refreshConcepts}). `applied` are the Concept IDs that resolved - * to a visible document and were acted on; `notFound` are those that did not resolve in - * this base (a no-op, reported back so the agent can re-check the id rather than assume success). - */ -export interface KnowledgeConceptMutationResult { - applied: string[] - notFound: string[] -} - -/** - * Concept ID (relative path, OKF §2) of a search hit's source document, or - * undefined when it has none: a directory (no material), or a url/note whose - * snapshot relativePath was not captured yet. A completed leaf normally has one, - * so this only swallows the rare unindexed-snapshot case rather than throwing - * out of the whole search. - */ -function deriveConceptId(item: KnowledgeItem): string | undefined { - if (item.type === 'directory') { - return undefined - } - try { - return toMaterialRelativePath(item) - } catch (error) { - // A completed url/note with no snapshot relativePath is an invariant violation. Swallow it (one - // unfollowable hit must not sink the whole search) but leave a diagnostic trail rather than a silent drop. - logger.warn('deriveConceptId: completed item has no material relativePath', { - itemId: item.id, - type: item.type, - error: error instanceof Error ? error.message : String(error) - }) - return undefined - } -} - -/** Compile a kb_read grep-mode pattern (always global; case-insensitive unless told otherwise), turning a bad pattern into a validation error. */ -function compileGrepRegex(pattern: string, ignoreCase: boolean): RegExp { - try { - return new RegExp(pattern, ignoreCase ? 'gi' : 'g') - } catch (error) { - const message = error instanceof Error ? error.message : String(error) - throw DataApiErrorFactory.validation({ pattern: [message] }, `Invalid kb_read regular expression: ${message}`) - } -} - -/** - * Scan `text` for every match of the global `regex`, one line at a time, returning the - * total count and the first `limit` matches with 1-based line numbers, document-absolute - * offsets, and padded snippets. Each line is matched independently and truncated at - * {@link CONCEPT_GREP_MAX_LINE_CHARS}, so a single regex evaluation never runs over more - * than one bounded line — a catastrophic-backtracking pattern therefore cannot freeze the - * main process by spanning the whole document (this mirrors the line-oriented fallback in - * the filesystem grep tool). Anchors (`^`/`$`) consequently bind to each line, and a match - * cannot span lines. `lastIndex` is advanced past zero-width matches so an empty-matching - * pattern cannot spin. `regex` MUST carry the global flag (the caller compiles it so); it - * is reset at the start of every line. - */ -function scanConceptMatches( - text: string, - regex: RegExp, - limit: number -): { totalMatches: number; matches: KnowledgeConceptGrepMatch[] } { - const matches: KnowledgeConceptGrepMatch[] = [] - let totalMatches = 0 - let lineNumber = 0 - // Absolute offset of the current line's first character within `text`. - let lineStart = 0 - - while (lineStart <= text.length) { - lineNumber++ - const newlineIndex = text.indexOf('\n', lineStart) - const lineEnd = newlineIndex === -1 ? text.length : newlineIndex - // Run the pattern over a single, truncated line so backtracking can't blow up across the - // whole document; a match past the per-line cap on an over-long line is dropped. - const line = text.slice(lineStart, Math.min(lineEnd, lineStart + CONCEPT_GREP_MAX_LINE_CHARS)) - - regex.lastIndex = 0 - for (let result = regex.exec(line); result !== null; result = regex.exec(line)) { - totalMatches++ - const matchLength = result[0].length - const start = lineStart + result.index - const end = start + matchLength - - if (matches.length < limit) { - matches.push({ - line: lineNumber, - charStart: start, - charEnd: end, - snippet: text.slice( - Math.max(0, start - CONCEPT_GREP_SNIPPET_PAD), - Math.min(text.length, end + CONCEPT_GREP_SNIPPET_PAD) - ) - }) - } - - // Zero-width match: bump lastIndex so exec() advances past the same position. - regex.lastIndex = result.index + (matchLength > 0 ? matchLength : 1) - } - - lineStart = lineEnd + 1 - } - - return { totalMatches, matches } -} - @Injectable('KnowledgeService') @ServicePhase(Phase.WhenReady) @DependsOn(['KnowledgeVectorStoreService', 'JobManager', 'FileProcessingService']) export class KnowledgeService extends BaseService { - private readonly knowledgeLockManager = new KnowledgeLockManager() - private readonly workflowService = new KnowledgeWorkflowService(this.knowledgeLockManager) + private readonly knowledgeLockManager = new KeyedMutex() + private readonly ingestionService = new KnowledgeIngestionService(this.knowledgeLockManager) + private readonly baseAdmin = new KnowledgeBaseAdminService(this.knowledgeLockManager, this.ingestionService) + private readonly queryService = new KnowledgeQueryService() + private readonly conceptService = new KnowledgeConceptService(this.ingestionService) protected onInit(): void { const jobManager = application.get('JobManager') jobManager.registerHandler( 'knowledge.prepare-root', - createPrepareRootJobHandler(this.knowledgeLockManager, this.workflowService) + createPrepareRootJobHandler(this.knowledgeLockManager, this.ingestionService) ) jobManager.registerHandler('knowledge.index-documents', createIndexDocumentsJobHandler(this.knowledgeLockManager)) jobManager.registerHandler( 'knowledge.check-file-processing-result', - createCheckFileProcessingResultJobHandler(this.knowledgeLockManager, this.workflowService) + createCheckFileProcessingResultJobHandler(this.knowledgeLockManager, this.ingestionService) ) jobManager.registerHandler('knowledge.delete-subtree', createDeleteSubtreeJobHandler(this.knowledgeLockManager)) jobManager.registerHandler( 'knowledge.reindex-subtree', - createReindexSubtreeJobHandler(this.knowledgeLockManager, this.workflowService) + createReindexSubtreeJobHandler(this.knowledgeLockManager, this.ingestionService) ) } protected async onAllReady(): Promise { - this.recoverDeletingItems() - this.recoverInterruptedItems() + this.ingestionService.recoverDeletingItems() + this.ingestionService.recoverInterruptedItems() } async createBase(dto: CreateKnowledgeBaseDto): Promise { - const base = knowledgeBaseService.create(dto) - const vectorStoreService = application.get('KnowledgeVectorStoreService') - - try { - await vectorStoreService.getIndexStore(base) - } catch (error) { - await this.rollbackFailedBaseCreation(base.id) - throw error - } - - return base - } - - /** - * Undo a half-created base after its index store failed to open: remove the - * orphaned `.cherry/` directory `getIndexStore` left on disk and drop the DB - * row. Both steps are best-effort and logged — a cleanup failure must never - * mask the original open error the caller needs to see. - */ - private async rollbackFailedBaseCreation(baseId: string): Promise { - const vectorStoreService = application.get('KnowledgeVectorStoreService') - try { - await vectorStoreService.deleteStore(baseId) - } catch (cleanupError) { - logger.warn('Failed to remove index store dir during createBase rollback', cleanupError as Error, { baseId }) - } - try { - knowledgeBaseService.delete(baseId) - } catch (cleanupError) { - logger.warn('Failed to delete knowledge base row during createBase rollback', cleanupError as Error, { baseId }) - } + return await this.baseAdmin.createBase(dto) } async deleteBase(baseId: string): Promise { - await this.cancelAllJobsForBase(baseId) - - await this.knowledgeLockManager.withBaseMutationLock(baseId, async () => { - try { - const vectorStoreService = application.get('KnowledgeVectorStoreService') - await vectorStoreService.deleteStore(baseId) - } catch (error) { - const normalizedError = error instanceof Error ? error : new Error(String(error)) - logger.error('Failed to delete knowledge base vector artifacts', normalizedError, { baseId }) - throw error - } - - try { - knowledgeBaseService.delete(baseId) - } catch (error) { - const normalizedError = error instanceof Error ? error : new Error(String(error)) - logger.error('Failed to delete knowledge base SQLite row after artifact cleanup', normalizedError, { - baseId - }) - throw DataApiErrorFactory.invalidOperation( - 'deleteBase', - `Vector artifacts were deleted, but SQLite knowledge base cleanup failed: ${normalizedError.message}` - ) - } - }) + await this.baseAdmin.deleteBase(baseId) } async restoreBase(dto: RestoreKnowledgeBaseDto): Promise { - const sourceBase = knowledgeBaseService.getById(dto.sourceBaseId) - - // Restore always supplies an embedding model (RestoreKnowledgeBaseSchema requires - // it). A BM25-only source's searchMode is pinned to 'bm25' by the no-model - // invariant, so carrying it over would leave the restored base's semantic search - // silently disabled despite it now paying for the full embedding backfill. Drop - // it (and the always-null hybridAlpha) so create() applies its own hybrid - // default; a source that already had a model keeps its existing preference. - const gainsEmbeddings = sourceBase.embeddingModelId === null - - const createDto: CreateKnowledgeBaseDto = { - name: dto.name?.trim() ?? sourceBase.name, - dimensions: dto.dimensions, - embeddingModelId: dto.embeddingModelId, - rerankModelId: sourceBase.rerankModelId, - fileProcessorId: sourceBase.fileProcessorId, - chunkSize: sourceBase.chunkSize, - chunkOverlap: sourceBase.chunkOverlap, - threshold: sourceBase.threshold, - documentCount: sourceBase.documentCount, - searchMode: gainsEmbeddings ? undefined : sourceBase.searchMode, - hybridAlpha: gainsEmbeddings ? undefined : sourceBase.hybridAlpha, - groupId: sourceBase.groupId ?? undefined - } - - const rootItems = knowledgeItemService.getRootItemsByBaseId(sourceBase.id) - - // Partial restore: probe each root's source and skip the ones whose source is genuinely gone, so - // a single missing source no longer aborts the entire restore. This is the common case for a - // failed base — a v1-migrated directory child has a virtual path with no raw/ file, and a file - // whose original was deleted has no material to copy; addItems would throw on the first such - // item and roll back the whole batch. An 'unverifiable' source (transient/permission error) is - // kept, not skipped — like reindex, we never drop a source we could not confirm is gone. - const restorableRootItems: KnowledgeItem[] = [] - for (const item of rootItems) { - if ((await classifyKnowledgeItemSource(sourceBase.id, item)) === 'missing') { - logger.warn('Skipping knowledge item with a missing source during restore', { - sourceBaseId: sourceBase.id, - itemId: item.id, - type: item.type - }) - continue - } - restorableRootItems.push(item) - } - const skippedMissingSourceCount = rootItems.length - restorableRootItems.length - if (skippedMissingSourceCount > 0) { - logger.info('Restore skipped knowledge items whose source no longer exists', { - sourceBaseId: sourceBase.id, - skippedMissingSourceCount, - restorableCount: restorableRootItems.length - }) - } + return await this.baseAdmin.restoreBase(dto) + } - const inputs = restorableRootItems.map((item) => this.toRestoreRuntimeInput(sourceBase.id, item)) - const restoredBase = await this.createBase(createDto) - try { - await this.addItems(restoredBase.id, inputs) - } catch (error) { - try { - await this.deleteBase(restoredBase.id) - } catch (cleanupError) { - const cleanupMessage = cleanupError instanceof Error ? cleanupError.message : String(cleanupError) - logger.error( - 'Failed to delete restored knowledge base after item restoration failed', - cleanupError instanceof Error ? cleanupError : new Error(cleanupMessage), - { - sourceBaseId: sourceBase.id, - restoredBaseId: restoredBase.id - } - ) - throw DataApiErrorFactory.invalidOperation( - 'restoreBase', - `Failed to restore knowledge items: ${ - error instanceof Error ? error.message : String(error) - }. Restored knowledge base '${restoredBase.id}' could not be cleaned up automatically: ${cleanupMessage}. Please delete it manually.` - ) - } - throw DataApiErrorFactory.invalidOperation( - 'restoreBase', - `Failed to restore knowledge items: ${error instanceof Error ? error.message : String(error)}` - ) - } + listBases(): KnowledgeBase[] { + return this.baseAdmin.listBases() + } - return { base: restoredBase, skippedMissingSourceCount } + /** Whether the user has any knowledge base at all — a cheap count (not a full list) for tool-availability gating. */ + hasAnyBase(): boolean { + return this.baseAdmin.hasAnyBase() } async addItems( @@ -450,748 +95,54 @@ export class KnowledgeService extends BaseService { items: KnowledgeAddItemInput[], conflictStrategy?: KnowledgeAddConflictStrategy ): Promise { - this.assertBaseCanRunRuntimeOperation(baseId, 'addItems') - return await this.workflowService.addItems(baseId, items, conflictStrategy) + return await this.ingestionService.addItems(baseId, items, conflictStrategy) } async deleteItems(baseId: string, itemIds: string[]): Promise { - const rootItemIds = knowledgeItemService.getOutermostSelectedItemIds(baseId, itemIds) - if (rootItemIds.length === 0) { - return - } - - await this.workflowService.deleteItems(baseId, rootItemIds) + await this.ingestionService.deleteItems(baseId, itemIds) } async reindexItems(baseId: string, itemIds: string[]): Promise { - this.assertBaseCanRunRuntimeOperation(baseId, 'reindexItems') - const rootItemIds = knowledgeItemService.getOutermostSelectedItemIds(baseId, itemIds) - if (rootItemIds.length === 0) { - return - } - - await this.assertSubtreesCanReindex(baseId, rootItemIds) - - await this.workflowService.reindexItems(baseId, rootItemIds) - } - - listBases(): KnowledgeBase[] { - const { items } = knowledgeBaseService.list({ page: 1, limit: KNOWLEDGE_BASES_MAX_LIMIT }) - return items - } - - /** Whether the user has any knowledge base at all — a cheap count (not a full list) for tool-availability gating. */ - hasAnyBase(): boolean { - const { total } = knowledgeBaseService.list({ page: 1, limit: 1 }) - return total > 0 + await this.ingestionService.reindexItems(baseId, itemIds) } listRootItems(baseId: string): KnowledgeItem[] { - return knowledgeItemService.getRootItemsByBaseId(baseId) + return this.queryService.listRootItems(baseId) } - @TraceMethod({ spanName: 'Knowledge.search', tag: 'Knowledge' }) async search(baseId: string, query: string): Promise { - this.assertBaseCanRunRuntimeOperation(baseId, 'search') - - if (!SEARCH_TOKEN_PATTERN.test(query)) { - throw DataApiErrorFactory.validation( - { query: ['Query has no searchable tokens'] }, - 'Query has no searchable tokens' - ) - } - - const base = knowledgeBaseService.getById(baseId) - // Stored search mode and the index store's mode are the same enum now, so no mapping. - // Vector/hybrid retrieval needs an embedding model; a base without one is BM25-only - // regardless of its stored preference. - const mode = isCompletedVectorKnowledgeBase(base) ? base.searchMode : 'bm25' - // BM25 is lexical only; skip the embedding round-trip when the query won't use it. - const queryEmbedding = mode === 'bm25' ? undefined : await embedKnowledgeQuery(base, query) - - const resolvedTopK = base.documentCount ?? 10 - const candidateLimit = Math.min(resolvedTopK * KNOWLEDGE_SEARCH_OVERFETCH_FACTOR, KNOWLEDGE_SEARCH_CANDIDATE_CAP) - - const vectorStoreService = application.get('KnowledgeVectorStoreService') - const store = await vectorStoreService.getIndexStore(base) - const matches = await this.runStoreOperation(store, baseId, 'search', () => - store.search({ - queryText: query, - queryEmbedding, - mode, - topK: candidateLimit, - alpha: base.hybridAlpha - }) - ) - - const scoreKind = getInitialSearchScoreKind(base) - const visibleSearchResults = this.toVisibleSearchResults(baseId, matches, scoreKind) - const topResults = this.trimToTopK(visibleSearchResults, resolvedTopK, baseId) - - if (base.rerankModelId) { - const rerankedResults = await rerankKnowledgeSearchResults(base, query, topResults) - return withSearchRanks(applyRelevanceThreshold(rerankedResults, base.threshold)) - } - - return withSearchRanks(applyRelevanceThreshold(topResults, base.threshold)) + return await this.queryService.search(baseId, query) } async listItemChunks(baseId: string, itemId: string): Promise { - const knowledgeBaseId = toKnowledgeBaseId(baseId) - const knowledgeItemId = toKnowledgeItemId(itemId) - this.assertBaseCanRunRuntimeOperation(knowledgeBaseId, 'listItemChunks') - const item = await this.assertItemCanRunChunkOperation(knowledgeBaseId, knowledgeItemId, 'list chunks') - this.assertCompletedContainerHasNoDeletingChildren(knowledgeBaseId, item) - - const base = knowledgeBaseService.getById(knowledgeBaseId) - const leafItems = knowledgeItemService.getSubtreeItems(knowledgeBaseId, [knowledgeItemId], { - includeRoots: true, - leafOnly: true - }) - if (leafItems.length === 0) { - return [] - } - - const vectorStoreService = application.get('KnowledgeVectorStoreService') - const store = await vectorStoreService.getIndexStore(base) - const chunkGroups = await this.runStoreOperation(store, knowledgeBaseId, 'listItemChunks', () => - Promise.all( - leafItems.map(async (leafItem) => { - const units = await store.listMaterialUnits(leafItem.id) - return units.map( - (unit): KnowledgeItemChunk => ({ - id: unit.unitId, - itemId: leafItem.id, - content: unit.text, - metadata: { - itemId: leafItem.id, - itemType: leafItem.type, - source: leafItem.data.source, - chunkIndex: unit.unitIndex, - tokenCount: estimateTokenCount(unit.text) - } - }) - ) - }) - ) - ) - - return chunkGroups.flat() + return await this.queryService.listItemChunks(baseId, itemId) } - /** - * Read a knowledge concept's indexed text by its Concept ID (the material's - * relative path, OKF §2), optionally a `[charStart, charEnd)` slice. The slice - * is capped at {@link CONCEPT_READ_MAX_CHARS}; `totalChars` + `truncated` let - * the caller page through a longer document. Throws NOT_FOUND when the Concept - * ID does not resolve to a readable, visible document in this base. - */ async readConcept( baseId: string, conceptId: string, range?: { charStart?: number; charEnd?: number } ): Promise { - const { item, text } = await this.resolveConcept(baseId, conceptId, 'readConcept') - - const totalChars = text.length - const start = Math.min(Math.max(range?.charStart ?? 0, 0), totalChars) - // Where the caller would have ended without the cap (an omitted charEnd reads to the document end). - const naturalEnd = Math.min(range?.charEnd ?? totalChars, totalChars) - const end = Math.max(start, Math.min(naturalEnd, start + CONCEPT_READ_MAX_CHARS)) - - return { - conceptId, - title: getKnowledgeItemDisplayTitle(item), - itemType: item.type, - totalChars, - charStart: start, - charEnd: end, - content: text.slice(start, end), - truncated: end < naturalEnd - } + return await this.conceptService.readConcept(baseId, conceptId, range) } - /** - * Search a knowledge concept's indexed text for a regular expression, returning - * the total match count and the first `maxMatches` matches (line, offsets, a - * padded snippet). The pattern is compiled global, case-insensitive by default; - * an invalid pattern throws a validation error. Throws NOT_FOUND when the - * Concept ID does not resolve to a readable, visible document in this base. - */ async grepConcept( baseId: string, conceptId: string, options: { pattern: string; ignoreCase?: boolean; maxMatches?: number } ): Promise { - const { item, text } = await this.resolveConcept(baseId, conceptId, 'grepConcept') - - const limit = Math.min( - Math.max(options.maxMatches ?? CONCEPT_GREP_DEFAULT_MAX_MATCHES, 1), - CONCEPT_GREP_MAX_MATCHES - ) - const regex = compileGrepRegex(options.pattern, options.ignoreCase ?? true) - const { totalMatches, matches } = scanConceptMatches(text, regex, limit) - - return { - conceptId, - title: getKnowledgeItemDisplayTitle(item), - itemType: item.type, - totalMatches, - matches - } + return await this.conceptService.grepConcept(baseId, conceptId, options) } - /** - * Delete knowledge documents addressed by Concept ID (the deep-read/kb_manage - * write counterpart to {@link readConcept}). Each Concept ID is resolved to its - * knowledge item and deleted via {@link deleteItems} (which expands to the - * outermost selected subtree); an id that does not resolve to a visible document - * in this base is reported in `notFound` rather than failing the whole batch. - */ async deleteConcepts(baseId: string, conceptIds: string[]): Promise { - const { found, notFound } = await this.resolveConceptItemIds(baseId, conceptIds, 'deleteConcepts') - if (found.length > 0) { - await this.deleteItems( - baseId, - found.map((entry) => entry.itemId) - ) - } - return { applied: found.map((entry) => entry.conceptId), notFound } + return await this.conceptService.deleteConcepts(baseId, conceptIds) } - /** - * Re-index knowledge documents addressed by Concept ID. Each Concept ID is - * resolved to its knowledge item and re-indexed via {@link reindexItems}; an id - * that does not resolve to a visible document in this base is reported in - * `notFound` rather than failing the whole batch. - */ async refreshConcepts(baseId: string, conceptIds: string[]): Promise { - const { found, notFound } = await this.resolveConceptItemIds(baseId, conceptIds, 'refreshConcepts') - if (found.length > 0) { - await this.reindexItems( - baseId, - found.map((entry) => entry.itemId) - ) - } - return { applied: found.map((entry) => entry.conceptId), notFound } + return await this.conceptService.refreshConcepts(baseId, conceptIds) } - /** - * Resolve a batch of Concept IDs to their knowledge item ids for a Concept - * ID-addressed mutation. Mirrors {@link resolveConcept}'s identity boundary — - * each relative path is looked up in this base's index store, then re-validated - * against the visible knowledge_item (same base, completed) — but resolves many - * ids and partitions them into resolved (`found`) vs unresolved (`notFound`) - * instead of throwing, so one bad id does not sink the batch. Duplicate ids in - * the input are collapsed to a single resolution. - */ - private async resolveConceptItemIds( - baseId: string, - conceptIds: string[], - operation: string - ): Promise<{ found: Array<{ conceptId: string; itemId: string }>; notFound: string[] }> { - const base = knowledgeBaseService.getById(baseId) - const vectorStoreService = application.get('KnowledgeVectorStoreService') - const store = await vectorStoreService.getIndexStore(base) - - const found: Array<{ conceptId: string; itemId: string }> = [] - const notFound: string[] = [] - const seen = new Set() - - for (const conceptId of conceptIds) { - if (seen.has(conceptId)) { - continue - } - seen.add(conceptId) - - const ref = await this.runStoreOperation(store, baseId, operation, () => - store.getMaterialByRelativePath(conceptId) - ) - // Identity re-check: the resolved material id must still be a visible item in this base. - const item = ref ? this.loadVisibleItems(baseId, [ref.materialId]).get(ref.materialId) : undefined - if (ref && item) { - found.push({ conceptId, itemId: ref.materialId }) - } else { - notFound.push(conceptId) - } - } - - return { found, notFound } - } - - /** - * Resolve a Concept ID to its content text for the deep-read tools. The tools - * address by relative path (a transparent DB key, never an fs path), so this - * re-validates the resolved material against the visible knowledge_item (same - * base, completed) before returning any text — the identity boundary that keeps - * a relative path from reaching another base's or a deleted item's content. - * Throws NOT_FOUND when the concept does not resolve to a readable, visible document. - */ - private async resolveConcept( - baseId: string, - conceptId: string, - operation: string - ): Promise<{ item: KnowledgeItem; text: string }> { - this.assertBaseCanRunRuntimeOperation(baseId, operation) - - const base = knowledgeBaseService.getById(baseId) - const vectorStoreService = application.get('KnowledgeVectorStoreService') - const store = await vectorStoreService.getIndexStore(base) - - const ref = await this.runStoreOperation(store, baseId, operation, () => store.getMaterialByRelativePath(conceptId)) - if (!ref) { - throw DataApiErrorFactory.notFound('Knowledge concept', conceptId) - } - - const item = this.loadVisibleItems(baseId, [ref.materialId]).get(ref.materialId) - if (!item) { - throw DataApiErrorFactory.notFound('Knowledge concept', conceptId) - } - - const text = await this.runStoreOperation(store, baseId, operation, () => store.readMaterialContent(ref.materialId)) - if (text == null) { - // The material resolved and the item is visible + completed, yet it has no content row — an - // invariant violation or a reindex TOCTOU race, NOT a bad conceptId. Use a distinct resource so the - // tool layer (conceptLookupError) can steer "retry shortly" instead of "verify the conceptId", which - // cannot fix a missing content row. The resource string mirrors KNOWLEDGE_CONCEPT_CONTENT_NOT_FOUND_RESOURCE - // in knowledgeLookup.ts. - logger.warn('resolveConcept: visible completed item has no content row', { baseId, conceptId, operation }) - throw DataApiErrorFactory.notFound('Knowledge concept content', conceptId) - } - - return { item, text } - } - - /** - * Build a knowledge base's organization tree from the `knowledge_item.groupId` - * hierarchy — directories are folders, file/url/note are leaves carrying a - * readable `conceptId` (when completed) for kb_read. This is the - * logical org layer the OKF `index.md` surfaces, deliberately decoupled from - * the flat physical `raw/` layout: no material scan, no path joins. Emitted as - * a pre-order DFS node list (each node's `depth` carries the hierarchy), - * capped at {@link KNOWLEDGE_TREE_MAX_NODES}. Throws NOT_FOUND if the base does - * not exist; available regardless of the base's runtime (search) state. - */ getOrganizationTree(baseId: string, options: { maxDepth?: number } = {}): KnowledgeOrganizationTree { - knowledgeBaseService.getById(baseId) - - const items = knowledgeItemService.getItemsByBaseId(baseId) - - // Index children by their parent groupId (null = base root); each bucket keeps - // getItemsByBaseId's createdAt/id order, so siblings emit in a stable order. - const childrenByGroupId = new Map() - for (const item of items) { - const key = item.groupId ?? null - const bucket = childrenByGroupId.get(key) - if (bucket) { - bucket.push(item) - } else { - childrenByGroupId.set(key, [item]) - } - } - - const maxDepth = options.maxDepth - const nodes: KnowledgeTreeNode[] = [] - let truncated = false - - const walk = (groupId: string | null, depth: number): void => { - if (truncated || (maxDepth !== undefined && depth > maxDepth)) { - return - } - for (const item of childrenByGroupId.get(groupId) ?? []) { - if (nodes.length >= KNOWLEDGE_TREE_MAX_NODES) { - truncated = true - return - } - nodes.push({ - depth, - title: getKnowledgeItemDisplayTitle(item), - itemType: item.type, - status: item.status, - // Only a completed leaf is readable; directories and pending leaves carry no Concept ID. - conceptId: item.type !== 'directory' && item.status === 'completed' ? deriveConceptId(item) : undefined - }) - if (item.type === 'directory') { - walk(item.id, depth + 1) - } - } - } - - walk(null, 0) - - return { baseId, totalItems: items.length, truncated, nodes } - } - - /** - * Turn raw index matches into visible search results: fetch each match's - * knowledge item once, drop any that is missing, in another base, or not - * completed, and reconstruct the chunk metadata (item type / source from the - * item; chunk index from the unit; token count recomputed from the body). - */ - private toVisibleSearchResults( - baseId: string, - matches: KnowledgeIndexSearchMatch[], - scoreKind: KnowledgeSearchResult['scoreKind'] - ): KnowledgeSearchResult[] { - const itemsById = this.loadVisibleItems( - baseId, - matches.map((match) => match.materialId) - ) - - const results: KnowledgeSearchResult[] = [] - for (const match of matches) { - const item = itemsById.get(match.materialId) - if (!item) { - continue - } - results.push({ - pageContent: match.text, - score: match.score, - scoreKind, - rank: results.length + 1, - metadata: { - itemId: match.materialId, - itemType: item.type, - source: item.data.source, - chunkIndex: match.unitIndex, - tokenCount: estimateTokenCount(match.text) - }, - itemId: match.materialId, - chunkId: match.unitId, - // Concept ID + title so a hit can be followed up with kb_read. - conceptId: deriveConceptId(item), - title: getKnowledgeItemDisplayTitle(item) - }) - } - return results - } - - /** Keep the highest-scored `topK` visible results, discarding the over-fetched tail. */ - private trimToTopK(results: KnowledgeSearchResult[], topK: number, baseId: string): KnowledgeSearchResult[] { - if (results.length <= topK) { - return results - } - logger.debug('Trimmed over-fetched knowledge search results to topK', { - baseId, - visibleCandidates: results.length, - topK - }) - return results.slice(0, topK) - } - - /** Fetch the distinct items behind the matches, keeping only those visible in this base (same base, completed). */ - private loadVisibleItems(baseId: string, materialIds: string[]): Map { - const uniqueIds = [...new Set(materialIds)] - const visibleItems = new Map() - - for (const materialId of uniqueIds) { - try { - const item = knowledgeItemService.getById(materialId) - if (item.baseId === baseId && item.status === 'completed') { - visibleItems.set(materialId, item) - } - } catch (error) { - if (isDataApiError(error) && error.code === ErrorCode.NOT_FOUND) { - continue - } - throw error - } - } - - return visibleItems - } - - /** - * Run a per-base index-store interaction, translating the error raised when the - * store is closed mid-flight — a concurrent {@link deleteBase} or app shutdown - * closed the driver — into a defined, retryable DataApiError instead of leaking - * the opaque driver-level error to the renderer. Genuine query errors rethrow - * unchanged. - */ - private async runStoreOperation( - store: KnowledgeIndexStore, - baseId: string, - operation: string, - run: () => Promise - ): Promise { - try { - return await run() - } catch (error) { - if (store.isClosed()) { - logger.warn('Knowledge index store was closed during operation', { baseId, operation }) - throw DataApiErrorFactory.invalidOperation( - operation, - `Knowledge base '${baseId}' index store was closed during ${operation}; retry the operation` - ) - } - throw error - } - } - - private async cancelAllJobsForBase(baseId: string): Promise { - const jobManager = application.get('JobManager') - const activeJobs = await jobManager.list({ - queue: knowledgeQueueName(toKnowledgeBaseId(baseId)), - status: [...KNOWLEDGE_ACTIVE_JOB_STATUSES], - limit: KNOWLEDGE_ACTIVE_JOB_LIMIT - }) - const jobsToCancel = activeJobs.filter((job) => KNOWLEDGE_JOB_TYPE_SET.has(job.type)) - const linkedFileProcessingJobIds = activeJobs.flatMap((job) => { - const narrowed = narrowKnowledgeJobInput(job) - return narrowed?.type === 'knowledge.check-file-processing-result' ? [narrowed.input.fileProcessingJobId] : [] - }) - - await Promise.all([ - ...jobsToCancel.map((job) => jobManager.cancel(job.id, 'delete-base')), - ...linkedFileProcessingJobIds.map((jobId) => jobManager.cancel(jobId, 'delete-base')) - ]) - } - - /** - * Park items stranded mid-indexing by an app quit / restart at `failed`. - * - * Indexing handlers declare `recovery: 'abandon'`, so an interrupted job is - * cancelled rather than silently resumed on the next launch — a deliberate - * quit must not auto-spend the (paid) embedding API. The job side is handled - * by JobManager's startup recovery; this closes the item side. The common case - * (handler settled the abort as cancelled) is already flipped to `failed` by - * the job's onSettled; this is the boot-time safety net for the stragglers - * onSettled lost the race to write (process exited first) or never ran (hard - * kill / crash). Marking them `failed` clears the perpetual spinner and makes - * them reindexable so the user can finish them on demand. - */ - private recoverInterruptedItems(): void { - try { - const failedCount = knowledgeItemService.failInterruptedItems(KNOWLEDGE_ITEM_ERROR_INDEXING_INTERRUPTED) - if (failedCount > 0) { - logger.info('Recovered interrupted knowledge items', { count: failedCount }) - } - } catch (error) { - logger.error('Failed to recover interrupted knowledge items', error as Error) - } - } - - private recoverDeletingItems(): void { - let deletingRootGroups: Awaited> - try { - deletingRootGroups = knowledgeItemService.getDeletingRootGroups() - } catch (error) { - logger.error('Failed to scan deleting knowledge items for recovery', error as Error) - return - } - - if (deletingRootGroups.length === 0) { - return - } - - const jobManager = application.get('JobManager') - for (const { baseId, rootItemIds } of deletingRootGroups) { - for (let i = 0; i < rootItemIds.length; i += DELETE_RECOVERY_ROOT_CHUNK_SIZE) { - const rootItemIdChunk = rootItemIds.slice(i, i + DELETE_RECOVERY_ROOT_CHUNK_SIZE) - try { - jobManager.enqueue( - 'knowledge.delete-subtree', - { baseId, rootItemIds: rootItemIdChunk }, - { - idempotencyKey: knowledgeDeleteSubtreeIdempotencyKey( - toKnowledgeBaseId(baseId), - toKnowledgeItemIds(rootItemIdChunk) - ), - queue: knowledgeQueueName(toKnowledgeBaseId(baseId)) - } - ) - } catch (error) { - logger.error('Failed to enqueue recovered knowledge delete cleanup', error as Error, { - baseId, - rootItemIds: rootItemIdChunk - }) - } - } - } - } - - private assertBaseCanRunRuntimeOperation(baseId: string, operation: string): void { - const base = knowledgeBaseService.getById(baseId) - - if (base.status !== 'failed') { - return - } - - throw DataApiErrorFactory.validation( - { - base: [`Knowledge base '${baseId}' is in failed state; restore it before ${operation}.`] - }, - `Cannot ${operation} failed knowledge base` - ) - } - - private async getRootItemsInBase(baseId: string, itemIds: string[]): Promise { - const rootIds = [...new Set(itemIds)] - const items = await Promise.all(rootIds.map((itemId) => knowledgeItemService.getById(itemId))) - const invalidItem = items.find((item) => item.baseId !== baseId) - - if (invalidItem) { - throw new Error(`Knowledge item '${invalidItem.id}' does not belong to base '${baseId}'`) - } - - return items - } - - private async assertItemCanRunChunkOperation( - baseId: string, - itemId: string, - operation: 'list chunks' | 'delete chunk' - ): Promise { - const [item] = await this.getRootItemsInBase(baseId, [itemId]) - - if (item.status !== 'completed') { - throw DataApiErrorFactory.validation( - { item: [`Knowledge item '${itemId}' must be completed before ${operation}`] }, - `Cannot ${operation} for a non-completed knowledge item` - ) - } - - return item - } - - private assertCompletedContainerHasNoDeletingChildren(baseId: string, item: KnowledgeItem): void { - if (item.type !== 'directory') { - return - } - - const subtreeItems = knowledgeItemService.getSubtreeItems(baseId, [item.id]) - if (subtreeItems.some((item) => item.status === 'deleting')) { - throw DataApiErrorFactory.validation( - { item: [`Knowledge item subtree '${item.id}' is being deleted`] }, - 'Cannot list chunks for a deleting knowledge item' - ) - } - } - - private async assertSubtreesCanReindex(baseId: string, rootItemIds: string[]): Promise { - const blockingStatusCounts = new Map() - const missingSourceItemIds: string[] = [] - const unverifiableSourceItemIds: string[] = [] - - for (const rootItemId of rootItemIds) { - const subtreeItems = knowledgeItemService.getSubtreeItems(baseId, [rootItemId], { includeRoots: true }) - - // Reindex deletes the subtree's vectors before re-reading the source (reindexSubtreeJobHandler), - // so a root whose source is gone would lose its vectors with nothing to rebuild from — reject up - // front. Only the root's own source matters: a directory is rescanned from data.source and its - // children recreated (never read from their raw/ files), a file leaf reads its own raw/ file, and - // note/url always rebuild from the DB / network. A v1-migrated folder child reindexed on its own - // is a file root whose raw/ file never existed, so this rejects it too. Distinguish a genuinely - // missing source (delete-and-re-add) from one we could not verify (transient/permission error, - // which should retry rather than be destroyed). - const root = subtreeItems.find((item) => item.id === rootItemId) - if (root) { - const sourceState = await classifyKnowledgeItemSource(baseId, root) - if (sourceState === 'missing') { - missingSourceItemIds.push(rootItemId) - } else if (sourceState === 'unverifiable') { - unverifiableSourceItemIds.push(rootItemId) - } - } - - for (const item of subtreeItems) { - if (REINDEX_ALLOWED_STATUSES.has(item.status)) { - continue - } - - blockingStatusCounts.set(item.status, (blockingStatusCounts.get(item.status) ?? 0) + 1) - } - } - - if (missingSourceItemIds.length > 0) { - throw DataApiErrorFactory.validation( - { - item: [`Knowledge item source no longer exists on disk for ${missingSourceItemIds.length} item(s)`] - }, - 'Cannot reindex a knowledge item whose source file or folder no longer exists; delete it and add it again to rebuild' - ) - } - - if (unverifiableSourceItemIds.length > 0) { - throw DataApiErrorFactory.validation( - { - item: [`Could not verify the knowledge item source on disk for ${unverifiableSourceItemIds.length} item(s)`] - }, - 'Could not verify the knowledge item source (it may be temporarily unavailable); please try again' - ) - } - - if (blockingStatusCounts.size === 0) { - return - } - - const statusSummary = [...blockingStatusCounts.entries()] - .sort(([left], [right]) => left.localeCompare(right)) - .map(([status, count]) => `${status}=${count}`) - .join(', ') - - throw DataApiErrorFactory.validation( - { - item: [`Knowledge item subtree is still running or being deleted: ${statusSummary}`] - }, - 'Cannot reindex knowledge item until the entire subtree is completed or failed' - ) - } - - private toRestoreRuntimeInput(sourceBaseId: string, item: KnowledgeItem): KnowledgeAddItemInput { - try { - if (item.type === 'file') { - return KnowledgeAddItemInputSchema.parse({ - type: 'file', - data: { - source: item.data.source, - path: getKnowledgeBaseFilePath(sourceBaseId, item.data.relativePath), - // Carry the processed artifact across so the new base indexes from it - // instead of re-running the (slow, paid) file processor. - ...(item.data.indexedRelativePath - ? { indexedPath: getKnowledgeBaseFilePath(sourceBaseId, item.data.indexedRelativePath) } - : {}) - } - }) - } - - if (item.type === 'url') { - return KnowledgeAddItemInputSchema.parse({ - type: 'url', - data: { - source: item.data.source, - url: item.data.url, - // Carry the captured snapshot across so the restored URL indexes offline - // instead of re-fetching the live page (which may have changed or died). - // If the source never captured one, omit it and let the first index capture. - ...(item.data.relativePath - ? { snapshotPath: getKnowledgeBaseFilePath(sourceBaseId, item.data.relativePath) } - : {}) - } - }) - } - - if (item.type === 'note') { - return KnowledgeAddItemInputSchema.parse({ - type: 'note', - // The snapshot relativePath is intentionally dropped: the content is the - // source of truth and re-capturing it into the new base on first index is - // free and deterministic, so there is no snapshot file to carry across. - data: { source: item.data.source, content: item.data.content } - }) - } - - return KnowledgeAddItemInputSchema.parse({ - type: item.type, - data: item.data - }) - } catch (error) { - throw DataApiErrorFactory.invalidOperation( - 'restoreBase', - `Cannot restore knowledge item '${item.id}' (${item.type}): ${ - error instanceof Error ? error.message : String(error) - }` - ) - } + return this.conceptService.getOrganizationTree(baseId, options) } } diff --git a/src/main/features/knowledge/README.md b/src/main/features/knowledge/README.md new file mode 100644 index 00000000000..a9bf5fcbe86 --- /dev/null +++ b/src/main/features/knowledge/README.md @@ -0,0 +1,75 @@ +# Knowledge Feature + +Per-base knowledge library: ingest sources (files, directories, urls, notes), convert them to +markdown, chunk + embed the text, and persist everything into a per-base `index.sqlite` +(better-sqlite3 + sqlite-vec) that serves hybrid vector/BM25 search and the Concept ID-addressed +agent tools (`kb_search` / `kb_read` / `kb_tree` / `kb_manage`). + +## Pipeline + +`pipeline/` spells out the ingestion pipeline in stage order: + +``` + input preprocess index persist + ┌──────────────┐ ┌────────────────┐ ┌───────────────┐ ┌───────────────┐ +pipeline/ │ sources/ │ ───> │ readers/ │ ──> │ indexing/ │ ──> │ vectorstore/ │ + │ expand dirs, │ │ file → md text │ │ chunk, embed, │ │ index.sqlite │ + │ url/note │ │ (pdf, docx, …) │ │ rerank │ │ (per base) │ + │ snapshots │ └────────────────┘ └───────────────┘ └───────────────┘ + └──────────────┘ heavy conversions (MinerU/PaddleOCR/…) run out-of-process + via FileProcessingService, polled by a knowledge job +``` + +Jobs in `tasks/` drive the stages; `ingestion/` decides which jobs to enqueue; `query/` reads the +result back out. Nothing under `pipeline/` enqueues jobs or mutates item status — that is +orchestration, and it lives in `ingestion/` and `tasks/`. + +## Directory map + +| Directory | Role | +| --- | --- | +| `KnowledgeService.ts` | Lifecycle facade: registers job handlers, runs boot recovery, delegates every public method, and creates the shared per-base mutation lock (`KeyedMutex`). No domain logic. | +| `base/` | Per-base domain: lifecycle admin (`KnowledgeBaseAdminService` — create with rollback, delete, restore, list), failed-base guard (`baseGuards.ts`). | +| `ingestion/` | Write-side orchestration: admission checks, item creation, add-conflict resolution, job enqueueing, subtree purge (`subtreePurge.ts`), boot recovery. | +| `pipeline/sources/` | Input stage: directory expansion, url fetch (Jina reader), url/note snapshot capture, OKF frontmatter. | +| `pipeline/readers/` | Preprocess stage: file → markdown/text `Document[]` readers (pdf/docx/epub/…). | +| `pipeline/indexing/` | Index stage: offset-preserving splitter + chunker, `AiService` embedding/rerank wrappers. | +| `pipeline/vectorstore/` | Persist stage: per-base `index.sqlite` lifecycle (`KnowledgeVectorStoreService`), the store itself (`indexStore/`, synchronous better-sqlite3 driver), vector deletion + index space reclamation (`vectorCleanup.ts`). | +| `query/` | Read side + Concept ID tool surface: hybrid search with visibility filtering (`KnowledgeQueryService`); Concept ID read/grep/tree plus the `kb_manage` delete/refresh writes, which delegate to `ingestion/` (`KnowledgeConceptService`). | +| `tasks/` | Job handlers — the pipeline executors (see below); `prepareItem.ts` is a prepare-root handler-private helper that expands a directory root into child items. | +| `pathStorage.ts` | `raw/` path allocation: collision-free names, reservation, base file paths. | +| `items.ts` / `types.ts` | Shared item vocabulary (type aliases, predicates, source probing, material-path derivation); branded ids, queue names, idempotency keys. | + +## Jobs + +All jobs run on the per-base queue `base.{baseId}`; idempotency keys prevent double-enqueues. + +| Job | Does | Enqueued by | +| --- | --- | --- | +| `knowledge.prepare-root` | Expand a directory root into child items, then enqueue leaf indexing. | `ingestion` (add), reindex handler | +| `knowledge.index-documents` | Read → chunk → embed → `rebuildMaterial` in one store transaction. | `ingestion`, prepare-root, fp-check | +| `knowledge.check-file-processing-result` | Poll a FileProcessingService job (5s delay per round); on success enqueue indexing. | `ingestion` (files needing conversion) | +| `knowledge.delete-subtree` | Cancel active jobs → delete vectors → delete files → delete rows. | `ingestion` (delete), boot recovery | +| `knowledge.reindex-subtree` | Verify source → delete vectors → reset statuses → re-enqueue indexing. | `ingestion` (reindex) | + +Indexing jobs and `knowledge.reindex-subtree` declare `recovery: 'abandon'` — an app restart never +silently resumes them (that would auto-spend the paid embedding API); boot recovery parks +interrupted items at `failed` instead. Only `knowledge.delete-subtree` uses `recovery: 'retry'`. + +Item status flow: `preparing` (directory) / `processing` → `completed` | `failed`; any status → +`deleting` → row removed. `reading`/`embedding` are transient sub-phases surfaced while the index +job runs. + +## Concurrency + +The per-base mutation lock is a core `KeyedMutex` (acquired via `runExclusive`), an **application-level** mutex serializing multi-step business +invariants that span the main DB, the index store, and the filesystem (e.g. add's +read-conflicts-then-create-rows sequence). It is not about protecting SQLite itself — the per-base +driver is synchronous, and single statements are atomic. Handlers acquire the lock only around the +mutation section, never across slow I/O (fetch, read, embed). + +## Related docs + +- Data-layer selection and patterns: [docs/references/data/README.md](../../../../docs/references/data/README.md) +- Concept ID = material relative path (OKF §2): the addressing primitive for `kb_read`/`kb_manage`; + resolved against the index store and re-validated against the visible `knowledge_item`. diff --git a/src/main/features/knowledge/__tests__/KnowledgeLockManager.test.ts b/src/main/features/knowledge/__tests__/KnowledgeLockManager.test.ts deleted file mode 100644 index 9c8cc884ac0..00000000000 --- a/src/main/features/knowledge/__tests__/KnowledgeLockManager.test.ts +++ /dev/null @@ -1,100 +0,0 @@ -import { describe, expect, it } from 'vitest' - -import { KnowledgeLockManager } from '../KnowledgeLockManager' - -function createDeferred() { - let resolve!: (value: T | PromiseLike) => void - let reject!: (reason?: unknown) => void - const promise = new Promise((promiseResolve, promiseReject) => { - resolve = promiseResolve - reject = promiseReject - }) - return { promise, resolve, reject } -} - -function flushMicrotasks(): Promise { - return new Promise((resolve) => setTimeout(resolve, 0)) -} - -function getBaseMutexCount(lockManager: KnowledgeLockManager): number { - return ( - lockManager as unknown as { - baseMutexes: Map - } - ).baseMutexes.size -} - -describe('KnowledgeLockManager', () => { - it('serializes overlapping tasks for the same base', async () => { - const lockManager = new KnowledgeLockManager() - const releaseFirst = createDeferred() - const order: string[] = [] - - const firstTask = lockManager.withBaseMutationLock('kb-1', async () => { - order.push('first-start') - await releaseFirst.promise - order.push('first-end') - }) - const secondTask = lockManager.withBaseMutationLock('kb-1', async () => { - order.push('second-start') - }) - - await flushMicrotasks() - expect(order).toEqual(['first-start']) - - releaseFirst.resolve() - await Promise.all([firstTask, secondTask]) - - expect(order).toEqual(['first-start', 'first-end', 'second-start']) - }) - - it('runs tasks for different bases concurrently', async () => { - const lockManager = new KnowledgeLockManager() - const releaseFirst = createDeferred() - const order: string[] = [] - - const firstTask = lockManager.withBaseMutationLock('kb-1', async () => { - order.push('first-start') - await releaseFirst.promise - order.push('first-end') - }) - const secondTask = lockManager.withBaseMutationLock('kb-2', async () => { - order.push('second-start') - }) - - await flushMicrotasks() - expect(order).toEqual(['first-start', 'second-start']) - - releaseFirst.resolve() - await firstTask - await secondTask - }) - - it('releases a base lock when the task throws', async () => { - const lockManager = new KnowledgeLockManager() - const order: string[] = [] - - await expect( - lockManager.withBaseMutationLock('kb-1', async () => { - order.push('throwing') - throw new Error('boom') - }) - ).rejects.toThrow('boom') - - await lockManager.withBaseMutationLock('kb-1', async () => { - order.push('after-throw') - }) - - expect(order).toEqual(['throwing', 'after-throw']) - }) - - it('removes idle base mutex entries after tasks complete', async () => { - const lockManager = new KnowledgeLockManager() - - await lockManager.withBaseMutationLock('kb-1', async () => { - expect(getBaseMutexCount(lockManager)).toBe(1) - }) - - expect(getBaseMutexCount(lockManager)).toBe(0) - }) -}) diff --git a/src/main/features/knowledge/__tests__/KnowledgeService.integration.test.ts b/src/main/features/knowledge/__tests__/KnowledgeService.integration.test.ts index 4e06fc56df4..3b33f085992 100644 --- a/src/main/features/knowledge/__tests__/KnowledgeService.integration.test.ts +++ b/src/main/features/knowledge/__tests__/KnowledgeService.integration.test.ts @@ -15,13 +15,16 @@ import { setupTestDatabase } from '@test-helpers/db' import { eq, isNull } from 'drizzle-orm' import { beforeEach, describe, expect, it, vi } from 'vitest' -const { getIndexStoreMock, deleteStoreMock, enqueueMock, listMock, registerHandlerMock } = vi.hoisted(() => ({ - getIndexStoreMock: vi.fn(), - deleteStoreMock: vi.fn(), - enqueueMock: vi.fn(), - listMock: vi.fn(), - registerHandlerMock: vi.fn() -})) +const { getIndexStoreMock, deleteStoreMock, enqueueMock, enqueueTxMock, listMock, registerHandlerMock } = vi.hoisted( + () => ({ + getIndexStoreMock: vi.fn(), + deleteStoreMock: vi.fn(), + enqueueMock: vi.fn(), + enqueueTxMock: vi.fn(), + listMock: vi.fn(), + registerHandlerMock: vi.fn() + }) +) vi.mock('@application', async () => { const { mockApplicationFactory } = await import('@test-mocks/main/application') @@ -30,6 +33,7 @@ vi.mock('@application', async () => { cancel: vi.fn(), cancelMany: vi.fn(), enqueue: enqueueMock, + enqueueTx: enqueueTxMock, list: listMock, registerHandler: registerHandlerMock }, @@ -70,6 +74,7 @@ describe('KnowledgeService integration', () => { getIndexStoreMock.mockResolvedValue({}) deleteStoreMock.mockResolvedValue(undefined) enqueueMock.mockResolvedValue({ id: 'job-1', snapshot: {}, finished: Promise.resolve({}) }) + enqueueTxMock.mockReturnValue({ id: 'job-1', snapshot: {}, finished: Promise.resolve({}) }) listMock.mockResolvedValue([]) const [providerOrderKey, embeddingModelOrderKey] = generateOrderKeySequence(2) @@ -117,7 +122,7 @@ describe('KnowledgeService integration', () => { groupId: null, type: 'note', data: { source: 'source-root', content: 'root content' }, - status: 'idle', + status: 'processing', error: null }, { @@ -126,7 +131,7 @@ describe('KnowledgeService integration', () => { groupId: SOURCE_ROOT_ITEM_ID, type: 'note', data: { source: 'source-child', content: 'child content' }, - status: 'idle', + status: 'processing', error: null } ]) @@ -181,7 +186,7 @@ describe('KnowledgeService integration', () => { expect(enqueueMock).toHaveBeenCalledWith( 'knowledge.index-documents', - { baseId: restoredBase.id, itemId: restoredItems[0].id, parentJobId: null }, + { baseId: restoredBase.id, itemId: restoredItems[0].id }, { idempotencyKey: `knowledge:${restoredBase.id}:${restoredItems[0].id}:index`, queue: `base.${restoredBase.id}`, @@ -207,6 +212,21 @@ describe('KnowledgeService integration', () => { expect(ungroupedRestoredItems.some((item) => item.baseId === restoredBase.id)).toBe(true) }) + it('rolls back the deleting status when enqueueTx fails inside deleteItems', async () => { + const service = new KnowledgeService() + enqueueTxMock.mockImplementationOnce(() => { + throw new Error('enqueue failed') + }) + + await expect(service.deleteItems(SOURCE_BASE_ID, [SOURCE_ROOT_ITEM_ID])).rejects.toThrow('enqueue failed') + + const [rootRow] = await dbh.db + .select() + .from(knowledgeItemTable) + .where(eq(knowledgeItemTable.id, SOURCE_ROOT_ITEM_ID)) + expect(rootRow.status).toBe('processing') + }) + describe('addItems conflict resolution', () => { const COMPLETED_BASE_ID = '33333333-3333-4333-8333-333333333333' const EXISTING_NOTE_ID = '0198f3f2-7d2a-7abc-8def-123456789abc' diff --git a/src/main/features/knowledge/__tests__/KnowledgeService.test.ts b/src/main/features/knowledge/__tests__/KnowledgeService.test.ts index 4b256dd0a49..33df0e3229c 100644 --- a/src/main/features/knowledge/__tests__/KnowledgeService.test.ts +++ b/src/main/features/knowledge/__tests__/KnowledgeService.test.ts @@ -10,7 +10,7 @@ import { } from '@shared/data/types/knowledge' import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' -import type * as PathStorage from '../utils/storage/pathStorage' +import type * as PathStorage from '../pathStorage' const { cancelManyMock, @@ -18,6 +18,7 @@ const { getIndexStoreMock, deleteStoreMock, enqueueMock, + enqueueTxMock, fileProcessingStartJobMock, getJobMock, aiEmbedManyMock, @@ -25,7 +26,7 @@ const { knowledgeBaseDeleteMock, knowledgeBaseGetByIdMock, knowledgeBaseListMock, - knowledgeItemCreateMock, + knowledgeItemCreateActiveMock, knowledgeItemDeleteMock, knowledgeItemGetDeletingRootGroupsMock, knowledgeItemFailInterruptedItemsMock, @@ -35,6 +36,7 @@ const { knowledgeItemGetRootItemsByBaseIdMock, knowledgeItemGetSubtreeItemsMock, knowledgeItemSetSubtreeStatusMock, + knowledgeItemSetSubtreeStatusTxMock, knowledgeItemUpdateStatusMock, listMock, registerHandlerMock, @@ -55,6 +57,7 @@ const { getIndexStoreMock: vi.fn(), deleteStoreMock: vi.fn(), enqueueMock: vi.fn(), + enqueueTxMock: vi.fn(), fileProcessingStartJobMock: vi.fn(), getJobMock: vi.fn(), aiEmbedManyMock: vi.fn(), @@ -62,7 +65,7 @@ const { knowledgeBaseDeleteMock: vi.fn(), knowledgeBaseGetByIdMock: vi.fn(), knowledgeBaseListMock: vi.fn(), - knowledgeItemCreateMock: vi.fn(), + knowledgeItemCreateActiveMock: vi.fn(), knowledgeItemDeleteMock: vi.fn(), knowledgeItemGetDeletingRootGroupsMock: vi.fn(), knowledgeItemFailInterruptedItemsMock: vi.fn(), @@ -72,6 +75,7 @@ const { knowledgeItemGetRootItemsByBaseIdMock: vi.fn(), knowledgeItemGetSubtreeItemsMock: vi.fn(), knowledgeItemSetSubtreeStatusMock: vi.fn(), + knowledgeItemSetSubtreeStatusTxMock: vi.fn(), knowledgeItemUpdateStatusMock: vi.fn(), listMock: vi.fn(), registerHandlerMock: vi.fn(), @@ -98,6 +102,7 @@ vi.mock('@application', async () => { cancel: cancelMock, cancelMany: cancelManyMock, enqueue: enqueueMock, + enqueueTx: enqueueTxMock, get: getJobMock, list: listMock, registerHandler: registerHandlerMock @@ -156,7 +161,7 @@ vi.mock('@data/services/KnowledgeBaseService', () => ({ vi.mock('@data/services/KnowledgeItemService', () => ({ knowledgeItemService: { - create: knowledgeItemCreateMock, + createActive: knowledgeItemCreateActiveMock, delete: knowledgeItemDeleteMock, getDeletingRootGroups: knowledgeItemGetDeletingRootGroupsMock, failInterruptedItems: knowledgeItemFailInterruptedItemsMock, @@ -166,16 +171,17 @@ vi.mock('@data/services/KnowledgeItemService', () => ({ getOutermostSelectedItemIds: knowledgeItemGetOutermostSelectedItemIdsMock, getRootItemsByBaseId: knowledgeItemGetRootItemsByBaseIdMock, setSubtreeStatus: knowledgeItemSetSubtreeStatusMock, + setSubtreeStatusTx: knowledgeItemSetSubtreeStatusTxMock, updateStatus: knowledgeItemUpdateStatusMock } })) -vi.mock('../utils/indexing/rerank', () => ({ +vi.mock('../pipeline/indexing/rerank', () => ({ rerankKnowledgeSearchResults: rerankKnowledgeSearchResultsMock })) -vi.mock('../utils/storage/pathStorage', async () => { - const actual = await vi.importActual('../utils/storage/pathStorage') +vi.mock('../pathStorage', async () => { + const actual = await vi.importActual('../pathStorage') return { ...actual, copyFileIntoKnowledgeBaseAt: copyFileIntoKnowledgeBaseAtMock, @@ -185,7 +191,8 @@ vi.mock('../utils/storage/pathStorage', async () => { } }) -const { KnowledgeService, KNOWLEDGE_TREE_MAX_NODES } = await import('../KnowledgeService') +const { KnowledgeService } = await import('../KnowledgeService') +const { KNOWLEDGE_TREE_MAX_NODES } = await import('../query/KnowledgeConceptService') const NOTE_ITEM_ID = '0198f3f2-7d1a-7abc-8def-123456789abc' const DELETING_NOTE_ITEM_ID = '0198f3f2-7d1b-7abc-8def-123456789abc' @@ -221,7 +228,7 @@ function createNoteItem( id = 'note-1', baseId = 'kb-1', groupId: string | null = null, - status: KnowledgeItemOf<'note'>['status'] = 'idle' + status: KnowledgeItemOf<'note'>['status'] = 'processing' ): KnowledgeItemOf<'note'> { const lifecycle = status === 'failed' ? ({ status, error: `failed ${id}` } as const) : ({ status, error: null } as const) @@ -241,7 +248,7 @@ function createNoteItem( function createDirectoryItem( id = 'dir-1', groupId: string | null = null, - status: KnowledgeItemOf<'directory'>['status'] = 'idle' + status: KnowledgeItemOf<'directory'>['status'] = 'processing' ): KnowledgeItemOf<'directory'> { const lifecycle = status === 'failed' ? ({ status, error: `failed ${id}` } as const) : ({ status, error: null } as const) @@ -262,7 +269,7 @@ function createFileItem( id = 'file-1', baseId = 'kb-1', source = '/docs/source.pdf', - status: KnowledgeItemOf<'file'>['status'] = 'idle' + status: KnowledgeItemOf<'file'>['status'] = 'processing' ): KnowledgeItemOf<'file'> { const lifecycle = status === 'failed' ? ({ status, error: `failed ${id}` } as const) : ({ status, error: null } as const) @@ -327,13 +334,15 @@ describe('KnowledgeService', () => { copyFileIntoKnowledgeBaseAtMock.mockImplementation( async (_baseId: string, _sourcePath: string, relativePath: string) => relativePath ) - knowledgeItemCreateMock.mockImplementation((baseId: string, input: { type?: string; data: { source: string } }) => { - createdItemBaseIds.set(input.data.source, baseId) - if (input.type === 'file') { - return createFileItem(input.data.source, baseId, input.data.source) + knowledgeItemCreateActiveMock.mockImplementation( + (baseId: string, input: { type?: string; data: { source: string } }) => { + createdItemBaseIds.set(input.data.source, baseId) + if (input.type === 'file') { + return createFileItem(input.data.source, baseId, input.data.source, 'processing') + } + return createNoteItem(input.data.source, baseId, null, 'processing') } - return createNoteItem(input.data.source, baseId) - }) + ) knowledgeItemDeleteMock.mockReturnValue(undefined) deleteKnowledgeItemFilesBestEffortMock.mockResolvedValue(undefined) knowledgeItemGetDeletingRootGroupsMock.mockReturnValue([]) @@ -355,20 +364,22 @@ describe('KnowledgeService', () => { } ) knowledgeItemSetSubtreeStatusMock.mockReturnValue(['note-1']) + knowledgeItemSetSubtreeStatusTxMock.mockReturnValue(['note-1']) knowledgeItemUpdateStatusMock.mockImplementation((id: string, status: KnowledgeItemOf<'note'>['status']) => { return createNoteItem(id, createdItemBaseIds.get(id) ?? 'kb-1', null, status) }) enqueueMock.mockReturnValue({ id: 'job-1', snapshot: {}, finished: Promise.resolve({}) }) + enqueueTxMock.mockReturnValue({ id: 'job-1', snapshot: {}, finished: Promise.resolve({}) }) fileProcessingStartJobMock.mockResolvedValue({ id: 'fp-job-1', snapshot: {}, finished: Promise.resolve({}) }) getJobMock.mockResolvedValue(null) listMock.mockResolvedValue([]) - getIndexStoreMock.mockResolvedValue({ + getIndexStoreMock.mockReturnValue({ search: storeSearchMock, listMaterialUnits: listMaterialUnitsMock, getMaterialByRelativePath: getMaterialByRelativePathMock, readMaterialContent: readMaterialContentMock }) - listMaterialUnitsMock.mockResolvedValue([]) + listMaterialUnitsMock.mockReturnValue([]) storeSearchMock.mockResolvedValue([]) getMaterialByRelativePathMock.mockResolvedValue(null) readMaterialContentMock.mockResolvedValue(null) @@ -528,7 +539,9 @@ describe('KnowledgeService', () => { ) expect(getIndexStoreMock).toHaveBeenCalledWith(base) - getIndexStoreMock.mockRejectedValueOnce(new Error('store failed')) + getIndexStoreMock.mockImplementationOnce(() => { + throw new Error('store failed') + }) await expect( service.createBase({ name: 'KB', dimensions: 3, embeddingModelId: 'provider::embed' }) ).rejects.toThrow('store failed') @@ -537,7 +550,9 @@ describe('KnowledgeService', () => { it('rollback removes the orphaned index dir and still surfaces the original error when cleanup itself fails', async () => { const service = new KnowledgeService() - getIndexStoreMock.mockRejectedValueOnce(new Error('store failed')) + getIndexStoreMock.mockImplementationOnce(() => { + throw new Error('store failed') + }) // Even if the orphan-dir cleanup throws, the caller must see the open error. deleteStoreMock.mockRejectedValueOnce(new Error('cleanup boom')) @@ -557,6 +572,13 @@ describe('KnowledgeService', () => { expect(listMock).toHaveBeenCalledWith({ queue: 'base.kb-1', status: ['pending', 'delayed', 'running'], + type: [ + 'knowledge.prepare-root', + 'knowledge.index-documents', + 'knowledge.check-file-processing-result', + 'knowledge.delete-subtree', + 'knowledge.reindex-subtree' + ], limit: 5000 }) expect(deleteStoreMock).toHaveBeenCalledWith('kb-1') @@ -579,7 +601,7 @@ describe('KnowledgeService', () => { fileProcessingJobId: 'fp-job-1', pollRound: 0, firstScheduledAt: 1779811200000, - parentJobId: null + processedRelativePath: 'source.md' } } ]) @@ -885,8 +907,7 @@ describe('KnowledgeService', () => { ...createFileItem('restored-file', 'restored-kb', '/docs/report.pdf', 'processing'), data: { source: '/docs/report.pdf', relativePath: 'report.pdf', indexedRelativePath: 'report.md' } } - knowledgeItemCreateMock.mockReturnValueOnce(restoredFile) - knowledgeItemUpdateStatusMock.mockReturnValueOnce(restoredFile) + knowledgeItemCreateActiveMock.mockReturnValueOnce(restoredFile) knowledgeItemGetByIdMock.mockReturnValue(restoredFile) await service.restoreBase({ @@ -902,7 +923,7 @@ describe('KnowledgeService', () => { ['restored-kb', '/mock/feature.knowledgebase.data/source-kb/raw/report.md', 'report.md'] ]) // The created item carries the artifact path. - expect(knowledgeItemCreateMock).toHaveBeenCalledWith( + expect(knowledgeItemCreateActiveMock).toHaveBeenCalledWith( 'restored-kb', expect.objectContaining({ type: 'file', @@ -946,7 +967,7 @@ describe('KnowledgeService', () => { 'example-page.md' ) // The created url item is pinned to the copied snapshot so first index reads it offline. - expect(knowledgeItemCreateMock).toHaveBeenCalledWith( + expect(knowledgeItemCreateActiveMock).toHaveBeenCalledWith( 'restored-kb', expect.objectContaining({ type: 'url', @@ -977,7 +998,7 @@ describe('KnowledgeService', () => { }) // No snapshot to carry: the restored url has no relativePath so first index re-captures it. - expect(knowledgeItemCreateMock).toHaveBeenCalledWith( + expect(knowledgeItemCreateActiveMock).toHaveBeenCalledWith( 'restored-kb', expect.objectContaining({ type: 'url', data: { source: 'https://example.com', url: 'https://example.com' } }) ) @@ -994,21 +1015,19 @@ describe('KnowledgeService', () => { expect(enqueueMock.mock.calls.map((call) => call[0])).toEqual([ 'knowledge.index-documents', - 'knowledge.delete-subtree', 'knowledge.reindex-subtree' ]) - expect(knowledgeItemSetSubtreeStatusMock).toHaveBeenCalledWith('kb-1', ['note-1'], 'deleting') + expect(enqueueTxMock.mock.calls.map((call) => call[1])).toEqual(['knowledge.delete-subtree']) + expect(knowledgeItemSetSubtreeStatusTxMock).toHaveBeenCalledWith(expect.anything(), 'kb-1', ['note-1'], 'deleting') }) it('starts file processing and schedules a check job for supported document files when the base has a processor', async () => { vi.useFakeTimers() vi.setSystemTime(new Date('2026-04-08T00:00:00.000Z')) const service = new KnowledgeService() - const createdFile = createFileItem('file-1', 'kb-1', '/docs/source.pdf') const processingFile = createFileItem('file-1', 'kb-1', '/docs/source.pdf', 'processing') knowledgeBaseGetByIdMock.mockReturnValue(createBase({ fileProcessorId: 'doc2x' })) - knowledgeItemCreateMock.mockReturnValueOnce(createdFile) - knowledgeItemUpdateStatusMock.mockReturnValueOnce(processingFile) + knowledgeItemCreateActiveMock.mockReturnValueOnce(processingFile) knowledgeItemGetByIdMock.mockReturnValueOnce(processingFile) await service.addItems('kb-1', [{ type: 'file', data: { source: '/docs/source.pdf', path: '/docs/source.pdf' } }]) @@ -1033,7 +1052,7 @@ describe('KnowledgeService', () => { fileProcessingJobId: 'fp-job-1', pollRound: 0, firstScheduledAt: expect.any(Number), - parentJobId: 'fp-job-1' + processedRelativePath: 'source.md' }, expect.objectContaining({ idempotencyKey: 'knowledge:kb-1:file-1:fp-check:fp-job-1:0', @@ -1048,10 +1067,7 @@ describe('KnowledgeService', () => { it('auto-renames a duplicate uploaded file name instead of rejecting the import', async () => { const service = new KnowledgeService() knowledgeBaseGetByIdMock.mockReturnValue(createBase({ fileProcessorId: null })) - knowledgeItemCreateMock - .mockReturnValueOnce(createFileItem('file-1', 'kb-1', '/Users/me/a/notes.md')) - .mockReturnValueOnce(createFileItem('file-2', 'kb-1', '/Users/me/b/notes.md')) - knowledgeItemUpdateStatusMock + knowledgeItemCreateActiveMock .mockReturnValueOnce(createFileItem('file-1', 'kb-1', '/Users/me/a/notes.md', 'processing')) .mockReturnValueOnce(createFileItem('file-2', 'kb-1', '/Users/me/b/notes.md', 'processing')) knowledgeItemGetByIdMock @@ -1064,7 +1080,7 @@ describe('KnowledgeService', () => { ]) // Both imports land; the second's relativePath is deduped (`_N`) rather than refused. - expect(knowledgeItemCreateMock).toHaveBeenCalledTimes(2) + expect(knowledgeItemCreateActiveMock).toHaveBeenCalledTimes(2) expect(copyFileIntoKnowledgeBaseAtMock).toHaveBeenNthCalledWith(1, 'kb-1', '/Users/me/a/notes.md', 'notes.md') expect(copyFileIntoKnowledgeBaseAtMock).toHaveBeenNthCalledWith(2, 'kb-1', '/Users/me/b/notes.md', 'notes_1.md') }) @@ -1072,10 +1088,7 @@ describe('KnowledgeService', () => { it('auto-renames a file whose processed-markdown name would collide', async () => { const service = new KnowledgeService() knowledgeBaseGetByIdMock.mockReturnValue(createBase({ fileProcessorId: 'doc2x' })) - knowledgeItemCreateMock - .mockReturnValueOnce(createFileItem('file-1', 'kb-1', '/Users/me/a/brief.pdf')) - .mockReturnValueOnce(createFileItem('file-2', 'kb-1', '/Users/me/b/brief.docx')) - knowledgeItemUpdateStatusMock + knowledgeItemCreateActiveMock .mockReturnValueOnce(createFileItem('file-1', 'kb-1', '/Users/me/a/brief.pdf', 'processing')) .mockReturnValueOnce(createFileItem('file-2', 'kb-1', '/Users/me/b/brief.docx', 'processing')) knowledgeItemGetByIdMock @@ -1089,7 +1102,7 @@ describe('KnowledgeService', () => { // brief.pdf reserves brief.pdf + its brief.md output; brief.docx would also emit // brief.md, so it is bumped to brief_1.docx (whose brief_1.md sibling is free). - expect(knowledgeItemCreateMock).toHaveBeenCalledTimes(2) + expect(knowledgeItemCreateActiveMock).toHaveBeenCalledTimes(2) expect(copyFileIntoKnowledgeBaseAtMock).toHaveBeenNthCalledWith(1, 'kb-1', '/Users/me/a/brief.pdf', 'brief.pdf') expect(copyFileIntoKnowledgeBaseAtMock).toHaveBeenNthCalledWith(2, 'kb-1', '/Users/me/b/brief.docx', 'brief_1.docx') }) @@ -1125,7 +1138,7 @@ describe('KnowledgeService', () => { '/captured/example-page.md', 'example-page_1.md' ) - expect(knowledgeItemCreateMock).toHaveBeenCalledWith( + expect(knowledgeItemCreateActiveMock).toHaveBeenCalledWith( 'kb-1', expect.objectContaining({ type: 'url', @@ -1139,7 +1152,7 @@ describe('KnowledgeService', () => { knowledgeBaseGetByIdMock.mockReturnValue(createBase({ fileProcessorId: null })) knowledgeItemGetItemsByBaseIdMock.mockReturnValue([]) // The url restore copies its snapshot to raw/ before the row is created; that create fails. - knowledgeItemCreateMock.mockImplementationOnce(() => { + knowledgeItemCreateActiveMock.mockImplementationOnce(() => { throw new Error('db down') }) @@ -1177,8 +1190,7 @@ describe('KnowledgeService', () => { data: { source: 'Meeting notes', content: 'hello', relativePath: 'Meeting notes.md' } } ]) - knowledgeItemCreateMock.mockReturnValueOnce(createFileItem('file-1', 'kb-1', '/Users/me/Meeting notes.md')) - knowledgeItemUpdateStatusMock.mockReturnValueOnce( + knowledgeItemCreateActiveMock.mockReturnValueOnce( createFileItem('file-1', 'kb-1', '/Users/me/Meeting notes.md', 'processing') ) knowledgeItemGetByIdMock.mockReturnValueOnce( @@ -1197,7 +1209,7 @@ describe('KnowledgeService', () => { '/Users/me/Meeting notes.md', 'Meeting notes_1.md' ) - expect(knowledgeItemCreateMock).toHaveBeenCalledWith( + expect(knowledgeItemCreateActiveMock).toHaveBeenCalledWith( 'kb-1', expect.objectContaining({ type: 'file', @@ -1220,17 +1232,17 @@ describe('KnowledgeService', () => { } ]) - const workflowService = ( + const ingestionService = ( service as unknown as { - workflowService: { + ingestionService: { scheduleItem(baseId: string, itemId: string, parentJobId?: string | null): Promise } } - ).workflowService + ).ingestionService // The processed-artifact reservation guard must treat the note snapshot as occupied (it lives // under `raw/` too), so it refuses the colliding `.md` output instead of overwriting it on disk. - await expect(workflowService.scheduleItem('kb-1', 'file-1')).rejects.toThrow( + await expect(ingestionService.scheduleItem('kb-1', 'file-1')).rejects.toThrow( 'Knowledge file already exists: source.md' ) expect(fileProcessingStartJobMock).not.toHaveBeenCalled() @@ -1274,7 +1286,7 @@ describe('KnowledgeService', () => { service.addItems('kb-1', [{ type: 'file', data: { source: '/Users/me/app.exe', path: '/Users/me/app.exe' } }]) ).rejects.toThrow('Unsupported knowledge file type: /Users/me/app.exe') - expect(knowledgeItemCreateMock).not.toHaveBeenCalled() + expect(knowledgeItemCreateActiveMock).not.toHaveBeenCalled() expect(copyFileIntoKnowledgeBaseAtMock).not.toHaveBeenCalled() expect(fileProcessingStartJobMock).not.toHaveBeenCalled() }) @@ -1285,14 +1297,14 @@ describe('KnowledgeService', () => { knowledgeBaseGetByIdMock.mockReturnValue(createBase({ fileProcessorId: 'doc2x' })) knowledgeItemGetByIdMock.mockReturnValueOnce(processingFile) - const workflowService = ( + const ingestionService = ( service as unknown as { - workflowService: { + ingestionService: { scheduleItem(baseId: string, itemId: string, parentJobId?: string | null): Promise } } - ).workflowService - await workflowService.scheduleItem('kb-1', 'file-1', 'reindex-job') + ).ingestionService + await ingestionService.scheduleItem('kb-1', 'file-1', 'reindex-job') expect(fileProcessingStartJobMock).toHaveBeenCalledWith( { @@ -1314,7 +1326,7 @@ describe('KnowledgeService', () => { fileProcessingJobId: 'fp-job-1', pollRound: 0, firstScheduledAt: expect.any(Number), - parentJobId: 'reindex-job' + processedRelativePath: 'source.md' }, expect.objectContaining({ idempotencyKey: 'knowledge:kb-1:file-1:fp-check:fp-job-1:0', @@ -1334,15 +1346,15 @@ describe('KnowledgeService', () => { throw new Error('check enqueue failed') }) - const workflowService = ( + const ingestionService = ( service as unknown as { - workflowService: { + ingestionService: { scheduleItem(baseId: string, itemId: string, parentJobId?: string | null): Promise } } - ).workflowService + ).ingestionService - await expect(workflowService.scheduleItem('kb-1', 'file-1')).rejects.toThrow('check enqueue failed') + await expect(ingestionService.scheduleItem('kb-1', 'file-1')).rejects.toThrow('check enqueue failed') expect(fileProcessingStartJobMock).toHaveBeenCalled() expect(cancelMock).toHaveBeenCalledWith('fp-job-1', 'knowledge-file-processing-check-enqueue-failed') @@ -1358,15 +1370,15 @@ describe('KnowledgeService', () => { }) cancelMock.mockRejectedValueOnce(new Error('cancel failed')) - const workflowService = ( + const ingestionService = ( service as unknown as { - workflowService: { + ingestionService: { scheduleItem(baseId: string, itemId: string, parentJobId?: string | null): Promise } } - ).workflowService + ).ingestionService - await expect(workflowService.scheduleItem('kb-1', 'file-1')).rejects.toThrow('check enqueue failed') + await expect(ingestionService.scheduleItem('kb-1', 'file-1')).rejects.toThrow('check enqueue failed') expect(cancelMock).toHaveBeenCalledWith('fp-job-1', 'knowledge-file-processing-check-enqueue-failed') }) @@ -1376,18 +1388,18 @@ describe('KnowledgeService', () => { knowledgeBaseGetByIdMock.mockReturnValue(createBase({ fileProcessorId: 'doc2x' })) knowledgeItemGetByIdMock.mockReturnValueOnce(processingFile) - const workflowService = ( + const ingestionService = ( service as unknown as { - workflowService: { + ingestionService: { scheduleItem(baseId: string, itemId: string, parentJobId?: string | null): Promise } } - ).workflowService - await workflowService.scheduleItem('kb-1', 'file-1', 'reindex-job') + ).ingestionService + await ingestionService.scheduleItem('kb-1', 'file-1', 'reindex-job') expect(enqueueMock).toHaveBeenCalledWith( 'knowledge.index-documents', - { baseId: 'kb-1', itemId: 'file-1', parentJobId: 'reindex-job' }, + { baseId: 'kb-1', itemId: 'file-1' }, { idempotencyKey: 'knowledge:kb-1:file-1:index:reindex-job', queue: 'base.kb-1', @@ -1400,23 +1412,29 @@ describe('KnowledgeService', () => { vi.useFakeTimers() vi.setSystemTime(new Date('2026-04-08T00:00:00.000Z')) const service = new KnowledgeService() - const workflowService = ( + const ingestionService = ( service as unknown as { - workflowService: { + ingestionService: { scheduleFileProcessingCheck( baseId: string, itemId: string, fileProcessingJobId: string, - options: { pollRound: number; firstScheduledAt: number; parentJobId: string | null } + options: { + pollRound: number + firstScheduledAt: number + parentJobId: string | null + processedRelativePath: string + } ): Promise } } - ).workflowService + ).ingestionService - await workflowService.scheduleFileProcessingCheck('kb-1', 'file-1', 'fp-job-1', { + await ingestionService.scheduleFileProcessingCheck('kb-1', 'file-1', 'fp-job-1', { pollRound: 1, firstScheduledAt: Date.parse('2026-04-08T00:00:00.000Z'), - parentJobId: 'check-job-0' + parentJobId: 'check-job-0', + processedRelativePath: 'source.md' }) expect(enqueueMock).toHaveBeenCalledWith( @@ -1427,7 +1445,7 @@ describe('KnowledgeService', () => { fileProcessingJobId: 'fp-job-1', pollRound: 1, firstScheduledAt: Date.parse('2026-04-08T00:00:00.000Z'), - parentJobId: 'check-job-0' + processedRelativePath: 'source.md' }, expect.objectContaining({ idempotencyKey: 'knowledge:kb-1:file-1:fp-check:fp-job-1:1', @@ -1440,11 +1458,9 @@ describe('KnowledgeService', () => { it('schedules direct indexing for file items when the extension does not need file processing', async () => { const service = new KnowledgeService() - const createdFile = createFileItem('file-1', 'kb-1', '/docs/source.md') const processingFile = createFileItem('file-1', 'kb-1', '/docs/source.md', 'processing') knowledgeBaseGetByIdMock.mockReturnValue(createBase({ fileProcessorId: 'doc2x' })) - knowledgeItemCreateMock.mockReturnValueOnce(createdFile) - knowledgeItemUpdateStatusMock.mockReturnValueOnce(processingFile) + knowledgeItemCreateActiveMock.mockReturnValueOnce(processingFile) knowledgeItemGetByIdMock.mockReturnValueOnce(processingFile) await service.addItems('kb-1', [{ type: 'file', data: { source: '/docs/source.md', path: '/docs/source.md' } }]) @@ -1452,7 +1468,7 @@ describe('KnowledgeService', () => { expect(fileProcessingStartJobMock).not.toHaveBeenCalled() expect(enqueueMock).toHaveBeenCalledWith( 'knowledge.index-documents', - { baseId: 'kb-1', itemId: 'file-1', parentJobId: null }, + { baseId: 'kb-1', itemId: 'file-1' }, { idempotencyKey: 'knowledge:kb-1:file-1:index', queue: 'base.kb-1', @@ -1463,11 +1479,9 @@ describe('KnowledgeService', () => { it('schedules direct indexing for document files when the base has no file processor', async () => { const service = new KnowledgeService() - const createdFile = createFileItem('file-1', 'kb-1', '/docs/source.pdf') const processingFile = createFileItem('file-1', 'kb-1', '/docs/source.pdf', 'processing') knowledgeBaseGetByIdMock.mockReturnValue(createBase({ fileProcessorId: null })) - knowledgeItemCreateMock.mockReturnValueOnce(createdFile) - knowledgeItemUpdateStatusMock.mockReturnValueOnce(processingFile) + knowledgeItemCreateActiveMock.mockReturnValueOnce(processingFile) knowledgeItemGetByIdMock.mockReturnValueOnce(processingFile) await service.addItems('kb-1', [{ type: 'file', data: { source: '/docs/source.pdf', path: '/docs/source.pdf' } }]) @@ -1475,7 +1489,7 @@ describe('KnowledgeService', () => { expect(fileProcessingStartJobMock).not.toHaveBeenCalled() expect(enqueueMock).toHaveBeenCalledWith( 'knowledge.index-documents', - { baseId: 'kb-1', itemId: 'file-1', parentJobId: null }, + { baseId: 'kb-1', itemId: 'file-1' }, { idempotencyKey: 'knowledge:kb-1:file-1:index', queue: 'base.kb-1', @@ -1499,20 +1513,18 @@ describe('KnowledgeService', () => { ]) ).rejects.toThrow('enqueue failed') - expect(knowledgeItemUpdateStatusMock).toHaveBeenCalledWith('note-1', 'processing') - expect(knowledgeItemUpdateStatusMock).toHaveBeenCalledWith('note-2', 'processing') - expect(knowledgeItemUpdateStatusMock).toHaveBeenCalledWith('note-2', 'failed', { + expect(knowledgeItemSetSubtreeStatusMock).toHaveBeenCalledWith('kb-1', ['note-2'], 'failed', { error: 'Failed to schedule knowledge item job: enqueue failed' }) - expect(knowledgeItemUpdateStatusMock).not.toHaveBeenCalledWith('note-1', 'failed', expect.anything()) + expect(knowledgeItemSetSubtreeStatusMock).not.toHaveBeenCalledWith('kb-1', ['note-1'], 'failed', expect.anything()) }) - it('rolls back every created addItems row when a status update fails', async () => { + it('rolls back already-created addItems rows when a later create fails mid-batch', async () => { const service = new KnowledgeService() - knowledgeItemUpdateStatusMock + knowledgeItemCreateActiveMock .mockReturnValueOnce(createNoteItem('note-1', 'kb-1', null, 'processing')) .mockImplementationOnce(() => { - throw new Error('status failed') + throw new Error('create failed') }) await expect( @@ -1520,16 +1532,17 @@ describe('KnowledgeService', () => { { type: 'note', data: { source: 'note-1', content: 'hello 1' } }, { type: 'note', data: { source: 'note-2', content: 'hello 2' } } ]) - ).rejects.toThrow('status failed') + ).rejects.toThrow('create failed') + // note-2's row was never created, so there is nothing to roll back for it. expect(knowledgeItemDeleteMock).toHaveBeenCalledWith('note-1') - expect(knowledgeItemDeleteMock).toHaveBeenCalledWith('note-2') + expect(knowledgeItemDeleteMock).not.toHaveBeenCalledWith('note-2') expect(enqueueMock).not.toHaveBeenCalled() }) it('runs best-effort copied-file cleanup and preserves the original addItems error', async () => { const service = new KnowledgeService() - knowledgeItemCreateMock.mockImplementationOnce(() => { + knowledgeItemCreateActiveMock.mockImplementationOnce(() => { throw new Error('create failed') }) @@ -1542,18 +1555,17 @@ describe('KnowledgeService', () => { expect(deleteKnowledgeItemFilesBestEffortMock).toHaveBeenCalledTimes(1) }) - it('keeps items deleting when delete cleanup enqueue fails', async () => { + it('runs delete marking and enqueue inside one transaction', async () => { const service = new KnowledgeService() knowledgeItemGetByIdMock.mockReturnValue(createNoteItem('note-1')) - enqueueMock.mockImplementationOnce(() => { + enqueueTxMock.mockImplementationOnce(() => { throw new Error('enqueue failed') }) await expect(service.deleteItems('kb-1', ['note-1'])).rejects.toThrow('enqueue failed') - expect(knowledgeItemSetSubtreeStatusMock).toHaveBeenCalledWith('kb-1', ['note-1'], 'deleting') - expect(knowledgeItemSetSubtreeStatusMock).not.toHaveBeenCalledWith('kb-1', ['note-1'], 'failed', expect.anything()) - expect(enqueueMock).toHaveBeenCalledTimes(1) + expect(knowledgeItemSetSubtreeStatusTxMock).toHaveBeenCalledWith(expect.anything(), 'kb-1', ['note-1'], 'deleting') + expect(enqueueTxMock).toHaveBeenCalledTimes(1) }) it('collapses nested delete and reindex inputs to top-level roots', async () => { @@ -1567,14 +1579,15 @@ describe('KnowledgeService', () => { expect(knowledgeItemGetOutermostSelectedItemIdsMock).toHaveBeenNthCalledWith(1, 'kb-1', ['dir-1', 'note-1']) expect(knowledgeItemGetOutermostSelectedItemIdsMock).toHaveBeenNthCalledWith(2, 'kb-1', ['dir-1', 'note-1']) - expect(enqueueMock).toHaveBeenNthCalledWith( + expect(enqueueTxMock).toHaveBeenNthCalledWith( 1, + expect.anything(), 'knowledge.delete-subtree', { baseId: 'kb-1', rootItemIds: ['dir-1'] }, expect.any(Object) ) expect(enqueueMock).toHaveBeenNthCalledWith( - 2, + 1, 'knowledge.reindex-subtree', { baseId: 'kb-1', rootItemIds: ['dir-1'] }, expect.any(Object) @@ -2106,11 +2119,11 @@ describe('KnowledgeService', () => { it('omits conceptId for a leaf that is not completed (not readable yet)', async () => { const service = new KnowledgeService() - knowledgeItemGetItemsByBaseIdMock.mockReturnValue([createFileItem('f1', 'kb-1', '/a.pdf', 'idle')]) + knowledgeItemGetItemsByBaseIdMock.mockReturnValue([createFileItem('f1', 'kb-1', '/a.pdf', 'processing')]) const tree = service.getOrganizationTree('kb-1') - expect(tree.nodes[0]).toMatchObject({ depth: 0, itemType: 'file', status: 'idle' }) + expect(tree.nodes[0]).toMatchObject({ depth: 0, itemType: 'file', status: 'processing' }) expect(tree.nodes[0].conceptId).toBeUndefined() }) @@ -2166,7 +2179,12 @@ describe('KnowledgeService', () => { expect(getMaterialByRelativePathMock).toHaveBeenCalledWith(CONCEPT_ID) expect(knowledgeItemGetOutermostSelectedItemIdsMock).toHaveBeenCalledWith('kb-1', [NOTE_ITEM_ID]) - expect(knowledgeItemSetSubtreeStatusMock).toHaveBeenCalledWith('kb-1', [NOTE_ITEM_ID], 'deleting') + expect(knowledgeItemSetSubtreeStatusTxMock).toHaveBeenCalledWith( + expect.anything(), + 'kb-1', + [NOTE_ITEM_ID], + 'deleting' + ) expect(result).toEqual({ applied: [CONCEPT_ID], notFound: [] }) }) @@ -2186,7 +2204,7 @@ describe('KnowledgeService', () => { const result = await service.deleteConcepts('kb-1', [CONCEPT_ID]) expect(result).toEqual({ applied: [], notFound: [CONCEPT_ID] }) - expect(knowledgeItemSetSubtreeStatusMock).not.toHaveBeenCalled() + expect(knowledgeItemSetSubtreeStatusTxMock).not.toHaveBeenCalled() }) it('collapses duplicate Concept IDs to a single resolution', async () => { @@ -2205,7 +2223,7 @@ describe('KnowledgeService', () => { const result = await service.deleteConcepts('kb-1', ['docs/missing.md']) expect(result).toEqual({ applied: [], notFound: ['docs/missing.md'] }) - expect(enqueueMock).not.toHaveBeenCalled() + expect(enqueueTxMock).not.toHaveBeenCalled() }) }) @@ -2333,7 +2351,7 @@ describe('KnowledgeService', () => { it('translates a search failure into a defined error when the store was closed mid-flight', async () => { const service = new KnowledgeService() - getIndexStoreMock.mockResolvedValueOnce({ + getIndexStoreMock.mockReturnValueOnce({ search: vi.fn().mockRejectedValue(new Error('Knowledge index store driver is closed')), listMaterialUnits: listMaterialUnitsMock, isClosed: () => true @@ -2348,7 +2366,7 @@ describe('KnowledgeService', () => { it('rethrows a genuine search failure unchanged when the store is still open', async () => { const service = new KnowledgeService() const queryError = new Error('disk I/O error') - getIndexStoreMock.mockResolvedValueOnce({ + getIndexStoreMock.mockReturnValueOnce({ search: vi.fn().mockRejectedValue(queryError), listMaterialUnits: listMaterialUnitsMock, isClosed: () => false @@ -2360,7 +2378,7 @@ describe('KnowledgeService', () => { it('lists chunks after checking item ownership', async () => { const service = new KnowledgeService() knowledgeItemGetByIdMock.mockReturnValue(createNoteItem('note-1', 'kb-1', null, 'completed')) - listMaterialUnitsMock.mockResolvedValueOnce([ + listMaterialUnitsMock.mockReturnValueOnce([ { unitId: 'chunk-1', materialId: 'note-1', @@ -2385,7 +2403,7 @@ describe('KnowledgeService', () => { knowledgeItemGetSubtreeItemsMock .mockReturnValueOnce([createNoteItem('note-1', 'kb-1', 'dir-1', 'completed')]) .mockReturnValueOnce([createNoteItem('note-1', 'kb-1', 'dir-1', 'completed')]) - listMaterialUnitsMock.mockResolvedValueOnce([ + listMaterialUnitsMock.mockReturnValueOnce([ { unitId: 'chunk-1', materialId: 'note-1', @@ -2417,7 +2435,7 @@ describe('KnowledgeService', () => { expect(listMaterialUnitsMock).not.toHaveBeenCalled() }) - it.each(['idle', 'processing', 'reading', 'embedding', 'failed', 'deleting'] as const)( + it.each(['processing', 'reading', 'embedding', 'failed', 'deleting'] as const)( 'rejects chunk operations for %s leaf items', async (status) => { const service = new KnowledgeService() @@ -2435,9 +2453,11 @@ describe('KnowledgeService', () => { it('translates a listItemChunks failure into a defined error when the store was closed mid-flight', async () => { const service = new KnowledgeService() knowledgeItemGetByIdMock.mockReturnValue(createNoteItem('note-1', 'kb-1', null, 'completed')) - getIndexStoreMock.mockResolvedValueOnce({ + getIndexStoreMock.mockReturnValueOnce({ search: storeSearchMock, - listMaterialUnits: vi.fn().mockRejectedValue(new Error('Knowledge index store driver is closed')), + listMaterialUnits: vi.fn().mockImplementation(() => { + throw new Error('Knowledge index store driver is closed') + }), isClosed: () => true }) diff --git a/src/main/features/knowledge/utils/__tests__/items.test.ts b/src/main/features/knowledge/__tests__/items.test.ts similarity index 62% rename from src/main/features/knowledge/utils/__tests__/items.test.ts rename to src/main/features/knowledge/__tests__/items.test.ts index bd6b1b26872..be806a31fc4 100644 --- a/src/main/features/knowledge/utils/__tests__/items.test.ts +++ b/src/main/features/knowledge/__tests__/items.test.ts @@ -1,15 +1,16 @@ import type { KnowledgeItem, KnowledgeItemOf } from '@shared/data/types/knowledge' import { beforeEach, describe, expect, it, vi } from 'vitest' -import type * as PathStorage from '../storage/pathStorage' +import type { MaterialFieldSource } from '../items' +import type * as PathStorage from '../pathStorage' const { probeKnowledgeFileMock, probeKnowledgeSourcePathMock } = vi.hoisted(() => ({ probeKnowledgeFileMock: vi.fn(), probeKnowledgeSourcePathMock: vi.fn() })) -vi.mock('../storage/pathStorage', async () => { - const actual = await vi.importActual('../storage/pathStorage') +vi.mock('../pathStorage', async () => { + const actual = await vi.importActual('../pathStorage') return { ...actual, probeKnowledgeFile: probeKnowledgeFileMock, @@ -17,16 +18,15 @@ vi.mock('../storage/pathStorage', async () => { } }) -const { canKnowledgeItemRebuildSource, classifyKnowledgeItemSource, isIndexableKnowledgeItem } = await import( - '../items' -) +const { canKnowledgeItemRebuildSource, classifyKnowledgeItemSource, isIndexableKnowledgeItem, toMaterialRelativePath } = + await import('../items') function createItem(type: KnowledgeItem['type']): KnowledgeItem { const base = { id: `${type}-1`, baseId: 'kb-1', groupId: null, - status: 'idle', + status: 'processing', error: null, createdAt: '2026-04-08T00:00:00.000Z', updatedAt: '2026-04-08T00:00:00.000Z' @@ -115,3 +115,59 @@ describe('canKnowledgeItemRebuildSource', () => { await expect(canKnowledgeItemRebuildSource('kb-1', createItem('directory'))).resolves.toBe(false) }) }) + +describe('toMaterialRelativePath', () => { + it('uses a file’s stored relativePath when there is no processed artifact', () => { + const file: MaterialFieldSource = { + id: 'file-1', + type: 'file', + data: { source: '/docs/report.pdf', relativePath: 'report.pdf' } + } + expect(toMaterialRelativePath(file)).toBe('report.pdf') + }) + + it('prefers a file’s processed-artifact path (indexedRelativePath) over the source path', () => { + const file: MaterialFieldSource = { + id: 'file-2', + type: 'file', + data: { source: '/docs/report.pdf', relativePath: 'report.pdf', indexedRelativePath: 'report.md' } + } + expect(toMaterialRelativePath(file)).toBe('report.md') + }) + + it('uses a url’s captured snapshot path once it has one (the real raw/ file, matching the migrator)', () => { + const url: MaterialFieldSource = { + id: 'url-1', + type: 'url', + data: { source: 'https://example.com', url: 'https://example.com', relativePath: 'example-page.md' } + } + expect(toMaterialRelativePath(url)).toBe('example-page.md') + }) + + it('uses a note’s captured snapshot path once it has one (the real raw/ file, matching the migrator)', () => { + const note: MaterialFieldSource = { + id: 'note-1', + type: 'note', + data: { source: 'My note', content: 'hello', relativePath: 'My note.md' } + } + expect(toMaterialRelativePath(note)).toBe('My note.md') + }) + + it('throws for a url that has not been captured yet — a snapshot is always materialized first', () => { + const url: MaterialFieldSource = { + id: 'url-2', + type: 'url', + data: { source: 'https://example.com', url: 'https://example.com' } + } + expect(() => toMaterialRelativePath(url)).toThrow('has no captured snapshot relativePath') + }) + + it('throws for a note that has not been captured yet — a snapshot is always materialized first', () => { + const note: MaterialFieldSource = { + id: 'note-2', + type: 'note', + data: { source: 'My note', content: 'hello' } + } + expect(() => toMaterialRelativePath(note)).toThrow('has no captured snapshot relativePath') + }) +}) diff --git a/src/main/features/knowledge/utils/storage/__tests__/pathStorage.test.ts b/src/main/features/knowledge/__tests__/pathStorage.test.ts similarity index 95% rename from src/main/features/knowledge/utils/storage/__tests__/pathStorage.test.ts rename to src/main/features/knowledge/__tests__/pathStorage.test.ts index a255e288931..cede79363b6 100644 --- a/src/main/features/knowledge/utils/storage/__tests__/pathStorage.test.ts +++ b/src/main/features/knowledge/__tests__/pathStorage.test.ts @@ -1,7 +1,7 @@ // Path-safety tests for the knowledge "store by relative path" core. The reject -// branches of `assertSafeKnowledgeRelativePath` / `isPathInsideBase` are the -// security boundary for every copied source file, so they are exercised here -// through the exported helpers (the guards themselves are private). +// branches of `assertSafeKnowledgeRelativePath` are the security boundary for +// every copied source file, so they are exercised here through the exported +// helpers (the guard itself is private). import path from 'node:path' import { beforeEach, describe, expect, it, vi } from 'vitest' @@ -43,11 +43,9 @@ vi.mock('@main/utils/file/fs', () => ({ })) const { - getKnowledgeBaseDir, getKnowledgeMaterialDir, getKnowledgeBaseFilePath, getKnowledgeSourceRelativePath, - toKnowledgeRelativePath, getProcessedMarkdownRelativePath, reserveImportedFileRelativePath, needsProcessedArtifactReservation, @@ -59,7 +57,6 @@ const { } = await import('../pathStorage') const BASE_ID = 'kb-1' -const BASE_DIR = getKnowledgeBaseDir(BASE_ID) // Material bytes resolve under the base's `raw/` material root, not the base dir itself. const MATERIAL_DIR = getKnowledgeMaterialDir(BASE_ID) @@ -99,19 +96,6 @@ describe('pathStorage relative-path safety', () => { }) }) - describe('toKnowledgeRelativePath', () => { - it('returns a POSIX relative path for a path inside the base', () => { - expect(toKnowledgeRelativePath(BASE_ID, path.join(MATERIAL_DIR, 'a', 'b.md'))).toBe('a/b.md') - }) - - it.each([ - ['an unrelated absolute path', '/etc/passwd'], - ['a sibling base', path.join(path.dirname(BASE_DIR), 'kb-2', 'x.md')] - ])('rejects %s', (_label, absolutePath) => { - expect(() => toKnowledgeRelativePath(BASE_ID, absolutePath)).toThrow() - }) - }) - describe('getProcessedMarkdownRelativePath', () => { it('swaps the extension to .md while keeping directories', () => { expect(getProcessedMarkdownRelativePath('sub/report.pdf')).toBe('sub/report.md') diff --git a/src/main/features/knowledge/base/KnowledgeBaseAdminService.ts b/src/main/features/knowledge/base/KnowledgeBaseAdminService.ts new file mode 100644 index 00000000000..339a84edf14 --- /dev/null +++ b/src/main/features/knowledge/base/KnowledgeBaseAdminService.ts @@ -0,0 +1,249 @@ +import { application } from '@application' +import { knowledgeBaseService } from '@data/services/KnowledgeBaseService' +import { knowledgeItemService } from '@data/services/KnowledgeItemService' +import { loggerService } from '@logger' +import type { KeyedMutex } from '@main/core/concurrency/KeyedMutex' +import { DataApiErrorFactory } from '@shared/data/api' +import { KNOWLEDGE_BASES_MAX_LIMIT } from '@shared/data/api/schemas/knowledges' +import { + type CreateKnowledgeBaseDto, + type KnowledgeAddItemInput, + KnowledgeAddItemInputSchema, + type KnowledgeBase, + type KnowledgeItem, + type RestoreKnowledgeBaseDto, + type RestoreKnowledgeBaseResult +} from '@shared/data/types/knowledge' + +import type { KnowledgeIngestionService } from '../ingestion/KnowledgeIngestionService' +import { classifyKnowledgeItemSource } from '../items' +import { getKnowledgeBaseFilePath } from '../pathStorage' +import { cancelActiveKnowledgeJobs } from '../tasks/utils/cancel' + +const logger = loggerService.withContext('Knowledge:BaseAdmin') + +/** Knowledge base lifecycle: create (with rollback), delete, restore, and list — everything about the base row + its on-disk artifacts, not about items. */ +export class KnowledgeBaseAdminService { + constructor( + private readonly knowledgeLockManager: KeyedMutex, + private readonly ingestionService: KnowledgeIngestionService + ) {} + + async createBase(dto: CreateKnowledgeBaseDto): Promise { + const base = knowledgeBaseService.create(dto) + const vectorStoreService = application.get('KnowledgeVectorStoreService') + + try { + vectorStoreService.getIndexStore(base) + } catch (error) { + await this.rollbackFailedBaseCreation(base.id) + throw error + } + + return base + } + + /** + * Undo a half-created base after its index store failed to open: remove the + * orphaned `.cherry/` directory `getIndexStore` left on disk and drop the DB + * row. Both steps are best-effort and logged — a cleanup failure must never + * mask the original open error the caller needs to see. + */ + private async rollbackFailedBaseCreation(baseId: string): Promise { + const vectorStoreService = application.get('KnowledgeVectorStoreService') + try { + await vectorStoreService.deleteStore(baseId) + } catch (cleanupError) { + logger.warn('Failed to remove index store dir during createBase rollback', cleanupError as Error, { baseId }) + } + try { + knowledgeBaseService.delete(baseId) + } catch (cleanupError) { + logger.warn('Failed to delete knowledge base row during createBase rollback', cleanupError as Error, { baseId }) + } + } + + async deleteBase(baseId: string): Promise { + await cancelActiveKnowledgeJobs(baseId, 'delete-base', { onCancelTimeout: 'proceed' }) + + await this.knowledgeLockManager.runExclusive(baseId, async () => { + try { + const vectorStoreService = application.get('KnowledgeVectorStoreService') + await vectorStoreService.deleteStore(baseId) + } catch (error) { + const normalizedError = error instanceof Error ? error : new Error(String(error)) + logger.error('Failed to delete knowledge base vector artifacts', normalizedError, { baseId }) + throw error + } + + try { + knowledgeBaseService.delete(baseId) + } catch (error) { + const normalizedError = error instanceof Error ? error : new Error(String(error)) + logger.error('Failed to delete knowledge base SQLite row after artifact cleanup', normalizedError, { + baseId + }) + throw DataApiErrorFactory.invalidOperation( + 'deleteBase', + `Vector artifacts were deleted, but SQLite knowledge base cleanup failed: ${normalizedError.message}` + ) + } + }) + } + + async restoreBase(dto: RestoreKnowledgeBaseDto): Promise { + const sourceBase = knowledgeBaseService.getById(dto.sourceBaseId) + + // Restore always supplies an embedding model (RestoreKnowledgeBaseSchema requires + // it). A BM25-only source's searchMode is pinned to 'bm25' by the no-model + // invariant, so carrying it over would leave the restored base's semantic search + // silently disabled despite it now paying for the full embedding backfill. Drop + // it (and the always-null hybridAlpha) so create() applies its own hybrid + // default; a source that already had a model keeps its existing preference. + const gainsEmbeddings = sourceBase.embeddingModelId === null + + const createDto: CreateKnowledgeBaseDto = { + name: dto.name?.trim() ?? sourceBase.name, + dimensions: dto.dimensions, + embeddingModelId: dto.embeddingModelId, + rerankModelId: sourceBase.rerankModelId, + fileProcessorId: sourceBase.fileProcessorId, + chunkSize: sourceBase.chunkSize, + chunkOverlap: sourceBase.chunkOverlap, + threshold: sourceBase.threshold, + documentCount: sourceBase.documentCount, + searchMode: gainsEmbeddings ? undefined : sourceBase.searchMode, + hybridAlpha: gainsEmbeddings ? undefined : sourceBase.hybridAlpha, + groupId: sourceBase.groupId ?? undefined + } + + const rootItems = knowledgeItemService.getRootItemsByBaseId(sourceBase.id) + + // Partial restore: probe each root's source and skip the ones whose source is genuinely gone, so + // a single missing source no longer aborts the entire restore. This is the common case for a + // failed base — a v1-migrated directory child has a virtual path with no raw/ file, and a file + // whose original was deleted has no material to copy; addItems would throw on the first such + // item and roll back the whole batch. An 'unverifiable' source (transient/permission error) is + // kept, not skipped — like reindex, we never drop a source we could not confirm is gone. + const restorableRootItems: KnowledgeItem[] = [] + for (const item of rootItems) { + if ((await classifyKnowledgeItemSource(sourceBase.id, item)) === 'missing') { + logger.warn('Skipping knowledge item with a missing source during restore', { + sourceBaseId: sourceBase.id, + itemId: item.id, + type: item.type + }) + continue + } + restorableRootItems.push(item) + } + const skippedMissingSourceCount = rootItems.length - restorableRootItems.length + if (skippedMissingSourceCount > 0) { + logger.info('Restore skipped knowledge items whose source no longer exists', { + sourceBaseId: sourceBase.id, + skippedMissingSourceCount, + restorableCount: restorableRootItems.length + }) + } + + const inputs = restorableRootItems.map((item) => this.toRestoreRuntimeInput(sourceBase.id, item)) + const restoredBase = await this.createBase(createDto) + try { + await this.ingestionService.addItems(restoredBase.id, inputs) + } catch (error) { + try { + await this.deleteBase(restoredBase.id) + } catch (cleanupError) { + const cleanupMessage = cleanupError instanceof Error ? cleanupError.message : String(cleanupError) + logger.error( + 'Failed to delete restored knowledge base after item restoration failed', + cleanupError instanceof Error ? cleanupError : new Error(cleanupMessage), + { + sourceBaseId: sourceBase.id, + restoredBaseId: restoredBase.id + } + ) + throw DataApiErrorFactory.invalidOperation( + 'restoreBase', + `Failed to restore knowledge items: ${ + error instanceof Error ? error.message : String(error) + }. Restored knowledge base '${restoredBase.id}' could not be cleaned up automatically: ${cleanupMessage}. Please delete it manually.` + ) + } + throw DataApiErrorFactory.invalidOperation( + 'restoreBase', + `Failed to restore knowledge items: ${error instanceof Error ? error.message : String(error)}` + ) + } + + return { base: restoredBase, skippedMissingSourceCount } + } + + listBases(): KnowledgeBase[] { + const { items } = knowledgeBaseService.list({ page: 1, limit: KNOWLEDGE_BASES_MAX_LIMIT }) + return items + } + + /** Whether the user has any knowledge base at all — a cheap count (not a full list) for tool-availability gating. */ + hasAnyBase(): boolean { + const { total } = knowledgeBaseService.list({ page: 1, limit: 1 }) + return total > 0 + } + + private toRestoreRuntimeInput(sourceBaseId: string, item: KnowledgeItem): KnowledgeAddItemInput { + try { + if (item.type === 'file') { + return KnowledgeAddItemInputSchema.parse({ + type: 'file', + data: { + source: item.data.source, + path: getKnowledgeBaseFilePath(sourceBaseId, item.data.relativePath), + // Carry the processed artifact across so the new base indexes from it + // instead of re-running the (slow, paid) file processor. + ...(item.data.indexedRelativePath + ? { indexedPath: getKnowledgeBaseFilePath(sourceBaseId, item.data.indexedRelativePath) } + : {}) + } + }) + } + + if (item.type === 'url') { + return KnowledgeAddItemInputSchema.parse({ + type: 'url', + data: { + source: item.data.source, + url: item.data.url, + // Carry the captured snapshot across so the restored URL indexes offline + // instead of re-fetching the live page (which may have changed or died). + // If the source never captured one, omit it and let the first index capture. + ...(item.data.relativePath + ? { snapshotPath: getKnowledgeBaseFilePath(sourceBaseId, item.data.relativePath) } + : {}) + } + }) + } + + if (item.type === 'note') { + return KnowledgeAddItemInputSchema.parse({ + type: 'note', + // The snapshot relativePath is intentionally dropped: the content is the + // source of truth and re-capturing it into the new base on first index is + // free and deterministic, so there is no snapshot file to carry across. + data: { source: item.data.source, content: item.data.content } + }) + } + + return KnowledgeAddItemInputSchema.parse({ + type: item.type, + data: item.data + }) + } catch (error) { + throw DataApiErrorFactory.invalidOperation( + 'restoreBase', + `Cannot restore knowledge item '${item.id}' (${item.type}): ${ + error instanceof Error ? error.message : String(error) + }` + ) + } + } +} diff --git a/src/main/features/knowledge/base/baseGuards.ts b/src/main/features/knowledge/base/baseGuards.ts new file mode 100644 index 00000000000..deed694056b --- /dev/null +++ b/src/main/features/knowledge/base/baseGuards.ts @@ -0,0 +1,22 @@ +import { knowledgeBaseService } from '@data/services/KnowledgeBaseService' +import { DataApiErrorFactory } from '@shared/data/api' +import type { KnowledgeBase } from '@shared/data/types/knowledge' + +/** + * A base in `failed` state must be restored before any runtime operation (search, add, reindex, + * deep-read). Returns the fetched base so synchronous callers need not re-fetch it. + */ +export function assertBaseCanRunRuntimeOperation(baseId: string, operation: string): KnowledgeBase { + const base = knowledgeBaseService.getById(baseId) + + if (base.status !== 'failed') { + return base + } + + throw DataApiErrorFactory.validation( + { + base: [`Knowledge base '${baseId}' is in failed state; restore it before ${operation}.`] + }, + `Cannot ${operation} failed knowledge base` + ) +} diff --git a/src/main/features/knowledge/index.ts b/src/main/features/knowledge/index.ts index 29435836b83..aa6e0589a36 100644 --- a/src/main/features/knowledge/index.ts +++ b/src/main/features/knowledge/index.ts @@ -1,3 +1 @@ -export { KnowledgeLockManager } from './KnowledgeLockManager' export { KnowledgeService } from './KnowledgeService' -export { KnowledgeWorkflowService } from './KnowledgeWorkflowService' diff --git a/src/main/features/knowledge/KnowledgeWorkflowService.ts b/src/main/features/knowledge/ingestion/KnowledgeIngestionService.ts similarity index 63% rename from src/main/features/knowledge/KnowledgeWorkflowService.ts rename to src/main/features/knowledge/ingestion/KnowledgeIngestionService.ts index 3f13b22ff4e..dc2e9311f55 100644 --- a/src/main/features/knowledge/KnowledgeWorkflowService.ts +++ b/src/main/features/knowledge/ingestion/KnowledgeIngestionService.ts @@ -1,24 +1,41 @@ -import './tasks/jobTypes' +import '../tasks/jobTypes' import { application } from '@application' import { knowledgeBaseService } from '@data/services/KnowledgeBaseService' import { knowledgeItemService } from '@data/services/KnowledgeItemService' import { loggerService } from '@logger' +import type { KeyedMutex } from '@main/core/concurrency/KeyedMutex' import { getFileExt } from '@main/utils/file' +import { DataApiErrorFactory } from '@shared/data/api' import { FileProcessorIdSchema } from '@shared/data/presets/fileProcessing' import { type CreateKnowledgeItemDto, DEFAULT_KNOWLEDGE_ADD_CONFLICT_STRATEGY, + KNOWLEDGE_ITEM_ERROR_INDEXING_INTERRUPTED, type KnowledgeAddConflictStrategy, type KnowledgeAddItemInput, type KnowledgeAddItemsResult, type KnowledgeBase, - type KnowledgeItem + type KnowledgeItem, + type KnowledgeItemStatus } from '@shared/data/types/knowledge' import { knowledgeSupportedFileExts } from '@shared/utils/file' -import type { KnowledgeLockManager } from './KnowledgeLockManager' -import { cancelJobOrThrow } from './tasks/utils/cancel' +import { assertBaseCanRunRuntimeOperation } from '../base/baseGuards' +import { classifyKnowledgeItemSource } from '../items' +import { + assertKnowledgeFileTargetAvailable, + collectKnowledgeReservedRelativePaths, + copyFileIntoKnowledgeBaseAt, + deleteKnowledgeItemFilesBestEffort, + getKnowledgeBaseFilePath, + getKnowledgeSourceRelativePath, + getProcessedMarkdownRelativePath, + needsProcessedArtifactReservation, + reserveImportedFileRelativePath +} from '../pathStorage' +import { planKnowledgeItemSource } from '../pipeline/sources/sourcePlanning' +import { cancelActiveKnowledgeJobs, cancelJobOrThrow } from '../tasks/utils/cancel' import { type KnowledgeBaseId, knowledgeDeleteSubtreeIdempotencyKey, @@ -31,43 +48,49 @@ import { toKnowledgeBaseId, toKnowledgeItemId, toKnowledgeItemIds -} from './types' -import { resolveKnowledgeAddConflicts } from './utils/addConflicts' -import { markUnscheduledKnowledgeItemsFailed } from './utils/cleanup/statusCleanup' -import { cancelActiveKnowledgeSubtreeJobs, purgeKnowledgeSubtreeWithinLock } from './utils/cleanup/subtreePurge' -import { isContainerKnowledgeItem } from './utils/items' -import { planKnowledgeItemSource } from './utils/sources/sourcePlanning' -import { - assertKnowledgeFileTargetAvailable, - collectKnowledgeReservedRelativePaths, - copyFileIntoKnowledgeBaseAt, - deleteKnowledgeItemFilesBestEffort, - getKnowledgeBaseFilePath, - getKnowledgeSourceRelativePath, - getProcessedMarkdownRelativePath, - needsProcessedArtifactReservation, - reserveImportedFileRelativePath -} from './utils/storage/pathStorage' +} from '../types' +import { resolveKnowledgeAddConflicts } from './addConflicts' +import { markUnscheduledKnowledgeItemsFailed } from './statusCleanup' +import { purgeKnowledgeSubtreeWithinLock } from './subtreePurge' -const logger = loggerService.withContext('Knowledge:WorkflowService') +const logger = loggerService.withContext('Knowledge:IngestionService') // Keep poll jobs delayed enough to avoid hot-looping while remote processors are still working. const FILE_PROCESSING_CHECK_DELAY_MS = 5_000 const KNOWLEDGE_SUPPORTED_FILE_EXT_SET = new Set(knowledgeSupportedFileExts) +const REINDEX_ALLOWED_STATUSES = new Set(['completed', 'failed']) +const DELETE_RECOVERY_ROOT_CHUNK_SIZE = 500 + +/** + * The workflow re-entry seam job handlers call back into (workflow-architecture.md): expand a + * container, index a leaf, or poll a file-processing job. Handlers depend on exactly this surface, + * not the full `KnowledgeIngestionService`. + */ +export interface KnowledgeItemScheduler { + scheduleItem(baseId: KnowledgeBaseId, itemId: KnowledgeItemId, parentJobId?: string | null): Promise + scheduleIndexing(baseId: KnowledgeBaseId, itemId: KnowledgeItemId, parentJobId?: string | null): Promise + scheduleFileProcessingCheck( + baseId: KnowledgeBaseId, + itemId: KnowledgeItemId, + fileProcessingJobId: string, + options: { pollRound: number; firstScheduledAt: number; parentJobId: string | null; processedRelativePath: string } + ): Promise +} -export class KnowledgeWorkflowService { - constructor(private readonly knowledgeLockManager: KnowledgeLockManager) {} +/** Write-side orchestration: admission checks, item creation, conflict handling, and job enqueueing for the add/delete/reindex flows. */ +export class KnowledgeIngestionService implements KnowledgeItemScheduler { + constructor(private readonly knowledgeLockManager: KeyedMutex) {} async addItems( baseId: string, inputs: KnowledgeAddItemInput[], conflictStrategy: KnowledgeAddConflictStrategy = DEFAULT_KNOWLEDGE_ADD_CONFLICT_STRATEGY ): Promise { + const base = assertBaseCanRunRuntimeOperation(baseId, 'addItems') + if (inputs.length === 0) { return { status: 'added' } } - const base = knowledgeBaseService.getById(baseId) - // rename (the default, and every internal caller — restore/migrator): keep all, // auto-rename on collision. detect/replace first resolve same-name conflicts // against the existing root items and earlier items in the same batch. @@ -88,11 +111,10 @@ export class KnowledgeWorkflowService { // holding it would deadlock. itemsToAdd = resolution.keptInputs if (resolution.conflictingExistingRootIds.length > 0) { - await cancelActiveKnowledgeSubtreeJobs( - base.id, - resolution.conflictingExistingRootIds, - 'knowledge-add-replace' - ) + await cancelActiveKnowledgeJobs(base.id, 'knowledge-add-replace', { + rootItemIds: resolution.conflictingExistingRootIds, + onCancelTimeout: 'throw' + }) } } } @@ -100,7 +122,7 @@ export class KnowledgeWorkflowService { const acceptedItems: KnowledgeItem[] = [] const copiedFileItems: Array> = [] - await this.knowledgeLockManager.withBaseMutationLock(base.id, async () => { + await this.knowledgeLockManager.runExclusive(base.id, async () => { try { if (conflictStrategy === 'replace') { // Purge the conflicting existing items synchronously inside the lock and @@ -123,13 +145,8 @@ export class KnowledgeWorkflowService { if (createInput.type === 'file' || (createInput.type === 'url' && createInput.data.relativePath)) { copiedFileItems.push(createInput) } - const createdItem = knowledgeItemService.create(base.id, createInput) + const createdItem = knowledgeItemService.createActive(base.id, createInput) acceptedItems.push(createdItem) - const activeItem = knowledgeItemService.updateStatus( - createdItem.id, - isContainerKnowledgeItem(createdItem) ? 'preparing' : 'processing' - ) - acceptedItems[acceptedItems.length - 1] = activeItem } } catch (error) { this.rollbackAcceptedItems(base.id, acceptedItems, error) @@ -177,36 +194,40 @@ export class KnowledgeWorkflowService { } async deleteItems(baseId: string, itemIds: string[]): Promise { + const rootItemIds = knowledgeItemService.getOutermostSelectedItemIds(baseId, itemIds) + if (rootItemIds.length === 0) { + return + } + knowledgeBaseService.getById(baseId) - const rootItemIds = [...new Set(itemIds)] const knowledgeBaseId = toKnowledgeBaseId(baseId) const knowledgeRootItemIds = toKnowledgeItemIds(rootItemIds) - const markedIds = await this.knowledgeLockManager.withBaseMutationLock(baseId, () => - knowledgeItemService.setSubtreeStatus(baseId, rootItemIds, 'deleting') - ) - try { - const jobManager = application.get('JobManager') - jobManager.enqueue( - 'knowledge.delete-subtree', - { baseId, rootItemIds }, - { - idempotencyKey: knowledgeDeleteSubtreeIdempotencyKey(knowledgeBaseId, knowledgeRootItemIds), - queue: knowledgeQueueName(knowledgeBaseId) - } - ) - } catch (error) { - logger.error('Failed to enqueue knowledge delete cleanup after marking items deleting', error as Error, { - baseId, - rootItemIds, - markedIds + await this.knowledgeLockManager.runExclusive(baseId, () => + application.get('DbService').withWriteTx((tx) => { + knowledgeItemService.setSubtreeStatusTx(tx, baseId, rootItemIds, 'deleting') + application.get('JobManager').enqueueTx( + tx, + 'knowledge.delete-subtree', + { baseId, rootItemIds }, + { + idempotencyKey: knowledgeDeleteSubtreeIdempotencyKey(knowledgeBaseId, knowledgeRootItemIds), + queue: knowledgeQueueName(knowledgeBaseId) + } + ) }) - throw error - } + ) } async reindexItems(baseId: string, itemIds: string[]): Promise { + assertBaseCanRunRuntimeOperation(baseId, 'reindexItems') + const rootItemIds = knowledgeItemService.getOutermostSelectedItemIds(baseId, itemIds) + if (rootItemIds.length === 0) { + return + } + + await this.assertSubtreesCanReindex(baseId, rootItemIds) + knowledgeBaseService.getById(baseId) - const rootItemIds = [...new Set(itemIds)] const knowledgeBaseId = toKnowledgeBaseId(baseId) const knowledgeRootItemIds = toKnowledgeItemIds(rootItemIds) const jobManager = application.get('JobManager') @@ -285,7 +306,8 @@ export class KnowledgeWorkflowService { firstScheduledAt: Date.now(), // Use the file-processing job as workflow parent when this is a direct add flow, // so retries keep a stable index idempotency key across poll rounds. - parentJobId: parentJobId ?? fileProcessingJob.id + parentJobId: parentJobId ?? fileProcessingJob.id, + processedRelativePath }) } catch (error) { try { @@ -308,9 +330,9 @@ export class KnowledgeWorkflowService { baseId: KnowledgeBaseId, itemId: KnowledgeItemId, fileProcessingJobId: string, - options: { pollRound: number; firstScheduledAt: number; parentJobId: string | null } + options: { pollRound: number; firstScheduledAt: number; parentJobId: string | null; processedRelativePath: string } ): Promise { - const { pollRound, firstScheduledAt, parentJobId } = options + const { pollRound, firstScheduledAt, parentJobId, processedRelativePath } = options const jobManager = application.get('JobManager') jobManager.enqueue( 'knowledge.check-file-processing-result', @@ -320,7 +342,7 @@ export class KnowledgeWorkflowService { fileProcessingJobId, pollRound, firstScheduledAt, - parentJobId + processedRelativePath }, { idempotencyKey: knowledgeFileProcessingCheckIdempotencyKey(baseId, itemId, fileProcessingJobId, pollRound), @@ -344,7 +366,7 @@ export class KnowledgeWorkflowService { const jobManager = application.get('JobManager') jobManager.enqueue( 'knowledge.index-documents', - { baseId, itemId, parentJobId }, + { baseId, itemId }, { idempotencyKey: knowledgeIndexIdempotencyKey(baseId, itemId, parentJobId), queue: knowledgeQueueName(baseId), @@ -353,6 +375,139 @@ export class KnowledgeWorkflowService { ) } + /** + * Park items stranded mid-indexing by an app quit / restart at `failed`. + * + * Indexing handlers declare `recovery: 'abandon'`, so an interrupted job is + * cancelled rather than silently resumed on the next launch — a deliberate + * quit must not auto-spend the (paid) embedding API. The job side is handled + * by JobManager's startup recovery; this closes the item side. The common case + * (handler settled the abort as cancelled) is already flipped to `failed` by + * the job's onSettled; this is the boot-time safety net for the stragglers + * onSettled lost the race to write (process exited first) or never ran (hard + * kill / crash). Marking them `failed` clears the perpetual spinner and makes + * them reindexable so the user can finish them on demand. + */ + recoverInterruptedItems(): void { + try { + const failedCount = knowledgeItemService.failInterruptedItems(KNOWLEDGE_ITEM_ERROR_INDEXING_INTERRUPTED) + if (failedCount > 0) { + logger.info('Recovered interrupted knowledge items', { count: failedCount }) + } + } catch (error) { + logger.error('Failed to recover interrupted knowledge items', error as Error) + } + } + + recoverDeletingItems(): void { + let deletingRootGroups: Awaited> + try { + deletingRootGroups = knowledgeItemService.getDeletingRootGroups() + } catch (error) { + logger.error('Failed to scan deleting knowledge items for recovery', error as Error) + return + } + + if (deletingRootGroups.length === 0) { + return + } + + const jobManager = application.get('JobManager') + for (const { baseId, rootItemIds } of deletingRootGroups) { + for (let i = 0; i < rootItemIds.length; i += DELETE_RECOVERY_ROOT_CHUNK_SIZE) { + const rootItemIdChunk = rootItemIds.slice(i, i + DELETE_RECOVERY_ROOT_CHUNK_SIZE) + try { + jobManager.enqueue( + 'knowledge.delete-subtree', + { baseId, rootItemIds: rootItemIdChunk }, + { + idempotencyKey: knowledgeDeleteSubtreeIdempotencyKey( + toKnowledgeBaseId(baseId), + toKnowledgeItemIds(rootItemIdChunk) + ), + queue: knowledgeQueueName(toKnowledgeBaseId(baseId)) + } + ) + } catch (error) { + logger.error('Failed to enqueue recovered knowledge delete cleanup', error as Error, { + baseId, + rootItemIds: rootItemIdChunk + }) + } + } + } + } + + private async assertSubtreesCanReindex(baseId: string, rootItemIds: string[]): Promise { + // rootItemIds comes from getOutermostSelectedItemIds, which guarantees the roots are mutually + // non-descendant (disjoint subtrees), so one batched query's union equals the per-root sum. + const subtreeItems = knowledgeItemService.getSubtreeItems(baseId, rootItemIds, { includeRoots: true }) + const rootIdSet = new Set(rootItemIds) + const roots = subtreeItems.filter((item) => rootIdSet.has(item.id)) + + // Reindex deletes the subtree's vectors before re-reading the source (reindexSubtreeJobHandler), + // so a root whose source is gone would lose its vectors with nothing to rebuild from — reject up + // front. Only the root's own source matters: a directory is rescanned from data.source and its + // children recreated (never read from their raw/ files), a file leaf reads its own raw/ file, and + // note/url always rebuild from the DB / network. A v1-migrated folder child reindexed on its own + // is a file root whose raw/ file never existed, so this rejects it too. Distinguish a genuinely + // missing source (delete-and-re-add) from one we could not verify (transient/permission error, + // which should retry rather than be destroyed). + const sourceStates = await Promise.all(roots.map((root) => classifyKnowledgeItemSource(baseId, root))) + + const missingSourceItemIds: string[] = [] + const unverifiableSourceItemIds: string[] = [] + roots.forEach((root, index) => { + if (sourceStates[index] === 'missing') { + missingSourceItemIds.push(root.id) + } else if (sourceStates[index] === 'unverifiable') { + unverifiableSourceItemIds.push(root.id) + } + }) + + const blockingStatusCounts = new Map() + for (const item of subtreeItems) { + if (REINDEX_ALLOWED_STATUSES.has(item.status)) { + continue + } + blockingStatusCounts.set(item.status, (blockingStatusCounts.get(item.status) ?? 0) + 1) + } + + if (missingSourceItemIds.length > 0) { + throw DataApiErrorFactory.validation( + { + item: [`Knowledge item source no longer exists on disk for ${missingSourceItemIds.length} item(s)`] + }, + 'Cannot reindex a knowledge item whose source file or folder no longer exists; delete it and add it again to rebuild' + ) + } + + if (unverifiableSourceItemIds.length > 0) { + throw DataApiErrorFactory.validation( + { + item: [`Could not verify the knowledge item source on disk for ${unverifiableSourceItemIds.length} item(s)`] + }, + 'Could not verify the knowledge item source (it may be temporarily unavailable); please try again' + ) + } + + if (blockingStatusCounts.size === 0) { + return + } + + const statusSummary = [...blockingStatusCounts.entries()] + .sort(([left], [right]) => left.localeCompare(right)) + .map(([status, count]) => `${status}=${count}`) + .join(', ') + + throw DataApiErrorFactory.validation( + { + item: [`Knowledge item subtree is still running or being deleted: ${statusSummary}`] + }, + 'Cannot reindex knowledge item until the entire subtree is completed or failed' + ) + } + private rollbackAcceptedItems(baseId: string, items: KnowledgeItem[], originalError: unknown): void { for (const item of items) { try { @@ -461,8 +616,7 @@ export class KnowledgeWorkflowService { errorMessage: message, failedStatusError: `Failed to schedule knowledge item job: ${message}`, logger, - logMessage: 'Failed to mark unscheduled knowledge item after addItems scheduling failure', - logContextKey: 'scheduleError' + logMessage: 'Failed to mark unscheduled knowledge item after addItems scheduling failure' }) } } diff --git a/src/main/features/knowledge/utils/__tests__/addConflicts.test.ts b/src/main/features/knowledge/ingestion/__tests__/addConflicts.test.ts similarity index 100% rename from src/main/features/knowledge/utils/__tests__/addConflicts.test.ts rename to src/main/features/knowledge/ingestion/__tests__/addConflicts.test.ts diff --git a/src/main/features/knowledge/utils/cleanup/__tests__/statusCleanup.integration.test.ts b/src/main/features/knowledge/ingestion/__tests__/statusCleanup.integration.test.ts similarity index 58% rename from src/main/features/knowledge/utils/cleanup/__tests__/statusCleanup.integration.test.ts rename to src/main/features/knowledge/ingestion/__tests__/statusCleanup.integration.test.ts index 50dc19a9b15..bc9cf0b730d 100644 --- a/src/main/features/knowledge/utils/cleanup/__tests__/statusCleanup.integration.test.ts +++ b/src/main/features/knowledge/ingestion/__tests__/statusCleanup.integration.test.ts @@ -87,44 +87,34 @@ describe('markUnscheduledKnowledgeItemsFailed integration', () => { ]) }) - it('falls back to subtree status without reviving deleting descendants', async () => { - const updateStatusSpy = vi.spyOn(knowledgeItemService, 'updateStatus') - updateStatusSpy.mockImplementationOnce(() => { - throw new Error('status busy') + it('does not revive deleting descendants when marking unscheduled items failed', async () => { + markUnscheduledKnowledgeItemsFailed({ + baseId: BASE_ID, + items: [ + { + id: ROOT_ID, + baseId: BASE_ID, + groupId: null, + type: 'directory', + data: { source: 'root' }, + status: 'processing', + error: null, + createdAt: '2026-04-08T00:00:00.000Z', + updatedAt: '2026-04-08T00:00:00.000Z' + } + ], + completedItemIds: new Set(), + errorMessage: 'enqueue failed', + failedStatusError: 'Failed to schedule knowledge child item job: enqueue failed', + logger: { + debug: vi.fn(), + error: vi.fn(), + info: vi.fn(), + warn: vi.fn() + } as unknown as LoggerService, + logMessage: 'Failed to mark unscheduled item' }) - try { - markUnscheduledKnowledgeItemsFailed({ - baseId: BASE_ID, - items: [ - { - id: ROOT_ID, - baseId: BASE_ID, - groupId: null, - type: 'directory', - data: { source: 'root' }, - status: 'processing', - error: null, - createdAt: '2026-04-08T00:00:00.000Z', - updatedAt: '2026-04-08T00:00:00.000Z' - } - ], - completedItemIds: new Set(), - errorMessage: 'enqueue failed', - failedStatusError: 'Failed to schedule knowledge child item job: enqueue failed', - logger: { - debug: vi.fn(), - error: vi.fn(), - info: vi.fn(), - warn: vi.fn() - } as unknown as LoggerService, - logMessage: 'Failed to mark unscheduled item', - logContextKey: 'scheduleError' - }) - } finally { - updateStatusSpy.mockRestore() - } - const rows = await dbh.db.select().from(knowledgeItemTable).where(eq(knowledgeItemTable.baseId, BASE_ID)) expect(rows).toEqual( expect.arrayContaining([ @@ -141,4 +131,51 @@ describe('markUnscheduledKnowledgeItemsFailed integration', () => { ]) ) }) + + it('aggregates unrecovered item ids and throws when setSubtreeStatus fails', () => { + const setSubtreeStatusSpy = vi.spyOn(knowledgeItemService, 'setSubtreeStatus') + setSubtreeStatusSpy.mockImplementationOnce(() => { + throw new Error('status busy') + }) + const errorLogger = vi.fn() + + try { + expect(() => + markUnscheduledKnowledgeItemsFailed({ + baseId: BASE_ID, + items: [ + { + id: ROOT_ID, + baseId: BASE_ID, + groupId: null, + type: 'directory', + data: { source: 'root' }, + status: 'processing', + error: null, + createdAt: '2026-04-08T00:00:00.000Z', + updatedAt: '2026-04-08T00:00:00.000Z' + } + ], + completedItemIds: new Set(), + errorMessage: 'enqueue failed', + failedStatusError: 'Failed to schedule knowledge child item job: enqueue failed', + logger: { + debug: vi.fn(), + error: errorLogger, + info: vi.fn(), + warn: vi.fn() + } as unknown as LoggerService, + logMessage: 'Failed to mark unscheduled item' + }) + ).toThrow(`Failed to mark unscheduled knowledge items failed; unrecovered item ids: ${ROOT_ID}`) + } finally { + setSubtreeStatusSpy.mockRestore() + } + + expect(errorLogger).toHaveBeenCalledWith( + 'Failed to mark unscheduled item', + expect.any(Error), + expect.objectContaining({ baseId: BASE_ID, itemId: ROOT_ID }) + ) + }) }) diff --git a/src/main/features/knowledge/utils/addConflicts.ts b/src/main/features/knowledge/ingestion/addConflicts.ts similarity index 100% rename from src/main/features/knowledge/utils/addConflicts.ts rename to src/main/features/knowledge/ingestion/addConflicts.ts diff --git a/src/main/features/knowledge/utils/cleanup/statusCleanup.ts b/src/main/features/knowledge/ingestion/statusCleanup.ts similarity index 68% rename from src/main/features/knowledge/utils/cleanup/statusCleanup.ts rename to src/main/features/knowledge/ingestion/statusCleanup.ts index 48b5e72759f..13386a89d57 100644 --- a/src/main/features/knowledge/utils/cleanup/statusCleanup.ts +++ b/src/main/features/knowledge/ingestion/statusCleanup.ts @@ -10,7 +10,6 @@ type MarkFailedInput = { failedStatusError: string logger: LoggerService logMessage: string - logContextKey: string } export function markUnscheduledKnowledgeItemsFailed(input: MarkFailedInput): void { @@ -22,35 +21,18 @@ export function markUnscheduledKnowledgeItemsFailed(input: MarkFailedInput): voi } try { - knowledgeItemService.updateStatus(item.id, 'failed', { + knowledgeItemService.setSubtreeStatus(input.baseId, [item.id], 'failed', { error: input.failedStatusError }) - continue } catch (cleanupError) { + unrecoveredItemIds.push(item.id) input.logger.error( input.logMessage, cleanupError instanceof Error ? cleanupError : new Error(String(cleanupError)), { baseId: input.baseId, itemId: item.id, - [input.logContextKey]: input.errorMessage - } - ) - } - - try { - knowledgeItemService.setSubtreeStatus(input.baseId, [item.id], 'failed', { - error: input.failedStatusError - }) - } catch (fallbackError) { - unrecoveredItemIds.push(item.id) - input.logger.error( - 'Failed to mark unscheduled knowledge item through subtree fallback', - fallbackError instanceof Error ? fallbackError : new Error(String(fallbackError)), - { - baseId: input.baseId, - itemId: item.id, - [input.logContextKey]: input.errorMessage + scheduleError: input.errorMessage } ) } diff --git a/src/main/features/knowledge/ingestion/subtreePurge.ts b/src/main/features/knowledge/ingestion/subtreePurge.ts new file mode 100644 index 00000000000..316fe5efed3 --- /dev/null +++ b/src/main/features/knowledge/ingestion/subtreePurge.ts @@ -0,0 +1,36 @@ +import { knowledgeItemService } from '@data/services/KnowledgeItemService' +import type { KnowledgeBase, KnowledgeItem } from '@shared/data/types/knowledge' + +import { isIndexableKnowledgeItem } from '../items' +import { deleteKnowledgeItemFilesBestEffort } from '../pathStorage' +import { deleteKnowledgeItemVectors } from '../pipeline/vectorstore/vectorCleanup' + +/** + * Remove a resolved subtree's vectors, on-disk files, and DB rows, in that order. + * MUST run INSIDE the base mutation lock so no indexer can write vectors for the + * rows being removed, and so a caller (e.g. replace-on-add) can purge and then + * recreate within one lock acquisition — keeping the freed name available to the + * incoming source. Callers resolve and filter `subtreeItems` themselves (the + * delete job keeps only `deleting` rows; replace passes the conflicting roots' + * subtrees), then run vector cleanup before DB deletion so a retry can still + * discover affected ids. + */ +export async function purgeKnowledgeSubtreeWithinLock( + base: KnowledgeBase, + subtreeItems: KnowledgeItem[], + logContext: Record +): Promise { + const subtreeItemIds = subtreeItems.map((item) => item.id) + if (subtreeItemIds.length === 0) { + return + } + const leafItemIds = subtreeItems.filter((item) => isIndexableKnowledgeItem(item)).map((item) => item.id) + + // Vector cleanup precedes DB deletion so a retry can still discover affected item ids. + await deleteKnowledgeItemVectors(base, leafItemIds) + // Best-effort: a file-removal failure must not abort the row deletion below, + // which would otherwise strand rows after their vectors are gone. + await deleteKnowledgeItemFilesBestEffort(base.id, subtreeItems, logContext) + + knowledgeItemService.deleteItemsByIds(base.id, subtreeItemIds) +} diff --git a/src/main/features/knowledge/utils/items.ts b/src/main/features/knowledge/items.ts similarity index 53% rename from src/main/features/knowledge/utils/items.ts rename to src/main/features/knowledge/items.ts index 426feb694d3..c3617e312ae 100644 --- a/src/main/features/knowledge/utils/items.ts +++ b/src/main/features/knowledge/items.ts @@ -1,8 +1,11 @@ import type { PathReadability } from '@main/utils/file/fs' -import type { KnowledgeItem } from '@shared/data/types/knowledge' +import type { KnowledgeItem, KnowledgeItemOf } from '@shared/data/types/knowledge' -import type { ContainerKnowledgeItem, IndexableKnowledgeItem } from '../types/items' -import { probeKnowledgeFile, probeKnowledgeSourcePath } from './storage/pathStorage' +import { probeKnowledgeFile, probeKnowledgeSourcePath } from './pathStorage' + +export type IndexableKnowledgeItem = KnowledgeItemOf<'file' | 'url' | 'note'> + +export type ContainerKnowledgeItem = KnowledgeItemOf<'directory'> export function isIndexableKnowledgeItem(item: KnowledgeItem): item is IndexableKnowledgeItem { return item.type === 'file' || item.type === 'url' || item.type === 'note' @@ -12,6 +15,36 @@ export function isContainerKnowledgeItem(item: KnowledgeItem): item is Container return item.type === 'directory' } +/** + * The subset of an indexable knowledge item needed to derive its index-store + * material fields. The `Pick` is distributed per member so `type` and `data` + * stay correlated (a single `Pick` over the union would collapse `data` to a bare + * union and lose the file-only `relativePath` / `indexedRelativePath`). Shared by + * the indexing job and the v1→v2 vector migrator so both stamp the material + * identically (knowledge-technical-design.md §4.2). + */ +export type MaterialFieldSource = + | Pick, 'id' | 'type' | 'data'> + | Pick, 'id' | 'type' | 'data'> + | Pick, 'id' | 'type' | 'data'> + +/** + * A material's stable relative path. A file uses its stored path (the processed + * artifact when present). A url or note uses its captured snapshot path — a real + * base file under `raw/`, materialized before the material is stamped (the index + * job's ensure-snapshot step, or the vector migrator), so it is always present + * here; a missing one is an invariant violation, not a fallback case. + */ +export function toMaterialRelativePath(item: MaterialFieldSource): string { + if (item.type === 'file') { + return item.data.indexedRelativePath ?? item.data.relativePath + } + if (!item.data.relativePath) { + throw new Error(`Knowledge ${item.type} item ${item.id} has no captured snapshot relativePath for its material`) + } + return item.data.relativePath +} + /** Whether a knowledge item's rebuild source is present, genuinely gone, or merely unverifiable. */ export type KnowledgeItemSourceState = 'rebuildable' | 'missing' | 'unverifiable' @@ -34,14 +67,14 @@ export async function classifyKnowledgeItemSource( return toSourceState(await probeKnowledgeSourcePath(item.data.source)) } if (item.type === 'file') { - return toSourceState(await probeKnowledgeFile(baseId, item.data.indexedRelativePath ?? item.data.relativePath)) + return toSourceState(await probeKnowledgeFile(baseId, toMaterialRelativePath(item))) } return 'rebuildable' } /** * Whether a knowledge item can rebuild from a still-readable source. Gates reindex both at admission - * (`KnowledgeService.assertSubtreesCanReindex`) and inside the reindex job's mutation lock right + * (`KnowledgeIngestionService.assertSubtreesCanReindex`) and inside the reindex job's mutation lock right * before the delete — a vanished or unverifiable source must never wipe vectors with nothing to * rebuild from. Admission additionally distinguishes the two via {@link classifyKnowledgeItemSource}. */ diff --git a/src/main/features/knowledge/utils/storage/pathStorage.ts b/src/main/features/knowledge/pathStorage.ts similarity index 94% rename from src/main/features/knowledge/utils/storage/pathStorage.ts rename to src/main/features/knowledge/pathStorage.ts index 8319395e183..1bac3a2aa1a 100644 --- a/src/main/features/knowledge/utils/storage/pathStorage.ts +++ b/src/main/features/knowledge/pathStorage.ts @@ -39,7 +39,7 @@ export function getKnowledgeMaterialDir(baseId: string): FilePath { return path.join(getKnowledgeBaseDir(baseId), MATERIAL_ROOT_DIR) as FilePath } -export function getKnowledgeBaseMetaDir(baseId: string): FilePath { +function getKnowledgeBaseMetaDir(baseId: string): FilePath { return path.join(getKnowledgeBaseDir(baseId), CHERRY_META_DIR) as FilePath } @@ -84,16 +84,6 @@ export function getKnowledgeSourceRelativePath(sourcePath: string): string { return fileName } -export function toKnowledgeRelativePath(baseId: string, absolutePath: string): string { - const materialDir = getKnowledgeMaterialDir(baseId) - const relativePath = path.relative(materialDir, absolutePath) - assertSafeKnowledgeRelativePath(relativePath) - if (!isPathInsideBase(materialDir, absolutePath)) { - throw new Error(`Path is outside knowledge base material root '${baseId}': ${absolutePath}`) - } - return normalizeRelativePath(relativePath) -} - export function getProcessedMarkdownRelativePath(relativePath: string): string { assertSafeKnowledgeRelativePath(relativePath) const parsed = path.parse(relativePath) @@ -319,11 +309,6 @@ function normalizeRelativePath(relativePath: string): string { return path.normalize(relativePath).replace(/\\/g, '/') } -function isPathInsideBase(baseDir: string, candidatePath: string): boolean { - const relativePath = path.relative(baseDir, candidatePath) - return Boolean(relativePath) && !relativePath.startsWith('..') && !path.isAbsolute(relativePath) -} - async function assertTargetAvailable(destPath: FilePath): Promise { try { await fs.lstat(destPath) diff --git a/src/main/features/knowledge/utils/indexing/__tests__/chunk.test.ts b/src/main/features/knowledge/pipeline/indexing/__tests__/chunk.test.ts similarity index 100% rename from src/main/features/knowledge/utils/indexing/__tests__/chunk.test.ts rename to src/main/features/knowledge/pipeline/indexing/__tests__/chunk.test.ts diff --git a/src/main/features/knowledge/utils/indexing/__tests__/embed.test.ts b/src/main/features/knowledge/pipeline/indexing/__tests__/embed.test.ts similarity index 100% rename from src/main/features/knowledge/utils/indexing/__tests__/embed.test.ts rename to src/main/features/knowledge/pipeline/indexing/__tests__/embed.test.ts diff --git a/src/main/features/knowledge/utils/indexing/__tests__/rerank.test.ts b/src/main/features/knowledge/pipeline/indexing/__tests__/rerank.test.ts similarity index 100% rename from src/main/features/knowledge/utils/indexing/__tests__/rerank.test.ts rename to src/main/features/knowledge/pipeline/indexing/__tests__/rerank.test.ts diff --git a/src/main/features/knowledge/utils/indexing/__tests__/splitter.test.ts b/src/main/features/knowledge/pipeline/indexing/__tests__/splitter.test.ts similarity index 100% rename from src/main/features/knowledge/utils/indexing/__tests__/splitter.test.ts rename to src/main/features/knowledge/pipeline/indexing/__tests__/splitter.test.ts diff --git a/src/main/features/knowledge/utils/indexing/chunk.ts b/src/main/features/knowledge/pipeline/indexing/chunk.ts similarity index 100% rename from src/main/features/knowledge/utils/indexing/chunk.ts rename to src/main/features/knowledge/pipeline/indexing/chunk.ts diff --git a/src/main/features/knowledge/utils/indexing/embed.ts b/src/main/features/knowledge/pipeline/indexing/embed.ts similarity index 100% rename from src/main/features/knowledge/utils/indexing/embed.ts rename to src/main/features/knowledge/pipeline/indexing/embed.ts diff --git a/src/main/features/knowledge/utils/indexing/rerank.ts b/src/main/features/knowledge/pipeline/indexing/rerank.ts similarity index 100% rename from src/main/features/knowledge/utils/indexing/rerank.ts rename to src/main/features/knowledge/pipeline/indexing/rerank.ts diff --git a/src/main/features/knowledge/utils/indexing/splitter.ts b/src/main/features/knowledge/pipeline/indexing/splitter.ts similarity index 100% rename from src/main/features/knowledge/utils/indexing/splitter.ts rename to src/main/features/knowledge/pipeline/indexing/splitter.ts diff --git a/src/main/features/knowledge/readers/KnowledgeFileReader.ts b/src/main/features/knowledge/pipeline/readers/KnowledgeFileReader.ts similarity index 90% rename from src/main/features/knowledge/readers/KnowledgeFileReader.ts rename to src/main/features/knowledge/pipeline/readers/KnowledgeFileReader.ts index 7f183edd3bf..cbbd641ee3d 100644 --- a/src/main/features/knowledge/readers/KnowledgeFileReader.ts +++ b/src/main/features/knowledge/pipeline/readers/KnowledgeFileReader.ts @@ -10,7 +10,8 @@ import { MarkdownReader } from '@vectorstores/readers/markdown' import { PDFReader } from '@vectorstores/readers/pdf' import { TextFileReader } from '@vectorstores/readers/text' -import { getKnowledgeBaseFilePath } from '../utils/storage/pathStorage' +import { toMaterialRelativePath } from '../../items' +import { getKnowledgeBaseFilePath } from '../../pathStorage' import { DraftsExportReader } from './files/DraftsExportReader' import { EpubReader } from './files/EpubReader' @@ -67,9 +68,5 @@ export async function loadDocumentsFromKnowledgeBaseFile( } export async function loadFileDocuments(item: KnowledgeItemOf<'file'>): Promise { - return loadDocumentsFromKnowledgeBaseFile( - item.baseId, - item.data.indexedRelativePath ?? item.data.relativePath, - item.data.source - ) + return loadDocumentsFromKnowledgeBaseFile(item.baseId, toMaterialRelativePath(item), item.data.source) } diff --git a/src/main/features/knowledge/readers/KnowledgeReader.ts b/src/main/features/knowledge/pipeline/readers/KnowledgeReader.ts similarity index 63% rename from src/main/features/knowledge/readers/KnowledgeReader.ts rename to src/main/features/knowledge/pipeline/readers/KnowledgeReader.ts index b7866448309..5818bdeb02d 100644 --- a/src/main/features/knowledge/readers/KnowledgeReader.ts +++ b/src/main/features/knowledge/pipeline/readers/KnowledgeReader.ts @@ -1,12 +1,11 @@ -import type { KnowledgeItem, KnowledgeItemOf } from '@shared/data/types/knowledge' +import type { KnowledgeItem } from '@shared/data/types/knowledge' import type { Document } from '@vectorstores/core' +import type { IndexableKnowledgeItem } from '../../items' import { loadFileDocuments } from './KnowledgeFileReader' import { loadSnapshotDocuments } from './KnowledgeSnapshotReader' -export type ReadableKnowledgeItem = KnowledgeItemOf<'file'> | KnowledgeItemOf<'url'> | KnowledgeItemOf<'note'> - -export async function loadKnowledgeItemDocuments(item: ReadableKnowledgeItem): Promise { +export async function loadKnowledgeItemDocuments(item: IndexableKnowledgeItem): Promise { switch (item.type) { case 'file': return await loadFileDocuments(item) diff --git a/src/main/features/knowledge/readers/KnowledgeSnapshotReader.ts b/src/main/features/knowledge/pipeline/readers/KnowledgeSnapshotReader.ts similarity index 91% rename from src/main/features/knowledge/readers/KnowledgeSnapshotReader.ts rename to src/main/features/knowledge/pipeline/readers/KnowledgeSnapshotReader.ts index 53b7da2cfe3..d45d9dbcf75 100644 --- a/src/main/features/knowledge/readers/KnowledgeSnapshotReader.ts +++ b/src/main/features/knowledge/pipeline/readers/KnowledgeSnapshotReader.ts @@ -2,8 +2,8 @@ import { read } from '@main/utils/file/fs' import type { KnowledgeItemOf, KnowledgeSourceMetadata } from '@shared/data/types/knowledge' import { Document } from '@vectorstores/core' -import { stripOkfFrontmatter } from '../utils/sources/okfFrontmatter' -import { getKnowledgeBaseFilePath } from '../utils/storage/pathStorage' +import { getKnowledgeBaseFilePath } from '../../pathStorage' +import { stripOkfFrontmatter } from '../sources/okfFrontmatter' /** * Read a url or note item from its captured on-disk snapshot — never the network diff --git a/src/main/features/knowledge/readers/__tests__/KnowledgeReaderMetadata.test.ts b/src/main/features/knowledge/pipeline/readers/__tests__/KnowledgeReaderMetadata.test.ts similarity index 97% rename from src/main/features/knowledge/readers/__tests__/KnowledgeReaderMetadata.test.ts rename to src/main/features/knowledge/pipeline/readers/__tests__/KnowledgeReaderMetadata.test.ts index 2bc86693cfb..612e459c6e7 100644 --- a/src/main/features/knowledge/readers/__tests__/KnowledgeReaderMetadata.test.ts +++ b/src/main/features/knowledge/pipeline/readers/__tests__/KnowledgeReaderMetadata.test.ts @@ -65,7 +65,7 @@ describe('knowledge reader metadata', () => { source: '/tmp/original.txt', relativePath: 'original.txt' }, - status: 'idle', + status: 'processing', error: null, createdAt: '2026-04-08T00:00:00.000Z', updatedAt: '2026-04-08T00:00:00.000Z' @@ -89,7 +89,7 @@ describe('knowledge reader metadata', () => { groupId: null, type: 'url', data: { source: 'https://example.com', url: 'https://example.com', relativePath: 'example.md' }, - status: 'idle', + status: 'processing', error: null, createdAt: '2026-04-08T00:00:00.000Z', updatedAt: '2026-04-08T00:00:00.000Z' @@ -114,7 +114,7 @@ describe('knowledge reader metadata', () => { groupId: null, type: 'url', data: { source: 'https://example.com', url: 'https://example.com' }, - status: 'idle', + status: 'processing', error: null, createdAt: '2026-04-08T00:00:00.000Z', updatedAt: '2026-04-08T00:00:00.000Z' @@ -134,7 +134,7 @@ describe('knowledge reader metadata', () => { groupId: null, type: 'note', data: { source: 'My note', content: '# Note title\n\nbody', relativePath: 'My note.md' }, - status: 'idle', + status: 'processing', error: null, createdAt: '2026-04-08T00:00:00.000Z', updatedAt: '2026-04-08T00:00:00.000Z' @@ -159,7 +159,7 @@ describe('knowledge reader metadata', () => { groupId: null, type: 'note', data: { source: 'My note', content: 'body' }, - status: 'idle', + status: 'processing', error: null, createdAt: '2026-04-08T00:00:00.000Z', updatedAt: '2026-04-08T00:00:00.000Z' diff --git a/src/main/features/knowledge/readers/__tests__/ReaderFactory.test.ts b/src/main/features/knowledge/pipeline/readers/__tests__/ReaderFactory.test.ts similarity index 98% rename from src/main/features/knowledge/readers/__tests__/ReaderFactory.test.ts rename to src/main/features/knowledge/pipeline/readers/__tests__/ReaderFactory.test.ts index bf8bd86dadd..3666c27b1bc 100644 --- a/src/main/features/knowledge/readers/__tests__/ReaderFactory.test.ts +++ b/src/main/features/knowledge/pipeline/readers/__tests__/ReaderFactory.test.ts @@ -109,7 +109,7 @@ function createFileItem(ext: string, sourcePath?: string): KnowledgeItemOf<'file baseId: 'base-1', groupId: null, type: 'file', - status: 'idle', + status: 'processing', error: null, createdAt: '2026-04-03T00:00:00.000Z', updatedAt: '2026-04-03T00:00:00.000Z', @@ -126,7 +126,7 @@ function createNoteItem(content: string, relativePath = 'note-1.md'): KnowledgeI baseId: 'base-1', groupId: null, type: 'note', - status: 'idle', + status: 'processing', error: null, createdAt: '2026-04-03T00:00:00.000Z', updatedAt: '2026-04-03T00:00:00.000Z', @@ -144,7 +144,7 @@ function createUrlItem(): KnowledgeItemOf<'url'> { baseId: 'base-1', groupId: null, type: 'url', - status: 'idle', + status: 'processing', error: null, createdAt: '2026-04-03T00:00:00.000Z', updatedAt: '2026-04-03T00:00:00.000Z', @@ -162,7 +162,7 @@ function createDirectoryItem(): KnowledgeItemOf<'directory'> { baseId: 'base-1', groupId: null, type: 'directory', - status: 'idle', + status: 'processing', error: null, createdAt: '2026-04-03T00:00:00.000Z', updatedAt: '2026-04-03T00:00:00.000Z', diff --git a/src/main/features/knowledge/readers/files/DraftsExportReader.ts b/src/main/features/knowledge/pipeline/readers/files/DraftsExportReader.ts similarity index 100% rename from src/main/features/knowledge/readers/files/DraftsExportReader.ts rename to src/main/features/knowledge/pipeline/readers/files/DraftsExportReader.ts diff --git a/src/main/features/knowledge/readers/files/EpubReader.ts b/src/main/features/knowledge/pipeline/readers/files/EpubReader.ts similarity index 100% rename from src/main/features/knowledge/readers/files/EpubReader.ts rename to src/main/features/knowledge/pipeline/readers/files/EpubReader.ts diff --git a/src/main/features/knowledge/readers/files/__tests__/DraftsExportReader.test.ts b/src/main/features/knowledge/pipeline/readers/files/__tests__/DraftsExportReader.test.ts similarity index 100% rename from src/main/features/knowledge/readers/files/__tests__/DraftsExportReader.test.ts rename to src/main/features/knowledge/pipeline/readers/files/__tests__/DraftsExportReader.test.ts diff --git a/src/main/features/knowledge/readers/files/__tests__/EpubReader.test.ts b/src/main/features/knowledge/pipeline/readers/files/__tests__/EpubReader.test.ts similarity index 100% rename from src/main/features/knowledge/readers/files/__tests__/EpubReader.test.ts rename to src/main/features/knowledge/pipeline/readers/files/__tests__/EpubReader.test.ts diff --git a/src/main/features/knowledge/utils/sources/__tests__/directory.test.ts b/src/main/features/knowledge/pipeline/sources/__tests__/directory.test.ts similarity index 63% rename from src/main/features/knowledge/utils/sources/__tests__/directory.test.ts rename to src/main/features/knowledge/pipeline/sources/__tests__/directory.test.ts index 0d6e48a21ff..e79eb0303c6 100644 --- a/src/main/features/knowledge/utils/sources/__tests__/directory.test.ts +++ b/src/main/features/knowledge/pipeline/sources/__tests__/directory.test.ts @@ -2,9 +2,10 @@ import type * as NodeFs from 'node:fs' import type * as NodeOs from 'node:os' import path from 'node:path' +import type { KnowledgeItem } from '@shared/data/types/knowledge' import { afterEach, describe, expect, it, vi } from 'vitest' -import type * as PathStorage from '../../storage/pathStorage' +import type * as PathStorage from '../../../pathStorage' const copyFileIntoKnowledgeBaseAtMock = vi.hoisted(() => vi.fn(async (_baseId: string, _externalPath: string, relativePath: string) => relativePath) @@ -15,15 +16,15 @@ vi.mock('@application', async () => { return mockApplicationFactory() }) -vi.mock('../../storage/pathStorage', async () => { - const actual = await vi.importActual('../../storage/pathStorage') +vi.mock('../../../pathStorage', async () => { + const actual = await vi.importActual('../../../pathStorage') return { ...actual, copyFileIntoKnowledgeBaseAt: copyFileIntoKnowledgeBaseAtMock } }) -const { expandDirectoryOwnerToTree } = await import('../directory') +const { chooseDirectoryPathPrefix, expandDirectoryOwnerToTree } = await import('../directory') const realFs = await vi.importActual('node:fs') const realOs = await vi.importActual('node:os') @@ -35,6 +36,45 @@ function createSignal() { return new AbortController().signal } +function createDirectoryOwner(source: string): KnowledgeItem { + return { + id: 'dir-owner-1', + baseId: 'kb-1', + groupId: null, + type: 'directory', + data: { source }, + status: 'processing', + error: null, + createdAt: '2026-04-08T00:00:00.000Z', + updatedAt: '2026-04-08T00:00:00.000Z' + } +} + +describe('chooseDirectoryPathPrefix', () => { + it('uses the directory basename when that top-level name is free', () => { + expect(chooseDirectoryPathPrefix(createDirectoryOwner('/some/path/anna'), new Set())).toBe('anna') + }) + + it('dedupes with a `_N` suffix when the top-level name is already taken', () => { + expect(chooseDirectoryPathPrefix(createDirectoryOwner('/some/path/project'), new Set(['project']))).toBe( + 'project_1' + ) + }) + + it('dedupes after the whole basename, not before a fake extension', () => { + // A folder literally named `report.v2`: the trailing `.v2` is part of the name, + // not a file extension, so the suffix must land after it (`report.v2_1`). + expect(chooseDirectoryPathPrefix(createDirectoryOwner('/some/path/report.v2'), new Set(['report.v2']))).toBe( + 'report.v2_1' + ) + }) + + it('rejects a non-directory owner', () => { + const fileOwner = { ...createDirectoryOwner('/some/path/x'), type: 'file' } as KnowledgeItem + expect(() => chooseDirectoryPathPrefix(fileOwner, new Set())).toThrow("must be type 'directory'") + }) +}) + describe('expandDirectoryOwnerToTree', () => { let tempRoot: string | undefined @@ -53,24 +93,9 @@ describe('expandDirectoryOwnerToTree', () => { realFs.writeFileSync(path.join(rootDir, '.dockerignore'), 'node_modules') realFs.writeFileSync(path.join(nestedDir, 'skill.md'), '# skill') - const { pathPrefix, children } = await expandDirectoryOwnerToTree( - { - id: 'dir-owner-1', - baseId: 'kb-1', - groupId: null, - type: 'directory', - data: { - source: rootDir - }, - status: 'idle', - error: null, - createdAt: '2026-04-08T00:00:00.000Z', - updatedAt: '2026-04-08T00:00:00.000Z' - }, - 'kb-1', - new Set(), - createSignal() - ) + const owner = createDirectoryOwner(rootDir) + const pathPrefix = chooseDirectoryPathPrefix(owner, new Set()) + const children = await expandDirectoryOwnerToTree(owner, 'kb-1', pathPrefix, createSignal()) expect(pathPrefix).toBe('anna') expect(children).toEqual([ @@ -106,22 +131,11 @@ describe('expandDirectoryOwnerToTree', () => { realFs.writeFileSync(path.join(rootDir, 'readme.md'), '# readme') realFs.writeFileSync(path.join(nestedDir, 'reference.md'), '# reference') - const { children } = await expandDirectoryOwnerToTree( - { - id: 'dir-owner-1', - baseId: 'kb-1', - groupId: null, - type: 'directory', - data: { - source: rootDir - }, - status: 'idle', - error: null, - createdAt: '2026-04-08T00:00:00.000Z', - updatedAt: '2026-04-08T00:00:00.000Z' - }, + const owner = createDirectoryOwner(rootDir) + const children = await expandDirectoryOwnerToTree( + owner, 'kb-1', - new Set(), + chooseDirectoryPathPrefix(owner, new Set()), createSignal() ) @@ -171,22 +185,11 @@ describe('expandDirectoryOwnerToTree', () => { realFs.writeFileSync(path.join(rootDir, 'sheet.ods'), 'ods') copyFileIntoKnowledgeBaseAtMock.mockClear() - const { children } = await expandDirectoryOwnerToTree( - { - id: 'dir-owner-1', - baseId: 'kb-1', - groupId: null, - type: 'directory', - data: { - source: rootDir - }, - status: 'idle', - error: null, - createdAt: '2026-04-08T00:00:00.000Z', - updatedAt: '2026-04-08T00:00:00.000Z' - }, + const owner = createDirectoryOwner(rootDir) + const children = await expandDirectoryOwnerToTree( + owner, 'kb-1', - new Set(), + chooseDirectoryPathPrefix(owner, new Set()), createSignal() ) @@ -218,22 +221,11 @@ describe('expandDirectoryOwnerToTree', () => { realFs.writeFileSync(path.join(dirB, 'notes.md'), '# b') copyFileIntoKnowledgeBaseAtMock.mockClear() - const { children } = await expandDirectoryOwnerToTree( - { - id: 'dir-owner-1', - baseId: 'kb-1', - groupId: null, - type: 'directory', - data: { - source: rootDir - }, - status: 'idle', - error: null, - createdAt: '2026-04-08T00:00:00.000Z', - updatedAt: '2026-04-08T00:00:00.000Z' - }, + const owner = createDirectoryOwner(rootDir) + const children = await expandDirectoryOwnerToTree( + owner, 'kb-1', - new Set(), + chooseDirectoryPathPrefix(owner, new Set()), createSignal() ) @@ -245,31 +237,16 @@ describe('expandDirectoryOwnerToTree', () => { expect(new Set(copiedTargets).size).toBe(copiedTargets.length) }) - it('dedupes the top-level directory name with a `_N` suffix when it is already taken', async () => { + it('stores children under a deduped prefix passed in by the caller', async () => { tempRoot = createTempRoot() const rootDir = path.join(tempRoot, 'project') realFs.mkdirSync(rootDir, { recursive: true }) realFs.writeFileSync(path.join(rootDir, 'notes.md'), '# notes') - const { pathPrefix, children } = await expandDirectoryOwnerToTree( - { - id: 'dir-owner-1', - baseId: 'kb-1', - groupId: null, - type: 'directory', - data: { - source: rootDir - }, - status: 'idle', - error: null, - createdAt: '2026-04-08T00:00:00.000Z', - updatedAt: '2026-04-08T00:00:00.000Z' - }, - 'kb-1', - // A prior `project` directory already occupies that top-level name under raw/. - new Set(['project']), - createSignal() - ) + const owner = createDirectoryOwner(rootDir) + // A prior `project` directory already occupies that top-level name under raw/. + const pathPrefix = chooseDirectoryPathPrefix(owner, new Set(['project'])) + const children = await expandDirectoryOwnerToTree(owner, 'kb-1', pathPrefix, createSignal()) expect(pathPrefix).toBe('project_1') expect(children).toEqual([ @@ -283,46 +260,6 @@ describe('expandDirectoryOwnerToTree', () => { ]) }) - it('dedupes a dotted directory name after the whole basename, not before a fake extension', async () => { - tempRoot = createTempRoot() - // A folder literally named `report.v2`: the trailing `.v2` is part of the name, - // not a file extension, so the suffix must land after it (`report.v2_1`). - const rootDir = path.join(tempRoot, 'report.v2') - realFs.mkdirSync(rootDir, { recursive: true }) - realFs.writeFileSync(path.join(rootDir, 'notes.md'), '# notes') - - const { pathPrefix, children } = await expandDirectoryOwnerToTree( - { - id: 'dir-owner-1', - baseId: 'kb-1', - groupId: null, - type: 'directory', - data: { - source: rootDir - }, - status: 'idle', - error: null, - createdAt: '2026-04-08T00:00:00.000Z', - updatedAt: '2026-04-08T00:00:00.000Z' - }, - 'kb-1', - // A prior `report.v2` directory already occupies that top-level name under raw/. - new Set(['report.v2']), - createSignal() - ) - - expect(pathPrefix).toBe('report.v2_1') - expect(children).toEqual([ - { - type: 'file', - data: { - source: path.join(rootDir, 'notes.md'), - relativePath: 'report.v2_1/notes.md' - } - } - ]) - }) - it('stops before reading when the runtime signal is already aborted', async () => { tempRoot = createTempRoot() const controller = new AbortController() @@ -330,24 +267,7 @@ describe('expandDirectoryOwnerToTree', () => { controller.abort(abortError) await expect( - expandDirectoryOwnerToTree( - { - id: 'dir-owner-1', - baseId: 'kb-1', - groupId: null, - type: 'directory', - data: { - source: tempRoot - }, - status: 'idle', - error: null, - createdAt: '2026-04-08T00:00:00.000Z', - updatedAt: '2026-04-08T00:00:00.000Z' - }, - 'kb-1', - new Set(), - controller.signal - ) + expandDirectoryOwnerToTree(createDirectoryOwner(tempRoot), 'kb-1', 'root', controller.signal) ).rejects.toBe(abortError) }) }) diff --git a/src/main/features/knowledge/utils/sources/__tests__/noteSnapshot.test.ts b/src/main/features/knowledge/pipeline/sources/__tests__/noteSnapshot.test.ts similarity index 95% rename from src/main/features/knowledge/utils/sources/__tests__/noteSnapshot.test.ts rename to src/main/features/knowledge/pipeline/sources/__tests__/noteSnapshot.test.ts index 5c2c145904e..c76cf0629b9 100644 --- a/src/main/features/knowledge/utils/sources/__tests__/noteSnapshot.test.ts +++ b/src/main/features/knowledge/pipeline/sources/__tests__/noteSnapshot.test.ts @@ -1,6 +1,6 @@ import { beforeEach, describe, expect, it, vi } from 'vitest' -import type * as PathStorage from '../../storage/pathStorage' +import type * as PathStorage from '../../../pathStorage' const { writeFileIntoKnowledgeBaseAtMock } = vi.hoisted(() => ({ writeFileIntoKnowledgeBaseAtMock: vi.fn() @@ -19,8 +19,8 @@ vi.mock('@logger', () => ({ // Keep reserveImportedFileRelativePath real (it is pure); only stub the disk write so // the test exercises name derivation + dedupe without touching the filesystem. -vi.mock('../../storage/pathStorage', async () => { - const actual = await vi.importActual('../../storage/pathStorage') +vi.mock('../../../pathStorage', async () => { + const actual = await vi.importActual('../../../pathStorage') return { ...actual, writeFileIntoKnowledgeBaseAt: writeFileIntoKnowledgeBaseAtMock } }) diff --git a/src/main/features/knowledge/utils/sources/__tests__/okfFrontmatter.test.ts b/src/main/features/knowledge/pipeline/sources/__tests__/okfFrontmatter.test.ts similarity index 100% rename from src/main/features/knowledge/utils/sources/__tests__/okfFrontmatter.test.ts rename to src/main/features/knowledge/pipeline/sources/__tests__/okfFrontmatter.test.ts diff --git a/src/main/features/knowledge/utils/sources/__tests__/sourcePlanning.test.ts b/src/main/features/knowledge/pipeline/sources/__tests__/sourcePlanning.test.ts similarity index 100% rename from src/main/features/knowledge/utils/sources/__tests__/sourcePlanning.test.ts rename to src/main/features/knowledge/pipeline/sources/__tests__/sourcePlanning.test.ts diff --git a/src/main/features/knowledge/utils/sources/__tests__/url.test.ts b/src/main/features/knowledge/pipeline/sources/__tests__/url.test.ts similarity index 100% rename from src/main/features/knowledge/utils/sources/__tests__/url.test.ts rename to src/main/features/knowledge/pipeline/sources/__tests__/url.test.ts diff --git a/src/main/features/knowledge/utils/sources/__tests__/urlSnapshot.test.ts b/src/main/features/knowledge/pipeline/sources/__tests__/urlSnapshot.test.ts similarity index 96% rename from src/main/features/knowledge/utils/sources/__tests__/urlSnapshot.test.ts rename to src/main/features/knowledge/pipeline/sources/__tests__/urlSnapshot.test.ts index 33e19616877..79e40f54365 100644 --- a/src/main/features/knowledge/utils/sources/__tests__/urlSnapshot.test.ts +++ b/src/main/features/knowledge/pipeline/sources/__tests__/urlSnapshot.test.ts @@ -1,6 +1,6 @@ import { beforeEach, describe, expect, it, vi } from 'vitest' -import type * as PathStorage from '../../storage/pathStorage' +import type * as PathStorage from '../../../pathStorage' const { writeFileIntoKnowledgeBaseAtMock } = vi.hoisted(() => ({ writeFileIntoKnowledgeBaseAtMock: vi.fn() @@ -19,8 +19,8 @@ vi.mock('@logger', () => ({ // Keep reserveImportedFileRelativePath real (it is pure); only stub the disk write so // the test exercises name derivation + dedupe without touching the filesystem. -vi.mock('../../storage/pathStorage', async () => { - const actual = await vi.importActual('../../storage/pathStorage') +vi.mock('../../../pathStorage', async () => { + const actual = await vi.importActual('../../../pathStorage') return { ...actual, writeFileIntoKnowledgeBaseAt: writeFileIntoKnowledgeBaseAtMock } }) diff --git a/src/main/features/knowledge/utils/sources/directory.ts b/src/main/features/knowledge/pipeline/sources/directory.ts similarity index 72% rename from src/main/features/knowledge/utils/sources/directory.ts rename to src/main/features/knowledge/pipeline/sources/directory.ts index 41cbd739421..f764c749978 100644 --- a/src/main/features/knowledge/utils/sources/directory.ts +++ b/src/main/features/knowledge/pipeline/sources/directory.ts @@ -5,7 +5,7 @@ import { nextFreeKnowledgeRelativePath } from '@main/utils/knowledge' import type { DirectoryItemData, FileItemData, KnowledgeItem } from '@shared/data/types/knowledge' import { knowledgeSupportedFileExts } from '@shared/utils/file' -import { copyFileIntoKnowledgeBaseAt } from '../storage/pathStorage' +import { copyFileIntoKnowledgeBaseAt } from '../../pathStorage' const KNOWLEDGE_SUPPORTED_FILE_EXT_SET = new Set(knowledgeSupportedFileExts) @@ -30,15 +30,6 @@ export type ExpandedDirectoryNode = data: Pick } -/** - * Result of expanding a directory owner: the deduped `raw/` prefix its files were - * stored under (to pin onto the owner's `relativePath`) plus the child tree. - */ -export interface ExpandedDirectoryTree { - pathPrefix: string - children: ExpandedDirectoryNode[] -} - async function readDirectoryTree( dirPath: string, signal: AbortSignal, @@ -110,10 +101,6 @@ async function expandDirectoryNode( } } - if (node.type !== 'folder') { - return null - } - const children: ExpandedDirectoryNode[] = [] for (const child of node.children ?? []) { @@ -136,29 +123,50 @@ async function expandDirectoryNode( } } -export async function expandDirectoryOwnerToTree( - owner: KnowledgeItem, - baseId: string, - reservedTopLevelNames: Set, - signal: AbortSignal -): Promise { +/** + * The deduped top-level `raw/` prefix a directory owner's files will be stored under — + * its own name (e.g. `raw/docs/...`) instead of the opaque owner UUID, so the on-disk + * layout mirrors what the user picked. When that name is already taken under raw/, + * dedupe it with a `_N` suffix (the same strategy file imports use, see + * reserveImportedFileRelativePath). Pure — no I/O — so the caller can pin it onto the + * container's `relativePath` BEFORE any byte is copied, making a mid-expansion crash + * recoverable (the retry reclaims `raw/` from the pinned row). + */ +export function chooseDirectoryPathPrefix(owner: KnowledgeItem, reservedTopLevelNames: Set): string { if (owner.type !== 'directory') { throw new Error(`Knowledge item '${owner.id}' must be type 'directory', received '${owner.type}'`) } // The original folder to scan lives in `source` (shared by every item type). `path` - // was retired in favour of a `relativePath` written back from `pathPrefix` below. + // was retired in favour of a `relativePath` written back from `pathPrefix`. const resolvedPath = path.resolve(owner.data.source) - // Store children under the directory's own name (e.g. `raw/docs/...`) instead of - // the opaque owner UUID, so the on-disk layout mirrors what the user picked. When - // that top-level name is already taken under raw/, dedupe it with a `_N` suffix — - // the same strategy file imports use (see reserveImportedFileRelativePath). - const pathPrefix = nextFreeKnowledgeRelativePath( + return nextFreeKnowledgeRelativePath( path.basename(resolvedPath), (candidate) => !reservedTopLevelNames.has(candidate), false // a directory basename is not a filename — keep any trailing ".ext" intact ) +} +/** + * Scan a directory owner's on-disk tree and durably copy every supported file into + * `raw//...`. The prefix is chosen and pinned by the caller + * (`chooseDirectoryPathPrefix`) before this runs, so a mid-expansion crash leaves the + * container row already pointing at `pathPrefix`; the next attempt's + * `deletePreviousLeafExpansion` reclaims the whole `raw/` shell. This + * function therefore does not clean up on failure — the retry-level reclaimer does, + * and it also survives a hard kill this local cleanup could not. + */ +export async function expandDirectoryOwnerToTree( + owner: KnowledgeItem, + baseId: string, + pathPrefix: string, + signal: AbortSignal +): Promise { + if (owner.type !== 'directory') { + throw new Error(`Knowledge item '${owner.id}' must be type 'directory', received '${owner.type}'`) + } + + const resolvedPath = path.resolve(owner.data.source) const children = await readDirectoryTree(resolvedPath, signal) const expandedChildren: ExpandedDirectoryNode[] = [] @@ -169,5 +177,5 @@ export async function expandDirectoryOwnerToTree( } } - return { pathPrefix, children: expandedChildren } + return expandedChildren } diff --git a/src/main/features/knowledge/utils/sources/noteSnapshot.ts b/src/main/features/knowledge/pipeline/sources/noteSnapshot.ts similarity index 97% rename from src/main/features/knowledge/utils/sources/noteSnapshot.ts rename to src/main/features/knowledge/pipeline/sources/noteSnapshot.ts index 26f225180f8..b794d28fcb3 100644 --- a/src/main/features/knowledge/utils/sources/noteSnapshot.ts +++ b/src/main/features/knowledge/pipeline/sources/noteSnapshot.ts @@ -1,6 +1,6 @@ import { sanitizeFilename } from '@shared/utils/file/filename' -import { reserveImportedFileRelativePath, writeFileIntoKnowledgeBaseAt } from '../storage/pathStorage' +import { reserveImportedFileRelativePath, writeFileIntoKnowledgeBaseAt } from '../../pathStorage' import { serializeOkfFrontmatter } from './okfFrontmatter' const SNAPSHOT_TITLE_MAX = 80 diff --git a/src/main/features/knowledge/utils/sources/okfFrontmatter.ts b/src/main/features/knowledge/pipeline/sources/okfFrontmatter.ts similarity index 100% rename from src/main/features/knowledge/utils/sources/okfFrontmatter.ts rename to src/main/features/knowledge/pipeline/sources/okfFrontmatter.ts diff --git a/src/main/features/knowledge/utils/sources/sourcePlanning.ts b/src/main/features/knowledge/pipeline/sources/sourcePlanning.ts similarity index 98% rename from src/main/features/knowledge/utils/sources/sourcePlanning.ts rename to src/main/features/knowledge/pipeline/sources/sourcePlanning.ts index bdf2f64111c..ef5dc4e6a74 100644 --- a/src/main/features/knowledge/utils/sources/sourcePlanning.ts +++ b/src/main/features/knowledge/pipeline/sources/sourcePlanning.ts @@ -3,7 +3,7 @@ import type { KnowledgeBase } from '@shared/data/types/knowledge' import type { KnowledgeItem } from '@shared/data/types/knowledge' import { knowledgeFileProcessingExts } from '@shared/utils/file' -import { isContainerKnowledgeItem, isIndexableKnowledgeItem } from '../items' +import { isContainerKnowledgeItem, isIndexableKnowledgeItem } from '../../items' const KNOWLEDGE_FILE_PROCESSING_EXT_SET = new Set(knowledgeFileProcessingExts) diff --git a/src/main/features/knowledge/utils/sources/url.ts b/src/main/features/knowledge/pipeline/sources/url.ts similarity index 100% rename from src/main/features/knowledge/utils/sources/url.ts rename to src/main/features/knowledge/pipeline/sources/url.ts diff --git a/src/main/features/knowledge/utils/sources/urlSnapshot.ts b/src/main/features/knowledge/pipeline/sources/urlSnapshot.ts similarity index 98% rename from src/main/features/knowledge/utils/sources/urlSnapshot.ts rename to src/main/features/knowledge/pipeline/sources/urlSnapshot.ts index 114817eb67d..651c45e3f0f 100644 --- a/src/main/features/knowledge/utils/sources/urlSnapshot.ts +++ b/src/main/features/knowledge/pipeline/sources/urlSnapshot.ts @@ -1,6 +1,6 @@ import { sanitizeFilename } from '@shared/utils/file/filename' -import { reserveImportedFileRelativePath, writeFileIntoKnowledgeBaseAt } from '../storage/pathStorage' +import { reserveImportedFileRelativePath, writeFileIntoKnowledgeBaseAt } from '../../pathStorage' import { serializeOkfFrontmatter } from './okfFrontmatter' const SNAPSHOT_TITLE_MAX = 80 diff --git a/src/main/features/knowledge/vectorstore/KnowledgeVectorStoreService.ts b/src/main/features/knowledge/pipeline/vectorstore/KnowledgeVectorStoreService.ts similarity index 89% rename from src/main/features/knowledge/vectorstore/KnowledgeVectorStoreService.ts rename to src/main/features/knowledge/pipeline/vectorstore/KnowledgeVectorStoreService.ts index d1dec47f056..8aed736cc7f 100644 --- a/src/main/features/knowledge/vectorstore/KnowledgeVectorStoreService.ts +++ b/src/main/features/knowledge/pipeline/vectorstore/KnowledgeVectorStoreService.ts @@ -7,8 +7,8 @@ import { DataApiErrorFactory } from '@shared/data/api' import type { CompletedKnowledgeBase, KnowledgeBase } from '@shared/data/types/knowledge' import { isCompletedKnowledgeBase } from '@shared/data/types/knowledge' -import { isIndexableKnowledgeItem } from '../utils/items' -import { deleteKnowledgeBaseDir, getKnowledgeVectorStoreFilePathSync } from '../utils/storage/pathStorage' +import { isIndexableKnowledgeItem } from '../../items' +import { deleteKnowledgeBaseDir, getKnowledgeVectorStoreFilePathSync } from '../../pathStorage' import { openBetterSqlite3IndexDriver } from './indexStore/BetterSqlite3Driver' import { betterSqlite3VectorIndex } from './indexStore/BetterSqlite3VectorIndex' import { ensureIndexMeta, hasAnyMaterial, readIndexSchemaVersion } from './indexStore/indexMeta' @@ -44,13 +44,14 @@ function assertVectorStoreReadyBase(base: KnowledgeBase): asserts base is Comple @ServicePhase(Phase.WhenReady) export class KnowledgeVectorStoreService extends BaseService { // Opening a store (better-sqlite3 connect + schema + meta) is fully synchronous - // (see openIndexStore), so it runs to completion in one JS turn — no concurrent - // getIndexStore call for the same base can ever observe an in-flight open, and a - // failed open never gets cached (the throw happens before .set() below runs). + // (see openIndexStore), so getIndexStore/getIndexStoreIfExists are plain sync + // methods that run to completion in one JS turn — no concurrent call for the + // same base can ever observe an in-flight open, and a failed open never gets + // cached (the throw happens before .set() below runs). private instanceCache = new Map() /** Open (or reuse) the base's index store, ensuring its schema exists. */ - async getIndexStore(base: KnowledgeBase): Promise { + getIndexStore(base: KnowledgeBase): KnowledgeIndexStore { assertVectorStoreReadyBase(base) const cached = this.instanceCache.get(base.id) @@ -66,7 +67,7 @@ export class KnowledgeVectorStoreService extends BaseService { } /** Reuse or open the store only if its file already exists on disk; used by cleanup paths that must not create one. */ - async getIndexStoreIfExists(base: KnowledgeBase): Promise { + getIndexStoreIfExists(base: KnowledgeBase): KnowledgeIndexStore | undefined { // No readiness assert here: cleanup must keep working on failed bases (see // operation-guards.md — deleteItems intentionally skips the guard, so its // delete-subtree job lands here for any base). A failed base never has a @@ -77,7 +78,7 @@ export class KnowledgeVectorStoreService extends BaseService { return cached } - if (!(await this.storeFileExists(base.id))) { + if (!this.storeFileExists(base.id)) { logger.debug('Knowledge index store does not exist on disk', { baseId: base.id }) return undefined } @@ -94,7 +95,7 @@ export class KnowledgeVectorStoreService extends BaseService { const store = this.instanceCache.get(baseId) try { - await this.closeStoreInstance(store) + this.closeStoreInstance(store) await deleteKnowledgeBaseDir(baseId) logger.info('Deleted knowledge index store', { baseId, hadCachedStore: Boolean(store) }) } finally { @@ -109,7 +110,7 @@ export class KnowledgeVectorStoreService extends BaseService { try { for (const [baseId, store] of this.instanceCache.entries()) { try { - await this.closeStoreInstance(store) + this.closeStoreInstance(store) } catch (error) { logger.error('Failed to close knowledge index store', error as Error, { baseId }) } @@ -186,10 +187,9 @@ export class KnowledgeVectorStoreService extends BaseService { } } - private async storeFileExists(baseId: string): Promise { + private storeFileExists(baseId: string): boolean { try { - const stat = await fs.promises.stat(getKnowledgeVectorStoreFilePathSync(baseId)) - return stat.isFile() + return fs.statSync(getKnowledgeVectorStoreFilePathSync(baseId)).isFile() } catch (error) { if ((error as NodeJS.ErrnoException).code === 'ENOENT') { return false @@ -198,10 +198,10 @@ export class KnowledgeVectorStoreService extends BaseService { } } - private async closeStoreInstance(store: KnowledgeIndexStore | undefined): Promise { + private closeStoreInstance(store: KnowledgeIndexStore | undefined): void { if (!store) { return } - await store.close() + store.close() } } diff --git a/src/main/features/knowledge/pipeline/vectorstore/__tests__/KnowledgeVectorStoreService.test.ts b/src/main/features/knowledge/pipeline/vectorstore/__tests__/KnowledgeVectorStoreService.test.ts new file mode 100644 index 00000000000..5aeb1129e81 --- /dev/null +++ b/src/main/features/knowledge/pipeline/vectorstore/__tests__/KnowledgeVectorStoreService.test.ts @@ -0,0 +1,466 @@ +import { mkdtempSync, rmSync, writeFileSync } from 'node:fs' +import { tmpdir } from 'node:os' +import { join } from 'node:path' + +import type * as LifecycleModule from '@main/core/lifecycle' +import { + DEFAULT_KNOWLEDGE_BASE_CHUNK_OVERLAP, + DEFAULT_KNOWLEDGE_BASE_CHUNK_SIZE, + type KnowledgeBase +} from '@shared/data/types/knowledge' +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' + +const { + loggerDebugMock, + loggerErrorMock, + loggerInfoMock, + loggerWarnMock, + getItemsByBaseIdMock, + getPathSyncMock, + deleteDirMock +} = vi.hoisted(() => ({ + loggerDebugMock: vi.fn(), + loggerErrorMock: vi.fn(), + loggerInfoMock: vi.fn(), + loggerWarnMock: vi.fn(), + getItemsByBaseIdMock: vi.fn(), + getPathSyncMock: vi.fn(), + deleteDirMock: vi.fn() +})) + +vi.mock('@main/core/lifecycle', async (importOriginal) => { + const actual = await importOriginal() + + class MockBaseService {} + + return { + ...actual, + BaseService: MockBaseService + } +}) + +vi.mock('@logger', () => ({ + loggerService: { + withContext: () => ({ + debug: loggerDebugMock, + info: loggerInfoMock, + error: loggerErrorMock, + warn: loggerWarnMock + }) + } +})) + +// The app-DB boundary: everything under indexStore/ (driver, schema, meta, store) runs +// for real against a temp-dir SQLite file, so these tests exercise the real DDL/meta +// contract instead of pinning mock call shapes. `pathStorage`'s path resolver is +// redirected into the temp dir; `deleteKnowledgeBaseDir` stays mocked (its own removeDir +// behavior is covered by pathStorage's own tests). +vi.mock('@data/services/KnowledgeItemService', () => ({ + knowledgeItemService: { getItemsByBaseId: getItemsByBaseIdMock } +})) + +vi.mock('../../../pathStorage', () => ({ + getKnowledgeVectorStoreFilePathSync: getPathSyncMock, + deleteKnowledgeBaseDir: deleteDirMock +})) + +const { KnowledgeVectorStoreService } = await import('../KnowledgeVectorStoreService') +const { KnowledgeIndexStore } = await import('../indexStore/KnowledgeIndexStore') +const { BetterSqlite3Driver, openBetterSqlite3IndexDriver } = await import('../indexStore/BetterSqlite3Driver') +const schemaModule = await import('../indexStore/schema') +const indexMetaModule = await import('../indexStore/indexMeta') +const { KNOWLEDGE_INDEX_SCHEMA_VERSION } = schemaModule + +function createBase(id = 'kb-1'): KnowledgeBase { + return { + id, + name: 'KB', + groupId: null, + dimensions: 1024, + embeddingModelId: 'ollama::nomic-embed-text', + status: 'completed', + error: null, + chunkSize: DEFAULT_KNOWLEDGE_BASE_CHUNK_SIZE, + chunkOverlap: DEFAULT_KNOWLEDGE_BASE_CHUNK_OVERLAP, + chunkStrategy: 'structured', + chunkSeparator: '\\n\\n', + searchMode: 'hybrid', + createdAt: '2026-04-08T00:00:00.000Z', + updatedAt: '2026-04-08T00:00:00.000Z' + } +} + +function createFailedBase(): KnowledgeBase { + return { + ...createBase(), + dimensions: null, + embeddingModelId: null, + status: 'failed', + error: 'missing_embedding_model' + } +} + +/** Opens (and immediately closes) a store for `base` so its file exists on disk at the current schema version. */ +function primeStoreOnDisk(base: KnowledgeBase): void { + const store = new KnowledgeVectorStoreService().getIndexStore(base) + store.close() +} + +/** Opens a fresh raw driver directly on `base`'s file (bypassing the service/cache) to inspect or mutate it. */ +function withRawDriver(base: KnowledgeBase, fn: (driver: InstanceType) => T): T { + const driver = openBetterSqlite3IndexDriver(getPathSyncMock(base.id)) + try { + return fn(driver) + } finally { + driver.close() + } +} + +describe('KnowledgeVectorStoreService', () => { + let tempDir: string + + beforeEach(() => { + vi.clearAllMocks() + tempDir = mkdtempSync(join(tmpdir(), 'cs-knowledge-vsstore-')) + getPathSyncMock.mockImplementation((baseId: string) => join(tempDir, `${baseId}.sqlite`)) + getItemsByBaseIdMock.mockReturnValue([]) + deleteDirMock.mockResolvedValue(undefined) + }) + + afterEach(() => { + vi.restoreAllMocks() + rmSync(tempDir, { recursive: true, force: true }) + }) + + it('opens an index store on first request and caches it per base', () => { + const service = new KnowledgeVectorStoreService() + const base = createBase() + + const first = service.getIndexStore(base) + const second = service.getIndexStore(base) + + expect(first).toBe(second) + expect(first).toBeInstanceOf(KnowledgeIndexStore) + expect(loggerInfoMock).toHaveBeenCalledWith('Opened knowledge index store', { baseId: base.id, cacheSize: 1 }) + expect(loggerDebugMock).toHaveBeenCalledWith('Reusing cached knowledge index store', { baseId: base.id }) + }) + + it('synchronous open makes a second call observe the already-cached store', () => { + // Opening a store (better-sqlite3 connect + schema + meta) is fully synchronous, so + // the first call's cache write completes before the second call runs — there is no + // in-flight state the second call could observe mid-open. Pinned via a call-count + // assertion so a future refactor that reintroduces an async open cannot silently + // double-open. + const service = new KnowledgeVectorStoreService() + const base = createBase() + const createSchemaSpy = vi.spyOn(schemaModule, 'createKnowledgeIndexSchema') + + const first = service.getIndexStore(base) + const second = service.getIndexStore(base) + + expect(first).toBe(second) + expect(createSchemaSpy).toHaveBeenCalledTimes(1) + }) + + it('evicts a failed open so a later call retries instead of re-throwing the failure', () => { + const service = new KnowledgeVectorStoreService() + const base = createBase() + // A regular file where the index dir must go makes the real mkdirSync in + // openBetterSqlite3IndexDriver fail with ENOTDIR before any Database is opened. + const blockerPath = join(tempDir, 'blocked-open') + writeFileSync(blockerPath, 'not a directory') + getPathSyncMock.mockReturnValueOnce(join(blockerPath, 'index.sqlite')) + + expect(() => service.getIndexStore(base)).toThrow() + + const store = service.getIndexStore(base) + expect(store).toBeInstanceOf(KnowledgeIndexStore) + }) + + it('stamps and verifies the meta identity row before handing out the store', () => { + const service = new KnowledgeVectorStoreService() + const base = createBase() + const ensureIndexMetaSpy = vi.spyOn(indexMetaModule, 'ensureIndexMeta') + + service.getIndexStore(base) + + expect(ensureIndexMetaSpy).toHaveBeenCalledTimes(1) + expect(ensureIndexMetaSpy).toHaveBeenCalledWith(expect.anything(), { baseId: base.id }) + }) + + it('creates the schema normally when the stored version matches (no rebuild)', () => { + const base = createBase() + primeStoreOnDisk(base) + const createSchemaSpy = vi.spyOn(schemaModule, 'createKnowledgeIndexSchema') + const resetSchemaSpy = vi.spyOn(schemaModule, 'resetKnowledgeIndexSchema') + const service = new KnowledgeVectorStoreService() + + service.getIndexStore(base) + + expect(createSchemaSpy).toHaveBeenCalledTimes(1) + expect(resetSchemaSpy).not.toHaveBeenCalled() + }) + + it('rebuilds the derived index when an existing index.sqlite is at a stale schema version', () => { + const base = createBase() + primeStoreOnDisk(base) + // Leave a marker row and stamp a stale version directly, so the rebuild below is + // verified against real DDL (dropped tables), not a mocked branch decision. + withRawDriver(base, (driver) => { + driver.execute('INSERT INTO content (content_hash, text, created_at) VALUES (?, ?, ?)', ['marker', 'x', 0]) + driver.execute('UPDATE meta SET schema_version = ?', [KNOWLEDGE_INDEX_SCHEMA_VERSION - 1]) + }) + const createSchemaSpy = vi.spyOn(schemaModule, 'createKnowledgeIndexSchema') + const resetSchemaSpy = vi.spyOn(schemaModule, 'resetKnowledgeIndexSchema') + const ensureIndexMetaSpy = vi.spyOn(indexMetaModule, 'ensureIndexMeta') + const service = new KnowledgeVectorStoreService() + + service.getIndexStore(base) + + // Drop+recreate via resetKnowledgeIndexSchema, NOT the plain create — an old layout + // (e.g. pre-fts_rowid) cannot be retrofitted by CREATE ... IF NOT EXISTS. + expect(resetSchemaSpy).toHaveBeenCalledTimes(1) + expect(createSchemaSpy).not.toHaveBeenCalled() + // Warn so the wipe-and-reindex is diagnosable. + expect(loggerWarnMock).toHaveBeenCalledWith( + 'Knowledge index schema version mismatch — rebuilding the derived index', + expect.objectContaining({ + baseId: base.id, + storedVersion: KNOWLEDGE_INDEX_SCHEMA_VERSION - 1, + expectedVersion: KNOWLEDGE_INDEX_SCHEMA_VERSION + }) + ) + // The reset drops meta, so the open path still restamps it afterwards. + expect(ensureIndexMetaSpy).toHaveBeenCalledTimes(1) + // The rebuild really dropped and recreated the schema: the marker inserted before + // the stale-version stamp is gone, and the restamped version is current. + expect( + withRawDriver(base, (driver) => driver.execute('SELECT 1 FROM content WHERE content_hash = ?', ['marker']).rows) + ).toEqual([]) + expect(withRawDriver(base, (driver) => indexMetaModule.readIndexSchemaVersion(driver))).toBe( + KNOWLEDGE_INDEX_SCHEMA_VERSION + ) + }) + + it('also rebuilds when the stored version is NEWER than this build (downgrade)', () => { + // The guard is `!==`, not `<` — a file written by a newer build (user downgraded the app) is + // just as incompatible a layout and must be rebuilt, not opened as-is. Pins that semantics so a + // future refactor to `<` does not silently start mounting newer files. + const base = createBase() + primeStoreOnDisk(base) + withRawDriver(base, (driver) => { + driver.execute('UPDATE meta SET schema_version = ?', [KNOWLEDGE_INDEX_SCHEMA_VERSION + 1]) + }) + const createSchemaSpy = vi.spyOn(schemaModule, 'createKnowledgeIndexSchema') + const resetSchemaSpy = vi.spyOn(schemaModule, 'resetKnowledgeIndexSchema') + const service = new KnowledgeVectorStoreService() + + service.getIndexStore(base) + + expect(resetSchemaSpy).toHaveBeenCalledTimes(1) + expect(createSchemaSpy).not.toHaveBeenCalled() + }) + + it('closes the driver and aborts the open when meta verification fails (wrong/corrupt base)', () => { + const base = createBase('kb-shared') + const otherBase = createBase('kb-other') + // Both bases resolve to the same on-disk file, so the second open's meta + // identity check genuinely finds a mismatched base_id. + getPathSyncMock.mockImplementation(() => join(tempDir, 'shared.sqlite')) + primeStoreOnDisk(base) + const closeSpy = vi.spyOn(BetterSqlite3Driver.prototype, 'close') + const service = new KnowledgeVectorStoreService() + + expect(() => service.getIndexStore(otherBase)).toThrow('belongs to a different base') + + expect(closeSpy).toHaveBeenCalledTimes(1) + }) + + it('closes the driver when schema creation fails so the file handle is not leaked', () => { + const service = new KnowledgeVectorStoreService() + const base = createBase() + const closeSpy = vi.spyOn(BetterSqlite3Driver.prototype, 'close') + vi.spyOn(schemaModule, 'createKnowledgeIndexSchema').mockImplementationOnce(() => { + throw new Error('disk full') + }) + + expect(() => service.getIndexStore(base)).toThrow('disk full') + + expect(closeSpy).toHaveBeenCalledTimes(1) + }) + + it('returns undefined from getIndexStoreIfExists when no backing file exists', () => { + const service = new KnowledgeVectorStoreService() + const base = createBase() + + expect(service.getIndexStoreIfExists(base)).toBeUndefined() + + expect(loggerDebugMock).toHaveBeenCalledWith('Knowledge index store does not exist on disk', { baseId: base.id }) + }) + + it('opens an existing store from disk when getIndexStoreIfExists detects a backing file', () => { + const base = createBase() + primeStoreOnDisk(base) + const service = new KnowledgeVectorStoreService() + + const store = service.getIndexStoreIfExists(base) + + expect(store).toBeInstanceOf(KnowledgeIndexStore) + }) + + it('returns the cached store from getIndexStoreIfExists without probing disk', () => { + const service = new KnowledgeVectorStoreService() + const base = createBase() + const created = service.getIndexStore(base) + getPathSyncMock.mockClear() + + expect(service.getIndexStoreIfExists(base)).toBe(created) + // storeFileExists resolves the store path before stat-ing it, so an + // untouched path resolver proves the disk probe never ran. + expect(getPathSyncMock).not.toHaveBeenCalled() + }) + + it('closes the cached store and removes the base directory on deleteStore', async () => { + const service = new KnowledgeVectorStoreService() + const base = createBase() + const closeSpy = vi.spyOn(BetterSqlite3Driver.prototype, 'close') + + const store = service.getIndexStore(base) + await service.deleteStore(base.id) + + expect(closeSpy).toHaveBeenCalledTimes(1) + expect(deleteDirMock).toHaveBeenCalledWith(base.id) + // Close must precede directory removal — on Windows a still-open sqlite + // handle makes the directory deletion fail. + expect(closeSpy.mock.invocationCallOrder[0]).toBeLessThan(deleteDirMock.mock.invocationCallOrder[0]) + + // Cache was evicted: the next open builds a fresh instance. + const reopened = service.getIndexStore(base) + expect(reopened).not.toBe(store) + }) + + it('deleteStore removes the directory even when no store was ever opened for the base', async () => { + // Opening a store (see openIndexStore) is fully synchronous — it either completes and + // caches a store, or throws before caching anything. There is no in-flight state + // deleteStore could observe mid-open, so this covers the "nothing cached" case: + // deleteStore must still close-if-present (a no-op here) and remove the directory. + const service = new KnowledgeVectorStoreService() + const base = createBase() + + await service.deleteStore(base.id) + + expect(deleteDirMock).toHaveBeenCalledWith(base.id) + }) + + it('evicts the cached store even when directory removal fails', async () => { + const service = new KnowledgeVectorStoreService() + const base = createBase() + deleteDirMock.mockRejectedValueOnce(new Error('delete failed')) + + const store = service.getIndexStore(base) + await expect(service.deleteStore(base.id)).rejects.toThrow('delete failed') + + const reopened = service.getIndexStore(base) + expect(reopened).not.toBe(store) + }) + + it('closes all cached stores during stop and continues when one close throws', async () => { + const service = new KnowledgeVectorStoreService() + + const first = service.getIndexStore(createBase('kb-1')) + const second = service.getIndexStore(createBase('kb-2')) + const closeError = new Error('close failed') + const firstCloseSpy = vi.spyOn(first, 'close').mockImplementationOnce(() => { + throw closeError + }) + const secondCloseSpy = vi.spyOn(second, 'close') + + await expect((service as unknown as { onStop: () => Promise }).onStop()).resolves.toBeUndefined() + + expect(firstCloseSpy).toHaveBeenCalledTimes(1) + expect(secondCloseSpy).toHaveBeenCalledTimes(1) + expect(loggerErrorMock).toHaveBeenCalledWith('Failed to close knowledge index store', closeError, { + baseId: 'kb-1' + }) + expect(loggerInfoMock).toHaveBeenCalledWith('Stopping knowledge index stores', { storeCount: 2 }) + expect(loggerInfoMock).toHaveBeenCalledWith('Stopped knowledge index stores', { storeCount: 2 }) + + // Cache cleared: reopening kb-2 builds a fresh instance. + const reopened = service.getIndexStore(createBase('kb-2')) + expect(reopened).not.toBe(second) + }) + + it('rejects bases that are not ready before touching disk', () => { + const service = new KnowledgeVectorStoreService() + const base = createFailedBase() + + expect(() => service.getIndexStore(base)).toThrow('not ready for vector store operations') + + // The readiness assert throws before openIndexStore ever resolves the store path. + expect(getPathSyncMock).not.toHaveBeenCalled() + }) + + it('lets cleanup on a failed base proceed: getIndexStoreIfExists returns undefined instead of asserting', () => { + // Failed bases never get a store file (the vector migrator skips them and + // getIndexStore asserts), so the existence probe is the path cleanup takes. + const service = new KnowledgeVectorStoreService() + const base = createFailedBase() + + expect(service.getIndexStoreIfExists(base)).toBeUndefined() + }) + + it('still asserts readiness when a failed base unexpectedly has a store file on disk', () => { + const service = new KnowledgeVectorStoreService() + const base = createFailedBase() + writeFileSync(getPathSyncMock(base.id), '') + + expect(() => service.getIndexStoreIfExists(base)).toThrow('not ready for vector store operations') + }) + + it('logs an error when an empty index mounts under a base with completed items', () => { + const service = new KnowledgeVectorStoreService() + const base = createBase() + getItemsByBaseIdMock.mockReturnValueOnce([ + { id: 'item-1', type: 'directory', status: 'completed' }, + { id: 'item-2', type: 'file', status: 'completed' } + ]) + + const store = service.getIndexStore(base) + + expect(store).toBeInstanceOf(KnowledgeIndexStore) + expect(loggerErrorMock).toHaveBeenCalledWith( + expect.stringContaining('zero materials while the base has completed items'), + expect.objectContaining({ baseId: base.id }) + ) + }) + + it('stays quiet when an empty index mounts under a base with no completed indexable items', () => { + const service = new KnowledgeVectorStoreService() + const base = createBase() + // A completed empty directory is legitimate without materials; in-flight leaves are too. + getItemsByBaseIdMock.mockReturnValueOnce([ + { id: 'item-1', type: 'directory', status: 'completed' }, + { id: 'item-2', type: 'file', status: 'processing' } + ]) + + service.getIndexStore(base) + + expect(loggerErrorMock).not.toHaveBeenCalled() + }) + + it('fails the open and closes the driver when the empty-index diagnostic cannot read the base items', () => { + const service = new KnowledgeVectorStoreService() + const base = createBase() + const closeSpy = vi.spyOn(BetterSqlite3Driver.prototype, 'close') + getItemsByBaseIdMock.mockImplementationOnce(() => { + throw new Error('app database unavailable') + }) + + // Deliberate fail-loud: swallowing the lookup failure would re-silence the + // deleted-base race (open racing deleteBase recreates an empty file, and the + // lookup's NOT_FOUND is what makes that loud instead of caching an empty store). + expect(() => service.getIndexStore(base)).toThrow('app database unavailable') + + expect(closeSpy).toHaveBeenCalledTimes(1) + }) +}) diff --git a/src/main/features/knowledge/utils/cleanup/__tests__/vectorCleanup.test.ts b/src/main/features/knowledge/pipeline/vectorstore/__tests__/vectorCleanup.test.ts similarity index 87% rename from src/main/features/knowledge/utils/cleanup/__tests__/vectorCleanup.test.ts rename to src/main/features/knowledge/pipeline/vectorstore/__tests__/vectorCleanup.test.ts index 929bc1aec1d..914cde626c9 100644 --- a/src/main/features/knowledge/utils/cleanup/__tests__/vectorCleanup.test.ts +++ b/src/main/features/knowledge/pipeline/vectorstore/__tests__/vectorCleanup.test.ts @@ -60,16 +60,16 @@ function createBase(): KnowledgeBase { describe('deleteKnowledgeItemVectors', () => { beforeEach(() => { vi.clearAllMocks() - getIndexStoreIfExistsMock.mockResolvedValue({ + getIndexStoreIfExistsMock.mockReturnValue({ deleteMaterials: deleteMaterialsMock, reclaimSpace: reclaimSpaceMock }) deleteMaterialsMock.mockResolvedValue(undefined) - reclaimSpaceMock.mockResolvedValue({ vacuumed: false, reclaimedBytes: 0 }) + reclaimSpaceMock.mockReturnValue({ vacuumed: false, reclaimedBytes: 0 }) }) it('skips cleanup when no vector store exists', async () => { - getIndexStoreIfExistsMock.mockResolvedValueOnce(undefined) + getIndexStoreIfExistsMock.mockReturnValueOnce(undefined) await deleteKnowledgeItemVectors(createBase(), ['note-1']) @@ -97,15 +97,15 @@ describe('deleteKnowledgeItemVectors', () => { describe('reclaimKnowledgeIndexSpace', () => { beforeEach(() => { vi.clearAllMocks() - getIndexStoreIfExistsMock.mockResolvedValue({ + getIndexStoreIfExistsMock.mockReturnValue({ deleteMaterials: deleteMaterialsMock, reclaimSpace: reclaimSpaceMock }) - reclaimSpaceMock.mockResolvedValue({ vacuumed: true, reclaimedBytes: 1024 }) + reclaimSpaceMock.mockReturnValue({ vacuumed: true, reclaimedBytes: 1024 }) }) it('skips reclaim when no vector store exists', async () => { - getIndexStoreIfExistsMock.mockResolvedValueOnce(undefined) + getIndexStoreIfExistsMock.mockReturnValueOnce(undefined) await reclaimKnowledgeIndexSpace(createBase()) @@ -121,7 +121,9 @@ describe('reclaimKnowledgeIndexSpace', () => { it('never throws when reclaim fails — the delete already succeeded', async () => { // Best-effort: a transient reclaim failure must not fail the delete job whose rows // and vectors are already gone; the freed pages just wait for a later index to reuse. - reclaimSpaceMock.mockRejectedValueOnce(new Error('database is locked')) + reclaimSpaceMock.mockImplementationOnce(() => { + throw new Error('database is locked') + }) await expect(reclaimKnowledgeIndexSpace(createBase())).resolves.toBeUndefined() // A transient failure stays a warn — not the corruption-class error path. @@ -133,9 +135,9 @@ describe('reclaimKnowledgeIndexSpace', () => { // reclaim's whole-file checkpoint/optimize/VACUUM is often the first op to touch the full file // post-delete, so SQLITE_CORRUPT/SQLITE_NOTADB surfaces here — it must be loud, not buried in the // benign "failed to reclaim" warn, yet still swallowed (the delete already succeeded). - reclaimSpaceMock.mockRejectedValueOnce( - Object.assign(new Error('database disk image is malformed'), { code: 'SQLITE_CORRUPT' }) - ) + reclaimSpaceMock.mockImplementationOnce(() => { + throw Object.assign(new Error('database disk image is malformed'), { code: 'SQLITE_CORRUPT' }) + }) await expect(reclaimKnowledgeIndexSpace(createBase())).resolves.toBeUndefined() expect(loggerErrorMock).toHaveBeenCalledWith( @@ -150,7 +152,9 @@ describe('reclaimKnowledgeIndexSpace', () => { // The open itself can throw (corrupt index, readiness/base_id mismatch, schema open failure). // It is inside the same best-effort try, so an open failure must not fail the delete job whose // rows and vectors are already gone. - getIndexStoreIfExistsMock.mockRejectedValueOnce(new Error('index store failed to open')) + getIndexStoreIfExistsMock.mockImplementationOnce(() => { + throw new Error('index store failed to open') + }) await expect(reclaimKnowledgeIndexSpace(createBase())).resolves.toBeUndefined() expect(reclaimSpaceMock).not.toHaveBeenCalled() diff --git a/src/main/features/knowledge/vectorstore/indexStore/BetterSqlite3Driver.ts b/src/main/features/knowledge/pipeline/vectorstore/indexStore/BetterSqlite3Driver.ts similarity index 96% rename from src/main/features/knowledge/vectorstore/indexStore/BetterSqlite3Driver.ts rename to src/main/features/knowledge/pipeline/vectorstore/indexStore/BetterSqlite3Driver.ts index 840e9491073..ec99188ad4f 100644 --- a/src/main/features/knowledge/vectorstore/indexStore/BetterSqlite3Driver.ts +++ b/src/main/features/knowledge/pipeline/vectorstore/indexStore/BetterSqlite3Driver.ts @@ -39,7 +39,7 @@ function toBindable(value: SqlValue): Bindable { * SqliteDriver backed by a synchronous better-sqlite3 connection (the only engine). * * No internal write mutex: every writer (rebuildMaterial / deleteMaterials / - * reclaimSpace) runs under `KnowledgeLockManager.withBaseMutationLock(baseId)`, a + * reclaimSpace) runs under `KeyedMutex.runExclusive(baseId, ...)`, a * per-base async-mutex that already serializes all writes to a base's index — so the * driver never self-serializes. better-sqlite3's single connection makes * `transaction` inherently atomic; reads run on the same connection and ride WAL / @@ -58,12 +58,17 @@ function toBindable(value: SqlValue): Bindable { */ export class BetterSqlite3Driver implements SqliteDriver { private closed = false + private readonly stmtCache = new Map() constructor(private readonly db: Database.Database) {} execute(sql: string, args: SqlValue[] = []): SqlQueryResult { this.assertOpen() - const stmt = this.db.prepare(sql) + let stmt = this.stmtCache.get(sql) + if (!stmt) { + stmt = this.db.prepare(sql) + this.stmtCache.set(sql, stmt) + } const bound = args.map(toBindable) // `reader` is true for statements that yield rows (SELECT, row-returning PRAGMA). // run() throws on those and all() throws on non-row statements, so split on it. diff --git a/src/main/features/knowledge/vectorstore/indexStore/BetterSqlite3VectorIndex.ts b/src/main/features/knowledge/pipeline/vectorstore/indexStore/BetterSqlite3VectorIndex.ts similarity index 100% rename from src/main/features/knowledge/vectorstore/indexStore/BetterSqlite3VectorIndex.ts rename to src/main/features/knowledge/pipeline/vectorstore/indexStore/BetterSqlite3VectorIndex.ts diff --git a/src/main/features/knowledge/vectorstore/indexStore/KnowledgeIndexStore.ts b/src/main/features/knowledge/pipeline/vectorstore/indexStore/KnowledgeIndexStore.ts similarity index 95% rename from src/main/features/knowledge/vectorstore/indexStore/KnowledgeIndexStore.ts rename to src/main/features/knowledge/pipeline/vectorstore/indexStore/KnowledgeIndexStore.ts index 9f664ba1adb..00621ea5ead 100644 --- a/src/main/features/knowledge/vectorstore/indexStore/KnowledgeIndexStore.ts +++ b/src/main/features/knowledge/pipeline/vectorstore/indexStore/KnowledgeIndexStore.ts @@ -7,7 +7,14 @@ import type { KnowledgeSearchUnit, RebuildMaterialInput } from './model' -import type { SqliteDriver, SqliteReclaimOutcome, SqliteTransaction, SqlValue, VectorIndex } from './types' +import type { + SqliteDriver, + SqliteExecutor, + SqliteReclaimOutcome, + SqliteTransaction, + SqlValue, + VectorIndex +} from './types' import { encodeVectorBlob } from './vectorBlob' /** RRF constant (1-indexed rank), matching the legacy hybrid fusion. */ @@ -45,7 +52,7 @@ export class KnowledgeIndexStore { * transaction so a crash or error can never leave old and new units mixed, and * an insert failure rolls back without destroying the prior index (§5.2). */ - async rebuildMaterial(materialId: string, input: RebuildMaterialInput): Promise { + rebuildMaterial(materialId: string, input: RebuildMaterialInput): void { // usesEmbeddings: false means a BM25-only rebuild — step 6 below writes whatever // `embeddings` holds unconditionally (only the step 6b coverage check is gated on // the flag), so a caller bug that sets both would silently write orphan vectors @@ -291,25 +298,12 @@ export class KnowledgeIndexStore { * re-reads (the hash is now absent) and re-embeds it. A stale "present" therefore * self-corrects rather than leaving a unit silently absent from vector search. */ - async listExistingEmbeddingHashes(hashes: string[]): Promise> { - const existing = new Set() - // Chunk to stay well under SQLite's bound-parameter limit for large materials. - for (let i = 0; i < hashes.length; i += EMBEDDING_HASH_QUERY_BATCH) { - const batch = hashes.slice(i, i + EMBEDDING_HASH_QUERY_BATCH) - const placeholders = batch.map(() => '?').join(', ') - const result = this.driver.execute( - `SELECT embedding_text_hash FROM embedding WHERE embedding_text_hash IN (${placeholders})`, - batch - ) - for (const row of result.rows) { - existing.add(row.embedding_text_hash as string) - } - } - return existing + listExistingEmbeddingHashes(hashes: string[]): Set { + return this.selectExistingEmbeddingHashes(this.driver, hashes) } /** Read back a material's units (with body text), ordered by unit index. */ - async listMaterialUnits(materialId: string): Promise { + listMaterialUnits(materialId: string): KnowledgeSearchUnit[] { const result = this.driver.execute( `SELECT su.unit_id, su.material_id, su.unit_type, su.unit_index, su.title, su.char_start, su.char_end, st.text AS body FROM search_unit su @@ -347,7 +341,7 @@ export class KnowledgeIndexStore { * index, so this is the lookup behind the deep-read tools — they re-validate * the resolved material against the visible knowledge_item before reading. */ - async getMaterialByRelativePath(relativePath: string): Promise { + getMaterialByRelativePath(relativePath: string): KnowledgeMaterialRef | null { const result = this.driver.execute(`SELECT material_id, relative_path FROM material WHERE relative_path = ?`, [ relativePath ]) @@ -368,7 +362,7 @@ export class KnowledgeIndexStore { * were sliced from, so `text.slice(unit.charStart, unit.charEnd) === unit.text` * holds (the same invariant rebuildMaterial enforces at write time). */ - async readMaterialContent(materialId: string): Promise { + readMaterialContent(materialId: string): string | null { const result = this.driver.execute( `SELECT c.text AS text FROM material m @@ -390,7 +384,7 @@ export class KnowledgeIndexStore { * text of a unit is the search source for both lanes * (knowledge-technical-design.md §6). */ - async search(input: KnowledgeIndexSearchInput): Promise { + search(input: KnowledgeIndexSearchInput): KnowledgeIndexSearchMatch[] { if (input.mode === 'bm25') { return this.bm25Search(input.queryText, input.topK) } @@ -420,11 +414,11 @@ export class KnowledgeIndexStore { * whole-index segment merge — and the FTS table name lives here, with the schema, not * in the engine-neutral driver. */ - async reclaimSpace(): Promise { + reclaimSpace(): SqliteReclaimOutcome { return this.driver.reclaim([`INSERT INTO search_text_fts(search_text_fts) VALUES('optimize')`]) } - async close(): Promise { + close(): void { this.driver.close() } @@ -514,20 +508,27 @@ export class KnowledgeIndexStore { return result.rows.map((row) => toMatch(row, -Number(row.len))) } - /** Throw (rolling back the surrounding rebuild) if any unit hash has no embedding row. */ - private assertEmbeddingCoverage(tx: SqliteTransaction, materialId: string, hashes: string[]): void { - const missing = new Set(hashes) + /** Of the given embedding-text hashes, return those already stored — chunked to stay well under SQLite's bound-parameter limit. */ + private selectExistingEmbeddingHashes(executor: SqliteExecutor, hashes: string[]): Set { + const existing = new Set() for (let i = 0; i < hashes.length; i += EMBEDDING_HASH_QUERY_BATCH) { const batch = hashes.slice(i, i + EMBEDDING_HASH_QUERY_BATCH) const placeholders = batch.map(() => '?').join(', ') - const result = tx.execute( + const result = executor.execute( `SELECT embedding_text_hash FROM embedding WHERE embedding_text_hash IN (${placeholders})`, batch ) for (const row of result.rows) { - missing.delete(row.embedding_text_hash as string) + existing.add(row.embedding_text_hash as string) } } + return existing + } + + /** Throw (rolling back the surrounding rebuild) if any unit hash has no embedding row. */ + private assertEmbeddingCoverage(tx: SqliteTransaction, materialId: string, hashes: string[]): void { + const present = this.selectExistingEmbeddingHashes(tx, hashes) + const missing = new Set(hashes.filter((hash) => !present.has(hash))) if (missing.size > 0) { throw new Error( `Knowledge index rebuild for material ${materialId} left ${missing.size} unit embedding hash(es) without a vector (first: ${[...missing][0]})` diff --git a/src/main/features/knowledge/vectorstore/indexStore/__tests__/BetterSqlite3Driver.test.ts b/src/main/features/knowledge/pipeline/vectorstore/indexStore/__tests__/BetterSqlite3Driver.test.ts similarity index 100% rename from src/main/features/knowledge/vectorstore/indexStore/__tests__/BetterSqlite3Driver.test.ts rename to src/main/features/knowledge/pipeline/vectorstore/indexStore/__tests__/BetterSqlite3Driver.test.ts diff --git a/src/main/features/knowledge/vectorstore/indexStore/__tests__/BetterSqlite3VectorIndex.test.ts b/src/main/features/knowledge/pipeline/vectorstore/indexStore/__tests__/BetterSqlite3VectorIndex.test.ts similarity index 100% rename from src/main/features/knowledge/vectorstore/indexStore/__tests__/BetterSqlite3VectorIndex.test.ts rename to src/main/features/knowledge/pipeline/vectorstore/indexStore/__tests__/BetterSqlite3VectorIndex.test.ts diff --git a/src/main/features/knowledge/vectorstore/indexStore/__tests__/KnowledgeIndexStore.integration.test.ts b/src/main/features/knowledge/pipeline/vectorstore/indexStore/__tests__/KnowledgeIndexStore.integration.test.ts similarity index 88% rename from src/main/features/knowledge/vectorstore/indexStore/__tests__/KnowledgeIndexStore.integration.test.ts rename to src/main/features/knowledge/pipeline/vectorstore/indexStore/__tests__/KnowledgeIndexStore.integration.test.ts index d3a61212352..4c57e7fd51e 100644 --- a/src/main/features/knowledge/vectorstore/indexStore/__tests__/KnowledgeIndexStore.integration.test.ts +++ b/src/main/features/knowledge/pipeline/vectorstore/indexStore/__tests__/KnowledgeIndexStore.integration.test.ts @@ -4,7 +4,7 @@ import { join } from 'node:path' import { afterEach, beforeEach, describe, expect, it } from 'vitest' -import { splitTextWithOffsets } from '../../../utils/indexing/splitter' +import { splitTextWithOffsets } from '../../../indexing/splitter' import { openBetterSqlite3IndexDriver } from '../BetterSqlite3Driver' import { betterSqlite3VectorIndex } from '../BetterSqlite3VectorIndex' import { hashEmbeddingText } from '../hashing' @@ -32,7 +32,7 @@ describe('KnowledgeIndexStore integration (real better-sqlite3)', () => { }) afterEach(async () => { - await store.close() + store.close() rmSync(tempDir, { recursive: true, force: true }) }) @@ -63,9 +63,9 @@ describe('KnowledgeIndexStore integration (real better-sqlite3)', () => { ) } - await store.rebuildMaterial('m1', input) + store.rebuildMaterial('m1', input) - const units = await store.listMaterialUnits('m1') + const units = store.listMaterialUnits('m1') expect(units).toHaveLength(chunks.length) units.forEach((unit, index) => { // §5.3: a unit's stored body is the exact content slice, never a rewritten copy. @@ -74,15 +74,15 @@ describe('KnowledgeIndexStore integration (real better-sqlite3)', () => { expect(unit.unitIndex).toBe(index) }) - const bm25Hits = await store.search({ queryText: 'embedding', mode: 'bm25', topK: 5 }) + const bm25Hits = store.search({ queryText: 'embedding', mode: 'bm25', topK: 5 }) expect(bm25Hits.length).toBeGreaterThan(0) expect(bm25Hits.every((hit) => hit.materialId === 'm1')).toBe(true) - const vectorHits = await store.search({ queryText: '', queryEmbedding: [1, 1, 0], mode: 'vector', topK: 5 }) + const vectorHits = store.search({ queryText: '', queryEmbedding: [1, 1, 0], mode: 'vector', topK: 5 }) expect(vectorHits.length).toBeGreaterThan(0) expect(vectorHits.every((hit) => hit.materialId === 'm1')).toBe(true) - const hybridHits = await store.search({ + const hybridHits = store.search({ queryText: 'embedding', queryEmbedding: [1, 1, 0], mode: 'hybrid', diff --git a/src/main/features/knowledge/vectorstore/indexStore/__tests__/KnowledgeIndexStore.rrf.test.ts b/src/main/features/knowledge/pipeline/vectorstore/indexStore/__tests__/KnowledgeIndexStore.rrf.test.ts similarity index 90% rename from src/main/features/knowledge/vectorstore/indexStore/__tests__/KnowledgeIndexStore.rrf.test.ts rename to src/main/features/knowledge/pipeline/vectorstore/indexStore/__tests__/KnowledgeIndexStore.rrf.test.ts index f097484fd0f..d4a2308668a 100644 --- a/src/main/features/knowledge/vectorstore/indexStore/__tests__/KnowledgeIndexStore.rrf.test.ts +++ b/src/main/features/knowledge/pipeline/vectorstore/indexStore/__tests__/KnowledgeIndexStore.rrf.test.ts @@ -59,7 +59,7 @@ describe('KnowledgeIndexStore hybrid RRF fusion', () => { const { driver, limits } = createFakeDriver([], []) const store = new KnowledgeIndexStore(driver, vectorIndex) - await store.search({ queryText: 'query', queryEmbedding: [1, 0], mode: 'hybrid', topK: 4 }) + store.search({ queryText: 'query', queryEmbedding: [1, 0], mode: 'hybrid', topK: 4 }) expect(limits.vector).toBe(20) expect(limits.bm25).toBe(20) @@ -74,7 +74,7 @@ describe('KnowledgeIndexStore hybrid RRF fusion', () => { const store = new KnowledgeIndexStore(driver, vectorIndex) const alpha = 0.75 - const results = await store.search({ queryText: 'query', queryEmbedding: [1, 0], mode: 'hybrid', topK: 10, alpha }) + const results = store.search({ queryText: 'query', queryEmbedding: [1, 0], mode: 'hybrid', topK: 10, alpha }) // score = alpha/(RRF_K + vRank + 1) + (1 - alpha)/(RRF_K + bRank + 1), ranks 0-indexed. const scoreA = alpha / (RRF_K + 1) + (1 - alpha) / (RRF_K + 2) @@ -94,7 +94,7 @@ describe('KnowledgeIndexStore hybrid RRF fusion', () => { const { driver } = createFakeDriver([{ unit_id: 'A', material_id: 'mA', unit_index: 0, body: null, dist: 0 }], []) const store = new KnowledgeIndexStore(driver, vectorIndex) - await expect(store.search({ queryText: '', queryEmbedding: [1, 0], mode: 'vector', topK: 5 })).rejects.toThrow( + expect(() => store.search({ queryText: '', queryEmbedding: [1, 0], mode: 'vector', topK: 5 })).toThrow( 'missing the body text for unit A' ) }) @@ -107,7 +107,7 @@ describe('KnowledgeIndexStore hybrid RRF fusion', () => { ) const store = new KnowledgeIndexStore(driver, vectorIndex) - const results = await store.search({ queryText: 'query', queryEmbedding: [1, 0], mode: 'hybrid', topK: 10 }) + const results = store.search({ queryText: 'query', queryEmbedding: [1, 0], mode: 'hybrid', topK: 10 }) const scoreById = Object.fromEntries(results.map((match) => [match.unitId, match.score])) expect(scoreById.A).toBeCloseTo(scoreById.B, 12) diff --git a/src/main/features/knowledge/vectorstore/indexStore/__tests__/KnowledgeIndexStore.search.test.ts b/src/main/features/knowledge/pipeline/vectorstore/indexStore/__tests__/KnowledgeIndexStore.search.test.ts similarity index 66% rename from src/main/features/knowledge/vectorstore/indexStore/__tests__/KnowledgeIndexStore.search.test.ts rename to src/main/features/knowledge/pipeline/vectorstore/indexStore/__tests__/KnowledgeIndexStore.search.test.ts index 9ee556f6953..8bee658d7a6 100644 --- a/src/main/features/knowledge/vectorstore/indexStore/__tests__/KnowledgeIndexStore.search.test.ts +++ b/src/main/features/knowledge/pipeline/vectorstore/indexStore/__tests__/KnowledgeIndexStore.search.test.ts @@ -24,7 +24,7 @@ describe('KnowledgeIndexStore.search', () => { }) afterEach(async () => { - await store.close() + store.close() rmSync(tempDir, { recursive: true, force: true }) }) @@ -39,33 +39,33 @@ describe('KnowledgeIndexStore.search', () => { }) it('vector mode ranks units by cosine similarity to the query embedding', async () => { - await indexMaterial('m1', 'a.md', 'apple pie', [1, 0, 0]) - await indexMaterial('m2', 'b.md', 'banana bread', [0, 1, 0]) + indexMaterial('m1', 'a.md', 'apple pie', [1, 0, 0]) + indexMaterial('m2', 'b.md', 'banana bread', [0, 1, 0]) - const matches = await store.search({ queryText: '', queryEmbedding: [0.95, 0.05, 0], mode: 'vector', topK: 10 }) + const matches = store.search({ queryText: '', queryEmbedding: [0.95, 0.05, 0], mode: 'vector', topK: 10 }) expect(matches.map((m) => m.materialId)).toEqual(['m1', 'm2']) expect(matches[0].score).toBeGreaterThan(matches[1].score) }) it('vector mode drops a degenerate zero-norm embedding instead of ranking it first', async () => { - await indexMaterial('m1', 'a.md', 'apple pie', [1, 0, 0]) + indexMaterial('m1', 'a.md', 'apple pie', [1, 0, 0]) // A zero vector has undefined cosine distance (sqlite-vec returns NaN, coerced to // NULL). Without the `dist IS NOT NULL` guard it sorts first under `ORDER BY dist` // and scores a perfect `1 - Number(null) = 1`, outranking the real hit — so it must // be excluded entirely. - await indexMaterial('m2', 'b.md', 'banana bread', [0, 0, 0]) + indexMaterial('m2', 'b.md', 'banana bread', [0, 0, 0]) - const matches = await store.search({ queryText: '', queryEmbedding: [1, 1, 0], mode: 'vector', topK: 10 }) + const matches = store.search({ queryText: '', queryEmbedding: [1, 1, 0], mode: 'vector', topK: 10 }) expect(matches.map((m) => m.materialId)).toEqual(['m1']) }) it('bm25 mode returns only units whose body matches the query tokens', async () => { - await indexMaterial('m1', 'a.md', 'apple pie', [1, 0, 0]) - await indexMaterial('m2', 'b.md', 'banana bread', [0, 1, 0]) + indexMaterial('m1', 'a.md', 'apple pie', [1, 0, 0]) + indexMaterial('m2', 'b.md', 'banana bread', [0, 1, 0]) - const matches = await store.search({ queryText: 'banana', mode: 'bm25', topK: 10 }) + const matches = store.search({ queryText: 'banana', mode: 'bm25', topK: 10 }) expect(matches.map((m) => m.materialId)).toEqual(['m2']) }) @@ -75,7 +75,7 @@ describe('KnowledgeIndexStore.search', () => { // been exercised against a real store — every other test in this file goes through // `indexMaterial`, which always supplies a vector. const text = 'lexical only content with no embedding model' - await store.rebuildMaterial('m1', { + store.rebuildMaterial('m1', { material: { relativePath: 'a.md' }, content: { text }, units: [{ unitType: 'chunk', unitIndex: 0, charStart: 0, charEnd: text.length }], @@ -83,33 +83,33 @@ describe('KnowledgeIndexStore.search', () => { embeddings: [] }) - const matches = await store.search({ queryText: 'lexical', mode: 'bm25', topK: 10 }) + const matches = store.search({ queryText: 'lexical', mode: 'bm25', topK: 10 }) expect(matches.map((m) => m.materialId)).toEqual(['m1']) }) it('bm25 mode returns nothing when the query has no usable token', async () => { - await indexMaterial('m1', 'a.md', 'apple pie', [1, 0, 0]) + indexMaterial('m1', 'a.md', 'apple pie', [1, 0, 0]) - expect(await store.search({ queryText: '!!!', mode: 'bm25', topK: 10 })).toEqual([]) + expect(store.search({ queryText: '!!!', mode: 'bm25', topK: 10 })).toEqual([]) }) it('bm25 mode falls back to a LIKE substring scan for short CJK queries the trigram FTS cannot index', async () => { - await indexMaterial('m1', 'a.md', '今天天气很好', [1, 0, 0]) - await indexMaterial('m2', 'b.md', '我喜欢编程', [0, 1, 0]) + indexMaterial('m1', 'a.md', '今天天气很好', [1, 0, 0]) + indexMaterial('m2', 'b.md', '我喜欢编程', [0, 1, 0]) // '天气' is 2 characters → produces no trigram → a bare MATCH returns nothing. - const matches = await store.search({ queryText: '天气', mode: 'bm25', topK: 10 }) + const matches = store.search({ queryText: '天气', mode: 'bm25', topK: 10 }) expect(matches.map((m) => m.materialId)).toEqual(['m1']) }) it('LIKE fallback ANDs every token, so a mixed short+long query still filters correctly', async () => { - await indexMaterial('m1', 'a.md', '系统 architecture overview', [1, 0, 0]) - await indexMaterial('m2', 'b.md', '系统 design notes', [0, 1, 0]) + indexMaterial('m1', 'a.md', '系统 architecture overview', [1, 0, 0]) + indexMaterial('m2', 'b.md', '系统 design notes', [0, 1, 0]) // The 2-char '系统' routes the whole query to LIKE; 'architecture' must still constrain it. - const matches = await store.search({ queryText: '系统 architecture', mode: 'bm25', topK: 10 }) + const matches = store.search({ queryText: '系统 architecture', mode: 'bm25', topK: 10 }) expect(matches.map((m) => m.materialId)).toEqual(['m1']) }) @@ -120,15 +120,15 @@ describe('KnowledgeIndexStore.search', () => { // the real-DB expectations below provably exercise the trigram index. expect(needsLikeFallback('天气预报')).toBe(false) - await indexMaterial('m1', 'a.md', '明天的天气预报说有雨', [1, 0, 0]) - await indexMaterial('m2', 'b.md', '我喜欢户外编程活动', [0, 1, 0]) + indexMaterial('m1', 'a.md', '明天的天气预报说有雨', [1, 0, 0]) + indexMaterial('m2', 'b.md', '我喜欢户外编程活动', [0, 1, 0]) - const matches = await store.search({ queryText: '天气预报', mode: 'bm25', topK: 10 }) + const matches = store.search({ queryText: '天气预报', mode: 'bm25', topK: 10 }) expect(matches.map((m) => m.materialId)).toEqual(['m1']) // A 3+ char CJK query whose trigrams appear nowhere must return empty via MATCH. expect(needsLikeFallback('量子计算')).toBe(false) - expect(await store.search({ queryText: '量子计算', mode: 'bm25', topK: 10 })).toEqual([]) + expect(store.search({ queryText: '量子计算', mode: 'bm25', topK: 10 })).toEqual([]) }) it('hybrid mode lifts a short-CJK LIKE-only hit above a closer vector-only competitor', async () => { @@ -136,10 +136,10 @@ describe('KnowledgeIndexStore.search', () => { // orthogonal in vector space but matches '天气' via the LIKE fallback. The BM25 // contribution must lift m1 above m2 — drop the LIKE fallback and the order // flips to ['m2', 'm1'], so this pins the fallback's effect on hybrid ranking. - await indexMaterial('m1', 'a.md', '今天天气', [0, 1, 0]) - await indexMaterial('m2', 'b.md', 'sunny day', [1, 0, 0]) + indexMaterial('m1', 'a.md', '今天天气', [0, 1, 0]) + indexMaterial('m2', 'b.md', 'sunny day', [1, 0, 0]) - const matches = await store.search({ + const matches = store.search({ queryText: '天气', queryEmbedding: [1, 0, 0], mode: 'hybrid', @@ -151,10 +151,10 @@ describe('KnowledgeIndexStore.search', () => { it('hybrid fusion ranks a unit hit by both lanes above one hit by a single lane', async () => { // Vector favors m1; BM25 favors m2. RRF should lift m2 because it appears in both lanes. - await indexMaterial('m1', 'a.md', 'apple pie', [1, 0, 0]) - await indexMaterial('m2', 'b.md', 'banana bread', [0, 1, 0]) + indexMaterial('m1', 'a.md', 'apple pie', [1, 0, 0]) + indexMaterial('m2', 'b.md', 'banana bread', [0, 1, 0]) - const matches = await store.search({ + const matches = store.search({ queryText: 'banana', queryEmbedding: [0.95, 0.05, 0], mode: 'hybrid', @@ -165,17 +165,17 @@ describe('KnowledgeIndexStore.search', () => { }) it('honors topK', async () => { - await indexMaterial('m1', 'a.md', 'alpha text', [1, 0, 0]) - await indexMaterial('m2', 'b.md', 'beta text', [0, 1, 0]) - await indexMaterial('m3', 'c.md', 'gamma text', [0, 0, 1]) + indexMaterial('m1', 'a.md', 'alpha text', [1, 0, 0]) + indexMaterial('m2', 'b.md', 'beta text', [0, 1, 0]) + indexMaterial('m3', 'c.md', 'gamma text', [0, 0, 1]) - expect(await store.search({ queryText: '', queryEmbedding: [1, 1, 1], mode: 'vector', topK: 2 })).toHaveLength(2) + expect(store.search({ queryText: '', queryEmbedding: [1, 1, 1], mode: 'vector', topK: 2 })).toHaveLength(2) }) it('rejects vector and hybrid search without a query embedding', async () => { - await indexMaterial('m1', 'a.md', 'apple pie', [1, 0, 0]) + indexMaterial('m1', 'a.md', 'apple pie', [1, 0, 0]) - await expect(store.search({ queryText: 'apple', mode: 'vector', topK: 5 })).rejects.toThrow(/query embedding/) - await expect(store.search({ queryText: 'apple', mode: 'hybrid', topK: 5 })).rejects.toThrow(/query embedding/) + expect(() => store.search({ queryText: 'apple', mode: 'vector', topK: 5 })).toThrow(/query embedding/) + expect(() => store.search({ queryText: 'apple', mode: 'hybrid', topK: 5 })).toThrow(/query embedding/) }) }) diff --git a/src/main/features/knowledge/vectorstore/indexStore/__tests__/KnowledgeIndexStore.test.ts b/src/main/features/knowledge/pipeline/vectorstore/indexStore/__tests__/KnowledgeIndexStore.test.ts similarity index 77% rename from src/main/features/knowledge/vectorstore/indexStore/__tests__/KnowledgeIndexStore.test.ts rename to src/main/features/knowledge/pipeline/vectorstore/indexStore/__tests__/KnowledgeIndexStore.test.ts index 5d3ba7abbb1..4d1ce5d3345 100644 --- a/src/main/features/knowledge/vectorstore/indexStore/__tests__/KnowledgeIndexStore.test.ts +++ b/src/main/features/knowledge/pipeline/vectorstore/indexStore/__tests__/KnowledgeIndexStore.test.ts @@ -47,7 +47,7 @@ describe('KnowledgeIndexStore', () => { }) afterEach(async () => { - await store.close() + store.close() rmSync(tempDir, { recursive: true, force: true }) }) @@ -68,7 +68,7 @@ describe('KnowledgeIndexStore', () => { driver.execute(`INSERT INTO search_text_fts(search_text_fts, rank) VALUES('integrity-check', 1)`) it('persists material, content, units, search_text and embeddings, then lists units in order', async () => { - await store.rebuildMaterial( + store.rebuildMaterial( 'm1', buildInput('hello world wide', [ [0, 5], @@ -76,7 +76,7 @@ describe('KnowledgeIndexStore', () => { ]) ) - const units = await store.listMaterialUnits('m1') + const units = store.listMaterialUnits('m1') expect(units.map((u) => u.text)).toEqual(['hello', 'world wide']) expect(units.map((u) => u.unitIndex)).toEqual([0, 1]) @@ -91,7 +91,7 @@ describe('KnowledgeIndexStore', () => { it('keeps body text equal to the content slice (search §5.3 invariant)', async () => { const text = 'alpha beta gamma' - await store.rebuildMaterial( + store.rebuildMaterial( 'm1', buildInput(text, [ [0, 5], @@ -100,24 +100,24 @@ describe('KnowledgeIndexStore', () => { ]) ) - for (const unit of await store.listMaterialUnits('m1')) { + for (const unit of store.listMaterialUnits('m1')) { expect(unit.text).toBe(text.slice(unit.charStart, unit.charEnd)) } }) it('throws on a unit whose body row is missing instead of fabricating an empty chunk', async () => { - await store.rebuildMaterial('m1', buildInput('alpha beta', [[0, 5]])) + store.rebuildMaterial('m1', buildInput('alpha beta', [[0, 5]])) // Corrupt the store: drop the body row out from under the unit. The same // damage silently excludes the unit from search (INNER JOIN); the list lane // must fail loudly rather than add a third symptom (existing-but-empty chunk). driver.execute(`DELETE FROM search_text WHERE target_type = 'search_unit' AND kind = 'body'`) - await expect(store.listMaterialUnits('m1')).rejects.toThrow('missing the body text for unit') + expect(() => store.listMaterialUnits('m1')).toThrow('missing the body text for unit') }) it('atomically replaces all prior units on rebuild (no old/new mix)', async () => { - await store.rebuildMaterial( + store.rebuildMaterial( 'm1', buildInput('one two three', [ [0, 3], @@ -127,8 +127,8 @@ describe('KnowledgeIndexStore', () => { ) expect(await count('search_unit')).toBe(3) - await store.rebuildMaterial('m1', buildInput('solo', [[0, 4]])) - const units = await store.listMaterialUnits('m1') + store.rebuildMaterial('m1', buildInput('solo', [[0, 4]])) + const units = store.listMaterialUnits('m1') expect(units).toHaveLength(1) expect(units[0].text).toBe('solo') expect(await count('search_unit')).toBe(1) @@ -137,7 +137,7 @@ describe('KnowledgeIndexStore', () => { it('reuses one embedding row for units sharing identical body text', async () => { // Two units with the same slice → same embedding_text_hash → one embedding row. - await store.rebuildMaterial( + store.rebuildMaterial( 'm1', buildInput('dup', [ [0, 3], @@ -150,16 +150,16 @@ describe('KnowledgeIndexStore', () => { }) it('keeps the FTS index in sync across rebuilds', async () => { - await store.rebuildMaterial('m1', buildInput('the knowledge base', [[0, 18]])) + store.rebuildMaterial('m1', buildInput('the knowledge base', [[0, 18]])) expect(await ftsMatchCount('knowledge')).toBe(1) - await store.rebuildMaterial('m1', buildInput('a different subject', [[0, 19]])) + store.rebuildMaterial('m1', buildInput('a different subject', [[0, 19]])) expect(await ftsMatchCount('knowledge')).toBe(0) expect(await ftsMatchCount('different')).toBe(1) }) it('rolls back the whole rebuild on a mid-transaction failure', async () => { - await store.rebuildMaterial( + store.rebuildMaterial( 'm1', buildInput('keep this safe', [ [0, 4], @@ -179,9 +179,9 @@ describe('KnowledgeIndexStore', () => { usesEmbeddings: true, embeddings: [{ embeddingTextHash: hashEmbeddingText('broken'), vector: [0.1, 0.2, 0.3] }] } - await expect(store.rebuildMaterial('m1', broken)).rejects.toThrow() + expect(() => store.rebuildMaterial('m1', broken)).toThrow() - const units = await store.listMaterialUnits('m1') + const units = store.listMaterialUnits('m1') expect(units.map((u) => u.text)).toEqual(['keep', 'this']) expect(await count('search_unit')).toBe(2) }) @@ -196,10 +196,10 @@ describe('KnowledgeIndexStore', () => { [0, 5], [6, 16] ]) - await store.rebuildMaterial('m1', input) - await expect(store.rebuildMaterial('m1', input)).resolves.toBeUndefined() + store.rebuildMaterial('m1', input) + expect(store.rebuildMaterial('m1', input)).toBeUndefined() - const units = await store.listMaterialUnits('m1') + const units = store.listMaterialUnits('m1') expect(units.map((u) => u.text)).toEqual(['hello', 'world wide']) expect(await count('search_unit')).toBe(2) expect(await count('search_text')).toBe(2) @@ -211,9 +211,7 @@ describe('KnowledgeIndexStore', () => { it('rejects a unit whose charEnd lies beyond the content text', async () => { // slice() would clamp silently and persist the lying offset — the store must // fail loud at write time instead of corrupting offset-based readers later. - await expect(store.rebuildMaterial('m1', buildInput('short', [[0, 99]]))).rejects.toThrow( - 'beyond the content length' - ) + expect(() => store.rebuildMaterial('m1', buildInput('short', [[0, 99]]))).toThrow('beyond the content length') expect(await count('material')).toBe(0) expect(await count('search_unit')).toBe(0) @@ -230,14 +228,14 @@ describe('KnowledgeIndexStore', () => { usesEmbeddings: false, embeddings: [{ embeddingTextHash: hashEmbeddingText('lexical only'), vector: [0.1, 0.2, 0.3] }] } - await expect(store.rebuildMaterial('m1', contradiction)).rejects.toThrow('usesEmbeddings: false') + expect(() => store.rebuildMaterial('m1', contradiction)).toThrow('usesEmbeddings: false') expect(await count('material')).toBe(0) expect(await count('search_unit')).toBe(0) }) it('rolls back a rebuild that leaves a unit embedding hash without a vector', async () => { - await store.rebuildMaterial('m1', buildInput('keep this safe', [[0, 4]])) + store.rebuildMaterial('m1', buildInput('keep this safe', [[0, 4]])) // A caller that hashes different text than the store re-slices (offset/hash // drift) supplies no vector for the unit's body — the coverage check must @@ -249,10 +247,10 @@ describe('KnowledgeIndexStore', () => { usesEmbeddings: true, embeddings: [{ embeddingTextHash: hashEmbeddingText('not the sliced body'), vector: [0.1, 0.2, 0.3] }] } - await expect(store.rebuildMaterial('m1', drifted)).rejects.toThrow('without a vector') + expect(() => store.rebuildMaterial('m1', drifted)).toThrow('without a vector') // Prior index intact (transaction rolled back). - const units = await store.listMaterialUnits('m1') + const units = store.listMaterialUnits('m1') expect(units.map((u) => u.text)).toEqual(['keep']) }) @@ -275,21 +273,21 @@ describe('KnowledgeIndexStore', () => { missingOne.embeddings = missingOne.embeddings.filter( (embedding) => embedding.embeddingTextHash !== hashEmbeddingText(words[500]) ) - await expect(store.rebuildMaterial('m1', missingOne)).rejects.toThrow('without a vector') + expect(() => store.rebuildMaterial('m1', missingOne)).toThrow('without a vector') expect(await count('material')).toBe(0) - await expect(store.rebuildMaterial('m1', buildInput(text, ranges))).resolves.toBeUndefined() + expect(store.rebuildMaterial('m1', buildInput(text, ranges))).toBeUndefined() expect(await count('search_unit')).toBe(501) expect(await count('embedding')).toBe(501) }) it('deletes a material and its derived rows, sweeping the now-orphaned embedding and content', async () => { - await store.rebuildMaterial('m1', buildInput('the knowledge base', [[0, 18]])) + store.rebuildMaterial('m1', buildInput('the knowledge base', [[0, 18]])) expect(await ftsMatchCount('knowledge')).toBe(1) await store.deleteMaterials(['m1']) - expect(await store.listMaterialUnits('m1')).toEqual([]) + expect(store.listMaterialUnits('m1')).toEqual([]) expect(await count('material')).toBe(0) expect(await count('search_unit')).toBe(0) expect(await count('search_text')).toBe(0) @@ -300,8 +298,8 @@ describe('KnowledgeIndexStore', () => { }) it('shares one content row across materials with identical content', async () => { - await store.rebuildMaterial('m1', buildInput('shared content', [[0, 14]], 'a.md')) - await store.rebuildMaterial('m2', buildInput('shared content', [[0, 14]], 'b.md')) + store.rebuildMaterial('m1', buildInput('shared content', [[0, 14]], 'a.md')) + store.rebuildMaterial('m2', buildInput('shared content', [[0, 14]], 'b.md')) expect(await count('material')).toBe(2) expect(await count('content')).toBe(1) @@ -310,8 +308,8 @@ describe('KnowledgeIndexStore', () => { it('keeps a shared embedding reachable for the remaining material after deleting one sharer', async () => { // m1 and m2 index the identical body → one embedding row, referenced by both. - await store.rebuildMaterial('m1', buildInput('shared body text', [[0, 16]], 'a.md')) - await store.rebuildMaterial('m2', buildInput('shared body text', [[0, 16]], 'b.md')) + store.rebuildMaterial('m1', buildInput('shared body text', [[0, 16]], 'a.md')) + store.rebuildMaterial('m2', buildInput('shared body text', [[0, 16]], 'b.md')) expect(await count('embedding')).toBe(1) await store.deleteMaterials(['m1']) @@ -321,16 +319,16 @@ describe('KnowledgeIndexStore', () => { // embedding when m1 was deleted would fail this behavioral assertion — the bare row // count in the delete test above cannot catch that. expect(await count('embedding')).toBe(1) - const matches = await store.search({ queryText: '', queryEmbedding: [0.1, 0.2, 0.3], mode: 'vector', topK: 10 }) + const matches = store.search({ queryText: '', queryEmbedding: [0.1, 0.2, 0.3], mode: 'vector', topK: 10 }) expect(matches.map((m) => m.materialId)).toEqual(['m2']) }) it('deleteMaterials removes the whole batch in one pass, sweeps only true orphans, and keeps the survivor searchable', async () => { // m1, m2 will be deleted; m3 survives. m1 and m3 index the IDENTICAL body, so they // share one content row and one embedding row; m2 has its own unique body. - await store.rebuildMaterial('m1', buildInput('shared knowledge body', [[0, 21]], 'a.md', [0.1, 0.2, 0.3])) - await store.rebuildMaterial('m2', buildInput('unique orphan body', [[0, 18]], 'b.md', [0.4, 0.5, 0.6])) - await store.rebuildMaterial('m3', buildInput('shared knowledge body', [[0, 21]], 'c.md', [0.1, 0.2, 0.3])) + store.rebuildMaterial('m1', buildInput('shared knowledge body', [[0, 21]], 'a.md', [0.1, 0.2, 0.3])) + store.rebuildMaterial('m2', buildInput('unique orphan body', [[0, 18]], 'b.md', [0.4, 0.5, 0.6])) + store.rebuildMaterial('m3', buildInput('shared knowledge body', [[0, 21]], 'c.md', [0.1, 0.2, 0.3])) expect(await count('material')).toBe(3) expect(await count('content')).toBe(2) // shared (m1+m3) + unique (m2) expect(await count('embedding')).toBe(2) @@ -340,8 +338,8 @@ describe('KnowledgeIndexStore', () => { await store.deleteMaterials(['m1', 'm2', 'm1']) expect(driver.execute(`SELECT material_id FROM material`).rows.map((r) => r.material_id)).toEqual(['m3']) - expect(await store.listMaterialUnits('m1')).toEqual([]) - expect(await store.listMaterialUnits('m2')).toEqual([]) + expect(store.listMaterialUnits('m1')).toEqual([]) + expect(store.listMaterialUnits('m2')).toEqual([]) // The single end-of-batch GC must sweep m2's now-orphaned body/embedding/content while // keeping the body m3 still references — i.e. the same end state N per-material GCs gave. expect(await count('content')).toBe(1) @@ -352,18 +350,16 @@ describe('KnowledgeIndexStore', () => { // the survivor m3 — whose search_text row is untouched — would drop out of keyword // search even though it is still present. It must stay both bm25- and vector-reachable. expect(await ftsMatchCount('knowledge')).toBe(1) - expect((await store.search({ queryText: 'knowledge', mode: 'bm25', topK: 10 })).map((h) => h.materialId)).toEqual([ - 'm3' - ]) + expect(store.search({ queryText: 'knowledge', mode: 'bm25', topK: 10 }).map((h) => h.materialId)).toEqual(['m3']) expect( - (await store.search({ queryText: '', queryEmbedding: [0.1, 0.2, 0.3], mode: 'vector', topK: 10 })).map( - (h) => h.materialId - ) + store + .search({ queryText: '', queryEmbedding: [0.1, 0.2, 0.3], mode: 'vector', topK: 10 }) + .map((h) => h.materialId) ).toEqual(['m3']) }) it('deleteMaterials is a no-op for an empty batch', async () => { - await store.rebuildMaterial('m1', buildInput('keep me', [[0, 7]])) + store.rebuildMaterial('m1', buildInput('keep me', [[0, 7]])) await expect(store.deleteMaterials([])).resolves.toBeUndefined() expect(await count('material')).toBe(1) }) @@ -374,9 +370,9 @@ describe('KnowledgeIndexStore', () => { // periodic yields. Prove the loop hands control back: drive the MONOTONIC performance.now() // past the time budget on every read (deterministic — no real-clock dependency) and confirm // the loop schedules a macrotask (setImmediate) per material while still deleting correctly. - await store.rebuildMaterial('m1', buildInput('alpha body one', [[0, 14]], 'a.md')) - await store.rebuildMaterial('m2', buildInput('bravo body two', [[0, 14]], 'b.md')) - await store.rebuildMaterial('m3', buildInput('gamma body six', [[0, 14]], 'c.md')) + store.rebuildMaterial('m1', buildInput('alpha body one', [[0, 14]], 'a.md')) + store.rebuildMaterial('m2', buildInput('bravo body two', [[0, 14]], 'b.md')) + store.rebuildMaterial('m3', buildInput('gamma body six', [[0, 14]], 'c.md')) // setImmediate is spied (call-through), not stubbed, so the yields still resolve. let clock = 0 @@ -402,9 +398,9 @@ describe('KnowledgeIndexStore', () => { // DELETE_YIELD_BUDGET_MS must schedule zero macrotasks. Freezing the monotonic clock keeps // every delta at 0 — so this fails if the gate is ever replaced by an unconditional // yield-per-row (the exact regression the test above cannot, on its own, rule out). - await store.rebuildMaterial('m1', buildInput('alpha body one', [[0, 14]], 'a.md')) - await store.rebuildMaterial('m2', buildInput('bravo body two', [[0, 14]], 'b.md')) - await store.rebuildMaterial('m3', buildInput('gamma body six', [[0, 14]], 'c.md')) + store.rebuildMaterial('m1', buildInput('alpha body one', [[0, 14]], 'a.md')) + store.rebuildMaterial('m2', buildInput('bravo body two', [[0, 14]], 'b.md')) + store.rebuildMaterial('m3', buildInput('gamma body six', [[0, 14]], 'c.md')) const perfNow = vi.spyOn(performance, 'now').mockReturnValue(1000) const immediate = vi.spyOn(global, 'setImmediate') @@ -427,34 +423,32 @@ describe('KnowledgeIndexStore', () => { // because search_text_fts keys on the stable fts_rowid (not the implicit rowid), m2 — whose // search_text is untouched — stays aligned. It must remain reachable, and the index intact. const hugeText = 'knowledge body filler '.repeat(600_000) - await store.rebuildMaterial('m1', buildInput(hugeText, [[0, 20]], 'big.md')) - await store.rebuildMaterial('m2', buildInput('shared knowledge body', [[0, 21]], 'keep.md')) + store.rebuildMaterial('m1', buildInput(hugeText, [[0, 20]], 'big.md')) + store.rebuildMaterial('m2', buildInput('shared knowledge body', [[0, 21]], 'keep.md')) await store.deleteMaterials(['m1']) - const outcome = await store.reclaimSpace() + const outcome = store.reclaimSpace() expect(outcome.vacuumed).toBe(true) expect(outcome.reclaimedBytes).toBeGreaterThan(0) expect(ftsIntegrityCheck()).toBeDefined() // The survivor stays both keyword- and vector-reachable after the rewrite. expect(await ftsMatchCount('knowledge')).toBe(1) - expect((await store.search({ queryText: 'knowledge', mode: 'bm25', topK: 10 })).map((h) => h.materialId)).toEqual([ - 'm2' - ]) + expect(store.search({ queryText: 'knowledge', mode: 'bm25', topK: 10 }).map((h) => h.materialId)).toEqual(['m2']) expect( - (await store.search({ queryText: '', queryEmbedding: [0.1, 0.2, 0.3], mode: 'vector', topK: 10 })).map( - (h) => h.materialId - ) + store + .search({ queryText: '', queryEmbedding: [0.1, 0.2, 0.3], mode: 'vector', topK: 10 }) + .map((h) => h.materialId) ).toEqual(['m2']) }) it('reclaimSpace skips the VACUUM (truncates the WAL only) when the freelist is below the threshold', async () => { // A tiny delete frees far less than the absolute floor, so the whole-file-rewrite block // is not worth it — reclaim just checkpoints and reports nothing reclaimed. - await store.rebuildMaterial('m1', buildInput('small knowledge body', [[0, 20]], 'a.md')) + store.rebuildMaterial('m1', buildInput('small knowledge body', [[0, 20]], 'a.md')) await store.deleteMaterials(['m1']) - const outcome = await store.reclaimSpace() + const outcome = store.reclaimSpace() expect(outcome).toEqual({ vacuumed: false, reclaimedBytes: 0 }) }) @@ -467,20 +461,18 @@ describe('KnowledgeIndexStore', () => { // before. content stays 13 MB so the freelist clears the VACUUM threshold; the 2 MB // body makes the shadow segments measurable. const hugeText = 'knowledge body filler '.repeat(600_000) - await store.rebuildMaterial('m1', buildInput(hugeText, [[0, 2_000_000]], 'big.md')) - await store.rebuildMaterial('m2', buildInput('shared knowledge body', [[0, 21]], 'keep.md')) + store.rebuildMaterial('m1', buildInput(hugeText, [[0, 2_000_000]], 'big.md')) + store.rebuildMaterial('m2', buildInput('shared knowledge body', [[0, 21]], 'keep.md')) const ftsSegmentsBefore = await count('search_text_fts_data') await store.deleteMaterials(['m1']) - const outcome = await store.reclaimSpace() + const outcome = store.reclaimSpace() expect(outcome.vacuumed).toBe(true) // optimize merged and dropped m1's dead segments instead of leaving them behind. expect(await count('search_text_fts_data')).toBeLessThan(ftsSegmentsBefore) // The survivor is still keyword-searchable after the optimize + VACUUM. - expect((await store.search({ queryText: 'shared', mode: 'bm25', topK: 10 })).map((h) => h.materialId)).toEqual([ - 'm2' - ]) + expect(store.search({ queryText: 'shared', mode: 'bm25', topK: 10 }).map((h) => h.materialId)).toEqual(['m2']) // VACUUM renumbers implicit rowids; assert the external-content FTS did NOT desync. Keyed on the // stable fts_rowid it stays aligned by construction — verified with the reliable integrity check // (the default integrity-check would pass even on a desync). @@ -492,10 +484,10 @@ describe('KnowledgeIndexStore', () => { // search_text's implicit rowid, and an external-content FTS keyed on that rowid would then // silently point at the wrong rows. search_text_fts keys on the stable fts_rowid column instead. // The index MUST be populated before the reshuffle, or it cannot expose the bug. - await store.rebuildMaterial('m1', buildInput('alpha apple body', [[0, 16]], 'a.md')) - await store.rebuildMaterial('m2', buildInput('bravo banana body', [[0, 17]], 'b.md')) - await store.rebuildMaterial('m3', buildInput('charlie cherry body', [[0, 19]], 'c.md')) - await store.rebuildMaterial('m4', buildInput('delta date body', [[0, 15]], 'd.md')) + store.rebuildMaterial('m1', buildInput('alpha apple body', [[0, 16]], 'a.md')) + store.rebuildMaterial('m2', buildInput('bravo banana body', [[0, 17]], 'b.md')) + store.rebuildMaterial('m3', buildInput('charlie cherry body', [[0, 19]], 'c.md')) + store.rebuildMaterial('m4', buildInput('delta date body', [[0, 15]], 'd.md')) // Delete a middle material to leave a rowid hole — the precondition the reshuffle needs. await store.deleteMaterials(['m2']) expect(await ftsMatchCount('cherry')).toBe(1) @@ -513,17 +505,15 @@ describe('KnowledgeIndexStore', () => { // fts_rowid was copied through the rebuild, so the index stays aligned: the reliable check passes // and the survivors resolve to the right materials (a rowid-keyed index would return wrong/empty). expect(ftsIntegrityCheck()).toBeDefined() - expect((await store.search({ queryText: 'cherry', mode: 'bm25', topK: 10 })).map((h) => h.materialId)).toEqual([ - 'm3' - ]) - expect((await store.search({ queryText: 'date', mode: 'bm25', topK: 10 })).map((h) => h.materialId)).toEqual(['m4']) + expect(store.search({ queryText: 'cherry', mode: 'bm25', topK: 10 }).map((h) => h.materialId)).toEqual(['m3']) + expect(store.search({ queryText: 'date', mode: 'bm25', topK: 10 }).map((h) => h.materialId)).toEqual(['m4']) // The reshuffle reassigned implicit rowids while leaving fts_rowid untouched and unique. expect(await count('search_text')).toBe(3) expect(Number(driver.execute(`SELECT COUNT(DISTINCT fts_rowid) AS n FROM search_text`).rows[0].n)).toBe(3) }) it('integrity-check,1 catches a NULL fts_rowid desync (and proves the detector is live)', async () => { - await store.rebuildMaterial('m1', buildInput('orphan searchable body', [[0, 22]], 'a.md')) + store.rebuildMaterial('m1', buildInput('orphan searchable body', [[0, 22]], 'a.md')) expect(await ftsMatchCount('searchable')).toBe(1) // Simulate a future bulk-insert / restore path that bypassed the AFTER INSERT trigger and left @@ -540,9 +530,9 @@ describe('KnowledgeIndexStore', () => { // current MAX and then inserting REUSES that just-freed rowid — the highest-risk path for a // stale external-content entry: the new row lands on the exact key the deleted row had, so a // delete trigger that failed to tombstone it would surface the old content under the new row. - await store.rebuildMaterial('m1', buildInput('alpha apple body', [[0, 16]], 'a.md')) - await store.rebuildMaterial('m2', buildInput('bravo banana body', [[0, 17]], 'b.md')) - await store.rebuildMaterial('m3', buildInput('charlie cherry body', [[0, 19]], 'c.md')) + store.rebuildMaterial('m1', buildInput('alpha apple body', [[0, 16]], 'a.md')) + store.rebuildMaterial('m2', buildInput('bravo banana body', [[0, 17]], 'b.md')) + store.rebuildMaterial('m3', buildInput('charlie cherry body', [[0, 19]], 'c.md')) // m3 holds the current MAX fts_rowid. Delete it to free that rowid. const maxBefore = Number(driver.execute(`SELECT MAX(fts_rowid) AS m FROM search_text`).rows[0].m) @@ -550,7 +540,7 @@ describe('KnowledgeIndexStore', () => { expect(await ftsMatchCount('cherry')).toBe(0) // The next insert's MAX(fts_rowid)+1 reuses the value m3 just vacated. - await store.rebuildMaterial('m4', buildInput('delta dragon body', [[0, 17]], 'd.md')) + store.rebuildMaterial('m4', buildInput('delta dragon body', [[0, 17]], 'd.md')) const reused = Number( driver.execute(`SELECT fts_rowid FROM search_text WHERE text LIKE 'delta%'`).rows[0].fts_rowid ) @@ -558,49 +548,47 @@ describe('KnowledgeIndexStore', () => { // The reused rowid resolves to m4 (not stale m3 content), the deleted term stays gone, and the // external-content index is consistent under the reliable detector. - expect((await store.search({ queryText: 'dragon', mode: 'bm25', topK: 10 })).map((h) => h.materialId)).toEqual([ - 'm4' - ]) + expect(store.search({ queryText: 'dragon', mode: 'bm25', topK: 10 }).map((h) => h.materialId)).toEqual(['m4']) expect(await ftsMatchCount('cherry')).toBe(0) expect(ftsIntegrityCheck()).toBeDefined() }) it('keeps a shared embedding reachable for the other material after rebuilding the one that introduced it', async () => { // m1 and m2 index the identical body → one shared embedding row, referenced by both. - await store.rebuildMaterial('m1', buildInput('shared body text', [[0, 16]], 'a.md')) - await store.rebuildMaterial('m2', buildInput('shared body text', [[0, 16]], 'b.md')) + store.rebuildMaterial('m1', buildInput('shared body text', [[0, 16]], 'a.md')) + store.rebuildMaterial('m2', buildInput('shared body text', [[0, 16]], 'b.md')) expect(await count('embedding')).toBe(1) // Rebuild m1 with unrelated content carrying a distinct vector, so m1 no longer // references the shared embedding — only m2 does now. - await store.rebuildMaterial('m1', buildInput('rebuilt unrelated body', [[0, 22]], 'a.md', [0.9, 0.8, 0.7])) + store.rebuildMaterial('m1', buildInput('rebuilt unrelated body', [[0, 22]], 'a.md', [0.9, 0.8, 0.7])) // The shared embedding must survive (m2 still references it), alongside m1's new // one → 2 rows. GC dropping the now-singly-referenced shared embedding on rebuild // would unjoin m2 from vector search below — the bare row count cannot catch that, // but searching the shared vector can. expect(await count('embedding')).toBe(2) - const matches = await store.search({ queryText: '', queryEmbedding: [0.1, 0.2, 0.3], mode: 'vector', topK: 10 }) + const matches = store.search({ queryText: '', queryEmbedding: [0.1, 0.2, 0.3], mode: 'vector', topK: 10 }) expect(matches.map((m) => m.materialId)).toContain('m2') }) it('sweeps the orphaned embedding and content when a material is rebuilt with new content', async () => { - await store.rebuildMaterial('m1', buildInput('original body', [[0, 13]], 'a.md', [0.1, 0.2, 0.3])) + store.rebuildMaterial('m1', buildInput('original body', [[0, 13]], 'a.md', [0.1, 0.2, 0.3])) expect(await count('embedding')).toBe(1) expect(await count('content')).toBe(1) // Rebuild with unrelated content + a distinct vector; nothing else references the // old embedding or old content, so GC must remove both, leaving only the new ones. - await store.rebuildMaterial('m1', buildInput('replacement body', [[0, 16]], 'a.md', [0.9, 0.8, 0.7])) + store.rebuildMaterial('m1', buildInput('replacement body', [[0, 16]], 'a.md', [0.9, 0.8, 0.7])) expect(await count('embedding')).toBe(1) expect(await count('content')).toBe(1) - const units = await store.listMaterialUnits('m1') + const units = store.listMaterialUnits('m1') expect(units.map((u) => u.text)).toEqual(['replacement body']) }) it('listExistingEmbeddingHashes reports only the hashes already stored', async () => { - await store.rebuildMaterial( + store.rebuildMaterial( 'm1', buildInput('alpha bravo', [ [0, 5], @@ -610,7 +598,7 @@ describe('KnowledgeIndexStore', () => { const stored = hashEmbeddingText('alpha') const absent = hashEmbeddingText('charlie') - const existing = await store.listExistingEmbeddingHashes([stored, absent]) + const existing = store.listExistingEmbeddingHashes([stored, absent]) expect(existing.has(stored)).toBe(true) expect(existing.has(absent)).toBe(false) @@ -618,24 +606,24 @@ describe('KnowledgeIndexStore', () => { }) it('listExistingEmbeddingHashes returns an empty set for empty input', async () => { - expect((await store.listExistingEmbeddingHashes([])).size).toBe(0) + expect(store.listExistingEmbeddingHashes([]).size).toBe(0) }) describe('deep-read lookups', () => { it('resolves a material by its relative path (the Concept ID) and returns null for an unknown path', async () => { - await store.rebuildMaterial('m1', buildInput('hello world', [[0, 11]], 'docs/intro.md')) + store.rebuildMaterial('m1', buildInput('hello world', [[0, 11]], 'docs/intro.md')) - const ref = await store.getMaterialByRelativePath('docs/intro.md') + const ref = store.getMaterialByRelativePath('docs/intro.md') expect(ref).not.toBeNull() expect(ref?.materialId).toBe('m1') expect(ref?.relativePath).toBe('docs/intro.md') - expect(await store.getMaterialByRelativePath('docs/missing.md')).toBeNull() + expect(store.getMaterialByRelativePath('docs/missing.md')).toBeNull() }) it('reads back the full content text a material was indexed from', async () => { const text = 'the quick brown fox jumps over the lazy dog' - await store.rebuildMaterial( + store.rebuildMaterial( 'm1', buildInput(text, [ [0, 19], @@ -643,12 +631,12 @@ describe('KnowledgeIndexStore', () => { ]) ) - expect(await store.readMaterialContent('m1')).toBe(text) + expect(store.readMaterialContent('m1')).toBe(text) }) it('keeps content readable as the exact source the units were sliced from', async () => { const text = 'alpha beta gamma' - await store.rebuildMaterial( + store.rebuildMaterial( 'm1', buildInput(text, [ [0, 5], @@ -656,14 +644,14 @@ describe('KnowledgeIndexStore', () => { ]) ) - const content = await store.readMaterialContent('m1') - for (const unit of await store.listMaterialUnits('m1')) { + const content = store.readMaterialContent('m1') + for (const unit of store.listMaterialUnits('m1')) { expect(content?.slice(unit.charStart, unit.charEnd)).toBe(unit.text) } }) it('returns null content for an unknown material', async () => { - expect(await store.readMaterialContent('nope')).toBeNull() + expect(store.readMaterialContent('nope')).toBeNull() }) }) }) diff --git a/src/main/features/knowledge/vectorstore/indexStore/__tests__/ftsQuery.test.ts b/src/main/features/knowledge/pipeline/vectorstore/indexStore/__tests__/ftsQuery.test.ts similarity index 100% rename from src/main/features/knowledge/vectorstore/indexStore/__tests__/ftsQuery.test.ts rename to src/main/features/knowledge/pipeline/vectorstore/indexStore/__tests__/ftsQuery.test.ts diff --git a/src/main/features/knowledge/vectorstore/indexStore/__tests__/hashing.test.ts b/src/main/features/knowledge/pipeline/vectorstore/indexStore/__tests__/hashing.test.ts similarity index 100% rename from src/main/features/knowledge/vectorstore/indexStore/__tests__/hashing.test.ts rename to src/main/features/knowledge/pipeline/vectorstore/indexStore/__tests__/hashing.test.ts diff --git a/src/main/features/knowledge/vectorstore/indexStore/__tests__/indexMeta.test.ts b/src/main/features/knowledge/pipeline/vectorstore/indexStore/__tests__/indexMeta.test.ts similarity index 100% rename from src/main/features/knowledge/vectorstore/indexStore/__tests__/indexMeta.test.ts rename to src/main/features/knowledge/pipeline/vectorstore/indexStore/__tests__/indexMeta.test.ts diff --git a/src/main/features/knowledge/vectorstore/indexStore/__tests__/schema.test.ts b/src/main/features/knowledge/pipeline/vectorstore/indexStore/__tests__/schema.test.ts similarity index 100% rename from src/main/features/knowledge/vectorstore/indexStore/__tests__/schema.test.ts rename to src/main/features/knowledge/pipeline/vectorstore/indexStore/__tests__/schema.test.ts diff --git a/src/main/features/knowledge/vectorstore/indexStore/__tests__/vectorBlob.test.ts b/src/main/features/knowledge/pipeline/vectorstore/indexStore/__tests__/vectorBlob.test.ts similarity index 100% rename from src/main/features/knowledge/vectorstore/indexStore/__tests__/vectorBlob.test.ts rename to src/main/features/knowledge/pipeline/vectorstore/indexStore/__tests__/vectorBlob.test.ts diff --git a/src/main/features/knowledge/vectorstore/indexStore/ftsQuery.ts b/src/main/features/knowledge/pipeline/vectorstore/indexStore/ftsQuery.ts similarity index 100% rename from src/main/features/knowledge/vectorstore/indexStore/ftsQuery.ts rename to src/main/features/knowledge/pipeline/vectorstore/indexStore/ftsQuery.ts diff --git a/src/main/features/knowledge/vectorstore/indexStore/hashing.ts b/src/main/features/knowledge/pipeline/vectorstore/indexStore/hashing.ts similarity index 100% rename from src/main/features/knowledge/vectorstore/indexStore/hashing.ts rename to src/main/features/knowledge/pipeline/vectorstore/indexStore/hashing.ts diff --git a/src/main/features/knowledge/vectorstore/indexStore/indexMeta.ts b/src/main/features/knowledge/pipeline/vectorstore/indexStore/indexMeta.ts similarity index 100% rename from src/main/features/knowledge/vectorstore/indexStore/indexMeta.ts rename to src/main/features/knowledge/pipeline/vectorstore/indexStore/indexMeta.ts diff --git a/src/main/features/knowledge/vectorstore/indexStore/model.ts b/src/main/features/knowledge/pipeline/vectorstore/indexStore/model.ts similarity index 100% rename from src/main/features/knowledge/vectorstore/indexStore/model.ts rename to src/main/features/knowledge/pipeline/vectorstore/indexStore/model.ts diff --git a/src/main/features/knowledge/vectorstore/indexStore/schema.ts b/src/main/features/knowledge/pipeline/vectorstore/indexStore/schema.ts similarity index 99% rename from src/main/features/knowledge/vectorstore/indexStore/schema.ts rename to src/main/features/knowledge/pipeline/vectorstore/indexStore/schema.ts index ebc790bf145..31be4bd1673 100644 --- a/src/main/features/knowledge/vectorstore/indexStore/schema.ts +++ b/src/main/features/knowledge/pipeline/vectorstore/indexStore/schema.ts @@ -159,7 +159,7 @@ export const KNOWLEDGE_INDEX_SCHEMA_STATEMENTS: readonly string[] = [ `CREATE INDEX IF NOT EXISTS search_text_kind_idx ON search_text(kind)`, // UNIQUE so its backing index makes the trigger's per-row MAX(fts_rowid)+1 an O(log N) lookup and // rejects any duplicate loudly (the assignment is race-free only because all writes to a base's - // index are serialized upstream by KnowledgeLockManager.withBaseMutationLock — see BetterSqlite3Driver). + // index are serialized upstream by KeyedMutex.runExclusive (per knowledge base) — see BetterSqlite3Driver). `CREATE UNIQUE INDEX IF NOT EXISTS search_text_fts_rowid_uniq ON search_text(fts_rowid)`, // embedding — current embedding vector keyed by embedding-text hash. diff --git a/src/main/features/knowledge/vectorstore/indexStore/types.ts b/src/main/features/knowledge/pipeline/vectorstore/indexStore/types.ts similarity index 96% rename from src/main/features/knowledge/vectorstore/indexStore/types.ts rename to src/main/features/knowledge/pipeline/vectorstore/indexStore/types.ts index 037d3eaab9a..5622290829d 100644 --- a/src/main/features/knowledge/vectorstore/indexStore/types.ts +++ b/src/main/features/knowledge/pipeline/vectorstore/indexStore/types.ts @@ -72,12 +72,6 @@ export interface SqliteDriver extends SqliteExecutor { close(): void } -/** One brute-force vector match: an embedding row and its distance to the query. */ -export interface VectorMatch { - embeddingTextHash: string - distance: number -} - /** * Engine-specific vector primitives. The store composes the brute-force scan * (`SELECT … ORDER BY dist LIMIT k`) over the plain-BLOB `embedding.vector_blob` diff --git a/src/main/features/knowledge/vectorstore/indexStore/vectorBlob.ts b/src/main/features/knowledge/pipeline/vectorstore/indexStore/vectorBlob.ts similarity index 100% rename from src/main/features/knowledge/vectorstore/indexStore/vectorBlob.ts rename to src/main/features/knowledge/pipeline/vectorstore/indexStore/vectorBlob.ts diff --git a/src/main/features/knowledge/utils/cleanup/vectorCleanup.ts b/src/main/features/knowledge/pipeline/vectorstore/vectorCleanup.ts similarity index 92% rename from src/main/features/knowledge/utils/cleanup/vectorCleanup.ts rename to src/main/features/knowledge/pipeline/vectorstore/vectorCleanup.ts index 0d916567329..507444bb088 100644 --- a/src/main/features/knowledge/utils/cleanup/vectorCleanup.ts +++ b/src/main/features/knowledge/pipeline/vectorstore/vectorCleanup.ts @@ -11,7 +11,7 @@ export async function deleteKnowledgeItemVectors(base: KnowledgeBase, itemIds: s } const vectorStoreService = application.get('KnowledgeVectorStoreService') - const store = await vectorStoreService.getIndexStoreIfExists(base) + const store = vectorStoreService.getIndexStoreIfExists(base) if (!store) { return } @@ -44,11 +44,11 @@ export async function reclaimKnowledgeIndexSpace(base: KnowledgeBase): Promise 0 ? matchLength : 1) + } + + lineStart = lineEnd + 1 + } + + return { totalMatches, matches } +} + +/** Concept ID-addressed agent tool surface: deep read (read/grep), organization tree, and concept-level delete/refresh. */ +export class KnowledgeConceptService { + constructor(private readonly ingestionService: KnowledgeIngestionService) {} + + /** + * Read a knowledge concept's indexed text by its Concept ID (the material's + * relative path, OKF §2), optionally a `[charStart, charEnd)` slice. The slice + * is capped at {@link CONCEPT_READ_MAX_CHARS}; `totalChars` + `truncated` let + * the caller page through a longer document. Throws NOT_FOUND when the Concept + * ID does not resolve to a readable, visible document in this base. + */ + async readConcept( + baseId: string, + conceptId: string, + range?: { charStart?: number; charEnd?: number } + ): Promise { + const { item, text } = await this.resolveConcept(baseId, conceptId, 'readConcept') + + const totalChars = text.length + const start = Math.min(Math.max(range?.charStart ?? 0, 0), totalChars) + // Where the caller would have ended without the cap (an omitted charEnd reads to the document end). + const naturalEnd = Math.min(range?.charEnd ?? totalChars, totalChars) + const end = Math.max(start, Math.min(naturalEnd, start + CONCEPT_READ_MAX_CHARS)) + + return { + conceptId, + title: getKnowledgeItemDisplayTitle(item), + itemType: item.type, + totalChars, + charStart: start, + charEnd: end, + content: text.slice(start, end), + truncated: end < naturalEnd + } + } + + /** + * Search a knowledge concept's indexed text for a regular expression, returning + * the total match count and the first `maxMatches` matches (line, offsets, a + * padded snippet). The pattern is compiled global, case-insensitive by default; + * an invalid pattern throws a validation error. Throws NOT_FOUND when the + * Concept ID does not resolve to a readable, visible document in this base. + */ + async grepConcept( + baseId: string, + conceptId: string, + options: { pattern: string; ignoreCase?: boolean; maxMatches?: number } + ): Promise { + const { item, text } = await this.resolveConcept(baseId, conceptId, 'grepConcept') + + const limit = Math.min( + Math.max(options.maxMatches ?? CONCEPT_GREP_DEFAULT_MAX_MATCHES, 1), + CONCEPT_GREP_MAX_MATCHES + ) + const regex = compileGrepRegex(options.pattern, options.ignoreCase ?? true) + const { totalMatches, matches } = scanConceptMatches(text, regex, limit) + + return { + conceptId, + title: getKnowledgeItemDisplayTitle(item), + itemType: item.type, + totalMatches, + matches + } + } + + /** + * Delete knowledge documents addressed by Concept ID (the deep-read/kb_manage + * write counterpart to {@link readConcept}). Each Concept ID is resolved to its + * knowledge item and deleted via the ingestion service's `deleteItems` (which + * expands to the outermost selected subtree); an id that does not resolve to a + * visible document in this base is reported in `notFound` rather than failing + * the whole batch. + */ + async deleteConcepts(baseId: string, conceptIds: string[]): Promise { + const { found, notFound } = await this.resolveConceptItemIds(baseId, conceptIds, 'deleteConcepts') + if (found.length > 0) { + await this.ingestionService.deleteItems( + baseId, + found.map((entry) => entry.itemId) + ) + } + return { applied: found.map((entry) => entry.conceptId), notFound } + } + + /** + * Re-index knowledge documents addressed by Concept ID. Each Concept ID is + * resolved to its knowledge item and re-indexed via the ingestion service's + * `reindexItems`; an id that does not resolve to a visible document in this + * base is reported in `notFound` rather than failing the whole batch. + */ + async refreshConcepts(baseId: string, conceptIds: string[]): Promise { + const { found, notFound } = await this.resolveConceptItemIds(baseId, conceptIds, 'refreshConcepts') + if (found.length > 0) { + await this.ingestionService.reindexItems( + baseId, + found.map((entry) => entry.itemId) + ) + } + return { applied: found.map((entry) => entry.conceptId), notFound } + } + + /** + * Build a knowledge base's organization tree from the `knowledge_item.groupId` + * hierarchy — directories are folders, file/url/note are leaves carrying a + * readable `conceptId` (when completed) for kb_read. This is the + * logical org layer the OKF `index.md` surfaces, deliberately decoupled from + * the flat physical `raw/` layout: no material scan, no path joins. Emitted as + * a pre-order DFS node list (each node's `depth` carries the hierarchy), + * capped at {@link KNOWLEDGE_TREE_MAX_NODES}. Throws NOT_FOUND if the base does + * not exist; available regardless of the base's runtime (search) state. + */ + getOrganizationTree(baseId: string, options: { maxDepth?: number } = {}): KnowledgeOrganizationTree { + knowledgeBaseService.getById(baseId) + + const items = knowledgeItemService.getItemsByBaseId(baseId) + + // Index children by their parent groupId (null = base root); each bucket keeps + // getItemsByBaseId's createdAt/id order, so siblings emit in a stable order. + const childrenByGroupId = new Map() + for (const item of items) { + const key = item.groupId ?? null + const bucket = childrenByGroupId.get(key) + if (bucket) { + bucket.push(item) + } else { + childrenByGroupId.set(key, [item]) + } + } + + const maxDepth = options.maxDepth + const nodes: KnowledgeTreeNode[] = [] + let truncated = false + + const walk = (groupId: string | null, depth: number): void => { + if (truncated || (maxDepth !== undefined && depth > maxDepth)) { + return + } + for (const item of childrenByGroupId.get(groupId) ?? []) { + if (nodes.length >= KNOWLEDGE_TREE_MAX_NODES) { + truncated = true + return + } + nodes.push({ + depth, + title: getKnowledgeItemDisplayTitle(item), + itemType: item.type, + status: item.status, + // Only a completed leaf is readable; directories and pending leaves carry no Concept ID. + conceptId: item.type !== 'directory' && item.status === 'completed' ? deriveConceptId(item) : undefined + }) + if (item.type === 'directory') { + walk(item.id, depth + 1) + } + } + } + + walk(null, 0) + + return { baseId, totalItems: items.length, truncated, nodes } + } + + /** + * Resolve a batch of Concept IDs to their knowledge item ids for a Concept + * ID-addressed mutation. Mirrors {@link resolveConcept}'s identity boundary — + * each relative path is looked up in this base's index store, then re-validated + * against the visible knowledge_item (same base, completed) — but resolves many + * ids and partitions them into resolved (`found`) vs unresolved (`notFound`) + * instead of throwing, so one bad id does not sink the batch. Duplicate ids in + * the input are collapsed to a single resolution. + */ + private async resolveConceptItemIds( + baseId: string, + conceptIds: string[], + operation: string + ): Promise<{ found: Array<{ conceptId: string; itemId: string }>; notFound: string[] }> { + const base = knowledgeBaseService.getById(baseId) + const vectorStoreService = application.get('KnowledgeVectorStoreService') + const store = vectorStoreService.getIndexStore(base) + + const found: Array<{ conceptId: string; itemId: string }> = [] + const notFound: string[] = [] + const seen = new Set() + + for (const conceptId of conceptIds) { + if (seen.has(conceptId)) { + continue + } + seen.add(conceptId) + + const ref = await runStoreOperation(store, baseId, operation, () => store.getMaterialByRelativePath(conceptId)) + // Identity re-check: the resolved material id must still be a visible item in this base. + const item = ref ? loadVisibleItems(baseId, [ref.materialId]).get(ref.materialId) : undefined + if (ref && item) { + found.push({ conceptId, itemId: ref.materialId }) + } else { + notFound.push(conceptId) + } + } + + return { found, notFound } + } + + /** + * Resolve a Concept ID to its content text for the deep-read tools. The tools + * address by relative path (a transparent DB key, never an fs path), so this + * re-validates the resolved material against the visible knowledge_item (same + * base, completed) before returning any text — the identity boundary that keeps + * a relative path from reaching another base's or a deleted item's content. + * Throws NOT_FOUND when the concept does not resolve to a readable, visible document. + */ + private async resolveConcept( + baseId: string, + conceptId: string, + operation: string + ): Promise<{ item: KnowledgeItem; text: string }> { + const base = assertBaseCanRunRuntimeOperation(baseId, operation) + + const vectorStoreService = application.get('KnowledgeVectorStoreService') + const store = vectorStoreService.getIndexStore(base) + + const ref = await runStoreOperation(store, baseId, operation, () => store.getMaterialByRelativePath(conceptId)) + if (!ref) { + throw DataApiErrorFactory.notFound('Knowledge concept', conceptId) + } + + const item = loadVisibleItems(baseId, [ref.materialId]).get(ref.materialId) + if (!item) { + throw DataApiErrorFactory.notFound('Knowledge concept', conceptId) + } + + const text = await runStoreOperation(store, baseId, operation, () => store.readMaterialContent(ref.materialId)) + if (text == null) { + // The material resolved and the item is visible + completed, yet it has no content row — an + // invariant violation or a reindex TOCTOU race, NOT a bad conceptId. Use a distinct resource so the + // tool layer (conceptLookupError) can steer "retry shortly" instead of "verify the conceptId", which + // cannot fix a missing content row. The resource string mirrors KNOWLEDGE_CONCEPT_CONTENT_NOT_FOUND_RESOURCE + // in knowledgeLookup.ts. + logger.warn('resolveConcept: visible completed item has no content row', { baseId, conceptId, operation }) + throw DataApiErrorFactory.notFound('Knowledge concept content', conceptId) + } + + return { item, text } + } +} diff --git a/src/main/features/knowledge/query/KnowledgeQueryService.ts b/src/main/features/knowledge/query/KnowledgeQueryService.ts new file mode 100644 index 00000000000..557582c845a --- /dev/null +++ b/src/main/features/knowledge/query/KnowledgeQueryService.ts @@ -0,0 +1,207 @@ +import { application } from '@application' +import { knowledgeBaseService } from '@data/services/KnowledgeBaseService' +import { knowledgeItemService } from '@data/services/KnowledgeItemService' +import { loggerService } from '@logger' +import { TraceMethod } from '@mcp-trace/trace-core' +import { DataApiErrorFactory } from '@shared/data/api' +import type { KnowledgeItem, KnowledgeItemChunk, KnowledgeSearchResult } from '@shared/data/types/knowledge' +import { getKnowledgeItemDisplayTitle, isCompletedVectorKnowledgeBase } from '@shared/data/types/knowledge' +import { estimateTokenCount } from 'tokenx' + +import { assertBaseCanRunRuntimeOperation } from '../base/baseGuards' +import { embedKnowledgeQuery } from '../pipeline/indexing/embed' +import { rerankKnowledgeSearchResults } from '../pipeline/indexing/rerank' +import { extractFtsTokens } from '../pipeline/vectorstore/indexStore/ftsQuery' +import type { KnowledgeIndexSearchMatch } from '../pipeline/vectorstore/indexStore/model' +import { toKnowledgeBaseId, toKnowledgeItemId } from '../types' +import { applyRelevanceThreshold, getInitialSearchScoreKind, withSearchRanks } from './search' +import { runStoreOperation } from './storeOperation' +import { deriveConceptId, loadVisibleItems } from './visibility' + +const logger = loggerService.withContext('Knowledge:QueryService') +/** + * Fetch this many × the requested result count as index candidates. The index + * store only filters by material state; the item-visibility filter (missing / + * other-base / not-completed) runs afterwards in the caller and can drop matches, + * so over-fetching keeps the final set from shrinking below topK. + */ +const KNOWLEDGE_SEARCH_OVERFETCH_FACTOR = 5 +/** Hard ceiling on fetched candidates, bounding the brute-force vector scan and rerank cost regardless of topK. */ +const KNOWLEDGE_SEARCH_CANDIDATE_CAP = 200 + +/** Read side of the knowledge feature: index search (with visibility filtering + rerank) and chunk/item listing. */ +export class KnowledgeQueryService { + @TraceMethod({ spanName: 'Knowledge.search', tag: 'Knowledge' }) + async search(baseId: string, query: string): Promise { + const base = assertBaseCanRunRuntimeOperation(baseId, 'search') + + // Same tokenization the FTS layer uses: no token means no BM25 hit is even possible. + if (extractFtsTokens(query).length === 0) { + throw DataApiErrorFactory.validation( + { query: ['Query has no searchable tokens'] }, + 'Query has no searchable tokens' + ) + } + + // Stored search mode and the index store's mode are the same enum now, so no mapping. + // Vector/hybrid retrieval needs an embedding model; a base without one is BM25-only + // regardless of its stored preference. + const mode = isCompletedVectorKnowledgeBase(base) ? base.searchMode : 'bm25' + // BM25 is lexical only; skip the embedding round-trip when the query won't use it. + const queryEmbedding = mode === 'bm25' ? undefined : await embedKnowledgeQuery(base, query) + + const resolvedTopK = base.documentCount ?? 10 + const candidateLimit = Math.min(resolvedTopK * KNOWLEDGE_SEARCH_OVERFETCH_FACTOR, KNOWLEDGE_SEARCH_CANDIDATE_CAP) + + const vectorStoreService = application.get('KnowledgeVectorStoreService') + const store = vectorStoreService.getIndexStore(base) + const matches = await runStoreOperation(store, baseId, 'search', () => + store.search({ + queryText: query, + queryEmbedding, + mode, + topK: candidateLimit, + alpha: base.hybridAlpha + }) + ) + + const scoreKind = getInitialSearchScoreKind(base) + const visibleSearchResults = this.toVisibleSearchResults(baseId, matches, scoreKind) + const topResults = this.trimToTopK(visibleSearchResults, resolvedTopK, baseId) + + const rerankedResults = await rerankKnowledgeSearchResults(base, query, topResults) + return withSearchRanks(applyRelevanceThreshold(rerankedResults, base.threshold)) + } + + async listItemChunks(baseId: string, itemId: string): Promise { + const knowledgeBaseId = toKnowledgeBaseId(baseId) + const knowledgeItemId = toKnowledgeItemId(itemId) + assertBaseCanRunRuntimeOperation(knowledgeBaseId, 'listItemChunks') + const item = this.assertItemCanRunChunkOperation(knowledgeBaseId, knowledgeItemId) + this.assertCompletedContainerHasNoDeletingChildren(knowledgeBaseId, item) + + const base = knowledgeBaseService.getById(knowledgeBaseId) + const leafItems = knowledgeItemService.getSubtreeItems(knowledgeBaseId, [knowledgeItemId], { + includeRoots: true, + leafOnly: true + }) + if (leafItems.length === 0) { + return [] + } + + const vectorStoreService = application.get('KnowledgeVectorStoreService') + const store = vectorStoreService.getIndexStore(base) + const chunkGroups = await runStoreOperation(store, knowledgeBaseId, 'listItemChunks', () => + leafItems.map((leafItem) => { + const units = store.listMaterialUnits(leafItem.id) + return units.map( + (unit): KnowledgeItemChunk => ({ + id: unit.unitId, + itemId: leafItem.id, + content: unit.text, + metadata: { + itemId: leafItem.id, + itemType: leafItem.type, + source: leafItem.data.source, + chunkIndex: unit.unitIndex, + tokenCount: estimateTokenCount(unit.text) + } + }) + ) + }) + ) + + return chunkGroups.flat() + } + + listRootItems(baseId: string): KnowledgeItem[] { + return knowledgeItemService.getRootItemsByBaseId(baseId) + } + + /** + * Turn raw index matches into visible search results: fetch each match's + * knowledge item once, drop any that is missing, in another base, or not + * completed, and reconstruct the chunk metadata (item type / source from the + * item; chunk index from the unit; token count recomputed from the body). + */ + private toVisibleSearchResults( + baseId: string, + matches: KnowledgeIndexSearchMatch[], + scoreKind: KnowledgeSearchResult['scoreKind'] + ): KnowledgeSearchResult[] { + const itemsById = loadVisibleItems( + baseId, + matches.map((match) => match.materialId) + ) + + const results: KnowledgeSearchResult[] = [] + for (const match of matches) { + const item = itemsById.get(match.materialId) + if (!item) { + continue + } + results.push({ + pageContent: match.text, + score: match.score, + scoreKind, + rank: results.length + 1, + metadata: { + itemId: match.materialId, + itemType: item.type, + source: item.data.source, + chunkIndex: match.unitIndex, + tokenCount: estimateTokenCount(match.text) + }, + itemId: match.materialId, + chunkId: match.unitId, + // Concept ID + title so a hit can be followed up with kb_read. + conceptId: deriveConceptId(item), + title: getKnowledgeItemDisplayTitle(item) + }) + } + return results + } + + /** Keep the highest-scored `topK` visible results, discarding the over-fetched tail. */ + private trimToTopK(results: KnowledgeSearchResult[], topK: number, baseId: string): KnowledgeSearchResult[] { + if (results.length <= topK) { + return results + } + logger.debug('Trimmed over-fetched knowledge search results to topK', { + baseId, + visibleCandidates: results.length, + topK + }) + return results.slice(0, topK) + } + + private assertItemCanRunChunkOperation(baseId: string, itemId: string): KnowledgeItem { + const item = knowledgeItemService.getById(itemId) + if (item.baseId !== baseId) { + throw new Error(`Knowledge item '${itemId}' does not belong to base '${baseId}'`) + } + + if (item.status !== 'completed') { + throw DataApiErrorFactory.validation( + { item: [`Knowledge item '${itemId}' must be completed before list chunks`] }, + 'Cannot list chunks for a non-completed knowledge item' + ) + } + + return item + } + + private assertCompletedContainerHasNoDeletingChildren(baseId: string, item: KnowledgeItem): void { + if (item.type !== 'directory') { + return + } + + const subtreeItems = knowledgeItemService.getSubtreeItems(baseId, [item.id]) + if (subtreeItems.some((item) => item.status === 'deleting')) { + throw DataApiErrorFactory.validation( + { item: [`Knowledge item subtree '${item.id}' is being deleted`] }, + 'Cannot list chunks for a deleting knowledge item' + ) + } + } +} diff --git a/src/main/features/knowledge/utils/__tests__/search.test.ts b/src/main/features/knowledge/query/__tests__/search.test.ts similarity index 100% rename from src/main/features/knowledge/utils/__tests__/search.test.ts rename to src/main/features/knowledge/query/__tests__/search.test.ts diff --git a/src/main/features/knowledge/utils/search.ts b/src/main/features/knowledge/query/search.ts similarity index 100% rename from src/main/features/knowledge/utils/search.ts rename to src/main/features/knowledge/query/search.ts diff --git a/src/main/features/knowledge/query/storeOperation.ts b/src/main/features/knowledge/query/storeOperation.ts new file mode 100644 index 00000000000..bbc26a4bdb9 --- /dev/null +++ b/src/main/features/knowledge/query/storeOperation.ts @@ -0,0 +1,33 @@ +import { loggerService } from '@logger' +import { DataApiErrorFactory } from '@shared/data/api' + +import type { KnowledgeIndexStore } from '../pipeline/vectorstore/indexStore/KnowledgeIndexStore' + +const logger = loggerService.withContext('Knowledge:Query') + +/** + * Run a per-base index-store interaction, translating the error raised when the + * store is closed mid-flight — a concurrent `deleteBase` or app shutdown + * closed the driver — into a defined, retryable DataApiError instead of leaking + * the opaque driver-level error to the renderer. Genuine query errors rethrow + * unchanged. + */ +export async function runStoreOperation( + store: KnowledgeIndexStore, + baseId: string, + operation: string, + run: () => T | Promise +): Promise { + try { + return await run() + } catch (error) { + if (store.isClosed()) { + logger.warn('Knowledge index store was closed during operation', { baseId, operation }) + throw DataApiErrorFactory.invalidOperation( + operation, + `Knowledge base '${baseId}' index store was closed during ${operation}; retry the operation` + ) + } + throw error + } +} diff --git a/src/main/features/knowledge/query/visibility.ts b/src/main/features/knowledge/query/visibility.ts new file mode 100644 index 00000000000..79a021e8073 --- /dev/null +++ b/src/main/features/knowledge/query/visibility.ts @@ -0,0 +1,55 @@ +import { knowledgeItemService } from '@data/services/KnowledgeItemService' +import { loggerService } from '@logger' +import { isDataApiNotFoundError } from '@shared/data/api' +import type { KnowledgeItem } from '@shared/data/types/knowledge' + +import { toMaterialRelativePath } from '../items' + +const logger = loggerService.withContext('Knowledge:Query') + +/** + * Concept ID (relative path, OKF §2) of a search hit's source document, or + * undefined when it has none: a directory (no material), or a url/note whose + * snapshot relativePath was not captured yet. A completed leaf normally has one, + * so this only swallows the rare unindexed-snapshot case rather than throwing + * out of the whole search. + */ +export function deriveConceptId(item: KnowledgeItem): string | undefined { + if (item.type === 'directory') { + return undefined + } + try { + return toMaterialRelativePath(item) + } catch (error) { + // A completed url/note with no snapshot relativePath is an invariant violation. Swallow it (one + // unfollowable hit must not sink the whole search) but leave a diagnostic trail rather than a silent drop. + logger.warn('deriveConceptId: completed item has no material relativePath', { + itemId: item.id, + type: item.type, + error: error instanceof Error ? error.message : String(error) + }) + return undefined + } +} + +/** Fetch the distinct items behind the matches, keeping only those visible in this base (same base, completed). */ +export function loadVisibleItems(baseId: string, materialIds: string[]): Map { + const uniqueIds = [...new Set(materialIds)] + const visibleItems = new Map() + + for (const materialId of uniqueIds) { + try { + const item = knowledgeItemService.getById(materialId) + if (item.baseId === baseId && item.status === 'completed') { + visibleItems.set(materialId, item) + } + } catch (error) { + if (isDataApiNotFoundError(error)) { + continue + } + throw error + } + } + + return visibleItems +} diff --git a/src/main/features/knowledge/tasks/__tests__/checkFileProcessingResultJobHandler.test.ts b/src/main/features/knowledge/tasks/__tests__/checkFileProcessingResultJobHandler.test.ts index 76989fea65d..fa4f6c78f1c 100644 --- a/src/main/features/knowledge/tasks/__tests__/checkFileProcessingResultJobHandler.test.ts +++ b/src/main/features/knowledge/tasks/__tests__/checkFileProcessingResultJobHandler.test.ts @@ -13,12 +13,12 @@ import { createJobSnapshot, FILE_ITEM_ID, getJobMock, + ingestionService, knowledgeItemGetByIdMock, knowledgeItemUpdateIndexedRelativePathMock, knowledgeItemUpdateStatusMock, knowledgeLockManager, - PROCESSED_RELATIVE_PATH, - workflowService + PROCESSED_RELATIVE_PATH } from './jobHandlerTestUtils' function createFileProcessingJobSnapshot(overrides: Partial> = {}) { @@ -48,14 +48,14 @@ function createCheckPayload( fileProcessingJobId: 'fp-job-1', pollRound: 0, firstScheduledAt: Date.now(), - parentJobId: null, + processedRelativePath: PROCESSED_RELATIVE_PATH, ...overrides } } describe('check-file-processing-result job handler', () => { it('declares the knowledge check job contract', () => { - const handler = createCheckFileProcessingResultJobHandler(knowledgeLockManager as never, workflowService as never) + const handler = createCheckFileProcessingResultJobHandler(knowledgeLockManager as never, ingestionService) expect(handler.recovery).toBe('abandon') expect(handler.defaultQueue?.(createCheckPayload())).toBe('base.kb-1') @@ -69,7 +69,7 @@ describe('check-file-processing-result job handler', () => { }) it('reschedules delayed polling while file processing is active', async () => { - const handler = createCheckFileProcessingResultJobHandler(knowledgeLockManager as never, workflowService as never) + const handler = createCheckFileProcessingResultJobHandler(knowledgeLockManager as never, ingestionService) knowledgeItemGetByIdMock.mockReturnValue(createFileItem()) getJobMock.mockResolvedValue(createFileProcessingJobSnapshot({ status: 'running' })) @@ -82,17 +82,18 @@ describe('check-file-processing-result job handler', () => { ) await handler.execute(ctx) - expect(workflowService.scheduleFileProcessingCheck).toHaveBeenCalledWith('kb-1', FILE_ITEM_ID, 'fp-job-1', { + expect(ingestionService.scheduleFileProcessingCheck).toHaveBeenCalledWith('kb-1', FILE_ITEM_ID, 'fp-job-1', { pollRound: 3, firstScheduledAt, - parentJobId: 'job-1' + parentJobId: 'job-1', + processedRelativePath: PROCESSED_RELATIVE_PATH }) - expect(workflowService.scheduleIndexing).not.toHaveBeenCalled() + expect(ingestionService.scheduleIndexing).not.toHaveBeenCalled() expect(ctx.reportProgress).toHaveBeenCalledWith(0, { stage: 'waiting', pollRound: 3 }) }) it('keeps polling follow-ups attached to the original workflow parent', async () => { - const handler = createCheckFileProcessingResultJobHandler(knowledgeLockManager as never, workflowService as never) + const handler = createCheckFileProcessingResultJobHandler(knowledgeLockManager as never, ingestionService) knowledgeItemGetByIdMock.mockReturnValue(createFileItem()) getJobMock.mockResolvedValue(createFileProcessingJobSnapshot({ status: 'running' })) @@ -101,22 +102,23 @@ describe('check-file-processing-result job handler', () => { createCtx( createCheckPayload({ pollRound: 2, - firstScheduledAt, - parentJobId: 'reindex-job' + firstScheduledAt }), - 'check-job-2' + 'check-job-2', + 'reindex-job' ) ) - expect(workflowService.scheduleFileProcessingCheck).toHaveBeenCalledWith('kb-1', FILE_ITEM_ID, 'fp-job-1', { + expect(ingestionService.scheduleFileProcessingCheck).toHaveBeenCalledWith('kb-1', FILE_ITEM_ID, 'fp-job-1', { pollRound: 3, firstScheduledAt, - parentJobId: 'reindex-job' + parentJobId: 'reindex-job', + processedRelativePath: PROCESSED_RELATIVE_PATH }) }) it('mirrors file-processing progress while polling', async () => { - const handler = createCheckFileProcessingResultJobHandler(knowledgeLockManager as never, workflowService as never) + const handler = createCheckFileProcessingResultJobHandler(knowledgeLockManager as never, ingestionService) knowledgeItemGetByIdMock.mockReturnValue(createFileItem()) getJobMock.mockResolvedValue(createFileProcessingJobSnapshot({ status: 'running' })) MockMainCacheServiceUtils.setSharedCacheValue(`${JOB_PROGRESS_KEY_PREFIX}fp-job-1`, { @@ -145,7 +147,7 @@ describe('check-file-processing-result job handler', () => { }) it('marks the item failed when file processing exceeds the wait limit', async () => { - const handler = createCheckFileProcessingResultJobHandler(knowledgeLockManager as never, workflowService as never) + const handler = createCheckFileProcessingResultJobHandler(knowledgeLockManager as never, ingestionService) knowledgeItemGetByIdMock.mockReturnValue(createFileItem()) getJobMock.mockResolvedValue(createFileProcessingJobSnapshot({ status: 'running' })) @@ -161,12 +163,12 @@ describe('check-file-processing-result job handler', () => { error: 'File processing job fp-job-1 did not finish within 30 minutes' }) expect(cancelMock).toHaveBeenCalledWith('fp-job-1', 'knowledge-file-processing-timeout') - expect(workflowService.scheduleFileProcessingCheck).not.toHaveBeenCalled() + expect(ingestionService.scheduleFileProcessingCheck).not.toHaveBeenCalled() expect(ctx.reportProgress).toHaveBeenCalledWith(100, { stage: 'failed' }) }) it('stores the processed artifact relative path and schedules indexing on completion', async () => { - const handler = createCheckFileProcessingResultJobHandler(knowledgeLockManager as never, workflowService as never) + const handler = createCheckFileProcessingResultJobHandler(knowledgeLockManager as never, ingestionService) knowledgeItemGetByIdMock.mockReturnValue(createFileItem()) getJobMock.mockResolvedValue( createFileProcessingJobSnapshot({ @@ -181,13 +183,13 @@ describe('check-file-processing-result job handler', () => { await handler.execute(ctx) expect(knowledgeItemUpdateIndexedRelativePathMock).toHaveBeenCalledWith(FILE_ITEM_ID, PROCESSED_RELATIVE_PATH) - expect(workflowService.scheduleIndexing).toHaveBeenCalledWith('kb-1', FILE_ITEM_ID, 'job-1') - expect(workflowService.scheduleFileProcessingCheck).not.toHaveBeenCalled() + expect(ingestionService.scheduleIndexing).toHaveBeenCalledWith('kb-1', FILE_ITEM_ID, 'job-1') + expect(ingestionService.scheduleFileProcessingCheck).not.toHaveBeenCalled() expect(ctx.reportProgress).toHaveBeenCalledWith(100, { stage: 'done' }) }) it('schedules indexing under the original workflow parent after polling completion', async () => { - const handler = createCheckFileProcessingResultJobHandler(knowledgeLockManager as never, workflowService as never) + const handler = createCheckFileProcessingResultJobHandler(knowledgeLockManager as never, ingestionService) knowledgeItemGetByIdMock.mockReturnValue(createFileItem()) getJobMock.mockResolvedValue( createFileProcessingJobSnapshot({ @@ -198,19 +200,13 @@ describe('check-file-processing-result job handler', () => { }) ) - await handler.execute( - createCtx( - createCheckPayload({ - parentJobId: 'reindex-job' - }) - ) - ) + await handler.execute(createCtx(createCheckPayload(), 'job-1', 'reindex-job')) - expect(workflowService.scheduleIndexing).toHaveBeenCalledWith('kb-1', FILE_ITEM_ID, 'reindex-job') + expect(ingestionService.scheduleIndexing).toHaveBeenCalledWith('kb-1', FILE_ITEM_ID, 'reindex-job') }) it('marks the item failed when the linked job is not the expected file-processing job', async () => { - const handler = createCheckFileProcessingResultJobHandler(knowledgeLockManager as never, workflowService as never) + const handler = createCheckFileProcessingResultJobHandler(knowledgeLockManager as never, ingestionService) knowledgeItemGetByIdMock.mockReturnValue(createFileItem()) getJobMock.mockResolvedValue( createFileProcessingJobSnapshot({ @@ -230,12 +226,12 @@ describe('check-file-processing-result job handler', () => { expect(knowledgeItemUpdateStatusMock).toHaveBeenCalledWith(FILE_ITEM_ID, 'failed', { error: 'Invalid file processing job for knowledge item: fp-job-1' }) - expect(workflowService.scheduleIndexing).not.toHaveBeenCalled() + expect(ingestionService.scheduleIndexing).not.toHaveBeenCalled() expect(ctx.reportProgress).toHaveBeenCalledWith(100, { stage: 'failed' }) }) it('skips attaching artifacts when the item becomes deleting before continuation side effects', async () => { - const handler = createCheckFileProcessingResultJobHandler(knowledgeLockManager as never, workflowService as never) + const handler = createCheckFileProcessingResultJobHandler(knowledgeLockManager as never, ingestionService) knowledgeItemGetByIdMock .mockReturnValueOnce(createFileItem()) .mockReturnValueOnce(createFileItem(FILE_ITEM_ID, 'deleting')) @@ -252,12 +248,12 @@ describe('check-file-processing-result job handler', () => { await handler.execute(ctx) expect(knowledgeItemUpdateIndexedRelativePathMock).not.toHaveBeenCalled() - expect(workflowService.scheduleIndexing).not.toHaveBeenCalled() + expect(ingestionService.scheduleIndexing).not.toHaveBeenCalled() expect(ctx.reportProgress).not.toHaveBeenCalledWith(100, { stage: 'done' }) }) it('marks the item failed when file processing fails', async () => { - const handler = createCheckFileProcessingResultJobHandler(knowledgeLockManager as never, workflowService as never) + const handler = createCheckFileProcessingResultJobHandler(knowledgeLockManager as never, ingestionService) knowledgeItemGetByIdMock.mockReturnValue(createFileItem()) getJobMock.mockResolvedValue( createFileProcessingJobSnapshot({ @@ -272,12 +268,12 @@ describe('check-file-processing-result job handler', () => { expect(knowledgeItemUpdateStatusMock).toHaveBeenCalledWith(FILE_ITEM_ID, 'failed', { error: 'File processing job fp-job-1 failed: processor failed' }) - expect(workflowService.scheduleIndexing).not.toHaveBeenCalled() + expect(ingestionService.scheduleIndexing).not.toHaveBeenCalled() expect(ctx.reportProgress).toHaveBeenCalledWith(100, { stage: 'failed' }) }) it('marks the item failed when the completed output has no markdown artifact', async () => { - const handler = createCheckFileProcessingResultJobHandler(knowledgeLockManager as never, workflowService as never) + const handler = createCheckFileProcessingResultJobHandler(knowledgeLockManager as never, ingestionService) knowledgeItemGetByIdMock.mockReturnValue(createFileItem()) getJobMock.mockResolvedValue( createFileProcessingJobSnapshot({ @@ -294,22 +290,22 @@ describe('check-file-processing-result job handler', () => { expect(knowledgeItemUpdateStatusMock).toHaveBeenCalledWith(FILE_ITEM_ID, 'failed', { error: 'Invalid file processing result for job fp-job-1' }) - expect(workflowService.scheduleIndexing).not.toHaveBeenCalled() + expect(ingestionService.scheduleIndexing).not.toHaveBeenCalled() }) it('cancels linked file-processing work before skipping deleting items', async () => { - const handler = createCheckFileProcessingResultJobHandler(knowledgeLockManager as never, workflowService as never) + const handler = createCheckFileProcessingResultJobHandler(knowledgeLockManager as never, ingestionService) knowledgeItemGetByIdMock.mockReturnValue(createFileItem(FILE_ITEM_ID, 'deleting')) await handler.execute(createCtx(createCheckPayload())) expect(cancelMock).toHaveBeenCalledWith('fp-job-1', 'knowledge-file-processing-item-unavailable') expect(knowledgeItemUpdateStatusMock).not.toHaveBeenCalled() - expect(workflowService.scheduleIndexing).not.toHaveBeenCalled() + expect(ingestionService.scheduleIndexing).not.toHaveBeenCalled() }) it('surfaces cancel failures before skipping deleting items', async () => { - const handler = createCheckFileProcessingResultJobHandler(knowledgeLockManager as never, workflowService as never) + const handler = createCheckFileProcessingResultJobHandler(knowledgeLockManager as never, ingestionService) knowledgeItemGetByIdMock.mockReturnValue(createFileItem(FILE_ITEM_ID, 'deleting')) cancelMock.mockRejectedValueOnce(new Error('cancel failed')) @@ -320,7 +316,7 @@ describe('check-file-processing-result job handler', () => { }) it('surfaces cancel timeouts before skipping deleting items', async () => { - const handler = createCheckFileProcessingResultJobHandler(knowledgeLockManager as never, workflowService as never) + const handler = createCheckFileProcessingResultJobHandler(knowledgeLockManager as never, ingestionService) knowledgeItemGetByIdMock.mockReturnValue(createFileItem(FILE_ITEM_ID, 'deleting')) cancelMock.mockResolvedValue({ outcome: 'timed-out' }) @@ -330,7 +326,7 @@ describe('check-file-processing-result job handler', () => { }) it('cancels linked file-processing work before skipping missing items', async () => { - const handler = createCheckFileProcessingResultJobHandler(knowledgeLockManager as never, workflowService as never) + const handler = createCheckFileProcessingResultJobHandler(knowledgeLockManager as never, ingestionService) knowledgeItemGetByIdMock.mockImplementation(() => { throw DataApiErrorFactory.notFound('KnowledgeItem', FILE_ITEM_ID) }) @@ -339,34 +335,11 @@ describe('check-file-processing-result job handler', () => { expect(cancelMock).toHaveBeenCalledWith('fp-job-1', 'knowledge-file-processing-item-unavailable') expect(knowledgeItemUpdateStatusMock).not.toHaveBeenCalled() - expect(workflowService.scheduleIndexing).not.toHaveBeenCalled() + expect(ingestionService.scheduleIndexing).not.toHaveBeenCalled() }) it('onSettled marks active items failed when the check job fails', async () => { - const handler = createCheckFileProcessingResultJobHandler(knowledgeLockManager as never, workflowService as never) - getJobMock.mockResolvedValue({ - id: 'job-1', - status: 'failed', - priority: 0, - queue: 'base.kb-1', - idempotencyKey: null, - scheduleId: null, - scheduledAt: '2026-04-08T00:00:00.000Z', - startedAt: '2026-04-08T00:00:00.000Z', - finishedAt: null, - attempt: 1, - maxAttempts: 3, - output: null, - error: null, - parentId: null, - cancelRequested: false, - metadata: {}, - timeoutMs: null, - createdAt: '2026-04-08T00:00:00.000Z', - updatedAt: '2026-04-08T00:00:00.000Z', - type: 'knowledge.check-file-processing-result', - input: createCheckPayload() - }) + const handler = createCheckFileProcessingResultJobHandler(knowledgeLockManager as never, ingestionService) knowledgeItemGetByIdMock.mockReturnValue(createFileItem()) await handler.onSettled?.({ @@ -385,30 +358,7 @@ describe('check-file-processing-result job handler', () => { }) it('onSettled falls back to the terminal status when a failed job has no error message', async () => { - const handler = createCheckFileProcessingResultJobHandler(knowledgeLockManager as never, workflowService as never) - getJobMock.mockResolvedValue({ - id: 'job-1', - status: 'failed', - priority: 0, - queue: 'base.kb-1', - idempotencyKey: null, - scheduleId: null, - scheduledAt: '2026-04-08T00:00:00.000Z', - startedAt: '2026-04-08T00:00:00.000Z', - finishedAt: null, - attempt: 1, - maxAttempts: 3, - output: null, - error: null, - parentId: null, - cancelRequested: false, - metadata: {}, - timeoutMs: null, - createdAt: '2026-04-08T00:00:00.000Z', - updatedAt: '2026-04-08T00:00:00.000Z', - type: 'knowledge.check-file-processing-result', - input: createCheckPayload() - }) + const handler = createCheckFileProcessingResultJobHandler(knowledgeLockManager as never, ingestionService) knowledgeItemGetByIdMock.mockReturnValue(createFileItem()) await handler.onSettled?.({ @@ -427,30 +377,7 @@ describe('check-file-processing-result job handler', () => { }) it('onSettled marks a cancelled (app-quit) item failed with the reindex-to-finish message', async () => { - const handler = createCheckFileProcessingResultJobHandler(knowledgeLockManager as never, workflowService as never) - getJobMock.mockResolvedValue({ - id: 'job-1', - status: 'cancelled', - priority: 0, - queue: 'base.kb-1', - idempotencyKey: null, - scheduleId: null, - scheduledAt: '2026-04-08T00:00:00.000Z', - startedAt: '2026-04-08T00:00:00.000Z', - finishedAt: null, - attempt: 1, - maxAttempts: 3, - output: null, - error: { code: 'CANCELLED', message: 'JobManager shutdown', retryable: false }, - parentId: null, - cancelRequested: false, - metadata: {}, - timeoutMs: null, - createdAt: '2026-04-08T00:00:00.000Z', - updatedAt: '2026-04-08T00:00:00.000Z', - type: 'knowledge.check-file-processing-result', - input: createCheckPayload() - }) + const handler = createCheckFileProcessingResultJobHandler(knowledgeLockManager as never, ingestionService) knowledgeItemGetByIdMock.mockReturnValue(createFileItem()) await handler.onSettled?.({ diff --git a/src/main/features/knowledge/tasks/__tests__/deleteSubtreeJobHandler.test.ts b/src/main/features/knowledge/tasks/__tests__/deleteSubtreeJobHandler.test.ts index 68243933c9b..b72f909f12a 100644 --- a/src/main/features/knowledge/tasks/__tests__/deleteSubtreeJobHandler.test.ts +++ b/src/main/features/knowledge/tasks/__tests__/deleteSubtreeJobHandler.test.ts @@ -37,7 +37,7 @@ describe('delete-subtree job handler', () => { createJobSnapshot({ id: 'index-job', type: 'knowledge.index-documents', - input: { baseId: 'kb-1', itemId: 'note-1', parentJobId: null } + input: { baseId: 'kb-1', itemId: 'note-1' } }), createJobSnapshot({ id: 'check-job', @@ -48,13 +48,13 @@ describe('delete-subtree job handler', () => { fileProcessingJobId: 'fp-job-1', pollRound: 0, firstScheduledAt: 1779811200000, - parentJobId: null + processedRelativePath: 'source.md' } }), createJobSnapshot({ id: 'unrelated-job', type: 'knowledge.index-documents', - input: { baseId: 'kb-1', itemId: 'other', parentJobId: null } + input: { baseId: 'kb-1', itemId: 'other' } }) ]) @@ -115,7 +115,7 @@ describe('delete-subtree job handler', () => { createJobSnapshot({ id: 'index-job', type: 'knowledge.index-documents', - input: { baseId: 'kb-1', itemId: 'note-1', parentJobId: null } + input: { baseId: 'kb-1', itemId: 'note-1' } }) ]) cancelMock.mockRejectedValue(new Error('cancel failed')) @@ -139,7 +139,7 @@ describe('delete-subtree job handler', () => { createJobSnapshot({ id: 'index-job', type: 'knowledge.index-documents', - input: { baseId: 'kb-1', itemId: 'note-1', parentJobId: null } + input: { baseId: 'kb-1', itemId: 'note-1' } }) ]) cancelMock.mockResolvedValue({ outcome: 'timed-out' }) diff --git a/src/main/features/knowledge/tasks/__tests__/indexDocumentsJobHandler.test.ts b/src/main/features/knowledge/tasks/__tests__/indexDocumentsJobHandler.test.ts index df0a0a471b1..255b8984460 100644 --- a/src/main/features/knowledge/tasks/__tests__/indexDocumentsJobHandler.test.ts +++ b/src/main/features/knowledge/tasks/__tests__/indexDocumentsJobHandler.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from 'vitest' -import { hashEmbeddingText } from '../../vectorstore/indexStore/hashing' -import type { RebuildMaterialInput } from '../../vectorstore/indexStore/model' +import { hashEmbeddingText } from '../../pipeline/vectorstore/indexStore/hashing' +import type { RebuildMaterialInput } from '../../pipeline/vectorstore/indexStore/model' import { captureNoteSnapshotFileMock, captureUrlSnapshotFileMock, @@ -10,14 +10,12 @@ import { createCtx, createFileItem, createIndexDocumentsJobHandler, - createJobSnapshot, createNoteItem, createUrlItem, embedKnowledgeTextsMock, fakeEmbedVector, fetchKnowledgeWebPageMock, FILE_ITEM_ID, - getJobMock, knowledgeBaseGetByIdMock, knowledgeItemGetByIdMock, knowledgeItemUpdateSnapshotRelativePathMock, @@ -47,7 +45,7 @@ describe('index-documents job handler', () => { knowledgeItemGetByIdMock.mockReturnValue(createNoteItem(NOTE_ITEM_ID)) knowledgeItemUpdateStatusMock.mockReturnValue(createNoteItem(NOTE_ITEM_ID)) - await handler.execute(createCtx({ baseId: 'kb-1', itemId: NOTE_ITEM_ID, parentJobId: null })) + await handler.execute(createCtx({ baseId: 'kb-1', itemId: NOTE_ITEM_ID })) expect(knowledgeItemUpdateStatusMock).toHaveBeenCalledWith(NOTE_ITEM_ID, 'reading') expect(knowledgeItemUpdateStatusMock).toHaveBeenCalledWith(NOTE_ITEM_ID, 'embedding') @@ -60,7 +58,12 @@ describe('index-documents job handler', () => { }) ) expect(knowledgeItemUpdateStatusMock).toHaveBeenCalledWith(NOTE_ITEM_ID, 'completed') - expect(handler.defaultQueue?.({ baseId: 'kb-1', itemId: NOTE_ITEM_ID, parentJobId: null })).toBe('base.kb-1') + expect(handler.defaultQueue?.({ baseId: 'kb-1', itemId: NOTE_ITEM_ID })).toBe('base.kb-1') + // README's lock-boundary contract: embedding (paid API call) must run before the + // mutation lock is taken, never inside it. + expect(embedKnowledgeTextsMock.mock.invocationCallOrder[0]).toBeLessThan( + knowledgeLockManager.runExclusive.mock.invocationCallOrder[0] + ) }) it('pairs every embedding vector with the hash of the body it was computed from', async () => { @@ -69,7 +72,7 @@ describe('index-documents job handler', () => { knowledgeItemUpdateStatusMock.mockReturnValue(createNoteItem(NOTE_ITEM_ID)) loadKnowledgeItemDocumentsMock.mockResolvedValueOnce(distinctDocuments()) - await handler.execute(createCtx({ baseId: 'kb-1', itemId: NOTE_ITEM_ID, parentJobId: null })) + await handler.execute(createCtx({ baseId: 'kb-1', itemId: NOTE_ITEM_ID })) const input = lastRebuildInput() expect(input.embeddings.length).toBeGreaterThanOrEqual(3) @@ -95,9 +98,9 @@ describe('index-documents job handler', () => { loadKnowledgeItemDocumentsMock.mockResolvedValueOnce(distinctDocuments()) // 'bravo' is already in the index; reindexing must not re-embed it. const storedHash = hashEmbeddingText('bravo') - listExistingEmbeddingHashesMock.mockResolvedValueOnce(new Set([storedHash])) + listExistingEmbeddingHashesMock.mockReturnValueOnce(new Set([storedHash])) - await handler.execute(createCtx({ baseId: 'kb-1', itemId: NOTE_ITEM_ID, parentJobId: null })) + await handler.execute(createCtx({ baseId: 'kb-1', itemId: NOTE_ITEM_ID })) // The paid embed call received only the two missing bodies. const embeddedBodies = embedKnowledgeTextsMock.mock.calls[0][1] as string[] @@ -116,9 +119,9 @@ describe('index-documents job handler', () => { knowledgeItemGetByIdMock.mockReturnValue(createNoteItem(NOTE_ITEM_ID)) knowledgeItemUpdateStatusMock.mockReturnValue(createNoteItem(NOTE_ITEM_ID)) loadKnowledgeItemDocumentsMock.mockResolvedValueOnce(distinctDocuments()) - listExistingEmbeddingHashesMock.mockResolvedValueOnce(new Set(DISTINCT_DOCS.map(hashEmbeddingText))) + listExistingEmbeddingHashesMock.mockReturnValueOnce(new Set(DISTINCT_DOCS.map(hashEmbeddingText))) - await handler.execute(createCtx({ baseId: 'kb-1', itemId: NOTE_ITEM_ID, parentJobId: null })) + await handler.execute(createCtx({ baseId: 'kb-1', itemId: NOTE_ITEM_ID })) // The paid embed seam receives zero bodies (embedKnowledgeTexts itself // short-circuits an empty input before AiService, pinned in embed.test.ts), @@ -142,7 +145,7 @@ describe('index-documents job handler', () => { knowledgeItemUpdateStatusMock.mockReturnValue(createNoteItem(NOTE_ITEM_ID)) loadKnowledgeItemDocumentsMock.mockResolvedValueOnce(distinctDocuments()) - await handler.execute(createCtx({ baseId: 'kb-1', itemId: NOTE_ITEM_ID, parentJobId: null })) + await handler.execute(createCtx({ baseId: 'kb-1', itemId: NOTE_ITEM_ID })) // No paid embed round-trip and no existing-hash lookup for a lexical base. expect(embedKnowledgeTextsMock).not.toHaveBeenCalled() @@ -161,7 +164,7 @@ describe('index-documents job handler', () => { knowledgeItemUpdateStatusMock.mockReturnValue(createNoteItem(NOTE_ITEM_ID)) loadKnowledgeItemDocumentsMock.mockResolvedValueOnce([]) - await handler.execute(createCtx({ baseId: 'kb-1', itemId: NOTE_ITEM_ID, parentJobId: null })) + await handler.execute(createCtx({ baseId: 'kb-1', itemId: NOTE_ITEM_ID })) // An image-only PDF or failed extraction must leave a diagnosable trace — // without the warn it would look indexed while matching nothing. @@ -180,7 +183,7 @@ describe('index-documents job handler', () => { knowledgeItemGetByIdMock.mockReturnValue(fileItem) knowledgeItemUpdateStatusMock.mockReturnValue(fileItem) - await handler.execute(createCtx({ baseId: 'kb-1', itemId: FILE_ITEM_ID, parentJobId: null })) + await handler.execute(createCtx({ baseId: 'kb-1', itemId: FILE_ITEM_ID })) expect(lastRebuildInput().material.relativePath).toBe('source.md') }) @@ -189,13 +192,7 @@ describe('index-documents job handler', () => { const handler = createIndexDocumentsJobHandler(knowledgeLockManager as never) knowledgeItemGetByIdMock.mockReturnValue(createFileItem(FILE_ITEM_ID)) - await handler.execute( - createCtx({ - baseId: 'kb-1', - itemId: FILE_ITEM_ID, - parentJobId: null - }) - ) + await handler.execute(createCtx({ baseId: 'kb-1', itemId: FILE_ITEM_ID })) expect(loadKnowledgeItemDocumentsMock).toHaveBeenCalledWith(expect.objectContaining({ id: FILE_ITEM_ID })) }) @@ -206,7 +203,7 @@ describe('index-documents job handler', () => { knowledgeItemUpdateStatusMock.mockReturnValue(createNoteItem(NOTE_ITEM_ID)) loadKnowledgeItemDocumentsMock.mockResolvedValueOnce([]) - await handler.execute(createCtx({ baseId: 'kb-1', itemId: NOTE_ITEM_ID, parentJobId: null })) + await handler.execute(createCtx({ baseId: 'kb-1', itemId: NOTE_ITEM_ID })) expect(knowledgeItemUpdateStatusMock).toHaveBeenCalledWith(NOTE_ITEM_ID, 'reading') expect(knowledgeItemUpdateStatusMock).toHaveBeenCalledWith(NOTE_ITEM_ID, 'embedding') @@ -223,7 +220,7 @@ describe('index-documents job handler', () => { .mockReturnValueOnce(createNoteItem(NOTE_ITEM_ID)) .mockReturnValueOnce(createNoteItem(NOTE_ITEM_ID, null, 'deleting')) - await handler.execute(createCtx({ baseId: 'kb-1', itemId: NOTE_ITEM_ID, parentJobId: null })) + await handler.execute(createCtx({ baseId: 'kb-1', itemId: NOTE_ITEM_ID })) expect(rebuildMaterialMock).not.toHaveBeenCalled() expect(knowledgeItemUpdateStatusMock).not.toHaveBeenCalledWith(NOTE_ITEM_ID, 'completed') @@ -232,11 +229,13 @@ describe('index-documents job handler', () => { it('does not mark completed when vector replacement fails', async () => { const handler = createIndexDocumentsJobHandler(knowledgeLockManager as never) knowledgeItemGetByIdMock.mockReturnValue(createNoteItem(NOTE_ITEM_ID)) - rebuildMaterialMock.mockRejectedValueOnce(new Error('vector write failed')) + rebuildMaterialMock.mockImplementationOnce(() => { + throw new Error('vector write failed') + }) - await expect( - handler.execute(createCtx({ baseId: 'kb-1', itemId: NOTE_ITEM_ID, parentJobId: null })) - ).rejects.toThrow('vector write failed') + await expect(handler.execute(createCtx({ baseId: 'kb-1', itemId: NOTE_ITEM_ID }))).rejects.toThrow( + 'vector write failed' + ) expect(knowledgeItemUpdateStatusMock).not.toHaveBeenCalledWith(NOTE_ITEM_ID, 'completed') }) @@ -244,9 +243,7 @@ describe('index-documents job handler', () => { it('stops before side effects when aborted before execution', async () => { const handler = createIndexDocumentsJobHandler(knowledgeLockManager as never) - await expect( - handler.execute(createAbortedCtx({ baseId: 'kb-1', itemId: NOTE_ITEM_ID, parentJobId: null })) - ).rejects.toThrow() + await expect(handler.execute(createAbortedCtx({ baseId: 'kb-1', itemId: NOTE_ITEM_ID }))).rejects.toThrow() expect(knowledgeBaseGetByIdMock).not.toHaveBeenCalled() expect(rebuildMaterialMock).not.toHaveBeenCalled() @@ -260,7 +257,7 @@ describe('index-documents job handler', () => { knowledgeItemGetByIdMock.mockReturnValue(createUrlItem('url-1')) captureUrlSnapshotFileMock.mockResolvedValue('example-page.md') - await handler.execute(createCtx({ baseId: 'kb-1', itemId: 'url-1', parentJobId: null })) + await handler.execute(createCtx({ baseId: 'kb-1', itemId: 'url-1' })) // Fetched exactly once, snapshot written, relativePath persisted. expect(fetchKnowledgeWebPageMock).toHaveBeenCalledTimes(1) @@ -280,13 +277,18 @@ describe('index-documents job handler', () => { // item-id virtual placeholder — so it points at the bytes captureUrlSnapshotFile // wrote and agrees with what the v1→v2 migrator stamps for the same url. expect(lastRebuildInput().material.relativePath).toBe('example-page.md') + // README's lock-boundary contract: the network fetch must complete before the + // (first) mutation lock is taken, never inside it. + expect(fetchKnowledgeWebPageMock.mock.invocationCallOrder[0]).toBeLessThan( + knowledgeLockManager.runExclusive.mock.invocationCallOrder[0] + ) }) it('does not fetch a URL that already has a captured snapshot', async () => { const handler = createIndexDocumentsJobHandler(knowledgeLockManager as never) knowledgeItemGetByIdMock.mockReturnValue(createUrlItem('url-1', 'cached.md')) - await handler.execute(createCtx({ baseId: 'kb-1', itemId: 'url-1', parentJobId: null })) + await handler.execute(createCtx({ baseId: 'kb-1', itemId: 'url-1' })) expect(fetchKnowledgeWebPageMock).not.toHaveBeenCalled() expect(captureUrlSnapshotFileMock).not.toHaveBeenCalled() @@ -303,7 +305,7 @@ describe('index-documents job handler', () => { .mockReturnValueOnce(createUrlItem('url-1')) .mockReturnValueOnce(createUrlItem('url-1', 'raced.md')) - await handler.execute(createCtx({ baseId: 'kb-1', itemId: 'url-1', parentJobId: null })) + await handler.execute(createCtx({ baseId: 'kb-1', itemId: 'url-1' })) // Fetched before the lock, but the duplicate write/persist is skipped. expect(fetchKnowledgeWebPageMock).toHaveBeenCalledTimes(1) @@ -322,9 +324,7 @@ describe('index-documents job handler', () => { knowledgeItemGetByIdMock.mockReturnValue(createUrlItem('url-1')) fetchKnowledgeWebPageMock.mockResolvedValueOnce('') - await expect(handler.execute(createCtx({ baseId: 'kb-1', itemId: 'url-1', parentJobId: null }))).rejects.toThrow( - 'empty markdown' - ) + await expect(handler.execute(createCtx({ baseId: 'kb-1', itemId: 'url-1' }))).rejects.toThrow('empty markdown') expect(captureUrlSnapshotFileMock).not.toHaveBeenCalled() expect(knowledgeItemUpdateStatusMock).not.toHaveBeenCalledWith('url-1', 'completed') @@ -335,9 +335,7 @@ describe('index-documents job handler', () => { const emptyNote = { ...createNoteItem(NOTE_ITEM_ID), data: { source: 'My note', content: ' ' } } knowledgeItemGetByIdMock.mockReturnValue(emptyNote) - await expect( - handler.execute(createCtx({ baseId: 'kb-1', itemId: NOTE_ITEM_ID, parentJobId: null })) - ).rejects.toThrow('empty content') + await expect(handler.execute(createCtx({ baseId: 'kb-1', itemId: NOTE_ITEM_ID }))).rejects.toThrow('empty content') expect(captureNoteSnapshotFileMock).not.toHaveBeenCalled() expect(knowledgeItemUpdateStatusMock).not.toHaveBeenCalledWith(NOTE_ITEM_ID, 'completed') @@ -351,7 +349,7 @@ describe('index-documents job handler', () => { knowledgeItemGetByIdMock.mockReturnValue(noSnapshotNote) captureNoteSnapshotFileMock.mockResolvedValue('My note.md') - await handler.execute(createCtx({ baseId: 'kb-1', itemId: NOTE_ITEM_ID, parentJobId: null })) + await handler.execute(createCtx({ baseId: 'kb-1', itemId: NOTE_ITEM_ID })) // No network fetch; the in-hand content is written and the relativePath persisted. expect(fetchKnowledgeWebPageMock).not.toHaveBeenCalled() @@ -370,7 +368,7 @@ describe('index-documents job handler', () => { const handler = createIndexDocumentsJobHandler(knowledgeLockManager as never) knowledgeItemGetByIdMock.mockReturnValue(createNoteItem(NOTE_ITEM_ID, null, 'processing', 'cached-note.md')) - await handler.execute(createCtx({ baseId: 'kb-1', itemId: NOTE_ITEM_ID, parentJobId: null })) + await handler.execute(createCtx({ baseId: 'kb-1', itemId: NOTE_ITEM_ID })) expect(captureNoteSnapshotFileMock).not.toHaveBeenCalled() expect(knowledgeItemUpdateSnapshotRelativePathMock).not.toHaveBeenCalled() @@ -387,7 +385,7 @@ describe('index-documents job handler', () => { .mockReturnValueOnce(noSnapshotNote) .mockReturnValueOnce(createNoteItem(NOTE_ITEM_ID, null, 'processing', 'raced-note.md')) - await handler.execute(createCtx({ baseId: 'kb-1', itemId: NOTE_ITEM_ID, parentJobId: null })) + await handler.execute(createCtx({ baseId: 'kb-1', itemId: NOTE_ITEM_ID })) expect(captureNoteSnapshotFileMock).not.toHaveBeenCalled() expect(knowledgeItemUpdateSnapshotRelativePathMock).not.toHaveBeenCalled() @@ -399,13 +397,6 @@ describe('index-documents job handler', () => { it('onSettled skips failed status when the item is deleting', async () => { const handler = createIndexDocumentsJobHandler(knowledgeLockManager as never) - getJobMock.mockResolvedValue( - createJobSnapshot({ - id: 'index-job', - type: 'knowledge.index-documents', - input: { baseId: 'kb-1', itemId: 'note-1', parentJobId: null } - }) - ) knowledgeItemGetByIdMock.mockReturnValue(createNoteItem('note-1', null, 'deleting')) await handler.onSettled?.({ @@ -414,7 +405,7 @@ describe('index-documents job handler', () => { scheduleId: null, parentId: null, status: 'failed', - input: { baseId: 'kb-1', itemId: 'note-1', parentJobId: null }, + input: { baseId: 'kb-1', itemId: 'note-1' }, error: { code: 'FAILED', message: 'cancelled', retryable: false }, attempt: 1, metadata: {} diff --git a/src/main/features/knowledge/tasks/__tests__/jobHandlerTestUtils.ts b/src/main/features/knowledge/tasks/__tests__/jobHandlerTestUtils.ts index 506ac4e473d..02d32db8810 100644 --- a/src/main/features/knowledge/tasks/__tests__/jobHandlerTestUtils.ts +++ b/src/main/features/knowledge/tasks/__tests__/jobHandlerTestUtils.ts @@ -1,10 +1,12 @@ import type { JobContext } from '@main/core/job/types' +import type * as FsUtils from '@main/utils/file/fs' import type { JobSnapshot } from '@shared/data/api/schemas/jobs' import type { KnowledgeBase, KnowledgeItemOf } from '@shared/data/types/knowledge' import { MockMainCacheServiceUtils } from '@test-mocks/main/CacheService' -import { beforeEach, vi } from 'vitest' +import { beforeEach, type Mocked, vi } from 'vitest' -import type * as PathStorage from '../../utils/storage/pathStorage' +import type { KnowledgeItemScheduler } from '../../ingestion/KnowledgeIngestionService' +import type * as PathStorage from '../../pathStorage' const mocks = vi.hoisted(() => ({ cancelMock: vi.fn(), @@ -36,7 +38,8 @@ const mocks = vi.hoisted(() => ({ listExistingEmbeddingHashesMock: vi.fn(), embedKnowledgeTextsMock: vi.fn(), loggerWarnMock: vi.fn(), - scheduleItemMock: vi.fn() + scheduleItemMock: vi.fn(), + removeDirMock: vi.fn() })) export const { @@ -69,7 +72,8 @@ export const { listExistingEmbeddingHashesMock, embedKnowledgeTextsMock, loggerWarnMock, - scheduleItemMock + scheduleItemMock, + removeDirMock } = mocks /** @@ -131,28 +135,28 @@ vi.mock('@data/services/KnowledgeItemService', () => ({ } })) -vi.mock('../../readers/KnowledgeReader', () => ({ +vi.mock('../../pipeline/readers/KnowledgeReader', () => ({ loadKnowledgeItemDocuments: loadKnowledgeItemDocumentsMock })) -vi.mock('../../utils/sources/prepare', () => ({ +vi.mock('../prepareItem', () => ({ prepareKnowledgeItem: prepareKnowledgeItemMock })) -vi.mock('../../utils/sources/url', () => ({ +vi.mock('../../pipeline/sources/url', () => ({ fetchKnowledgeWebPage: fetchKnowledgeWebPageMock })) -vi.mock('../../utils/sources/urlSnapshot', () => ({ +vi.mock('../../pipeline/sources/urlSnapshot', () => ({ captureUrlSnapshotFile: captureUrlSnapshotFileMock })) -vi.mock('../../utils/sources/noteSnapshot', () => ({ +vi.mock('../../pipeline/sources/noteSnapshot', () => ({ captureNoteSnapshotFile: captureNoteSnapshotFileMock })) -vi.mock('../../utils/storage/pathStorage', async () => { - const actual = await vi.importActual('../../utils/storage/pathStorage') +vi.mock('../../pathStorage', async () => { + const actual = await vi.importActual('../../pathStorage') return { ...actual, // Stub the best-effort cleanup the handlers call. Its swallow-on-failure @@ -167,7 +171,17 @@ vi.mock('../../utils/storage/pathStorage', async () => { } }) -vi.mock('../../utils/indexing/embed', () => ({ +// prepare-root reclaims the container's own `raw/` shell via removeDir on retry; +// spy on it while keeping every other fs helper real. Default no-op resolve in beforeEach. +vi.mock('@main/utils/file/fs', async () => { + const actual = await vi.importActual('@main/utils/file/fs') + return { + ...actual, + removeDir: removeDirMock + } +}) + +vi.mock('../../pipeline/indexing/embed', () => ({ embedKnowledgeTexts: embedKnowledgeTextsMock })) @@ -280,12 +294,12 @@ export function createDirectoryItem( } } -export function createCtx(input: TInput, jobId = 'job-1'): JobContext { +export function createCtx(input: TInput, jobId = 'job-1', parentId: string | null = null): JobContext { return { jobId, input, attempt: 1, - parentId: null, + parentId, signal: new AbortController().signal, metadata: {}, patchMetadata: vi.fn().mockResolvedValue(undefined), @@ -334,10 +348,10 @@ export function createJobSnapshot(overrides: KnowledgeJobSnapshotInput): JobSnap } export const knowledgeLockManager = { - withBaseMutationLock: vi.fn(async (_baseId: string, task: () => Promise) => await task()) + runExclusive: vi.fn(async (_key: string, task: () => Promise) => await task()) } -export const workflowService = { +export const ingestionService: Mocked = { scheduleFileProcessingCheck: vi.fn(), scheduleIndexing: vi.fn(), scheduleItem: scheduleItemMock @@ -346,8 +360,8 @@ export const workflowService = { beforeEach(() => { vi.clearAllMocks() MockMainCacheServiceUtils.resetMocks() - knowledgeLockManager.withBaseMutationLock.mockImplementation( - async (_baseId: string, task: () => Promise) => await task() + knowledgeLockManager.runExclusive.mockImplementation( + async (_key: string, task: () => Promise) => await task() ) knowledgeBaseGetByIdMock.mockReturnValue(createBase()) knowledgeItemGetByIdMock.mockReturnValue(createNoteItem()) @@ -375,13 +389,13 @@ beforeEach(() => { reclaimSpace: reclaimSpaceMock, listExistingEmbeddingHashes: listExistingEmbeddingHashesMock } - getIndexStoreMock.mockResolvedValue(indexStore) - getIndexStoreIfExistsMock.mockResolvedValue(indexStore) - rebuildMaterialMock.mockResolvedValue(undefined) + getIndexStoreMock.mockReturnValue(indexStore) + getIndexStoreIfExistsMock.mockReturnValue(indexStore) + rebuildMaterialMock.mockReturnValue(undefined) deleteMaterialsMock.mockResolvedValue(undefined) - reclaimSpaceMock.mockResolvedValue({ vacuumed: false, reclaimedBytes: 0 }) + reclaimSpaceMock.mockReturnValue({ vacuumed: false, reclaimedBytes: 0 }) // No vectors stored yet by default → every chunk is embedded (prior behavior). - listExistingEmbeddingHashesMock.mockResolvedValue(new Set()) + listExistingEmbeddingHashesMock.mockReturnValue(new Set()) embedKnowledgeTextsMock.mockImplementation(async (_base: KnowledgeBase, values: string[]) => values.map(fakeEmbedVector) ) @@ -391,10 +405,11 @@ beforeEach(() => { knowledgeItemUpdateIndexedRelativePathMock.mockReturnValue(createFileItem()) deleteItemsByIdsMock.mockReturnValue(undefined) deleteKnowledgeItemFilesBestEffortMock.mockResolvedValue(undefined) + removeDirMock.mockResolvedValue(undefined) probeKnowledgeFileMock.mockResolvedValue('readable') probeKnowledgeSourcePathMock.mockResolvedValue('readable') cancelMock.mockResolvedValue({ outcome: 'cancelled' }) - workflowService.scheduleFileProcessingCheck.mockResolvedValue(undefined) - workflowService.scheduleIndexing.mockResolvedValue(undefined) + ingestionService.scheduleFileProcessingCheck.mockResolvedValue(undefined) + ingestionService.scheduleIndexing.mockResolvedValue(undefined) scheduleItemMock.mockResolvedValue({ id: 'scheduled-job' }) }) diff --git a/src/main/features/knowledge/utils/sources/__tests__/prepare.test.ts b/src/main/features/knowledge/tasks/__tests__/prepareItem.test.ts similarity index 67% rename from src/main/features/knowledge/utils/sources/__tests__/prepare.test.ts rename to src/main/features/knowledge/tasks/__tests__/prepareItem.test.ts index 18c4e7765a7..a93ac25fdf3 100644 --- a/src/main/features/knowledge/utils/sources/__tests__/prepare.test.ts +++ b/src/main/features/knowledge/tasks/__tests__/prepareItem.test.ts @@ -2,15 +2,17 @@ import type { KnowledgeItem } from '@shared/data/types/knowledge' import { beforeEach, describe, expect, it, vi } from 'vitest' const { + chooseDirectoryPathPrefixMock, expandDirectoryOwnerToTreeMock, - knowledgeItemCreateMock, + knowledgeItemCreateActiveMock, knowledgeItemUpdateStatusMock, knowledgeItemUpdateDirectoryRelativePathMock, knowledgeItemGetItemsByBaseIdMock, loggerWarnMock } = vi.hoisted(() => ({ + chooseDirectoryPathPrefixMock: vi.fn(), expandDirectoryOwnerToTreeMock: vi.fn(), - knowledgeItemCreateMock: vi.fn(), + knowledgeItemCreateActiveMock: vi.fn(), knowledgeItemUpdateStatusMock: vi.fn(), knowledgeItemUpdateDirectoryRelativePathMock: vi.fn(), knowledgeItemGetItemsByBaseIdMock: vi.fn(), @@ -19,7 +21,7 @@ const { vi.mock('@data/services/KnowledgeItemService', () => ({ knowledgeItemService: { - create: knowledgeItemCreateMock, + createActive: knowledgeItemCreateActiveMock, updateStatus: knowledgeItemUpdateStatusMock, updateDirectoryRelativePath: knowledgeItemUpdateDirectoryRelativePathMock, getItemsByBaseId: knowledgeItemGetItemsByBaseIdMock @@ -34,23 +36,22 @@ vi.mock('@logger', () => ({ } })) -vi.mock('../directory', () => ({ +vi.mock('../../pipeline/sources/directory', () => ({ + chooseDirectoryPathPrefix: chooseDirectoryPathPrefixMock, expandDirectoryOwnerToTree: expandDirectoryOwnerToTreeMock })) -import type { PrepareKnowledgeItemOptions } from '../prepare' +import type { PrepareKnowledgeItemOptions } from '../prepareItem' -const { prepareKnowledgeItem } = await import('../prepare') +const { prepareKnowledgeItem } = await import('../prepareItem') const baseId = 'kb-1' -function createPrepareOptions(item: KnowledgeItem, onCreatedItem = vi.fn()): PrepareKnowledgeItemOptions { +function createPrepareOptions(item: KnowledgeItem): PrepareKnowledgeItemOptions { const signal = new AbortController().signal return { baseId, item, - onCreatedItem, - runMutation: async (task) => await task(), signal } } @@ -104,15 +105,16 @@ describe('prepareKnowledgeItem', () => { beforeEach(() => { vi.clearAllMocks() - expandDirectoryOwnerToTreeMock.mockResolvedValue({ pathPrefix: 'docs', children: [] }) + chooseDirectoryPathPrefixMock.mockReturnValue('docs') + expandDirectoryOwnerToTreeMock.mockResolvedValue([]) knowledgeItemGetItemsByBaseIdMock.mockReturnValue([]) - knowledgeItemCreateMock.mockImplementation((_baseId: string, item: Partial) => ({ + knowledgeItemCreateActiveMock.mockImplementation((_baseId: string, item: Partial) => ({ id: `${item.type}-created`, baseId, groupId: item.groupId ?? null, type: item.type, data: item.data, - status: 'idle', + status: item.type === 'directory' ? 'preparing' : 'processing', error: null, createdAt: '2026-04-08T00:00:00.000Z', updatedAt: '2026-04-08T00:00:00.000Z' @@ -137,7 +139,7 @@ describe('prepareKnowledgeItem', () => { await expect(prepareKnowledgeItem(createPrepareOptions(note))).resolves.toEqual([note]) - expect(knowledgeItemCreateMock).not.toHaveBeenCalled() + expect(knowledgeItemCreateActiveMock).not.toHaveBeenCalled() expect(knowledgeItemUpdateStatusMock).not.toHaveBeenCalled() }) @@ -145,42 +147,47 @@ describe('prepareKnowledgeItem', () => { const root = createDirectoryItem('dir-root') const childDir = createDirectoryItem('dir-child', root.id) const childFile = createFileItem('file-child', childDir.id) - knowledgeItemCreateMock.mockReturnValueOnce(childDir).mockReturnValueOnce(childFile) - knowledgeItemUpdateStatusMock.mockReturnValueOnce(childDir).mockReturnValueOnce(childFile) - expandDirectoryOwnerToTreeMock.mockResolvedValueOnce({ - pathPrefix: 'dir-root-prefix', - children: [ - { - type: 'directory', - data: childDir.data, - children: [ - { - type: 'file', - data: childFile.data - } - ] - } - ] - }) + knowledgeItemCreateActiveMock.mockReturnValueOnce(childDir).mockReturnValueOnce(childFile) + chooseDirectoryPathPrefixMock.mockReturnValueOnce('dir-root-prefix') + expandDirectoryOwnerToTreeMock.mockResolvedValueOnce([ + { + type: 'directory', + data: childDir.data, + children: [ + { + type: 'file', + data: childFile.data + } + ] + } + ]) const options = createPrepareOptions(root) await expect(prepareKnowledgeItem(options)).resolves.toEqual([childFile]) - expect(expandDirectoryOwnerToTreeMock).toHaveBeenCalledWith(root, baseId, expect.any(Set), options.signal) + expect(expandDirectoryOwnerToTreeMock).toHaveBeenCalledWith(root, baseId, 'dir-root-prefix', options.signal) expect(knowledgeItemUpdateDirectoryRelativePathMock).toHaveBeenCalledWith(root.id, 'dir-root-prefix') - expect(knowledgeItemCreateMock).toHaveBeenNthCalledWith(1, baseId, { + // The container prefix must be pinned BEFORE any byte is copied (expansion) or any child + // row is created, so a mid-expansion crash leaves the pinned row for the retry to reclaim. + expect(knowledgeItemUpdateDirectoryRelativePathMock.mock.invocationCallOrder[0]).toBeLessThan( + expandDirectoryOwnerToTreeMock.mock.invocationCallOrder[0] + ) + expect(knowledgeItemUpdateDirectoryRelativePathMock.mock.invocationCallOrder[0]).toBeLessThan( + knowledgeItemCreateActiveMock.mock.invocationCallOrder[0] + ) + expect(knowledgeItemCreateActiveMock).toHaveBeenNthCalledWith(1, baseId, { groupId: root.id, type: 'directory', data: childDir.data }) - expect(knowledgeItemCreateMock).toHaveBeenNthCalledWith(2, baseId, { + expect(knowledgeItemCreateActiveMock).toHaveBeenNthCalledWith(2, baseId, { groupId: childDir.id, type: 'file', data: childFile.data }) - expect(knowledgeItemUpdateStatusMock).toHaveBeenCalledWith(childDir.id, 'preparing') + // `childDir` is a directory: it's created active as `preparing` by `createActive`, + // then flipped to `processing` once its own children finish being created. expect(knowledgeItemUpdateStatusMock).toHaveBeenCalledWith(childDir.id, 'processing') - expect(knowledgeItemUpdateStatusMock).toHaveBeenCalledWith(childFile.id, 'processing') }) it('excludes the container itself from the reserved names so a reindex keeps its own prefix', () => { @@ -191,13 +198,12 @@ describe('prepareKnowledgeItem', () => { root.data = { source: '/abs/docs', relativePath: 'docs' } const sibling = createFileItem('file-sibling') // top-level relativePath `file-sibling.md` knowledgeItemGetItemsByBaseIdMock.mockReturnValueOnce([root, sibling]) - expandDirectoryOwnerToTreeMock.mockResolvedValueOnce({ - pathPrefix: 'docs', - children: [{ type: 'file', data: { source: '/abs/docs/a.md', relativePath: 'docs/a.md' } }] - }) + expandDirectoryOwnerToTreeMock.mockResolvedValueOnce([ + { type: 'file', data: { source: '/abs/docs/a.md', relativePath: 'docs/a.md' } } + ]) return prepareKnowledgeItem(createPrepareOptions(root)).then(() => { - const reserved = expandDirectoryOwnerToTreeMock.mock.calls[0][2] as Set + const reserved = chooseDirectoryPathPrefixMock.mock.calls[0][1] as Set expect(reserved.has('docs')).toBe(false) // Sibling names are still reserved — only the container under reindex is exempt. expect(reserved.has('file-sibling.md')).toBe(true) @@ -206,7 +212,8 @@ describe('prepareKnowledgeItem', () => { it('marks empty directory roots failed and returns no leaves', async () => { const root = createDirectoryItem('dir-root') - expandDirectoryOwnerToTreeMock.mockResolvedValueOnce({ pathPrefix: 'dir-root', children: [] }) + chooseDirectoryPathPrefixMock.mockReturnValueOnce('dir-root') + expandDirectoryOwnerToTreeMock.mockResolvedValueOnce([]) await expect(prepareKnowledgeItem(createPrepareOptions(root))).resolves.toEqual([]) @@ -218,38 +225,9 @@ describe('prepareKnowledgeItem', () => { expect(knowledgeItemUpdateStatusMock).toHaveBeenCalledWith(root.id, 'failed', { error: 'Directory contains no indexable files' }) - expect(knowledgeItemUpdateDirectoryRelativePathMock).not.toHaveBeenCalled() - }) - - it('reports created children before marking them processing', async () => { - const root = createDirectoryItem('dir-root') - const fileChild: KnowledgeItem = { - id: 'file-child', - baseId, - groupId: root.id, - type: 'file', - data: { - source: '/docs/file-child.md', - relativePath: 'file-child.md' - }, - status: 'idle', - error: null, - createdAt: '2026-04-08T00:00:00.000Z', - updatedAt: '2026-04-08T00:00:00.000Z' - } - const onCreatedItem = vi.fn() - expandDirectoryOwnerToTreeMock.mockResolvedValueOnce({ - pathPrefix: 'dir-root', - children: [{ type: 'file', data: fileChild.data }] - }) - knowledgeItemCreateMock.mockReturnValueOnce(fileChild) - knowledgeItemUpdateStatusMock.mockImplementationOnce(() => { - throw new Error('status failed') - }) - - await expect(prepareKnowledgeItem(createPrepareOptions(root, onCreatedItem))).rejects.toThrow('status failed') - - expect(onCreatedItem).toHaveBeenCalledWith(fileChild) + // Pin now precedes the empty check (pin-first crash-safety), so an empty dir still + // reserves its top-level name — a failure-path change we accept as more correct. + expect(knowledgeItemUpdateDirectoryRelativePathMock).toHaveBeenCalledWith(root.id, 'dir-root') }) it('stops creating children when the runtime signal is aborted after expansion', async () => { @@ -258,18 +236,15 @@ describe('prepareKnowledgeItem', () => { const abortError = new Error('interrupted') expandDirectoryOwnerToTreeMock.mockImplementationOnce(async () => { controller.abort(abortError) - return { - pathPrefix: 'dir-root', - children: [ - { - type: 'file', - data: { - source: '/docs/file-child.md', - relativePath: 'file-child.md' - } + return [ + { + type: 'file', + data: { + source: '/docs/file-child.md', + relativePath: 'file-child.md' } - ] - } + } + ] }) await expect( @@ -279,7 +254,7 @@ describe('prepareKnowledgeItem', () => { }) ).rejects.toBe(abortError) - expect(knowledgeItemCreateMock).not.toHaveBeenCalled() + expect(knowledgeItemCreateActiveMock).not.toHaveBeenCalled() expect(knowledgeItemUpdateStatusMock).not.toHaveBeenCalled() }) diff --git a/src/main/features/knowledge/tasks/__tests__/prepareRootJobHandler.test.ts b/src/main/features/knowledge/tasks/__tests__/prepareRootJobHandler.test.ts index 9b211cfa4a5..07d6109cb00 100644 --- a/src/main/features/knowledge/tasks/__tests__/prepareRootJobHandler.test.ts +++ b/src/main/features/knowledge/tasks/__tests__/prepareRootJobHandler.test.ts @@ -1,41 +1,48 @@ import { describe, expect, it } from 'vitest' +import { getKnowledgeBaseFilePath } from '../../pathStorage' import { createCtx, createDirectoryItem, - createJobSnapshot, createNoteItem, createPrepareRootJobHandler, deleteItemsByIdsMock, deleteKnowledgeItemFilesBestEffortMock, deleteMaterialsMock, - getJobMock, + ingestionService, knowledgeItemGetByIdMock, knowledgeItemGetSubtreeItemsMock, knowledgeItemSetSubtreeStatusMock, knowledgeItemUpdateStatusMock, knowledgeLockManager, prepareKnowledgeItemMock, - scheduleItemMock, - workflowService + removeDirMock, + scheduleItemMock } from './jobHandlerTestUtils' +/** A directory container whose `raw/` was pinned by a prior attempt (pin-before-copy). */ +function createPinnedDirectoryItem(relativePath: string) { + const container = createDirectoryItem() + return { ...container, data: { ...container.data, relativePath } } +} + describe('prepare-root job handler', () => { it('clears stale expansion and schedules recreated leaves', async () => { - const handler = createPrepareRootJobHandler(knowledgeLockManager as never, workflowService as never) + const handler = createPrepareRootJobHandler(knowledgeLockManager as never, ingestionService) knowledgeItemGetByIdMock.mockReturnValue(createDirectoryItem()) await handler.execute(createCtx({ baseId: 'kb-1', itemId: 'dir-1' }, 'prepare-job')) expect(knowledgeItemGetSubtreeItemsMock).toHaveBeenCalledWith('kb-1', ['dir-1']) - expect(deleteItemsByIdsMock).toHaveBeenCalledWith('kb-1', []) + // Nothing to purge (no prior expansion) — purgeKnowledgeSubtreeWithinLock is a no-op for an empty subtree. + expect(deleteItemsByIdsMock).not.toHaveBeenCalled() expect(prepareKnowledgeItemMock).toHaveBeenCalledWith(expect.objectContaining({ baseId: 'kb-1' })) expect(scheduleItemMock).toHaveBeenCalledWith('kb-1', 'leaf-1', 'prepare-job') expect(handler.defaultQueue?.({ baseId: 'kb-1', itemId: 'dir-1' })).toBe('base.kb-1') }) it('clears stale expansion vectors before deleting rows', async () => { - const handler = createPrepareRootJobHandler(knowledgeLockManager as never, workflowService as never) + const handler = createPrepareRootJobHandler(knowledgeLockManager as never, ingestionService) const activeChild = createNoteItem('active-note', 'dir-1') knowledgeItemGetByIdMock.mockReturnValue(createDirectoryItem()) knowledgeItemGetSubtreeItemsMock.mockReturnValue([activeChild]) @@ -50,7 +57,7 @@ describe('prepare-root job handler', () => { }) it('routes stale-expansion cleanup through best-effort delete before deleting rows', async () => { - const handler = createPrepareRootJobHandler(knowledgeLockManager as never, workflowService as never) + const handler = createPrepareRootJobHandler(knowledgeLockManager as never, ingestionService) const activeChild = createNoteItem('active-note', 'dir-1') knowledgeItemGetByIdMock.mockReturnValue(createDirectoryItem()) knowledgeItemGetSubtreeItemsMock.mockReturnValue([activeChild]) @@ -69,7 +76,7 @@ describe('prepare-root job handler', () => { }) it('leaves deleting descendants for delete-subtree cleanup', async () => { - const handler = createPrepareRootJobHandler(knowledgeLockManager as never, workflowService as never) + const handler = createPrepareRootJobHandler(knowledgeLockManager as never, ingestionService) const activeChild = createNoteItem('active-note', 'dir-1') const deletingChild = createNoteItem('deleting-note', 'dir-1', 'deleting') knowledgeItemGetByIdMock.mockReturnValue(createDirectoryItem()) @@ -83,11 +90,54 @@ describe('prepare-root job handler', () => { expect(deleteItemsByIdsMock).not.toHaveBeenCalledWith('kb-1', expect.arrayContaining(['deleting-note'])) }) + it('reclaims the container pinned prefix shell even when no descendant rows exist to purge', async () => { + const handler = createPrepareRootJobHandler(knowledgeLockManager as never, ingestionService) + // A crash in the pin-before-copy window leaves orphan bytes under raw/docs but zero + // descendant rows. getSubtreeItems excludes the container, so the row-scoped purge sees + // nothing — only the explicit container removeDir can reclaim those orphan bytes. + knowledgeItemGetByIdMock.mockReturnValue(createPinnedDirectoryItem('docs')) + knowledgeItemGetSubtreeItemsMock.mockReturnValue([]) + + await handler.execute(createCtx({ baseId: 'kb-1', itemId: 'dir-1' }, 'prepare-job')) + + expect(deleteItemsByIdsMock).not.toHaveBeenCalled() + expect(removeDirMock).toHaveBeenCalledWith(getKnowledgeBaseFilePath('kb-1', 'docs')) + }) + + it('does not reclaim a container shell when no prefix was ever pinned', async () => { + const handler = createPrepareRootJobHandler(knowledgeLockManager as never, ingestionService) + // Default createDirectoryItem() carries data.source only, no relativePath — the guard + // must skip removeDir so a never-expanded directory does not blow away an unrelated path. + knowledgeItemGetByIdMock.mockReturnValue(createDirectoryItem()) + knowledgeItemGetSubtreeItemsMock.mockReturnValue([]) + + await handler.execute(createCtx({ baseId: 'kb-1', itemId: 'dir-1' }, 'prepare-job')) + + expect(removeDirMock).not.toHaveBeenCalled() + }) + + it('purges descendant rows before reclaiming the container shell', async () => { + const handler = createPrepareRootJobHandler(knowledgeLockManager as never, ingestionService) + const activeChild = createNoteItem('active-note', 'dir-1') + knowledgeItemGetByIdMock.mockReturnValue(createPinnedDirectoryItem('docs')) + knowledgeItemGetSubtreeItemsMock.mockReturnValue([activeChild]) + + await handler.execute(createCtx({ baseId: 'kb-1', itemId: 'dir-1' }, 'prepare-job')) + + expect(deleteItemsByIdsMock).toHaveBeenCalledWith('kb-1', ['active-note']) + expect(removeDirMock).toHaveBeenCalledWith(getKnowledgeBaseFilePath('kb-1', 'docs')) + // Row-scoped purge runs first; the container removeDir then sweeps whatever the purge + // could not see (nested dir shells, bytes of rows that were never committed). + expect(deleteItemsByIdsMock.mock.invocationCallOrder[0]).toBeLessThan(removeDirMock.mock.invocationCallOrder[0]) + }) + it('skips expansion when the root becomes deleting inside the mutation lock', async () => { - const handler = createPrepareRootJobHandler(knowledgeLockManager as never, workflowService as never) + const handler = createPrepareRootJobHandler(knowledgeLockManager as never, ingestionService) + // Live on the pre-lock load, then deleting for every in-lock re-resolve (the shell-reclaim + // guard and the expansion guard both re-read the row); expansion must be skipped. knowledgeItemGetByIdMock .mockReturnValueOnce(createDirectoryItem()) - .mockReturnValueOnce(createDirectoryItem('dir-1', 'deleting')) + .mockReturnValue(createDirectoryItem('dir-1', 'deleting')) const ctx = createCtx({ baseId: 'kb-1', itemId: 'dir-1' }, 'prepare-job') await handler.execute(ctx) @@ -99,7 +149,7 @@ describe('prepare-root job handler', () => { }) it('keeps terminal failure from an empty expansion', async () => { - const handler = createPrepareRootJobHandler(knowledgeLockManager as never, workflowService as never) + const handler = createPrepareRootJobHandler(knowledgeLockManager as never, ingestionService) knowledgeItemGetByIdMock.mockReturnValue(createDirectoryItem()) prepareKnowledgeItemMock.mockResolvedValue([]) @@ -110,7 +160,7 @@ describe('prepare-root job handler', () => { }) it('marks unscheduled child leaves failed when enqueueing a child fails', async () => { - const handler = createPrepareRootJobHandler(knowledgeLockManager as never, workflowService as never) + const handler = createPrepareRootJobHandler(knowledgeLockManager as never, ingestionService) const leaves = [ createNoteItem('leaf-1', 'dir-1'), createNoteItem('leaf-2', 'dir-1'), @@ -127,50 +177,21 @@ describe('prepare-root job handler', () => { expect(scheduleItemMock).toHaveBeenCalledWith('kb-1', 'leaf-1', 'prepare-job') expect(scheduleItemMock).toHaveBeenCalledWith('kb-1', 'leaf-2', 'prepare-job') expect(scheduleItemMock).not.toHaveBeenCalledWith('kb-1', 'leaf-3', 'prepare-job') - expect(knowledgeItemUpdateStatusMock).not.toHaveBeenCalledWith('leaf-1', 'failed', expect.anything()) - expect(knowledgeItemUpdateStatusMock).toHaveBeenCalledWith('leaf-2', 'failed', { + expect(knowledgeItemSetSubtreeStatusMock).not.toHaveBeenCalledWith('kb-1', ['leaf-1'], 'failed', expect.anything()) + expect(knowledgeItemSetSubtreeStatusMock).toHaveBeenCalledWith('kb-1', ['leaf-2'], 'failed', { error: 'Failed to schedule knowledge child item job: enqueue failed' }) - expect(knowledgeItemUpdateStatusMock).toHaveBeenCalledWith('leaf-3', 'failed', { - error: 'Failed to schedule knowledge child item job: enqueue failed' - }) - }) - - it('falls back to subtree failed status when marking an unscheduled leaf fails', async () => { - const handler = createPrepareRootJobHandler(knowledgeLockManager as never, workflowService as never) - const leaves = [createNoteItem('leaf-1', 'dir-1'), createNoteItem('leaf-2', 'dir-1')] - knowledgeItemGetByIdMock.mockReturnValue(createDirectoryItem()) - prepareKnowledgeItemMock.mockResolvedValue(leaves) - scheduleItemMock.mockRejectedValueOnce(new Error('enqueue failed')) - knowledgeItemUpdateStatusMock - .mockReturnValueOnce(createDirectoryItem('dir-1', 'processing')) - .mockImplementationOnce(() => { - throw new Error('status busy') - }) - - await expect(handler.execute(createCtx({ baseId: 'kb-1', itemId: 'dir-1' }, 'prepare-job'))).rejects.toThrow( - 'enqueue failed' - ) - - expect(knowledgeItemSetSubtreeStatusMock).toHaveBeenCalledWith('kb-1', ['leaf-1'], 'failed', { - error: 'Failed to schedule knowledge child item job: enqueue failed' - }) - expect(knowledgeItemUpdateStatusMock).toHaveBeenCalledWith('leaf-2', 'failed', { + expect(knowledgeItemSetSubtreeStatusMock).toHaveBeenCalledWith('kb-1', ['leaf-3'], 'failed', { error: 'Failed to schedule knowledge child item job: enqueue failed' }) }) it('reports unrecovered leaves when failed status cleanup also fails', async () => { - const handler = createPrepareRootJobHandler(knowledgeLockManager as never, workflowService as never) + const handler = createPrepareRootJobHandler(knowledgeLockManager as never, ingestionService) const leaves = [createNoteItem('leaf-1', 'dir-1')] knowledgeItemGetByIdMock.mockReturnValue(createDirectoryItem()) prepareKnowledgeItemMock.mockResolvedValue(leaves) scheduleItemMock.mockRejectedValueOnce(new Error('enqueue failed')) - knowledgeItemUpdateStatusMock - .mockReturnValueOnce(createDirectoryItem('dir-1', 'processing')) - .mockImplementationOnce(() => { - throw new Error('status busy') - }) knowledgeItemSetSubtreeStatusMock.mockImplementationOnce(() => { throw new Error('subtree busy') }) @@ -181,14 +202,7 @@ describe('prepare-root job handler', () => { }) it('onSettled skips failed status when the item is deleting', async () => { - const handler = createPrepareRootJobHandler(knowledgeLockManager as never, workflowService as never) - getJobMock.mockResolvedValue( - createJobSnapshot({ - id: 'prepare-job', - type: 'knowledge.prepare-root', - input: { baseId: 'kb-1', itemId: 'dir-1' } - }) - ) + const handler = createPrepareRootJobHandler(knowledgeLockManager as never, ingestionService) knowledgeItemGetByIdMock.mockReturnValue(createDirectoryItem('dir-1', 'deleting')) await handler.onSettled?.({ diff --git a/src/main/features/knowledge/tasks/__tests__/reindexSubtreeJobHandler.test.ts b/src/main/features/knowledge/tasks/__tests__/reindexSubtreeJobHandler.test.ts index 1bf995f79a2..00ead97f310 100644 --- a/src/main/features/knowledge/tasks/__tests__/reindexSubtreeJobHandler.test.ts +++ b/src/main/features/knowledge/tasks/__tests__/reindexSubtreeJobHandler.test.ts @@ -13,7 +13,7 @@ import { deleteKnowledgeItemFilesBestEffortMock, deleteMaterialsMock, FILE_ITEM_ID, - getJobMock, + ingestionService, knowledgeItemGetSubtreeItemsMock, knowledgeItemSetSubtreeStatusMock, knowledgeItemUpdateStatusMock, @@ -21,13 +21,12 @@ import { listMock, loggerWarnMock, probeKnowledgeSourcePathMock, - scheduleItemMock, - workflowService + scheduleItemMock } from './jobHandlerTestUtils' describe('reindex-subtree job handler', () => { it('clears old artifacts, resets selected roots, and schedules selected roots', async () => { - const handler = createReindexSubtreeJobHandler(knowledgeLockManager as never, workflowService as never) + const handler = createReindexSubtreeJobHandler(knowledgeLockManager as never, ingestionService) const root = createDirectoryItem('dir-1') const child = createNoteItem('note-1', 'dir-1') knowledgeItemGetSubtreeItemsMock.mockImplementation( @@ -50,7 +49,7 @@ describe('reindex-subtree job handler', () => { }) it('routes container descendant cleanup through best-effort delete before deleting rows', async () => { - const handler = createReindexSubtreeJobHandler(knowledgeLockManager as never, workflowService as never) + const handler = createReindexSubtreeJobHandler(knowledgeLockManager as never, ingestionService) const root = createDirectoryItem('dir-1') const child = createNoteItem('note-1', 'dir-1') knowledgeItemGetSubtreeItemsMock.mockImplementation( @@ -75,7 +74,7 @@ describe('reindex-subtree job handler', () => { }) it('skips deleting subtrees before reset', async () => { - const handler = createReindexSubtreeJobHandler(knowledgeLockManager as never, workflowService as never) + const handler = createReindexSubtreeJobHandler(knowledgeLockManager as never, ingestionService) const root = createDirectoryItem('dir-1', 'deleting') const child = createNoteItem('note-1', 'dir-1', 'deleting') const ctx = createCtx({ baseId: 'kb-1', rootItemIds: ['dir-1'] }, 'reindex-job') @@ -93,7 +92,7 @@ describe('reindex-subtree job handler', () => { }) it('skips reset when the subtree becomes deleting inside the mutation lock', async () => { - const handler = createReindexSubtreeJobHandler(knowledgeLockManager as never, workflowService as never) + const handler = createReindexSubtreeJobHandler(knowledgeLockManager as never, ingestionService) const root = createDirectoryItem('dir-1') const child = createNoteItem('note-1', 'dir-1') const deletingChild = createNoteItem('note-1', 'dir-1', 'deleting') @@ -110,7 +109,7 @@ describe('reindex-subtree job handler', () => { }) it('leaves a root untouched when its source vanished before the mutation lock', async () => { - const handler = createReindexSubtreeJobHandler(knowledgeLockManager as never, workflowService as never) + const handler = createReindexSubtreeJobHandler(knowledgeLockManager as never, ingestionService) const root = createDirectoryItem('dir-1') const child = createNoteItem('note-1', 'dir-1') const ctx = createCtx({ baseId: 'kb-1', rootItemIds: ['dir-1'] }, 'reindex-job') @@ -140,7 +139,7 @@ describe('reindex-subtree job handler', () => { }) it('reindexes only the roots whose source still exists', async () => { - const handler = createReindexSubtreeJobHandler(knowledgeLockManager as never, workflowService as never) + const handler = createReindexSubtreeJobHandler(knowledgeLockManager as never, ingestionService) const presentRoot = createDirectoryItem('dir-1') const presentChild = createNoteItem('note-1', 'dir-1') const missingRoot = createDirectoryItem('dir-2') @@ -176,7 +175,7 @@ describe('reindex-subtree job handler', () => { }) it('clears old artifacts for selected leaf roots', async () => { - const handler = createReindexSubtreeJobHandler(knowledgeLockManager as never, workflowService as never) + const handler = createReindexSubtreeJobHandler(knowledgeLockManager as never, ingestionService) const root = createNoteItem('note-1') knowledgeItemGetSubtreeItemsMock.mockImplementation( (_baseId: string, _rootIds: string[], options: { includeRoots?: boolean; leafOnly?: boolean } = {}) => { @@ -194,7 +193,7 @@ describe('reindex-subtree job handler', () => { }) it('marks only unscheduled reset roots failed when rescheduling fails', async () => { - const handler = createReindexSubtreeJobHandler(knowledgeLockManager as never, workflowService as never) + const handler = createReindexSubtreeJobHandler(knowledgeLockManager as never, ingestionService) const firstRoot = createDirectoryItem('dir-1') const secondRoot = createDirectoryItem('dir-2') const firstChild = createNoteItem('note-1', 'dir-1') @@ -220,7 +219,7 @@ describe('reindex-subtree job handler', () => { }) it('never fails a left-untouched missing-source root when rescheduling a rebuildable root fails', async () => { - const handler = createReindexSubtreeJobHandler(knowledgeLockManager as never, workflowService as never) + const handler = createReindexSubtreeJobHandler(knowledgeLockManager as never, ingestionService) const presentRoot = createDirectoryItem('dir-1') const presentChild = createNoteItem('note-1', 'dir-1') const missingRoot = createDirectoryItem('dir-2') @@ -255,7 +254,7 @@ describe('reindex-subtree job handler', () => { }) it('stores the localized interruption code when a shutdown abort interrupts rescheduling', async () => { - const handler = createReindexSubtreeJobHandler(knowledgeLockManager as never, workflowService as never) + const handler = createReindexSubtreeJobHandler(knowledgeLockManager as never, ingestionService) const root = createDirectoryItem('dir-1') const child = createNoteItem('note-1', 'dir-1') knowledgeItemGetSubtreeItemsMock.mockImplementation( @@ -285,14 +284,7 @@ describe('reindex-subtree job handler', () => { }) it('onSettled marks active roots without follow-up jobs failed', async () => { - const handler = createReindexSubtreeJobHandler(knowledgeLockManager as never, workflowService as never) - getJobMock.mockResolvedValue( - createJobSnapshot({ - id: 'reindex-job', - type: 'knowledge.reindex-subtree', - input: { baseId: 'kb-1', rootItemIds: ['dir-1', 'note-1'] } - }) - ) + const handler = createReindexSubtreeJobHandler(knowledgeLockManager as never, ingestionService) listMock.mockResolvedValue([]) knowledgeItemGetSubtreeItemsMock.mockReturnValue([ createDirectoryItem('dir-1', 'preparing'), @@ -317,14 +309,7 @@ describe('reindex-subtree job handler', () => { }) it('onSettled stores the localized interruption code when the reindex job was cancelled', async () => { - const handler = createReindexSubtreeJobHandler(knowledgeLockManager as never, workflowService as never) - getJobMock.mockResolvedValue( - createJobSnapshot({ - id: 'reindex-job', - type: 'knowledge.reindex-subtree', - input: { baseId: 'kb-1', rootItemIds: ['dir-1', 'note-1'] } - }) - ) + const handler = createReindexSubtreeJobHandler(knowledgeLockManager as never, ingestionService) listMock.mockResolvedValue([]) knowledgeItemGetSubtreeItemsMock.mockReturnValue([ createDirectoryItem('dir-1', 'preparing'), @@ -351,14 +336,7 @@ describe('reindex-subtree job handler', () => { }) it('onSettled skips deleting roots and roots with follow-up jobs', async () => { - const handler = createReindexSubtreeJobHandler(knowledgeLockManager as never, workflowService as never) - getJobMock.mockResolvedValue( - createJobSnapshot({ - id: 'reindex-job', - type: 'knowledge.reindex-subtree', - input: { baseId: 'kb-1', rootItemIds: ['dir-1', 'note-1'] } - }) - ) + const handler = createReindexSubtreeJobHandler(knowledgeLockManager as never, ingestionService) listMock.mockResolvedValue([ createJobSnapshot({ id: 'prepare-dir-1', @@ -388,14 +366,7 @@ describe('reindex-subtree job handler', () => { }) it('onSettled treats file-processing check jobs as follow-up jobs', async () => { - const handler = createReindexSubtreeJobHandler(knowledgeLockManager as never, workflowService as never) - getJobMock.mockResolvedValue( - createJobSnapshot({ - id: 'reindex-job', - type: 'knowledge.reindex-subtree', - input: { baseId: 'kb-1', rootItemIds: [FILE_ITEM_ID] } - }) - ) + const handler = createReindexSubtreeJobHandler(knowledgeLockManager as never, ingestionService) listMock.mockResolvedValue([ createJobSnapshot({ id: 'check-file-1', @@ -407,7 +378,7 @@ describe('reindex-subtree job handler', () => { fileProcessingJobId: 'fp-job-1', pollRound: 0, firstScheduledAt: 1779811200000, - parentJobId: 'reindex-job' + processedRelativePath: 'source.md' } }) ]) diff --git a/src/main/features/knowledge/tasks/checkFileProcessingResultJobHandler.ts b/src/main/features/knowledge/tasks/checkFileProcessingResultJobHandler.ts index ae8a4b8d98f..fd312bed123 100644 --- a/src/main/features/knowledge/tasks/checkFileProcessingResultJobHandler.ts +++ b/src/main/features/knowledge/tasks/checkFileProcessingResultJobHandler.ts @@ -3,6 +3,7 @@ import './jobTypes' import { application } from '@application' import { knowledgeItemService } from '@data/services/KnowledgeItemService' import { loggerService } from '@logger' +import type { KeyedMutex } from '@main/core/concurrency/KeyedMutex' import type { JobContext, JobHandler } from '@main/core/job/types' import { JOB_PROGRESS_KEY_PREFIX } from '@main/core/job/types' import { isTerminalStatus, type JobSnapshot } from '@shared/data/api/schemas/jobs' @@ -12,13 +13,12 @@ import { getFileProcessingMarkdownArtifactPath } from '../../fileProcessing/persistence/artifacts' import type { FileProcessingJobPayload } from '../../fileProcessing/tasks/shared' -import type { KnowledgeLockManager } from '../KnowledgeLockManager' -import type { KnowledgeWorkflowService } from '../KnowledgeWorkflowService' -import { knowledgeQueueName, toKnowledgeBaseId, toKnowledgeItemId } from '../types' -import { toKnowledgeRelativePath } from '../utils/storage/pathStorage' +import type { KnowledgeItemScheduler } from '../ingestion/KnowledgeIngestionService' +import { knowledgeQueueName, reportKnowledgeProgress, toKnowledgeBaseId, toKnowledgeItemId } from '../types' import type { KnowledgeCheckFileProcessingResultPayload } from './jobTypes' import { cancelJobOrThrow } from './utils/cancel' -import { isDataApiNotFoundError, markKnowledgeItemFailedOnSettled } from './utils/settled' +import { resolveLiveKnowledgeItem } from './utils/liveItem' +import { markKnowledgeItemFailedOnSettled } from './utils/settled' const logger = loggerService.withContext('Knowledge:CheckFileProcessingResultJobHandler') // Remote document processors can be slow, but a stale paid job should not poll forever. @@ -26,8 +26,8 @@ const FILE_PROCESSING_MAX_WAIT_MS = 30 * 60 * 1000 const FILE_PROCESSING_ITEM_UNAVAILABLE_CANCEL_REASON = 'knowledge-file-processing-item-unavailable' export function createCheckFileProcessingResultJobHandler( - knowledgeLockManager: KnowledgeLockManager, - workflowService: KnowledgeWorkflowService + knowledgeLockManager: KeyedMutex, + ingestionService: KnowledgeItemScheduler ): JobHandler { return { // Don't auto-resume on restart — a deliberate app quit must not re-spend the @@ -46,7 +46,7 @@ export function createCheckFileProcessingResultJobHandler( async execute(ctx) { const { baseId, itemId, fileProcessingJobId } = ctx.input const firstScheduledAt = ctx.input.firstScheduledAt - const workflowParentJobId = ctx.input.parentJobId ?? ctx.jobId + const workflowParentJobId = ctx.parentId ?? ctx.jobId ctx.signal.throwIfAborted() if (shouldSkipMissingOrDeletingItem(baseId, itemId, ctx.jobId)) { @@ -59,13 +59,13 @@ export function createCheckFileProcessingResultJobHandler( if (!snapshot) { markItemFailed(itemId, `File processing job not found: ${fileProcessingJobId}`) - ctx.reportProgress(100, { stage: 'failed' }) + reportKnowledgeProgress(ctx, 100, { stage: 'failed' }) return } if (!isExpectedFileProcessingJob(snapshot, itemId)) { markItemFailed(itemId, `Invalid file processing job for knowledge item: ${fileProcessingJobId}`) - ctx.reportProgress(100, { stage: 'failed' }) + reportKnowledgeProgress(ctx, 100, { stage: 'failed' }) return } @@ -73,19 +73,20 @@ export function createCheckFileProcessingResultJobHandler( if (Date.now() - firstScheduledAt >= FILE_PROCESSING_MAX_WAIT_MS) { await cancelJobOrThrow(fileProcessingJobId, 'knowledge-file-processing-timeout') markItemFailed(itemId, `File processing job ${fileProcessingJobId} did not finish within 30 minutes`) - ctx.reportProgress(100, { stage: 'failed' }) + reportKnowledgeProgress(ctx, 100, { stage: 'failed' }) return } const nextPollRound = ctx.input.pollRound + 1 - await workflowService.scheduleFileProcessingCheck( + await ingestionService.scheduleFileProcessingCheck( toKnowledgeBaseId(baseId), toKnowledgeItemId(itemId), fileProcessingJobId, { pollRound: nextPollRound, firstScheduledAt, - parentJobId: workflowParentJobId + parentJobId: workflowParentJobId, + processedRelativePath: ctx.input.processedRelativePath } ) reportWaitingProgress(ctx, fileProcessingJobId, nextPollRound) @@ -97,24 +98,24 @@ export function createCheckFileProcessingResultJobHandler( itemId, `File processing job ${fileProcessingJobId} ${snapshot.status}: ${getFileProcessingFailureMessage(snapshot)}` ) - ctx.reportProgress(100, { stage: 'failed' }) + reportKnowledgeProgress(ctx, 100, { stage: 'failed' }) return } - const indexedRelativePath = parseMarkdownArtifactRelativePathOrNull(baseId, snapshot) - if (!indexedRelativePath) { + if (!isCompletedMarkdownArtifact(snapshot)) { markItemFailed(itemId, `Invalid file processing result for job ${fileProcessingJobId}`) - ctx.reportProgress(100, { stage: 'failed' }) + reportKnowledgeProgress(ctx, 100, { stage: 'failed' }) return } + const indexedRelativePath = ctx.input.processedRelativePath - const canContinue = await knowledgeLockManager.withBaseMutationLock(baseId, async () => { + const canContinue = await knowledgeLockManager.runExclusive(baseId, async () => { if (shouldSkipMissingOrDeletingItem(baseId, itemId, ctx.jobId)) { return false } knowledgeItemService.updateIndexedRelativePath(itemId, indexedRelativePath) - await workflowService.scheduleIndexing( + await ingestionService.scheduleIndexing( toKnowledgeBaseId(baseId), toKnowledgeItemId(itemId), workflowParentJobId @@ -124,7 +125,7 @@ export function createCheckFileProcessingResultJobHandler( if (!canContinue) { return } - ctx.reportProgress(100, { stage: 'done' }) + reportKnowledgeProgress(ctx, 100, { stage: 'done' }) }, async onSettled(event) { @@ -144,11 +145,11 @@ function reportWaitingProgress( ): void { const childProgress = application.get('CacheService').getShared(`${JOB_PROGRESS_KEY_PREFIX}${fileProcessingJobId}`) if (!childProgress) { - ctx.reportProgress(0, { stage: 'waiting', pollRound }) + reportKnowledgeProgress(ctx, 0, { stage: 'waiting', pollRound }) return } - ctx.reportProgress(childProgress.progress, { + reportKnowledgeProgress(ctx, childProgress.progress, { stage: 'waiting', pollRound, fileProcessingJobId, @@ -168,50 +169,48 @@ function isExpectedFileProcessingJob(snapshot: JobSnapshot, itemId: string): boo } function shouldSkipMissingOrDeletingItem(baseId: string, itemId: string, jobId: string): boolean { - try { - const item = knowledgeItemService.getById(itemId) - if (item.baseId !== baseId) { - throw new Error(`Knowledge item '${itemId}' does not belong to base '${baseId}'`) - } - if (item.status === 'deleting') { + const result = resolveLiveKnowledgeItem(itemId) + if ('skip' in result) { + if (result.skip === 'deleting') { logger.info('Skipping file-processing check for deleting item', { baseId, itemId, jobId }) - return true - } - return false - } catch (error) { - if (isDataApiNotFoundError(error)) { + } else { logger.info('Skipping file-processing check for missing item', { baseId, itemId, jobId }) - return true } - throw error + return true } + if (result.item.baseId !== baseId) { + throw new Error(`Knowledge item '${itemId}' does not belong to base '${baseId}'`) + } + return false } function markItemFailed(itemId: string, error: string): void { - try { - const item = knowledgeItemService.getById(itemId) - if (item.status === 'deleting') { + const result = resolveLiveKnowledgeItem(itemId) + if ('skip' in result) { + if (result.skip === 'deleting') { logger.info('Skipping mark failed for deleting item', { itemId, error }) - return - } - knowledgeItemService.updateStatus(itemId, 'failed', { error }) - } catch (updateError) { - if (isDataApiNotFoundError(updateError)) { + } else { logger.info('Skipping mark failed for missing item', { itemId, error }) - return } - throw updateError + return } + knowledgeItemService.updateStatus(itemId, 'failed', { error }) } -function parseMarkdownArtifactRelativePathOrNull(baseId: string, snapshot: JobSnapshot): string | null { +/** + * Validates that a completed file-processing job actually produced a markdown-file + * artifact (not a degraded inline-text result) — the item's `indexedRelativePath` + * is the job's own `processedRelativePath` input, not derived from this artifact. + */ +function isCompletedMarkdownArtifact(snapshot: JobSnapshot): boolean { try { - return toKnowledgeRelativePath(baseId, getFileProcessingMarkdownArtifactPath(snapshot)) + getFileProcessingMarkdownArtifactPath(snapshot) + return true } catch (error) { logger.warn('Invalid file-processing result for knowledge item', { jobId: snapshot.id, error: error instanceof Error ? error.message : String(error) }) - return null + return false } } diff --git a/src/main/features/knowledge/tasks/deleteSubtreeJobHandler.ts b/src/main/features/knowledge/tasks/deleteSubtreeJobHandler.ts index b1dc54a85f4..8e1781d89fd 100644 --- a/src/main/features/knowledge/tasks/deleteSubtreeJobHandler.ts +++ b/src/main/features/knowledge/tasks/deleteSubtreeJobHandler.ts @@ -3,18 +3,19 @@ import './jobTypes' import { knowledgeBaseService } from '@data/services/KnowledgeBaseService' import { knowledgeItemService } from '@data/services/KnowledgeItemService' import { loggerService } from '@logger' +import type { KeyedMutex } from '@main/core/concurrency/KeyedMutex' import type { JobHandler } from '@main/core/job/types' -import type { KnowledgeLockManager } from '../KnowledgeLockManager' +import { purgeKnowledgeSubtreeWithinLock } from '../ingestion/subtreePurge' +import { reclaimKnowledgeIndexSpace } from '../pipeline/vectorstore/vectorCleanup' import { knowledgeQueueName, reportKnowledgeProgress, toKnowledgeBaseId } from '../types' -import { cancelActiveKnowledgeSubtreeJobs, purgeKnowledgeSubtreeWithinLock } from '../utils/cleanup/subtreePurge' -import { reclaimKnowledgeIndexSpace } from '../utils/cleanup/vectorCleanup' import type { KnowledgeDeleteSubtreePayload } from './jobTypes' +import { cancelActiveKnowledgeJobs } from './utils/cancel' const logger = loggerService.withContext('Knowledge:DeleteSubtreeJobHandler') export function createDeleteSubtreeJobHandler( - knowledgeLockManager: KnowledgeLockManager + knowledgeLockManager: KeyedMutex ): JobHandler { return { recovery: 'retry', @@ -43,10 +44,14 @@ export function createDeleteSubtreeJobHandler( } // Stop active work touching deleting rows before removing vectors and rows. - await cancelActiveKnowledgeSubtreeJobs(baseId, deletingSubtreeItemIds, 'knowledge-delete-subtree', ctx.jobId) + await cancelActiveKnowledgeJobs(baseId, 'knowledge-delete-subtree', { + rootItemIds: deletingSubtreeItemIds, + excludeJobId: ctx.jobId, + onCancelTimeout: 'throw' + }) // Cleanup is locked so no indexer can write vectors for rows being removed. - await knowledgeLockManager.withBaseMutationLock(baseId, async () => { + await knowledgeLockManager.runExclusive(baseId, async () => { const base = knowledgeBaseService.getById(baseId) const subtreeItems = knowledgeItemService .getSubtreeItems(baseId, rootItemIds, { includeRoots: true }) diff --git a/src/main/features/knowledge/tasks/indexDocumentsJobHandler.ts b/src/main/features/knowledge/tasks/indexDocumentsJobHandler.ts index ade892c4562..d86855f4404 100644 --- a/src/main/features/knowledge/tasks/indexDocumentsJobHandler.ts +++ b/src/main/features/knowledge/tasks/indexDocumentsJobHandler.ts @@ -4,26 +4,27 @@ import { application } from '@application' import { knowledgeBaseService } from '@data/services/KnowledgeBaseService' import { knowledgeItemService } from '@data/services/KnowledgeItemService' import { loggerService } from '@logger' +import type { KeyedMutex } from '@main/core/concurrency/KeyedMutex' import type { JobContext, JobHandler } from '@main/core/job/types' +import { isDataApiNotFoundError } from '@shared/data/api' import type { KnowledgeBase } from '@shared/data/types/knowledge' import { isCompletedVectorKnowledgeBase } from '@shared/data/types/knowledge' -import type { KnowledgeLockManager } from '../KnowledgeLockManager' -import { loadKnowledgeItemDocuments } from '../readers/KnowledgeReader' +import type { IndexableKnowledgeItem } from '../items' +import { isIndexableKnowledgeItem, toMaterialRelativePath } from '../items' +import { collectKnowledgeReservedRelativePaths } from '../pathStorage' +import { type ChunkedKnowledgeContent, chunkKnowledgeDocuments } from '../pipeline/indexing/chunk' +import { embedKnowledgeTexts } from '../pipeline/indexing/embed' +import { loadKnowledgeItemDocuments } from '../pipeline/readers/KnowledgeReader' +import { captureNoteSnapshotFile } from '../pipeline/sources/noteSnapshot' +import { fetchKnowledgeWebPage } from '../pipeline/sources/url' +import { captureUrlSnapshotFile } from '../pipeline/sources/urlSnapshot' +import { hashEmbeddingText } from '../pipeline/vectorstore/indexStore/hashing' +import type { RebuildMaterialEmbeddingInput, RebuildMaterialInput } from '../pipeline/vectorstore/indexStore/model' import { knowledgeQueueName, reportKnowledgeProgress, toKnowledgeBaseId } from '../types' -import type { IndexableKnowledgeItem } from '../types/items' -import { type ChunkedKnowledgeContent, chunkKnowledgeDocuments } from '../utils/indexing/chunk' -import { embedKnowledgeTexts } from '../utils/indexing/embed' -import { toMaterialRelativePath } from '../utils/indexing/materialFields' -import { isIndexableKnowledgeItem } from '../utils/items' -import { captureNoteSnapshotFile } from '../utils/sources/noteSnapshot' -import { fetchKnowledgeWebPage } from '../utils/sources/url' -import { captureUrlSnapshotFile } from '../utils/sources/urlSnapshot' -import { collectKnowledgeReservedRelativePaths } from '../utils/storage/pathStorage' -import { hashEmbeddingText } from '../vectorstore/indexStore/hashing' -import type { RebuildMaterialEmbeddingInput, RebuildMaterialInput } from '../vectorstore/indexStore/model' import type { KnowledgeIndexDocumentsPayload } from './jobTypes' -import { isDataApiNotFoundError, markKnowledgeItemFailedOnSettled } from './utils/settled' +import { resolveLiveKnowledgeItem } from './utils/liveItem' +import { markKnowledgeItemFailedOnSettled } from './utils/settled' const logger = loggerService.withContext('Knowledge:IndexDocumentsJobHandler') @@ -34,7 +35,7 @@ type LoadedIndexDocumentsInput = { type LoadedDocuments = Awaited> export function createIndexDocumentsJobHandler( - knowledgeLockManager: KnowledgeLockManager + knowledgeLockManager: KeyedMutex ): JobHandler { return { // Don't auto-resume on restart — a deliberate app quit must not re-spend the @@ -72,7 +73,7 @@ export function createIndexDocumentsJobHandler( // lock; these phases can be slow and do not mutate shared state. const readableItem = await ensureSnapshot(ctx, item, knowledgeLockManager) const documents = await readItemDocuments(ctx, readableItem) - const chunked = chunkItemDocuments(base, documents) + const chunked = chunkKnowledgeDocuments(base, documents) if (chunked.chunks.length === 0) { // Deliberate: the item still completes (an empty material is written) so the // UI doesn't show a stuck/failed item, but leave a trace — an image-only PDF @@ -112,26 +113,9 @@ function loadIndexDocumentsInputOrSkip( ): LoadedIndexDocumentsInput | null { const { baseId, itemId } = ctx.input + let base: KnowledgeBase try { - const base = knowledgeBaseService.getById(baseId) - const item = knowledgeItemService.getById(itemId) - - if (item.status === 'deleting') { - logger.info('Skipping index-documents for deleting item', { baseId, itemId, jobId: ctx.jobId }) - reportKnowledgeProgress(ctx, 100, { stage: 'deleting', currentFile: 1, totalFiles: 1 }) - return null - } - - if (!isIndexableKnowledgeItem(item)) { - throw new Error(`indexDocumentsJobHandler received non-leaf knowledge item: id=${itemId} type=${item.type}`) - } - - if (item.status === 'completed') { - reportKnowledgeProgress(ctx, 100, { stage: 'already-completed', currentFile: 1, totalFiles: 1 }) - return null - } - - return { base, item } + base = knowledgeBaseService.getById(baseId) } catch (error) { if (isDataApiNotFoundError(error)) { logger.info('Skipping index-documents for missing base or item', { baseId, itemId, jobId: ctx.jobId }) @@ -140,6 +124,30 @@ function loadIndexDocumentsInputOrSkip( } throw error } + + const result = resolveLiveKnowledgeItem(itemId) + if ('skip' in result) { + if (result.skip === 'deleting') { + logger.info('Skipping index-documents for deleting item', { baseId, itemId, jobId: ctx.jobId }) + reportKnowledgeProgress(ctx, 100, { stage: 'deleting', currentFile: 1, totalFiles: 1 }) + } else { + logger.info('Skipping index-documents for missing base or item', { baseId, itemId, jobId: ctx.jobId }) + reportKnowledgeProgress(ctx, 100, { stage: 'item-gone', currentFile: 1, totalFiles: 1 }) + } + return null + } + const { item } = result + + if (!isIndexableKnowledgeItem(item)) { + throw new Error(`indexDocumentsJobHandler received non-leaf knowledge item: id=${itemId} type=${item.type}`) + } + + if (item.status === 'completed') { + reportKnowledgeProgress(ctx, 100, { stage: 'already-completed', currentFile: 1, totalFiles: 1 }) + return null + } + + return { base, item } } async function readItemDocuments( @@ -216,7 +224,7 @@ function resolveSnapshotCaptureSpec(item: IndexableKnowledgeItem): SnapshotCaptu async function ensureSnapshot( ctx: JobContext, item: IndexableKnowledgeItem, - knowledgeLockManager: KnowledgeLockManager + knowledgeLockManager: KeyedMutex ): Promise { const spec = resolveSnapshotCaptureSpec(item) if (!spec) { @@ -225,7 +233,7 @@ async function ensureSnapshot( const markdown = await spec.produce(ctx.signal) - return await knowledgeLockManager.withBaseMutationLock(ctx.input.baseId, async () => { + return await knowledgeLockManager.runExclusive(ctx.input.baseId, async () => { const latest = knowledgeItemService.getById(ctx.input.itemId) if (latest.type !== spec.type || latest.data.relativePath) { // Another job captured the snapshot (or the item changed) while we produced. @@ -238,10 +246,6 @@ async function ensureSnapshot( }) } -function chunkItemDocuments(base: KnowledgeBase, documents: LoadedDocuments): ChunkedKnowledgeContent { - return chunkKnowledgeDocuments(base, documents) -} - /** * Embed the distinct chunk bodies and assemble the atomic rebuild input. Bodies * are deduped by embedding-text hash so identical chunks are embedded once; the @@ -269,8 +273,8 @@ async function buildRebuildMaterialInput( let embeddings: RebuildMaterialEmbeddingInput[] = [] if (usesEmbeddings) { const vectorStoreService = application.get('KnowledgeVectorStoreService') - const store = await vectorStoreService.getIndexStore(base) - const existingHashes = await store.listExistingEmbeddingHashes([...bodyByHash.keys()]) + const store = vectorStoreService.getIndexStore(base) + const existingHashes = store.listExistingEmbeddingHashes([...bodyByHash.keys()]) const missing = [...bodyByHash.entries()].filter(([hash]) => !existingHashes.has(hash)) const vectors = await embedKnowledgeTexts( base, @@ -302,21 +306,21 @@ async function writeItemMaterial( ctx: JobContext, base: KnowledgeBase, input: RebuildMaterialInput, - knowledgeLockManager: KnowledgeLockManager + knowledgeLockManager: KeyedMutex ): Promise { const { baseId, itemId } = ctx.input - await knowledgeLockManager.withBaseMutationLock(baseId, async () => { + await knowledgeLockManager.runExclusive(baseId, async () => { ctx.signal.throwIfAborted() - const latestItem = knowledgeItemService.getById(itemId) - if (latestItem.status === 'deleting') { + const result = resolveLiveKnowledgeItem(itemId) + if ('skip' in result) { logger.info('Skipping material rebuild for deleting item', { baseId, itemId, jobId: ctx.jobId }) return } const vectorStoreService = application.get('KnowledgeVectorStoreService') - const store = await vectorStoreService.getIndexStore(base) - await store.rebuildMaterial(itemId, input) + const store = vectorStoreService.getIndexStore(base) + store.rebuildMaterial(itemId, input) knowledgeItemService.updateStatus(itemId, 'completed') }) } diff --git a/src/main/features/knowledge/tasks/jobTypes.ts b/src/main/features/knowledge/tasks/jobTypes.ts index 9d537241df6..c9dc88cfa6c 100644 --- a/src/main/features/knowledge/tasks/jobTypes.ts +++ b/src/main/features/knowledge/tasks/jobTypes.ts @@ -15,7 +15,6 @@ declare module '@main/core/job/jobRegistry' { 'knowledge.index-documents': { baseId: string itemId: string - parentJobId: string | null } 'knowledge.check-file-processing-result': { baseId: string @@ -23,7 +22,7 @@ declare module '@main/core/job/jobRegistry' { fileProcessingJobId: string pollRound: number firstScheduledAt: number - parentJobId: string | null + processedRelativePath: string } 'knowledge.delete-subtree': { baseId: string diff --git a/src/main/features/knowledge/utils/sources/prepare.ts b/src/main/features/knowledge/tasks/prepareItem.ts similarity index 60% rename from src/main/features/knowledge/utils/sources/prepare.ts rename to src/main/features/knowledge/tasks/prepareItem.ts index 19e0891379f..2ae2d1d6f62 100644 --- a/src/main/features/knowledge/utils/sources/prepare.ts +++ b/src/main/features/knowledge/tasks/prepareItem.ts @@ -7,10 +7,13 @@ import { type KnowledgeItemType } from '@shared/data/types/knowledge' -import type { IndexableKnowledgeItem } from '../../types/items' -import { isContainerKnowledgeItem, isIndexableKnowledgeItem } from '../items' -import { collectKnowledgeReservedRelativePaths } from '../storage/pathStorage' -import { expandDirectoryOwnerToTree, type ExpandedDirectoryNode } from './directory' +import { type IndexableKnowledgeItem, isIndexableKnowledgeItem } from '../items' +import { collectKnowledgeReservedRelativePaths } from '../pathStorage' +import { + chooseDirectoryPathPrefix, + expandDirectoryOwnerToTree, + type ExpandedDirectoryNode +} from '../pipeline/sources/directory' const logger = loggerService.withContext('KnowledgePrepare') const EMPTY_DIRECTORY_ERROR = 'Directory contains no indexable files' @@ -18,16 +21,12 @@ const EMPTY_DIRECTORY_ERROR = 'Directory contains no indexable files' export interface PrepareKnowledgeItemOptions { baseId: string item: KnowledgeItem - onCreatedItem: (item: KnowledgeItem) => void - runMutation: (task: () => T) => Promise signal: AbortSignal } export async function prepareKnowledgeItem({ baseId, item, - onCreatedItem, - runMutation, signal }: PrepareKnowledgeItemOptions): Promise { signal.throwIfAborted() @@ -36,20 +35,27 @@ export async function prepareKnowledgeItem({ return [item] } - return await prepareDirectoryForRuntime(baseId, item, onCreatedItem, runMutation, signal) + return await prepareDirectoryForRuntime(baseId, item, signal) } async function prepareDirectoryForRuntime( baseId: string, item: KnowledgeItemOf<'directory'>, - onCreatedItem: (item: KnowledgeItem) => void, - runMutation: (task: () => T) => Promise, signal: AbortSignal ): Promise { // Exclude this container itself: on reindex it already owns its `relativePath` // prefix, and counting it as reserved would self-collide it to `_1` every time. const reservedTopLevelNames = collectReservedTopLevelNames(baseId, item.id) - const { pathPrefix, children } = await expandDirectoryOwnerToTree(item, baseId, reservedTopLevelNames, signal) + const pathPrefix = chooseDirectoryPathPrefix(item, reservedTopLevelNames) + + // Pin the deduped `raw/` prefix onto the container BEFORE any byte is copied. Expansion + // durably writes files under `raw//...`; if a crash/kill/abort lands mid-copy, + // the pinned `relativePath` lets the next attempt's `deletePreviousLeafExpansion` reclaim + // the whole shell (orphan bytes are always a subset of `raw/`). The UI also + // shows the on-disk name (e.g. `docs_2`) and delete removes the shell by it. + knowledgeItemService.updateDirectoryRelativePath(item.id, pathPrefix) + + const children = await expandDirectoryOwnerToTree(item, baseId, pathPrefix, signal) signal.throwIfAborted() if (children.length === 0) { @@ -58,15 +64,11 @@ async function prepareDirectoryForRuntime( itemId: item.id, source: item.data.source }) - await runMutation(() => knowledgeItemService.updateStatus(item.id, 'failed', { error: EMPTY_DIRECTORY_ERROR })) + knowledgeItemService.updateStatus(item.id, 'failed', { error: EMPTY_DIRECTORY_ERROR }) return [] } - // Pin the deduped `raw/` prefix the children were stored under onto the container, so the - // UI shows the on-disk name (e.g. `docs_2`) and delete can remove the whole shell by it. - await runMutation(() => knowledgeItemService.updateDirectoryRelativePath(item.id, pathPrefix)) - - return await createDirectoryChildren(baseId, item.id, children, onCreatedItem, runMutation, signal) + return await createDirectoryChildren(baseId, item.id, children, signal) } /** @@ -92,8 +94,6 @@ async function createDirectoryChildren( baseId: string, parentId: string, children: ExpandedDirectoryNode[], - onCreatedItem: (item: KnowledgeItem) => void, - runMutation: (task: () => T) => Promise, signal: AbortSignal ): Promise { const leafItems: IndexableKnowledgeItem[] = [] @@ -109,8 +109,6 @@ async function createDirectoryChildren( type: 'file', data: child.data }, - onCreatedItem, - runMutation, signal ) leafItems.push(createdFile) @@ -124,19 +122,10 @@ async function createDirectoryChildren( type: 'directory', data: child.data }, - onCreatedItem, - runMutation, - signal - ) - const childLeafItems = await createDirectoryChildren( - baseId, - createdDirectory.id, - child.children, - onCreatedItem, - runMutation, signal ) - await runMutation(() => knowledgeItemService.updateStatus(createdDirectory.id, 'processing')) + const childLeafItems = await createDirectoryChildren(baseId, createdDirectory.id, child.children, signal) + knowledgeItemService.updateStatus(createdDirectory.id, 'processing') leafItems.push(...childLeafItems) } @@ -146,20 +135,11 @@ async function createDirectoryChildren( async function createRuntimeItem( baseId: string, item: Extract, - onCreatedItem: (item: KnowledgeItem) => void, - runMutation: (task: () => TResult) => Promise, signal: AbortSignal ): Promise> { signal.throwIfAborted() - const createdItem = await runMutation(() => knowledgeItemService.create(baseId, item)) - onCreatedItem(createdItem) - - const processingItem = await runMutation(() => - isContainerKnowledgeItem(createdItem) - ? knowledgeItemService.updateStatus(createdItem.id, 'preparing') - : knowledgeItemService.updateStatus(createdItem.id, 'processing') - ) + const createdItem = knowledgeItemService.createActive(baseId, item) signal.throwIfAborted() - return processingItem as KnowledgeItemOf + return createdItem as KnowledgeItemOf } diff --git a/src/main/features/knowledge/tasks/prepareRootJobHandler.ts b/src/main/features/knowledge/tasks/prepareRootJobHandler.ts index c41f9495b73..b4685876e90 100644 --- a/src/main/features/knowledge/tasks/prepareRootJobHandler.ts +++ b/src/main/features/knowledge/tasks/prepareRootJobHandler.ts @@ -3,25 +3,27 @@ import './jobTypes' import { knowledgeBaseService } from '@data/services/KnowledgeBaseService' import { knowledgeItemService } from '@data/services/KnowledgeItemService' import { loggerService } from '@logger' +import type { KeyedMutex } from '@main/core/concurrency/KeyedMutex' import type { JobContext, JobHandler } from '@main/core/job/types' +import { removeDir } from '@main/utils/file/fs' +import { isDataApiNotFoundError } from '@shared/data/api' import type { KnowledgeItem } from '@shared/data/types/knowledge' -import type { KnowledgeLockManager } from '../KnowledgeLockManager' -import type { KnowledgeWorkflowService } from '../KnowledgeWorkflowService' +import type { KnowledgeItemScheduler } from '../ingestion/KnowledgeIngestionService' +import { markUnscheduledKnowledgeItemsFailed } from '../ingestion/statusCleanup' +import { purgeKnowledgeSubtreeWithinLock } from '../ingestion/subtreePurge' +import { getKnowledgeBaseFilePath } from '../pathStorage' import { knowledgeQueueName, reportKnowledgeProgress, toKnowledgeBaseId, toKnowledgeItemId } from '../types' -import { markUnscheduledKnowledgeItemsFailed } from '../utils/cleanup/statusCleanup' -import { deleteKnowledgeItemVectors } from '../utils/cleanup/vectorCleanup' -import { isIndexableKnowledgeItem } from '../utils/items' -import { prepareKnowledgeItem } from '../utils/sources/prepare' -import { deleteKnowledgeItemFilesBestEffort } from '../utils/storage/pathStorage' import type { KnowledgePrepareRootPayload } from './jobTypes' -import { isDataApiNotFoundError, markKnowledgeItemFailedOnSettled } from './utils/settled' +import { prepareKnowledgeItem } from './prepareItem' +import { resolveLiveKnowledgeItem } from './utils/liveItem' +import { markKnowledgeItemFailedOnSettled } from './utils/settled' const logger = loggerService.withContext('Knowledge:PrepareRootJobHandler') export function createPrepareRootJobHandler( - knowledgeLockManager: KnowledgeLockManager, - workflowService: KnowledgeWorkflowService + knowledgeLockManager: KeyedMutex, + ingestionService: KnowledgeItemScheduler ): JobHandler { return { // Don't auto-resume on restart — a deliberate app quit must not re-spend the @@ -56,7 +58,7 @@ export function createPrepareRootJobHandler( // Source expansion creates child items, so it runs under the base mutation lock. const leafItems = await scanRootItem(ctx, knowledgeLockManager) // Child indexing is scheduled after expansion succeeds so partial scans do not enqueue stale leaves. - await enqueueLeafItems(ctx, leafItems, workflowService) + await enqueueLeafItems(ctx, leafItems, ingestionService) }, async onSettled(event) { @@ -70,15 +72,6 @@ function loadPrepareRootItemOrSkip(ctx: JobContext) try { knowledgeBaseService.getById(baseId) - const item = knowledgeItemService.getById(itemId) - - if (item.status === 'deleting') { - logger.info('Skipping prepare-root for deleting item', { baseId, itemId, jobId: ctx.jobId }) - reportKnowledgeProgress(ctx, 100, { stage: 'deleting' }) - return null - } - - return item } catch (error) { if (isDataApiNotFoundError(error)) { logger.info('Skipping prepare-root for missing base or item', { baseId, itemId, jobId: ctx.jobId }) @@ -87,57 +80,70 @@ function loadPrepareRootItemOrSkip(ctx: JobContext) } throw error } + + const result = resolveLiveKnowledgeItem(itemId) + if ('skip' in result) { + if (result.skip === 'deleting') { + logger.info('Skipping prepare-root for deleting item', { baseId, itemId, jobId: ctx.jobId }) + reportKnowledgeProgress(ctx, 100, { stage: 'deleting' }) + } else { + logger.info('Skipping prepare-root for missing base or item', { baseId, itemId, jobId: ctx.jobId }) + reportKnowledgeProgress(ctx, 100, { stage: 'item-gone' }) + } + return null + } + + return result.item } async function deletePreviousLeafExpansion( baseId: string, itemId: string, - knowledgeLockManager: KnowledgeLockManager + knowledgeLockManager: KeyedMutex ): Promise { - await knowledgeLockManager.withBaseMutationLock(baseId, async () => { + await knowledgeLockManager.runExclusive(baseId, async () => { const base = knowledgeBaseService.getById(baseId) const descendants = knowledgeItemService.getSubtreeItems(baseId, [itemId]) const removableDescendants = descendants.filter((item) => item.status !== 'deleting') - const removableDescendantIds = removableDescendants.map((item) => item.id) - const removableLeafIds = removableDescendants.filter(isIndexableKnowledgeItem).map((item) => item.id) - - await deleteKnowledgeItemVectors(base, removableLeafIds) - // Best-effort: a file-removal failure must not abort the row deletion below. - await deleteKnowledgeItemFilesBestEffort(baseId, removableDescendants, { baseId, itemId }) - knowledgeItemService.deleteItemsByIds(baseId, removableDescendantIds) + await purgeKnowledgeSubtreeWithinLock(base, removableDescendants, { baseId, itemId }) + + // `getSubtreeItems` excludes the container row, so the purge above never touches the + // container's own `raw/` shell. A prior attempt pins `relativePath` before + // copying any byte (see prepareDirectoryForRuntime), so if orphan bytes exist the row + // records their prefix — reclaim the whole shell here. removeDir is idempotent (ENOENT + // no-op) when the shell was never created. + const result = resolveLiveKnowledgeItem(itemId) + if ('item' in result && result.item.type === 'directory') { + const prefix = result.item.data.relativePath + if (prefix) { + await removeDir(getKnowledgeBaseFilePath(baseId, prefix)) + } + } }) } async function scanRootItem( ctx: JobContext, - knowledgeLockManager: KnowledgeLockManager + knowledgeLockManager: KeyedMutex ): Promise { const { baseId, itemId } = ctx.input - return await knowledgeLockManager.withBaseMutationLock(baseId, async () => { - let currentItem: KnowledgeItem - try { - currentItem = knowledgeItemService.getById(itemId) - } catch (error) { - if (isDataApiNotFoundError(error)) { + return await knowledgeLockManager.runExclusive(baseId, async () => { + const result = resolveLiveKnowledgeItem(itemId) + if ('skip' in result) { + if (result.skip === 'deleting') { + logger.info('Skipping prepare-root for deleting item before expansion', { baseId, itemId, jobId: ctx.jobId }) + reportKnowledgeProgress(ctx, 100, { stage: 'deleting' }) + } else { logger.info('Skipping prepare-root for missing item before expansion', { baseId, itemId, jobId: ctx.jobId }) reportKnowledgeProgress(ctx, 100, { stage: 'item-gone' }) - return [] } - throw error - } - - if (currentItem.status === 'deleting') { - logger.info('Skipping prepare-root for deleting item before expansion', { baseId, itemId, jobId: ctx.jobId }) - reportKnowledgeProgress(ctx, 100, { stage: 'deleting' }) return [] } const leaves = await prepareKnowledgeItem({ baseId, - item: currentItem, - onCreatedItem: () => {}, - runMutation: async (task) => await task(), + item: result.item, signal: ctx.signal }) if (leaves.length > 0) { @@ -150,7 +156,7 @@ async function scanRootItem( async function enqueueLeafItems( ctx: JobContext, leafItems: KnowledgeItem[], - workflowService: KnowledgeWorkflowService + ingestionService: KnowledgeItemScheduler ): Promise { const { baseId } = ctx.input @@ -160,7 +166,7 @@ async function enqueueLeafItems( for (const [index, leaf] of leafItems.entries()) { ctx.signal.throwIfAborted() try { - await workflowService.scheduleItem(baseIdInput, toKnowledgeItemId(leaf.id), ctx.jobId) + await ingestionService.scheduleItem(baseIdInput, toKnowledgeItemId(leaf.id), ctx.jobId) completedSchedulingLeafIds.add(leaf.id) } catch (error) { markUnscheduledLeafItemsFailed(baseId, leafItems, completedSchedulingLeafIds, error) @@ -190,7 +196,6 @@ function markUnscheduledLeafItemsFailed( errorMessage: message, failedStatusError: `Failed to schedule knowledge child item job: ${message}`, logger, - logMessage: 'Failed to mark unscheduled knowledge child item after prepare-root scheduling failure', - logContextKey: 'scheduleError' + logMessage: 'Failed to mark unscheduled knowledge child item after prepare-root scheduling failure' }) } diff --git a/src/main/features/knowledge/tasks/reindexSubtreeJobHandler.ts b/src/main/features/knowledge/tasks/reindexSubtreeJobHandler.ts index ef571f05825..252c75627c1 100644 --- a/src/main/features/knowledge/tasks/reindexSubtreeJobHandler.ts +++ b/src/main/features/knowledge/tasks/reindexSubtreeJobHandler.ts @@ -4,32 +4,26 @@ import { application } from '@application' import { knowledgeBaseService } from '@data/services/KnowledgeBaseService' import { knowledgeItemService } from '@data/services/KnowledgeItemService' import { loggerService } from '@logger' +import type { KeyedMutex } from '@main/core/concurrency/KeyedMutex' import type { JobHandler, JobSettledEvent } from '@main/core/job/types' -import type { JobSnapshot } from '@shared/data/api/schemas/jobs' +import { ACTIVE_JOB_STATUSES, type JobSnapshot } from '@shared/data/api/schemas/jobs' import { KNOWLEDGE_ITEM_ERROR_INDEXING_INTERRUPTED, type KnowledgeItemStatus } from '@shared/data/types/knowledge' -import type { KnowledgeLockManager } from '../KnowledgeLockManager' -import type { KnowledgeWorkflowService } from '../KnowledgeWorkflowService' -import { - KNOWLEDGE_ACTIVE_JOB_LIMIT, - KNOWLEDGE_ACTIVE_JOB_STATUSES, - knowledgeQueueName, - reportKnowledgeProgress, - toKnowledgeBaseId, - toKnowledgeItemId -} from '../types' -import { deleteKnowledgeItemVectors } from '../utils/cleanup/vectorCleanup' -import { canKnowledgeItemRebuildSource, isContainerKnowledgeItem } from '../utils/items' -import { deleteKnowledgeItemFilesBestEffort } from '../utils/storage/pathStorage' +import type { KnowledgeItemScheduler } from '../ingestion/KnowledgeIngestionService' +import { canKnowledgeItemRebuildSource, isContainerKnowledgeItem } from '../items' +import { deleteKnowledgeItemFilesBestEffort } from '../pathStorage' +import { deleteKnowledgeItemVectors } from '../pipeline/vectorstore/vectorCleanup' +import { knowledgeQueueName, reportKnowledgeProgress, toKnowledgeBaseId, toKnowledgeItemId } from '../types' import type { KnowledgeReindexSubtreePayload } from './jobTypes' import { narrowKnowledgeJobInput } from './utils/jobInput' +import { resolveLiveKnowledgeSubtree } from './utils/liveItem' const logger = loggerService.withContext('Knowledge:ReindexSubtreeJobHandler') const REINDEX_RECOVERY_ACTIVE_STATUSES = new Set(['preparing', 'processing']) export function createReindexSubtreeJobHandler( - knowledgeLockManager: KnowledgeLockManager, - workflowService: KnowledgeWorkflowService + knowledgeLockManager: KeyedMutex, + ingestionService: KnowledgeItemScheduler ): JobHandler { return { // Don't auto-resume on restart — a deliberate app quit must not re-spend the @@ -58,15 +52,16 @@ export function createReindexSubtreeJobHandler( } // Reset vectors, expanded children, and root statuses as one base-level mutation. - const resetResult = await knowledgeLockManager.withBaseMutationLock(baseId, async () => { + const resetResult = await knowledgeLockManager.runExclusive(baseId, async () => { const base = knowledgeBaseService.getById(baseId) - const rootItems = knowledgeItemService.getSubtreeItems(baseId, rootItemIds, { includeRoots: true }) // Re-check under the mutation lock so reindex cannot turn a just-deleted // subtree back into preparing/processing during cleanup/reset. - if (rootItems.some((item) => item.status === 'deleting')) { + const subtreeResult = resolveLiveKnowledgeSubtree(baseId, rootItemIds) + if ('skip' in subtreeResult) { logger.info('Skipping reindex-subtree reset for deleting subtree', { baseId, rootItemIds, jobId: ctx.jobId }) return { roots: [], skippedDeleting: true, skippedMissingSource: 0 } } + const rootItems = subtreeResult.items const selectedRoots = rootItems.filter((item) => rootItemIds.includes(item.id)) // Admission (assertSubtreesCanReindex) already rejected roots whose source is gone, but the @@ -133,7 +128,7 @@ export function createReindexSubtreeJobHandler( try { for (const item of resetResult.roots) { ctx.signal.throwIfAborted() - await workflowService.scheduleItem(toKnowledgeBaseId(baseId), toKnowledgeItemId(item.id), ctx.jobId) + await ingestionService.scheduleItem(toKnowledgeBaseId(baseId), toKnowledgeItemId(item.id), ctx.jobId) completedSchedulingRootIds.add(item.id) } } catch (error) { @@ -174,36 +169,36 @@ export function createReindexSubtreeJobHandler( } function shouldSkipDeletingSubtreeReindex(baseId: string, rootItemIds: string[], jobId: string): boolean { - const subtreeItems = knowledgeItemService.getSubtreeItems(baseId, rootItemIds, { includeRoots: true }) - const hasDeletingItem = subtreeItems.some((item) => item.status === 'deleting') - if (hasDeletingItem) { + const result = resolveLiveKnowledgeSubtree(baseId, rootItemIds) + const isSkipped = 'skip' in result + if (isSkipped) { logger.info('Skipping reindex-subtree for deleting subtree', { baseId, rootItemIds, jobId }) } - return hasDeletingItem + return isSkipped } -async function markReindexSubtreeFailedOnSettled(event: JobSettledEvent): Promise { +async function markReindexSubtreeFailedOnSettled( + event: JobSettledEvent +): Promise { if (event.status === 'completed') return - const jobManager = application.get('JobManager') - const snapshot = await jobManager.get(event.jobId) - const narrowed = snapshot ? narrowKnowledgeJobInput(snapshot) : null - if (!narrowed || !('rootItemIds' in narrowed.input) || narrowed.input.rootItemIds.length === 0) return - const { input } = narrowed + const { baseId, rootItemIds } = event.input + if (rootItemIds.length === 0) return const reason = event.error?.message?.trim() || `Job ${event.status}` try { + const jobManager = application.get('JobManager') const activeJobs = await jobManager.list({ - queue: knowledgeQueueName(toKnowledgeBaseId(input.baseId)), - status: [...KNOWLEDGE_ACTIVE_JOB_STATUSES], - limit: KNOWLEDGE_ACTIVE_JOB_LIMIT + queue: knowledgeQueueName(toKnowledgeBaseId(baseId)), + status: [...ACTIVE_JOB_STATUSES], + parentId: event.jobId }) - const rootsWithFollowUpJobs = getRootsWithFollowUpJobs(activeJobs, event.jobId, input.rootItemIds) - const rootItems = knowledgeItemService.getSubtreeItems(input.baseId, input.rootItemIds, { + const rootsWithFollowUpJobs = getRootsWithFollowUpJobs(activeJobs, rootItemIds) + const rootItems = knowledgeItemService.getSubtreeItems(baseId, rootItemIds, { includeRoots: true }) const rootsToFail = rootItems - .filter((item) => input.rootItemIds.includes(item.id)) + .filter((item) => rootItemIds.includes(item.id)) .filter((item) => REINDEX_RECOVERY_ACTIVE_STATUSES.has(item.status)) .filter((item) => !rootsWithFollowUpJobs.has(item.id)) .map((item) => item.id) @@ -218,26 +213,24 @@ async function markReindexSubtreeFailedOnSettled(event: JobSettledEvent): Promis event.status === 'cancelled' ? KNOWLEDGE_ITEM_ERROR_INDEXING_INTERRUPTED : `Reindex job ${event.status}: ${reason}` - knowledgeItemService.setSubtreeStatus(input.baseId, rootsToFail, 'failed', { error }) + knowledgeItemService.setSubtreeStatus(baseId, rootsToFail, 'failed', { error }) } catch (error) { logger.error( 'Failed to flip reindex-subtree targets to failed in onSettled', error instanceof Error ? error : new Error(String(error)), { jobId: event.jobId, - baseId: input.baseId, - rootItemIds: input.rootItemIds + baseId, + rootItemIds } ) } } -function getRootsWithFollowUpJobs(activeJobs: JobSnapshot[], reindexJobId: string, rootItemIds: string[]): Set { +function getRootsWithFollowUpJobs(activeJobs: JobSnapshot[], rootItemIds: string[]): Set { const rootItemIdSet = new Set(rootItemIds) const rootsWithFollowUpJobs = new Set() for (const job of activeJobs) { - if (job.parentId !== reindexJobId) continue - const narrowed = narrowKnowledgeJobInput(job) if (narrowed && 'itemId' in narrowed.input && rootItemIdSet.has(narrowed.input.itemId)) { rootsWithFollowUpJobs.add(narrowed.input.itemId) diff --git a/src/main/features/knowledge/tasks/utils/__tests__/cancel.test.ts b/src/main/features/knowledge/tasks/utils/__tests__/cancel.test.ts new file mode 100644 index 00000000000..983610f41ba --- /dev/null +++ b/src/main/features/knowledge/tasks/utils/__tests__/cancel.test.ts @@ -0,0 +1,231 @@ +import type { JobSnapshot } from '@shared/data/api/schemas/jobs' +import type { KnowledgeItem } from '@shared/data/types/knowledge' +import { beforeEach, describe, expect, it, vi } from 'vitest' + +import { KNOWLEDGE_ACTIVE_JOB_LIMIT } from '../../../types' + +const { cancelMock, listMock, knowledgeItemGetSubtreeItemsMock } = vi.hoisted(() => ({ + cancelMock: vi.fn(), + listMock: vi.fn(), + knowledgeItemGetSubtreeItemsMock: vi.fn() +})) + +vi.mock('@application', async () => { + const { mockApplicationFactory } = await import('@test-mocks/main/application') + return mockApplicationFactory({ + JobManager: { + cancel: cancelMock, + list: listMock + } + } as Parameters[0]) +}) + +vi.mock('@data/services/KnowledgeItemService', () => ({ + knowledgeItemService: { + getSubtreeItems: knowledgeItemGetSubtreeItemsMock + } +})) + +const { cancelActiveKnowledgeJobs } = await import('../cancel') + +type JobSnapshotInput = Pick & Partial + +function createJobSnapshot(overrides: JobSnapshotInput): JobSnapshot { + return { + id: 'job-1', + status: 'running', + priority: 0, + queue: 'base.kb-1', + idempotencyKey: null, + scheduleId: null, + scheduledAt: '2026-04-08T00:00:00.000Z', + startedAt: '2026-04-08T00:00:00.000Z', + finishedAt: null, + attempt: 1, + maxAttempts: 3, + output: null, + error: null, + parentId: null, + cancelRequested: false, + metadata: {}, + timeoutMs: null, + createdAt: '2026-04-08T00:00:00.000Z', + updatedAt: '2026-04-08T00:00:00.000Z', + ...overrides + } +} + +function createItem(id: string): KnowledgeItem { + return { + id, + baseId: 'kb-1', + groupId: null, + type: 'note', + data: { source: 'note', content: 'text' }, + status: 'processing', + error: null, + createdAt: '2026-04-08T00:00:00.000Z', + updatedAt: '2026-04-08T00:00:00.000Z' + } +} + +describe('cancelActiveKnowledgeJobs', () => { + beforeEach(() => { + cancelMock.mockReset().mockResolvedValue({ outcome: 'cancelled' }) + listMock.mockReset().mockResolvedValue([]) + knowledgeItemGetSubtreeItemsMock.mockReset() + }) + + it('cancels every active knowledge job in the base queue when scoped base-wide', async () => { + listMock.mockResolvedValue([ + createJobSnapshot({ + id: 'prepare-job', + type: 'knowledge.prepare-root', + input: { baseId: 'kb-1', itemId: 'dir-1' } + }), + createJobSnapshot({ + id: 'check-job', + type: 'knowledge.check-file-processing-result', + input: { + baseId: 'kb-1', + itemId: 'file-1', + fileProcessingJobId: 'fp-job-1', + pollRound: 0, + firstScheduledAt: 1779811200000, + processedRelativePath: 'source.md' + } + }) + ]) + + await cancelActiveKnowledgeJobs('kb-1', 'delete-base', { onCancelTimeout: 'proceed' }) + + expect(knowledgeItemGetSubtreeItemsMock).not.toHaveBeenCalled() + expect(cancelMock).toHaveBeenCalledWith('prepare-job', 'delete-base') + expect(cancelMock).toHaveBeenCalledWith('fp-job-1', 'delete-base') + }) + + it('returns without listing jobs when the requested subtree resolves empty', async () => { + knowledgeItemGetSubtreeItemsMock.mockReturnValue([]) + + await cancelActiveKnowledgeJobs('kb-1', 'knowledge-delete-subtree', { + rootItemIds: ['missing-root'], + onCancelTimeout: 'throw' + }) + + expect(listMock).not.toHaveBeenCalled() + expect(cancelMock).not.toHaveBeenCalled() + }) + + it('scopes cancellation to jobs touching the resolved subtree and excludes the given job id', async () => { + knowledgeItemGetSubtreeItemsMock.mockReturnValue([createItem('dir-1'), createItem('note-1')]) + listMock.mockResolvedValue([ + createJobSnapshot({ + id: 'current-job', + type: 'knowledge.delete-subtree', + input: { baseId: 'kb-1', rootItemIds: ['dir-1'] } + }), + createJobSnapshot({ + id: 'index-job', + type: 'knowledge.index-documents', + input: { baseId: 'kb-1', itemId: 'note-1' } + }), + createJobSnapshot({ + id: 'unrelated-job', + type: 'knowledge.index-documents', + input: { baseId: 'kb-1', itemId: 'other' } + }) + ]) + + await cancelActiveKnowledgeJobs('kb-1', 'knowledge-delete-subtree', { + rootItemIds: ['dir-1'], + excludeJobId: 'current-job', + onCancelTimeout: 'throw' + }) + + expect(cancelMock).toHaveBeenCalledTimes(1) + expect(cancelMock).toHaveBeenCalledWith('index-job', 'knowledge-delete-subtree') + }) + + it('throws when a scoped cancel times out', async () => { + knowledgeItemGetSubtreeItemsMock.mockReturnValue([createItem('note-1')]) + listMock.mockResolvedValue([ + createJobSnapshot({ + id: 'index-job', + type: 'knowledge.index-documents', + input: { baseId: 'kb-1', itemId: 'note-1' } + }) + ]) + cancelMock.mockResolvedValue({ outcome: 'timed-out' }) + + await expect( + cancelActiveKnowledgeJobs('kb-1', 'knowledge-delete-subtree', { + rootItemIds: ['note-1'], + onCancelTimeout: 'throw' + }) + ).rejects.toThrow('Job cancel timed out: index-job') + }) + + it('does not throw on a cancel timeout when proceeding base-wide', async () => { + listMock.mockResolvedValue([ + createJobSnapshot({ + id: 'index-job', + type: 'knowledge.index-documents', + input: { baseId: 'kb-1', itemId: 'note-1' } + }) + ]) + cancelMock.mockResolvedValue({ outcome: 'timed-out' }) + + await expect( + cancelActiveKnowledgeJobs('kb-1', 'delete-base', { onCancelTimeout: 'proceed' }) + ).resolves.toBeUndefined() + }) + + it('drains a full page and re-queries until a page comes back short', async () => { + const fullPage = Array.from({ length: KNOWLEDGE_ACTIVE_JOB_LIMIT }, (_, index) => + createJobSnapshot({ + id: `index-job-${index}`, + type: 'knowledge.index-documents', + input: { baseId: 'kb-1', itemId: `note-${index}` } + }) + ) + listMock.mockResolvedValueOnce(fullPage).mockResolvedValueOnce([]) + + await cancelActiveKnowledgeJobs('kb-1', 'delete-base', { onCancelTimeout: 'proceed' }) + + expect(listMock).toHaveBeenCalledTimes(2) + expect(cancelMock).toHaveBeenCalledTimes(KNOWLEDGE_ACTIVE_JOB_LIMIT) + }) + + it('terminates a full page of scoped-out jobs instead of re-querying it forever', async () => { + knowledgeItemGetSubtreeItemsMock.mockReturnValue([createItem('target')]) + // A full page whose every job touches an item OUTSIDE the requested subtree: nothing + // is cancellable, so the active set never shrinks. `mockResolvedValue` (not `Once`) + // returns the same page on every call, so a loop that only breaks on a short page + // would spin forever here — the guard must stop on the first no-progress round. + const fullPageOfMisses = Array.from({ length: KNOWLEDGE_ACTIVE_JOB_LIMIT }, (_, index) => + createJobSnapshot({ + id: `index-job-${index}`, + type: 'knowledge.index-documents', + input: { baseId: 'kb-1', itemId: `other-${index}` } + }) + ) + listMock.mockResolvedValue(fullPageOfMisses) + + await cancelActiveKnowledgeJobs('kb-1', 'knowledge-delete-subtree', { + rootItemIds: ['target'], + onCancelTimeout: 'throw' + }) + + expect(listMock).toHaveBeenCalledTimes(1) + expect(cancelMock).not.toHaveBeenCalled() + }) + + it('stops after a single empty page without cancelling anything', async () => { + listMock.mockResolvedValueOnce([]) + + await cancelActiveKnowledgeJobs('kb-1', 'delete-base', { onCancelTimeout: 'proceed' }) + + expect(listMock).toHaveBeenCalledTimes(1) + expect(cancelMock).not.toHaveBeenCalled() + }) +}) diff --git a/src/main/features/knowledge/tasks/utils/__tests__/jobInput.test.ts b/src/main/features/knowledge/tasks/utils/__tests__/jobInput.test.ts index 9d13fb5aa22..adbf2b35d41 100644 --- a/src/main/features/knowledge/tasks/utils/__tests__/jobInput.test.ts +++ b/src/main/features/knowledge/tasks/utils/__tests__/jobInput.test.ts @@ -7,34 +7,13 @@ describe('narrowKnowledgeJobInput', () => { expect( narrowKnowledgeJobInput({ type: 'knowledge.index-documents', - input: { baseId: 'kb-1', itemId: 'note-1', parentJobId: null } + input: { baseId: 'kb-1', itemId: 'note-1' } }) ).toEqual({ type: 'knowledge.index-documents', input: { baseId: 'kb-1', - itemId: 'note-1', - parentJobId: null - } - }) - }) - - it('preserves index-documents payload fields', () => { - expect( - narrowKnowledgeJobInput({ - type: 'knowledge.index-documents', - input: { - baseId: 'kb-1', - itemId: 'file-1', - parentJobId: 'reindex-job' - } - }) - ).toEqual({ - type: 'knowledge.index-documents', - input: { - baseId: 'kb-1', - itemId: 'file-1', - parentJobId: 'reindex-job' + itemId: 'note-1' } }) }) @@ -49,7 +28,7 @@ describe('narrowKnowledgeJobInput', () => { fileProcessingJobId: 'fp-job-1', pollRound: 2, firstScheduledAt: 1779811200000, - parentJobId: 'reindex-job' + processedRelativePath: 'source.md' } }) ).toEqual({ @@ -60,7 +39,7 @@ describe('narrowKnowledgeJobInput', () => { fileProcessingJobId: 'fp-job-1', pollRound: 2, firstScheduledAt: 1779811200000, - parentJobId: 'reindex-job' + processedRelativePath: 'source.md' } }) }) @@ -100,13 +79,7 @@ describe('narrowKnowledgeJobInput', () => { expect( narrowKnowledgeJobInput({ type: 'knowledge.index-documents', - input: { baseId: 'kb-1', itemId: 'file-1' } - }) - ).toBeNull() - expect( - narrowKnowledgeJobInput({ - type: 'knowledge.index-documents', - input: { baseId: 'kb-1', itemId: 'file-1', parentJobId: 1 } + input: { baseId: 'kb-1' } }) ).toBeNull() }) diff --git a/src/main/features/knowledge/tasks/utils/__tests__/liveItem.test.ts b/src/main/features/knowledge/tasks/utils/__tests__/liveItem.test.ts new file mode 100644 index 00000000000..06b68e3cb8e --- /dev/null +++ b/src/main/features/knowledge/tasks/utils/__tests__/liveItem.test.ts @@ -0,0 +1,88 @@ +import { DataApiErrorFactory } from '@shared/data/api' +import type { KnowledgeItemOf } from '@shared/data/types/knowledge' +import { beforeEach, describe, expect, it, vi } from 'vitest' + +const { knowledgeItemGetByIdMock, knowledgeItemGetSubtreeItemsMock } = vi.hoisted(() => ({ + knowledgeItemGetByIdMock: vi.fn(), + knowledgeItemGetSubtreeItemsMock: vi.fn() +})) + +vi.mock('@data/services/KnowledgeItemService', () => ({ + knowledgeItemService: { + getById: knowledgeItemGetByIdMock, + getSubtreeItems: knowledgeItemGetSubtreeItemsMock + } +})) + +const { resolveLiveKnowledgeItem, resolveLiveKnowledgeSubtree } = await import('../liveItem') + +function createItem( + id: string, + status: Exclude['status'], 'failed'> = 'processing' +): KnowledgeItemOf<'note'> { + return { + id, + baseId: 'kb-1', + groupId: null, + type: 'note', + data: { source: 'note', content: 'text' }, + status, + error: null, + createdAt: '2026-04-08T00:00:00.000Z', + updatedAt: '2026-04-08T00:00:00.000Z' + } +} + +describe('resolveLiveKnowledgeItem', () => { + beforeEach(() => { + knowledgeItemGetByIdMock.mockReset() + }) + + it('returns the item when it is live', () => { + const item = createItem('note-1') + knowledgeItemGetByIdMock.mockReturnValue(item) + + expect(resolveLiveKnowledgeItem('note-1')).toEqual({ item }) + }) + + it('classifies a deleting item as a deleting skip', () => { + knowledgeItemGetByIdMock.mockReturnValue(createItem('note-1', 'deleting')) + + expect(resolveLiveKnowledgeItem('note-1')).toEqual({ skip: 'deleting' }) + }) + + it('classifies a NOT_FOUND row as a missing skip', () => { + knowledgeItemGetByIdMock.mockImplementation(() => { + throw DataApiErrorFactory.notFound('KnowledgeItem', 'note-1') + }) + + expect(resolveLiveKnowledgeItem('note-1')).toEqual({ skip: 'missing' }) + }) + + it('rethrows a non-NOT_FOUND error', () => { + knowledgeItemGetByIdMock.mockImplementation(() => { + throw new Error('db busy') + }) + + expect(() => resolveLiveKnowledgeItem('note-1')).toThrow('db busy') + }) +}) + +describe('resolveLiveKnowledgeSubtree', () => { + beforeEach(() => { + knowledgeItemGetSubtreeItemsMock.mockReset() + }) + + it('returns the resolved items when none are deleting', () => { + const items = [createItem('dir-1'), createItem('note-1')] + knowledgeItemGetSubtreeItemsMock.mockReturnValue(items) + + expect(resolveLiveKnowledgeSubtree('kb-1', ['dir-1'])).toEqual({ items }) + }) + + it('skips when any resolved item is deleting', () => { + knowledgeItemGetSubtreeItemsMock.mockReturnValue([createItem('dir-1'), createItem('note-1', 'deleting')]) + + expect(resolveLiveKnowledgeSubtree('kb-1', ['dir-1'])).toEqual({ skip: 'deleting' }) + }) +}) diff --git a/src/main/features/knowledge/tasks/utils/cancel.ts b/src/main/features/knowledge/tasks/utils/cancel.ts index f1fc8ca9068..90c161e4a54 100644 --- a/src/main/features/knowledge/tasks/utils/cancel.ts +++ b/src/main/features/knowledge/tasks/utils/cancel.ts @@ -1,4 +1,9 @@ import { application } from '@application' +import { knowledgeItemService } from '@data/services/KnowledgeItemService' +import { ACTIVE_JOB_STATUSES } from '@shared/data/api/schemas/jobs' + +import { KNOWLEDGE_ACTIVE_JOB_LIMIT, KNOWLEDGE_JOB_TYPES, knowledgeQueueName, toKnowledgeBaseId } from '../../types' +import { narrowKnowledgeJobInput } from './jobInput' export async function cancelJobOrThrow(jobId: string, reason: string): Promise { const result = await application.get('JobManager').cancel(jobId, reason) @@ -6,3 +11,106 @@ export async function cancelJobOrThrow(jobId: string, reason: string): Promise { + const { rootItemIds, excludeJobId, onCancelTimeout } = options + + let subtreeItemIds: Set | undefined + if (rootItemIds) { + const subtreeItems = knowledgeItemService.getSubtreeItems(baseId, rootItemIds, { includeRoots: true }) + subtreeItemIds = new Set(subtreeItems.map((item) => item.id)) + if (subtreeItemIds.size === 0) { + return + } + } + + const jobManager = application.get('JobManager') + const queue = knowledgeQueueName(toKnowledgeBaseId(baseId)) + const cancelOne: (jobId: string) => Promise = + onCancelTimeout === 'throw' + ? (jobId) => cancelJobOrThrow(jobId, reason) + : (jobId) => jobManager.cancel(jobId, reason) + + // Re-query from the top each round (no offset): cancelling shrinks the active-job + // set between rounds, so offset-based paging could skip rows as the window shifts, + // and `list()`'s createdAt-only ordering has no tie-breaker to make offsets stable. + while (true) { + const activeJobs = await jobManager.list({ + queue, + status: [...ACTIVE_JOB_STATUSES], + type: [...KNOWLEDGE_JOB_TYPES], + limit: KNOWLEDGE_ACTIVE_JOB_LIMIT + }) + + const jobIds = activeJobs + .filter((job) => job.id !== excludeJobId) + .filter((job) => !subtreeItemIds || jobTouchesSubtree(job, subtreeItemIds)) + .map((job) => job.id) + const fileProcessingJobIds = activeJobs.flatMap((job) => getLinkedFileProcessingJobIds(job, subtreeItemIds)) + + await Promise.all([...jobIds, ...fileProcessingJobIds].map(cancelOne)) + + // Stop when the page came back short (nothing more to drain) or when a full page + // yielded nothing cancellable this round. The short-page check alone is not enough: + // a scoped cancel whose entire full page misses the subtree cancels nothing, so the + // active set never shrinks and the same page would be re-queried forever. + if (activeJobs.length < KNOWLEDGE_ACTIVE_JOB_LIMIT || jobIds.length + fileProcessingJobIds.length === 0) { + break + } + } +} + +function jobTouchesSubtree(job: { type: string; input: unknown }, subtreeIds: Set): boolean { + const narrowed = narrowKnowledgeJobInput(job) + if (!narrowed) { + return false + } + if ('itemId' in narrowed.input && subtreeIds.has(narrowed.input.itemId)) { + return true + } + return 'rootItemIds' in narrowed.input && narrowed.input.rootItemIds.some((itemId) => subtreeIds.has(itemId)) +} + +/** Linked file-processing jobs for a knowledge job, optionally filtered to a subtree scope. */ +function getLinkedFileProcessingJobIds( + job: { type: string; input: unknown }, + subtreeIds: Set | undefined +): string[] { + const narrowed = narrowKnowledgeJobInput(job) + if ( + narrowed?.type === 'knowledge.check-file-processing-result' && + (!subtreeIds || subtreeIds.has(narrowed.input.itemId)) && + narrowed.input.fileProcessingJobId + ) { + return [narrowed.input.fileProcessingJobId] + } + return [] +} diff --git a/src/main/features/knowledge/tasks/utils/jobInput.ts b/src/main/features/knowledge/tasks/utils/jobInput.ts index f77f9c0c860..4ed1f3d669c 100644 --- a/src/main/features/knowledge/tasks/utils/jobInput.ts +++ b/src/main/features/knowledge/tasks/utils/jobInput.ts @@ -67,9 +67,7 @@ function narrowFileProcessingCheckJobPayload( if (typeof input.fileProcessingJobId !== 'string') return null if (typeof input.pollRound !== 'number') return null if (typeof input.firstScheduledAt !== 'number') return null - if (!('parentJobId' in input)) return null - const parentJobId = input.parentJobId - if (parentJobId !== null && typeof parentJobId !== 'string') return null + if (typeof input.processedRelativePath !== 'string') return null return { baseId: basePayload.baseId, @@ -77,7 +75,7 @@ function narrowFileProcessingCheckJobPayload( fileProcessingJobId: input.fileProcessingJobId, pollRound: input.pollRound, firstScheduledAt: input.firstScheduledAt, - parentJobId + processedRelativePath: input.processedRelativePath } } @@ -96,19 +94,7 @@ function narrowItemJobPayload( } function narrowIndexDocumentsJobPayload(rawInput: JobSnapshot['input']): KnowledgeIndexDocumentsPayload | null { - const input = narrowJobInputObject(rawInput) - if (!input) return null - const basePayload = narrowItemJobPayload(rawInput) - if (!basePayload) return null - if (!('parentJobId' in input)) return null - const parentJobId = input.parentJobId - if (parentJobId !== null && typeof parentJobId !== 'string') return null - - return { - baseId: basePayload.baseId, - itemId: basePayload.itemId, - parentJobId - } + return narrowItemJobPayload(rawInput) } function narrowSubtreeJobPayload( diff --git a/src/main/features/knowledge/tasks/utils/liveItem.ts b/src/main/features/knowledge/tasks/utils/liveItem.ts new file mode 100644 index 00000000000..fa56d9d4e5a --- /dev/null +++ b/src/main/features/knowledge/tasks/utils/liveItem.ts @@ -0,0 +1,33 @@ +import { knowledgeItemService } from '@data/services/KnowledgeItemService' +import { isDataApiNotFoundError } from '@shared/data/api' +import type { KnowledgeItem } from '@shared/data/types/knowledge' + +export type LiveKnowledgeItemSkipReason = 'missing' | 'deleting' +export type LiveKnowledgeItemResult = { item: KnowledgeItem } | { skip: LiveKnowledgeItemSkipReason } + +/** Classify a single item: found and live, or a skip (`item.status === 'deleting'`, or the row is NOT_FOUND). Does not check the base. */ +export function resolveLiveKnowledgeItem(itemId: string): LiveKnowledgeItemResult { + try { + const item = knowledgeItemService.getById(itemId) + if (item.status === 'deleting') { + return { skip: 'deleting' } + } + return { item } + } catch (error) { + if (isDataApiNotFoundError(error)) { + return { skip: 'missing' } + } + throw error + } +} + +export type LiveKnowledgeSubtreeResult = { items: KnowledgeItem[] } | { skip: 'deleting' } + +/** Classify a resolved subtree (roots + descendants): live, or a skip if any member is deleting. */ +export function resolveLiveKnowledgeSubtree(baseId: string, rootItemIds: string[]): LiveKnowledgeSubtreeResult { + const items = knowledgeItemService.getSubtreeItems(baseId, rootItemIds, { includeRoots: true }) + if (items.some((item) => item.status === 'deleting')) { + return { skip: 'deleting' } + } + return { items } +} diff --git a/src/main/features/knowledge/tasks/utils/settled.ts b/src/main/features/knowledge/tasks/utils/settled.ts index ea0b34d0fe5..bee2b8b27e1 100644 --- a/src/main/features/knowledge/tasks/utils/settled.ts +++ b/src/main/features/knowledge/tasks/utils/settled.ts @@ -1,28 +1,17 @@ -import { application } from '@application' import { knowledgeItemService } from '@data/services/KnowledgeItemService' import type { JobSettledEvent } from '@main/core/job/types' import type { LoggerService } from '@main/core/logger/LoggerService' -import { ErrorCode, isDataApiError } from '@shared/data/api' +import { isDataApiNotFoundError } from '@shared/data/api' import { KNOWLEDGE_ITEM_ERROR_INDEXING_INTERRUPTED } from '@shared/data/types/knowledge' -import { narrowKnowledgeJobInput } from './jobInput' - -export function isDataApiNotFoundError(error: unknown): boolean { - return isDataApiError(error) && error.code === ErrorCode.NOT_FOUND -} - export async function markKnowledgeItemFailedOnSettled( - event: JobSettledEvent, + event: JobSettledEvent<{ itemId: string }>, logger: LoggerService, logMessage: string ): Promise { if (event.status === 'completed') return - const jobManager = application.get('JobManager') - const snapshot = await jobManager.get(event.jobId) - const narrowed = snapshot ? narrowKnowledgeJobInput(snapshot) : null - if (!narrowed || !('itemId' in narrowed.input)) return - const { input } = narrowed + const { itemId } = event.input // A cancelled indexing job for a non-deleting item means it was aborted by an // app quit / restart (knowledge has no per-item user cancel), so store the @@ -33,15 +22,15 @@ export async function markKnowledgeItemFailedOnSettled( ? KNOWLEDGE_ITEM_ERROR_INDEXING_INTERRUPTED : event.error?.message?.trim() || `Job ${event.status}` try { - const item = knowledgeItemService.getById(input.itemId) + const item = knowledgeItemService.getById(itemId) if (item.status === 'deleting') return - knowledgeItemService.updateStatus(input.itemId, 'failed', { error: reason }) + knowledgeItemService.updateStatus(itemId, 'failed', { error: reason }) } catch (error) { if (isDataApiNotFoundError(error)) return logger.error(logMessage, error instanceof Error ? error : new Error(String(error)), { jobId: event.jobId, - itemId: input.itemId + itemId }) } } diff --git a/src/main/features/knowledge/types.ts b/src/main/features/knowledge/types.ts index 7f431ab289a..045c73cd298 100644 --- a/src/main/features/knowledge/types.ts +++ b/src/main/features/knowledge/types.ts @@ -1,6 +1,6 @@ import type { JobType } from '@main/core/job/jobRegistry' -export type KnowledgeWorkflowJobType = Extract +type KnowledgeWorkflowJobType = Extract export const KNOWLEDGE_JOB_TYPES = [ 'knowledge.prepare-root', 'knowledge.index-documents', @@ -45,7 +45,6 @@ export type KnowledgeProgressDetail = stage: 'failed' } -export const KNOWLEDGE_ACTIVE_JOB_STATUSES = ['pending', 'delayed', 'running'] as const export const KNOWLEDGE_ACTIVE_JOB_LIMIT = 5000 export function toKnowledgeBaseId(baseId: string): KnowledgeBaseId { diff --git a/src/main/features/knowledge/types/items.ts b/src/main/features/knowledge/types/items.ts deleted file mode 100644 index e6c57392b11..00000000000 --- a/src/main/features/knowledge/types/items.ts +++ /dev/null @@ -1,6 +0,0 @@ -import type { KnowledgeItemOf } from '@shared/data/types/knowledge' - -export type IndexableKnowledgeItem = KnowledgeItemOf<'file' | 'url' | 'note'> - -export type ContainerKnowledgeItem = KnowledgeItemOf<'directory'> -export type ContainerKnowledgeItemType = ContainerKnowledgeItem['type'] diff --git a/src/main/features/knowledge/utils/cleanup/subtreePurge.ts b/src/main/features/knowledge/utils/cleanup/subtreePurge.ts deleted file mode 100644 index 665c62d2983..00000000000 --- a/src/main/features/knowledge/utils/cleanup/subtreePurge.ts +++ /dev/null @@ -1,109 +0,0 @@ -import { application } from '@application' -import { knowledgeItemService } from '@data/services/KnowledgeItemService' -import type { KnowledgeBase, KnowledgeItem } from '@shared/data/types/knowledge' - -import { cancelJobOrThrow } from '../../tasks/utils/cancel' -import { narrowKnowledgeJobInput } from '../../tasks/utils/jobInput' -import { - KNOWLEDGE_ACTIVE_JOB_LIMIT, - KNOWLEDGE_ACTIVE_JOB_STATUSES, - knowledgeQueueName, - toKnowledgeBaseId -} from '../../types' -import { isIndexableKnowledgeItem } from '../items' -import { deleteKnowledgeItemFilesBestEffort } from '../storage/pathStorage' -import { deleteKnowledgeItemVectors } from './vectorCleanup' - -/** - * Cancel any in-flight job touching the given subtree (the roots plus their - * descendants). MUST run OUTSIDE the base mutation lock: `cancel` awaits each - * running handler's settlement, and knowledge index/prepare/reindex handlers - * themselves acquire `withBaseMutationLock`, so cancelling while holding the lock - * would deadlock (the handler can never reach its abort check). This is why both - * the delete job and the replace-on-add path cancel first, then purge under the - * lock. - */ -export async function cancelActiveKnowledgeSubtreeJobs( - baseId: string, - rootItemIds: string[], - reason: string, - currentJobId?: string -): Promise { - const subtreeItems = knowledgeItemService.getSubtreeItems(baseId, rootItemIds, { includeRoots: true }) - const subtreeIds = new Set(subtreeItems.map((item) => item.id)) - if (subtreeIds.size === 0) { - return - } - - const jobManager = application.get('JobManager') - const activeJobs = await jobManager.list({ - queue: knowledgeQueueName(toKnowledgeBaseId(baseId)), - status: [...KNOWLEDGE_ACTIVE_JOB_STATUSES], - limit: KNOWLEDGE_ACTIVE_JOB_LIMIT - }) - const jobIds = activeJobs - .filter((job) => job.id !== currentJobId && jobTouchesSubtree(job, subtreeIds)) - .map((job) => job.id) - const fileProcessingJobIds = activeJobs.flatMap((job) => getFileProcessingJobIdsForTouchedSubtree(job, subtreeIds)) - - await Promise.all([ - ...jobIds.map((jobId) => cancelJobOrThrow(jobId, reason)), - ...fileProcessingJobIds.map((jobId) => cancelJobOrThrow(jobId, reason)) - ]) -} - -/** - * Remove a resolved subtree's vectors, on-disk files, and DB rows, in that order. - * MUST run INSIDE the base mutation lock so no indexer can write vectors for the - * rows being removed, and so a caller (e.g. replace-on-add) can purge and then - * recreate within one lock acquisition — keeping the freed name available to the - * incoming source. Callers resolve and filter `subtreeItems` themselves (the - * delete job keeps only `deleting` rows; replace passes the conflicting roots' - * subtrees), then run vector cleanup before DB deletion so a retry can still - * discover affected ids. - */ -export async function purgeKnowledgeSubtreeWithinLock( - base: KnowledgeBase, - subtreeItems: KnowledgeItem[], - logContext: Record -): Promise { - const subtreeItemIds = subtreeItems.map((item) => item.id) - if (subtreeItemIds.length === 0) { - return - } - const leafItemIds = subtreeItems.filter((item) => isIndexableKnowledgeItem(item)).map((item) => item.id) - - // Vector cleanup precedes DB deletion so a retry can still discover affected item ids. - await deleteKnowledgeItemVectors(base, leafItemIds) - // Best-effort: a file-removal failure must not abort the row deletion below, - // which would otherwise strand rows after their vectors are gone. - await deleteKnowledgeItemFilesBestEffort(base.id, subtreeItems, logContext) - - knowledgeItemService.deleteItemsByIds(base.id, subtreeItemIds) -} - -function jobTouchesSubtree(job: { type: string; input: unknown }, subtreeIds: Set): boolean { - const narrowed = narrowKnowledgeJobInput(job) - if (!narrowed) { - return false - } - if ('itemId' in narrowed.input && subtreeIds.has(narrowed.input.itemId)) { - return true - } - return 'rootItemIds' in narrowed.input && narrowed.input.rootItemIds.some((itemId) => subtreeIds.has(itemId)) -} - -function getFileProcessingJobIdsForTouchedSubtree( - job: { type: string; input: unknown }, - subtreeIds: Set -): string[] { - const narrowed = narrowKnowledgeJobInput(job) - if ( - narrowed?.type === 'knowledge.check-file-processing-result' && - subtreeIds.has(narrowed.input.itemId) && - narrowed.input.fileProcessingJobId - ) { - return [narrowed.input.fileProcessingJobId] - } - return [] -} diff --git a/src/main/features/knowledge/utils/indexing/__tests__/materialFields.test.ts b/src/main/features/knowledge/utils/indexing/__tests__/materialFields.test.ts deleted file mode 100644 index 31208bbd362..00000000000 --- a/src/main/features/knowledge/utils/indexing/__tests__/materialFields.test.ts +++ /dev/null @@ -1,59 +0,0 @@ -import { describe, expect, it } from 'vitest' - -import { type MaterialFieldSource, toMaterialRelativePath } from '../materialFields' - -describe('toMaterialRelativePath', () => { - it('uses a file’s stored relativePath when there is no processed artifact', () => { - const file: MaterialFieldSource = { - id: 'file-1', - type: 'file', - data: { source: '/docs/report.pdf', relativePath: 'report.pdf' } - } - expect(toMaterialRelativePath(file)).toBe('report.pdf') - }) - - it('prefers a file’s processed-artifact path (indexedRelativePath) over the source path', () => { - const file: MaterialFieldSource = { - id: 'file-2', - type: 'file', - data: { source: '/docs/report.pdf', relativePath: 'report.pdf', indexedRelativePath: 'report.md' } - } - expect(toMaterialRelativePath(file)).toBe('report.md') - }) - - it('uses a url’s captured snapshot path once it has one (the real raw/ file, matching the migrator)', () => { - const url: MaterialFieldSource = { - id: 'url-1', - type: 'url', - data: { source: 'https://example.com', url: 'https://example.com', relativePath: 'example-page.md' } - } - expect(toMaterialRelativePath(url)).toBe('example-page.md') - }) - - it('uses a note’s captured snapshot path once it has one (the real raw/ file, matching the migrator)', () => { - const note: MaterialFieldSource = { - id: 'note-1', - type: 'note', - data: { source: 'My note', content: 'hello', relativePath: 'My note.md' } - } - expect(toMaterialRelativePath(note)).toBe('My note.md') - }) - - it('throws for a url that has not been captured yet — a snapshot is always materialized first', () => { - const url: MaterialFieldSource = { - id: 'url-2', - type: 'url', - data: { source: 'https://example.com', url: 'https://example.com' } - } - expect(() => toMaterialRelativePath(url)).toThrow('has no captured snapshot relativePath') - }) - - it('throws for a note that has not been captured yet — a snapshot is always materialized first', () => { - const note: MaterialFieldSource = { - id: 'note-2', - type: 'note', - data: { source: 'My note', content: 'hello' } - } - expect(() => toMaterialRelativePath(note)).toThrow('has no captured snapshot relativePath') - }) -}) diff --git a/src/main/features/knowledge/utils/indexing/materialFields.ts b/src/main/features/knowledge/utils/indexing/materialFields.ts deleted file mode 100644 index eddbfe65052..00000000000 --- a/src/main/features/knowledge/utils/indexing/materialFields.ts +++ /dev/null @@ -1,31 +0,0 @@ -import type { KnowledgeItemOf } from '@shared/data/types/knowledge' - -/** - * The subset of an indexable knowledge item needed to derive its index-store - * material fields. The `Pick` is distributed per member so `type` and `data` - * stay correlated (a single `Pick` over the union would collapse `data` to a bare - * union and lose the file-only `relativePath` / `indexedRelativePath`). Shared by - * the indexing job and the v1→v2 vector migrator so both stamp the material - * identically (knowledge-technical-design.md §4.2). - */ -export type MaterialFieldSource = - | Pick, 'id' | 'type' | 'data'> - | Pick, 'id' | 'type' | 'data'> - | Pick, 'id' | 'type' | 'data'> - -/** - * A material's stable relative path. A file uses its stored path (the processed - * artifact when present). A url or note uses its captured snapshot path — a real - * base file under `raw/`, materialized before the material is stamped (the index - * job's ensure-snapshot step, or the vector migrator), so it is always present - * here; a missing one is an invariant violation, not a fallback case. - */ -export function toMaterialRelativePath(item: MaterialFieldSource): string { - if (item.type === 'file') { - return item.data.indexedRelativePath ?? item.data.relativePath - } - if (!item.data.relativePath) { - throw new Error(`Knowledge ${item.type} item ${item.id} has no captured snapshot relativePath for its material`) - } - return item.data.relativePath -} diff --git a/src/main/features/knowledge/vectorstore/__tests__/KnowledgeVectorStoreService.test.ts b/src/main/features/knowledge/vectorstore/__tests__/KnowledgeVectorStoreService.test.ts deleted file mode 100644 index a2eeaf81861..00000000000 --- a/src/main/features/knowledge/vectorstore/__tests__/KnowledgeVectorStoreService.test.ts +++ /dev/null @@ -1,507 +0,0 @@ -import type * as LifecycleModule from '@main/core/lifecycle' -import { - DEFAULT_KNOWLEDGE_BASE_CHUNK_OVERLAP, - DEFAULT_KNOWLEDGE_BASE_CHUNK_SIZE, - type KnowledgeBase -} from '@shared/data/types/knowledge' -import { beforeEach, describe, expect, it, vi } from 'vitest' - -const { - loggerDebugMock, - loggerErrorMock, - loggerInfoMock, - loggerWarnMock, - openDriverMock, - createSchemaMock, - resetSchemaMock, - ensureIndexMetaMock, - readIndexSchemaVersionMock, - hasAnyMaterialMock, - getItemsByBaseIdMock, - indexStoreCtorMock, - getPathSyncMock, - deleteDirMock, - statMock -} = vi.hoisted(() => ({ - loggerDebugMock: vi.fn(), - loggerErrorMock: vi.fn(), - loggerInfoMock: vi.fn(), - loggerWarnMock: vi.fn(), - openDriverMock: vi.fn(), - createSchemaMock: vi.fn(), - resetSchemaMock: vi.fn(), - ensureIndexMetaMock: vi.fn(), - readIndexSchemaVersionMock: vi.fn(), - hasAnyMaterialMock: vi.fn(), - getItemsByBaseIdMock: vi.fn(), - indexStoreCtorMock: vi.fn(), - getPathSyncMock: vi.fn(), - deleteDirMock: vi.fn(), - statMock: vi.fn() -})) - -/** Mirrors KNOWLEDGE_INDEX_SCHEMA_VERSION in schema.ts; the service test pins the open-time - * branching (rebuild vs create), while schema.test.ts pins the real constant's value. */ -const MOCK_SCHEMA_VERSION = 2 - -vi.mock('node:fs', () => ({ - default: { promises: { stat: statMock } } -})) - -vi.mock('@main/core/lifecycle', async (importOriginal) => { - const actual = await importOriginal() - - class MockBaseService {} - - return { - ...actual, - BaseService: MockBaseService - } -}) - -vi.mock('@logger', () => ({ - loggerService: { - withContext: () => ({ - debug: loggerDebugMock, - info: loggerInfoMock, - error: loggerErrorMock, - warn: loggerWarnMock - }) - } -})) - -vi.mock('../indexStore/KnowledgeIndexStore', () => ({ - KnowledgeIndexStore: indexStoreCtorMock -})) - -vi.mock('../indexStore/BetterSqlite3Driver', () => ({ - openBetterSqlite3IndexDriver: openDriverMock -})) - -vi.mock('../indexStore/BetterSqlite3VectorIndex', () => ({ - betterSqlite3VectorIndex: { kind: 'better-sqlite3' } -})) - -vi.mock('../indexStore/schema', () => ({ - createKnowledgeIndexSchema: createSchemaMock, - resetKnowledgeIndexSchema: resetSchemaMock, - KNOWLEDGE_INDEX_SCHEMA_VERSION: MOCK_SCHEMA_VERSION -})) - -vi.mock('../indexStore/indexMeta', () => ({ - ensureIndexMeta: ensureIndexMetaMock, - readIndexSchemaVersion: readIndexSchemaVersionMock, - hasAnyMaterial: hasAnyMaterialMock -})) - -vi.mock('@data/services/KnowledgeItemService', () => ({ - knowledgeItemService: { getItemsByBaseId: getItemsByBaseIdMock } -})) - -vi.mock('../../utils/storage/pathStorage', () => ({ - getKnowledgeVectorStoreFilePathSync: getPathSyncMock, - deleteKnowledgeBaseDir: deleteDirMock -})) - -const { KnowledgeVectorStoreService } = await import('../KnowledgeVectorStoreService') - -function createBase(id = 'kb-1'): KnowledgeBase { - return { - id, - name: 'KB', - groupId: null, - dimensions: 1024, - embeddingModelId: 'ollama::nomic-embed-text', - status: 'completed', - error: null, - chunkSize: DEFAULT_KNOWLEDGE_BASE_CHUNK_SIZE, - chunkOverlap: DEFAULT_KNOWLEDGE_BASE_CHUNK_OVERLAP, - chunkStrategy: 'structured', - chunkSeparator: '\\n\\n', - searchMode: 'hybrid', - createdAt: '2026-04-08T00:00:00.000Z', - updatedAt: '2026-04-08T00:00:00.000Z' - } -} - -/** The store instance built by the most recent `new KnowledgeIndexStore(...)` call. */ -function lastStore() { - const results = indexStoreCtorMock.mock.results - return results[results.length - 1]?.value as { close: ReturnType } -} - -describe('KnowledgeVectorStoreService', () => { - beforeEach(() => { - vi.clearAllMocks() - getPathSyncMock.mockImplementation((baseId: string) => `/tmp/${baseId}/index.sqlite`) - // Each open returns a fresh closeable driver so failure paths can assert close(). - // The real driver port is synchronous (see BetterSqlite3Driver) — these mocks - // must return plain values, not promises, or the store under test would try to - // call .execute() on a Promise object. - openDriverMock.mockImplementation(() => ({ - kind: 'driver', - close: vi.fn() - })) - createSchemaMock.mockReturnValue(undefined) - resetSchemaMock.mockReturnValue(undefined) - ensureIndexMetaMock.mockReturnValue(undefined) - // Default: a fresh/blank file has no stored version → the open path takes the normal - // create branch (no rebuild). Mismatch tests override this per-case. - readIndexSchemaVersionMock.mockReturnValue(null) - // A non-empty material probe keeps the invisible-contents diagnostic quiet - // unless a test opts in. - hasAnyMaterialMock.mockReturnValue(true) - getItemsByBaseIdMock.mockReturnValue([]) - deleteDirMock.mockResolvedValue(undefined) - indexStoreCtorMock.mockImplementation(() => ({ close: vi.fn().mockResolvedValue(undefined) })) - }) - - it('opens an index store on first request and caches it per base', async () => { - const service = new KnowledgeVectorStoreService() - const base = createBase() - - const first = await service.getIndexStore(base) - const second = await service.getIndexStore(base) - - expect(first).toBe(second) - expect(indexStoreCtorMock).toHaveBeenCalledTimes(1) - expect(openDriverMock).toHaveBeenCalledTimes(1) - expect(createSchemaMock).toHaveBeenCalledTimes(1) - expect(loggerInfoMock).toHaveBeenCalledWith('Opened knowledge index store', { baseId: base.id, cacheSize: 1 }) - expect(loggerDebugMock).toHaveBeenCalledWith('Reusing cached knowledge index store', { baseId: base.id }) - }) - - it('shares a single open across concurrent callers for the same base (single-flight)', async () => { - const service = new KnowledgeVectorStoreService() - const base = createBase() - - // Both calls are issued before the first open resolves; the second must join - // the first's in-flight open rather than starting its own (which would leak a - // store no one closes). - const [first, second] = await Promise.all([service.getIndexStore(base), service.getIndexStore(base)]) - - expect(first).toBe(second) - expect(openDriverMock).toHaveBeenCalledTimes(1) - expect(indexStoreCtorMock).toHaveBeenCalledTimes(1) - }) - - it('evicts a failed open so a later call retries instead of re-awaiting the failure', async () => { - const service = new KnowledgeVectorStoreService() - const base = createBase() - openDriverMock.mockImplementationOnce(() => { - throw new Error('open failed') - }) - - await expect(service.getIndexStore(base)).rejects.toThrow('open failed') - - const store = await service.getIndexStore(base) - expect(store).toBe(lastStore()) - expect(openDriverMock).toHaveBeenCalledTimes(2) - expect(indexStoreCtorMock).toHaveBeenCalledTimes(1) - }) - - it('stamps and verifies the meta identity row before handing out the store', async () => { - const service = new KnowledgeVectorStoreService() - const base = createBase() - - await service.getIndexStore(base) - - expect(ensureIndexMetaMock).toHaveBeenCalledTimes(1) - expect(ensureIndexMetaMock).toHaveBeenCalledWith(expect.objectContaining({ kind: 'driver' }), { - baseId: base.id - }) - }) - - it('creates the schema normally when the stored version matches (no rebuild)', async () => { - const service = new KnowledgeVectorStoreService() - const base = createBase() - readIndexSchemaVersionMock.mockReturnValueOnce(MOCK_SCHEMA_VERSION) - - await service.getIndexStore(base) - - expect(createSchemaMock).toHaveBeenCalledTimes(1) - expect(resetSchemaMock).not.toHaveBeenCalled() - }) - - it('rebuilds the derived index when an existing index.sqlite is at a stale schema version', async () => { - const service = new KnowledgeVectorStoreService() - const base = createBase() - readIndexSchemaVersionMock.mockReturnValueOnce(MOCK_SCHEMA_VERSION - 1) - - await service.getIndexStore(base) - - // Drop+recreate via resetKnowledgeIndexSchema, NOT the plain create — an old layout - // (e.g. pre-fts_rowid) cannot be retrofitted by CREATE ... IF NOT EXISTS. - expect(resetSchemaMock).toHaveBeenCalledTimes(1) - expect(createSchemaMock).not.toHaveBeenCalled() - // Warn so the wipe-and-reindex is diagnosable. - expect(loggerWarnMock).toHaveBeenCalledWith( - 'Knowledge index schema version mismatch — rebuilding the derived index', - expect.objectContaining({ - baseId: base.id, - storedVersion: MOCK_SCHEMA_VERSION - 1, - expectedVersion: MOCK_SCHEMA_VERSION - }) - ) - // The reset drops meta, so the open path still restamps it afterwards. - expect(ensureIndexMetaMock).toHaveBeenCalledTimes(1) - }) - - it('also rebuilds when the stored version is NEWER than this build (downgrade)', async () => { - // The guard is `!==`, not `<` — a file written by a newer build (user downgraded the app) is - // just as incompatible a layout and must be rebuilt, not opened as-is. Pins that semantics so a - // future refactor to `<` does not silently start mounting newer files. - const service = new KnowledgeVectorStoreService() - const base = createBase() - readIndexSchemaVersionMock.mockReturnValueOnce(MOCK_SCHEMA_VERSION + 1) - - await service.getIndexStore(base) - - expect(resetSchemaMock).toHaveBeenCalledTimes(1) - expect(createSchemaMock).not.toHaveBeenCalled() - }) - - it('closes the driver and aborts the open when meta verification fails (wrong/corrupt base)', async () => { - const service = new KnowledgeVectorStoreService() - const base = createBase() - let openedDriver: { close: ReturnType } | undefined - openDriverMock.mockImplementationOnce(() => { - openedDriver = { kind: 'driver', close: vi.fn().mockResolvedValue(undefined) } as never - return openedDriver - }) - ensureIndexMetaMock.mockImplementationOnce(() => { - throw new Error('belongs to a different base') - }) - - await expect(service.getIndexStore(base)).rejects.toThrow('belongs to a different base') - - expect(openedDriver?.close).toHaveBeenCalledTimes(1) - expect(indexStoreCtorMock).not.toHaveBeenCalled() - }) - - it('closes the driver when schema creation fails so the file handle is not leaked', async () => { - const service = new KnowledgeVectorStoreService() - const base = createBase() - let openedDriver: { close: ReturnType } | undefined - openDriverMock.mockImplementationOnce(() => { - openedDriver = { kind: 'driver', close: vi.fn().mockResolvedValue(undefined) } as never - return openedDriver - }) - createSchemaMock.mockImplementationOnce(() => { - throw new Error('disk full') - }) - - await expect(service.getIndexStore(base)).rejects.toThrow('disk full') - - expect(openedDriver?.close).toHaveBeenCalledTimes(1) - expect(indexStoreCtorMock).not.toHaveBeenCalled() - }) - - it('returns undefined from getIndexStoreIfExists when no backing file exists', async () => { - const service = new KnowledgeVectorStoreService() - const base = createBase() - statMock.mockRejectedValueOnce(Object.assign(new Error('missing'), { code: 'ENOENT' })) - - await expect(service.getIndexStoreIfExists(base)).resolves.toBeUndefined() - - expect(indexStoreCtorMock).not.toHaveBeenCalled() - expect(loggerDebugMock).toHaveBeenCalledWith('Knowledge index store does not exist on disk', { baseId: base.id }) - }) - - it('opens an existing store from disk when getIndexStoreIfExists detects a backing file', async () => { - const service = new KnowledgeVectorStoreService() - const base = createBase() - statMock.mockResolvedValueOnce({ isFile: () => true }) - - const store = await service.getIndexStoreIfExists(base) - - expect(store).toBe(lastStore()) - expect(indexStoreCtorMock).toHaveBeenCalledTimes(1) - }) - - it('returns the cached store from getIndexStoreIfExists without probing disk', async () => { - const service = new KnowledgeVectorStoreService() - const base = createBase() - - const created = await service.getIndexStore(base) - - await expect(service.getIndexStoreIfExists(base)).resolves.toBe(created) - expect(statMock).not.toHaveBeenCalled() - }) - - it('closes the cached store and removes the base directory on deleteStore', async () => { - const service = new KnowledgeVectorStoreService() - const base = createBase() - - const store = await service.getIndexStore(base) - await service.deleteStore(base.id) - - expect(store.close).toHaveBeenCalledTimes(1) - expect(deleteDirMock).toHaveBeenCalledWith(base.id) - // Close must precede directory removal — on Windows a still-open sqlite - // handle makes the directory deletion fail. - expect((store.close as unknown as ReturnType).mock.invocationCallOrder[0]).toBeLessThan( - deleteDirMock.mock.invocationCallOrder[0] - ) - - // Cache was evicted: the next open builds a fresh instance. - const reopened = await service.getIndexStore(base) - expect(reopened).not.toBe(store) - expect(indexStoreCtorMock).toHaveBeenCalledTimes(2) - }) - - it('deleteStore removes the directory even when no store was ever opened for the base', async () => { - // Opening a store (see openIndexStore) is fully synchronous — it either completes and - // caches a store, or throws before caching anything. There is no longer an in-flight - // state deleteStore could observe mid-open, so this covers the remaining "nothing - // cached" case: deleteStore must still close-if-present (a no-op here) and remove - // the directory. - const service = new KnowledgeVectorStoreService() - const base = createBase() - - await service.deleteStore(base.id) - - expect(deleteDirMock).toHaveBeenCalledWith(base.id) - expect(indexStoreCtorMock).not.toHaveBeenCalled() - }) - - it('evicts the cached store even when directory removal fails', async () => { - const service = new KnowledgeVectorStoreService() - const base = createBase() - deleteDirMock.mockRejectedValueOnce(new Error('delete failed')) - - const store = await service.getIndexStore(base) - await expect(service.deleteStore(base.id)).rejects.toThrow('delete failed') - - const reopened = await service.getIndexStore(base) - expect(reopened).not.toBe(store) - expect(indexStoreCtorMock).toHaveBeenCalledTimes(2) - }) - - it('closes all cached stores during stop and continues when one close throws', async () => { - const service = new KnowledgeVectorStoreService() - - const first = await service.getIndexStore(createBase('kb-1')) - const second = await service.getIndexStore(createBase('kb-2')) - const closeError = new Error('close failed') - ;(first.close as unknown as ReturnType).mockRejectedValueOnce(closeError) - - await expect((service as any).onStop()).resolves.toBeUndefined() - - expect(first.close).toHaveBeenCalledTimes(1) - expect(second.close).toHaveBeenCalledTimes(1) - expect(loggerErrorMock).toHaveBeenCalledWith('Failed to close knowledge index store', closeError, { - baseId: 'kb-1' - }) - expect(loggerInfoMock).toHaveBeenCalledWith('Stopping knowledge index stores', { storeCount: 2 }) - expect(loggerInfoMock).toHaveBeenCalledWith('Stopped knowledge index stores', { storeCount: 2 }) - - // Cache cleared: reopening kb-2 builds a fresh instance. - const reopened = await service.getIndexStore(createBase('kb-2')) - expect(reopened).not.toBe(second) - }) - - it('rejects bases that are not ready before touching disk', async () => { - const service = new KnowledgeVectorStoreService() - const base = { - ...createBase(), - dimensions: null, - embeddingModelId: null, - status: 'failed', - error: 'missing_embedding_model' - } satisfies KnowledgeBase - - await expect(service.getIndexStore(base)).rejects.toThrow('not ready for vector store operations') - - expect(indexStoreCtorMock).not.toHaveBeenCalled() - }) - - it('lets cleanup on a failed base proceed: getIndexStoreIfExists returns undefined instead of asserting', async () => { - const service = new KnowledgeVectorStoreService() - const base = { - ...createBase(), - dimensions: null, - embeddingModelId: null, - status: 'failed', - error: 'missing_embedding_model' - } satisfies KnowledgeBase - // Failed bases never get a store file (the vector migrator skips them and - // getIndexStore asserts), so the existence probe is the path cleanup takes. - statMock.mockRejectedValueOnce(Object.assign(new Error('missing'), { code: 'ENOENT' })) - - await expect(service.getIndexStoreIfExists(base)).resolves.toBeUndefined() - - expect(indexStoreCtorMock).not.toHaveBeenCalled() - }) - - it('still asserts readiness when a failed base unexpectedly has a store file on disk', async () => { - const service = new KnowledgeVectorStoreService() - const base = { - ...createBase(), - dimensions: null, - embeddingModelId: null, - status: 'failed', - error: 'missing_embedding_model' - } satisfies KnowledgeBase - statMock.mockResolvedValueOnce({ isFile: () => true }) - - await expect(service.getIndexStoreIfExists(base)).rejects.toThrow('not ready for vector store operations') - - expect(indexStoreCtorMock).not.toHaveBeenCalled() - }) - - it('logs an error when an empty index mounts under a base with completed items', async () => { - const service = new KnowledgeVectorStoreService() - const base = createBase() - hasAnyMaterialMock.mockReturnValueOnce(false) - getItemsByBaseIdMock.mockReturnValueOnce([ - { id: 'item-1', type: 'directory', status: 'completed' }, - { id: 'item-2', type: 'file', status: 'completed' } - ]) - - const store = await service.getIndexStore(base) - - expect(store).toBe(lastStore()) - expect(loggerErrorMock).toHaveBeenCalledWith( - expect.stringContaining('zero materials while the base has completed items'), - expect.objectContaining({ baseId: base.id }) - ) - }) - - it('stays quiet when an empty index mounts under a base with no completed indexable items', async () => { - const service = new KnowledgeVectorStoreService() - const base = createBase() - hasAnyMaterialMock.mockReturnValueOnce(false) - // A completed empty directory is legitimate without materials; in-flight leaves are too. - getItemsByBaseIdMock.mockReturnValueOnce([ - { id: 'item-1', type: 'directory', status: 'completed' }, - { id: 'item-2', type: 'file', status: 'processing' } - ]) - - await service.getIndexStore(base) - - expect(loggerErrorMock).not.toHaveBeenCalled() - }) - - it('fails the open and closes the driver when the empty-index diagnostic cannot read the base items', async () => { - const service = new KnowledgeVectorStoreService() - const base = createBase() - let openedDriver: { close: ReturnType } | undefined - openDriverMock.mockImplementationOnce(() => { - openedDriver = { kind: 'driver', close: vi.fn().mockResolvedValue(undefined) } as never - return openedDriver - }) - hasAnyMaterialMock.mockReturnValueOnce(false) - getItemsByBaseIdMock.mockImplementationOnce(() => { - throw new Error('app database unavailable') - }) - - // Deliberate fail-loud: swallowing the lookup failure would re-silence the - // deleted-base race (open racing deleteBase recreates an empty file, and the - // lookup's NOT_FOUND is what makes that loud instead of caching an empty store). - await expect(service.getIndexStore(base)).rejects.toThrow('app database unavailable') - - expect(openedDriver?.close).toHaveBeenCalledTimes(1) - expect(indexStoreCtorMock).not.toHaveBeenCalled() - }) -}) diff --git a/src/renderer/i18n/locales/en-us.json b/src/renderer/i18n/locales/en-us.json index 735d137482c..d8a1bd8f57a 100644 --- a/src/renderer/i18n/locales/en-us.json +++ b/src/renderer/i18n/locales/en-us.json @@ -2743,7 +2743,8 @@ "indexing_interrupted": "Indexing was interrupted because the app closed. Reindex this item to finish.", "missing_embedding_model": "The embedding model used by this knowledge base was not found during migration. Rebuild the knowledge base and choose a new embedding model.", "missing_vector_store": "This knowledge base's vector store could not be read during migration (missing, empty, or locked). The knowledge base was kept; re-index it to recover.", - "model_invalid": "No model selected" + "model_invalid": "No model selected", + "never_indexed": "This item never finished indexing during migration. Reindex it to complete." }, "groups": { "add": "New Group", diff --git a/src/renderer/i18n/locales/zh-cn.json b/src/renderer/i18n/locales/zh-cn.json index 1175bc96d61..9819126ea96 100644 --- a/src/renderer/i18n/locales/zh-cn.json +++ b/src/renderer/i18n/locales/zh-cn.json @@ -2743,7 +2743,8 @@ "indexing_interrupted": "索引因应用关闭而中断，请重新索引以完成。", "missing_embedding_model": "迁移时未找到原知识库使用的嵌入模型，请重建知识库并选择新的嵌入模型。", "missing_vector_store": "迁移时未能读取该知识库的向量库（缺失、为空或被占用），知识库已保留，请重新索引以恢复。", - "model_invalid": "未选择模型" + "model_invalid": "未选择模型", + "never_indexed": "该项在迁移时从未开始索引，请重新索引以完成。" }, "groups": { "add": "新建分组", diff --git a/src/renderer/i18n/locales/zh-tw.json b/src/renderer/i18n/locales/zh-tw.json index dce9ffc6cb5..9575156a598 100644 --- a/src/renderer/i18n/locales/zh-tw.json +++ b/src/renderer/i18n/locales/zh-tw.json @@ -2743,7 +2743,8 @@ "indexing_interrupted": "索引因應用程式關閉而中斷，請重新索引以完成。", "missing_embedding_model": "遷移時未找到原知識庫使用的嵌入模型，請重建知識庫並選擇新的嵌入模型。", "missing_vector_store": "遷移時未能讀取該知識庫的向量庫（缺失、為空或被占用），知識庫已保留，請重新索引以恢復。", - "model_invalid": "未選擇模型" + "model_invalid": "未選擇模型", + "never_indexed": "該項在遷移時從未開始索引，請重新索引以完成。" }, "groups": { "add": "新建分组", diff --git a/src/renderer/pages/knowledge/panels/dataSource/__tests__/KnowledgeItemRow.test.tsx b/src/renderer/pages/knowledge/panels/dataSource/__tests__/KnowledgeItemRow.test.tsx index ad4565d209b..aaba64ddc21 100644 --- a/src/renderer/pages/knowledge/panels/dataSource/__tests__/KnowledgeItemRow.test.tsx +++ b/src/renderer/pages/knowledge/panels/dataSource/__tests__/KnowledgeItemRow.test.tsx @@ -449,7 +449,7 @@ describe('KnowledgeItemRow', () => { expect(handleClick).not.toHaveBeenCalled() }) - it.each(['idle', 'processing', 'reading', 'embedding', 'failed', 'deleting'] as const)( + it.each(['processing', 'reading', 'embedding', 'failed', 'deleting'] as const)( 'hides view chunks for %s leaf items', (status) => { render() @@ -551,15 +551,12 @@ describe('KnowledgeItemRow', () => { expect(screen.getByRole('button', { name: '重新索引' })).toBeInTheDocument() }) - it.each(['idle', 'processing', 'reading', 'embedding', 'deleting'] as const)( - 'hides reindex for %s leaf items', - (status) => { - render() + it.each(['processing', 'reading', 'embedding', 'deleting'] as const)('hides reindex for %s leaf items', (status) => { + render() - fireEvent.contextMenu(screen.getByRole('row')) + fireEvent.contextMenu(screen.getByRole('row')) - expect(screen.queryByRole('button', { name: '重新索引' })).not.toBeInTheDocument() - expect(screen.getByRole('button', { name: '删除' })).toBeInTheDocument() - } - ) + expect(screen.queryByRole('button', { name: '重新索引' })).not.toBeInTheDocument() + expect(screen.getByRole('button', { name: '删除' })).toBeInTheDocument() + }) }) diff --git a/src/renderer/pages/knowledge/panels/dataSource/utils/models.ts b/src/renderer/pages/knowledge/panels/dataSource/utils/models.ts index db46ea6235f..262abcac4a6 100644 --- a/src/renderer/pages/knowledge/panels/dataSource/utils/models.ts +++ b/src/renderer/pages/knowledge/panels/dataSource/utils/models.ts @@ -105,7 +105,7 @@ export const resolveDataSourceStatusViewModel = (status: KnowledgeItemStatus): D } } - if (status === 'idle' || status === 'preparing') { + if (status === 'preparing') { return { kind: 'processing', labelKey: 'knowledge.data_source.status.pending', diff --git a/src/renderer/pages/knowledge/utils/error.ts b/src/renderer/pages/knowledge/utils/error.ts index 9184166d30d..7a3759f461a 100644 --- a/src/renderer/pages/knowledge/utils/error.ts +++ b/src/renderer/pages/knowledge/utils/error.ts @@ -14,6 +14,7 @@ type KnowledgeErrorTranslator = ( | 'knowledge.error.missing_vector_store' | 'knowledge.error.directory_not_migrated' | 'knowledge.error.indexing_interrupted' + | 'knowledge.error.never_indexed' ) => string /** Localized copy for a known base error code. Exhaustive over `KnowledgeBaseErrorCode`. */ @@ -50,6 +51,8 @@ const translateKnowledgeItemErrorCode = (code: KnowledgeItemErrorCode, t: Knowle return t('knowledge.error.directory_not_migrated') case 'indexing_interrupted': return t('knowledge.error.indexing_interrupted') + case 'never_indexed': + return t('knowledge.error.never_indexed') default: return code satisfies never } diff --git a/src/shared/data/api/apiErrors.ts b/src/shared/data/api/apiErrors.ts index 9d437102682..d3fe8d605e1 100644 --- a/src/shared/data/api/apiErrors.ts +++ b/src/shared/data/api/apiErrors.ts @@ -806,6 +806,15 @@ export function isDataApiError(error: unknown): error is DataApiError { return error instanceof DataApiError } +/** + * Check if an error is a DataApiError carrying the NOT_FOUND code. + * @param error - Any thrown value + * @returns true if the error is a NOT_FOUND DataApiError + */ +export function isDataApiNotFoundError(error: unknown): boolean { + return isDataApiError(error) && error.code === ErrorCode.NOT_FOUND +} + /** * Check if an object is a serialized DataApiError. * @param error - Any object diff --git a/src/shared/data/api/index.ts b/src/shared/data/api/index.ts index a90c4e2c518..42664ab3b7d 100644 --- a/src/shared/data/api/index.ts +++ b/src/shared/data/api/index.ts @@ -80,6 +80,7 @@ export { DataApiError, DataApiErrorFactory, isDataApiError, + isDataApiNotFoundError, isSerializedDataApiError, toDataApiError } from './apiErrors' diff --git a/src/shared/data/api/schemas/__tests__/knowledges.test.ts b/src/shared/data/api/schemas/__tests__/knowledges.test.ts index cef118ebc5d..0514b0dd009 100644 --- a/src/shared/data/api/schemas/__tests__/knowledges.test.ts +++ b/src/shared/data/api/schemas/__tests__/knowledges.test.ts @@ -435,7 +435,7 @@ describe('Knowledge base schemas', () => { groupId: null, type: 'note', data: { source: 'hello', content: 'hello' }, - status: 'idle', + status: 'processing', error: null, createdAt: '2026-04-10T00:00:00.000Z', updatedAt: '2026-04-10T00:00:00.000Z' @@ -449,7 +449,7 @@ describe('Knowledge base schemas', () => { groupId: null, type: 'note', data: { source: 'hello', content: 'hello' }, - status: 'idle', + status: 'processing', createdAt: '2026-04-10T00:00:00.000Z', updatedAt: '2026-04-10T00:00:00.000Z' }).success @@ -497,7 +497,6 @@ describe('Knowledge base schemas', () => { updatedAt: '2026-04-10T00:00:00.000Z' } - expect(KnowledgeItemSchema.safeParse({ ...validItem, status: 'idle', error: null }).success).toBe(true) expect(KnowledgeItemSchema.safeParse({ ...validItem, status: 'completed', error: null }).success).toBe(true) expect(KnowledgeItemSchema.safeParse({ ...validItem, status: 'processing', error: null }).success).toBe(true) expect(KnowledgeItemSchema.safeParse({ ...validItem, status: 'reading', error: null }).success).toBe(true) @@ -505,9 +504,10 @@ describe('Knowledge base schemas', () => { expect(KnowledgeItemSchema.safeParse({ ...validItem, status: 'failed', error: 'read failed' }).success).toBe(true) expect(KnowledgeItemSchema.safeParse({ ...validItem, status: 'deleting', error: null }).success).toBe(true) - expect(KnowledgeItemSchema.safeParse({ ...validItem, status: 'idle', phase: 'reading', error: null }).success).toBe( - false - ) + expect(KnowledgeItemSchema.safeParse({ ...validItem, status: 'idle', error: null }).success).toBe(false) + expect( + KnowledgeItemSchema.safeParse({ ...validItem, status: 'processing', phase: 'reading', error: null }).success + ).toBe(false) expect(KnowledgeItemSchema.safeParse({ ...validItem, status: 'completed', error: 'stale' }).success).toBe(false) expect(KnowledgeItemSchema.safeParse({ ...validItem, status: 'processing', error: 'stale' }).success).toBe(false) expect( diff --git a/src/shared/data/types/knowledge.ts b/src/shared/data/types/knowledge.ts index 9dab17bd6fd..599c406f280 100644 --- a/src/shared/data/types/knowledge.ts +++ b/src/shared/data/types/knowledge.ts @@ -26,19 +26,18 @@ export type KnowledgeItemType = z.infer * * ```text * file/url/note: - * idle -> processing -> reading -> embedding -> completed - * \ \ \ \ - * +--------------------+-------------+-----------> failed - * \---------------------------------------------> deleting + * processing -> reading -> embedding -> completed + * \ \ \ + * +-------------+-----------> failed + * \---------------------------------------------> deleting * * directory: - * idle -> preparing -> processing -> completed - * \ \ \ \ - * +--------+-------------+-----------> failed - * \---------------------------------> deleting + * preparing -> processing -> completed + * \ \ \ + * +-------------+-----------> failed + * \---------------------------------> deleting * ``` * - * - `idle`: item row exists but indexing has not started. * - `preparing`: container expansion is running; only `directory` items may use it. * - `processing`: work has been queued or is running before a more specific phase is known. * - `reading`: leaf source documents are being read; only `file` / `url` / `note` items may use it. @@ -50,7 +49,6 @@ export type KnowledgeItemType = z.infer * - `deleting`: delete cleanup is in progress; default list/search/RAG reads hide the item. */ export const KNOWLEDGE_ITEM_STATUSES = [ - 'idle', 'preparing', 'processing', 'reading', @@ -79,7 +77,7 @@ export const DEFAULT_KNOWLEDGE_BASE_STATUS: KnowledgeBaseStatus = 'completed' // user_model, so the base needs a new embedding model on restore. // `missing_vector_store`: the embedding model resolved, but the per-base legacy vector store // was missing/empty/locked so its dimensions could not be determined. The base (name, model, -// config, idle items) is kept as a restorable `failed` row instead of being dropped, so the +// config, unindexed items) is kept as a restorable `failed` row instead of being dropped, so the // user can re-index it — a transient lock is recoverable by re-running rather than a data loss. export const KNOWLEDGE_BASE_ERROR_CODES = ['missing_embedding_model', 'missing_vector_store'] as const export const KnowledgeBaseErrorCodeSchema = z.enum(KNOWLEDGE_BASE_ERROR_CODES) @@ -88,20 +86,23 @@ export const KNOWLEDGE_BASE_ERROR_MISSING_EMBEDDING_MODEL: KnowledgeBaseErrorCod export const KNOWLEDGE_BASE_ERROR_MISSING_VECTOR_STORE: KnowledgeBaseErrorCode = 'missing_vector_store' /** - * Item-level error codes stored on `knowledge_item.error`. Two are set today: + * Item-level error codes stored on `knowledge_item.error`. Three are set today: * - `directory_not_migrated`: a v1-indexed `directory` whose container-level vectors could not * be re-attributed to per-file children (unreadable legacy sources, or no migratable vectors). * - `indexing_interrupted`: an indexing job was abandoned by an app quit / restart, so the item - * was parked at `failed` instead of silently resumed (see KnowledgeService.recoverInterruptedItems). + * was parked at `failed` instead of silently resumed (see KnowledgeIngestionService.recoverInterruptedItems). + * - `never_indexed`: a migrated v1 item never started indexing (no `uniqueId`), so there is + * nothing to resume from and it is parked at `failed` instead of a synthetic in-progress state. * Modeled as a zod enum (the same shape as the base error codes above) so the renderer's * code → i18n switch in `error.ts` stays exhaustive-checkable and the code ↔ translator-key * triple is tied together. Codes are localized by the UI; any other value is a free-form message. */ -export const KNOWLEDGE_ITEM_ERROR_CODES = ['directory_not_migrated', 'indexing_interrupted'] as const +export const KNOWLEDGE_ITEM_ERROR_CODES = ['directory_not_migrated', 'indexing_interrupted', 'never_indexed'] as const export const KnowledgeItemErrorCodeSchema = z.enum(KNOWLEDGE_ITEM_ERROR_CODES) export type KnowledgeItemErrorCode = z.infer export const KNOWLEDGE_ITEM_ERROR_DIRECTORY_NOT_MIGRATED: KnowledgeItemErrorCode = 'directory_not_migrated' export const KNOWLEDGE_ITEM_ERROR_INDEXING_INTERRUPTED: KnowledgeItemErrorCode = 'indexing_interrupted' +export const KNOWLEDGE_ITEM_ERROR_NEVER_INDEXED: KnowledgeItemErrorCode = 'never_indexed' export const KnowledgeChunkSizeSchema = z.number().int().positive() export const KnowledgeChunkOverlapSchema = z.number().int().min(0) @@ -152,7 +153,14 @@ export const KnowledgeBaseEntitySchema = z.strictObject({ updatedAt: z.iso.datetime() }) -export const KnowledgeBaseSchema = KnowledgeBaseEntitySchema.superRefine((value, ctx) => { +/** + * Cross-field invariants for a knowledge base row, shared by the read-side entity + * schema and the pre-write candidate schema so a rule is defined exactly once. + */ +function refineKnowledgeBaseInvariants( + value: Omit, 'id' | 'createdAt' | 'updatedAt'>, + ctx: z.RefinementCtx +): void { if (value.status === 'completed') { if (value.error !== null) { ctx.addIssue({ @@ -206,9 +214,30 @@ export const KnowledgeBaseSchema = KnowledgeBaseEntitySchema.superRefine((value, message: 'Hybrid alpha requires hybrid search mode' }) } -}) + + if (value.chunkStrategy === 'delimiter' && !value.chunkSeparator) { + ctx.addIssue({ + code: 'custom', + path: ['chunkSeparator'], + message: 'Separator is required when chunk strategy is delimiter' + }) + } +} + +export const KnowledgeBaseSchema = KnowledgeBaseEntitySchema.superRefine(refineKnowledgeBaseInvariants) export type KnowledgeBase = z.infer +/** + * The full row about to be inserted/updated, validated against the same + * invariants as the read-side schema before it ever reaches the DB CHECK + * constraints — `id`/`createdAt`/`updatedAt` don't exist yet at write time. + */ +export const KnowledgeBaseWriteSchema = KnowledgeBaseEntitySchema.omit({ + id: true, + createdAt: true, + updatedAt: true +}).superRefine(refineKnowledgeBaseInvariants) + /** * A knowledge base that has finished setup and is ready for runtime operations * (opening its index store, BM25 search, indexing). Covers both a vector base @@ -262,7 +291,7 @@ const KnowledgeItemSharedSchema = z.strictObject({ */ export const FileItemDataSchema = KnowledgeItemSharedSchema.extend({ // relativePath / indexedRelativePath are always produced by main-side helpers - // (copyFileIntoKnowledgeBaseAt, toKnowledgeRelativePath, ...), never raw caller + // (copyFileIntoKnowledgeBaseAt, toMaterialRelativePath, ...), never raw caller // input. The base-relative, POSIX-normalized, no-traversal invariant is // enforced imperatively by assertSafeKnowledgeRelativePath at the filesystem // boundary (getKnowledgeBaseFilePath). This schema only validates shape, so a @@ -361,11 +390,6 @@ const KnowledgeItemEntityBaseSchema = z.strictObject({ updatedAt: z.iso.datetime().describe('ISO timestamp when the item row was last updated.') }) -const IdleKnowledgeItemLifecycleSchema = { - status: z.literal('idle').describe('Item row exists but indexing has not started.'), - error: z.null().describe('No error is stored for non-failed lifecycle states.') -} as const - const PreparingKnowledgeItemLifecycleSchema = { status: z.literal('preparing').describe('Container expansion is running; only directory items use it.'), error: z.null().describe('No error is stored for non-failed lifecycle states.') @@ -408,11 +432,6 @@ const createLeafKnowledgeItemEntitySchemas = [ - KnowledgeItemEntityBaseSchema.extend({ - type: z.literal(type), - data, - ...IdleKnowledgeItemLifecycleSchema - }), KnowledgeItemEntityBaseSchema.extend({ type: z.literal(type), data, @@ -450,11 +469,6 @@ const createContainerKnowledgeItemEntitySchemas = [ - KnowledgeItemEntityBaseSchema.extend({ - type: z.literal(type), - data, - ...IdleKnowledgeItemLifecycleSchema - }), KnowledgeItemEntityBaseSchema.extend({ type: z.literal(type), data, diff --git a/tests/__mocks__/README.md b/tests/__mocks__/README.md index 7a008f9f1fc..2077c6f1757 100644 --- a/tests/__mocks__/README.md +++ b/tests/__mocks__/README.md @@ -333,7 +333,7 @@ Database service providing access to the mock SQLite database. | Method | Signature | |--------|-----------| | `getDb` | `() => MockDb` | -| `withWriteTx` | `(fn: (tx) => Promise) => Promise` (passthrough — calls `fn(db)`) | +| `withWriteTx` | `(fn: (tx) => T) => T` (synchronous, mirrors production) | | `isReady` | `boolean` (getter) | ```typescript @@ -348,7 +348,7 @@ MockMainDbServiceUtils.getDefaultMockDb() MockMainDbServiceUtils.setDb(customMockDb) ``` -> **`withWriteTx`**: passthrough (`async (fn) => fn(this.db)`) — no mutex / BUSY retry. Use `vi.spyOn(dbServiceInstance, 'withWriteTx')` to inject custom behavior. Hand-rolled DbService mocks MUST include this method or production code throws `TypeError: dbService.withWriteTx is not a function`. +> **`withWriteTx`**: once a real DB is attached (`setupTestDatabase()` → `setDb()`), delegates to `db.transaction(fn, { behavior: 'immediate' })` for real transaction semantics (rollback on throw, etc.); in pure-mock mode (default db, no `setDb()`) falls through to `fn(this.db)` — no mutex / BUSY retry. Use `vi.spyOn(dbServiceInstance, 'withWriteTx')` to inject custom behavior. Hand-rolled DbService mocks MUST include this method or production code throws `TypeError: dbService.withWriteTx is not a function`. --- diff --git a/tests/__mocks__/main/DbService.ts b/tests/__mocks__/main/DbService.ts index f3a504c8350..944f82a583c 100644 --- a/tests/__mocks__/main/DbService.ts +++ b/tests/__mocks__/main/DbService.ts @@ -69,12 +69,19 @@ export class MockMainDbService { public getDb = vi.fn(() => this.db) /** - * Write transaction mock. Mirrors `DbService.withWriteTx`: synchronously passes - * the current db (or whatever was set via `setDb`) to the synchronous `fn` so - * tests exercising the write path do not need a real transaction. Tests can - * replace this mock with `vi.spyOn(...)` to assert call order, etc. + * Write transaction mock. Mirrors `DbService.withWriteTx`: when a real + * better-sqlite3 connection is attached (via `setDb`, e.g. `setupTestDatabase()`), + * delegates to its `.transaction()` for real transaction semantics (rollback on + * throw, etc.); otherwise falls through to the plain (non-transactional) db stub. + * Tests can replace this mock with `vi.spyOn(...)` to assert call order, etc. */ - public withWriteTx = vi.fn((fn: (tx: unknown) => T): T => fn(this.db)) + public withWriteTx = vi.fn((fn: (tx: unknown) => T): T => { + const db = this.db as { transaction?: (fn: (tx: unknown) => unknown, options?: unknown) => unknown } + if (typeof db?.transaction === 'function') { + return db.transaction(fn, { behavior: 'immediate' }) as T + } + return fn(this.db) + }) public get isReady() { return this._isReady