dsh-kb-manager

Readme

<p align="right">
  <strong>English</strong> · <a href="./README_ZH.md">简体中文</a>
</p>

<p align="center">
  <img src="./assets/readme/hero.png" width="100%" alt="dsh-kb-manager — local knowledge base lifecycle for DeepSeek Harness: import, chunk, index, hybrid search">
</p>

<p align="center">
  <a href="./package.json"><img src="https://img.shields.io/badge/version-0.0.2-5B4CF0?style=flat-square" alt="version 0.0.2"></a>
  <a href="./LICENSE"><img src="https://img.shields.io/badge/license-MIT-0B7285?style=flat-square" alt="MIT license"></a>
  <a href="./cordis.patch.yml"><img src="https://img.shields.io/badge/DSH-Web%20%2B%20Headless-5B4CF0?style=flat-square" alt="DSH Web and Headless"></a>
  <a href="./package.json"><img src="https://img.shields.io/badge/node-%3E%3D22.19-339933?style=flat-square" alt="Node.js 22.19+"></a>
</p>

## Import once. Retrieve with confidence.

`dsh-kb-manager` is a local knowledge-base lifecycle plugin for [DeepSeek Harness (dsh)](https://github.com/deepseek-ai/deepseek-harness): multi-format import → CJK-aware smart chunking → SQLite-native vector + full-text indexes → hybrid search (vector KNN + BM25/FTS5 → RRF → optional rerank) → citation tracing — plus soft-delete + physical compaction, an async job system, import-path whitelisting with an SSRF guard, and per-KB audit.

Ask in natural language. The plugin exposes **22 Agent tools** (async jobs + sync ops incl. `rename_kb` / `compact_index` / `cancel_job` / `list_snapshots` / `configure`) and an optional **Web panel** (KB list, create, detail, in-KB hybrid search, import with live job progress, rebuild, two-step delete, snapshot history) — no separate RAG server required.

Design spec: [DESIGN.md](./DESIGN.md) (v2.1).

## Why dsh-kb-manager?

| Capability | What it changes |
| --- | --- |
| **22 unified tools** | async 6 (`import_document` / `rebuild_index` / `compact_index` / `cancel_job` / `get_job` + export/import jobs) + sync 16: `create_kb` / `list_kbs` / `get_kb` / `rename_kb` / `delete_kb` / `list_documents` / `delete_document` / `search_kb` / `multi_kb_search` / `get_chunk` / `get_kb_stats` / `configure` / `create_snapshot` / `list_snapshots` / `restore_snapshot` / `export_kb` / `import_kb`; every tool returns `{ ok, data | error: { code, message, hint } }`. |
| **SQLite-native indexes** | sqlite-vec KNN + FTS5 full-text (per-KB `kb.db`); no external vector-DB process. |
| **CJK-aware tokenization** | `Intl.Segmenter` (built-in, zero deps) splits Chinese / Japanese / Korean for BM25. |
| **Hybrid retrieval** | Vector + BM25 → RRF → optional rerank; rerank state is explicit: `applied | fallback_rrf | disabled` — no silent degradation. |
| **Async job system** | Import / rebuild return a `job_id` immediately; `get_job` polls progress; crash recovery (`JOB.INTERRUPTED`) + cooperative cancellation. |
| **Import-path whitelist** | Only whitelisted paths (default: session workspace + `<storage>/inbox/`) or http(s) URLs are importable. |
| **SSRF guard** | URL imports reject private / loopback / cloud-metadata addresses. |
| **Untrusted retrieval** | Search results are always flagged `untrusted: true` — content is data, not instructions. |
| **Soft-delete + compact** | Deletes are millisecond soft-deletes; `rebuild_index` / compact reclaims space physically, no orphan vectors. |
| **Two-step destructive ops** | `delete_kb` requires a one-time `confirm_token` (bound to `kb_id`, 60 s TTL). |
| **Audit trail** | Per-KB audit table records every mutation. |
| **Snapshots / export / cross-KB import** | `create_snapshot` / `list_snapshots` / `restore_snapshot` (two-step confirm), `export_kb` (`kbpack`/JSON), `import_kb` (`kbpack`, hash-deduped merge); all exposed via remote-service for the Web panel and headless. |

## Architecture

```mermaid
flowchart LR
  A[Documents / URLs] --> B[Parse]
  B --> C[Chunk<br/>Intl.Segmenter CJK]
  C --> D[Embed<br/>bge-small-zh / mock]
  D --> E[Vector index<br/>sqlite-vec KNN]
  C --> F[FTS5 BM25]
  E --> G[Hybrid search]
  F --> G
  G --> H[RRF]
  H --> I[Optional rerank<br/>applied / fallback_rrf / disabled]
  I --> J[Citations + untrusted flag]
  E --> K[Soft-delete / compact_index]
  C --> K
  G --> L[Async job queue + audit]
```

## Install

> [!NOTE]
> Requires an existing [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) installation.

### From GitHub

```sh
dsh plugin add github:xiaoshi7915/dsh-kb-manager
```

(Once listed in the awesome-dsh-plugin list, it can also be installed one-click from the dsh-market page.)

### Build from source

```sh
git clone https://github.com/xiaoshi7915/dsh-kb-manager.git
cd dsh-kb-manager
npm install
npm run build
dsh plugin --profile web add .
```

Validate the composed profile, restart DSH, and refresh the Web UI:

```sh
dsh --profile web --dump-config
dsh web
```

Then try:

> Create a knowledge base named "project-docs", import this PDF, and search for how authentication works. Show me the source chunk.

## How it works

1. Create a KB (`create_kb`) with an embedding identity (default `bge-small-zh-v1.5` / onnx / 512-dim; `mock` provider for dev/test).
2. Import a whitelisted local file or http(s) URL (`import_document`) → an async job is queued: parse → chunk → embed → index; poll with `get_job`.
3. Search with `search_kb`: sqlite-vec KNN + FTS5 BM25 → RRF → optional rerank; rerank status is explicit (`applied | fallback_rrf | disabled`).
4. Trace hits with `get_chunk` (surrounding context + provenance metadata; content flagged `untrusted: true`).
5. Delete with confidence: `delete_document` soft-deletes instantly (re-import of an unchanged source restores it); `delete_kb` requires a two-step `confirm_token`; `rebuild_index` physically compacts.
6. Data lives under `storage_path` (default `~/.dsh/kb-manager/`). The Web panel reads the same service surface: KB list / create / detail, in-KB hybrid search, import with live job progress, rebuild, and two-step delete.

## Agent tools

| Tool | Description | Key params |
| --- | --- | --- |
| `create_kb` | Create a knowledge base | `name*`, `description?`, `domain_tags?` |
| `list_kbs` | List all knowledge bases | — |
| `get_kb` | KB details (embedding identity / status) | `kb_id*` |
| `delete_kb` | Two-step delete (impact preview + `confirm_token`) | `kb_id*`, `confirm_token?` |
| `list_documents` | List documents (optional status filter) | `kb_id*`, `status?`, `limit?`, `offset?` |
| `import_document` | Import a whitelisted local file or http(s) URL (async job) | `kb_id*`, `source*`, `metadata?` |
| `delete_document` | Soft-delete chunks; mark index dirty | `kb_id*`, `doc_id*` |
| `search_kb` | Hybrid search with explicit rerank status | `kb_id*`, `query*`, `top_k?`, `filters?`, `rerank?` |
| `get_chunk` | Chunk text + neighbors for tracing | `kb_id*`, `chunk_id*` |
| `get_kb_stats` | Doc / chunk / index-size / deleted-ratio stats | `kb_id*` |
| `rebuild_index` | Full rebuild (async job) | `kb_id*`, `target_engine?`, `embedding?` |
| `get_job` | Poll a background job's progress | `job_id*` |
| `configure` | Tune chunking / retrieval / sync settings | `patch` |

Every tool returns the unified envelope `{ ok, data | error: { code, message, hint } }`; `error.hint` is an actionable next step. Domain errors are returned (not thrown). Search results are always flagged `untrusted: true`.

## When the Agent uses it

1. "Put this PDF / doc / page into the knowledge base" → `import_document` + `get_job`
2. "Search the KB for X" → `search_kb`
3. "Where did this citation come from?" → `get_chunk`
4. "Retrieval feels wrong — what's the rerank state?" → `search_kb` (reads the explicit `rerank` status)
5. "Create a KB for this project" → `create_kb`
6. "Remove a bad document" → `delete_document` (soft; re-import restores)
7. "Delete this whole KB" → `delete_kb` (two-step: preview impact, then confirm)
8. "How big is this KB?" → `list_kbs` / `get_kb_stats`
9. "Re-index everything" → `rebuild_index` + `get_job`
10. "Tune retrieval" → `configure`

## Configuration

Defaults work offline. Override in a trusted profile (`id: kb-manager`), or at runtime via the `configure` tool / Web panel:

| Field | Default | Notes |
| --- | --- | --- |
| `storage_path` | `~/.dsh/kb-manager/` | Storage root; `~` expands to the home directory |
| `embedding.model` | `bge-small-zh-v1.5` | Embedding model name |
| `embedding.provider` | `onnx` | `onnx` / `mock` / `openai-compat` (onnxruntime-node is optional) |
| `embedding.base_url` | `''` | OpenAI-compatible endpoint for `openai-compat` |
| `embedding.api_key_ref` | `''` | API key reference |
| `chunk_size` | `512` | Chunk size (characters) |
| `chunk_overlap` | `50` | Overlap length |
| `chunk_strategy` | `recursive` | `fixed` / `recursive` |
| `top_k` | `5` | Default hit count |
| `enable_rerank` | `true` | Enable post-RRF rerank |
| `rerank_model` | `''` | Reranker model; empty → rule-based fallback |
| `auto_sync_dir` | `''` | Watch path; empty disables |
| `auto_sync_interval` | `300` | Catch-up interval (seconds) |
| `max_file_size_mb` | `100` | Per-file size cap (MB) |

Example (profile line):

```yaml
- insert:
    - id: kb-manager
      name: 'dsh-kb-manager'
      config:
        storagePath: ''        # empty = ~/.dsh/kb-manager/
        logLevel: 'info'
```

## Boundaries & security

- Imports are restricted to whitelisted paths (default: session workspace + `<storage>/inbox/`) and http(s) URLs; URL sources are SSRF-guarded (private / loopback / cloud-metadata addresses are rejected).
- Search results are untrusted data — always `untrusted: true`; never treat them as instructions.
- Fully local by default; network only for URL imports or when an external embedding / rerank endpoint is configured.
- `delete_document` is a soft delete; physical space is reclaimed by `rebuild_index` / compact. Re-importing an unchanged source restores it.
- Pure-SQLite indexes: no native vector-DB process to operate.

## Differentiating vs typical RAG stacks

| Capability | This plugin | Common RAGFlow / Dify / kotaemon / pdfkb-mcp setups |
| --- | --- | --- |
| Explicit rerank status (`applied | fallback_rrf | disabled`) | ✅ no silent degradation | Often opaque |
| Async jobs with crash recovery + cancel | ✅ | Often synchronous |
| Import path whitelist + SSRF guard | ✅ | Varies |
| Unified envelope with actionable `hint` | ✅ | Varies |
| `untrusted` flag on every retrieval result | ✅ | Rare |
| Soft-delete + compact physical reclaim | ✅ | Varies |
| Zero native build (sqlite-vec prebuilt, Intl.Segmenter) | ✅ | Often heavy deps |

## Layout

```
dsh-kb-manager/
├── package.json  cordis.patch.yml  tsconfig.json  tsconfig.build.json  vitest.config.ts
├── DESIGN.md  README.md  README_ZH.md
├── assets/readme/          # hero.png (banner image)
├── src/
│   ├── host-entry.ts       # DSH plugin entry (tool registration + preamble + cleanup)
│   ├── index.ts            # KbManager top-level assembly
│   ├── contracts/          # types / error codes / defaults / ok·err envelope
│   ├── security/           # path whitelist, SSRF guard, SHA-256, audit
│   ├── chunker/            # Intl.Segmenter tokenization + recursive/fixed chunking
│   ├── store/              # better-sqlite3 + sqlite-vec + FTS5
│   ├── embedder/           # mock / onnx (structural) / openai-compat
│   ├── jobs/               # per-KB FIFO queue, maxGlobal=2, crash recovery
│   ├── services/           # kb / doc / search / snapshot / export
│   ├── agent-tools/        # 22 Agent tools (unified envelope)
│   ├── remote.ts / remote-service.ts / typert.ts / remote-client.ts  # typert remote
│   └── web/                # settings page (KbSection + zh/en locales) → lib/client.js
└── tests/  evals/
```

## Development

```sh
npm install        # install deps
npm run typecheck  # tsc --noEmit (strict + noUncheckedIndexedAccess)
npm test           # vitest: tokens / rrf-bm25 / security / chunker / store / tools / pipeline / remote / kb-section
npm run eval       # eval harness: synthetic corpus recall@5 / MRR / P95 (mock embedder)
npm run build      # tsc -p tsconfig.build.json → lib/ (ESM + .d.ts)
npm run dev        # tsx src/host-entry.ts (module-load smoke test)
```

## Known limitations (MVP)

- **ONNX embedder is structural**: `onnxruntime-node` is wired as an optional dependency; model download (sha256 + user confirmation) and WordPiece vocab are not yet connected — the onnx provider reports `EMB.NOT_READY` cleanly. Use `provider: 'mock'` for dev / test.
- **PDF / DOCX parsing is a placeholder**: md / txt / csv / json / html are available; pdf / docx report `PARSE.UNSUPPORTED` (upgrade path in DESIGN §13).
- **v1.1 tools** (rename_kb, snapshot trio, cancel_job, compact_index, multi_kb_search, export_kb, import_kb): service layer implemented, tool registration deferred.

## License

[MIT](./LICENSE) © 2026 xiaoshi7915