dsh-knowledge-sync

Readme

# dsh-knowledge-sync

English | [中文](README.zh.md)

Freeze every round of a [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) conversation into a knowledge-base document, and let later conversations find it.

A round ends, and what was asked, what was concluded, and which tools ran become one Markdown file. Nothing is sent to a model to make that happen. Later, an agent working in the same directory is told the base exists — two sentences, not the documents — and decides for itself whether to look.

## What you get

| | |
|---|---|
| **A file per document** | Markdown with YAML front matter under `<root>/<session>/<id>-<slug>.md` — a captured round (`raw`), a distilled finding, or a note the agent chose to record |
| **Worth keeping, or not written** | A round is captured only if it clears the salience bar — a terse "12 passed" status is not knowledge, so it is not saved |
| **Redaction before disk** | Tool arguments are scrubbed of credentials (tokens, `--password=`, `.env` values, PEM keys…) before the file exists, with configurable extra patterns |
| **Readable by everything** | Greppable, editable, and ready for version control — the files are the source of truth, with no sidecar index |
| **Recall, not injection** | A short pointer in the system prompt plus `knowledge_search` and `knowledge_read`; the documents never enter the context uninvited |
| **Full-text search** | An in-memory BM25 index searches document bodies, not just titles, and returns a snippet per hit |
| **Scoped to a workspace** | An agent recalls what was recorded in its own directory, so another project's knowledge is not noise in this one |
| **A page to read it** | A **Knowledge** section in settings: filter, open, read, with kind and tag badges |
| **A page to configure it** | A **Config** tab in the same section edits redaction, capture, and recall live — writes persist to settings, values apply without a restart |

## Install

```sh
dsh plugin --profile web add dsh-knowledge-sync
dsh --profile web --dump-config   # shows a "# == dsh-knowledge-sync" layer
dsh web
```

Documents land in `./knowledge` beside wherever you launched, or wherever `DSH_KNOWLEDGE_ROOT` points. Override anything in the profile's `cordis.patch.yml`:

```yaml
- id: knowledge
  config:
    root: /srv/project/knowledge
```

## What a document looks like

```markdown
---
id: "abc12345-t1"
title: "Why is the build red?"
session: "abc12345-0000-4000-8000-000000000000"
turn: 1
created: "2026-08-18T10:00:00.000Z"
kind: "raw"
outcome: "completed"
cwd: "/srv/project"
tools: ["bash"]
---

# Why is the build red?

## Question

Why is the build red?

## Answer

The lockfile was stale — `pnpm install` had not been re-run after the
dependency bump.

## Tools

- `bash` — {"command":"pnpm install"}
```

Reasoning is left out on purpose: a document records the conclusion, not the working that reached it. A round that committed no answer is not written at all — an abandoned turn records that something was attempted, not something learned.

### Three kinds of document

| `kind` | Where it comes from | Example body |
|---|---|---|
| `raw` | A round that cleared the salience filter | `## Question` / `## Answer` / `## Tools` |
| `note` | The agent called `knowledge_note` mid-work | `## Finding` / `## Evidence` / `## Scope` |
| `distilled` | A kept round condensed by a small model call (opt-in) | `## Summary` / `## Finding` / then the raw round as appendix |

## How recall works

The plugin does **not** paste the knowledge base into the conversation. A previous round is usually irrelevant to the current one, and injecting the lot would spend the context window on that irrelevance and invalidate the prompt prefix every time a round is written.

Instead an agent working in a directory with recorded rounds sees one short section:

> **Knowledge base** — 12 documents from earlier conversation rounds for this workspace (/srv/project) are available. They are NOT included in this conversation.
>
> Call `knowledge_search` when the task resembles work that may already have been done here — a recurring build failure, a decision that was already made, a file whose purpose was already established. Search reads the documents themselves, so a phrase from the conclusion will find its round. Then `knowledge_read` the ones worth reading in full. Prefer checking over repeating work, and prefer your own judgement over a stale document when the two disagree.

`knowledge_search` searches document bodies through the in-memory BM25 index — not just titles and tool names — and returns titles, kinds, tags, and a snippet of the match. Notes and distilled findings rank above raw transcripts, because a finding written down deliberately is worth more than a round that happened to record one. `knowledge_search` can also filter by `kind`, `tag`, and `tool`. `knowledge_read` returns one document in full, by id, and only for documents the same rule made visible.

A workspace with nothing recorded contributes **no section at all**. A pointer to an empty shelf teaches the model to stop believing the pointer.

## Configuration

| Row | Field | Meaning |
|---|---|---|
| `knowledge` | `root` | Where documents are written |
| | `redact.enabled` | Scrub credentials before they reach disk (default on) |
| | `redact.mask` | The replacement for a redacted value |
| | `redact.patterns` | Extra regular-expression patterns for redaction |
| `knowledge-capture` | `enabled` | Freeze rounds as they end |
| | `salience.minScore` | How a round earns its place; 0 keeps every round that has an answer |
| | `distill.enabled` | Condense a kept round with one small model call (default off) |
| | `distill.provider` / `distill.model` | The route to call when distillation is on |
| `knowledge-note` | `enabled` | Register the `knowledge_note` tool |
| `knowledge-recall` | `announce` | Tell the model the base exists |
| | `sameWorkspaceOnly` | Limit recall to the session's own directory |
| | `searchLimit` | Results per search |
| `knowledge-http` | `path` | Where the page reads from |

Each row is independent: drop `knowledge-http` in a composition with no web server, or `knowledge-recall` to record without offering recall, and the rest still works.

> **Install note.** The package is a Cordis bundle: its `cordis.patch.yml` must stay in step with its build output. Rebuild (`pnpm run build`) after changing source so `lib/` carries every module the patch references (the capture, note, recall, http, distill, policy, redact, search and settings entries), then add `dsh-knowledge-sync` to your profile's `dsh.profile.bundles` and restate in your profile's `cordis.patch.yml` any row you override. A patch row pointing at a not-yet-built `lib/<module>.js` will fail the profile boot.
>
> **Config is optional.** The store registers a `knowledge` settings namespace so the page's Config tab is writable. If no settings service is mounted, nothing changes behavior: the rows' `Config` defaults stay authoritative and the tab reports that live settings are unavailable.

## Development

The repository is self-contained — it develops against the published `@deepseek-ai/*` packages, the same versions a `dsh` installation ships:

```sh
pnpm install
pnpm run typecheck
pnpm run test
pnpm run build
```

Pin those devDependencies exactly, never by range: the registry's `latest` tag for several harness packages points at a much older release than the CLI installs, and a range resolves to that stale set.

The tests run the real thing — a real agent loop and a scripted model — because the shape of a session log is the one thing this plugin cannot guess at.

## Known limitations

- **Lexical search, not semantic.** The in-memory BM25 index matches words (with CJK bigrams) — it cannot match a phrase it has never seen. Fitting in this data scale; the LLM seam has no embedding support yet.
- **Recall is per directory.** Sessions in a subdirectory of a recorded workspace do not see it, because the match is exact.
- **No relation tracking yet.** Documents can still contradict each other silently; the `supersedes` / staleness signals from the plan's third phase are not built.
- **Distillation is opt-in.** Off by default; enabling it costs one small model call per kept round.
- **Documents are read-only on the page.** The knowledge base is written by the conversations that produced it; a page that could rewrite history would make a document mean less than the round it froze. (Config is editable; documents are not.)

## License

MIT