@zzdream67/dsh-vision-bridge

Readme

# @zzdream67/dsh-vision-bridge

**Let text-only models read images, without switching models.**

[中文文档](./README.zh.md)

In DeepSeek Harness, pasting an image at a text-only model is refused outright:

```
Model "xxx" does not support image input.
```

This plugin intercepts the `llm/stream` waterfall and substitutes each image block with a vision model's transcription before the request reaches the provider. You paste a picture; the text-only model receives words it can read.

```
you paste an image + a question
   ↓
the plugin captions it with YOUR configured vision model
   ↓
the text-only model receives: [image "shot.png" transcribed: a Python TypeError…] + your question
   ↓
it answers from that text
```

---

## Installation

### Prerequisite: the `dsh` command

Every command below starts with `dsh`. If it is not on your PATH:

```
dsh : The term 'dsh' is not recognized as the name of a cmdlet, function, script file, or operable program.
```

Then either install DSH globally:

```powershell
npm install -g @deepseek-ai/dsh
```

Or leave it uninstalled and prefix every `dsh` in this README with `npx -y @deepseek-ai/dsh`:

```powershell
npx -y @deepseek-ai/dsh plugin --profile web add @zzdream67/dsh-vision-bridge
```

The `npx` form downloads DSH on demand and works on a machine that has only Node.js.

### Prerequisite: pnpm

`dsh plugin` forwards to pnpm internally. Many environments lack it:

```
'pnpm' is not recognized as an internal or external command
dsh: pnpm failed in profile directory ...
```

Install it:

```powershell
npm install -g pnpm
```

> `corepack enable pnpm` often fails on Windows because it needs to write into `C:\Program Files\nodejs\` (`EPERM`). The `npm install -g` route is more reliable.

### From npm (recommended)

```powershell
dsh plugin --profile web add @zzdream67/dsh-vision-bridge
```

Installs prebuilt output; no build authorization needed.

### From GitHub

```powershell
dsh plugin --profile web add github:zzdream67/dsh-vision-bridge
```

A git install fetches source and builds it on install. pnpm ≥10 refuses to run a git dependency's build script without explicit permission, so the first `add` fails with `ERR_PNPM_GIT_DEP_PREPARE_NOT_ALLOWED`.

That error prints the exact key to allow, ending with the commit it resolved:

```
allowBuilds:
  @zzdream67/dsh-vision-bridge@https://codeload.github.com/zzdream67/dsh-vision-bridge/tar.gz/<sha>: true
```

Copy that key verbatim into the profile's `pnpm-workspace.yaml` — quoting it, because it contains `@` and `:`. The file already exists, so append rather than replace:

```yaml
allowBuilds:
  '@zzdream67/dsh-vision-bridge@https://codeload.github.com/zzdream67/dsh-vision-bridge/tar.gz/<sha>': true
```

Then re-run `add`. The key includes a commit hash, so it changes whenever the branch moves. Pin a commit (`github:zzdream67/dsh-vision-bridge#<sha>`) to keep it stable, or install from npm and skip this entirely.

### From a tarball

```powershell
npm pack                                    # in this repo
dsh plugin --profile web add ./zzdream67-dsh-vision-bridge-0.1.0.tgz
```

### Uninstall

```powershell
dsh plugin --profile web remove @zzdream67/dsh-vision-bridge
```

The dependency and its config layer are both removed. If `manageDeclarations` was on, the plugin withdraws the declarations it wrote as it unloads.

Withdrawal restores the exact prior value. Each declaration records what the row's `input` looked like beforehand, because after removing `image` a leftover `['text']` is indistinguishable from a field that was never there. A rule-based revert would delete a line you wrote by hand; the recorded value prevents that.

The record is a note about a row that existed when the note was written, and the file belongs to you. If you delete the model, delete the whole route, or hand-edit the row before withdrawal, the plugin declines rather than reconstructs:

| You did this | Withdrawal does |
|---|---|
| Deleted the model | Skips it; a deleted model is never recreated from a stale note |
| Deleted the whole route | Skips it, and still cleans up every other route |
| Hand-edited `input` into something malformed | Leaves it for you to fix |
| Nothing; the row is as the plugin left it | Restores the exact prior value |

One missing target does not abandon the rest of the withdrawal. Recreating a row you deliberately removed would be worse than leaving a declaration on a row that no longer exists, so missing rows are skipped.

The inline-comment alignment is the one thing that does not come back; see Development for the details.

---

## Configuration

The plugin ships a browser-side bundle (`dist/client.js`) that registers its own section into the Web settings page's `settings.section` slot. Every field is editable there, applies live, and needs no restart.

Three buttons in that page are worth knowing about:

- **Test and enable** sends one 16×16 PNG to the selected vision model. The declaration is written only if the endpoint accepts the image, and is withdrawn again on failure. Capability is tested rather than assumed, because an OpenAI-compatible `/v1/models` reply carries no modality information.
- **Enable directly** skips the test and writes the declaration. Use it when you already know the model can see.
- **Reload** re-reads the configuration from disk. The desktop client has no reload gesture, so this is the only way to pick up a change made in the Models page, in another window, or by hand. It asks first if you have unsaved edits.

> Host-side schema registration alone does **not** render a form. The section exists because this plugin authors that browser bundle by hand; the build preset for DSH's client format is not published, but the format itself is plain and stable.

### Why a declaration is required first

The admission gate (`dsh-host-apiproxy`) checks the model's declaration before the image enters the session:

```js
if (modelInfo.inputModalities !== undefined && !modelInfo.inputModalities.includes('image'))
  return err(request, { details: { reason: 'MODEL_DOES_NOT_SUPPORT_IMAGES' } })
```

A text-only model must declare `input: [text, image]`, or the image never arrives and this plugin's conversion never runs.

`manageDeclarations` (default: on) writes that declaration for each model in `bridge` and withdraws it when you turn the switch off, disable the plugin, or unload it. With the plugin loaded and the model bridged, the route genuinely does accept image input, because conversion happens before the provider is called.

`input: [text, image]` is an admission label. Every place the host reads it is guarded by "does this request contain an image", so a text-only request behaves identically whether or not the declaration is present. It does not affect token accounting, context window, sampling, or model behaviour.

Declaring a model without bridging it moves the failure later: the host admits the image and the provider then rejects it. Declarations therefore follow the `bridge` list rather than being applied broadly.

### Fields

| Field | Default | Meaning |
|---|---|---|
| `enabled` | `true` | Master switch. When off the plugin stays loaded but intervenes in nothing and withdraws its declarations, like uninstalling without touching node_modules |
| `visionProvider` | `''` | Vision route id. Empty leaves the plugin inert (see Behavior details) |
| `visionModel` | `''` | Vision model id in that route. Empty leaves the plugin inert (see Behavior details) |
| `prompt` | see below | Caption instruction sent with each image |
| `bridge` | `[]` | Text-only models to bridge |
| `manageDeclarations` | `true` | Let the plugin write/withdraw modality declarations |
| `cacheSize` | `64` | Cached transcriptions; `0` disables caching |
| `timeoutMs` | `120000` | Per-image caption timeout |
| `verbose` | `false` | One log line per conversion (never logs transcription text) |

Or in `cordis.patch.yml`:

```yaml
- id: zz-vision-bridge
  name: '@zzdream67/dsh-vision-bridge'
  config:
    visionProvider: my-local-route
    visionModel: my-vision-model
    bridge:
      - provider: my-text-route
        model: my-text-model
    manageDeclarations: true
```

A model not listed in `bridge` is untouched and keeps the host's exact behavior.

Models under the built-in `deepseek-official` route cannot be bridged: its modality is hardcoded in the plugin that provides it, and its adapter refuses image content. To bridge DeepSeek, add a custom model provider with a route using `openai-completions` at `https://api.deepseek.com/v1`, then bridge the models under that route.

### The default prompt

It asks for transcription, not interpretation: transcribe all visible text faithfully, describe layout and structure, never speculate or fill gaps, and state plainly what is illegible. The reasoning belongs to the model consuming the transcription; a vision model that editorializes costs the caller information it can never recover.

---

## Behavior details

| Situation | Behavior |
|---|---|
| No vision backend configured | The plugin is inert: it loads, does nothing, and warns once when an image arrives |
| Target model not in `bridge` | Passes through with zero impact |
| The request is the plugin's own caption call | Skipped (otherwise it would recurse forever) |
| No image in the request | Fast path, skipped |
| Same image appears repeatedly | Resolved sequentially, hits the cache, captioned once |
| Caption fails or times out | Injects a failure notice telling the model not to pretend it saw the image |
| The request object (always deep-frozen) | Not mutated. A fresh request carrying the rewritten messages is issued through `ctx.llm.stream()` |

Caption calls go through `ctx.llm.stream()`, so the vision route's credentials, retry policy, attribution headers, and observability all still apply. The plugin issues no HTTP requests of its own.

---

## Before you install

**The model receives words, not pixels.**

| Works | Does not work |
|---|---|
| Reading errors, code, logs in a screenshot | Comparing two images precisely |
| Describing UI layout and button positions | Picking colors, measuring pixels |
| Transcribing tables and chart trends | Fine spatial or geometric reasoning |
| Reading handwriting and printed text | Judgments resting on subtle visual detail |

Every substitution is labeled as a transcription, so the model does not treat it as having seen the image itself.

**You need your own vision model.** The plugin ships no vision service. It uses a vision model you configure: a local LM Studio / Ollama instance, or a remote service you hold a key for. A local model costs nothing and sends nothing off your machine; a remote one bills your own account.

**The plugin writes to `settings.yaml`.** `manageDeclarations` writes `input: [text, image]` into the `llm-pi-ai` section for each bridged model and withdraws it when you turn the switch off, disable the plugin, or unload it. The switch is on by default, because the declaration is what makes the feature work. What is preserved and what is not is detailed in Development. Back up `settings.yaml` before enabling it the first time.

---

## Development

```powershell
npm install
npm run build      # tsc -> dist/
npm test           # node --test, 68 cases
npm run typecheck
```

`src/rewrite.ts` is pure and I/O-free, so the rewriting logic is fully testable without a live model. `test/uninstall.test.mjs` asserts the declare/withdraw round trip is byte-identical for an absent field, a hand-written `[ text ]`, unusual modality lists, and untouched sibling rows.

### Why `llm/stream` instead of a new route

The common approach registers a twin route (e.g. `xxx (vision)`) with hardcoded modalities. This plugin does not:

- It registers no route and no adapter, so the host's model registry keeps telling the truth and no extra entry appears in the model picker.
- It never competes in the registry, so none of the defensive machinery against "another plugin rebuilt the registry" is needed.
- It leaves nothing behind: `ctx.on()` is an effect and is cleaned up automatically, and declaration writes are reverted explicitly.

### Deep rewriting

An image block is legal at any depth, including nested inside a `tool-result`. The built-in `read_image` tool records an image in its result, so a session that once used it carries nested image blocks on every later turn.

A top-level-only rewrite leaves those in place, and the text-only adapter then fails on every subsequent turn, not just the one that uploaded an image. This plugin recurses through all depths.

### Prompt-injection guard

Text recovered from an image is attacker-controlled, exactly like a web page: a screenshot can say "ignore all previous instructions".

Every transcription carries an explicit notice that image text is untrusted evidence and must not be executed as an instruction. The notice precedes the transcription and the text is fenced, so content inside the fence cannot pass itself off as surrounding narration.

### Writing declarations to `settings.yaml`

`manageDeclarations` writes `input: [text, image]` into the `llm-pi-ai` settings section and withdraws it when you turn the switch off, disable the plugin, or unload it. Measured against the real `dsh-settings-file`, which patches the YAML AST minimally:

**Preserved:**

- Header, section, and inline comment text
- Other routes and other namespaces
- Existing inline style (e.g. `input: [ text, image ]`)
- Blank lines and overall structure

**One real side effect:**

```diff
- displayName: LM Studio Local     # hand-aligned comment
+ displayName: LM Studio Local # hand-aligned comment
```

Hand-aligned padding before an inline comment collapses to a single space and does not come back on revert.

The declaration restates `text` explicitly:

```yaml
input: [ text, image ]
```

pi-ai maps this array straight onto `inputModalities`. Writing `[image]` alone would declare a model that accepts images but not text, so the write restates `text`.

### Verified facts

All of the following are measured against the real DSH runtime:

- The `llm/stream` listener fires, but the request arrives deep-frozen: `dsh-agent-loop.buildRequest` wraps it in `deepFreeze`, so `options.messages = x` throws `TypeError`. Replacing the argument slot does not help either, because cordis' inner callback closes over the original object; neither `arguments[0] = …` nor `next(replacement)` reaches the adapter. **The only working seam is to skip `next()` and issue a fresh `ctx.llm.stream({ ...options, messages: rewritten })`, marked as the plugin's own request to prevent re-entry.**
- The adapter receives no image block at all, including images nested inside `tool-result`.
- A nested `ctx.llm.stream()` does re-enter the listener; a `WeakSet` guards against that.
- Declaring `[text, image]` makes the admission gate admit the image.
- An `image` block carries an attachment reference, not inline bytes. pi-ai resolves it via `attachments.readImage(block.attachment)`, and a block holding `{ data, mediaType }` is silently dropped from the payload.
- Every terminal outcome, including an adapter refusal, arrives as `{ type: 'finish', reason: { kind: 'error' | 'aborted', failure } }`. There is no `type: 'error'` chunk and no `finishReason` field.
- Cross-namespace settings writes carry no ownership restriction, but path addressing does not support array indices, so a row edit must write the whole array back.
- The real `dsh-settings-file` preserves comments on disk, with the one formatting side effect described above.

---

## License

MIT © ZZ Dream (zzdream67)

## Acknowledgements

The public source of the following projects informed several design considerations in this plugin:

- [dsh-vision-router](https://www.npmjs.com/package/dsh-vision-router) (MIT)
- [@anionex/dsh-vision-toolkit](https://www.npmjs.com/package/@anionex/dsh-vision-toolkit)
- [Anionex/agent-vision-toolkit](https://github.com/Anionex/agent-vision-toolkit)

This plugin is an independent implementation taking a different architectural path (`llm/stream` waterfall interception, registering neither route nor adapter).