@deepseek-ai/dsh-read-image

Readme

# dsh-read-image

[简体中文](docs/README.zh.md)

> Plug-and-play image reading for text-only DeepSeek Harness models: pasted images are admitted, projected as `[Image #N]`, and read back through a first-class `read_image` tool backed by a configurable vision model — no preset changes required.

A [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) plugin that lets **non-multimodal models "see" images**.

- **Pasted images are no longer rejected** — text-only routes are declared to accept image input, so the api-proxy admission gate lets them through.
- **`[Image #N]` projection** — on text-only routes, image blocks in the model request are replaced in place with `[Image #N]` text; pixels never reach a text API. Native multimodal routes pass through untouched.
- **First-class `read_image` tool** (registered automatically in every session, shadowing the built-in tool of the same name):
  - `image_index` — read the Nth image in the conversation (`[Image #N]`);
  - `file_path` — read an image file from a path (PNG/JPEG/WebP/GIF);
  - `prompt` / `reasoning_effort` / `timeout_ms` / `max_tokens` / `max_thinking_tokens` — optional overrides; omitted parameters use the configured defaults (the current real defaults are interpolated into the tool description, so the agent never has to guess);
  - on text-only routes the configured **vision model** converts the image into a text description; on native multimodal routes the image itself is returned;
  - stateless and repeatable, including re-reading the same image.
  - **Native multimodal routes fall back to the built-in tool**: when the session's base route itself declares image input (e.g. `mimo-v2.5`), the custom tool is not registered and no `[Image #N]` prompt section is injected — the model sees real image blocks and uses the harness's built-in `read_image` (`file_path` only, "return the image itself").
- **Visual configuration** — a "Read image" section in the settings panel (gear icon in the sidebar) edits the vision model and defaults; the browser talks to the host over a typert Remote bridge (`readImageConfig`), working around the harness's settings allowlist for plugin namespaces.

## Installation

Requires DeepSeek Harness (`dsh`) `0.1.0-rc.6` or later (the harness is in developer preview; a newer release candidate may need a compatibility pass).

> **Windows**: the plugin runtime is pure Node.js and runs on Windows as-is — the harness natively handles Windows paths (drive letters, case-insensitive filesystems, PowerShell command execution). `dsh` is installed via npm as `dsh.cmd` (on your PATH), and `~/.dsh` maps to `%USERPROFILE%\.dsh` (e.g. `C:\Users\<you>\.dsh`). Every command below works unchanged; wherever a literal path is shown, substitute `%USERPROFILE%\.dsh\...` for `~/.dsh/...`.

### Recommended: `dsh plugin add`

```sh
dsh plugin --profile web add github:OoWJZZoO/dsh-read-image
```

Then restart `dsh web`. This package ships a `cordis.patch.yml` via its `dsh.bundle` manifest, so the profile bundle mechanism composes the plugin row automatically — no manual patch editing.

### Manual install

Add the package to the web profile and register the plugin row yourself:

```json
// ~/.dsh/profiles/web/package.json → dependencies
"@deepseek-ai/dsh-read-image": "github:OoWJZZoO/dsh-read-image#v0.1.0"
```

```sh
cd ~/.dsh/profiles/web && pnpm install
```

Insert one row into `~/.dsh/profiles/web/cordis.patch.yml`:

```yaml
- insert:
    - id: read-image
      name: '@deepseek-ai/dsh-read-image'
      config: {}   # optional: deployment defaults (user layer can override)
```

Then restart `dsh web`.

> **Do not combine the two paths**: `dsh plugin add` already composes the plugin row, so adding the dependency manually *and* inserting the row would register the plugin twice.

## Configuration

Configuration lives in `$DSH_HOME/settings.yaml` (hot-reloaded, no restart) — on Windows, `%USERPROFILE%\.dsh\settings.yaml` when `DSH_HOME` is unset — and can also be edited in Web at **Settings → Read image** (writes to the user layer, overriding the corresponding settings.yaml keys):

```yaml
# 1) Declare image input for your vision model (pi-ai route)
llm-pi-ai:
  providers:
    <your-provider>:
      models:
        - id: <your-vision-model>
          # …contextWindow / maxTokens…
          input: [text, image]      # ← required

# 2) This plugin's configuration
dsh-read-image:
  visionProvider: <your-provider>   # the route provider hosting the vision model
  visionModel: <your-vision-model>  # the multimodal model id (must declare input: [text, image])
```

| Key | Default | Description |
| --- | --- | --- |
| `visionProvider` | `''` | Route provider of the vision model (dropdown fed by the Models page) |
| `visionModel` | `''` | The multimodal model that performs read_image tasks (dropdown under that provider) |
| `defaultPrompt` | English step-by-step description prompt (classify → verbatim text to Markdown / visual description) | Used when `read_image` is called without `prompt` |
| `defaultReasoningEffort` | `low` | Default reasoning intensity. `low` is the lowest tier widely accepted and honored; on many adapters `off` is equivalent to "omit the field" and does not disable thinking on models that default it on, so thinking eats into `max_tokens` and small budgets come back empty |
| `defaultTimeoutMs` | `300000` | Default vision-call timeout (5 minutes, generous) |
| `defaultMaxThinkingTokens` | `4096` | Default max thinking tokens (separate budget, not counted against output); when thinking exceeds the budget and the output comes back empty, `read_image` reports an explicit error instead of silently returning nothing |
| `defaultMaxTokens` | `8192` | Default **actual output** token cap; the API `max_tokens` sent is this + `defaultMaxThinkingTokens` (budget is 0 with `reasoning_effort=off`, passed through unchanged) |
| `guard.enabled` | `true` | Environment self-check switch; `false` skips the self-check and force-loads the plugin (see [Safety](#safety)) |

> Why not the settings protocol? The harness allowlists `WEB_SETTINGS_NAMESPACES`, so a plugin `settings.register()` namespace only answers `settings-not-exposed` to the browser. The "Read image" page therefore talks to a host-side typert Remote bridge (`readImageConfig.get/set`, see `lib/config-remote.js`) instead — settings.yaml stays the base layer, Web writes go to the user layer, `scope.watch` hot-syncs the runtime, and headless and Web stay consistent.

## Usage

- Paste an image: the text model sees `[Image #1]`; call `read_image image_index=1` to view it.
- `read_image file_path=/path/to/image.png` reads an image file (on Windows, both `C:\Users\...` and `C:/Users/...` spellings work — backslashes are handled by the harness file service).
- Pass `prompt` (a specific question for the vision model), `reasoning_effort`, `timeout_ms`, `max_tokens`, `max_thinking_tokens` to override the defaults.
- Call it repeatedly, including re-reading the same image.

## Safety

This plugin rides on harness internals that can change shape between releases. On startup it runs an environment self-check over every harness contract it touches. If any check fails, the plugin **fails safe**: it loads nothing and the harness boots normally — the full diagnostics are written to `~/.dsh/logs/dsh-read-image-guard.log` (on Windows: `%USERPROFILE%\.dsh\logs\dsh-read-image-guard.log`) and one short notice is logged. To force-load anyway (at your own risk): `guard.enabled: false` under the `dsh-read-image` section of settings.yaml.

## How it works

| Mechanism | Layer | What it does |
| --- | --- | --- |
| M1 admission | host | Wraps `llm.resolveModelInfo` so text-only routes report image input (restored on dispose) → send/switch-model gates let images through |
| M2 projection | host | Synchronous `llm/stream` waterfall listener: on text-only routes, image blocks become `[Image #N]` text, then re-dispatches the harness-`deepFreeze`d request |
| M3 tool | per-agent | Registers `read_image` into the agent's own scope at `session/created` (shadowing the built-in tool) plus a prompt section |
| Settings | host | Registers the `dsh-read-image` settings namespace (schema defaults = fallback), hot-syncs runtime config via `scope.watch` |
| Config bridge | host | `ReadImageConfigService` (typert Remote, `readImageConfig.get/set`) reads/writes the same namespace; auto-discovered by the gateway |
| Config page | browser | `settings.section` registers the "Read image" page (order 12); `ctx.remote.$mount` self-mounts `remote.readImageConfig` |
| G1 self-check | host + browser | Total fuse: probes every harness contract at apply; any failure → plugin loads nothing, harness unaffected (see [Safety](#safety)) |

- Capability truth table: built from the **original** `resolveModelInfo` (never the M1-wrapped one), rebuilt on `llm/adapters-updated` with startup retries;
- Vision calls reuse `ctx.llm` (same credentials / retries / logging as the harness), with `AbortSignal.any([exec.signal, timeout])` for timeouts;
- Session logs are the single source of truth: image references persist as usual, and only the model-visible boundary performs the `[Image #N]` replacement.

## Development

Contributor guide and the full list of harness-workaround hacks (and the reasons they exist) live in `AGENTS.md` and `docs/known-hacks.md`. Tests: `node --test test/` (run from the dev-profile copy — see the test file headers).

The plugin code itself is cross-platform (pure Node.js, no OS-specific APIs). The dev/deploy scripts (`scripts/*.sh`) are POSIX bash — on Windows run them under Git Bash / WSL / MSYS2, or mirror the copy steps by hand (they only copy `package.json`, `lib/` and `cordis.patch.yml` into the profile's `node_modules/@deepseek-ai/dsh-read-image`).

## License

MIT