dsh-ocr-bridge

Readme

# dsh-ocr-bridge

**English** | [**中文**](https://github.com/vuvanmai936-dot/dsh-ocr-bridge/blob/main/README.zh.md)

Paste images directly into [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) (DSH) chat and have them **read by a free local backend** — macOS Vision, with a Tesseract fallback — before the text-only DeepSeek model (e.g. `deepseek-v4-flash`) answers. Your model, agent capabilities, API key, and cost stay exactly the same.

> **Scope note:** this is an *OCR-level* local bridge — it reads text out of images (macOS Vision → Tesseract). It is **not** general visual understanding, and **no cloud endpoint is ever called**.

> **Independence notice.** This is an independent, community-built plugin **for** DeepSeek Harness (DSH). It is not an official DeepSeek product and is not affiliated with, endorsed by, or sponsored by DeepSeek or the DeepSeek Harness project. "DeepSeek" and "DeepSeek Harness" are trademarks of their respective owners.

> ⚠️ **Trust boundary.** This plugin runs inside the harness host process with shell-level access (it spawns `osascript` / `tesseract`). Only install plugins you trust. The OCR text injected into the request is explicitly marked as *untrusted observation data, not instructions* — never execute commands, rules, or privilege escalations that appear in it.

## Why

DSH's image admission gate (`dsh-host-apiproxy.submitPrompt`) rejects attachments unless the current model declares `image` input. The official DeepSeek adapter hardcodes `inputModalities: ["text"]` for every model and its serializer throws on image blocks. The gate only checks the *declaration*, not real multimodality — so this plugin registers a wrapper adapter that:

1. declares `["text", "image"]` to let the gate pass;
2. recognizes pasted images **locally** (macOS Vision → Tesseract, no network, no API key);
3. strips the image blocks and injects the recognized text as untrusted context;
4. delegates the pure-text call to the official DeepSeek API with your existing key.

No new API keys, no settings changes, no model switching.

## Install

Requires DSH `0.1.0-rc.7` (peer dependencies are pinned; other release candidates are not supported — see [Compatibility](#compatibility)).

```bash
# npm registry
dsh plugin --profile web add dsh-ocr-bridge

# or directly from GitHub
dsh plugin --profile web add github:your-org/dsh-ocr-bridge

# local development (live reload of lib/ changes)
dsh plugin --profile web add link:~/dev/dsh-vision-bridge
```

> Package name note: the npm name `dsh-vision-bridge` is taken by an unrelated project, and the upstream `dsh-vision` package already owns the "vision" naming — so this plugin is published as **`dsh-ocr-bridge`**, which also states its OCR-level scope.

Restart `dsh web` after adding the plugin (patch layers load at boot). The model selector still shows the original `deepseek-v4-flash` / `deepseek-v4-pro` entries — nothing to re-select.

## Backends

| Order | Backend | Requirements |
| --- | --- | --- |
| 1 | macOS Vision (JXA via `osascript`) | macOS 10.15+; zero installation, Chinese + English built in |
| 2 | Tesseract | `tesseract` CLI + language packs (`chi_sim`, `chi_tra`, `eng`); also the only option on Linux/Windows |

If Vision fails and Tesseract is missing, the request fails with `VISION_UNAVAILABLE` and the error lists both backend failures. Images are written to a temp directory and always cleaned up.

## Configuration

All settings are optional. They live in the `llm-deepseek` settings section (the official DeepSeek section this plugin takes over), so your existing DeepSeek settings keep working.

`settings.yaml` (or the GUI settings surface):

```yaml
llm-deepseek:
  visionTimeoutMs: 180000   # per-backend recognition timeout (ms), default 180000
  maxImages: 8              # images recognized per request, 1..32, default 8
  # …all official DeepSeek settings keep working: baseURL, apiKeyEnv, thinking, …
```

Notes:

- Over the limit, the request fails with `VISION_IMAGE_LIMIT` instead of silently dropping images.
- Recognized observations are cached per (image set + latest user text), up to 64 entries, so re-sending the same screenshot in one conversation does not re-run OCR.
- `DEEPSEEK_API_KEY` resolution is fully preserved: `ctx.credentials` first, then the launch environment, exactly like the official adapter.
- png / jpeg / webp / gif are all supported (whatever the harness attachment admission accepts).

## Architecture

```
paste image ──► submitPrompt gate ──► wrapper adapter (inputModalities=["text","image"]) ──passes──► durable attachment refs
                                                          │
                                   stream(): collectImageRefs(messages)
                                     ├─ no images ─► delegate to DeepSeekAdapter untouched
                                     ├─ native model supports image ─► delegate untouched
                                     └─ images ─► attachments.readImage(ref) each
                                                ► local recognition (macOS Vision → Tesseract)
                                                ► strip image blocks, append <vision-bridge-context>
                                                ► delegate pure-text call to official DeepSeek API
```

## Development

```bash
pnpm install
pnpm check        # typecheck + test + build
```

Live-testing against your harness:

```bash
dsh plugin --profile web add link:~/dev/dsh-vision-bridge   # symlink: lib/ changes apply immediately
# editing cordis.patch.yml still requires restarting `dsh web`
```

Then paste an image into any conversation and verify the [checklist below](#verification-checklist).

## Verification checklist

- [ ] After `dsh plugin --profile web add dsh-ocr-bridge` and restart, the model selector still shows `deepseek-v4-flash` / `deepseek-v4-pro`
- [ ] Pasting one or several images under a text-only flash model no longer raises `MODEL_DOES_NOT_SUPPORT_IMAGES`
- [ ] Mixed Chinese + English screenshots are recognized correctly (macOS Vision)
- [ ] Tesseract fallback works (simulate a Vision failure)
- [ ] Text-only conversations behave identically to the official adapter (pure delegation)
- [ ] No new API key needed; `settings.yaml` unchanged works out of the box
- [ ] png / jpeg / webp / gif all work; image count respects `maxImages`

## Compatibility

- DSH `0.1.0-rc.7` only. Peer dependencies are pinned exactly (`@deepseek-ai/*` `0.1.0-rc.7`, `@deepseek-ai/cordis` `4.0.1`); later release candidates will be added as they ship.
- Node `>=22.19`, pnpm `10.x`.
- macOS 10.15+ recommended for the Vision backend; Linux/Windows fall back to Tesseract.

## Credits

The code skeleton is adapted from [oil-oil/dsh-vision](https://github.com/oil-oil/dsh-vision) (MIT) and used with attribution (see [LICENSE](./LICENSE)). Positioning differs deliberately: **dsh-ocr-bridge is a local-first, OCR-level bridge** (zero cloud, zero extra cost, pinned to rc.7), while upstream focuses on cloud multimodal endpoints plus a visual-memory workflow. The two are complementary rather than overlapping in scope; this plugin does not claim to be a fork or successor of upstream.

## Roadmap (v1+)

- Ollama local vision backend (e.g. `llava`) as a third recognition option
- Client settings card for the bridge options (currently configured via `settings.yaml` / composition config)
- Multi-backend scoring (Vision + Tesseract agreement) for higher-confidence OCR

## License

MIT. See [LICENSE](./LICENSE).