@deepseek-ai/dsh-vision-bridge

Readme

# @deepseek-ai/dsh-vision-bridge

English | [中文](README.zh.md)

Host plugin and Web settings page for routing image-bearing messages to a user-configured vision model when the active text model cannot accept image input.

## Install

```bash
./install.sh            # installs into the default `web` profile
DSH_PROFILE=myprofile ./install.sh
```

The installer registers this repository as a profile bundle with `dsh plugin`, then
applies the small `dsh-host-apiproxy` compatibility patch this plugin needs on
dsh `0.1.0-rc.x`: exposing the `vision-bridge` settings namespace and admitting
new image prompts while the bridge is enabled. Restart `dsh web` after
installing.

## How it works

The bridge listens on the `agent/pre-step` waterfall. Before the loop appends claimed messages to the session log, it inspects each message for `image` content blocks:

1. If the active provider/model reports `image` input modality, the message passes through unchanged.
2. If the model explicitly reports text-only input and the settings namespace names a configured endpoint, the bridge:
   - asks the active text model to write one precise vision prompt from the accompanying message text;
   - assembles the same scoped system prompt the loop would use;
   - POSTs `system prompt + written prompt + image data` to the configured OpenAI-compatible endpoint;
   - replaces the image blocks with the returned text in a fresh user message.
3. The loop logs that text-only message, so the transcript remains reconstructable and the text model never receives raw image bytes.

## Settings

The Web Settings page adds a **Vision** section under the `vision-bridge` settings namespace:

| Field | Default | Purpose |
| --- | --- | --- |
| `enabled` | `false` | Whether image bridging runs at all |
| `url` | `''` | OpenAI-compatible API base URL, or a full URL ending in `/chat/completions` |
| `apiKey` | `''` | Bearer API key; `role('secret')`, redacted on the wire |
| `model` | `''` | Vision model id for the configured endpoint |
| `timeoutMs` | `60000` | Per-request timeout |

The plugin's cordis.yml `config` supplies the settings composition `base`, so a deployment can set defaults without writing the user document. Bridging requires all of `enabled`, `url`, `apiKey`, and `model`.

## Endpoint protocol

`POST <url>/chat/completions` with `Authorization: Bearer <apiKey>` and an OpenAI chat-completions body:

```json
{
  "model": "<settings.model>",
  "messages": [
    { "role": "system", "content": "<assembled harness system prompt>" },
    {
      "role": "user",
      "content": [
        { "type": "text", "text": "<text-model-written prompt>" },
        { "type": "image_url", "image_url": { "url": "data:<mediaType>;base64,<bytes>" } }
      ]
    }
  ]
}
```

The response must be a chat completion whose first choice carries a string or text-part-array `message.content`.

## Model Experience

### Bridged conversation

#### What the model sees

When bridging activates, the text model sees the original message text plus one appended text block: `[image analysis from vision model <model>]` followed by the vision model's answer. The image block itself is removed. The text model's prompt-writing auxiliary call uses a fixed instruction and `temperature: 0`; it carries no session id.

#### Token effect

The original image blocks contribute no tokens to the text model. The appended vision answer contributes its full text token count, plus the fixed marker line and the prompt-writing call's input/output tokens.

#### KV Cache effect

System and tool sections are unchanged. The text-model prompt-writing call is a separate request with a fixed system prompt and does not reuse the conversation prefix. A changed vision endpoint, model, or enable state changes future conversation content and therefore future prefix caches.

## Known Limitations and Deferred Work

- Bridging triggers only when the active adapter's `resolveModel()` explicitly omits `image` from `inputModalities`. Models with unknown modality metadata pass through untouched.
- The `agent/request` waterfall may replace the provider/model after pre-step; this plugin reads the agent's declared options and therefore does not follow such a late model switch.
- Images that already reached the session log through another path (for example a custom tool result) are not rewritten; only claimed pre-step inbox messages are bridged.
- Only OpenAI-compatible `/chat/completions` endpoints are supported.
- The settings page keeps the saved API key hidden and treats a blank key field as "keep the saved key"; clearing a stored key requires the raw settings document.