dsh-plugin-multimodal

Readme

# dsh-plugin-multimodal

[English](README.md) | [中文](README.zh.md)

Image recognition and image generation for [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) (`dsh-plugin`, a Cordis plugin).

**The plugin is just tools**: `image_recognize`, `image_generate`, and `vision_providers` call whichever external API models the deployer declares — vision via multimodal `chat/completions`; image generation via OpenAI-compatible `images/generations` or the Aliyun DashScope native async-task protocol. **No model is bundled or defaulted** — a tool invoked without a configured model fails with a clear, actionable error. It never guesses and never fails silently.

## Demo

| Image recognition | Image generation |
|---|---|
| ![Image recognition demo](./data/bfd22e009098d2d8d9d1788fb88e62cb.png) | ![Image generation demo](./data/d2d795fc9b2dd76a4f5a419ebb3c44af.png) |

## Features

- **`image_recognize`** — analyze an image (local path / http(s) URL / data URI) through a configured vision model: describe, OCR, read charts/screenshots, review content, …
  - Local files are inlined as base64 data URIs (25 MiB cap)
  - `prompt` (what to ask), `max_tokens` / `temperature`, per-call `provider` selection
- **`image_generate`** — generate or transform an image through a configured image model, save it to disk, and return its file path plus a Web-renderable URL when the static server is enabled
  - **Text-to-image**: OpenAI-compatible `images/generations` (prefers `b64_json`, falls back to URL download), or `protocol: dashscope-native` for the DashScope async task API (submit → poll → download), needed when an OpenAI-compatible gateway exposes no image routes (some Aliyun deployments)
  - **Image-to-image**: pass an `image` argument (source image) to generate from it (edit / variant / style transfer). OpenAI-compatible endpoints use the `image` field; DashScope native uses `base_image_url`
  - **Watermark-free by default**: OpenAI-compatible requests carry `watermark: false` (Doubao Seedream etc. support it; endpoints rejecting the param are retried without it). Set `watermark: true` on the provider to keep the vendor's mark
  - **Inline in the Web chat**: a built-in static server (default 127.0.0.1:3081) exposes generated images as absolute http(s) URLs; the tool returns Markdown `![](url)` lines the model can paste into its reply so the GUI renders them (the GUI's Markdown renderer allows only absolute http(s) image URLs — local paths, relative links and data URIs never render; the harness webserver serves no arbitrary files, and writing into the frontend dist directory does not work, tested)
  - **Real-format extensions**: base64 payloads get their extension from the magic bytes (PNG/JPEG/WebP/GIF) so Content-Type matches the content
  - `size` (e.g. `1024x1024`), `n` (1–4, auto-suffixed filenames), `output_path` (directory or file)
  - Default save location: `<caller workspace>/generated/`, timestamped filenames
- **`vision_providers`** — list every configured external model (id, kind, model, endpoint, apiKey status) so the model can pick a provider id
- **Multiple models** — the `providers` list accepts any number of entries; each tool call may select one via `provider`, defaulting to the first of the matching kind
- **Not configured → clear error** — no provider of the kind, unknown provider id, empty `apiBase`, `model`, or `apiKey` all raise actionable Chinese errors when the tool is called; missing connection settings never prevent DSH Web from starting

## Install into a DSH profile

The DSH plugin command delegates installation to `pnpm`, so `pnpm` must be on
`PATH` regardless of how DSH itself is launched. Build this local checkout
before installing it (the package entry point is `lib/index.js`):

```sh
cd C:/projects/dsh-harness/dsh-multimodal
pnpm install
pnpm run build
```

Then use the command that matches how you run DSH:

```sh
# DSH run through npx (no global `dsh` command)
npx @deepseek-ai/dsh plugin --profile web add file:C:/projects/dsh-harness/dsh-multimodal

# DSH run from a deepseek-harness source checkout (run from that repo root)
pnpm dsh plugin --profile web add file:C:/projects/dsh-harness/dsh-multimodal

# DSH installed with a global `dsh` command
dsh plugin --profile web add file:C:/projects/dsh-harness/dsh-multimodal
```

The package also has a `prepare` lifecycle script so Git-hosted installs build
before use. A local `file:` checkout should still be built explicitly before
`add`, which makes the required runtime artifact easy to verify.

Then restart the Web profile with the same launcher (`npx @deepseek-ai/dsh web`,
`pnpm dsh web`, or `dsh web`) and open a **new session** (tool sets are fixed
per session). The in-package `cordis.patch.yml` is the deployment config source
(in-package config overrides outer layers; `settings.yaml` and user profile
patches only fill undeclared keys).

> **⚠️ Applying config changes (important)**: the web profile uses `nodeLinker: hoisted`, so a `file:` plugin is **copied** into the profile (not symlinked). After editing `cordis.patch.yml` you must resync:
> 1. Delete `<DSH_HOME>\profiles\web\node_modules\dsh-plugin-multimodal` (the snapshot dir)
> 2. Run `<your DSH launcher> plugin --profile web install`
> 3. Restart the Web profile with the same launcher
>
> Re-running `dsh plugin add` alone will **not** recopy existing content; deleting the snapshot first is what makes `install` reliable.

## Usage

### 1. Configure external API models (required)

Fill in `apiBase`, `apiKey`, and `model` in the `providers` list of the in-package `cordis.patch.yml` (**none of these connection values has a working default**). The plugin and DSH Web still start while they are empty; validation happens when the corresponding tool is called. The protocol is OpenAI-compatible, so any vendor works:

```yaml
providers:
  # Image recognition: multimodal chat/completions
  - id: my-vision
    kind: vision
    apiBase: https://api.openai.com/v1        # or https://api.siliconflow.cn/v1 etc.
    apiKey: sk-xxxx                            # your API key
    model: gpt-4o                              # or Qwen/Qwen2.5-VL-72B-Instruct etc.
    # maxTokens: 1024      # optional
    # temperature: 0.2     # optional

  # Image generation: ONE provider covers text-to-image AND image-to-image
  # (Volcengine Ark Doubao Seedream uses this protocol too)
  - id: my-image
    kind: image
    apiBase: https://api.openai.com/v1        # or https://ark.cn-beijing.volces.com/api/v3
    apiKey: sk-xxxx
    model: gpt-image-1                         # or doubao-seedream-5-0-pro-260628 etc.
    # i2iModel: doubao-seedream-3-0-i2i-250528 # optional: i2i model used when an
    #                                          #   image arg is passed; omit when the
    #                                          #   model handles both (e.g. Seedream 5.0 Pro)
    # watermark: false      # default false (watermark-free), no need to set
    # size: 1024x1024       # optional default size

  # Image generation: Aliyun DashScope native protocol (use when the
  # compatible-mode gateway exposes no image routes; tested: wanx2.1-t2i-turbo
  # is watermark-free). Aliyun t2i/i2i are separate models → i2iModel.
  - id: my-image-wanx
    kind: image
    protocol: dashscope-native
    apiBase: https://dashscope.aliyuncs.com
    apiKey: sk-xxxx
    model: wanx2.1-t2i-turbo
    i2iModel: wanx-v1                          # i2i model (input image must be a public http(s) URL)
    size: '1024*1024'      # DashScope sizes use an asterisk
```

`id` must be unique; the same endpoint may host several models under different ids; vision and image generation may point at different vendors. `image_generate` does **not** distinguish t2i/i2i: passing an `image` argument switches to `i2iModel` (or the same `model` when unset); omitting it means text-to-image.

### 2. Verify the plugin is active

Open a **new session** and ask: *"Do you have image_recognize / image_generate tools?"* — or have the model run `vision_providers` to list the configured models.

### 3. What the model can do

- **Recognize** — *"识别这张图 C:\path\to\photo.png 里有什么"* → `image_recognize`
- **Text-to-image** — *"画一只戴帽子的橘猫，保存到 output 目录"* → `image_generate` returns the saved file path
- **Image-to-image** — *"把这张图 https://.../a.png 里的猫变成蓝色（用 my-image-wanx）"* → `image_generate` with `image` + `provider`
- **Inspect** — *"你现在能用哪些图像模型？"* → `vision_providers`

### When nothing is configured

DSH Web and the plugin remain available. Only the invoked tool fails, e.g.:

```
multimodal: provider "my-vision" 未配置 apiBase。请在插件包内 cordis.patch.yml 的 providers 中填写 OpenAI 兼容端点，然后重启 Web profile。
```

## Configuration

| Key | Default | Meaning |
|---|---|---|
| `providers` | `[]` at schema level; the bundled patch contains two empty editable examples | External API models. Each entry: `id` (unique), `kind` (`vision` / `image`), `apiBase`, `apiKey`, and `model` (required when calling the provider), `protocol` (image generation only: `openai` = images/generations (default); `dashscope-native` = Aliyun DashScope async task API), optional `i2iModel` / `watermark` (default `false`) / `size` / `maxTokens` / `temperature` |
| `outputDir` | empty (= `<caller workspace>/generated`) | Default directory for generated images; relative paths resolve against the caller workspace, absolute paths are used as-is |
| `staticPort` | `3081` | Built-in static image server port (127.0.0.1); generated images are exposed at `http://127.0.0.1:<port>/<file>` for the Web GUI to render via markdown. `0` disables it (paths only); port conflicts degrade gracefully |

`image_recognize` args: `image` (required: path / URL / data URI), `prompt`, `provider`, `max_tokens`, `temperature`.
`image_generate` args: `prompt` (required), `image` (optional: image-to-image source, path / URL / data URI), `provider`, `size`, `n` (1–4), `output_path` (directory ends with `/` or `\`).

## Development

```sh
npm install --cache ./.npm-cache   # build & test deps
npm run build                      # tsc → lib/
npm test                           # unit tests (stubbed API, no network)
node scripts/verify.mjs            # mount check: loads, tools register, error contract
```

Peer dependencies (`@deepseek-ai/cordis`, `@deepseek-ai/dsh-tools`, and `@deepseek-ai/schemastery`) are also development dependencies for local builds and tests. An installed plugin consumes the compatible peer versions supplied by its DSH profile/runtime.

## Known limitations

- **25 MiB inline cap for local images** — compress or use a URL for larger files
- **Generated-image format detection** — PNG/JPEG/WebP/GIF base64 payloads use their detected extension; unknown base64 formats fall back to PNG, while downloaded URL results use the response Content-Type
- **DashScope image-to-image needs a public image URL** — `base_image_url` rejects local paths and data URIs (tested). Upload the source image to a public location (Alibaba Cloud OSS / any image host) first and pass the URL. Some models (e.g. `wanx2.1-imageedit`) are pickier about URLs — `wanx-v1` is the most reliable for i2i (tested)
- **Old sessions don't see new tools** — open a new session after install/upgrade
- **Non-standard compatible endpoints** — the plugin retries once without `response_format`; if the endpoint still errors, the remote message is surfaced

## License

MIT