dsh-plugin-modality-fallback

Readme

# dsh-plugin-modality-fallback

English | [中文](README.zh.md)

A [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) (`dsh`) plugin. Route **one request** to a modality-capable fallback model instead of forcing the whole session onto a single model.

## The problem

A `dsh` session (Agent) selects one `provider`/`model` for its entire lifetime. That single model may not accept every modality that shows up in the session's history — today that means images. Without this plugin:

- The built-in `read_image` tool refuses outright when the session's model does not declare `image` input: *"switch to an image-capable model to read images."*
- `ApiProxy` refuses to send a message, or to switch models, when the session's history already contains an image the target model can't accept.

Every one of these paths tells the user to manually switch the **whole session** to an image-capable model and back — losing the differentiated model choice they made for everything else in that conversation.

## What this plugin does

It wraps `dsh`'s `agent/request` waterfall (the extension point `dsh-agent-default-model`'s own README documents as deferred: *"per-session selection remains the entry point's responsibility"*). Before a request goes out:

1. It reads the session's derived message history (`agent.session.deriveMessages()`).
2. If that history needs a modality beyond plain text (currently: `image`) and the model resolved by every other listener does not declare that modality (`llm.resolveModelInfo(...).inputModalities`), it looks up a configured fallback route for that modality.
3. If one is configured, it swaps `provider`/`model` for **that request only**. The session's own selection is untouched — the next request (once the image scrolls out of context, or the user switches models) resolves normally again.

No core `deepseek-harness` code is modified. This is an ordinary Cordis plugin, loaded alongside the rest of your `dsh` composition.

## Install

This package declares a `dsh.bundle` manifest, so `dsh plugin` installs and wires it into a profile in one step:

```sh
dsh plugin --profile web add dsh-plugin-modality-fallback
# or straight from GitHub, no npm publish needed:
dsh plugin --profile web add github:lilei0311/dsh-plugin-modality-fallback
```

That appends this package to the profile's `dsh.profile.bundles` and applies [`cordis.patch.yml`](cordis.patch.yml), which inserts the plugin row with an empty `fallback: {}` (no routes configured yet — every request behaves exactly as before). Configure a real route by restating that row's `config` in your own profile's or `$DSH_HOME`'s `cordis.patch.yml` (a patch replaces the whole `config`, so restate the id too):

```yaml
- id: modality-fallback
  config:
    fallback:
      image: { provider: deepseek-official, model: deepseek-vision }
```

`dsh --profile web --dump-config` shows the composed row so you can confirm it landed. See [deepseek-harness's plugin-install tutorial](https://github.com/deepseek-ai/deepseek-harness/blob/main/docs/user/develop/basic/publish.md) for the full bundle/profile mechanics this relies on.

### Programmatic use (embedding `dsh` yourself)

```ts
import ModalityFallback from 'dsh-plugin-modality-fallback'

await ctx.plugin(ModalityFallback, {
  fallback: {
    image: { provider: 'deepseek-official', model: 'deepseek-vision' },
  },
})
```

Load it after your `llm` and `agent`/`agent-loop` plugins so `ctx.llm` and the `agent/request` waterfall already exist.

## Known limitations

- **Only `image` is detected today.** The modality vocabulary (`ModelModality`) is open-ended, but this plugin's content check only walks for image blocks. Extending it to another modality means adding a content predicate, not changing the routing mechanism.
- **At most one missing modality is resolved per request.** If a future modality check finds more than one unmet modality at once, only the first is routed; the rest fall through unchanged.
- **`read_image` and `ApiProxy`'s own gates are unaffected.** Those refuse *before* a request is ever built, based only on the session's currently selected model, so they refuse even when this plugin has a working fallback configured for the very modality they're gating. Fixing that requires a change in `deepseek-harness` core itself (those gates would need to consult this plugin, or an equivalent capability, before refusing) — out of scope for a plugin that doesn't touch core.
- **Unknown model capability is treated as capable.** When `resolveModelInfo(...).inputModalities` is `undefined` (capability unknown), the plugin does not redirect — matching `ApiProxy`'s existing send/switch-model gates, not the stricter `read_image` gate (which refuses on unknown capability). A deployment that wants redirection on unknown capability too should have its adapter declare `inputModalities` explicitly.
- **A route switch drops the inherited reasoning effort** rather than forwarding one the fallback model may not support; the fallback route's own adapter/provider default applies instead.
- **A capability-probe failure fails open.** `resolveModelInfo` is adapter-owned I/O and can reject (network, an adapter returning invalid metadata, etc.). This plugin only ever *helps* route around a missing modality, so a probe failure logs a warning (`ctx.logger.warn`) and leaves the route unchanged rather than failing the whole request — even one whose already-resolved model didn't need a fallback in the first place. The probe itself is also skipped entirely when none of the request's needed modalities have a configured route (the default install, `fallback: {}`, never calls it at all), so this failure mode only matters once you've actually configured a route.

## Why a plugin, not a `deepseek-harness` PR

`deepseek-harness` is still at an early developer-preview stage and its `CONTRIBUTING.md` states the project does not accept external pull requests yet. Its own guidance for this situation is to build a plugin and share it — this repository does that, tagged with the `dsh-plugin` GitHub topic for discoverability.

## License

MIT