@launchmaniac/dsh-media-tools

Readme

# dsh-media-tools

Image, video, and speech generation as [deepseek-harness](https://github.com/deepseek-ai/deepseek-harness) tools, backed by OpenRouter. Ships as an out-of-tree profile bundle — no harness fork, no patched core.

## What it registers

| Tool | Backing endpoint | Result reaches the model as |
|---|---|---|
| `generate_image` | `POST /images` | **the image itself** — committed to the attachment store, returned as an image block. Also written to `outputDir` so a person can open it. |
| `generate_video` | `POST /videos` + polling | a **file path** |
| `generate_speech` | `POST /audio/speech` | a **file path**, plus a play link |
| `transcribe_audio` | `POST /audio/transcriptions` | **the transcript text** — the only tool here that runs inward |

That split is the whole design, and it is not arbitrary. The harness content vocabulary (`ContentBlockMap`) has `text`, `reasoning`, `image`, `tool-call`, and `tool-result` — no video, no audio — and a model route may declare only `text` and `image` input. So a generated image can re-enter model context and a generated video cannot. Adding a video or audio block would require changing the harness itself plus its adapter, UI, and compaction consumers, which no out-of-tree bundle can do.

The practical consequence: **the model never sees the video or hears the audio.** It learns the path and the byte count. The rendered tool result stays strictly factual for that reason — the model has no way to check any claim about how an artifact looks or sounds.

Generated images **do** appear inline in the Web UI, by a route worth understanding before you rely on it. No harness surface renders a tool-result image — the card shows the durable reference, not pixels — and the client's markdown renderer accepts images only from `http(s)`, refusing `data:` URIs and `file://` paths. So this bundle serves each written image on the harness web server behind an unguessable process-local token, and the tool result hands the model that URL with an instruction to include it in its reply.

That last step is the caveat: **display depends on the model following that instruction.** A model that ignores it leaves you with the file path and no picture. Every image is therefore also written to `outputDir` with a real extension, and `revealImages: true` opens each one in your desktop viewer as it is produced — those two paths do not depend on model behaviour.

## Install

```sh
dsh plugin --profile <name> add @launchmaniac/dsh-media-tools
dsh --profile <name> --dump-config   # verify the layer without booting
```

That is the recommended route: the npm tarball ships `lib/` already built, so
nothing executes on your machine at install time and no build allowance is
involved.

### Installing from GitHub instead

A git install fetches **sources, not built artifacts**, so pnpm runs this
package's `prepare` script to build `lib/`. pnpm ≥10 refuses to run a git
dependency's build script until you allow it, so the first `add` fails with the
package key it wants. Add it to your profile's `pnpm-workspace.yaml`:

```yaml
allowBuilds:
  "@launchmaniac/dsh-media-tools": true
```

then re-run the `add`.

npm v12 gates git dependencies separately from build scripts — installing one
also needs `--allow-git`, and a remote URL dependency needs `--allow-remote`.
`npm approve-scripts --allow-scripts-pending` reviews what is waiting. This is
another reason to prefer the npm install above: it is a plain registry
dependency and trips none of these.

Treat that allowance as what it is: **permission to execute this package's code
on your machine at install time**, outside any sandbox the agent runs under.
Read the source first, and pin a commit so a later push cannot silently change
what runs:

```sh
dsh plugin --profile <name> add github:launchmaniac/dsh-media-tools#<sha>
```

If you would rather not grant a build allowance, `pnpm pack` a tarball from a
checkout and `dsh plugin --profile <name> add ./launchmaniac-dsh-media-tools-0.1.0.tgz` —
prebuilt, so no build permission is involved.

Set your key:

```sh
export OPENROUTER_API_KEY=sk-or-...
```

The key is read through the harness credential seam (`ctx.credentials`) when one is mounted, falling back to the process environment. It is resolved per call and never cached, so a rotated key takes effect on the next tool invocation.

## Configuration

Three places, in increasing precedence.

**1. Per call, in the conversation.** `generate_image` takes an optional `model`, so you can say *"generate a logo with `recraft/recraft-v4-vector`"* and it applies to that call only.

**2. `$DSH_HOME/settings.yaml`** under a `media-tools` key. This is the live one — a change takes effect on the next tool call, with no restart:

```yaml
media-tools:
  imageModel: black-forest-labs/flux.2-pro
```

The Settings dialog's **Open configuration file** button reaches this file. The bundle registers its schema through `installSettingsSection`, so the values are validated on write. It does not yet appear as a card in **Settings → Plugins**: that needs a browser half declared with `dsh.client`, and the build preset for client bundles is not published outside the harness monorepo.

**3. Your profile's `cordis.patch.yml`** — the deployment default everything else layers over:

```yaml
- id: media-tools
  name: "@launchmaniac/dsh-media-tools"
  config:
    imageModel: bytedance-seed/seedream-4.5
    videoModel: bytedance/seedance-2.5
    # A voice-requiring model must be given one, or every call 400s.
    speechModel: google/gemini-3.1-flash-tts-preview
    speechVoice: Kore
    outputDir: ./media
    videoTimeoutMs: 900000
```

| Key | Default | Meaning |
|---|---|---|
| `apiKeyEnv` | `OPENROUTER_API_KEY` | Environment variable holding the key |
| `baseURL` | `https://openrouter.ai/api/v1` | API root |
| `attributionTitle` | `deepseek-harness` | Sent as `X-Title` |
| `attributionUrl` | — | Sent as `HTTP-Referer` |
| `imageModel` | `bytedance-seed/seedream-4.5` | Used by `generate_image` |
| `videoModel` | `bytedance/seedance-2.5` | Used by `generate_video` |
| `speechModel` | `fish-audio/s2.1-pro` | Used by `generate_speech` |
| `speechVoice` | — | Default voice, overridable per call |
| `transcribeModel` | `openai/whisper-1` | Used by `transcribe_audio`; `deepgram/nova-3` and `openai/gpt-4o-transcribe` also work |
| `maxTranscribeBytes` | `26214400` | Largest audio file sent for transcription |
| `outputDir` | `./media` | Where artifacts are written. **Relative paths resolve against the session's working directory**, so each workspace keeps its own; an absolute path pools everything in one place |
| `revealImages` | `false` | Open each generated image in the desktop viewer |
| `requestTimeoutMs` | `120000` | Per-HTTP-request budget |
| `videoPollIntervalMs` | `5000` | Delay between video job polls |
| `videoTimeoutMs` | `900000` | Total wait for one video job |

A patch replaces a row's entire `config` value rather than merging keys, so an override must restate every key it needs.

## Requirements and refusals

`generate_image` refuses, rather than degrading, in two cases:

- **No attachment store mounted.** The tool is registered inside `ctx.inject(['attachments'], …)`, so it does not exist without one. `@deepseek-ai/dsh-attachment-local` ships in `@deepseek-ai/dsh-base`, so a normal profile has it.
- **The calling model does not declare `image` input.** Generating an image the current route cannot carry would put a block into durable session history that the route cannot replay. The refusal names the model and tells you to switch.

Both checks run before any HTTP request, so a refusal never spends a call.

## Audio in: transcription

`transcribe_audio` is the one tool that runs inward. No chat route accepts audio
input — `ModelModality` is text and image only — so a transcript is how recorded
speech reaches a model at all. Point it at a file and the model reads what was
said.

Three models are served: `openai/whisper-1` (default), `deepgram/nova-3`, and
`openai/gpt-4o-transcribe`. None appear in the ordinary `/models` listing, the
same way the TTS models do not.

This is turn-based, not a conversation. Speaking to the agent live would need
microphone capture in the browser, which a plugin cannot reach — that is a
harness-side change.

## Text to speech: voices and containers

Two things about TTS are worth knowing before you configure it, both learned by
running the endpoint rather than reading its docs:

- **Most providers require an explicit `voice`, and voice names are not
  portable.** Gemini TTS rejects a voiceless request and takes names like
  `Kore` or `Puck` (`alloy` returns a 500). Deepgram takes `flux-*-en` names
  and lists the valid set in its error. `fish-audio/s2.1-pro` synthesizes with
  no voice at all, which is why it is the shipped default — the tool works out
  of the box, and you opt into a voice-requiring model deliberately.
- **Raw PCM is re-containered as WAV.** Both verified providers return
  `audio/pcm` with the sample rate only in the media-type parameters. Written
  as-is those bytes are unplayable, so `generate_speech` prepends a RIFF/WAVE
  header (16-bit little-endian, rate and channels taken from the media type)
  and writes `.wav`. Non-PCM audio passes through untouched.

## Choosing models

`GET https://openrouter.ai/api/v1/images/models` and `.../videos/models` list what is currently available for each endpoint. The default `/api/v1/models` listing returns text-output models only, which is why generation models do not appear there.

## Known limitations

- **Video and audio writes bypass the `ctx.fs` seam.** Artifacts are written with `node:fs` straight into `outputDir`, so they are not subject to the session filesystem sandbox or observation policy. This is bounded by the destination being a single deployment-configured directory and never a model-supplied path, but it is a real gap — see the `FIXME` in `src/output.ts`.
- **No streaming.** Image and speech calls buffer the whole artifact in memory before writing.
- **PCM is assumed to be 16-bit.** `audio/pcm` media types carry a rate and a channel count but no sample width. Every provider observed emits signed 16-bit little-endian; one emitting 24- or 32-bit would produce a WAV that plays at the wrong speed rather than failing. See `PCM_BITS_PER_SAMPLE` in `src/output.ts`.
- **Video content filters reject on the audio track.** A `generate_audio: true` request can fail with a copyright refusal even when the visual prompt is innocuous. Pass `generate_audio: false` if you keep hitting it.
- **Video jobs outlive their timeout.** When `videoTimeoutMs` expires the tool reports a timeout naming the job id; the job keeps running server-side and the id stays pollable.
- **Config is read at load, not live.** The plugin reads its section once. Wire `installSettingsSection` from `@deepseek-ai/dsh-settings` if you want live edits.
- **No declaration emit.** `prepare` must work on a git install where peers may not be resolvable, so `tsdown` runs with `dts: false`. Run `pnpm typecheck` in a checkout with peers installed for the real type signal.

## Development

```sh
pnpm install
pnpm check       # self-check for the pure helpers; needs no key and no peers
pnpm typecheck   # real type signal; needs peers installed
pnpm build
```

`pnpm check` exercises the only branching logic no request covers — extension
selection, file naming, and replay-safe presenter narrowing. It runs standalone
because `src/output.ts` has no runtime peer imports.

## License

MIT