@dsh-voice/bundle

Readme

# dsh-voice

**Voice notes in, spoken answers out.** Dictate audio that becomes user messages, and have the agent read replies aloud. A hands-free terminal for DSH.

`dsh-voice` is a DeepSeek Harness bundle. Two tools, one durable event, one toggle:

- **`transcribe({ source })`** — speech-to-text. Pass `{ file }` (an existing audio file) or `{ record }` (record from the mic for a few seconds). The transcript becomes a **user message** the agent responds to — never tool output — and the chat shows a compact **audio card** with play/pause, duration, backend badge, and the transcript as caption.
- **`speak({ text, voice?, rate? })`** — text-to-speech on a background job. The tool returns `{ jobId, audioRef }` immediately and never blocks the turn; playback happens async and a failure surfaces as an injected note. `speak` doubles as **walk-away narration** for long builds and headless runs ("build finished, 0 failures").
- **`readReplies` + `/voice`** — a per-session toggle that auto-narrates the assistant's reply text. Off by default; flip it live with `/voice on`.

The design center is **local-first**: audio is plain files under `~/.dsh/voice/` (inspect them, `rm` them), nothing leaves the machine unless you explicitly configure a cloud backend, and nothing audio-related ever auto-runs — the model must call a tool.

## Why this shape

A terminal agent handles two everyday moments badly: you're away from the keyboard and want to leave an instruction (dictation), and you're mid-task and don't want to read a wall of output (narration). dsh-voice is a thin layer over things DSH already exposes — `ctx.shell`, `ctx.jobs`, `ctx.settings`, `ctx.attachments`, `ctx.conversationEvents` — so it stays useful without owning any audio pipeline itself. Audio is **plain files**, the session log holds only **refs + transcripts** (the attachment/image-ref pattern), and replay reproduces the audio card without re-reading audio.

## Install

```sh
dsh plugin --profile web add @dsh-voice/bundle
```

The bundle installs the `dsh-voice` entry (tools + `/voice` command + the web audio cards). Nothing runs until the model calls a tool.

## Config

All fields optional (profile patch or `cordis.patch.yml`):

```yaml
plugins:
  dsh-voice:
    stt:
      backend: whisper-local | openai | macos | fake   # absent = auto (whisper-local → macos)
      model: whisper-1                                  # STT model
      whisperLocal: { bin: whisper-cli, model: tiny }   # whisper.cpp binary + model
      openai: { baseUrl: https://api.openai.com/v1, apiKeyEnv: OPENAI_API_KEY }
    tts:
      backend: say | piper | edge-tts | fake            # absent = auto (say → piper)
      voice: Samantha                                   # default voice
      rate: 180                                         # say words per minute
      piper: { bin: piper, model: /path/to/model.onnx }
      edgeTts: { voice: en-US-GuyNeural }
    readReplies: false                                  # narrate replies when on
    audioDir: ~/.dsh/voice                              # artifact root
```

Defaults: `stt.backend` auto-selected offline (whisper-local → macos), `tts.backend: say`, `readReplies: false`, `audioDir: ~/.dsh/voice`. **Cloud backends are never auto-selected** — `openai` and `edge-tts` are reachable only when you pin them. The `openai` backend reads its key through the standard credential seam (`OPENAI_API_KEY`, the same convention a polyglot preset would use), falling back to the launching environment.

## Tools

### `transcribe({ source, to? })`

`source` is **exactly one** of:

- `{ file: <path> }` — transcribe an existing audio file.
- `{ record: { seconds? } }` — record from the microphone (default 5s), gated on a recording path being available (ffmpeg or the bundled swift shim on macOS).

The transcript is **inserted as a user message**, not returned as tool output: a `voice/note` session event renders the audio card as a user-authored turn, and the text is delivered to the agent as user input. The canonical return is a compact handle — `{ transcript, audioRef, backend, durationMs }` — so Code Mode callers get structured data.

With **dsh-crosstalk** installed, `transcribe({ source, to: <peer> })` delivers the note to another local session as a labeled peer message with the audio path attached (crosstalk owns provenance framing; the option simply isn't offered without it).

### `speak({ text, voice?, rate? })`

Synthesizes + plays on a **background job** (`ctx.jobs`, kind `voice-speak`), returns `{ jobId, audioRef }` immediately. Every backend writes a durable file under `audioDir` first (the unit-testable seam), then plays it as a separate best-effort step. A job failure is injected as a note, never a thrown turn.

Because it's a plain tool over `ctx.jobs`, `speak` is callable from routines and headless runs — **narration *is* speak called from a job context**. No new surface.

## Voice notes in chat

Audio never enters the session log. The file lands under `audioDir`; the log holds one durable event:

| Event | Role | Required durable facts |
|---|---|---|
| `voice/note` | unique start | `noteId`, turn/step coords, `audioRef` (path + mime + durationMs), `transcript`, `direction: 'in' \| 'out'`, `backend` |

Single-event business in v0.1 — `noteId` is the stable id, no update events. The web client renders `voice-note` cards: inbound notes (STT) read as user turns, outbound (`speak`) as agent-side cards. A missing or deleted file degrades to a transcript-only card — you're free to `rm` audio.

## `/voice`

```sh
/voice on          # narrate the assistant's replies aloud
/voice off         # stop
/voice status      # current state + backend + audioDir
/voice speak <text>  # speak a line directly from the composer
```

`readReplies` defaults follow config; the toggle is per-session and live.

## Backends

Speech-to-text (`dsh-voice-backends` module owns selection + the fake):

- **`whisper-local`** — a whisper.cpp binary on PATH (or configured), invoked via `ctx.shell`. Fully offline.
- **`openai`** — an OpenAI-compatible `whisper-1` endpoint via the standard credential seam. The only STT path that sends audio off-machine; only when configured.
- **`macos`** — built-in `SFSpeechRecognizer` via a tiny bundled swift shim through `ctx.shell`. No install, no network setup.
- **`fake`** — text-to-text fixture mapping (a file whose content is `{"transcript": "…"}` — or whose basename is `fixture-<text>.m4a` — transcribes to that text). Runs the whole tool path with no mic and no network; the CI default.

Text-to-speech:

- **`say`** (default) — macOS `say -o <file> --file-format=m4af --data-format=aac`, then `afplay`. Zero install; writes Chrome/Safari-playable m4a.
- **`piper`** — local Piper binary, offline neural TTS.
- **`edge-tts`** — cloud; only when explicitly configured.
- **`fake`** — writes `{"transcript": "<text>"}` so speak output round-trips through the fake STT exactly.

Selection is pure and unit-tested: configured backend always wins; otherwise offline fallback order (`whisper-local → macos`, `say → piper`); cloud never auto-selected; no offline backend → a clear error telling you what to configure.

## Safety / privacy defaults

- **Local-first** — audio never leaves the machine unless you explicitly set `stt.backend: openai` or `tts.backend: edge-tts`.
- **Plain files** — every artifact is a file under `audioDir` you can inspect or `rm`; the session log holds only refs + transcripts.
- **No auto-run** — recording and playback happen only on an explicit tool call. `readReplies` narrates existing replies; it never records, and it's off unless configured.

## Non-goals (v0.1)

Real-time streaming conversation; outbound synthesized voice calls; audio in group WeChat contexts; speaker diarization; music/effects; storing raw audio in the session log; wake-word / always-listening capture.

## Testing

```sh
pnpm install
pnpm typecheck   # host + client tsconfigs
pnpm test        # node --test (46 tests)
pnpm build       # tsc host + client declarations + the web client bundle
pnpm pack        # publishable tarball
```

The suite covers the spec's testing goals: arg-schema units (the exact-one `{file|record}` union, `speak`'s optional `voice`/`rate`), backend selection with faked probes, the fake text-to-text backend end-to-end through both tool pipelines, the `voice-note` renderer (expected `node.data` from a logged event, transcript-only degradation, replay purity), and a macOS `say` integration test (synthesizes a non-empty m4a under audioDir).

The client bundle (`lib/client.js`) is built by `scripts/build-client.mjs` into the web client's lazy-CJS handoff format and served at `/plugins/@dsh-voice/bundle/client.js` when the bundle is installed in a web profile.

## Development

The repo mirrors the sibling plugin layout: `src/backends/` is the `dsh-voice-backends` module (interfaces, pure selection, probes, the fake, and every concrete backend); `src/tools/` holds the `transcribe`/`speak` pipelines with injected deps so tests run with fakes; `src/client/` is the web half (pure Definition + React audio card); `shims/` are the bundled swift scripts for macOS STT and mic recording.

## License

MIT