dsh-plugin-loud-failure

Readme

# dsh-plugin-loud-failure

[![ci](https://github.com/Rhymer-Lcy/dsh-plugin-loud-failure/actions/workflows/ci.yml/badge.svg)](https://github.com/Rhymer-Lcy/dsh-plugin-loud-failure/actions/workflows/ci.yml)
[![license: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)

A [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) plugin that turns silent tool failures into loud ones.

A tool call can exit 0 and still have failed. `pandoc` drops a glyph and only warns; a pipe swallows a Python traceback; a `;` chain hides `command not found`; NumPy prints a `RuntimeWarning` and hands back `nan`. The exit code says success, the model reads success, and the mistake travels downstream. This plugin is one `tools/post-execute` waterfall listener: it matches the text a tool returned against a rule table and, on a hit, either **blocks** the result (it becomes an `isError` result whose content leads with the explanation and keeps the original output) or **attaches a notice** that lands in the model's next request. It changes no tool and no loop: it takes part in the ordered post-execute waterfall (context-only matches delegate and keep downstream decisions; blocking matches short-circuit on purpose) and unmounts cleanly.

## Contents

- [Why](#why)
- [How it works](#how-it-works)
- [Install](#install)
- [Configuration](#configuration)
- [Built-in rules](#built-in-rules)
- [What the model sees](#what-the-model-sees)
- [Verification](#verification)
- [Design notes](#design-notes)
- [Model Experience](#model-experience)
- [Known Limitations and Deferred Work](#known-limitations-and-deferred-work)
- [Development](#development)
- [License](#license)

## Why

Every built-in rule comes from a failure that was observed to hide behind a successful exit code:

| Observed | What the model saw | What had happened |
|---|---|---|
| `pandoc ... --pdf-engine=xelatex` prints `Missing character: There is no ₂ in font ...` | a PDF was written, exit 0 | the subscript in `SpO₂` was silently dropped from the PDF |
| `python script.py \| tail -n 20` | the last 20 lines, exit 0 | a traceback scrolled by; `tail` supplied the exit status |
| `pandocc in.md -o out.pdf; ls -l out.pdf` | `bash: pandocc: command not found` followed by a listing, exit 0 | nothing was built; `ls` supplied the exit status |
| `python calc.py` prints `RuntimeWarning: invalid value encountered in divide` | an array, exit 0 | the array contained `nan` |
| a PowerShell 5.1 command run with `2>&1` | `NativeCommandError`, `$?` false | the program had exited 0; PowerShell wrapped its stderr |
| a Windows console printing `���` | text, exit 0 | GBK/UTF-8 code-page mismatch; the text was corrupted |

The harness already records everything the model sees and lets any plugin rewrite a tool result before the model sees it. This plugin uses exactly that seam to make the failure visible at the moment it happens, instead of three steps later.

## How it works

```mermaid
flowchart LR
  call["tool/call"] --> pre["tools/pre-execute"] --> exec["tools/execute"] --> post["tools/post-execute<br/>(this plugin)"] --> result["tool/result"]
  post -->|"error rule matched"| block["block: isError result<br/>header + original output"]
  post -->|"context rule matched"| notice["accept + additionalContexts<br/>plugin-sourced notice"]
  post -->|"no match"| next["next()"]
```

1. At load, the plugin merges the built-in rules with the configured ones (a user rule with a built-in id replaces it) and compiles every pattern. An unusable table (invalid regular expression, duplicate id, stateful flag, empty message) rejects the plugin load with the offending rule id, so a broken config fails at boot, not at the first tool call.
2. On every `tools/post-execute`, the text blocks of the result (optionally also the JSON of a successful canonical value) are matched against the rules whose `when` and `tools` filters apply. A result counts as failed for the `when` filter when it is `isError` **or** its text carries a failure marker the shipped shell tools append themselves: a `[exit code: N]` line with N != 0, a `[status: ...]` trailer with a non-zero exit code, or a `[sandbox: file access denied ...]` notice. The model already sees those; they are not silent, so `when: success` rules stay quiet.
3. If any matching rule has `action: error`, the listener returns `{ kind: 'block' }` with feedback that begins with a header naming every matched rule and its message, followed by the original content blocks unchanged; context the tool body deferred is carried on the decision so a block does not drop it. The registry turns that into an `isError` result: the canonical value is gone, so a Code Mode program cannot consume a poisoned value either.
4. Otherwise (only `context` matches) the listener delegates with `next()` and appends one `UserMessage` to the decision's `additionalContexts`. Its source is `{ kind: 'plugin', plugin: 'loud-failure', form: 'notice', summary }`, so the Web UI shows a collapsed one-line row and the session log records exactly what the model was told.
5. No match: `next()`. Rules with `action: off` never run.

Because the listener is registered through `ctx.on`, it is torn down with the plugin; a config change reloads the plugin and registers a fresh listener.

## Install

Requires DeepSeek Harness `0.1.0-rc.6` (`@deepseek-ai/dsh-tools` and `@deepseek-ai/dsh-llm` at `0.1.0-rc.6`) and Node.js 20+. That is the only version this plugin has been tested against; the harness is a developer preview that may break compatibility, so the peer ranges are pinned and bumped per release. `dsh plugin add` prints peer-dependency warnings for `@deepseek-ai/*`: expected, because the profile resolves those packages from the harness installation rather than installing them next to the plugin.

**From a release tarball (no build step, no build authorization):**

```sh
dsh plugin --profile web add https://github.com/Rhymer-Lcy/dsh-plugin-loud-failure/releases/download/v0.1.1/dsh-plugin-loud-failure-0.1.1.tgz
dsh --profile web --dump-config   # shows a "# == dsh-plugin-loud-failure" layer
```

**From GitHub at a pinned commit:** a git install fetches source, so `pnpm` must be allowed to run this package's `prepare` script (it runs `tsc`). The first `add` fails and prints the exact key to allow (for a pinned commit the key includes the codeload URL); append it to the profile's `pnpm-workspace.yaml`, then rerun:

```sh
dsh plugin --profile web add github:Rhymer-Lcy/dsh-plugin-loud-failure#<commit-sha>
```

```yaml
# $DSH_HOME/profiles/web/pnpm-workspace.yaml (key copied from pnpm's message)
allowBuilds:
  "dsh-plugin-loud-failure@https://codeload.github.com/Rhymer-Lcy/dsh-plugin-loud-failure/tar.gz/<commit-sha>": true
```

Treat that authorization as what it is: code from this repository runs on your machine at install time, outside any agent sandbox. Pin the commit.

**From a source checkout, without installing:**

```sh
git clone https://github.com/Rhymer-Lcy/dsh-plugin-loud-failure.git
cd dsh-plugin-loud-failure && pnpm install && pnpm run build
```

```yaml
# overlay.yml
- insert:
    - id: loud-failure
      name: /absolute/path/to/dsh-plugin-loud-failure/lib/index.js
```

```sh
dsh web --patch ./overlay.yml
```

Uninstall with `dsh plugin --profile web remove dsh-plugin-loud-failure`.

## Configuration

The bundle inserts one row, `id: loud-failure`, with the schema defaults restated. A patch replaces a row's whole `config`, so an override in your profile's `cordis.patch.yml` must restate every key it keeps.

| Key | Type | Default | Meaning |
|---|---|---|---|
| `builtinRules` | boolean | `true` | Load the [built-in rule table](#built-in-rules) before user rules. |
| `rules` | `Rule[]` | `[]` | User rules. An id shared with a built-in rule replaces that rule in place; new ids append in order; two user rules with the same id fail the load. |
| `shellTools` | `string[]` | `[bash, pwsh, job_output]` | Tools inspected by rules that omit `tools`. Add other shell-like tools here (for example a terminal-read tool from another bundle). |
| `excerptChars` | natural | `240` | Maximum characters of the matched excerpt shown to the model. |
| `scanValue` | boolean | `false` | Also scan the JSON of a successful canonical value, for tools whose text projection hides stderr. Note: this `JSON.stringify`s the whole value before matching, and a match shows up to `excerptChars` characters of that otherwise hidden value to the model and the session log. |

A rule:

| Field | Type | Meaning |
|---|---|---|
| `id` | string, required | Stable identifier. |
| `pattern` | string, required | JavaScript regular-expression source. |
| `flags` | string | Any of `i`, `m`, `s`, `u`. `g` and `y` are rejected because they carry state between calls. |
| `tools` | `string[]` | Tools the rule applies to. Omitted or empty: `shellTools`. `['*']`: every tool. |
| `when` | `success` \| `error` \| `any` | Which outcomes the rule inspects. Default `success`, so a result that already failed is not decorated twice. |
| `action` | `error` \| `context` \| `off`, required | `error` blocks, `context` attaches a notice, `off` disables the rule. |
| `message` | string, required | What the signature means and what the model should do next. It is shown to the model verbatim. |

Disable one built-in rule and add one of your own:

```yaml
# $DSH_HOME/profiles/web/cordis.patch.yml
- id: loud-failure
  config:
    builtinRules: true
    rules:
      - id: no-such-file-with-success
        pattern: unused
        action: off
        message: unused
      - id: cuda-oom
        pattern: 'CUDA out of memory|torch\.OutOfMemoryError'
        tools: [bash, pwsh, job_output]
        action: error
        message: 'The GPU ran out of memory; whatever this run was supposed to produce is incomplete. Reduce the batch size or free VRAM, then rerun.'
    shellTools: [bash, pwsh, job_output]
    excerptChars: 240
    scanValue: false
```

## Built-in rules

All built-in rules use `when: success` and apply to `shellTools`.

| id | action | Fires on |
|---|---|---|
| `pandoc-missing-character` | error | a line starting with `Missing character: There is no ` or `[WARNING] Missing character: There is no ` (xelatex/lualatex through pandoc; the PDF is missing glyphs) |
| `python-traceback-with-success` | error | a line starting with `Traceback (most recent call last):` (leading whitespace or ANSI colour codes allowed) in a result that reported success |
| `shell-command-not-found` | error | `bash: [line N:] name: command not found` and `sh: N: name: not found` lines |
| `windows-command-not-recognized` | error | a line of the form `'name' is not recognized as an internal or external command` or `[prefix: ][The term ]'name' is not recognized as a/the name of a cmdlet` |
| `fatal-signal-with-success` | error | a shell crash line: `Segmentation fault` (with or without `(core dumped)`), or `Bus error` / `Aborted` with `(core dumped)`, optionally prefixed by `bash: line N: PID` |
| `numpy-runtime-warning` | context | `RuntimeWarning: invalid value encountered` / `divide by zero encountered` / `overflow encountered` |
| `powershell-native-command-error` | context | `NativeCommandError` |
| `latex-undefined-references` | context | `LaTeX Warning: There were undefined references` / `Citation ... undefined` / `Reference ... undefined` |
| `latex-rerun-needed` | context | `Rerun to get cross-references right`, `Rerun to get outlines right`, `rerunfilecheck Warning` |
| `pandoc-could-not-fetch-resource` | context | `[WARNING] Could not fetch resource` |
| `no-such-file-with-success` | context | `No such file or directory` |
| `permission-denied-with-success` | context | `Permission denied` |
| `unicode-replacement-character` | context | any U+FFFD replacement character in the output |

The `error` rules are line-anchored to the shape the tools actually print, which rules out the common quoted-substring mention (`grep -r "command not found" logs/`, a README quoting the pandoc warning); a file that echoes a genuine log line verbatim will still match, and that is a known limitation. The `context` rules are deliberately broader because a notice is cheap and the model keeps the original result.

## What the model sees

A blocked result, exactly as rendered (the second content block is the untouched original output):

```text
[loud-failure] Tool "bash" reported success, but its output matched a rule (pandoc-missing-character) that indicates a silent failure. dsh-plugin-loud-failure marked this result as an error; do not treat the call as successful.

- pandoc-missing-character (error): pandoc/xelatex dropped one or more glyphs; the produced PDF is missing characters even though the exit code was 0. Rewrite the character (for example write SpO2 instead of a subscript digit) or switch to a font that has it, then rebuild and re-check for this warning.
  excerpt: "pandoc thesis.md -o thesis.pdf --pdf-engine=xelatex\n[WARNING] Missing character: There is no ₂ in font Microsoft YaHei/OT!\n[exit code: 0]"

The original tool output follows unchanged.
```

A notice attached to a successful result (the result itself is unchanged; this arrives as a `user`-role message with source `{"kind":"plugin","plugin":"loud-failure","form":"notice","summary":"loud-failure: bash output matched numpy-runtime-warning"}`):

```text
[loud-failure] Notice for tool "bash": its output matched a rule (numpy-runtime-warning) that often means a silent failure. Verify before relying on this result.

- numpy-runtime-warning (context): NumPy raised a RuntimeWarning about NaN, division by zero, or overflow. Downstream numbers may be NaN or inf while the exit code stays 0. Check the affected arrays before using the result.
  excerpt: "python calc.py\ncalc.py:12: RuntimeWarning: invalid value encountered in divide\n[0.5 nan 0.25]"
```

## Verification

`pnpm run check` runs typecheck, tests, build, and a pack dry-run; CI runs it on Ubuntu and Windows with Node 20 and 22. 87 tests in total.

- `tests/rules.test.ts` covers merging (including the duplicate-user-id rejection), compilation failures (each with its message), filters, statelessness, excerpt bounds, the failure-marker detector, and every built-in rule against real signatures plus the negative cases (clean output, non-shell tools, already-failed results, quoted mentions, `Aborted` as a plain word).
- `tests/plugin.test.ts` mounts the plugin next to the real `ToolRuntime` and drives tool calls through the complete pipeline: block with original output preserved, notice with the plugin source, precedence of `error` over `context`, tool and `when` filters, results carrying `[exit code: N]` / job-status / sandbox markers left alone by `success` rules, `when: any` still applying to them, deferred tool context surviving a block, built-in override, a user rule with `tools` omitted, `scanValue`, composition with a downstream `tools/post-execute` listener, clean unmount, and fail-loud on a bad rule table.
- The bundle was also exercised against `@deepseek-ai/dsh@0.1.0-rc.6` on Windows: `dsh plugin add` from a checkout, from the v0.1.0 release tarball URL, and from `github:...#<sha>` (with the `allowBuilds` key) all compose, and `--dump-config` shows the layer; booting `dsh --profile web` with an overlay that injects an invalid rule fails at boot with `RuleConfigError: rule "bad": invalid pattern` raised from the installed plugin's `apply`, and booting with the shipped config serves the Web UI.

## Design notes

- **`tools/post-execute`, not `tools/result`.** `tools/result` only observes the frozen outcome; the point here is to change what the model sees, which is exactly what post-execute is for (replace content, block with feedback, attach context).
- **Block keeps the original output and the deferred context.** A blocked result would otherwise erase the evidence the model needs to fix the problem. The header comes first so the model reads the verdict before the noise; the original blocks follow unchanged, and context the tool body deferred rides on the decision instead of being dropped.
- **Two severities, no third.** `error` is for signatures that mean the work did not happen; `context` is for signatures that mean "check before you trust". Anything softer than that is not worth a token.
- **Regular expressions, not a model.** The signatures are literal strings emitted by tools; a regex is deterministic, cheap, testable, and auditable in the config dump. Semantics belong to the model, which gets the rule message and the excerpt.
- **Fail loud at load.** A rule table is configuration; a bad one should stop the boot with the rule id, not degrade into a listener that never fires.
- **Waterfall etiquette.** A blocking decision short-circuits, as a decision-owning policy listener should; a context decision delegates first and enriches whatever came back, so downstream replacements survive. Listeners registered before this one wrap it and can still override either outcome.

## Model Experience

### Request context and condition

#### What the model sees

Only when a rule matches. For an `error` match: the tool result becomes `isError: true` and its content is one text block rendered as in [What the model sees](#what-the-model-sees), followed by the original content blocks. For a `context`-only match: the tool result is unchanged and one `user`-role notice message with source `{ kind: 'plugin', plugin: 'loud-failure', form: 'notice', summary }` is appended after the recorded tool results of that batch. Rule messages are configuration and appear verbatim; the excerpt is bounded by `excerptChars`.

#### Token effect

Conditional. Zero when nothing matches. On a match: the header plus, per matched rule, its message and up to `excerptChars` characters of excerpt. A block adds only that header; the original blocks that follow are the same bytes the model would have received anyway.

#### KV Cache effect

Append-only. The plugin contributes no system-prompt section and never rewrites earlier history; a block changes the content of the current tool result before it is first recorded, and a notice is appended after that batch's tool results. Neither invalidates any previously reusable prefix.

## Known Limitations and Deferred Work

- **Text only.** The listener inspects text blocks (and, with `scanValue`, the JSON of a successful value). A failure that leaves no trace in the output, such as a library returning a negative status that the script never prints, cannot be caught here; the pattern for those is to print the value and let a rule catch it.
- **Regular expressions can false-positive.** The `context` rules are broad by design; disable one with `action: off` if it is noisy in your workload. A blocked result still carries the original output, so a false `error` costs the model one corrective step, not the information.
- **Only three failure markers are recognized.** `[exit code: N]`, a `[status: ...]` trailer with a non-zero exit code, and the sandbox denial notice. Other ways a shell tool can report failure in text (signal names, timeouts) are not yet treated as failed outcomes, so a `success` rule can still fire on them.
- **Whole-text scan.** Very large tool outputs are scanned once per rule; the shipped shell tools already truncate long output, so this has not been a cost in practice, but there is no size cap yet.
- **`run_code` aggregate output.** Code Mode sub-dispatches each pass through post-execute and are inspected individually; the outer `run_code` result (program logs and return value) is inspected as ordinary text when `run_code` is listed in `shellTools`, which it is not by default.
- **No per-agent scoping.** The listener is registered on the plugin context and sees every agent's calls. Scoping to a subagent preset is a Cordis-level exercise not attempted here.

## Development

```sh
pnpm install
pnpm run check          # typecheck + test + build + pack dry-run
pnpm run test:watch
```

Source lives in `src/` (`rules.ts` is pure and unit-tested; `index.ts` is the Cordis adapter); tests in `tests/`; `lib/` is build output produced by `tsc` and by the `prepare` script on git installs.

## License

[MIT](LICENSE)