dsh-tool-chaos

Readme

# dsh-tool-chaos

**English** · [简体中文](README.zh-CN.md)

[![CI](https://github.com/cyanseek/dsh-tool-chaos/actions/workflows/ci.yml/badge.svg)](https://github.com/cyanseek/dsh-tool-chaos/actions/workflows/ci.yml)

> Deterministic fault injection and evidence-backed resilience tests for the DeepSeek Harness tool pipeline.

`dsh-tool-chaos` runs an isolated DSH experiment through **baseline → dry-run → fault injection → PASS / FAIL / INCONCLUSIVE**. It helps plugin and Agent authors verify retry, fallback, cancellation, policy-denial, and untrusted-result behavior before those paths fail unexpectedly.

Status: **`0.1.0-next.0` prerelease**. DSH is still in developer preview, so pin versions and review the [tested compatibility matrix](docs/COMPATIBILITY.md).

## Highlights

- Five DSH-native actions: `deny`, `error`, `delay`, `abort`, and `block`.
- Reproducible matching and sampling; no runtime randomness.
- Disabled and dry-run by default, with bounded injections and explicit global-wildcard opt-in.
- JSON and Markdown reports with structured decision events and a three-state verdict.
- Native Cordis plugin, zero-runtime-dependency CLI, TypeScript engine API, and Codex/Agent Skill surfaces.

## Quick start

The npm prerelease has not been published yet. The currently available path is a source checkout and a locally built tarball:

```bash
git clone https://github.com/cyanseek/dsh-tool-chaos.git
cd dsh-tool-chaos
npm ci
npm run build
npm pack --silent

dsh plugin --profile chaos-demo add ./dsh-tool-chaos-0.1.0-next.0.tgz
dsh --profile chaos-demo --dump-config
```

The effective config should contain `tool-chaos` and `tool-chaos-reporter`. Both are inert after installation: the main plugin is disabled and the reporter is disabled.

Generate a complete dry-run overlay:

```bash
node bin/dsh-tool-chaos.mjs init .dsh-chaos/retry.patch.yml --preset retry
```

Review the generated tool matcher, task, and assertion, then run it against the disposable profile:

```bash
node bin/dsh-tool-chaos.mjs run \
  --mode dry-run \
  --profile chaos-demo \
  --patch .dsh-chaos/retry.patch.yml \
  --task "Use web_fetch and recover from one transient failure" \
  --expect "expected stable phrase" \
  --no-install \
  --json
```

A dry-run must show the intended `dryRun: true` decision while leaving tool execution unchanged. Run a real experiment only after that match is exact.

## Install from a pinned GitHub commit

DSH can install this repository directly:

```bash
dsh plugin --profile chaos-demo add \
  github:cyanseek/dsh-tool-chaos#<commit-sha>
```

pnpm 10+ can intentionally block a Git dependency's `prepare` script. If it does, review the pinned source and copy the **exact package key printed by pnpm** into that profile's `pnpm-workspace.yaml`:

```yaml
allowBuilds:
  dsh-tool-chaos@https://codeload.github.com/cyanseek/dsh-tool-chaos/tar.gz/<commit-sha>: true
```

Rerun the install and verify the effective config. This approval allows package code to execute during installation; a reviewed tarball remains the simpler option.

To uninstall and verify removal:

```bash
dsh plugin --profile chaos-demo remove dsh-tool-chaos
dsh --profile chaos-demo --dump-config
```

## Configuration

Installed defaults cannot inject a fault:

```yaml
- id: tool-chaos
  config:
    enabled: false
    dryRun: true
    seed: dsh-tool-chaos
    allowGlobalWildcard: false
    rules: []
```

A bounded preview rule looks like this:

```yaml
- id: tool-chaos
  config:
    enabled: true
    dryRun: true
    seed: retry-suite-v1
    allowGlobalWildcard: false
    rules:
      - id: first-github-error
        tool: github_*
        action: error
        scope: root
        argumentsPattern: ''
        probability: 1
        afterMatches: 0
        every: 1
        maxInjections: 1
        delayMs: 1000
        message: deterministic GitHub transport failure
```

`tool` is an anchored, case-sensitive glob. `argumentsPattern` runs against stable key-sorted JSON. DSH patch rows replace a row's complete `config`, so every overlay must restate all keys it relies on.

## Fault actions

| Action | DSH pipeline phase | Behavior |
|---|---|---|
| `deny` | `tools/pre-execute` | Reject before dispatch |
| `error` | `tools/execute` | Return a structured injected error without running the tool body |
| `delay` | `tools/execute` | Add abort-aware latency |
| `abort` | `tools/execute` | Abort a derived signal and restore the upstream signal |
| `block` | `tools/post-execute` | Run the tool, then withhold its result |

Each decision emits `tool-chaos/decision`. The optional reporter converts it to a stable stderr JSONL boundary prefixed with `DSH_TOOL_CHAOS_EVENT `.

## Evidence and verdicts

The CLI writes a JSON source-of-truth report and a Markdown summary. Reports include configuration hashes, bounded process metadata, parsed chaos decisions, assertions, verdict, and a reproduction command. Raw command output is excluded unless explicitly requested.

| Verdict | Meaning |
|---|---|
| `PASS` | The intended fault occurred and the declared recovery behavior is supported by evidence |
| `FAIL` | The intended fault occurred and at least one recovery assertion failed |
| `INCONCLUSIVE` | Baseline, installation, matching, safety, or observation evidence is incomplete |

Exit code alone is never sufficient evidence of resilience.

## CLI

```text
dsh-tool-chaos doctor [--profile name] [--json]
dsh-tool-chaos setup [--profile name] [--source npm-or-path] [--codex] [--json]
dsh-tool-chaos init [path] [--preset retry|timeout|nested|result-block|deny] [--force]
dsh-tool-chaos run --patch file --task prompt [--mode baseline|dry-run|experiment]
                   [--expect regex] [--profile name] [--json]
dsh-tool-chaos install-plugin [--profile name] [--source npm-or-path] [--json]
dsh-tool-chaos install-skill [--global | --project path]
dsh-tool-chaos install-codex-plugin [--dir path] [--json]
dsh-tool-chaos verify [--json]
dsh-tool-chaos prompt [--lang en|zh] <requirement...>
dsh-tool-chaos codex [--cwd path] [--json] <requirement...>
```

Automation should consume `--json` instead of scraping human-readable output.

## Codex and Agent Skill

The repository contains a Codex marketplace plugin and an open Agent Skill. Install the Codex plugin from a checkout:

```bash
codex plugin marketplace add /absolute/path/to/dsh-tool-chaos
codex plugin add dsh-tool-chaos@dsh-tool-chaos
```

Start a new thread and invoke:

```text
$dsh-chaos-test Prove the retry path survives one deterministic GitHub tool error.
```

See [Codex integration](docs/CODEX.md) for the source-checkout workflow. `npx dsh-tool-chaos ...` commands become available only after the npm prerelease is published.

## Programmatic engine

The `dsh-tool-chaos/engine` export has no Cordis or DSH runtime import:

```ts
import { ChaosEngine, resolveConfig } from 'dsh-tool-chaos/engine'

const engine = new ChaosEngine(resolveConfig({
  enabled: true,
  dryRun: false,
  seed: 'ci-retry-v1',
  rules: [{
    id: 'first-web-error',
    tool: 'web_fetch',
    action: 'error',
    maxInjections: 1,
  }],
}))

const decision = engine.decide('execute', {
  callId: 'call-1',
  rootCallId: 'call-1',
  name: 'web_fetch',
  arguments: { url: 'https://example.test' },
})
```

See [the engine example](examples/engine-api.mjs).

## Safety boundaries

- Use disposable profiles and synthetic or read-only test inputs.
- Keep `dryRun: true` until the exact intended match is visible.
- Exact global `*` requires `allowGlobalWildcard: true`.
- Every rule injects at most once by default.
- The runtime plugin does not rewrite arguments, read files, or access the network.
- `abort` is cooperative; a same-process tool that ignores `AbortSignal` cannot be hard-killed.
- Never represent skipped or unobservable checks as success.

Production/shared profiles, external mutations, credentials, broad targeting, or privilege changes require separate safeguards and explicit authorization.

## Compatibility

| Surface | Verified state |
|---|---|
| Tests | Strict TypeScript compile and 32 unit/CLI/real-runtime tests pass |
| Node | 24.11.1 locally; GitHub CI passes on 22.19.0 and 24.x |
| DSH | `@deepseek-ai/dsh-tools@0.1.0-rc.6` runtime integration passes |
| Tarball consumer | Install, repeated add, config dump, and read-only headless dry-run preview pass |
| GitHub source | Pinned commit install, repeated install, config dump, uninstall, and reinstall pass with pnpm 11.7.0 |
| npm | Not published; no npm availability claim is made |

Evidence was collected on August 14, 2026 with isolated profiles and synthetic read-only input. See [Compatibility](docs/COMPATIBILITY.md) for the precise matrix and limitations.

## Documentation

- [CLI and machine API](docs/CLI.md)
- [Report schema](docs/REPORT-SCHEMA.md)
- [Architecture](docs/ARCHITECTURE.md)
- [Codex integration](docs/CODEX.md)
- [Compatibility](docs/COMPATIBILITY.md)
- [Contributing](CONTRIBUTING.md)
- [Security](SECURITY.md)
- [Support](SUPPORT.md)
- [Changelog](CHANGELOG.md)

## Development

```bash
npm ci
npm run sync:skill
npm run check
```

Compatibility claims must be backed by the exact environment and command that passed.

## License

[MIT](LICENSE)