dsh-tool-squeeze

Readme

# dsh-tool-squeeze

**Evidence-preserving tool-output compression for DeepSeek Harness.**

[![DeepSeek Harness](https://img.shields.io/badge/DeepSeek_Harness-0.1.0--rc.8-4c6fff)](https://github.com/deepseek-ai/DeepSeek-Harness)
[![Tests](https://img.shields.io/badge/tests-21_passing-22c55e)](#benchmark)
[![License: MIT](https://img.shields.io/badge/license-MIT-0f172a.svg)](LICENSE)
[![Local first](https://img.shields.io/badge/runtime-local--first-f59e0b)](#safety)

Turn massive logs, JSON, HTML, and text into compact, model-ready evidence—
**before noisy tool output consumes the agent's context window.**

```text
Synthetic Maven failure fixture

Without                     With dsh-tool-squeeze
~22,998 estimated tokens    ~1,736 estimated tokens

                            ↓ 92.5%
                            Critical evidence: PASS
```

`dsh-tool-squeeze` is a deterministic, local-first DeepSeek Harness plugin. It
compresses oversized log, JSON, HTML, and plain-text tool results at the public
`tools/post-execute` lifecycle seam—before the result is repeatedly sent to the
model. It uses no LLM and does not patch Harness Core.

> Safety comes first. Small and mixed-media results pass through unchanged,
> errors and boundaries are prioritized, every omission is disclosed, and the
> full formatted original is retained through the official DSH spill store.

[中文说明](README.zh-CN.md)

## Why agent builders use it

- **More useful context:** compresses repetitive output while prioritizing errors,
  failed tests, stack roots, warnings, exit information, and structural samples.
- **No extra model call:** deterministic local processing keeps latency, cost, and
  data flow predictable.
- **Evidence, not blind truncation:** every omission is disclosed and the complete
  formatted original is retained through the official DSH spill store.
- **Safe integration:** uses the public `tools/post-execute` lifecycle, never
  patches Harness Core, and fails open when retention or processing is unsafe.
- **Reproducible claims:** committed fixtures, exact evidence gates, and raw
  benchmark results let you verify the behavior yourself.

## Problem

A tool can return tens of thousands of low-information lines. Blind truncation
may remove the one failure in the middle, while leaving the full result inline
spends context on every later model request.

DeepSeek Harness rc.8 already has an excellent hard-size fallback: its official
spill policy stores very large plain text and shows a head/tail preview. It also
prunes old results during compaction. This plugin complements those features by
making the first model-facing result content-aware: it preserves errors, failed
tests, structured samples, document semantics, and exact omission counts before
the native spill cap is applied.

## Before / after

```text
[dsh-tool-squeeze]
Output compressed: 92.5% (estimated).
Original ~22,998 tokens → ~1,736 tokens.
Some repetitive or low-information content was omitted.
Preserved: head, tail, errors, warnings, stack trace roots, exit information.
Original retained at: .../maven-original.txt
Use read with offset/limit, or grep this path.

[retained head]
...
[middle error and root cause]
...
[retained tail]
```

The notice is part of the model-visible content. Token and reduction numbers are
explicitly estimates.

## Install

The package is an ordinary DSH bundle, not a Skill. From GitHub:

```bash
dsh plugin --profile web add github:w2829562572-dev/dsh-tool-squeeze
```

For a local checkout:

```bash
pnpm install --ignore-scripts
pnpm build
dsh plugin --profile web add -w .
dsh --profile web --dump-config
```

Pin a reviewed commit for reproducible installs while DSH is in developer
preview.

## How it works

```text
Tool result
  → inspect size, type, and tool name
  → threshold / exclusion decision
  → log | JSON | HTML | text processor
  → evidence and token-budget guard
  → retain original through ctx.spillStore
  → transparent notice + compressed content
  → official spill-policy hard-size fallback
  → model context
```

The plugin returns a content-only `PostToolDecision`. It never replaces the
canonical tool `value`. See [architecture](docs/architecture.md) and
[API research](docs/research.md).

## Supported output types

| Type | V0.1 behavior |
| --- | --- |
| Logs | Head/tail, error and warning windows, stack roots, failed tests, exit information, exact repeated line/block counts |
| JSON | Structure, representative indices, error-bearing records, nested summaries, exact omitted item/key counts |
| HTML | Rules-based readable Markdown retaining titles, headings, paragraphs, tables, lists, links, and code; removes scripts/styles/SVG/comments |
| Text | Conservative whitespace cleanup and exact line/paragraph deduplication, then evidence-aware head/tail selection if still over budget |

Mixed non-text content passes through unchanged.

## Safety

The priority order is:

```text
Safety > information preservation > compression ratio
```

- Outputs below `minTokens` are unchanged.
- Compression must be smaller and within `maxTokens` after its notice.
- Error, warning, failure, traceback, assertion, cause, and non-zero exit
  evidence receives priority.
- The full formatted original is retained by default using the official DSH
  spill service. If retention is unavailable or fails, the result passes through.
- Parser or compressor exceptions are caught; a plugin failure never turns the
  tool call into an error.
- `excludeTools` provides a durable per-tool bypass, and `enabled: false` is a
  complete bypass.

See the full [safety model](docs/safety.md).

## Configuration

Defaults:

```yaml
enabled: true
minTokens: 4000
targetTokens: 6000
maxTokens: 12000

processors:
  log: true
  json: true
  html: true
  text: true

preserve:
  headLines: 80
  tailLines: 120
  errors: true
  warnings: true

report:
  enabled: true

excludeTools: []
retainOriginal: true
```

Override the installed row in your profile's `cordis.patch.yml`:

```yaml
- id: dsh-tool-squeeze
  config:
    minTokens: 6000
    targetTokens: 5000
    excludeTools: [database_export]
```

Set `retainOriginal: false` only if losing the retrievable formatted original is
acceptable. The compression notice will state that retention is disabled.

## Benchmark

The committed deterministic fixtures produce:

| Fixture | Original Tokens | Compressed | Reduction | Critical Evidence |
| --- | ---: | ---: | ---: | --- |
| Maven failure | 22,998 | 1,736 | 92.5% | PASS |
| Gradle failure | 60,943 | 123 | 99.8% | PASS |
| npm build | 44,527 | 3,028 | 93.2% | PASS |
| pytest | 42,732 | 2,669 | 93.8% | PASS |
| JSON API | 159,459 | 352 | 99.8% | PASS |
| HTML page | 57,321 | 145 | 99.7% | PASS |
| Generic log | 85,027 | 106 | 99.9% | PASS |

These are synthetic high-repetition fixtures designed to verify processor and
evidence behavior, not a claim that production workloads always save 90%+. Token
counts use the provider-neutral V0.1 estimator and include the notice. Reproduce
them with:

```bash
pnpm benchmark
```

Raw results: [`benchmark/results.json`](benchmark/results.json). Method and
caveats: [`docs/benchmarks.md`](docs/benchmarks.md).

## AI and search discovery

For AI assistants, catalogues, and technical search engines, the canonical
project name is **dsh-tool-squeeze**: a local-first, evidence-preserving tool
output compressor for **DeepSeek Harness / DSH**. Machine-readable project facts
and citation guidance are available in [`llms.txt`](llms.txt) and
[`CITATION.cff`](CITATION.cff).

## Architecture

The host-only Cordis plugin has no Web UI bundle and no replacement tool. Its
normal-order post-execute listener sits downstream of the official prepended
spill listener, so content-aware compression runs first and the native spill cap
remains the outer safety bound. Session stats are held in memory in V0.1; a Web
dashboard and `/squeeze` command are intentionally deferred.

## Compatibility

V0.1 is developed and integration-tested against:

- `@deepseek-ai/dsh@0.1.0-rc.8`
- `@deepseek-ai/dsh-tools@0.1.0-rc.8`
- `@deepseek-ai/cordis@4.0.1`
- Node.js 22.19+

The peer range deliberately pins the DSH tool lifecycle RC. Upstream states that
developer-preview releases may introduce breaking changes.

## Limitations

- Compression is lossy even when the original is retrievable; exact downstream
  reasoning quality needs task-level evaluation, not only string evidence gates.
- V0.1 uses an approximation instead of a provider tokenizer.
- JSON sampling can pass through when too many unique critical records exceed
  the maximum budget—safety is preferred over forced compression.
- HTML extraction is semantic and conservative, not a readability classifier.
- Nested `run_code` dispatch-log copies are left to the official spill policy.
- No command-specific parser catalog, UI dashboard, or one-call transient bypass
  is included yet.

## Contributing

Keep changes deterministic, fail-open, and evidence-backed. Add a fixture and an
exact critical-evidence gate for new processor behavior, then run:

```bash
pnpm lint
pnpm typecheck
pnpm test
pnpm build
pnpm benchmark
```

Please do not add LLM calls or copy third-party implementations. Competitive
context and licenses are documented in
[`docs/competitive-analysis.md`](docs/competitive-analysis.md).

## License

[MIT](LICENSE). The only runtime libraries are DeepSeek's MIT-licensed
Schemastery package and the MIT-licensed `htmlparser2` parser stack. No
third-party source code is vendored.