dsh-agent-quality-diagnosis

Readme

# dsh-agent-quality-diagnosis

English | [中文](README.zh.md)

`dsh-agent-quality-diagnosis` is a DSH Web plugin that turns the current session's event log into an execution-quality report for agent work. It complements DSH Trajectory: Trajectory shows the event ledger, while this plugin answers whether anything still blocks delivery and what concrete action closes it.

The plugin is an MVP and currently provides:

- a `质量诊断` tab registered through `dsh-better-sidebar`;
- a Host API that reads `ctx.sessions.get(sessionId).events` and builds a tool-call-level report;
- a Web runtime fallback when the Host report is unavailable;
- a user-facing report with `ready`, `needs_action`, or `blocked` status;
- a primary next action at the top of the report;
- open actions with a target, action, optional command or related command, done condition, and a copyable prompt for continuing the work in the current agent conversation;
- separate resolved and observed sections so mid-session issues fixed later do not block delivery;
- a comparison against the previous cached report that shows findings closed by a refresh;
- user-readable impact, folded evidence, and a Markdown report;
- local browser caching for the latest generated report;
- manual report refresh, report copy, command copy, and agent-prompt copy actions in the sidebar tab;
- a `defaultVerificationCommand` Host config value for project-specific mutation verification;
- copyable evidence locators with rule id, event sequence, tool name, call id, and evidence text;
- a browser error fallback so a rendering defect in the tab does not affect the active session;
- no changes to `agent-loop`, subagent scheduling, tool permissions, or model calls.

## Installation

Install `dsh-better-sidebar` in the web profile first:

```powershell
pnpm dsh plugin --profile web add dsh-better-sidebar@latest
```

Install this plugin from the repository path:

```powershell
pnpm dsh plugin --profile web add link:/path/to/your/dsh-agent-quality-diagnosis
```

Restart or refresh DSH Web, then open `质量诊断` in the sidebar.

## Screenshots

The gallery shows the main report, summary metrics, finding details, and copyable actions/evidence.

| Overview | Report summary |
| --- | --- |
| ![Quality diagnosis overview](assets/overview.png) | ![Report summary](assets/report-summary.png) |

| Finding detail | Actions and evidence |
| --- | --- |
| ![Finding detail](assets/finding-detail.png) | ![Actions and evidence](assets/actions-and-evidence.png) |

Optional Host config:

```yaml
- id: agent-quality-diagnosis
  name: 'dsh-agent-quality-diagnosis'
  config:
    defaultVerificationCommand: 'pnpm run check'
```

## Diagnosis signals

Host session-event reports detect:

- failed tool calls;
- consecutive duplicate tool calls;
- file mutation without a later successful verification result;
- verification failure without a later passing verification;
- risky operations such as deletion, secret access, external network commands, and permission changes.

The Web runtime fallback detects:

- failed or killed background jobs;
- agents waiting for user approval or input;
- unresolved collaboration with running, waiting, failed, or killed child work;
- repeated subagent titles that may indicate duplicated work;
- long-running workers.

## Boundaries

The report is a heuristic diagnostic signal, not an absolute quality score. Each finding carries user-readable impact, an action, confidence, and expandable evidence so users can judge false positives.

Overall status only considers `open` findings. A failed tool call with a later successful retry, a failed verification with a later passing verification, or a risky operation with an `allowed-once` approval record moves into the resolved section. Duplicate attempts and non-sensitive risky operations are observations unless they remain failed or lack the authorization evidence required for delivery.

Risky operations keep their original command as a related command, not as a command to rerun. The agent prompt asks for authorization, impact, and rollback explanation before any further high-risk operation.

All analysis runs locally inside the plugin. The plugin does not upload the session log and does not change execution behavior. If DSH session-event fields change, update `src/analyzer/trace-builder.ts`; rule logic stays under `src/rules/`.

## Local checks

The plugin carries its own lightweight checks because it lives outside the main workspace package list:

```powershell
pnpm run check
pnpm exec tsc -p tsconfig.json
pnpm exec vitest run --config vitest.config.ts
node --check lib\index.js
node --check lib\client.js
```

## Demo report

Generate a Markdown report from the included sample sessions:

```powershell
pnpm run demo:report
pnpm run demo:report -- fixtures/missing-verification-session.json
```

The fixtures cover a normal session, a failed tool call, and a mutation without later verification.
They also include resolved examples for a failed tool retry and a failed verification followed by a passing verification.

## Browser smoke

Use the linked web profile, start DSH Web, then open `质量诊断` on a session with tool calls:

```powershell
$env:DSH_HOME='<path-to-your-dsh-home>'
pnpm dsh --profile web
```

The tab should show Host session-event metrics when the Host route is available. If the route fails, it should show the Web runtime fallback and the Host error note.
After fixing an open issue, run refresh again. The tab should show the closed finding under `本次已关闭` when the previous cached report had that finding open.
Expanded evidence rows should provide a copyable locator that can be pasted into Trajectory search or a bug note.