dsh-model-probe

Readme

# DSH Model Probe

Persistent model-health monitoring for the **DeepSeek Harness (DSH)** Web UI.

DSH Model Probe discovers the models advertised by active DSH providers, probes them through DSH's own LLM runtime, keeps a persistent **Alive / Dead** health ledger, observes failures from normal DSH model usage, and automatically rechecks unhealthy models.

> This is a third-party DSH plugin maintained by **AGSQ11**. It is not an official DeepSeek project.

## Screenshots

### Alive models

![DSH Model Probe — Alive models](docs/screenshots/model-probe-alive.png)

### Dead models and retry diagnostics

![DSH Model Probe — Dead models and retry diagnostics](docs/screenshots/model-probe-dead.png)

## Features

- Native DSH Web UI integration under **Settings → Model Probe**.
- **App-wide runtime picker health badges:** Alive models get a green dot and Dead models get a red dot in DSH model-selection lists.
- **Health-first model ordering:** runtime pickers show Alive models first, unknown/unprobed models next, and Dead models last; each tier is sorted alphabetically A→Z.
- **Fast model filtering:** the composer Model list gains a case-insensitive prefix filter over model display name and model ID (for example, `nem` → `nemotron-3-ultra`). DSH's `/model` popup keeps its existing native search field.
- Uses DSH semantic `--dsw-*` theme tokens and follows Light, Dark, and System themes.
- Higher-contrast diagnostic/secondary text for dense health data in both light and dark palettes.
- Enumerates every model advertised by every currently registered DSH provider adapter.
- **Alive / Dead** tabs with provider, model, latency, TTFT, result/error, attempt count, and next retry time.
- Manual **Probe all** with configurable timeout and concurrency.
- Persistent host-side health state that survives browser and DSH restarts.
- Runtime monitoring of normal DSH `llm/stream` calls.
- Two independent health retries after a real model error before declaring the model Dead.
- Dead models are rechecked automatically every **30 minutes**.
- Successful normal DSH traffic immediately revives a previously Dead model.
- Explicit user cancellation/abort is not treated as model death.
- Bounded health history (latest 500 outcomes).

## Health policy

### Manual probe

A manual probe has up to three attempts:

1. Initial minimal probe.
2. Retry #1 if the initial probe fails.
3. Retry #2 if retry #1 also fails.
4. The model is marked **Dead** only if all three attempts fail.

### Failure during normal DSH use

If a normal DSH request ends in a terminal provider/model error:

1. That failed user request counts as the initial failure.
2. Model Probe sends health retry #1.
3. If necessary, it sends health retry #2.
4. The model becomes **Dead** only if both health retries fail and no successful normal request has revived it in the meantime.

The plugin intentionally does **not** replay the user's original request. Replaying a partially streamed response could duplicate assistant output or tool calls. Health verification uses a separate, minimal request:

```text
Reply with exactly OK.
```

with `maxTokens=8`.

### Dead-model recovery

A Dead model receives one background health probe every 30 minutes:

- success → **Alive** immediately;
- failure → remains **Dead** and gets another check 30 minutes later.

The scheduler runs in the DSH host process, so the Model Probe page does not need to remain open. If DSH is stopped, overdue checks resume after DSH starts again.


## Runtime model-picker integration

Model Probe reuses its persistent provider+model health ledger in DSH's runtime model selectors:

- **green dot** — the exact provider/model is currently **Alive**;
- **red dot** — the exact provider/model is currently **Dead**;
- **no dot** — the model has not reached a known Alive/Dead state yet.

The composer **Model** pane gains a search box above the provider-grouped list. Matching is case-insensitive and prefix-based against both the displayed model name and the provider model ID. Typing `nem` therefore matches an ID such as `nemotron-3-ultra` even when its human-readable display name is different. Provider groups with no matches are hidden while the filter is active.

DSH's `/model` popup already provides its own local search input, so Model Probe does not replace it; it adds health dots to those rows and leaves DSH's native keyboard/search behavior intact.


### Health-first ordering

Runtime model lists are reorganized globally by health rather than retaining provider-first visual grouping:

1. **Alive** models (green dot), A→Z
2. **Unknown / not yet probed** models (no dot), A→Z
3. **Dead** models (red dot), A→Z

Provider context remains visible inline, and deterministic provider/model-id tie-breakers are used when display names are identical. The composer and `/model` picker use the same ordering. Filtering is applied on top of that ordering, so typing a prefix keeps the matching results health-sorted.

### Compatibility note

At the time of v0.3.0, DSH's public model-selection client face exposes the shared model directory plus load/select operations, but no third-party per-row decorator/header slot. To avoid replacing or forking DSH's selector, Model Probe's picker enhancer attaches only to accessibility semantics (`role=menu`, `role=group`, `role=menuitemradio`, `role=listbox`, and `role=option`) and deliberately does **not** depend on DSH's generated CSS-module class names. This is substantially less brittle than styling private class names, but a future DSH change to the selector's semantic markup could still require a plugin update.

The provider configuration/editor model rows under **Settings → Models** are intentionally not treated as runtime health selectors: those rows can describe dormant or not-yet-active routes that cannot truthfully be labeled Alive or Dead.

## Persistence

The host stores current health state and a bounded result history in `state.json`.

Default paths:

| Platform | Path |
| --- | --- |
| Windows | `%LOCALAPPDATA%\\dsh-model-probe\\state.json` |
| macOS | `~/Library/Application Support/dsh-model-probe/state.json` |
| Linux/Unix | `$XDG_STATE_HOME/dsh-model-probe/state.json` or `~/.local/state/dsh-model-probe/state.json` |

Override the location with:

```text
DSH_MODEL_PROBE_STATE_FILE=/custom/path/state.json
```

No provider credentials are stored by Model Probe. Probe requests use the provider adapters and credentials already configured in DSH.

## Install

Clone or download this repository, then add the local plugin directory to the DSH profile:

```bash
dsh plugin --profile default add ./dsh-model-probe
```

Replace `default` with the profile actually in use, restart DSH, and open:

```text
Settings → Model Probe
```

## Update

Replace the plugin directory with the newer release/source, keep the same DSH profile entry, and restart DSH. Persistent health state is kept separately from the plugin directory.

## Model discovery limitation

DSH provider catalogs are advisory: an adapter may accept model IDs that it does not advertise. Model Probe therefore checks **all advertised models**, but it cannot infer arbitrary hidden model IDs that no provider exposes through its catalog.

Health is keyed by **provider + model**, not model name alone. For example, the same model ID can be Alive on one provider and Dead on another.

## Development

The distributed package is intentionally small and prebuilt. The browser and host entry points live in both `src/` and `lib/` so DSH can load the package directly.

Run the test suite with:

```bash
npm test
```

The tests cover failure classification, persistence, manual retry policy, runtime-error verification, dead-model scheduling/recovery behavior, API routes, DSH theme integration, picker health resolution, ambiguity handling, and prefix filtering.

## License

MIT © 2026 **AGSQ11**. See [LICENSE](LICENSE).