dsh-pc-pilot

Readme

# dsh-pc-pilot

English | [中文](./README.zh.md)

[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
![Platform: Windows](https://img.shields.io/badge/platform-Windows%2010%2F11-lightgrey)
![Node](https://img.shields.io/badge/node-%E2%89%A522.12-green)
[![npm](https://img.shields.io/npm/v/dsh-pc-pilot.svg)](https://www.npmjs.com/package/dsh-pc-pilot)
[![CI](https://github.com/JeremyWangCY/dsh-pc-pilot/actions/workflows/ci.yml/badge.svg)](https://github.com/JeremyWangCY/dsh-pc-pilot/actions/workflows/ci.yml)

A **[DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) host plugin** that gives the model a single `computer` tool to observe and operate the local Windows desktop: an indexed UIA accessibility tree, per-window screenshots, background synthetic-cursor input that never steals focus, and — when a task truly requires it — real SendInput mouse/keyboard control.

While acting, the model moves a small **on-screen cursor indicator** (a rounded arrow with a soft blue radial glow) to each target point, and a frosted-glass status pill at the top center of the screen shows that PC-Pilot is running (with a breathing green dot on the right). Both are click-through, never take focus, and auto-hide three seconds after the last action.

## Runtime scope

**PC-Pilot targets one interactive Windows session: the user keeps working normally while the AI prefers background-safe paths.** Browser CDP, UIA patterns, target-window messages, WGC and PrintWindow are used to avoid focus theft whenever the application supports them. Virtual machines, a second desktop, or a hidden alternate Windows session are not part of the architecture. If an application truly requires real foreground SendInput, PC-Pilot reports that its background path is unavailable rather than pretending independent user/AI input can be guaranteed.

Optimization priority: **browser batched/local observation → UIA caching and capability routing → reusable WGC capture sessions → conditional batching.** Capture support and background-input support are evaluated separately: being able to capture a covered window does not imply that the same application can accept safe background input. Application-specific object models, automation APIs, and domain integrations belong in separate plugins rather than PC-Pilot core.

## Features

- **One tool, full desktop plus browser** — the desktop surface uses canonical Windows Computer Use names directly: `list_apps`, `list_windows`, `get_window`, `launch_app`, `get_window_state`, `click`, `press_key`, `type_text`, `scroll`, `drag`, `set_value`, `perform_secondary_action`, `activate_window`, and `minimize_window`.
- **Reusable Windows targets** — observations return `window: { id, app }`, which can be supplied unchanged to the next action. Element actions use `element_index`; screenshot coordinates use `scrollX` / `scrollY` and `mouse_button` where applicable.
- **Background-first input with capability routing** — observations cache per-window/per-control support for UIA patterns (Invoke / Value / Toggle / Selection / ExpandCollapse / Scroll / RangeValue). Known-unsupported paths are skipped instead of reprobed on every action; verified target-window `WM_CHAR` / `WM_KEY` / `WM_MOUSEWHEEL` messages cover traditional controls without consulting an occluding foreground window. A fresh UIA observation rebuilds the capability map when the window/control tree changes.
- **State-bound actions** — `get_window_state` returns a `snapshot_id` and `screenshot_id`; element actions must present that snapshot and coordinate actions can bind to the screenshot. Expired, moved, wrong-window, changed-element, or consumed state is rejected instead of falling back to a potentially wrong control.
- **Persistent browser-use session** — one bounded CDP WebSocket is reused per isolated AI browser endpoint, with reusable per-tab sessions rather than reconnecting for every action. `browser_tabs`, `browser_history`, `browser_back` / `browser_forward`, and condition-based `browser_wait` provide session-level navigation; `browser_events` returns cursor-based console/network/lifecycle evidence and `browser_downloads` tracks Chromium download progress plus completed files. Screenshot observation has its own non-fatal timeout so a slow frame cannot tear down an otherwise healthy browser session. `browser_state` preserves raw tokens plus compact `@eN` refs for compatibility, while `browser_observe` returns the compact refs without UUID token noise; both remain bound to the exact tab/document/name/role. `browser_click`, `browser_type`, `browser_replace`, and `browser_key` accept either form. `launch_app { app: "msedge.exe", headless: true }` still starts a fresh isolated profile and `browser_*` never attaches to the user's own browser.
- **Stable UIA identity + incremental state** — `include_text: true` assigns every returned control a stable `element_id`, plus monotonic `accessibility_revision` and `accessibility_delta` metadata (`added`, `removed`, `changed`, `unchanged_count`). Existing snapshot-bound `element_index` actions remain fully compatible; the stable identity is for reasoning across observations, not for bypassing stale-snapshot checks.
- **Verified postconditions and deterministic recovery** — actions can carry an `expect` contract for window presence/closure, accessibility change, stable-element value, desktop text, browser URL/text/readiness, or completed downloads. A failed postcondition is reported as `postcondition_failed` and is never blindly replayed. `recovery: "foreground_once"` is intentionally narrow: it may retry only after an explicit `not_executed + background_unavailable`; element recovery requires the observed stable `element_id` so PC-Pilot can re-observe and remap a fresh index/snapshot first.
- **Stable app identity and exact targeting** — `list_apps`, `list_windows`, and `get_window` expose an `app_identity` with Win32 executable identity or packaged-app AUMID, parent pid, and same-process-family root pid. Reuse `identity_key` to target a known app exactly instead of relying on title/process-name matching. `get_app_identity` lazily adds product/version/company metadata and optionally verifies Authenticode signer publisher/subject/thumbprint with executable-level caching.
- **Bounded failure semantics** — one-shot helper calls have an external deadline watchdog. A timeout, disconnect, or post-dispatch transport error reports `outcome: "unknown"`; mutating actions are never replayed automatically.
- **Lightweight conditional batching** — send up to 20 ordered actions through `actions`. Any step may use `when` as a precondition and `expect` as a verified postcondition; the next step runs only after the current gate is verified. `when` checks current state once by default (`timeout_ms: 0`) and never dispatches the action when unmet. There is deliberately no branching/workflow DSL or hidden replay engine; execution still stops at the first failed, unmet, or uncertain step.
- **Structured safety classification** — obvious consequential target names and sensitive browser fields are classified in the result for future host policy integration; the current PC-Pilot profile does not interpose confirmation.
- **Occlusion-immune background clicks** — with an `app` specified, coordinate clicks aim at the target window's own UIA tree / hwnd, so a fully covered window can be operated unattended while the user keeps working on top.
- **Rich mouse vocabulary** — `click` supports standard `left`, `right`, `wheel` (middle), `back`, and `forward` buttons plus a bounded `click_count`; `scroll` preserves simultaneous `scrollX` / `scrollY` axes; `drag` uses `path`.
- **Computer-use action spelling compatibility** — `double_click`, `type`, `keypress { keys: [...] }`, and `move` are accepted alongside the Windows canonical `click`, `type_text`, `press_key`, and `mouse_move` actions.
- **O(1) element lookup** — `get_window_state` caches the UIA element list inside the persistent helper daemon, so `click { element_index }`, `set_value`, `perform_secondary_action`, `select_text`, and `type_text` resolve without a second full-tree traversal.
- **Occlusion-immune screenshots with reusable WGC sessions** — a bundled Windows Graphics Capture bridge captures the target HWND even when it is covered and reuses a bounded `GraphicsCaptureItem + FramePool + CaptureSession` slot per HWND. Window-size changes recreate only the frame pool; closed, failed, or idle slots are evicted. `PrintWindow` (multi-flag ladder) remains the occlusion-immune fallback. Neither tier copies the visible screen, so a covering window can never leak into the shot; when neither can render, the result is a legible `screenshot_black` error instead of a frame of the occluder. The bridge activates when .NET 8 is available; PrintWindow keeps the plugin usable without it.
- **Per-task dispatch** — `dispatch: "foreground"` (real SendInput) exists for the cases that genuinely need it (canvas clicks, unsupported drags, apps with no background path); the tool guidance keeps background as the default and asks the model to be explicit when it goes foreground.
- **Virtual-cursor indicator** — per-pixel-alpha layered window (`UpdateLayeredWindow` + `CreateDIBSection`): rounded white arrow with black outline over a soft blue radial glow. Click-through (`WS_EX_TRANSPARENT`), non-activating (`WS_EX_NOACTIVATE` + `SW_SHOWNOACTIVATE`), always-on-top. Auto-hides 3 s after the last action and reappears on the next one.
- **High-DPI accurate** — the overlay calls `SetProcessDPIAware` at startup and positions itself in physical pixels, matching the physical coordinates the helper reports from UIA. Correct placement at 100% / 125% / 150% scaling.
- **PowerShell 5.1 + 7 (Core) compatible** — the overlay adds the `System.Private.Windows.GdiPlus` / `System.Private.Windows.Core` references under .NET Core so both runtimes render the cursor identically.
- **Frosted status pill** — while the overlay is active, a dark frosted-glass pill sits top-center on the primary screen reading "PC-Pilot 运行中" with a breathing green dot on the right (Apple-style: rounded capsule, subtle top sheen, sine-wave breathing). Same per-pixel-alpha layered-window technique as the cursor, `TOPMOST` + `NOACTIVATE` + click-through, so it never intercepts input; it hides 4 s after the last activity.

## Requirements

- Windows 10 or 11
- DeepSeek Harness (DSH) with the `dsh-pc-pilot` bundle loaded in the `web` profile
- PowerShell 5.1 (built into Windows) and Node.js ≥ 22.12 (shipped with DSH)
- .NET 8 Desktop/Runtime is optional; when present, the bundled WGC bridge provides occlusion-independent native HWND screenshots. Without it, occlusion-immune `PrintWindow` capture keeps the plugin usable.

## Installation

Published on [npm](https://www.npmjs.com/package/dsh-pc-pilot). npm itself is included with Node.js.

### From the DSH plugin market

Once listed, search for *dsh-pc-pilot* in the market and click install.

### From npm

```powershell
npm install dsh-pc-pilot
```

Run this inside the DSH profile (`~/.dsh/profiles/web`), then restart the host.

### From source

```powershell
git clone https://github.com/JeremyWangCY/dsh-pc-pilot.git
cd dsh-pc-pilot
npm install ./dsh-pc-pilot
```

Or link it manually: add `"dsh-pc-pilot": "link:./vendor/dsh-pc-pilot"` to the profile's `package.json` dependencies, add the bundle to `dsh.profile.bundles`, run `npm install`, and restart the host.

## CLI and runtime API

PC-Pilot also exposes a small standalone runtime surface. The CLI is intentionally thin: it does not maintain an action whitelist, invent workflows, or retry uncertain mutations for the agent.

```powershell
npm install -g dsh-pc-pilot
pc-pilot status
pc-pilot doctor
pc-pilot list_apps --json
'{"action":"list_apps"}' | pc-pilot request --stdin --json
```

`act` / direct action mode is a convenience layer. `request` is the escape hatch for agents: it forwards a complete computer request, including batches and future fields, to the same core used by the DSH `computer` tool.

```js
import { createPcPilotRuntime } from 'dsh-pc-pilot/runtime'

const pc = createPcPilotRuntime()
const apps = await pc.act('list_apps')
const batch = await pc.run({ actions: [{ action: 'wait', seconds: 1 }] })

// Optional convenience only: bind a reusable target without creating a hidden session.
const launched = await pc.act('launch_app', { name: 'msedge.exe', headless: true })
const opened = await pc.act('browser_open', { browser: launched.browser, url: 'https://example.com' })
const page = pc.bind({ browser: opened.browser })
const observed = await page.act('browser_observe')
await page.act('browser_click', { browser_element: observed.elements[0].ref })

pc.close()
```

The runtime returns the core outcome unchanged. In particular, `outcome: "unknown"` is evidence for the agent to inspect state; the CLI does not turn it into an automatic replay. `runtime.bind(defaults)` is only a shallow request convenience: per-call fields win, nested `window` / `browser` targets merge, and no session daemon, lifecycle, page choice, observation, navigation, or retry is added behind the agent's back.

The browser backend is also a small injectable provider rather than a second policy layer. The default provider keeps the current persistent CDP session implementation; alternate providers only need an `execute(action, args, signal)` function. This leaves room for a BrowserSkill-compatible backend without forcing its workflow onto every agent.

## Status pill

While the overlay is enabled (default), each first action launches two tiny resident PowerShell loops from the helper directory: the virtual-cursor indicator and the frosted status pill. They read a state file under `%TEMP%\dsh-cua` — the pill is visible top-center while activity is fresh (≤ 4 s) and fades out afterwards; both processes idle-exit after 120 s and respawn on demand. No driver, no UAC, no display changes — PC-Pilot runs entirely on the user's real desktop, in the background.

## Usage

The plugin registers one global tool, `computer`. Typical flow:

1. `computer { action: "list_apps" }` — running apps with pids, exact `app_identity` / `identity_key`, window titles, hwnds and rects.
2. `computer { action: "get_window_state", window: { id, app }, include_screenshot: true, include_text: true }` — indexed accessibility tree with stable `element_id`, revision/delta metadata, a window screenshot and `snapshot_id`.
3. Act on the state — element actions include the `snapshot_id` from the same observation. Browser results return a reusable `browser: { endpoint, tab_id }` target that can be passed back unchanged; element actions can use either the raw `browser_element` token or the short `@eN` ref returned by `browser_state` / `browser_observe`.
4. Observe again only when state is stale/unknown, the target changed, or the next decision needs information you do not already have. Desktop `element_index` values remain bound to the `get_window_state` that produced them.

### Action reference (57 actions)

| Action | Purpose |
| --- | --- |
| `list_apps` / `list_windows` / `list_displays` | Enumerate apps with exact identity / per-app windows / display topology |
| `get_app_identity` | Resolve a process/window to stable Win32 path or packaged AUMID identity; lazily return product/version/company and optionally verified Authenticode signer evidence |
| `get_window_state` | Indexed UIA tree + stable element ids + revision/delta metadata + per-window PNG screenshot + document text |
| `click` | Coordinate click or snapshot-bound `element_index` click |
| `set_value` / `type_text` / `perform_secondary_action` / `select_text` | Element-level write, text entry, named UIA pattern, text-range selection |
| `press_key` / `hold_key` | Key chords and timed holds |
| `scroll` | Standard `scroll_x/scroll_y` (including target-window horizontal UIA scrolling), or legacy `amount` plus `direction` |
| `move` / `mouse_move` / `mouse_down` / `mouse_up` | Standard move plus raw mouse primitives |
| `drag` | Standard `path` of `{x, y}` points or legacy endpoints; element move (background) or real SendInput drag (foreground) |
| `screenshot` / `zoom` | Full-display or region capture; crop `screenshot_path` from a prior capture (`path` remains accepted by the runtime for compatibility) |
| `switch_display` / `cursor_position` | Default capture display; real cursor location |
| `launch_app` / `wait` | Launch an app behind the active work without activating it; the window stays normally renderable for WGC/UIA instead of remaining minimized. Registered Windows activation protocols such as `ms-settings:display` are supported. Delegated app launches return a target only when exactly one new window is safely identifiable; pause between actions |
| `activate_window` / `minimize_window` / `close_window` / `get_window` | Bring window to foreground / minimize it directly with Win32 (no title-bar coordinates) / request a graceful WM_CLOSE and verify disappearance (otherwise returns `window_close_unconfirmed`) / query fresh window geometry & metadata |
| `read_clipboard` / `write_clipboard` | Clipboard round-trip |
| `browser_tabs` / `browser_state` / `browser_observe` / `browser_history` / `browser_back` / `browser_forward` / `browser_wait` | Manage exact tabs; `browser_observe` gives compact ref-only semantic state while `browser_state` keeps raw tokens for compatibility; navigate history and wait on page readiness/URL/text without synthetic sleeps |
| `browser_events` / `browser_downloads` | Cursor-based console/network/lifecycle evidence; Chromium download progress and completed AI-profile files |
| `browser_shutdown` | Close the entire browser only when this PC-Pilot tool instance launched it |
| `browser_click` / `browser_type` / `browser_replace` / `browser_key` | Operate a raw token or compact `@eN` ref from the latest state/observe result; stale or changed targets are rejected |
| `browser_click_point` | Click browser-viewport coordinates only when bound to an exact `screenshot_id`; both explicit observations and post-action screenshots remain chainable for 30 seconds while the tab/document/URL identity is unchanged |
| `browser_upload` | Select 1–20 explicit absolute local files on an already-observed `<input type=file>` and verify the browser received them; does not submit the surrounding form |

### Key parameters

| Parameter | Default | Notes |
| --- | --- | --- |
| `dispatch` | `background` | UIA patterns + window messages; never steals focus. `foreground` uses real SendInput — pick it per task only when the user asked for real control or the essential action has no background path. |
| `activate` | `false` | `launch_app` only: normal foreground launch. The default keeps the new window renderable but non-activated and places it behind the active work; use `true` only when the user explicitly wants it brought forward. |
| `overlay` | `true` | Show the click-through cursor at each action point; it auto-hides 3 s after the last action. |
| `include_screenshot` | `true` | Capture a per-window PNG in `get_window_state`. |
| `include_text` | `false` | Include the indexed accessibility tree and document text when an element action is needed. On a window-targeted `wait`, include it in the post-wait observation to check application readiness without a second round-trip. |
| `wait_for` | — | On a window-targeted `wait`, wait for `accessibility_present` (any UIA descendant) or `accessibility_available` (a complete UIA tree). A timeout is an explicit, retry-safe `wait_condition_timeout`. |
| `app` | — | pid number, process name, or window-title substring; same-titled duplicate windows are rejected unless `window_index` or `hwnd` identifies one, while differently titled windows of one app auto-resolve and return `chosen_hwnd`. |
| `identity_key` | — | Exact application identity returned by discovery. Packaged apps use `aumid:<AUMID>`; Win32 apps use `win32:<full executable path>`. Prefer it for continuation targeting once an app is known. |
| `verify_signature` | `true` | `get_app_identity` only: verify Authenticode signer/publisher and cache the result by executable path. |
| `snapshot_id` | — | Required for desktop element actions; use the id from the latest `get_window_state { include_text: true }`. |
| `element_id` | — | Stable UIA identity returned by `get_window_state`; optional for normal actions, required for safe element remapping during `foreground_once` recovery. |
| `expect` | — | Optional postcondition: verify window state, accessibility change, element value, text, browser URL/text/readiness, or completed download after the action. |
| `when` | — | Optional precondition using the same condition vocabulary as `expect`; checked before dispatch, with default `timeout_ms: 0`. If unmet the action is `not_executed`. Useful as a lightweight gate inside short `actions` batches. |
| `recovery` | `none` | `foreground_once` retries only a conclusively non-executed `background_unavailable` action. Unknown outcomes are never replayed. |
| `browser` / `browser_endpoint` / `tab_id` / `browser_element` / `event_cursor` | — | Prefer the reusable `browser: { endpoint, tab_id }` target returned by PC-Pilot; the separate endpoint/tab fields remain compatible. `browser_element` accepts a raw semantic token or compact `@eN` ref. |
| `x` / `y` | — | Window-local pixels with `app`/`hwnd`, matching the Computer Use coordinate model; screen coordinates without a target. Set `coordinate_space: "screen"` only for an explicit absolute click. |
| `button` / `click_count` / `keys` | `left` / `1` / — | Mouse button and legacy click repetitions; `keys` supplies standard keypress chords and mouse modifiers. Foreground mouse actions and validated native-window background clicks preserve the modifier state; unsupported background paths report `background_unavailable`. |
| `actions` | — | Ordered batch of action objects (maximum 20). Steps may use `when` before dispatch and `expect` after dispatch; the batch stops at the first unmet/failed/uncertain step. Intended for short predictable sequences, not scripted branching. |
| `display` | primary | 1-based display index for `screenshot` / `switch_display`. |

## How it works

```
model ── computer tool ──> host (Node ESM bundle)
                              │  persistent PowerShell daemon, bounded one-shot fallback (JSON in / JSON out)
                              ├─> UIA accessibility tree (IUIAutomation)
                              ├─> Windows Graphics Capture → PrintWindow screenshots (occlusion-immune, no screen-DC tier)
                              ├─> background input: UIA patterns → pixel hit-test → WM_* messages
                              ├─> foreground input: SendInput
                              └─> overlay: UpdateLayeredWindow per-pixel-alpha layered window
```

The helper is a single self-contained `pc-pilot-helper.ps1` copied to `%TEMP%` once per host start; the overlay is `virtual-cursor-overlay.ps1` and the status pill is `pcpilot-statusbar.ps1` — resident low-frequency loops that read state files and blit per-pixel-alpha layered windows with `UpdateLayeredWindow`. Registration uses the harness tool API (`defineTool` + `ctx.tools.register`) with `isConcurrencySafe: false`, so desktop actions serialize.

## Security considerations

- The tool can read window titles, accessibility trees and screenshots, and can drive input into the user's applications. The bundled tool description instructs the model to operate **only** what the user explicitly asked for and to never submit forms, send messages, make purchases, delete data, or change account/settings without explicit instruction.
- Background actions never move the user's cursor or steal focus. Foreground actions do — the guidance requires the model to say so.
- No telemetry. The plugin performs no network access and no persistence beyond `%TEMP%\dsh-cua-*` state files.

## Troubleshooting

- **The cursor indicator does not appear** — check `%TEMP%\dsh-cua-diag.log` (boot diagnostics) and make sure the host was restarted after installation.
- **The indicator is visible but misplaced** — ensure the installed version calls `SetProcessDPIAware` (all ≥ 0.1.0 builds do); mismatched DPI awareness shifts the overlay by the scaling factor.
- **Desktop icons vanish / gray boxes appear** — this is a Windows shell (WorkerW) glitch typically caused by desktop-organizer or wallpaper tools, not by this plugin; restarting `explorer.exe` restores the desktop.
- **`background_unavailable`** — the target has no verified background path (canvas, some WinUI surfaces, unsupported native controls). Decide per task whether to go `foreground`; the helper refuses an unverified coordinate fallback.
- **`accessibility_status: unavailable`** — the HWND and screenshot are valid, but the application exposed no usable UI Automation descendants. Wait and observe again; if it remains unavailable, use a screenshot-bound foreground path only when the task permits it. Never invent an `element_index`.
- **`foreground_activation_unconfirmed`** — an explicitly foreground-targeted app did not become the foreground window after the verified activation attempt. No real input was sent; when `launch_succeeded: true` is returned, do not retry launch—observe the returned window instead.
- **`screenshot_black`** — the target is DirectComposition/UWP without the WGC bridge, hardware-accelerated, or hung, so neither occlusion-immune tier could render it. Bring the window forward with `dispatch: "foreground"` (or `activate_window`) and retry.

## Development

```powershell
git clone https://github.com/JeremyWangCY/dsh-pc-pilot.git
cd dsh-pc-pilot
pwsh -File scripts/smoke-test.ps1
# Rebuild the optional WGC bridge after changing native/wgc-capture:
dotnet publish native/wgc-capture/wgc-capture.csproj -c Release -r win-x64 --self-contained false -p:PublishSingleFile=true -o lib/wgc
```

The smoke test exercises helper actions (`list_apps`, `get_window_state`, background clicks) against a real window. To run the plugin from a local checkout, link it into a DSH profile as shown above.

## License

[MIT](LICENSE)


Background browser mode, helper source modules and timing: [2026-09-10 implementation notes](docs/background-refactor.zh.md).