@wangyong1972/dsh-computer-use-macos

Readme

# @wangyong1972/dsh-computer-use-macos

[English](#english) | [中文](./README.zh.md)

[![npm version](https://img.shields.io/npm/v/@wangyong1972/dsh-computer-use-macos)](https://www.npmjs.com/package/@wangyong1972/dsh-computer-use-macos)
[![npm downloads](https://img.shields.io/npm/dm/@wangyong1972/dsh-computer-use-macos)](https://www.npmjs.com/package/@wangyong1972/dsh-computer-use-macos)
[![license](https://img.shields.io/npm/l/@wangyong1972/dsh-computer-use-macos)](./LICENSE)

<a id="english"></a>

A [DeepSeek Harness (DSH)](https://github.com/deepseek-ai) plugin that gives
models a native macOS implementation of Anthropic's "Computer Use"
capability: one `computer` tool that can screenshot the real display and
drive the real mouse/keyboard, using only binaries that ship with macOS
(`screencapture`, `sips`, `osascript`) — **zero installed dependencies, zero
network calls, zero compiled helper binaries**.

## What it does

The plugin registers a single tool, `computer`, that lets the model:

- take a screenshot of the real screen and see it inline in the conversation;
- move the mouse, left-/right-/double-click at a pixel coordinate;
- type text and send key combos (`Return`, `cmd+c`, `ctrl+shift+t`, ...);
- scroll up/down/left/right;
- read the current cursor position (best-effort);
- wait for a short, bounded duration.

Every OS interaction goes through `execFile('screencapture', [...])`,
`execFile('sips', [...])`, or `execFile('osascript', ['-l', 'JavaScript', ...])`
with argv arrays — never a shell string built from model input. See
"Security posture" below.

**Screenshots are automatically downscaled** before being sent to the model:
any capture whose longest edge exceeds 1568px is resized in place (via the
macOS-native `sips -Z`, preserving aspect ratio) so it stays comfortably
under common vision-model per-image pixel limits — this is what makes
`screenshot` work at all on a Retina or 6K display, whose raw capture would
otherwise be rejected outright by many hosts with
"Image exceeds the configured per-side pixel limit." **No setting changes
this target size.** If on-screen text or controls become too small to read
reliably in the downscaled image, increase the size of that text/UI *in the
app being controlled* (zoom in, increase font size, etc.) rather than trying
to change the display's resolution or this plugin's downscale target —
matching [Claude Code's own documented
behavior](https://code.claude.com/docs/zh-CN/computer-use) for its native
macOS computer-use tool. Every `coordinate` the model reads or supplies is
always in this *downscaled* screenshot's own pixel space, never the
display's raw native pixel space — the tool converts internally.

**Claude-Code-parity UX features** (all individually toggleable, see the
config table below): a machine-wide lock so a second DSH process on the same
Mac can't drive the mouse/keyboard concurrently; optionally hiding other
visible apps while acting; optionally excluding the host/terminal from the
screenshot it takes of itself; macOS notifications when the tool starts and
stops acting; and extra warning text in the approval prompt when the
frontmost app is a terminal/IDE, Finder, or a system-settings app. A global
Esc-to-abort hotkey was investigated and found infeasible under this
plugin's architecture — see `DESIGN.md` §10.5 for the full analysis.

## Requirements

- macOS only (`"os": ["darwin"]` — installation fails loudly on any other
  platform).
- A DSH host with the `tools` and `attachments` services mounted (both are
  standard in a DSH web/desktop profile).
- A vision-capable model route for the `screenshot` action specifically (a
  text-only route gets a clear error instead of a wasted screenshot).

## One-time macOS permission setup

macOS requires two manual, one-time permission grants before this plugin can
do anything. **This plugin cannot grant these to itself** — macOS
deliberately makes TCC (privacy) grants un-scriptable from the requesting
process. This is the same kind of one-time dance you already went through for
apps needing Full Disk Access or Screen Recording.

1. **Accessibility** — required for every mouse/keyboard action
   (`mouse_move`, `left_click`, `right_click`, `double_click`, `type`, `key`,
   `scroll`, and best-effort for `cursor_position`).
   Open **System Settings → Privacy & Security → Accessibility** and enable
   the relevant app.
2. **Screen Recording** — required for `screenshot` to return real pixels
   instead of a blank image (recent macOS versions silently return a blank
   capture, with no error, when this permission is missing).
   Open **System Settings → Privacy & Security → Screen Recording** and
   enable the relevant app.

**Which app shows up in those lists depends on how DSH was launched.** Every
mouse/keyboard/screenshot action in this plugin runs through the `osascript`
binary (or `screencapture` for screenshots), so TCC attributes the
permission to whichever process actually executed it — in practice, this is
either `osascript` itself, or your terminal/host app in the responsible-
process chain, depending on macOS version and how DSH was started. **Trigger
one `computer` action first** (it will fail with a clear message naming the
missing permission), *then* check System Settings — the correct entry only
appears in that list after the first attempt. Look for `osascript`,
`Terminal`, or your terminal app.

Once both permissions are granted, retry the failed action.

## Installing into a DSH profile

**npm (recommended):**

```sh
dsh plugin --profile <name> add @wangyong1972/dsh-computer-use-macos
```

Published package: [@wangyong1972/dsh-computer-use-macos](https://www.npmjs.com/package/@wangyong1972/dsh-computer-use-macos)

**Local checkout (development):**

```sh
dsh plugin --profile <name> add /path/to/dsh-computer-use-macos
```

This plugin ships its own `cordis.patch.yml` (referenced from
`package.json`'s `dsh.bundle.patch` field), so adding it registers the
`computer-use-macos` plugin with sensible defaults automatically.

## Config reference

| field | default | description |
|---|---|---|
| `enabled` | `true` | Master switch. When `false`, the `computer` tool is not registered at all — zero prompt-token cost. |
| `requireConfirmation` | `true` | Route every mutating action (click/move/type/key/scroll) through the approval seam before executing. `screenshot`/`cursor_position`/`wait` are never gated. Fails closed (denies) when no approval service is mounted. |
| `allowedDisplayIndex` | `0` | Backward-compatible default 0-based display index used whenever an action omits `display`. Per-action selection can override it without changing config. |
| `screenshotFormat` | `'png'` | Reserved for future formats; only `png` is supported in v1. |
| `actionTimeoutMs` | `10000` | Hard timeout for every `screencapture`/`sips`/`osascript`/`ps` subprocess call. |
| `maxTypeTextLength` | `4096` | Reject `action=type` calls whose text exceeds this many UTF-16 code units (rejected, never truncated). |
| `enableMachineLock` | `true` | Hold a machine-wide lockfile while a mutating action runs, so a second DSH process on this Mac can't drive the real mouse/keyboard concurrently. A stale lock (dead holder, or held too long) is reclaimed automatically. |
| `hideOtherAppsWhileActing` | `false` | Hide every other visible app while a mutating action runs, keeping only this plugin's own host process visible, then restore exactly what was hidden. Off by default — visibly intrusive, opt in deliberately. |
| `excludeHostFromScreenshots` | `false` | Momentarily hide the host process for just the `screencapture` call so `screenshot` never shows the terminal instead of the real target app. Off by default for the same reason as above. |
| `enableSessionNotifications` | `true` | Post a macOS notification when the tool starts/stops acting. Best-effort; never fails the underlying action. |
| `enableAppRiskWarnings` | `true` | Add extra warning text to the approval prompt when the frontmost app is a terminal/IDE, Finder, or a system-settings app. Purely additive — never changes what gets approved, no effect when `requireConfirmation` is `false`. |

## Click diagnostics and logging

Every `mouse_move`/`left_click`/`right_click`/`double_click` call emits one
structured, privacy-safe diagnostic record via DSH's own `ctx.logger()`
facade (never a bespoke log file, never `console.*`) — named
`computer-use-macos`, so a host's own log exporter can filter on it. Example
line (real format, live-verified against a real `@deepseek-ai/cordis`
`Context`):

```
computer.left_click display=0 pixel=[500,400] point=(500,400) pid=69234 exit=0 elapsed=112ms outcome=ok cursorVerified=true
```

Fields: the action, `allowedDisplayIndex`, the requested pixel coordinate,
the resolved Quartz point, the `osascript` subprocess's pid/exit code,
elapsed time, a coarse outcome classification (`ok` /
`accessibility-denied` / `timeout` / `aborted` / `other-error`), and whether
a post-action cursor-position self-check confirmed the cursor actually
arrived at the requested point. **Never logged**: raw stderr/stdout text,
typed `text`, or key-combo content — see `DESIGN.md` §11.2 for the full
privacy contract. `warn`-level for anything other than a clean, verified
outcome; `info`-level otherwise.

**What this can and cannot prove**: the cursor self-check confirms a click's
underlying `CGEventPost` call reached the OS and the cursor warped to the
right screen point — it cannot confirm the target application actually
received or reacted to the click as a click (that would need per-app
accessibility-tree introspection, out of scope; see `DESIGN.md` §11.3). Take
a follow-up `screenshot` to confirm a click's actual on-screen effect.

## Supported actions (v1)

`list_displays`, `screenshot`, `left_click`, `right_click`, `double_click`,
`mouse_move`, `type`, `key`, `scroll`, `wait`, `cursor_position` (best-effort).

### Selecting a display

Call `list_displays` to obtain each active display's 0-based `index`, numeric
`CGDirectDisplayID` (`id`), main-display flag, Quartz origin/size, backing
scale, and explicit 1-based `screenshotOrdinal`. `screenshot`, `cursor_position`, mouse move/click actions, and `scroll`
accept an optional `display` field:

- positional strings: `main`, `leftmost`, `rightmost`, `topmost`, `bottommost`;
- a JSON number for a 0-based active-display index;
- a decimal string for a `CGDirectDisplayID`, optionally prefixed with `id:`.

If `display` is omitted, the existing `allowedDisplayIndex` behavior is
preserved. The inventory is refreshed for every call, invalid/out-of-range
selectors fail before any OS input event, and coordinates remain local to the
selected display's screenshot space. `main` must resolve to exactly one display;
a positional extreme shared by two displays is ambiguous and fails closed.
For `scroll` without `coordinate`, the `display` selector is accepted but does
not move the pointer; scrolling occurs at the current cursor position as before.

**Screenshot identity limitation:** macOS `screencapture -D` accepts only a
1-based ordinal, not a `CGDirectDisplayID`. `screenshotOrdinal` is the best-known
mapping derived from the same `NSScreen` inventory and is passed explicitly; it
is not an identity-safe guarantee that Apple will keep `NSScreen` and
`screencapture` ordering identical. Mouse/cursor geometry is ID-backed through
`CGDisplayBounds`, but callers must not treat CG-ID-selected screenshots as
absolutely identity-safe until the backend is replaced with direct CoreGraphics
capture.

**Not yet supported (deferred to v2):** `left_click_drag`, `middle_click`,
`triple_click`, `hold_key`, `zoom`, compensating for the macOS "natural
scrolling" preference, and per-window/per-app (accessibility-tree) targeting.
Please don't file bugs for these — they're known, intentional v1 scope cuts,
not oversights.

## Security posture

- **No shell-string execution anywhere.** Every OS command is invoked via
  Node's `execFile` with an argv array, and every mouse/keyboard action's
  untrusted input (`text`, `coordinate`) crosses the JXA subprocess boundary
  as one JSON string in `argv`, never as interpolated AppleScript or shell
  source text.
- **No network calls of any kind, anywhere in this plugin.**
- **All numeric input is validated and clamped** against the real,
  freshly-queried screen bounds before any mouse/keyboard event is
  synthesized — never trusted from the model, even if the model is only
  echoing a coordinate it was previously shown.
- **No arbitrary file path acceptance.** The only filesystem paths this
  plugin ever touches are its own randomly-named temp screenshot file
  (created, optionally downscaled in place via `sips`, and deleted within the
  same tool call) and a single fixed-path machine-wide lockfile (§10.1).
- **The machine-wide lock fails closed on contention** — a second process
  holding it is a hard denial, never a silent proceed; stale-lock reclaim is
  bounded by both PID-liveness and an absolute age ceiling.
- **App-hiding always restores exactly what it itself hid**, never a blanket
  "show everything," so an app the user had already hidden stays hidden.
- **Per-app risk warnings are purely additive** to the existing approval
  prompt — they can never turn an approval "ask" into an automatic "allow,"
  and the risk table is fixed, not model- or config-editable.
- **Every mutating action is gate-able** via `requireConfirmation`, which
  fails closed (denies) when no approval service is mounted — never fails
  open.
- **`enabled: false` fully disables** the plugin: no tool registration, zero
  token cost.
- **No `postinstall`/`preinstall` scripts**, ever.
- **`peerDependencies`, not `dependencies`**, for every `@deepseek-ai/*`
  package this plugin composes against.

## Development

```sh
pnpm install
pnpm run build      # tsc -> lib/, then copy the JXA helper verbatim
pnpm run typecheck   # tsc --noEmit
pnpm run verify      # static checks, no macOS side effects, safe for CI
pnpm test            # unit tests over lib/ (key-spec, validate, permission-errors, screenshot-scale, lock, app-tiers, notify)
```

Everything requiring real Accessibility/Screen Recording permissions is
covered by the manual checklist in `tests/e2e.manual.md`.

## License

MIT