dsh-webfetch

Readme

# dsh-webfetch

> 为 DeepSeek Harness 智能体装上「阅读器」：给定 URL，抓取网页并提取干净的 Markdown / 纯文本正文，附带链接清单与 RSS/Atom 订阅源解析。零运行时依赖，只读，不发送任何凭证。
> [English](#english) | 中文简介

A web page reader plugin for [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) (`dsh`).
`dsh` agents can search, but until now they could not *read the page behind a URL*.
`dsh-webfetch` closes that gap with four read-only tools and **zero runtime dependencies** (Node built-ins + global `fetch` only).

## Tools

### `web_fetch`

Fetch a web page and extract its readable content.

| Parameter     | Type                 | Default    | Description                                                    |
| ------------- | -------------------- | ---------- | -------------------------------------------------------------- |
| `url`         | string (required)    | —          | Full http/https URL of the page to fetch.                      |
| `format`      | `'markdown' \| 'text'` | `markdown` | Markdown keeps headings, links, lists and code fences; `text` is plain prose. |
| `extractLinks`| boolean              | `false`    | Also return every link found on the page (resolved, absolute). |
| `maxChars`    | number               | `50000`    | Cap on extracted content length (1000–200000).                 |

Returns `{ url, finalUrl, status, title, content, length, truncated, links? }`.

```text
user: what does the dsh README say about the architecture?
agent: web_fetch("https://github.com/deepseek-ai/deepseek-harness")
  → HTTP 200 — title: deepseek-harness
    ## DeepSeek Harness
    ..."everything is a plugin"...
```

### `web_links`

Collect every link on a page with its visible label, resolved to absolute
URLs, deduplicated, capped at `limit` (1–200, default 50). Useful for mapping
what a page points to or crawling site structure.

### `web_feed`

Read an RSS 2.0 or Atom feed and return its entries as a clean,
LLM-friendly listing. Feed URLs are XML documents that `web_fetch` cannot
extract from — `web_feed` parses them into structured entries instead.

| Parameter        | Type              | Default | Description                                                       |
| ---------------- | ----------------- | ------- | ----------------------------------------------------------------- |
| `url`            | string (required) | —       | Full http/https URL of the RSS or Atom feed.                      |
| `maxItems`       | number            | `10`    | Max entries to return (1–50).                                     |
| `includeContent` | boolean           | `false` | Also return each entry's full content (else summaries only).      |

Returns `{ url, finalUrl, status, feedTitle, entryCount, truncated, entries }`,
where each entry is `{ title, url, published?, author?, summary?, content? }`
with CDATA unwrapped, HTML entities decoded, markup stripped and relative
links resolved against the feed URL.

```text
user: what did the example blog post this week?
agent: web_feed("https://blog.example.com/feed.xml", maxItems: 5)
  → feed: Example Blog
    5 entries from https://blog.example.com/feed.xml
    1. First & foremost post — https://blog.example.com/posts/first
       published: Mon, 01 Jan 2024 10:00:00 GMT
       author: Alice
       Hello world — café & tea.
```

### `web_headers`

Inspect the HTTP status, response headers and redirect chain of a URL
**without downloading the page body** — the diagnostic companion to
`web_fetch`: check status codes, content types, redirects, caching or
security headers before (or instead of) pulling the content. Uses `HEAD`
by default and falls back to `GET` automatically when the server answers
405/501; unlike `web_fetch`, *any* status is reported (404/500/… included)
rather than thrown.

| Parameter         | Type              | Default | Description                                                              |
| ----------------- | ----------------- | ------- | ------------------------------------------------------------------------ |
| `url`             | string (required) | —       | Full http/https URL to inspect.                                          |
| `method`          | `'HEAD' \| 'GET'` | `HEAD`  | `HEAD` downloads no body; `GET` always works but transfers the body.     |
| `followRedirects` | boolean           | `true`  | Follow redirects and report every hop of the chain.                      |

Returns `{ url, finalUrl, status, statusText, method, headers, redirects }`
where `headers` is the full response-header map (lower-cased keys) and
`redirects` lists each hop as `{ url, status, location }`.

```text
user: is that download link still alive, and does it redirect?
agent: web_headers("https://example.com/downloads/latest")
  → HTTP 200 OK — HEAD https://example.com/downloads/latest
    redirect chain:
      1. 301 https://example.com/downloads/latest → https://cdn.example.com/releases/latest.zip
    final URL: https://cdn.example.com/releases/latest.zip
    12 header(s):
      content-type: application/zip
      content-length: 4821337
      ...
```

## Install

```sh
dsh plugin --profile web add github:TYEclipse/dsh-webfetch
# or a pinned release:
dsh plugin --profile web add github:TYEclipse/dsh-webfetch#v0.3.0
```

Restart your agent session and the tools are available to the model.

## Configuration

All settings are optional (defaults shown):

```yaml
plugins:
  dsh-webfetch:
    timeoutMs: 10000        # per-request timeout (1000–60000)
    maxBytes: 1500000       # response size cap in bytes (10000–5000000)
    maxChars: 50000         # extracted content cap in chars (1000–200000)
    maxRedirects: 3         # redirect hops to follow (0–10)
    userAgent: "dsh-webfetch/0.3 (DeepSeek Harness plugin)"
```

## Proxy support

Node's built-in `fetch` ignores `HTTP_PROXY`/`HTTPS_PROXY`, so on networks
that require a proxy every request would fail. dsh-webfetch ships a
zero-dependency http-proxy transport (CONNECT tunnelling for https,
absolute-URI form for http, `NO_PROXY` matching with wildcard and IPv4 CIDR
support) and uses it automatically:

- `httpProxy` / `httpsProxy` (default: `HTTP_PROXY` / `HTTPS_PROXY` env) —
  `http://host:port` URL; set to an empty string to disable.
- `noProxy` (default: `NO_PROXY` env) — comma-separated bypass list
  (exact hosts, `.suffix`, `*.wildcard`, IPv4 CIDRs, optional `:port`).

```yaml
plugins:
  dsh-webfetch:
    httpsProxy: "http://127.0.0.1:7897"   # override env
    noProxy: "localhost,.internal,10.0.0.0/8"
```

Proxy credentials embedded in the proxy URL are sent as
`Proxy-Authorization: Basic` (to the proxy only, never to the target).

## Safety model

- **http/https only** — `file:`, `ftp:`, `javascript:` and friends are rejected.
- **No credentials ever** — URLs with embedded credentials are rejected; no
  cookies or authorization headers are attached; nothing is persisted.
- **Bounded everything** — hard timeout per request, redirect hop limit,
  response size cap, extracted-text cap. Oversized bodies are cut off and
  flagged (`truncated: true`), never buffered past the cap.
- **Content-type gated** — `web_fetch`/`web_links` parse only `text/html` and
  `text/plain`; `web_feed` additionally accepts `application/rss+xml`,
  `application/atom+xml`, `application/xml` and `text/xml`. Scripts, styles,
  comments and embedded content are stripped by the extractor/parser.
- **Charset-aware** — honours the `Content-Type` charset, falls back to XML
  declaration / `<meta charset>` sniffing, then UTF-8.

## Development

```sh
pnpm install
pnpm build      # tsc
pnpm test       # vitest — 97 tests, fully offline (local fixture servers)
pnpm lint       # oxlint src test
```

## License

[MIT](LICENSE)

---

## 中文简介

dsh-webfetch 是 DeepSeek Harness 的网页阅读插件：智能体拿到 URL 后可以直接抓取页面并提取干净的 Markdown 或纯文本（保留标题、链接、列表与代码块，剥离脚本/样式），`web_links` 可列出页面全部链接（解析为绝对地址、去重、限量），`web_feed` 可解析 RSS 2.0 / Atom 订阅源为条目清单（标题/链接/发布时间/作者/摘要/正文，处理 CDATA、HTML 实体与相对链接），`web_headers` 可用 HEAD 请求探测任意 URL 的 HTTP 状态码、响应头与重定向链而不下载正文（服务器不支持 HEAD 时自动回退 GET，非 2xx 状态照常报告）。零运行时依赖、只读、不发送凭证；http/https 协议限定、超时/重定向/体积/文本长度全部有上限，字符集自动识别（Content-Type → XML 声明/meta → UTF-8）；内置零依赖 http 代理支持（CONNECT 隧道 + NO_PROXY 白名单，自动读环境变量），在必须走代理的网络也能正常工作。与内置搜索互补：搜索给线索，webfetch 读正文，web_headers 读前诊断。