Skip to content
Novaterra

web.fetch

Schema

Input = {
  url: string,                 // absolute http(s) URL
  maxChars?: number,           // 500-200000, default 12000
  preferProxy?: boolean,       // default false — skip the direct fetch, go straight to the proxy
}


Output = {
  url: string,
  title: string,
  markdown: string,
  links: Array<{ text: string, href: string }>,   // up to 60
  extractor: string,           // 'readability' | 'text' | 'jina'
  truncated: boolean,
  status?: number,
}

How it reads a page

  1. Direct fetch (browser-like User-Agent, 15s timeout, 3MB cap) — if the response is HTML, it’s parsed with a jsdom document (scripts/styles/nav/footer/aside/header/forms stripped first), then @mozilla/readability extracts the main article, and Turndown converts that to markdown (ATX headings, fenced code blocks). If Readability can’t find at least 200 characters of real content, the whole <body> is turned to markdown instead (extractor: 'text') rather than returning nothing.
  2. Plain text/JSON/XML responses skip HTML extraction entirely and are returned as-is (still truncated to maxChars).
  3. r.jina.ai proxy fallback — used whenever the direct fetch fails, is blocked, returns an unsupported content type, or yields too little text. r.jina.ai/<url> returns pre-cleaned markdown directly; its Title: / Markdown Content: sections are parsed apart, and links are pulled out of the markdown itself with a regex since there’s no DOM to query.

The skill only throws if both the direct fetch and the proxy fail — every partial failure along the way is recorded in an internal notes list that becomes the thrown error’s message.

Example

Terminal window
curl -s -X POST http://localhost:4000/api/studio/skills/invoke \
  -H 'content-type: application/json' -b cookies.txt \
  -d '{"name":"web.fetch","input":{"url":"https://openrouter.ai/docs","maxChars":3000}}'
{
  "ok": true,
  "output": {
    "url": "https://openrouter.ai/docs",
    "title": "OpenRouter Docs",
    "markdown": "# OpenRouter\n\nOpenRouter provides a unified API...",
    "links": [{ "text": "Quickstart", "href": "https://openrouter.ai/docs/quickstart" }],
    "extractor": "readability",
    "truncated": false,
    "status": 200
  },
  "durationMs": 890
}

Used together with web.search to read a result in depth, and internally by web.research to read every chosen source.