anthropic-research¶

Local-first RAG archive of Anthropic's published research.

Crawls the research corpus from Anthropic's sitemap, archives each entry (HTML + any Anthropic-hosted full-paper PDF) as browsable local files, indexes it semantically, and exposes it to coding agents through MCP tools — with citations and local file paths. A sibling to calibre-librarian, sharing the same store / embed / retrieve architecture.

Requirements¶

• Python 3.11+
• Ollama with nomic-embed-text for embeddings
• Network access to anthropic.com for crawling

Install¶

cd research
uv sync

Configure¶

Defaults work out of the box: the archive and index database live under tool-chest's shared private data root (tool-chest.toml's [workspace].data_root, default ~/tool-chest-data), at <data_root>/research/archive and <data_root>/research/index.db. To override, use the workbench (Settings → Research channels) or research config set; the values are kept in <data_root>/config/research.toml under [general]. The old ~/.config/research/config.toml is no longer read (make config-migrate moves it):

[general]
archive_dir = "~/tool-chest-data/research/archive"
db = "~/tool-chest-data/research/index.db"
embedding_model = "nomic-embed-text"
ollama_base_url = "http://localhost:11434"
include_pdfs = true
fetch_delay = 1.0   # seconds between requests (be polite)

Usage¶

# List every research entry discovered in the sitemap (no crawl)
uv run research show-scope

# Plan a sync without fetching anything
uv run research sync --dry-run

# Crawl, archive, and index the whole corpus (needs Ollama running)
uv run research sync

# Archive + chunk without embeddings (no Ollama required)
uv run research sync --no-embed

# Index a single entry (useful for testing)
uv run research sync --only alignment-faking

# Search from the CLI
uv run research search "alignment faking in language models" --limit 5

# List what's indexed
uv run research list

# Start the MCP server for Cursor / Claude Code
uv run research serve

What gets archived¶

For each /research/* entry, research sync writes to archive_dir/<slug>/:

The SQLite index (db) holds chunks + embeddings and points back at these files. Re-running sync only re-fetches entries whose sitemap lastmod changed.

External venues (arXiv, transformer-circuits.pub) are recorded as reference links, not crawled.

MCP tools¶

Scanning inbound material (research.scan)¶

Anything that enters the corpus is meant to be scanned by a deterministic step before an agent can retrieve it (#80). research.scan.scan() takes the original bytes, the declared type, a document name and the extracted text per unit (a PDF page, or an HTML/markdown section) and returns flags: wrong file type, size and decompression-bomb limits, invisible/bidi/tag characters, known injection phrases, and obfuscated variants of them (leetspeak, spaced letters, base64). It is a tripwire, not a wall: heuristic, with real false negatives. The sandbox around extraction (research.sandbox) is the strong control.

The rules and each rule's historical false-positive rate (the share of the trusted archive's entries it fired on) are in src/research/scan_rules.toml. To re-measure after changing a rule:

research scan-archive                # report only
research scan-archive --write-rates  # also record the rates in the rules file

A rule that fires on a large share of real entries is demoted or dropped, not shipped (see the thresholds in scan_archive.py). The scan decides nothing by itself; a flag on a security article that quotes an attack is expected and is what the human review step is for.

The staged tier (research staged)¶

Material that is not yet trusted for the corpus (flagged by the scan at medium or high confidence, or, later, found by an agent) is held in <data_root>/research/staged/<slug>/: the archive's file shape plus a staged.json saying why, when and from where. It is excluded by construction: a staged entry has no row in the index and no directory under the archive, so search, passage, list and every MCP tool cannot return it. There is no "staged" flag on an index row that a query could forget.

research staged list           # what is held, and why
research staged show <slug>    # its flags (with short excerpts) and where its files are
research promote <slug>        # a human override: moves it into the archive and indexes it
research reject <slug> --yes   # deletes it (an untrusted fetch can be regenerated); without --yes nothing happens

research is not on your PATH; from the repository root use make research-staged to list, or make research ARGS="staged show <slug>", make research ARGS="promote <slug>" (needs Ollama) and make research ARGS="reject <slug> --yes". Each runs the command in the research project's own environment.

promote records promoted_at and the overridden scan flags in the entry's meta.json. Read the entry's article.md in the staged directory before promoting; the point of the tier is that a person looked.

Reviewing a batch: the workbench. The primary way to review is Library → Staged review in the workbench (/next/library/staged): every staged entry on one page with why it is staged and each flag's rule, level, location and excerpt, a checkbox per entry, select all (never including agent-found entries), filters by rule and by reason, the text around each flag on demand, and Promote or Delete for the selection (delete asks once for the whole selection; a promote is refused up front if Ollama is unreachable, and one failing entry does not stop the others). The page calls research review list|detail|promote|delete (JSON on stdin, by slug); it is deliberately not a bulk CLI for people, and no MCP tool lists, promotes or deletes staged entries (a test lists the tools).

Adding a file by hand: the inbox (research ingest-inbox)¶

For something the maintainer already has (a paywalled paper, a file received directly), drop it in <data_root>/research/inbox/ with a sidecar next to it. Copy sidecar.txt.template (written into the inbox the first time you run the command), name the copy <file>.sidecar.txt, and fill in url, title, published, note.

research ingest-inbox

Each file with a sidecar goes through one inbound path: the original bytes are hashed (original_hash, so the same file is ingested once), the text is extracted inside the sandbox-exec helper (no network, writes confined to a scratch directory; if it is unavailable the file is refused, never run unsandboxed), the scan runs, and the entry is archived and indexed, or staged if the scan flagged it at medium or high confidence. Low flags only are archived with the flags recorded in meta.json (scan_flags). Entries carry channel = manual-inbox.

• No public URL? Leave url empty. It is recorded as "no public URL: manually supplied" and shown without a link. Do not invent an address: a value that is not a real http(s) address makes the file be left in place.
• A file with no sidecar, an unreadable or invalid sidecar, an unsupported type (PDF, HTML, markdown and plain text are accepted) or a size over the limit is reported and left where it is. Processed files move to inbox/processed/.

Adding a page or PDF by URL (research add-url)¶

research add-url https://example.org/paper.pdf
research add-url --staged https://a.example/x https://b.example/y   # agent-suggested: always staged

The fetch is the risky step, so it refuses to be pointed at the machine itself: only http and https, no credentials in the address, and the host must resolve only to global addresses (no loopback, private, link-local or cloud-metadata ranges, and not through a redirect either: every hop is re-checked, and the connection goes to the address that was checked). The body is size- and time-capped, and only PDF, HTML, markdown and plain text are accepted. After the fetch it is the same path as the inbox: hash, sandboxed extraction, scan, archive-or-stage, with channel = manual-url.

Agent research: pointers only¶

An agent doing research does not add to the corpus. It returns a report of candidate URLs with a one-line reason each, as ordinary output. A person decides to submit them with research add-url --staged, and everything submitted that way is staged for review whatever the scan says. The agent has no tool that writes to the corpus or the staged tier (the research MCP has read tools only, and a test keeps it that way), and the text always comes from the deterministic pipeline, never from the agent. A prompt-injected agent can at worst suggest a bad URL, which the address checks, the size cap and the scan then handle.

Channels, and where an entry came from¶

A channel is three steps: an optional discover (what is new, used only by scheduled polling), a fetch and an extract. The Anthropic site is the first channel (research.channels.AnthropicSiteChannel); the inbox and URL routes skip discovery. Every entry records its channel (anthropic-site, manual-inbox, manual-url, ...) and the original_hash of its original bytes in meta.json, and the channel is carried into the index and shown in search results, list and the MCP tools, so the corpus can be filtered or labelled by origin.

A full research sync only ever purges entries of its own channel. Entries that arrived by hand are not in the sitemap, so they are left alone.

Entries made before channels existed have neither field. Fill them in once, with no network, from the stored copy:

research backfill-channels --dry-run   # what would change
research backfill-channels             # sets channel = anthropic-site and original_hash = sha256 of source.html

original_hash here is the hash of the stored source.html, not of a re-fetch: if the site has changed since, this records what the archive actually holds. The command is idempotent and leaves complete entries byte-for-byte alone; an entry with no source.html is reported, not guessed.

arXiv, the first scheduled channel (research poll-arxiv)¶

Saved feeds live in the private <data_root>/research/feeds.toml (never in the repository). A missing file means no feeds, and no error.

[[feed]]
name = "ml"                  # unique; letters, digits, . _ -
category = "cs.LG"           # exactly one of: category, author (Hinton_G) or a raw arXiv query
max_results = 25             # optional, 1 to 100: how far back each poll looks
enabled = true               # optional; false pauses the feed (the poll skips it and counts it as paused)

[[feed]]
name = "interp"
query = "cat:cs.LG AND ti:interpretability"

You can edit the list in the workbench (Settings → Research channels) instead of by hand. The form asks the research CLI, which holds every rule: research feeds list|validate|set (JSON on stdin, --data-root). The query is checked as you type (known field prefixes ti au abs co jr cat rn all submittedDate, closed quotes and parentheses, AND/OR/ANDNOT, the length cap) and the exact request URL is shown. A save replaces the file atomically (mode 0600, previous version kept as feeds.toml.bak) and does not keep hand-written comments. The caps (20 feeds, 100 results per feed, 300 characters) are the research settings limits.*; poll-arxiv refuses a file that breaks them. Poll now starts the arxiv-poll automation, nothing else.

research poll-arxiv --dry-run   # what would be ingested; no PDF is downloaded
research poll-arxiv             # query the feeds, download new PDFs, archive or stage each one

Each feed is queried with sortBy=submittedDate (newest first). The poller remembers what it has seen per feed (arxiv-state.json) and does not ingest a paper twice: not by arXiv id (a new version of a seen paper is not re-ingested), and not by the hash of the PDF bytes. New papers go through the same path as the inbox and URL routes, with channel = arxiv; one flagged by the scan, including its title or abstract, is staged for review. At most 20 new papers are ingested per run; the rest wait for the next. A paper whose download fails is retried next run.

Politeness is binding, not a courtesy. arXiv's terms of use say to "make no more than one request every three seconds, and limit requests to a single connection at a time". Requests are strictly sequential and at least three seconds apart, across the query API and the PDF downloads alike. The Atom response is refused if it declares a DTD or an entity (a feed has no reason to), every id is validated against arXiv's id grammar, and the PDF address is built from the validated id, never taken from the response body.

It is built as an outbound = true automation (make automations-arxiv-poll; see workflow/automations), so the auto-pause rule applies: three failures in a row pause it.

Ollama must be running when it polls: papers are embedded for search as they are indexed. The embedder is checked once before anything is fetched; if it is not reachable the run fails (it fetches nothing, and says to start Ollama) rather than downloading PDFs it cannot index. A scheduled run needs the Ollama app to be up at that hour.

Storage posture (decided 2026-10-11)¶

This channel stores the full PDF, its extracted text and its metadata under the private data root, for the maintainer's own use and for reading by the maintainer's own agent sessions. Nothing is committed, published or redistributed. arXiv's terms of use for its APIs (read at build time, 2026-10-11) say you must not "Store and serve arXiv e-prints (PDFs, source files, or other content) from your servers, unless you have the permission of the copyright holder or are permitted to do so by the license", and that you should "Direct users to arXiv.org to retrieve e-print content". They also say you may "Retrieve, store, transform, and share descriptive metadata". The maintainer's reading is that downloading papers for personal use, and never serving them to anyone else, is within those terms. That is the maintainer's reading, not a confirmation from arXiv, and it holds only while the condition does: do not share this archive, serve it to other people, or publish any index of it. If that ever changes, revisit this first; the fallback is metadata and abstract only, keeping the PDF link.

Privacy & etiquette¶

• All archiving, embedding, and search runs locally.
• Crawling respects robots.txt (Anthropic allows it), identifies via a descriptive User-Agent, and rate-limits with fetch_delay.

Development¶

make test