Skip to content

anthropic-research

Local-first RAG archive of Anthropic's published research.

Crawls the research corpus from Anthropic's sitemap, archives each entry (HTML + any Anthropic-hosted full-paper PDF) as browsable local files, indexes it semantically, and exposes it to coding agents through MCP tools — with citations and local file paths. A sibling to calibre-librarian, sharing the same store / embed / retrieve architecture.

Requirements

  • Python 3.11+
  • Ollama with nomic-embed-text for embeddings
  • Network access to anthropic.com for crawling

Install

cd research
uv sync

Configure

Defaults work out of the box. To override, create ~/.config/research/config.toml (or add a [research] section to tool-chest.toml):

archive_dir = "~/reference-archive/anthropic-research"
db = "~/.local/share/research/index.db"
embedding_model = "nomic-embed-text"
ollama_base_url = "http://localhost:11434"
include_pdfs = true
fetch_delay = 1.0   # seconds between requests (be polite)

Usage

# List every research entry discovered in the sitemap (no crawl)
uv run research show-scope

# Plan a sync without fetching anything
uv run research sync --dry-run

# Crawl, archive, and index the whole corpus (needs Ollama running)
uv run research sync

# Archive + chunk without embeddings (no Ollama required)
uv run research sync --no-embed

# Index a single entry (useful for testing)
uv run research sync --only alignment-faking

# Search from the CLI
uv run research search "alignment faking in language models" --limit 5

# List what's indexed
uv run research list

# Start the MCP server for Cursor / Claude Code
uv run research serve

What gets archived

For each /research/* entry, research sync writes to archive_dir/<slug>/:

File Contents
source.html Raw fetched HTML (redirects followed)
article.md Extracted main-content markdown + PDF text
paper.pdf The Anthropic-hosted full-paper PDF, when linked
meta.json URL, title, date, references, fetch timestamp

The SQLite index (db) holds chunks + embeddings and points back at these files. Re-running sync only re-fetches entries whose sitemap lastmod changed.

External venues (arXiv, transformer-circuits.pub) are recorded as reference links, not crawled.

MCP tools

Tool Description
search_research Semantic search over the archived corpus
get_passage Retrieve a passage with surrounding context
list_articles List indexed research entries

Privacy & etiquette

  • All archiving, embedding, and search runs locally.
  • Crawling respects robots.txt (Anthropic allows it), identifies via a descriptive User-Agent, and rate-limits with fetch_delay.

Development

make test