Skip to content

Interface redesign

Status: framing settled (2026-09-28); Workbench (A2) chosen, all surfaces drawn in round 3 (2026-09-29), awaiting review. This supersedes the page layout in dashboard-ux-plan.md. The decisions there about data, privacy and write gates (P1–P6, A1–A5, T1–T3) still hold as constraints.

Diagnosis (agreed)

The current dashboard is an admin panel: its navigation mirrors the back-end modules, and each page is a form around one endpoint. It configures the things that do work elsewhere (Claude Code, Cursor) instead of being a place to do the work. The back end (scripts/dashboard.py + tested modules, ~40 JSON endpoints) is sound and does not depend on the page layout, so the front end can be replaced wholesale.

Problems seen on main at b64b10c:

  • Home shows wiring (article counts, client connections), not what needs attention today.
  • Status vocabulary is inconsistent: an automation reads "healthy" while also "never run" and "not installed"; the header's grey dot can mean not-run or failed.
  • Pages are mostly controls with little content.
  • No model is reachable from the UI; all intelligence lives in agent clients.

Accepted surfaces

  1. Today: due tasks, anything broken or degraded, untriaged captures, projects that changed.
  2. Projects as a hub: per project, its repo state, AnyType object, tasks, attachments, relevant patterns and agent connections.
  3. Command bar (⌘K): capture, search every corpus, jump anywhere, run an automation. Replaces the separate Inbox, Voice and Search pages.
  4. Reading / knowledge mode: library, research and patterns as places to browse and read in.
  5. Lab (added by Nathan): a project-agnostic workspace for creating and modifying the tools themselves: skills, design patterns, prompts, rules, MCP servers. Nathan's work as a solutions architect means he often authors new tools or adapts old ones to new purposes. Sometimes this happens inside a project, but tool work needs its own home.
  6. Motivating example: while refining a SKILL.md, run it against several models in parallel, or run slight variations of it across several instances of the same model, and compare the results.
  7. Generalize that shape: an artifact under edit, variants of it, a set of runs (artifact variant × model × input), and a side-by-side comparison. The Lab is that working bench, not a skill-specific editor.

Configuration (MCP matrix, code-library categories, source root, client connections) moves to a single, rarely visited Settings area.

Decisions

  • D1. A place to work in, open all day (2026-09-28). Persistent layout with panes, tabs or splits; work survives a reload. Today is the landing view; ⌘K is on every page. A system-wide summon hotkey is a later add-on, not a starting constraint.

  • D2. Models: the Lab plus an assistant you call on (option B) (2026-09-28). ⌘K, project and reading views can hand a request to an agent. Two rules: a model runs only on an explicit action (no background generation in v1), and every call shows its model, route (local Ollama or cloud via OmniRoute) and the data classification it works on. Ambient generation (option C) is deferred until the model paths exist and have been used.

  • Consequence: the AnyType write gate is a Claude Code PreToolUse hook, so an agent in the UI must either run as Claude Code (hooks apply unchanged) or duplicate the gate. Settled under 2a.
  • D3. Two new admin surfaces follow from D2 (2026-09-28):
  • Agent tracing: review what agents did, validate routing, check behavior and reasoning, export raw run logs to other services. Adopt an existing open-source tracer; do not build one.
  • Inference cost: spend per model, route, surface and project. Small enough to build in-house.
  • Findings (verified 2026-09-28):
    • LangSmith (Nathan's first suggestion) is not open source. Its docs say: "Self-hosted LangSmith is an add-on to the Enterprise plan". The free tier is cloud-hosted, which conflicts with "sensitive stays local".
    • Langfuse: "MIT licensed, except for the ee folders"; self-hosts via Docker Compose; accepts OTLP at /api/public/otel (v3.22.0+, HTTP only, no gRPC). Not yet checked: per-model cost tracking and whether its UI covers the "behavior and thought process" need.
    • Claude Code exports OpenTelemetry: metric claude_code.cost.usage (USD), event claude_code.api_request (with model, cost_usd), events claude_code.tool_result and claude_code.tool_decision. Prompt content is "Redacted by default" (OTEL_LOG_USER_PROMPTS=1 to include).
    • OmniRoute's README claims usage/cost analytics. That is the project's own description, not verified.
  • Constraint: a trace store detailed enough for reasoning review holds full prompts and tool inputs, so it is sensitive. It stays local, under the data root, outside the repo and out of every RAG index.

  • Open (Nathan): tracer choice, given LangSmith's licensing.

  • D4. The Lab has two run modes; models from every provider (2026-09-28).

  • Prompt mode: direct model calls (via OmniRoute or local Ollama) with the artifact as text. Fast, cheap, cross-provider; tests whether a model follows the text, not whether a skill triggers or uses tools.
  • Agent mode: headless Claude Code sessions (claude -p), each with its own model and an isolated copy of the artifact (worktree and/or the sandbox-exec helper from #84). Tests real triggering, file loading and tool use; hooks, including the AnyType write gate, apply unchanged; cost and tool events arrive through Claude Code's OpenTelemetry output.
  • Both modes send traces to the same tracer, so comparison and cost views are mode-agnostic. The D2 assistant always runs in agent mode.
  • Scope: Claude models and other cloud providers. Local Ollama models were in scope but are parked (below).
  • To verify empirically: agent mode with non-Claude models assumes OmniRoute's Anthropic-compatible endpoint carries a full Claude Code session, tool use included. Code evidence (2026-09-28): the fork has src/app/api/v1/messages/route.ts, and bin/cli/commands/launch.mjs starts Claude Code with ANTHROPIC_BASE_URL pointed at OmniRoute and ANTHROPIC_AUTH_TOKEN taken from OMNIROUTE_API_KEY. That shows the design intent, not that tool use survives translation for every provider, so it needs a run per provider: plain text → tool_use block → a real claude -p session that executes a tool, calibrated first on a Claude model.
  • Probe run 1 (2026-09-28, output in notes/omniroute-anthropic-probe.txt):
    • claude/claude-sonnet-5-5: text PASS, tool_use PASS. The endpoint carries Anthropic tool use for a Claude model.
    • The claude -p step failed for every model because of a probe bug (--allowedTools is variadic and swallowed the prompt), so it says nothing about OmniRoute. Fixed; re-run needed.
    • zc/glm-5.3 (502 "spawn zcode ENOENT") and cxa/gpt-5.5 (503 "Codex app-server transport is not configured") failed because those providers are not working locally: inconclusive about the endpoint.
    • Calibration lesson (Nathan): /v1/models is not a list of configured models. It lists models from connections marked active, and "active" does not mean "working" (both failures above were listed). OmniRoute's /api/providers has connection state, but needs its management login and can return stored keys, so it is not used. Instead the probe gains a discover step: one max_tokens 8 call per provider to a cheap representative model, calibrated first on the known-good Claude model.
  • Discover run (2026-09-29, notes/omniroute-discover.json): calibration passed (claude/claude-sonnet-5-5 WORKS). Of 19 providers, 4 answer: claude, cc, ac-omni (all Claude) and cfp (gpt-oss-20b; 20 open models). So the only working non-Claude route today is cfp. Failures:
    • not installed locally (never works until set up): aug (Auggie CLI), cxa (Codex transport), dva (Devin bridge path), zc (zcode binary), pepper;
    • account or billing: cinf/cheaperinference (402 wallet), moonshot (account suspended);
    • temporary, worth a retry: gemini (503 demand), mistral (429), cerebras (403 upstream), ddgw (418 anti-abuse), unc (502);
    • bad representative id: anthropic (404 on claude-haiku-4.5; the native API spells it claude-haiku-4-5) and oc (401 "not supported"). A probe or catalog-id problem, not a provider verdict.
    • New finding: Claude Code rejects model ids it doesn't know ("isn't described by this version's model catalog; update Claude Code, or map it with behavesAs…"). Agent mode with non-Claude models needs a model mapping in Claude Code's config. Mechanism not yet looked up.

  • Local models (Ollama) are parked (Nathan, 2026-09-29): this machine can only run the smallest models. Revisit after a hardware upgrade, with quantized models. Until then "local" is not a Lab route.

  • D5. Activity-first structure; project is a carried filter (option B) (2026-09-28). A left rail lists the surfaces (Today, Projects, Lab, Library, Agents for tracing and cost; Settings apart). A current project setting carries across every surface as a filter. "Lab scoped to project X" covers tools built inside a project; "Lab, no project" is the default. Rejected: a project-first IDE shape (tool work would have to pose as a project) and an object-first shape (every screen becomes a database view).

  • D6. Runs as an installed web app (PWA) now; room for a native wrapper later (2026-09-28). Same Python back end on 127.0.0.1, installed via Chrome for its own window, Dock/⌘-Tab entry and fuller keyboard shortcuts. The core uses no native-only features, so a Tauri shell (global hotkey, menubar, notifications) can wrap it later without a rewrite.

  • Essential security requirement (Nathan: "emphatically"). Once the UI can start agent sessions (D4), the local server is a code-execution surface for any web page that can reach it. The existing checks (scripts/dashboard.py:202: Host validated on every request against DNS rebinding; own Origin plus per-launch token on every POST against CSRF) become acceptance criteria, with tests covering every agent-launching endpoint.

  • Not asked yet: access from other devices. Assumed this Mac only until Nathan says otherwise.

  • D7. Built for Nathan first, but works for anyone who clones it (option B) (2026-09-28). Every surface works with an empty data root and with services missing (AnyType, Ollama, OmniRoute), has a designed empty state, and offers a first-run setup path in Settings that mirrors the setup skill. No personal data (space names, project lists, topics) in front-end code.

  • Why (Nathan): few others are likely to use it, but designing as if they might forces good data hygiene and structure, and is the guard against leaking personal data while building it.

  • Consequence: design samples built against the live API show real project names, including work/ material, so they are served locally only. Any sample that is published uses fictional data.

  • D8. Round-3 choices confirmed (Nathan, 2026-09-29): one tab per surface (the label shows the current item); Library holds reading and tools (skills, MCP servers, code library), with the Lab as the place to edit and test them; Automations live under Agents; the all-projects connection matrix lives in Settings, and each project hub shows only its own connections; this Mac only (no remote access, no auth beyond the localhost checks). The budget stays.

  • D9. Project context is a selector at the top, like the Google Cloud console (Nathan, 2026-09-29). Services stay on the left rail; a project dropdown at the top sets context for every service, and includes "create a new project". Browsing a project hub does not change the context by itself. Supersedes round-3 choice 2.
  • "All projects" is the only non-project entry (Nathan, 2026-09-29). It means both "show everything" (Today, costs and the budget, the projects index) and "no project" for anything created while it is selected. There is no separate "None"; showing only unscoped items is a filter chip where it matters (Traces). Selector entries: All projects, each project, New project…, Manage projects…
  • The selected project is in the URL (?project=), so links and reloads keep their context.
  • Surfaces the selector doesn't affect (Settings, Automations, most of the Library) show a "not project-scoped" marker.
  • Proposed, not yet confirmed: a project's domain sets the default classification for Ask, capture and Lab runs made under it (overridable per action). To settle with the sensitive-data routing question.

  • D10. A Lab experiment's scope follows the selector, and setup makes it visible (option C, Nathan, 2026-09-29). The setup step shows the scope prominently with a one-click "make general". Supersedes D5's "Lab, no project is the default".

  • D11. Front end: Preact + htm, copied into the repo, no build step (option B, Nathan, 2026-09-29). Pinned single-file ES modules checked into the repo; make dashboard stays one command with no Node toolchain. JSDoc plus optional tsc --noEmit for types; Node's built-in test runner for logic and Playwright for browser checks (today's front-end "tests" only match strings in dashboard.js). One accepted exception: CodeMirror 6 for the Lab editor, bundled once into a committed file.

  • D12. Ask answers appear in a side panel with context chips (option A, Nathan, 2026-09-29). The panel is available on every surface (⌘J, or from ⌘K), persists while navigating, and can widen into a tab. Before sending, a header shows model, route and classification, all editable (D2). What you are looking at (the selected project, a Library passage, a Lab trace) is attached as removable chips, never silently. Answers stream; tool calls are collapsed; write-gate approvals appear inline as Approve / Deny. Follow-ups continue the same session, "Open in Claude Code" hands it to a terminal (claude --resume), and every conversation appears in Traces.

  • Verified from claude --help (2026-09-29): -p supports --output-format stream-json, --input-format stream-json (multi-turn over stdin), --resume <session-id>, and --permission-prompts host with --permission-prompt-tool, so the hosting program can answer permission prompts (none auto-denies them).
  • D13. Lab runs use stand-in tools first; a throwaway AnyType space is the fallback (Nathan, 2026-09-29). Each run gets its own temporary Claude Code config directory holding only Nathan's hooks, the variant under test, and the MCP servers the experiment names, so the write gate still applies and nothing else leaks in. Real write tools are replaced by a local stand-in MCP server with the same tool names that records every call and returns plausible success; checks assert on the recorded calls. If stand-ins prove too limited, fall back to a throwaway AnyType space created and torn down per run batch, like an ephemeral pipeline test environment. Ask (the side panel) uses the real tools under the real gate.
  • To verify before the fallback is built: whether the scoped API key may create and delete spaces.
  • Defaults stated and not objected to: one claude -p process per conversation or run, streamed to the page over server-sent events; at most 3 concurrent runs; the dashboard answers permission prompts (--permission-prompts host).

  • D14. New structured state goes in SQLite (Nathan, 2026-09-29): data_root/tool-chest.db, through Python's built-in sqlite3. It holds experiments, runs, check results, stand-in call logs, the conversation index, the budget, pins and tabs. Variants stay plain files under data_root/lab/<experiment>/variants/; "promote" copies one into shared/skills/ on a new git branch. Existing JSON/JSONL stores are not migrated. Tabs and pins are server-side. Every experiment carries its data classification.

  • D15. Tracing through three seams, not an adapter layer (Nathan, 2026-09-29). Aim: no lock-in to one tracing provider, without enterprise machinery. Open source is not required if a free tier covers personal use.

  • Write: everything emits standard OTLP (Claude Code sessions, and each Ask or Lab run through OTEL_* variables on its claude -p process). No vendor SDK in tool-chest. Admission test for a backend: accepts OTLP over HTTP, and is self-hostable or has a free tier covering personal use.
  • Route: one OTLP endpoint in [tracing] in tool-chest.toml. An OpenTelemetry Collector is added only for a second backend or any cloud backend. Privacy rule: full-content traces go only to local backends; a cloud backend only receives spans the collector has stripped of content.
  • Read: the dashboard writes its own index row per run it launches to tool-chest.db (run id, kind, model, route, classification, project, status, duration, cost, session id). Traces, Costs and the budget read that index. The vendor UI is only an "open trace ↗" link from a URL template in config. Losing a vendor loses the detailed span view until the endpoint is repointed, and nothing else.
  • Not built: provider adapters, a query API across vendors, plugins.
  • Defaults (confirmed): first backend is Phoenix, run locally (uvx arize-phoenix serve; Elastic License 2.0), no collector yet; Costs and the budget cover dashboard-launched runs only in v1 (ordinary Claude Code sessions still send traces to the tracer, but aren't in the Workbench's cost figures); Laminar and Galileo are checked against the admission test only when a second backend is wanted.
  • Verified 2026-09-29: Langfuse self-hosting needs Postgres, Redis/Valkey, ClickHouse, S3/blob storage and separate web and worker containers (too heavy for this laptop for now). Phoenix accepts OTLP and runs as one process; license ELv2 (quoted). Claude Code span export is a beta: it needs CLAUDE_CODE_ENABLE_TELEMETRY=1, CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1 and OTEL_TRACES_EXPORTER=otlp.

  • To verify before building: whether Claude Code takes a parent trace context from the launching process (otherwise link by session id), and whether claude -p's stream-json result message reports cost.

  • D16. Ollama local models stay a supported feature; tool-chest will run on two machines (Nathan, 2026-09-29). Supersedes "Local models (Ollama) are parked" under D4. Nathan has a new, more powerful work laptop and plans to clone tool-chest there and run much of the day-to-day on it; the personal machine can only run the smallest models. So "local" is a real route wherever Ollama and a suitable model are present, and absent elsewhere. The routing policy table in tool-chest.toml is confirmed as the mechanism (enforced on the server before any run starts; the Lab model picker and the Ask header both respect it).

  • D8's "this Mac only" now reads: each machine runs its own instance; still no cross-machine access and no auth beyond the localhost checks.

  • D17. The routing policy (Nathan, 2026-09-29), in tool-chest.toml, enforced on the server before any run starts:

Classification Allowed routes Confirm
low_sensitive any configured route no
internal local, plus a named cloud list (starts as Anthropic direct only) once per conversation or experiment, for cloud
private_sensitive local only; blocked where no suitable local model exists; no cloud override n/a
  • An optional per-machine policy file in each data root may only narrow the shared policy, never widen it (e.g. restrict internal to an employer endpoint on the work laptop).
  • A project's domain sets the default classification (work → internal). Raising it per action is free; lowering it below the project default needs a confirmation. (Confirms the proposal under D9.)
  • Each machine is its own island: data root, tool-chest.db, experiments, traces and budget don't sync.
  • Not decided by tool-chest, recorded for Nathan: AnyType syncs every space in the account to every device, including an employer laptop; and whether an employment agreement allows skills written on the work machine to be pushed to the public repo is not yet checked (same kind of question as A5). Until checked, promoting a Lab variant into the repo on the work machine needs an explicit confirmation.

  • Test: a known-bad case must fail (an internal run routed to a free gateway is refused; a private_sensitive run with no local model is refused).

  • D18. Rollout: build alongside, then switch all at once (option A, Nathan, 2026-09-29). The new interface grows at /next while the old dashboard stays at /, frozen (fixes only). Vertical slices, in order: (1) shell + Settings, (2) Today, (3) Projects, (4) Library, (5) run infrastructure + routing policy + security tests, (6) Ask panel, (7) Agents, (8) Lab, (9) switch over and delete the old dashboard.

  • Amended 2026-10-12 (Nathan, in the slicing grill): one cut-over, at parity. The switch no longer waits for slices 5 to 8. It happens when every section of the old dashboard has a new home: slices 1 to 4, plus Capture and the Automations page (both need no model run and were pulled out of slices 6 and 7), plus the Harnesses, OmniRoute and make check sections the old page also has. Slices 5 to 8 land afterwards, additive. Research-channels management (feeds editor, staged review) is in v1, kept thin. Slices are now far smaller than the nine above; see epic #113 and spec §11.
  • The security epic and red-team session happen before slice 5, the first slice that launches agents. No agent-launching endpoint ships before its security tests.
  • Tracker changes made 2026-09-29, approved by Nathan: #58, #59, #60 closed after checking each against origin/main (merge commit is an ancestor, tests pass). #48, #49, #52, #53, #54, #56, #79 relabelled ready-for-agent → needs-info with a note: they add UI to the page being replaced; their back-end parts get absorbed into the matching slice. #51 was left ready-for-agent: it has no UI, and it is already blocked by #48.
  • Related issues: #67 (harness design and testing lab) is blocked on "a decision on the run log"; the run index in D14/D15 is the candidate. #83 (one structured private data directory) agrees with D14.

Roadmap additions

  • Security epic and red-team session for tool-chest as a whole (Nathan, 2026-09-28). The attack surface has grown a lot. A seed list of surfaces, not yet refined, to be grilled before it becomes an epic:
  • the dashboard server, especially agent-launching endpoints (D4, D6);
  • the AnyType write gate (the README already notes "a focused threat-model pass is still to do");
  • the Lab's isolation of parallel agent runs (worktrees, sandbox-exec);
  • the trace store, which holds full prompts and tool inputs (D3);
  • OmniRoute: a fork carrying provider keys, on the path of all cloud traffic;
  • automations (launchd jobs, Keychain credentials, Pushover);
  • reference clones and their updates (supply chain), and code-library scans;
  • capture inputs, including the deferred URL fetch (SSRF, per A2).

Still open

  • Tracer choice (D3) settled in D15.
  • OmniRoute probe results per provider (D4): run the discover step, then the three-step ladder on providers that answer.
  • Access from other devices settled in D8/D16: each machine runs its own instance, no remote access.
  • The model-mapping step for non-Claude models in agent mode (behavesAs), deferred by Nathan until the design is settled.
  • Before building: the verify items under D13 and D15.
  • Sensitive-data routing settled in D16/D17.

Design samples (2026-09-29)

Three directions in one throwaway page, scripts/dashboard_static/prototype-shell.html, served at /prototype (?variant=A|B|C, &surface=lab; ← → to switch):

  • A Workbench: icon rail, explorer sidebar, tabs, split panes, status bar.
  • B Desk: single calm column, reading-first, a docked ask bar.
  • C Command-first: a command line is the way in; results stack as blocks.

Each draws the shell with the project filter, Today (live data), the Lab (mocked: 3 variants × 3 models, checks, cost and trace per run) and ⌘K. They also try out a single status vocabulary (ok · warn · fail · off · not-run · info).

Round 1 verdict (Nathan, 2026-09-29): Workbench (A) is the best fit overall and is the starting point. Desk (B)'s Today reads well but "hides too much utility behind a simple calm design". Command-first (C) is dropped.

Round 2: ?variant=A2 puts a briefing Today inside the Workbench: Desk's reading order (date, greeting, one-sentence summary whose counts jump to their sections, then Needs attention / Commitments / Where you left off / Inbox) in Workbench's theme and density. Every row keeps its actions visible: specific ones per item (install all, update 9, reset to upstream…, open / terminal / diff), inline task add, keyboard hints. A right-hand "glance" column holds systems, spend, the Lab and recent agent runs. A keeps the three-column Today for comparison, B stays as the reference. Lab mock models are now ones that answered the discover run.

Round 2 verdict (Nathan, 2026-09-29): A2 is the right direction. Flesh it out and draw the missing surfaces.

Round 3 (2026-09-29): the page is now Workbench only (A2), with every surface drawn: Today (briefing), Projects (index + per-project hub), Lab, Library, Agents (traces, costs, automations), Settings (services, agent clients, project connections, code library, security posture, first-run setup). Live data wherever an endpoint exists; traces, costs and Lab runs are mocked and labelled; every action is a stub. Deep links: /prototype?surface=…&sub=…[&q=…].

Design choices made in round 3 that Nathan has not confirmed:

  1. One tab per surface (its current item shown in the tab label), not one tab per item.
  2. Opening a project in Projects also sets the carried project filter (my reading of D5).
  3. Library holds both reading (research, books, wiki, patterns) and tools (skills, MCP servers, code library). The Lab is where tools are edited and tested.
  4. Automations move from "Operate" into Agents, next to traces and costs.
  5. The per-project connection matrix, client configs, security posture and first-run setup (D7) all go in Settings. Each project hub shows only its own connections.
  6. Today's sidebar holds views, pinned items and open tabs instead of the project list, which avoids repeating "Where you left off".
  7. A monthly budget appears in Costs as a proposal. Confirmed (Nathan, 2026-09-29): keep the budget (warn on Today at 80%, hold Lab batch runs at 100%).

OmniRoute enabled = true in tool-chest.toml (2026-09-29) was Nathan's manual change, needed for the model probe. Not a prototype side effect.

Verdict: pending.