Automations¶

Small deterministic commands that can be run by name, with every run recorded, and optionally run on a schedule. Part of the Automations v1 epic (#68); this folder holds the definitions, which are public and generic. Anything personal (topics, sources, thresholds) and all live state (paused, what is installed on this machine) lives in the private data root, never here.

A definition¶

One file per automation, <name>.toml:

name = "heartbeat"                 # must match the file name
description = "what it does"       # optional
command = ["{python}", "scripts/automation_heartbeat.py"]   # argv, no shell; run from the repository root
timeout_seconds = 30               # optional; 1 to 3600, default 300
outbound = false                   # optional: does it call outside services? (auto-pause applies if so)
headline = "beats"                 # optional: which reported fact is its headline count
env = ["SOME_VARIABLE"]            # optional: environment variables the command may see (default: none; not AUTOMATION_*)

[anytype]                          # optional: the AnyType access it needs (default: none, and no key is ever fetched for it)
permission = "read"                # the word AnyType reports for the key's permission: "read" or "readwrite"
spaces = ["agent"]                 # exactly the spaces (spaces.json ids) the key may reach
kind = "command"                   # optional; the only kind for now, reserved for later ones

[schedule]                         # optional; without it the automation is manual-only
every_minutes = 5                  # an interval of 1 to 1440 minutes ...
# daily_at = "08:00"               # ... or a fixed local time every day (HH:MM, 24 hour): exactly one of the two

The runner gives a command a clean environment: only the variables it names under env (plus a few basics such as PATH and HOME) get through, and a scheduled run started by launchd has none of your shell's variables anyway. {python} stands for the interpreter running the runner. A definition may not name the home directory or the data root; make automations-defs-check (part of make check) validates every definition.

Running¶

make automations-run NAME=heartbeat     # run one and print a one-line summary
make automations-list                   # what is defined, paused or not, and the last run
make automations-runs [NAME=heartbeat]  # recent runs (never their output)
make automations-pause NAME=heartbeat REASON="why"
make automations-resume NAME=heartbeat

Each run writes a started and a finished event to <data_root>/automations/runs.jsonl, so a run that dies half way shows as interrupted. The runner enforces the timeout on the command and its children, refuses to start a second copy of an automation that is already running, keeps only the tail of the output, and gives the command a clean environment.

An automation reports what it found through scripts/automation_result.py: named facts and one headline count, which the runner validates and records.

The task digest¶

task-digest is a read-only digest of the open tasks in the Agent space. make automations-digest runs it by hand and records the run. Its output lists every task, earliest due first, with its due date and days remaining; its result holds counts only (overdue, due_today, open, unreadable, and needs_attention, the headline: overdue plus due today), so a task name never reaches the log's facts.

Same rules as the task browser. Dates are read in the local timezone, a date with no time is due at the end of that day, and a timestamp at exactly midnight UTC means a date with no time. The browser (dashboard.js) and the digest (scripts/task_due.py) are both tested against one file, scripts/fixtures/due_vectors.json, so they cannot drift apart; a new rule or edge case goes there.
Read-only by construction. The digest is given a client that can only search and read one object.
A failure is never an empty digest. AnyType being down or refusing the search fails the run and records the failure class. One task that cannot be read is counted as unreadable and shown, and the others are still reported. A full search page (100 tasks) sets possibly_more.
Its own key. It declares [anytype] and has its own key in the Keychain (see below). A run without a stored key, or with a key that does not match the declaration, fails with the reason and never starts the command.

AnyType keys (Keychain)¶

An automation that needs AnyType declares it ([anytype], above) and gets its own key, separate from your everyday one. The key lives in the macOS Keychain under the service tool-chest.automation.<name>; nothing else stores it.

make automations-key-store NAME=task-digest    # run it yourself, in a terminal: paste the key (hidden)
make automations-key-check NAME=task-digest    # what the stored key reaches, against the declaration
make automations-key-delete NAME=task-digest

The gate. Before every run the runner fetches the key, asks AnyType what it really allows (GET /v2/auth/whoami) and refuses to run unless it matches the declaration exactly: the same permission (on the key and on each space), exactly the declared spaces (an extra or a missing one is refused), and a restricted, scoped, non-full, unexpired key. An all-spaces key, a legacy key, or an answer it cannot read is a refusal, never a pass. The refusal is a failed run whose error says which differences it found. key-store runs the same check on the pasted key first, so a key that does not match is never stored.
A key may write only where agents may write without asking. The key sits in a Keychain item that an agent in the sandbox can read without a prompt (an automation runs unattended, so it can't be made to ask), so it must grant no more than agent policy already does. "read" is allowed on any agent-readable space. Any other permission needs every declared space to be agent_write = "full" in spaces.json (today, only Agent); a definition that declares a write key for Work or Job Hunt is refused, naming the space and its policy. The rule fails closed: any word other than read counts as a write.
Declaring nothing means no key. An automation with no [anytype] never has the Keychain asked for it, and a definition may not ask for a key through env (any ANYTYPE_* name is refused).
The key stays out of the log. It goes to security on stdin (never on a command line), is handed to the command only as ANYTYPE_OPENAPI_MCP_HEADERS, and is removed from the recorded output, error and facts if the command prints it, including a piece cut by the output limit. It is never in a refusal message.
Fails closed. A locked Keychain, or one waiting for a prompt nobody can see (it gives up after 20 seconds), is a failed run with that reason.
Which word means read-only? The declaration compares against whatever word AnyType reports, so it does not guess. Create the key in AnyType, run make automations-key-check, and declare the permission it reports. Whether Desktop offers read-only or single-space keys is recorded on #72.

Health¶

make automations-health [NAME=...] says whether each automation is healthy or degraded, with the reasons, computed from the run log each time (nothing is stored, so what "reliable" means can change without rewriting history). It exits 1 when any automation is degraded. The output holds names, counts and class names, never run content.

Success rate over the last 20 runs (degraded below 80%, once there are 3 runs). A skipped run is not a run; an interrupted one (started, never finished) is a failure and is also reported by name.
Age of the last success, for installed, scheduled automations, in awake time: a sleeping Mac runs nothing, so the time it slept (from pmset -g log) is taken out. An interval job is stale after 3 intervals awake; a daily one after 30 awake hours. Without the sleep history the age is wall-clock and the command says so.
Missed days, for daily_at automations only: days with no scheduled run at all (degraded above one day in the last 14). Slots coalesce across sleep, so after a two-day sleep there is one run on wake and no way to say which day it covered; per-slot counting would be fiction. A run late on wake counts for its day, a skipped run counts (launchd fired while it was paused), and the days since a current pause are excused. An interval automation is never judged from the wall clock.
Headline change, opt-in per automation: the latest success against the median of the seven before it, flagging a large rise or a drop to zero after usually being non-zero. It does not apply until enough successes exist.
Outside calls by class: client side (4xx), server side (5xx) and no status at all, each flagged when it is a quarter or more of the recent calls.
A scheduled run that could not find its automation (deleted, renamed or broken) is recorded as a failed run, so it lowers the success rate; a name with no definition that still fails on schedule shows up as degraded, telling you to reinstall or uninstall its agent.

Thresholds have defaults and can be overridden, for every automation or one, in the private <data_root>/automations/health.toml (an unknown or out-of-range setting is refused, naming it):

[defaults]
window = 30
[automations.task-digest]
headline_check = true          # opt in to the headline check for this automation
headline_min_successes = 5

The settings are the fields of Thresholds in scripts/automation_health.py.

The clone freshness check¶

clone-check fetches every reference clone under the code library's source root and counts how many are behind upstream, using the existing check code (scripts/clone_sync.py). It only fetches: it never pulls, merges or changes a working tree, so updating a clone stays an explicit action (the dashboard's Code library page, or clone_sync.update).

Facts (counts only): clones, checked, behind, unreachable, and, for the unreachable ones, origin_unreachable and fetch_failed. The headline is behind. Names appear only in the run's output.
ok vs failed. Some clones unreachable (offline, private, gone) is an ordinary ok run; those are counted. It is failed only when it could not do its job: no configured or usable source root, or not one clone could be checked.
It calls outside services (outbound = true), so #77's auto-pause applies: three runs where nothing could be checked pause it, with the reason naming what git reported (remote unreachable, fetch failed), the same vocabulary automation_health.failure_classes uses for HTTP failures.
Default schedule: once a day at 04:30, adjustable in the definition.

The arXiv poll¶

arxiv-poll is the research project's first scheduled channel (#107). It runs research poll-arxiv, which reads the saved feeds from the private <data_root>/research/feeds.toml, asks arXiv what is new (sortBy=submittedDate), downloads each new paper's PDF, and sends it through the sandboxed extraction, the scan and the archive-or-stage path. It makes no more than one request every three seconds and one connection at a time, as arXiv's terms of use require.

Facts (counts only): feeds, feeds_ok, new_papers, archived, staged, duplicates, failed, skipped_invalid, deferred. The headline is new_papers. No title, author or address is recorded.
ok vs failed. No feeds configured, nothing new, or a paper that failed to download (it is retried next run) are all ok. It is failed only when feeds were configured and not one could be queried, which is what an offline machine, an arXiv outage or a block looks like.
It calls an outside service (outbound = true), so #77's auto-pause applies: three such runs in a row pause it.
Ollama must be running (papers are embedded as they are indexed). If it is not reachable the run fails before fetching anything, with embedder_unavailable = 1 in the facts, so three such mornings in a row pause it.
The command does not depend on PATH: it runs the research project's own interpreter (research/.venv), because a scheduled job has no ~/.local/bin. The run log's stderr tail carries the reason each paper failed.
Nothing is polled until you write a feed into feeds.toml (see research/README.md), and nothing runs on a schedule until you install it (make automations-install NAME=arxiv-poll). Run it once by hand first: make automations-arxiv-poll.
Storage posture. It keeps full PDFs under your private data root, for personal use only. arXiv's terms say not to "store and serve" them to other people; the maintainer's reading (2026-10-11) is that personal use that never serves them is within the terms, which holds only while the archive is never shared or served. The wording is in research/README.md and research/src/research/arxiv.py.
Default schedule: once a day at 05:15, adjustable in the definition.

Auto-pause¶

An automation whose definition says outbound = true (it calls outside services) is paused by the runner after 3 consecutive failures, so a broken or blocked job does not keep hammering a service and get the machine blocked. The reason and the kinds of failure are recorded (auto-paused: 3 failures in a row (server errors (5xx), no network or no answer). Fix the cause, then resume.), and it stays paused until you resume it (make automations-resume NAME=... or the dashboard).

A local-only automation is never auto-paused. It is marked degraded and keeps running; pausing it would only hide a fixable bug.
No counter. The streak is read from the run log each time, so it cannot drift: a success ends it, a skipped run is not a run, and resuming starts a fresh count (only runs that started after the resume count, compared to the millisecond).
The kinds of failure are named from the recorded outside calls: blocked (403), rate limited (429), server errors (5xx), other client errors (4xx), no network or no answer, plus a timeout, an interrupted run, or, when no outside call went wrong, the command itself failing.
The number can be set per automation in the private <data_root>/automations/health.toml ([automations.<name>], auto_pause_after = 5); it defaults to 3.
It shows in the health view (make automations-health: paused automatically: ... and the failing streak), on the dashboard page, and as one notification, <name> is degraded: paused after repeated failures. A pause you made by hand is not shown as automatic.

Notifications¶

Kept deliberately small, and built only from an allow-list of fixed words and numbers, so free text (a task name, an error message, a secret that got into a run) has no way into one.

A change from healthy to degraded sends one notification: task-digest is degraded: low success rate, missed runs. The last verdict each automation was given is kept in the private state, so repeated degraded runs are silent, recovery is silent, and a new degradation after a recovery fires again. If a notification could not be delivered it is tried again on the next look, since you never heard. The runner looks after every run; the health-check automation (hourly, awake time) looks at everything on a schedule, because a missed day or a job that stopped produces no run to look after. A name with no definition that still fails on schedule, and a broken definition, count too.
The task digest sends its counts, Task digest: 3 overdue, 1 due today, and nothing on a day with neither (--no-notify turns it off).
A notification that fails is printed to stderr (launchd keeps that in the automation's own file under <data_root>/automations/launchd/) and never fails a run.
Delivery is Pushover (a phone push service), not a local macOS notification: osascript needed a per-app permission (Script Editor) that a macOS update or a settings cleanup can silently revoke, and could not be relied on to show anything with the screen locked or at a DarkWake. TOOL_CHEST_NOTIFY=off turns delivery off (the tests and CI set it).
Credentials. pushover_notifier reads PUSHOVER_USER_KEY and PUSHOVER_API_TOKEN from the environment first (a manual override), and otherwise from the Keychain, the shared pair stored under the service tool-chest.pushover (scripts/automation_pushover_key.py). This is what makes a scheduled run able to notify at all: the generated agent's environment is fixed to PATH only (automation_launchd.py), so it never has the environment variables, only the Keychain. A pasted pair is checked against Pushover's /1/users/validate.json before it is stored (no notification sent), so a mistyped key is never saved silently. A failed notification is printed to the automation's own log and retried on the next look; it never fails the run.

Setting up and rotating the Pushover credentials¶

One shared pair, not per automation: a user key (from your Pushover dashboard) and an API token (from a registered application there). Run these yourself, in a terminal — store refuses without one.

make automations-notify-key-store    # paste the user key, then the API token (both hidden)
make automations-notify-key-check    # confirm the stored pair still validates; sends nothing
make automations-notify-key-delete   # remove the stored pair

First setup: automations-notify-key-store. It validates the pair with Pushover before writing anything, so a typo leaves the Keychain untouched and prints Pushover's own reason. On success it prints the account's active devices and licenses (never the credentials themselves) and stores the pair as one Keychain item.
Rotating (a new API token, a revoked one, moving to a different Pushover application): generate the new token on Pushover's dashboard first, then run automations-notify-key-store again with the new pair. It overwrites the existing Keychain item in place (security add-generic-password -U), so there is no separate "update" command and no need to delete first. automations-notify-key-check confirms the new pair took.
Verifying: automations-notify-key-check re-validates the stored pair against Pushover without sending a notification. To confirm delivery itself, either run an automation that notifies (make automations-digest when something is due, or the heartbeat degrade recipe in the private runbook) or call automation_notify.pushover_notifier directly from a Python one-liner.
Removing: automations-notify-key-delete, then revoke or delete the application on Pushover's own dashboard if you no longer want it there either.
The pair never appears on a command line, in a log, or in any error message this codebase raises; only Pushover's own diagnostic text (which field was invalid) can appear, never the values.
Pushover's limits (verified against its API docs): a message up to 1024 UTF-8 characters, a title up to 250; the title and message are truncated to these before sending. Exceeding the monthly quota answers HTTP 429.

Install the check with make automations-install NAME=health-check (it asks first).

The dashboard page¶

Operate > Automations in the dashboard (make dashboard) shows every automation: the health verdict and its reasons, the last run (time, outcome, duration, trigger), the latest result (facts, and the run's output as escaped text), the last ten runs, and the next scheduled time. It lists names from the definitions and from the log, so a broken definition, or a name that still runs on schedule after its definition was deleted, shows up as degraded instead of disappearing. Nothing installed and no runs yet is shown as such, and an unreadable state or configuration is shown as an error, never a blank page.

Run now starts scripts/automations.py run <name> in the background, the same runner a schedule uses, so the page stays responsive and the run is recorded like any other (trigger manual). Only the name of a definition in the repository is accepted; nothing else from the request reaches a command, and a run that is already running is refused. The page polls while anything runs and shows running, then finished.
Pause and resume take effect at once (the next run, scheduled or by hand, is skipped or runs) and are shown on the page.
Actions use the dashboard's Host, Origin and token checks, like every other write.
The latest result can hold private task names. It is local like the rest of the dashboard, and shown as text, never as markup.

Outside calls (the HTTP client)¶

An automation that calls an outside service uses scripts/automation_http.py, so every call is recorded and classified without the automation having to remember to:

import automation_http
client = automation_http.default_client()
response = client.get("https://example.org/feed")     # any status comes back as a response
if not response.ok: ...                               # 4xx and 5xx are answers, not exceptions
# no status at all (no network, refused, timeout, bad certificate) raises HttpFailure, with .detail

For each request it records the host, the status, the elapsed time and a class: 2xx, 3xx (a redirect that was not followed or ran past the limit), 4xx (with blocked for 403 and rate_limited for 429 called out), 5xx, or none (no status: dns, refused, timeout, tls or other). A dead network has no HTTP code, and must never look like "found nothing". The finished run record carries the summary under http: counts per class, the first 50 requests, and how many more were only counted.

Nothing private is recorded. Never a query string, a path, a request or response body, or a header, and a failure message names only the host. The runner also validates the snapshot strictly, so nothing but hosts, statuses, times and class names can reach the log.
No silent retries. A 503 is reported once. A retry hides exactly the failures the metrics exist to show.
Redirects are followed by hand, at most five. Credentials never cross to a different scheme, host or port: only harmless headers go along, a request with a body is not repeated at another origin, and a redirect to anything but http(s) is not followed.
The timeout is a total deadline for the request (default 20 seconds, at most 120), not a pause between bytes, and a response is read only up to a limit (5 MB by default).
A run that is killed by its timeout still shows what it did, because the client rewrites a small private snapshot after every request.

Scheduling (launchd)¶

Nothing is scheduled until you install it. Installing an automation generates a launchd agent from its definition, loads it, and records on this machine that you did.

make automations-plist NAME=heartbeat                 # print the agent a definition generates
make automations-install NAME=heartbeat DRY=1         # preview: what would change (safe anywhere)
make automations-install NAME=heartbeat               # install it: run this yourself, in a terminal
make automations-check                                # do installed agents still match the definitions?
make automations-uninstall NAME=heartbeat             # unload, remove and forget it (also asks first)

Generated, never edited. The definitions are the single source of truth. make automations-check fails when an installed agent differs from what its definition would generate: changed (a definition, the checkout path, the interpreter or the data root moved), missing, extra (an agent of ours that was never recorded), or orphaned (recorded, but its definition is gone or lost its schedule). It is machine-local, so like projects-check it is not part of make check.
Not for agents. install and uninstall refuse unless stdin and stdout are a terminal, then ask for a typed y. There is no flag that skips either. Run them from your own terminal.
Only our own agents. They touch exactly ~/Library/LaunchAgents/local.tool-chest.automation.<name>.plist and ask launchctl only to print, bootstrap or bootout that one label. Other agents in the folder are never modified or removed.
Safe failure. If launchd refuses a new agent the previous one is put back; an agent that will not unload is never replaced or removed.
A daily job that was missed while the Mac slept runs once, at the first wake. Verified on the maintainer's Mac (2026-09-26, sleep on AC and on battery): one run per missed slot, none skipped, none doubled. "First wake" includes a maintenance DarkWake, so the run can start while the Mac is still asleep to the user and the network may be only partly up. Do not assume the lid is open, and let an outside call fail visibly (see the HTTP client) rather than assume it will succeed.
An interval job is not a catch-up mechanism. every_minutes becomes launchd's StartInterval, which counts awake time only: sleep pauses the timer and it resumes on wake. A 30 minute sleep produces no backlog and no immediate run; the next run comes after the remaining awake seconds (observed: 300 s of awake time between runs, whatever the wall clock said). Use daily_at for anything that must happen once a day.
Moving the checkout. The agent file contains the absolute checkout path, so moving the repository (a workspace layout-apply, say) shows up as changed; reinstall to fix it.