Recover from Failed or Partial Orchestration Runs¶

Bring the system back to a known-good state safely.

Failure taxonomy¶

Class	Examples
Validation failure	repo-contract invalid; automation exit nonzero
Promotion failure	release-policy gate failed; tag rollback required
Credential failure	rotation broke a downstream integration
Runtime failure	agent runtime mismatch; data class violated
Audit failure	log gap detected; tamper suspicion

Decision tree¶

Failure detected
└── Is data integrity at risk?
    ├── yes  → freeze namespace writes (deny new state-changing actions); escalate to platform + security owners
    └── no   → continue
        └── Is a credential or secret possibly exposed?
            ├── yes  → invoke credential-rotation-jit runbook with emergency rotation
            └── no   → continue
                └── Is the failed action partially applied?
                    ├── yes  → run rollback path defined in release-policy
                    └── no   → fix root cause, re-run from last good checkpoint

Rollback / retry checklist¶

Capture the failing event id and audit row.
Identify the last known-good state (release tag, last validation artifact, last successful onboarding).
If state-changing: revert. If read-only: just retry after fix.
Confirm rollback path is itself a logged audit event.
Validate the system at the rolled-back state (run compatibility-validation.md).
Document in incident-log.md (template below).

Incident log template¶

## [<yyyy-mm-dd>] <short title>

- detected_at: <ISO-8601>
- detected_by: <human|agent>
- class: <from taxonomy>
- impact: <repos/agents/credentials affected>
- root_cause: <text>
- recovery_actions: <bulleted>
- audit_events: [<list of event ids>]
- post_mortem_due: <date>
- preventive_change: <SPB-15 candidate? yes/no>

Tabletop drill¶

Run a recovery tabletop at least once per quarter against a hypothetical class-2 (promotion) failure.
Record outcome and any policy gaps in ../security-privacy-baseline.v1.md review notes.