Incident Runbook (on-call)

Symptom → diagnosis → action, for the operator on call for a NovaFabric deployment (nova server API, nova serve dashboard, or a local install). Labels follow the docs honesty rule: works today, experimental, planned, or future design. Every command below is part of the shipped CLI (check --help for exact flags on your version — most of these surfaces are experimental).

First moves for almost any incident:

nova doctor --check-storage      # backend, schema revision, migration state
nova support-bundle              # secret-safe diagnostics tarball (§8)

1. Server won't start / /readyz failing

Status: experimental (ADR-0182).

The two probes mean different things:

Symptom Diagnosis Action
Process exits at startup, key-file error Offline key file is world-readable — the server refuses to start chmod 600 the offline_key_path file; owner must be the server user
Process exits at startup, config error Bad nova-server.yaml / env override Check resolution order: --config flag → $NOVA_SERVER_CONFIG → ~/.config/novafabric/nova-server.yaml → /etc/novafabric/nova-server.yaml (Server Deployment Guide)
Bind error Port 7433 (server) / 4321 (serve) already in use --port, or find the stale process. Never start a second writer against the same DB (ADR-0180 fencing invariant)
/readyz → "db": "fail" Postgres unreachable / DSN wrong / pool exhausted Verify $NOVA_DSN, DB up, network path; nova doctor --check-storage --backend postgres --postgres-dsn "$NOVA_DSN"
/readyz → "migrations": "fail" Schema behind the code (upgrade rolled out without migrating) nova db upgrade (or alembic -c alembic-postgres.ini upgrade head), then re-probe. See Upgrade Guide §3
/readyz → "object_store": "fail" Configured WORM store unreachable Check endpoint/credentials; see §4. ("skipped" = not configured — that is normal, not a failure)
/livez OK but /readyz red for minutes Dependency down, process healthy Fix the named dependency; do not restart-loop the pod — liveness is fine
Kubernetes restart-loops the pod Probes pointed at the wrong endpoint Liveness must target /livez, readiness /readyz — a wedged-DB pod is not-ready, not dead

2. Auth lockout

Status: experimental (ADR-0184, ADR-0060).

Symptom Diagnosis Action
Everything returns 401 after upgrading to ≥ v0.61 (local mode) Breaking default: with OIDC off, the server now requires the local bearer token Read the token from ~/.novafabric/.server-token ($NOVAFABRIC_HOME/.server-token, mode 0600) — also printed at startup, on stderr only: since v0.98.0 the secret is deliberately never written to the application logger, so do not expect to find it in journald or an aggregated log sink (the logger records only the token-file path). Pin a stable one via NOVAFABRIC_SERVER_TOKEN for CI/Docker
Lost the local token Token file is the credential It survives restarts in .server-token; with shell access to the host you can read it. To force a known value, set NOVAFABRIC_SERVER_TOKEN and restart (env var wins over the file)
Role revoke returns 409 Conflict The last-admin guard: rbac_store.revoke_role refuses any revoke that would leave zero admin rows in role_assignments while no OIDC issuer is configured (LastAdminError, ADR-0060) This is working as designed. Assign a second admin first (nova server assign-role), then revoke
SCIM deprovision returns SCIM 409 Same guard: a deprovision that would remove the last admin is refused and nothing is mutated Ensure another admin exists before deactivating that user in the IdP
No admin exists at all (e.g. someone edited the DB directly) The guard only protects the HTTP surface; direct sqlite3/SQL edits can bypass it With shell/DB access, re-insert an admin row into role_assignments, or use nova server assign-role <subject> admin on the host. Restrict filesystem access to the DB to prevent this class
OIDC logins fail after IdP key rotation Stale JWKS cache nova server flush-jwks-cache --server <url> (Server Deployment Guide — Scenario 3)
Air-gapped/CI identity broken Offline token expired or revoked Issue a new one: nova server issue-token --subject <who> --roles <roles> --expires-in <days>d; revocations live in the token_audit table (nova server revoke-token <jti>)

3. Clients getting 429s (rate limits / quotas)

Status: experimental (ADR-0179). Both mechanisms are off by default — if you see 429s, someone enabled server.rate_limits in the config. Two distinct 429s:

Signal Meaning Action
429, error.code = "rate_limited", with Retry-After + X-RateLimit-Limit/-Remaining/-Reset headers Token-bucket rate limit hit (per principal → tenant → client-IP; separate ingest/read/admin classes) Client should honor Retry-After. Operator: identify the hot key (sustained limiting emits an audit event per window), raise the class budget in server.rate_limits, or fix the runaway client
429, error.code = "quota_exceeded", no Retry-After Hard storage quota (capsule count or total bytes) — does not decay on a clock Free space (retention/ADR-0134), raise the quota, or accept the block. Usage is derived from the capsule store with a short TTL cache — recounts lag by seconds
Writes succeed but carry X-NovaFabric-Quota-Warning: <kind> <usage>/<limit> Soft quota crossed — warn-then-reject phase Act now, before the hard limit: this is the early warning, and it also emits one audit event per window
Probes failing under load Should be impossible /health, /livez, /readyz, /metrics are never rate limited by contract — if a probe 429s, that is a bug: file it

Buckets are in process memory: they reset on restart, and a restart is a (crude) way to clear a wedged limiter.

4. Full disk / WORM store issues

Status: local paths work today; WORM object-store tier experimental.

Symptom Diagnosis Action
Captures failing, SQLite errors, server 5xx Disk full under ~/.novafabric (capsules, registry.db, Merkle log) Free space first. Apply retention policies rather than hand-deleting capsule directories — capsules are the evidence; deletion should itself be recorded (ADR-0134). Then run the §5 drift checks
WORM writes rejected Bucket unreachable, credentials expired, or Object Lock/immutability policy blocking a non-compliant write Check /readyz's object_store check, endpoint config and credentials. Remember WORM is supposed to reject overwrites/deletes of locked objects — verify you are not trying to mutate immutable evidence
"Cleanup" of the WORM bucket fails Object Lock / legal hold doing its job Working as designed. Retention on WORM data is policy-driven, not rm-driven
Metadata DB lost or corrupted, WORM bucket intact Derived state lost; evidence safe The metadata DB is rebuildable from the object store's manifest chain: nova rebuild-metadata-db --backend <s3|minio|ceph_rgw|azure_blob|local> --target-db <path>; then nova doctor --check-storage. Losing the app node loses no evidence (ADR-0180 D2)

5. Capsule index drift (dashboard shows wrong/missing runs)

Status: works today. The dashboard's runs_cache table (inside registry.db) is a cache, never the source of truth — it is always rebuildable from the capsule filesystem (src/novafabric/registry/runs_cache.py).

Symptom Diagnosis Action
Runs on disk but missing in the dashboard runs_cache drifted from the capsule dir (e.g. capsules copied in out-of-band) Restart nova serve — it rebuilds the runs index from capsule files on startup and keeps it current via the background stats-refresh thread. Confirm --capsule-dir/--db-path point where you think they do
Dashboard lists runs whose capsule is gone — every drill-in 404s The rebuild is additive by design (it never deletes a row for a capsule it cannot see), so rows for capsules deleted or moved out-of-band survive restarts forever v0.98.0, opt-in: POST /api/admin/reindex-runs with {"confirmed": true, "prune": true} drops index rows whose capsule directory no longer exists; the response adds pruned. Default is false — a rebuild without it stays additive. Capsules are never touched, only the derived row. Reachable from the dashboard's Infra → Maintenance card. Rebuild the row later by copying the capsule back and restarting
Lineage queries missing edges for capsules you restored/copied in Lineage graph is also a derived cache Re-import: nova lineage import <capsule-dir>... (imports lineage edges from capsule directories)
Doubt about DB health itself — nova doctor --check-storage (schema version, migration state, per-table row counts); server-mode rebuild path: §4 last row

6. Seal verification failures (nova verify)

Status: works today. nova verify <capsule-dir> checks three layers in the capsule's .seal/ directory — DSSE ECDSA P-256 signature, RFC 3161 timestamp (structural hash), Merkle log inclusion — printing each check and exiting 0 only if all pass, 1 otherwise (there are no distinct per-cause exit codes; read the itemized ✓/✗ output).

Output Diagnosis Action
No .seal/ directory found (exit 1) Capsule was never sealed — NovaSeal is opt-in Not tampering. Configure ~/.novafabric/novaseal.yaml (or NOVAFABRIC_SEAL_CONFIG) so future captures are sealed (NovaSeal configuration)
NovaSeal is not configured (exit 1) Verification host has no seal config, so it cannot locate the Merkle DB Point --seal-config/NOVAFABRIC_SEAL_CONFIG at the same profile (esp. merkle_db:) used at signing time
Signature … FAIL Capsule content changed since signing, or wrong key/cert profile Treat as a potential integrity incident: preserve the capsule, compare against backups, check which profile signed it
Timestamp … FAIL Timestamp token missing/damaged — e.g. capsule sealed with no tsa_url (air-gapped installs legitimately skip RFC 3161) Check whether your seal profile sets tsa_url; if it never did, an absent timestamp is expected, not tampering
Merkle log inclusion … FAIL Verifier's Merkle DB is not the one that recorded the seal (common on shared FS/multi-host setups), or the log is damaged Verify merkle_db: paths match across hosts; audit the log itself with nova seal log verify (sampled; --full for O(N) re-hash)
Sigstore backend errors --backend sigstore needs the novafabric[sigstore] extra and a stored bundle; Rekor/Fulcio need network Use the local backend in air-gapped deployments (Air-gapped guide)

Escalation: repeated signature failures across capsules ⇒ treat as key compromise until proven otherwise — NovaSeal key management has the compromise procedure.

7. Emergency bypass of maker-checker SoD (nova seal bypass)

Status: works today (ADR-0059, ADR-0058). Use only when the two-person Separation-of-Duties (SoD) requirement for a capsule promotion genuinely cannot be satisfied in time (e.g. the only other authorized approver is unreachable during a production incident) — every bypass creates a permanent, signed audit trail and is visible in the dashboard SealTab.

Symptom Diagnosis Action
nova promote approve / nova seal verify exits 8 or 9 (Approval bundle not found / Proposal bundle not found) and the missing party is genuinely unavailable Normal two-step SoD flow cannot complete in the required window Do not work around this by sharing keys or approving as the proposer — that is exactly what SoD prevents. Use the bypass below instead, so the deviation is recorded rather than hidden.
Need to move a capsule to production right now, no second approver reachable Emergency bypass is the documented escape hatch nova seal bypass <capsule-id> --reason "<≥50-char justification>" --duration 24h --key <your-key.pem> --cert <your-cert.pem> --target-env production --notify oncall@example.com. --reason is enforced at ≥50 characters — write what happened and why, not a placeholder. --duration caps at 168h (7 days); prefer the shortest window that covers the incident.
Need to confirm a bypass is actually in effect before proceeding nova seal verify <capsule-id> (command name corrected 2026-07-30 — the CLI registers it as verify, not verify-sod) prints SoD verification passed (bypass active — SoD checks skipped) when a live bypass covers it, vs. a normal pass/fail otherwise Treat "bypass active" the same as any other incident action — note it in the incident log, not just in the tool's own audit trail
Bypass created but --notify recipients report never hearing about it Verified 2026-07-30: nothing you configure will fix this today. NOVA_BYPASS_NOTIFY_FILE / NOVA_BYPASS_NOTIFY_WEBHOOK (promote/bypass_notify.py::build_notifier_from_env) are implemented and unit-tested, but no production call site wires them up — every PromoteBundleStore(...) construction in cli/seal_propose.py, serve/app.py, and server/routes/seal.py uses the default NullBypassNotifier. Setting either env var currently has zero effect; treat --notify as documentation-of-intent recorded in the signed predicate, not a live delivery Manually notify the listed recipients yourself — this is the actual, currently-required step, not a fallback. Do not rely on the env vars until this wiring gap is closed (flagged as a real code gap, not a doc gap)
Post-incident: need to review all bypasses used during an incident window Every bypass is a signed, stored predicate (promote_bypass_v1.json schema) in the NovaSeal bundle store, plus a human_approval capture event when a run was active Dashboard SealTab lists active/expired bypasses with approver identity; for a full audit trail export, include the bundle store in a support bundle

Authorization model, stated honestly: the bypass is gated by possession of an ECDSA P-256 key + X.509 certificate (--key/--cert), not by an RBAC admin role lookup — the signer's certificate Common Name becomes the recorded bypass_authorized_by identity. Restrict who holds a signing key the same way you would restrict an admin role; the tool cannot enforce an org-chart policy it has no way to observe.

8. Restoring from backup

Follow the Backup & Restore Runbook — do not improvise:

9. Collecting a support bundle

Status: experimental (ADR-0187). Works in local mode and offline.

nova support-bundle                          # ./nova-support-bundle-<ts>.tar.gz
nova support-bundle -o /tmp/diag.tar.gz --log-window-hours 24

Allowlist-only and deny-by-default: doctor output, versions, env-var names only, health snapshot, secret-redacted config, bounded recent logs (via NOVAFABRIC_LOG_DIR / $NOVAFABRIC_HOME/logs), plus a manifest with the SHA-256 of every member and the redaction-ruleset version. It never contains tokens, keys, credentials, capsule payloads, prompts, responses, or env-var values — safe to attach to a ticket as-is.