---
title: "Incident Runbook (on-call)"
description: "Symptom → diagnosis → action, for the operator on call for a NovaFabric deployment (nova server API, nova serve dashboard, or a local install)."
canonical: https://novafabric.ai/docs/ops/incident-runbook/
last_updated: 2026-10-23
source: https://github.com/MSKazemi/novafabric/blob/main/docs/ops/incident-runbook.md
---

# Incident Runbook (on-call)

Symptom → diagnosis → action, for the operator on call for a NovaFabric
deployment (`nova server` API, `nova serve` dashboard, or a local install).
Labels follow the [docs honesty rule](https://github.com/MSKazemi/novafabric/blob/main/CONTRIBUTING.md#documentation-status-labels): **works today**,
**experimental**, **planned**, or **future design**. Every command below is
part of the shipped CLI (check `--help` for exact flags on your version —
most of these surfaces are **experimental**).

First moves for almost any incident:

```bash
nova doctor --check-storage      # backend, schema revision, migration state
nova support-bundle              # secret-safe diagnostics tarball (§8)
```

---

## 1. Server won't start / `/readyz` failing

**Status: experimental**
([ADR-0182](https://novafabric.ai/docs/decisions/)).

The two probes mean different things:

- **`/livez`** — process liveness only. Failing ⇒ the process is wedged:
  restart it. It performs *no* dependency checks.
- **`/readyz`** — dependency readiness, with each check **itemized in the
  JSON body**, e.g.
  `{"checks": {"db": "ok", "migrations": "ok", "object_store": "skipped"}}` —
  a failing probe names its cause. (`/health` remains as a compatibility
  alias.)

| Symptom | Diagnosis | Action |
|---|---|---|
| Process exits at startup, TLS key error | `tls.key_path` is group/world-accessible — the server refuses to start | `chmod 600` the TLS key file |
| Offline tokens all return 401 | Server was started without `NOVAFABRIC_OFFLINE_KEY_PATH`, so offline-JWT auth is disabled | Restart with `NOVAFABRIC_OFFLINE_KEY_PATH=<key-dir>/offline-key.pub` (environment only) |
| Process exits at startup, config error | Bad `server.yaml` / env override | The server reads `--config <path>` if given, else `~/.config/novafabric/server.yaml` — no other search path. Unknown keys are ignored silently, so also check that keys are top-level (no `server:` block) ([Server Deployment Guide](https://novafabric.ai/docs/ops/server-deployment/#config-file-reference)) |
| Bind error | Port 7433 (server) / 4321 (serve) already in use | `--port`, or find the stale process. Never start a second writer against the same DB ([ADR-0180](https://novafabric.ai/docs/decisions/) fencing invariant) |
| `/readyz` → `"db": "fail"` | Postgres unreachable / DSN wrong | Verify `$NOVAFABRIC_POSTGRES_DSN`, DB up, network path; `nova doctor --check-storage --backend postgres --postgres-dsn "$NOVAFABRIC_POSTGRES_DSN"` |
| `/readyz` → `"migrations": "fail"` | Schema behind the code (upgrade rolled out without migrating) | `nova db upgrade --track registry --backend postgres` (or `--backend sqlite`) — the probe checks the **registry** track, which a bare `nova db upgrade` does not migrate — then re-probe. See [Upgrade Guide](https://novafabric.ai/docs/ops/upgrade-guide/) §3 |
| `/readyz` → `"object_store"` | Always `"skipped"` today — neither app has an object-store check wired into the probe | Check WORM reachability directly; see §4 |
| `/livez` OK but `/readyz` red for minutes | Dependency down, process healthy | Fix the named dependency; do **not** restart-loop the pod — liveness is fine |
| Kubernetes restart-loops the pod | Probes pointed at the wrong endpoint | Liveness must target `/livez`, readiness `/readyz` — a wedged-DB pod is *not-ready*, not *dead* |

---

## 2. Auth lockout

**Status: experimental**
([ADR-0184](https://novafabric.ai/docs/decisions/),
[ADR-0060](https://novafabric.ai/docs/decisions/)).

| Symptom | Diagnosis | Action |
|---|---|---|
| Everything returns 401 after upgrading to ≥ v0.61 (local mode) | Breaking default: with OIDC off, the server now requires the local bearer token | Read the token from `~/.novafabric/.server-token` (`$HOME`/`$NOVAFABRIC_HOME`, mode 0600). It is printed at startup **only when stderr is an interactive terminal** (ADR-0263): under systemd, Docker/Kubernetes, `nohup`, or CI the start-up line shows a non-reversible *fingerprint* plus the file path instead, and the application logger has never carried the secret. So do **not** expect to find the token in journald or an aggregated log sink — read the file, or pin `NOVAFABRIC_SERVER_TOKEN`. Pin a stable one via `NOVAFABRIC_SERVER_TOKEN` for CI/Docker |
| Lost the local token | Token file is the credential | It survives restarts in `.server-token`; with shell access to the host you can read it. To force a known value, set `NOVAFABRIC_SERVER_TOKEN` and restart (env var wins over the file) |
| Role revoke returns **409 Conflict** | The **last-admin guard**: `rbac_store.revoke_role` refuses any revoke that would leave zero `admin` rows in `role_assignments` while the server config has no OIDC (`oidc:` block; `LastAdminError`, ADR-0060) | This is working as designed. Assign a second admin first (`nova server assign-role`), then revoke |
| SCIM deprovision returns SCIM 409 | Same guard: a deprovision that would remove the last admin is refused and nothing is mutated | Ensure another admin exists before deactivating that user in the IdP |
| No admin exists at all (e.g. someone edited the DB directly) | The guard only protects the HTTP surface; direct `sqlite3`/SQL edits can bypass it | With shell/DB access, re-insert an admin row into `role_assignments`, or use `nova server assign-role <subject> admin` on the host. Restrict filesystem access to the DB to prevent this class |
| OIDC logins fail after IdP key rotation | Stale JWKS cache | `nova server flush-jwks-cache --server <url>` ([Server Deployment Guide — Scenario 3](https://novafabric.ai/docs/ops/server-deployment/)) |
| Air-gapped/CI identity broken | Offline token expired or revoked | Issue a new one: `nova server issue-token --subject <who> --roles <roles> --expires-in <days>d`; revocations live in the `token_audit` table (`nova server revoke-token <jti>`) |

---

## 3. Clients getting 429s (rate limits / quotas)

**Status: experimental**
([ADR-0179](https://novafabric.ai/docs/decisions/)). Both
mechanisms are **off by default** — if you see 429s, someone enabled
the top-level `rate_limits:` block in `server.yaml`. Two distinct 429s:

| Signal | Meaning | Action |
|---|---|---|
| `429`, `error.code = "rate_limited"`, with `Retry-After` + `X-RateLimit-Limit/-Remaining/-Reset` headers | Token-bucket rate limit hit (per principal → tenant → client-IP; separate `ingest`/`read`/`admin` classes) | Client should honor `Retry-After`. Operator: identify the hot key (sustained limiting emits an audit event per window), raise the class budget in `rate_limits`, or fix the runaway client |
| `429`, `error.code = "quota_exceeded"`, **no** `Retry-After` | **Hard storage quota** (capsule count or total bytes) — does not decay on a clock | Free space (retention/[ADR-0134](https://novafabric.ai/docs/decisions/)), raise the quota, or accept the block. Usage is derived from the capsule store with a short TTL cache — recounts lag by seconds |
| Writes succeed but carry `X-NovaFabric-Quota-Warning: <kind> <usage>/<limit>` | **Soft** quota crossed — warn-then-reject phase | Act now, before the hard limit: this is the early warning, and it also emits one audit event per window |
| Probes failing under load | Should be impossible | `/health`, `/livez`, `/readyz`, `/metrics` are **never** rate limited by contract — if a probe 429s, that is a bug: file it |

Buckets are in process memory: they reset on restart, and a restart is a
(crude) way to clear a wedged limiter.

---

## 4. Full disk / WORM store issues

**Status: local paths work today; WORM object-store tier experimental.**

| Symptom | Diagnosis | Action |
|---|---|---|
| Captures failing, SQLite errors, server 5xx | Disk full under `~/.novafabric` (capsules, `registry.db`, Merkle log) | Free space first. Apply retention policies rather than hand-deleting capsule directories — capsules are the evidence; deletion should itself be recorded ([ADR-0134](https://novafabric.ai/docs/decisions/)). Then run the §5 drift checks |
| WORM writes rejected | Bucket unreachable, credentials expired, or Object Lock/immutability policy blocking a non-compliant write | Check endpoint config and credentials (`/readyz` has no object-store check today — it always reports `skipped`). Remember WORM is *supposed* to reject overwrites/deletes of locked objects — verify you are not trying to mutate immutable evidence |
| "Cleanup" of the WORM bucket fails | Object Lock / legal hold doing its job | Working as designed. Retention on WORM data is policy-driven, not `rm`-driven |
| Metadata DB lost or corrupted, WORM bucket intact | Derived state lost; evidence safe | The metadata DB is rebuildable from the object store's manifest chain: `nova rebuild-metadata-db --backend <s3\|minio\|ceph_rgw\|azure_blob\|gcs> --target-db <path>` (the in-memory `local` backend is refused — it holds nothing to rebuild from); then `nova doctor --check-storage`. Losing the app node loses no evidence (ADR-0180 D2) |

---

## 5. Capsule index drift (dashboard shows wrong/missing runs)

**Status: works today.** The dashboard's `runs_cache` table (inside
`registry.db`) is **a cache, never the source of truth** — it is always
rebuildable from the capsule filesystem
(`src/novafabric/registry/runs_cache.py`).

| Symptom | Diagnosis | Action |
|---|---|---|
| Runs on disk but missing in the dashboard | `runs_cache` drifted from the capsule dir (e.g. capsules copied in out-of-band) | Restart `nova serve` — it rebuilds the runs index from capsule files on startup and keeps it current via the background stats-refresh thread. Confirm `--capsule-dir`/`--db-path` point where you think they do |
| Dashboard **lists runs whose capsule is gone** — every drill-in 404s | The rebuild is **additive** by design (it never deletes a row for a capsule it cannot see), so rows for capsules deleted or moved out-of-band survive restarts forever | **v0.98.0, opt-in:** `POST /api/admin/reindex-runs` with `{"confirmed": true, "prune": true}` drops index rows whose capsule directory no longer exists; the response adds `pruned`. Default is `false` — a rebuild without it stays additive. Capsules are never touched, only the derived row. Reachable from the dashboard's **Infra → Maintenance** card. Rebuild the row later by copying the capsule back and restarting |
| Lineage queries missing edges for capsules you restored/copied in | Lineage graph is also a derived cache | Re-import: `nova lineage import <capsule-or-parent-dir>` (one path: a capsule, or a directory of capsules) |
| Doubt about DB health itself | — | `nova doctor --check-storage` (schema version, migration state, per-table row counts); server-mode rebuild path: §4 last row |

---

## 6. Seal verification failures (`nova verify`)

**Status: works today.** `nova verify <capsule-dir>` checks three layers in
the capsule's `.seal/` directory — DSSE ECDSA P-256 signature, RFC 3161
timestamp (structural hash), Merkle log inclusion — printing each check and
**exiting 0 only if all pass, 1 on a failed check, 2 on a usage error** (there
are no distinct per-cause exit codes; read the itemized ✓/✗ output).

| Output | Diagnosis | Action |
|---|---|---|
| `No .seal/ directory found` (exit 1) | Capsule was never sealed — NovaSeal is opt-in | Not tampering. Configure `~/.novafabric/novaseal.yaml` (or `NOVAFABRIC_SEAL_CONFIG`) so future captures are sealed ([NovaSeal configuration](https://novafabric.ai/docs/novaseal-configuration/)) |
| `⊘ Merkle log inclusion: NOT CHECKED — log not available` (exit 0) | The capsule carries no inclusion proof (sealed by ≤ v0.102.x) and this host has no Merkle log holding its entry — e.g. an auditor's machine | Expected on an independent host. To also check inclusion, verify with `--seal-config` pointing at the profile (`merkle_db:`) used at signing time. Since v0.103 there is no `NovaSeal is not configured` failure: verification needs only the capsule |
| `Signature … FAIL` | Capsule content changed since signing, or wrong key/cert profile | Treat as a potential integrity incident: preserve the capsule, compare against backups, check which profile signed it |
| `Timestamp … FAIL` | A timestamp token is present but does not verify (damaged, or does not cover the signature) | Treat as a potential integrity incident |
| `⊘ Timestamp (RFC 3161): NOT PRESENT` (exit unaffected) | Capsule sealed without a `tsa_url` — timestamping is opt-in; air-gapped installs legitimately skip RFC 3161 | Expected, not tampering; it means the seal carries no trusted time |
| `Merkle log inclusion … FAIL` | `log-entry.json` is inconsistent: its entry does not hash to `leaf_hash`, names a different capsule, carries an inclusion proof that does not recompute its root, or the local log holds the entry at a different index — or the local log itself is damaged. (A log that simply lacks the entry is *not checked*, not failed.) | Treat as a potential integrity incident; audit the log itself with `nova seal log verify` (sampled; `--full` for O(N) re-hash) |
| Sigstore backend errors | `--backend sigstore` needs the `novafabric[sigstore]` extra and a stored bundle; Rekor/Fulcio need network | Use the local backend in air-gapped deployments ([Air-gapped guide](https://novafabric.ai/docs/ops/air-gapped-install/)) |

Escalation: repeated signature failures across capsules ⇒ treat as key
compromise until proven otherwise —
[NovaSeal key management](https://novafabric.ai/docs/novaseal-key-management/) has the compromise
procedure.

---

## 7. Emergency bypass of maker-checker SoD (`nova seal bypass`)

**Status: works today**
([ADR-0059](https://novafabric.ai/docs/decisions/),
[ADR-0058](https://novafabric.ai/docs/decisions/)). Use only
when the two-person Separation-of-Duties (SoD) requirement for a capsule
promotion genuinely cannot be satisfied in time (e.g. the only other
authorized approver is unreachable during a production incident) — **every
bypass creates a permanent, signed audit trail** and is visible in the
dashboard **SealTab**.

| Symptom | Diagnosis | Action |
|---|---|---|
| `nova seal verify` exits 8 or 9 (`Approval bundle not found` / `Proposal bundle not found`; `nova promote approve` uses 1/3/4/6 instead) and the missing party is genuinely unavailable | Normal two-step SoD flow cannot complete in the required window | Do **not** work around this by sharing keys or approving as the proposer — that is exactly what SoD prevents. Use the bypass below instead, so the deviation is recorded rather than hidden. |
| Need to move a capsule to production right now, no second approver reachable | Emergency bypass is the documented escape hatch | `nova seal bypass <capsule-id> --reason "<≥50-char justification>" --duration 24h --key <your-key.pem> --cert <your-cert.pem> --target-env production --notify oncall@example.com`. `--reason` is enforced at ≥50 characters — write what happened and why, not a placeholder. `--duration` caps at `168h` (7 days); prefer the shortest window that covers the incident. |
| Need to confirm a bypass is actually in effect before proceeding | `nova seal verify <capsule-id>` (command name corrected 2026-07-30 — the CLI registers it as `verify`, not `verify-sod`) prints `SoD verification passed (bypass active — SoD checks skipped)` when a live bypass covers it, vs. a normal pass/fail otherwise | Treat "bypass active" the same as any other incident action — note it in the incident log, not just in the tool's own audit trail |
| Bypass created but `--notify` recipients report never hearing about it | **Verified 2026-07-30: nothing you configure will fix this today.** `NOVA_BYPASS_NOTIFY_FILE` / `NOVA_BYPASS_NOTIFY_WEBHOOK` (`promote/bypass_notify.py::build_notifier_from_env`) are implemented and unit-tested, but no production call site wires them up — every `PromoteBundleStore(...)` construction in `cli/seal_propose.py`, `serve/app.py`, and `server/routes/seal.py` uses the default `NullBypassNotifier`. Setting either env var currently has **zero effect**; treat `--notify` as documentation-of-intent recorded in the signed predicate, not a live delivery | Manually notify the listed recipients yourself — this is the actual, currently-required step, not a fallback. Do not rely on the env vars until this wiring gap is closed (flagged as a real code gap, not a doc gap) |
| Post-incident: need to review all bypasses used during an incident window | Every bypass is a signed, stored predicate (`promote_bypass_v1.json` schema) in the NovaSeal bundle store, plus a `human_approval` capture event when a run was active | Dashboard **SealTab** lists active/expired bypasses with approver identity; for a full audit trail export, include the bundle store in a [support bundle](#9-collecting-a-support-bundle) |

**Authorization model, stated honestly:** the bypass is gated by *possession
of an ECDSA P-256 key + X.509 certificate* (`--key`/`--cert`), not by an RBAC
`admin` role lookup — the signer's certificate Common Name becomes the
recorded `bypass_authorized_by` identity. Restrict who holds a signing key
the same way you would restrict an admin role; the tool cannot enforce an
org-chart policy it has no way to observe.

---

## 8. Restoring from backup

Follow the [Backup & Restore Runbook](https://novafabric.ai/docs/ops/backup-restore/) — do not improvise:

- Local-profile sets: `nova restore <set.tar.gz>` (verifies first, extracts
  safely, migrates to head, replays crypto-shreds, then runs the
  verification chain — it reports ok **only** when verification passes).
- Postgres: `pg_restore --clean --if-exists` per runbook §1.2, then
  `nova db upgrade` and the §1.5 verification chain.
- **A restore is not done until verification passes** (`nova doctor
  --check-storage`, `nova seal log verify`, `nova ledger verify`,
  spot-check `nova validate`).

---

## 9. Collecting a support bundle

**Status: experimental**
([ADR-0187](https://novafabric.ai/docs/decisions/)). Works in
local mode and offline.

```bash
nova support-bundle                          # ./nova-support-bundle-<ts>.tar.gz
nova support-bundle -o /tmp/diag.tar.gz --log-window-hours 24
```

Allowlist-only and deny-by-default: doctor output, versions, env-var
**names** only, health snapshot, secret-redacted config, bounded recent logs
(via `NOVAFABRIC_LOG_DIR` / `$NOVAFABRIC_HOME/logs`), plus a manifest with
the SHA-256 of every member and the redaction-ruleset version. It never
contains tokens, keys, credentials, capsule payloads, prompts, responses, or
env-var values — safe to attach to a ticket as-is.