---
title: "Server Administration Guide (multi-user nova server)"
description: "This guide is for the administrator of a shared, multi-user NovaFabric deployment — the nova server REST API (/v0), not the single-operator nova serve…"
canonical: https://novafabric.ai/docs/ops/server-admin-guide/
last_updated: 2026-10-24
source: https://github.com/MSKazemi/novafabric/blob/main/docs/ops/server-admin-guide.md
---

# Server Administration Guide (multi-user `nova server`)

This guide is for the administrator of a shared, multi-user NovaFabric
deployment — the `nova server` REST API (`/v0`), not the single-operator
`nova serve` dashboard. It covers identity, roles, tenancy, and audit
administration. For installation and deployment topologies see the
[Server Deployment Guide](https://novafabric.ai/docs/ops/server-deployment/); this guide does not repeat it.

Everything below is labelled per the docs honesty rule: **works today**,
**experimental**, **planned**, or **future design**.

> **The two servers, in one line:** `nova server` is the team/API surface
> (OIDC, RBAC, `/v0`); `nova serve` is the localhost dashboard (single shared
> token). See [`docs/api-reference.md`](https://novafabric.ai/docs/api-reference/) for the full
> disambiguation table.

---

## 1. Enabling server mode

**Status: experimental** (v0.7+,
[ADR-0017](https://novafabric.ai/docs/decisions/),
[ADR-0029](https://novafabric.ai/docs/decisions/)).

```bash
pip install 'novafabric[server]'
nova server start                       # 127.0.0.1:7433 by default
nova server start --host 0.0.0.0 --port 8080
```

Configuration lives in `~/.config/novafabric/server.yaml`; only `--config <path>`
points the server at another file (for example `/etc/novafabric/server.yaml`).
`NOVAFABRIC_SERVER_*` environment variables override individual settings, and are held to the
same bounds as the YAML keys: an out-of-range value (`NOVAFABRIC_SERVER_PORT=99999`,
`NOVAFABRIC_SERVER_INGEST_MAX_CONCURRENT_UNPACK=0`) makes `nova server start` exit `2` with
`Invalid server config`, naming the field. Secrets
are **environment-only** and stripped from YAML: `NOVAFABRIC_POSTGRES_DSN`,
`NOVAFABRIC_OFFLINE_KEY_PATH`, `NOVAFABRIC_SCIM_TOKEN` and `NOVAFABRIC_SERVER_TOKEN`
(`server/config.py::load_config`). See [Server Deployment Guide — Scenario 2](https://novafabric.ai/docs/ops/server-deployment/) for the
Postgres backend and migration from SQLite.

> **Local auth (ADR-0184, experimental — shipped 2026-07-16).** With OIDC
> disabled, the server requires an auto-generated local bearer token (printed
> at startup, stored at `$NOVAFABRIC_HOME/.server-token` mode 0600, pinnable via
> `NOVAFABRIC_SERVER_TOKEN`). The old anonymous-admin behavior needs the
> explicit `--insecure-no-auth` opt-out and refuses non-loopback binds
> without `--i-know-this-is-public`
> ([ADR-0184](https://novafabric.ai/docs/decisions/)). Each insecure start logs a warning **and**
> appends a `server.insecure_no_auth` entry (bind host, port, loopback,
> public-bind confirmation) to the hash-chained audit log at
> `NOVAFABRIC_AUDIT_LOG_PATH` (default `$XDG_DATA_HOME/novafabric/audit.jsonl`, else
> `~/.local/share/novafabric/audit.jsonl`; `NOVAFABRIC_HOME` does not move it — set the
> variable explicitly to put the log on a data volume).
> If the entry cannot be written the server refuses to start in insecure mode.

## 2. Identity backends

| Backend | Status | Notes |
|---|---|---|
| OIDC (JWT Bearer + JWKS) | **experimental** | The primary team backend. Setup: [Server Deployment Guide — Scenario 3](https://novafabric.ai/docs/ops/server-deployment/). JWKS cache flush: `nova server flush-jwks-cache`. [ADR-0018](https://novafabric.ai/docs/decisions/) |
| Offline ed25519 tokens | **experimental** | Air-gapped/CI/SLURM machine identity without an IdP. `nova server issue-token --subject worker-01 --roles reader,writer --expires-in 30d`; revoke with `nova server revoke-token <jti>`. Revocations are recorded in the `token_audit` table. |
| SCIM 2.0 provisioning | **experimental** | Off by default (endpoints 404). Enable with `NOVAFABRIC_SERVER_SCIM_ENABLED=1` **and** `NOVAFABRIC_SCIM_TOKEN`. `/scim/v2/*` per RFC 7644; all provisioning actions land in the append-only `scim_audit_events` table. [ADR-0139](https://novafabric.ai/docs/decisions/) |
| SAML 2.0 SSO | **experimental, opt-in** | SP metadata via `nova server saml-metadata`. `POST /v0/auth/saml/acs` consumes assertions only when `saml.experimental_acs_enabled: true` is set in `server.yaml` **and** `novafabric[saml]` (signxml + lxml, cleared the license gate in v0.73.0) is installed; otherwise it returns **501**. Even then a verified assertion is answered **501** `saml_session_unavailable`: the server cannot yet issue a session for a SAML login. NovaFabric never skips assertion signature verification. A Security-Architect review remains a pre-production blocking condition ([ADR-0138](https://novafabric.ai/docs/decisions/) §D5). |
| Device-grant demo flow | **off by default (endpoints 404)** | The RFC 8628 `/v0/auth/device/code\|token\|approve` flow is local/testing scaffolding whose HS256 tokens the real verifier never honours; `/approve` is unauthenticated. It stays unmounted unless you set `NOVAFABRIC_SERVER_DEMO_DEVICE_GRANT=1`. **Never enable it in production** — use OIDC or offline tokens instead. [ADR-0198](https://novafabric.ai/docs/decisions/) |

## 3. Roles and authorization

**Status: experimental.**

[ADR-0018](https://novafabric.ai/docs/decisions/) defines six built-in roles.
Four are enforced as route-level checks:

| Role | Grants |
|---|---|
| `reader` | Read access to runs, capsules, lineage |
| `writer` | `reader` + create/ingest |
| `admin` | Everything, including role management |
| `auditor` | Orthogonal — audit trails and evidence only; cannot write |

The hierarchy is `reader < writer < admin`; `admin` satisfies every check;
`auditor` satisfies only auditor checks. The remaining two roles —
`promoter` and `approver` — implement separation of duties in the
maker-checker promotion and NovaSeal approval flows
([ADR-0058](https://novafabric.ai/docs/decisions/)); the same
identity can never serve as both maker and checker for one proposal.

Assigning roles (either surface writes the same `role_assignments` table):

```bash
nova server assign-role alice@example.com writer
# or over HTTP (admin-gated):
#   GET/POST /v0/admin/roles, DELETE /v0/admin/roles/{subject}/{role}
```

⚠ A `role_assignments` row is consulted only once the workspace/org model is in
use (a scoped membership or service account exists). In a stock deployment it
grants nothing: `assign-role` then warns `this assignment is not in force` on
stderr and `GET /v0/admin/roles` reports `effective_now: false`. Carry the role
in the token instead (OIDC `nova_roles`/`roles` claim, or `issue-token --roles`).

**Experimental:** org/workspace-scoped role bindings and named service accounts
(`/v0/orgs`, `/v0/workspaces`, `/v0/service-accounts`) —
[ADR-0178](https://novafabric.ai/docs/decisions/). Creating the first membership or service account is
what switches `role_assignments` rows on.

What a binding grants, when a credential's own roles fall short:

- A **global** `role_assignments` row grants its role everywhere.
- A **workspace or org membership** grants its role in full only on that
  workspace's own routes (`/v0/workspaces/{ws_id}/…`): a workspace `admin` can
  manage that workspace's memberships, not another's. On every other route a
  membership grants at most `writer` — enough for the data plane (the capsule store
  is not partitioned by workspace), never server-wide `admin`: a workspace admin
  cannot mint API keys, create orgs or assign global roles.
- A credential that carries **its own role set** — an API key, or an offline token
  issued with `--roles` — gets exactly those roles and never its owner's bindings,
  so a least-privilege key stays least-privilege. A role-less service-account token
  takes its roles from its bindings.
- **SCIM:** a user who is `active: false` gets no role from any mapped group, even
  when the IdP adds them to one; reactivation restores the roles of their groups.

## 4. Tenancy and isolation

**Status: experimental** (Postgres backend only).

With the Postgres metadata store, tenant isolation is enforced **in the
database**, not in application code: `FORCE ROW LEVEL SECURITY` with a
`tenant_isolation` policy on `runs`, `capsules`, `signatures`, and
`retention_policies`, a per-transaction `SET LOCAL app.current_tenant_id`
(safe under pgBouncer transaction pooling), and a split between the
`novafabric_app` role (no `BYPASSRLS`) and `novafabric_migrator`
([ADR-0040](https://novafabric.ai/docs/decisions/),
[ADR-0052](https://novafabric.ai/docs/decisions/)).
The policy holds only when the server connects as `novafabric_app` (or another
role without `BYPASSRLS`). A Postgres superuser bypasses row-level security even
under `FORCE`, so never point `NOVAFABRIC_METADATA_DSN` at a superuser. The app
role is granted `SELECT, INSERT, UPDATE` on the tenant tables. Capsule deletion
(`DELETE /v0/capsules/{run_id}`, `POST /v0/capsules/bulk-delete`) also needs `DELETE` on
`runs`, `capsules` and `signatures`, which migration `v005` grants — run
`nova db upgrade --backend postgres` after upgrading.
A CI gate (`metadata_store_security_gate`) re-proves cross-tenant isolation on
every change.

Tenancy today is a single flat `tenant_id` per deployment-defined scope. There
is **no** organization/workspace/team hierarchy yet — that is
[ADR-0178](https://novafabric.ai/docs/decisions/)
(`future design`), which keeps `tenant_id` as the sole RLS key.

## 5. Audit trails

**Status: works today** (local + server).

- Hash-chained append-only audit log; domain trails for SCIM
  (`scim_audit_events`), tokens (`token_audit`), SAML (redacted records), and
  retention decisions ("deletion is evidence", [ADR-0134](https://novafabric.ai/docs/decisions/)).
- `nova audit …` maps capsule evidence to regulatory controls;
  `nova ledger anchor|verify|status` runs the adversary-anchored
  accountability ledger ([ADR-0094](https://novafabric.ai/docs/decisions/));
  `nova seal log …` operates the Merkle log.
- Give compliance staff the `auditor` role — read-only by construction.

## 6. API behavior an admin should know

- **Versioning:** all resource routes sit under `/v0`; the canonical contract
  is [`api/openapi.yaml`](https://github.com/MSKazemi/novafabric/blob/main/api/openapi.yaml). A formal deprecation/sunset
  policy is proposed in [ADR-0188](https://novafabric.ai/docs/decisions/)
  (`future design`).
- **Ids in requests are names, never paths** (**works today**): a `run_id`
  in a request (`POST /v0/evidence`, `POST /v0/replays`, `GET|DELETE
  /v0/capsules/{run_id}`) and a `run_id`/`parent_run_id` in an uploaded
  `capsule.yaml` must be one path segment: no `/`, `\`, `..`, NUL or other
  control character, not absolute, at most 255 bytes, else **400**. An entry of
  the capsule store that is a symbolic link is refused with **400** by the read,
  evidence and replay routes, not followed. `POST /v0/evidence` writes the bundle only to
  `<evidence dir>/<run_id>.zip` (`NOVAFABRIC_EVIDENCE_DIR`, default
  `$NOVAFABRIC_HOME/evidence`), written to a new file and renamed into place, and
  returns that path as `bundle_path`. A request that names an `output_path` is
  refused with **400**: before v0.107.0 a `writer` used it to overwrite any file
  the server could write (review server-01). The signing key the server creates on
  first export (`$NOVAFABRIC_HOME/keys/signing_key.pem`) is written mode `0600`.
- **Pagination:** cursor-based, default 50 / max 500 per page (**works
  today**). `GET /v0/capsules` and `GET /v0/assets` now serve **keyset**
  cursors — see 6a (**experimental**).
- **Rate limiting/quotas:** **experimental, default off** — in-process rate
  limiting + storage quotas ([ADR-0179](https://novafabric.ai/docs/decisions/)),
  plus per-workspace usage metering, `GET /v0/usage` reporting,
  per-workspace budgets, and the `nova server usage reconcile|export`
  drift-reconciliation / chargeback-export CLI
  ([ADR-0208](https://novafabric.ai/docs/decisions/)).
  See [Quotas & rate limits](https://novafabric.ai/docs/ops/quotas-and-rate-limits/); with the master
  switch off (the default), plan capacity accordingly.
- **Health and self-observability** (**experimental**,
  [ADR-0182](https://novafabric.ai/docs/decisions/)): `GET /health` and `GET /livez` (unauthenticated
  liveness), `GET /readyz` (database, migration-head and object-store checks),
  `GET /v0/version` (`reader` role) and Prometheus `GET /metrics` (`reader` role
  unless `observability.metrics_exempt: true`; needs `prometheus-client` from the
  `[server]` extra). All four are exempt from rate limiting. For deeper diagnostics
  use `nova doctor` (add `--check-storage` for schema/migration state).

### 6a. Bulk capsule operations + keyset pagination (experimental, ADR-0206)

**Status: experimental** — shipped by
[ADR-0206](https://novafabric.ai/docs/decisions/). The normative contract is
specified in the maintainers' private `design/` tree and is not published.

**Keyset pagination on `GET /v0/capsules`:**

- List order is pinned to `created_at DESC, run_id DESC`; `next_cursor` is
  an opaque v1 keyset cursor — pass it back verbatim. Page requests are
  O(page) against a derived runs-cache index (lazily backfilled from the
  capsule directory, which stays the source of truth) instead of re-parsing
  every `capsule.yaml`.
- `total` appears on first-page (and legacy-cursor) responses only; cursor
  pages omit it by design.
- A corrupted/tampered cursor is now a **400 `invalid_cursor`** — it no
  longer silently restarts from the first page.
- Legacy `{"offset": N}` cursors still work for one deprecation cycle
  (ADR-0188); those responses carry `Deprecation: true`. Set
  `pagination.legacy_offset_cursors: false` (a top-level `pagination:` block in `server.yaml`)
  (`NOVAFABRIC_SERVER_PAGINATION_LEGACY_OFFSET_CURSORS=false`) to refuse
  them early.

**Keyset pagination on `GET /v0/assets`** (experimental, ADR-0206 P2) — same
cursor format and rules as above: order `created_at DESC, id DESC`, seek in
the registry SQL (`spec_json` is never read for a list page), `total` on the
first page only, 400 `invalid_cursor` on garbage (previously a silent restart
at page one), legacy `{"offset": N}` cursors served with `Deprecation: true`
until `legacy_offset_cursors: false`. Cursors are not bound to the
`asset_type`/`status` filters — reuse a cursor only with the filters it was
issued under.

**Keyset pagination on `GET /v0/lineage/nodes`** (experimental, ADR-0206 P2)
— order `node_id ASC` (the primary key, so the seek needs no new index), the
same v1 cursor with `k = [null, node_id]`; a cursor from another listing is a
400 `invalid_cursor`. `total` on the first page only (and on legacy pages),
legacy `{"offset": N}` cursors served with `Deprecation: true` until
`legacy_offset_cursors: false`. Behaviour change: items used to come back in
insertion order and a garbage cursor silently restarted at page one. Cursors
are not bound to the `kind` filter — reuse a cursor only with the filter it
was issued under. With `kind=`, SQLite walks the primary key and filters; a
`(kind, node_id)` index would make that seek O(page) (planned — registry
schema change).

**Governed deletion (`admin` role only — writer/reader/auditor get 403):**

- `DELETE /v0/capsules/{run_id}` — removes the capsule directory plus its
  derived index rows (runs-cache, content-search, and the MetadataStore row
  when the caller carries a tenant) and appends a `capsule_delete` audit entry.
  Re-deleting is a 404. The capsule is first renamed into a sibling
  `.deleting/` tombstone, then the index rows go, then the tombstone is purged;
  if an index step fails the capsule is restored (500 `delete_failed`) so
  files and index never silently disagree, and a purge failure leaves hidden
  residue that is reaped after an hour. Also served by `nova serve`'s
  `DELETE /api/runs/{id}`.
  **If the rollback also fails** the response is 500 `delete_inconsistent`: the
  tombstone then holds the capsule's *only* copy. It is renamed to
  `<data>/.deleting/<run_id>.<epoch>.<hex>.inconsistent` (the exact path is in
  the error text, the `details.tombstone` field and the audit entry) and is
  **never reaped, whatever its age**. Restore it with
  `mv <data>/.deleting/<run_id>.<epoch>.<hex>.inconsistent <data>/capsules/<run_id>`,
  then the next list request re-syncs the runs-cache; or remove the
  directory yourself once you are sure the capsule is meant to be gone. Until
  then a retry of the same delete is refused (409 / `delete_inconsistent_pending`,
  per-item `error` in bulk) instead of returning a bare 404 or creating a second
  tombstone. `nova serve`'s `DELETE /api/runs/{id}` gives the same 409. If even
  the marking rename fails, the reported path is a plain tombstone; the reaper
  still keeps it (whatever its age) while the run has a runs-cache or
  MetadataStore row or an unfinished-delete audit entry (`capsule_delete_failed`
  with no later `capsule_delete`), or if either source cannot be read. Only
  orphans whose index rows are gone are reaped after an hour.
- `POST /v0/capsules/bulk-delete` — body
  `{"run_ids": [...], "dry_run": false}`; per-item outcomes
  (`deleted | held | not_found | invalid_id | duplicate | error`) plus
  summary counts; no rollback — partial progress is reported, not undone.
  Batch size is capped by `bulk.max_items` (top-level `bulk:` block)
  (`NOVAFABRIC_SERVER_BULK_MAX_ITEMS`, default 100, ceiling 1000);
  oversized batches are a 422 before any work. `dry_run: true` returns the
  identical report and deletes nothing.
- **Legal holds always win:** any unreleased hold in any registry's
  `holds.jsonl` — the one hold store `nova hold` and `POST /api/holds` both write
  (`$NOVAFABRIC_HOME/registries/`), plus the pre-unification locations
  `<capsule-dir>/../registries/` and `./.novafabric/registries/` — refuses deletion (409 `legal_hold_active` / per-item
  `held`) and there is **no force override**. Honest limit: holds are
  registry-global today — one active hold freezes the whole delete
  surface. Unexpired WORM locks refuse with 409 `worm_hold`. **NovaSeal-sealed
  capsules (`.seal/` present) refuse with 409 `sealed_capsule`** unless
  `NOVAFABRIC_ALLOW_SEALED_DELETE=1` (experimental; never overrides a hold or
  WORM). If a configured `worm.db` (any `registries/*/worm.db` under
  `$NOVAFABRIC_HOME`, `./.novafabric` or the capsule store's parent) exists but cannot be
  read, deletion refuses with 409 `worm_state_unavailable` instead of assuming the capsule
  is unlocked; single, bulk, dashboard and metadata-index deletion all apply it, with no
  override. Repair the file first ([runbook](https://novafabric.ai/docs/ops/backup-restore/#unreadable-worm-or-retention-policy-files)).
  No `worm.db` anywhere means no local WORM is configured. The `LocalWormAdapter` is
  **dev/test-only** and is not compliance-grade immutability.
  Refusals append `run.index_delete_refused` to the chained audit log;
  removed MetadataStore rows append `run.index_delete`.
  **Since v0.98.0 a corrupt line in `holds.jsonl` fails closed:** an
  unparseable line is treated as an *active, blocking* hold (and logged as a
  warning) rather than skipped, because a truncated or damaged line may encode
  the very hold that must block the delete. Symptom: deletion refuses with
  `legal_hold_active` and `nova hold list` shows an active
  `__corrupt__:<registry>` hold — repair or remove the damaged line before retrying.
- **Order of checks and a runnable probe** (experimental, ADR-0206 P2): hold, then WORM, then
  seal. The same function backs `DELETE /v0/capsules/{id}`, bulk delete and `nova serve`'s
  `DELETE /api/runs/{id}`. Against a throwaway directory (no server needed):

  ```python
  from novafabric.server.capsule_delete import check_deletable, DeleteBlockedError
  # base/run-1/.seal/ exists, NOVAFABRIC_ALLOW_SEALED_DELETE unset
  try:
      check_deletable(base, "run-1")
  except DeleteBlockedError as e:
      print(e.code, "|", e)
  ```

  ```text
  sealed_capsule | Deletion blocked: 'run-1' is NovaSeal-sealed.
  ```

  With `NOVAFABRIC_ALLOW_SEALED_DELETE=1` (exactly `1`; any other value is off) the same call
  returns without raising. **Behaviour change:** before this release a sealed capsule could be
  deleted through these routes; it is now refused unless you opt out. The refusal `details`
  name the env var.
- Audit actions: `capsule_delete` (per deletion, `via: api|bulk`),
  `capsule_delete_refused` (single-delete 409s), `capsule_bulk_delete`
  (one summary per bulk request, dry runs included).
- Bulk **export** is separate and already shipped —
  [ADR-0141](https://novafabric.ai/docs/decisions/)
  (`nova export-blob`).

## 7. Backup and restore

See the dedicated [Backup & Restore Runbook](https://novafabric.ai/docs/ops/backup-restore/) — manual
procedures that work today, the `nova backup` tooling
([ADR-0181](https://novafabric.ai/docs/decisions/),
`experimental`), and the automated Postgres restore
(`nova restore` — pg profile auto-detected, ADR-0217,
[ADR-0211](https://novafabric.ai/docs/decisions/),
`experimental`).

### 7a. Startup schema-skew guard (experimental, ADR-0211)

Before the server touches its database at startup, the lifespan compares the
DB's Alembic stamp against the installed build's migration head (registry
track; both `sqlite` and `postgres` backends):

- schema **behind** the build → the server **refuses to start**
  (`E-SKEW-BEHIND`), naming both revisions and the fix:
  `nova db upgrade --track registry --backend <backend>`;
- stamp **unknown to this build** (migrated by a newer/different NovaFabric)
  → refuses (`E-SKEW-AHEAD`): upgrade the package, do **not** downgrade the
  schema;
- **unstamped** DB (any `init_schema()`-bootstrapped deployment) or an
  **unknown** state → starts with one structured warning — never a refusal
  based on ignorance, never a fake `ok`.

Break-glass: `NOVAFABRIC_ALLOW_SCHEMA_SKEW=1` downgrades refusals to a
structured warning (`event=schema_skew_overridden`) and starts — for
emergency read-mostly access only; unset after the incident. Refusal happens
**before** `init_schema()` and the org bootstrap, so a refused server mutates
nothing. Error/warning text carries backend name and revisions only — never
DSNs or hostnames. `/readyz`'s `migrations` check uses the same comparator
and reports real `ok`/`fail` for Postgres too.

## 8. Enterprise hardening at a glance

First slices shipped `experimental` 2026-07-16; tracked in the
enterprise-readiness plan:

| Feature | ADR | Status |
|---|---|---|
| Secure-by-default local auth (no anonymous admin) | [0184](https://novafabric.ai/docs/decisions/) | experimental |
| Workspaces, organizations, service accounts (`/v0/orgs`, `/v0/workspaces`, `/v0/service-accounts`) | [0178](https://novafabric.ai/docs/decisions/) | experimental |
| Rate limiting + storage quotas (default off; quota enforcement is warn-then-reject) | [0179](https://novafabric.ai/docs/decisions/) | experimental |
| `/metrics`, `/livez`, `/readyz`, `/v0/version` | [0182](https://novafabric.ai/docs/decisions/) | experimental |
| Support bundle (`nova support-bundle`) | [0187](https://novafabric.ai/docs/decisions/) | experimental |
| Backup sets (`nova backup create/verify`, `nova restore` local profile) | [0181](https://novafabric.ai/docs/decisions/) | experimental |
| Automated pg restore (`nova restore`, manifest-driven — [0217](https://novafabric.ai/docs/decisions/)) + startup schema-skew guard ([0211](https://novafabric.ai/docs/decisions/) Part B) | 0217 / 0211 | experimental |

## 9. Webhook subscriptions (experimental)

**Status: experimental** ([ADR-0205](https://novafabric.ai/docs/decisions/),
spec `webhook-registry-v0.md`) — API
shapes may change. Server mode only; the env-configured `NOVA_EVENTS_*` /
`NOVA_ALERTS_*` sinks are unchanged.

API-managed outbound event subscriptions: `/v0/webhooks` CRUD + test ping, a
persisted per-attempt delivery log with explicit redelivery, and HMAC-signed
delivery through the ADR-0137 sink core.

```yaml
# ~/.config/novafabric/server.yaml
webhooks:
  enabled: true                  # default false — off ⇒ no dispatch worker at all
  queue_max: 1000                # bounded dispatch queue (overflow = drop-with-audit)
  max_attempts: 5                # POSTs per delivery chain (1–10); backoff 0s/30s/2m/10m/1h
  timeout_s: 5.0                 # per-POST timeout
  delivery_retention_days: 30    # delivery-log age cap
  delivery_retention_rows: 10000 # delivery-log per-webhook row cap
  allow_insecure_url: false      # permit non-loopback http:// endpoints (audited opt-out)
  allow_internal_targets: false  # v0.98.0 SSRF guard opt-out — see below
```

Env overrides follow `NOVAFABRIC_SERVER_WEBHOOKS_*` (plus the spec's
`NOVAFABRIC_WEBHOOKS_QUEUE_MAX` alias for the queue bound).

Operating notes:

- **RBAC:** mutations (create/update/delete/ping/redeliver) are `admin`-only;
  list/get and the delivery log are `admin` or `auditor`; `reader`/`writer`
  have no access.
- **SSRF guard (v0.98.0, works today).** A webhook URL whose host resolves to a
  private, link-local, reserved, or unspecified address is **rejected at
  create/update time** — the server refuses to be used as a probe of your
  internal network. The host is resolved and checked again before **every
  delivery attempt**, so a name that later resolves to an internal address
  (DNS rebinding) receives nothing: the attempt is recorded as failed, with the
  blocked address in `last_error`, and retried on the normal schedule. A
  rebinding faster than the gap between that check and the connection itself is
  not caught. **Loopback is always allowed**, so local webhooks keep
  working unchanged. A self-hosted deployment that legitimately posts to an
  internal receiver (`http://alertmanager.internal/...`) sets
  `webhooks.allow_internal_targets: true` — a documented, auditable opt-out,
  exactly like `allow_insecure_url`. The rejection message names the blocked
  address and the setting that overrides it. **Config-file only:** unlike the
  other webhook settings there is no `NOVAFABRIC_SERVER_WEBHOOKS_*` env
  override for this one today, so it must be set in `server.yaml`.
- **Signing secret** (`nvwh_…`) is returned **exactly once** by
  `POST /v0/webhooks` and is never retrievable afterwards. At rest it is
  wrapped via the ADR-0185 key-wrapping path when
  `NOVAFABRIC_WEBHOOKS_KEK_PATH` names a local 256-bit KEK file; otherwise it
  is stored **as-is** in the 0600 registry DB (a documented fallback —
  `secret_at_rest: plaintext|wrapped` on every GET shows your posture).
  Secret rotation is planned (P2); until then, delete + recreate.
- **Verify deliveries** with the `X-NovaFabric-Signature: t=<unix>,v1=<hex>`
  header: `v1 = HMAC-SHA256(secret, "{t}." + raw_body)`; reject when
  `|now − t| > 300 s`. A reference verifier ships as
  `novafabric.server.webhooks.verify_delivery_signature`.
- **Delivery contract:** best-effort notification, not evidence transport —
  5 bounded attempts, then the row is terminal `failed` and visible in
  `GET /v0/webhooks/{id}/deliveries`; recover with
  `POST /v0/webhooks/{id}/deliveries/{delivery_id}/redeliver`. A dead endpoint
  never delays or fails ingest (bounded queue, drop-with-audit on overflow).
  Scheduled retries are in-memory (P1): rows left `pending`/`retrying` across
  a restart are not auto-resumed — redeliver them manually.
- **Audit:** every lifecycle change, delivery attempt, and queue overflow
  appends to the hash-chained audit log (and flows to SIEM export as
  `webhook.*` OCSF rows). The secret never appears in any audit entry.