Server Administration Guide (multi-user `nova server`)

This guide is for the administrator of a shared, multi-user NovaFabric deployment — the nova server REST API (/v0), not the single-operator nova serve dashboard. It covers identity, roles, tenancy, and audit administration. For installation and deployment topologies see the Server Deployment Guide; this guide does not repeat it.

Everything below is labelled per the docs honesty rule: works today, experimental, planned, or future design.

The two servers, in one line: nova server is the team/API surface (OIDC, RBAC, /v0); nova serve is the localhost dashboard (single shared token). See docs/api-reference.md for the full disambiguation table.

1. Enabling server mode

Status: experimental (v0.7+, ADR-0017, ADR-0029).

pip install 'novafabric[server]'
nova server start                       # 127.0.0.1:7433 by default
nova server start --host 0.0.0.0 --port 8080

Configuration lives in ~/.config/novafabric/server.yaml; only --config <path> points the server at another file (for example /etc/novafabric/server.yaml). NOVAFABRIC_SERVER_* environment variables override individual settings, and are held to the same bounds as the YAML keys: an out-of-range value (NOVAFABRIC_SERVER_PORT=99999, NOVAFABRIC_SERVER_INGEST_MAX_CONCURRENT_UNPACK=0) makes nova server start exit 2 with Invalid server config, naming the field. Secrets are environment-only and stripped from YAML: NOVAFABRIC_POSTGRES_DSN, NOVAFABRIC_OFFLINE_KEY_PATH, NOVAFABRIC_SCIM_TOKEN and NOVAFABRIC_SERVER_TOKEN (server/config.py::load_config). See Server Deployment Guide — Scenario 2 for the Postgres backend and migration from SQLite.

Local auth (ADR-0184, experimental — shipped 2026-07-16). With OIDC disabled, the server requires an auto-generated local bearer token (printed at startup, stored at $NOVAFABRIC_HOME/.server-token mode 0600, pinnable via NOVAFABRIC_SERVER_TOKEN). The old anonymous-admin behavior needs the explicit --insecure-no-auth opt-out and refuses non-loopback binds without --i-know-this-is-public (ADR-0184). Each insecure start logs a warning and appends a server.insecure_no_auth entry (bind host, port, loopback, public-bind confirmation) to the hash-chained audit log at NOVAFABRIC_AUDIT_LOG_PATH (default $XDG_DATA_HOME/novafabric/audit.jsonl, else ~/.local/share/novafabric/audit.jsonl; NOVAFABRIC_HOME does not move it — set the variable explicitly to put the log on a data volume). If the entry cannot be written the server refuses to start in insecure mode.

2. Identity backends

Backend	Status	Notes
OIDC (JWT Bearer + JWKS)	experimental	The primary team backend. Setup: Server Deployment Guide — Scenario 3. JWKS cache flush: `nova server flush-jwks-cache`. ADR-0018
Offline ed25519 tokens	experimental	Air-gapped/CI/SLURM machine identity without an IdP. `nova server issue-token --subject worker-01 --roles reader,writer --expires-in 30d`; revoke with `nova server revoke-token <jti>`. Revocations are recorded in the `token_audit` table.
SCIM 2.0 provisioning	experimental	Off by default (endpoints 404). Enable with `NOVAFABRIC_SERVER_SCIM_ENABLED=1` and `NOVAFABRIC_SCIM_TOKEN`. `/scim/v2/*` per RFC 7644; all provisioning actions land in the append-only `scim_audit_events` table. ADR-0139
SAML 2.0 SSO	experimental, opt-in	SP metadata via `nova server saml-metadata`. `POST /v0/auth/saml/acs` consumes assertions only when `saml.experimental_acs_enabled: true` is set in `server.yaml` and `novafabric[saml]` (signxml + lxml, cleared the license gate in v0.73.0) is installed; otherwise it returns 501. Even then a verified assertion is answered 501 `saml_session_unavailable`: the server cannot yet issue a session for a SAML login. NovaFabric never skips assertion signature verification. A Security-Architect review remains a pre-production blocking condition (ADR-0138 §D5).
Device-grant demo flow	off by default (endpoints 404)	The RFC 8628 `/v0/auth/device/code\|token\|approve` flow is local/testing scaffolding whose HS256 tokens the real verifier never honours; `/approve` is unauthenticated. It stays unmounted unless you set `NOVAFABRIC_SERVER_DEMO_DEVICE_GRANT=1`. Never enable it in production — use OIDC or offline tokens instead. ADR-0198

3. Roles and authorization

Status: experimental.

ADR-0018 defines six built-in roles. Four are enforced as route-level checks:

Role	Grants
`reader`	Read access to runs, capsules, lineage
`writer`	`reader` + create/ingest
`admin`	Everything, including role management
`auditor`	Orthogonal — audit trails and evidence only; cannot write

The hierarchy is reader < writer < admin; admin satisfies every check; auditor satisfies only auditor checks. The remaining two roles — promoter and approver — implement separation of duties in the maker-checker promotion and NovaSeal approval flows (ADR-0058); the same identity can never serve as both maker and checker for one proposal.

Assigning roles (either surface writes the same role_assignments table):

nova server assign-role alice@example.com writer
# or over HTTP (admin-gated):
#   GET/POST /v0/admin/roles, DELETE /v0/admin/roles/{subject}/{role}

⚠ A role_assignments row is consulted only once the workspace/org model is in use (a scoped membership or service account exists). In a stock deployment it grants nothing: assign-role then warns this assignment is not in force on stderr and GET /v0/admin/roles reports effective_now: false. Carry the role in the token instead (OIDC nova_roles/roles claim, or issue-token --roles).

Experimental: org/workspace-scoped role bindings and named service accounts (/v0/orgs, /v0/workspaces, /v0/service-accounts) — ADR-0178. Creating the first membership or service account is what switches role_assignments rows on.

What a binding grants, when a credential's own roles fall short:

A global role_assignments row grants its role everywhere.
A workspace or org membership grants its role in full only on that workspace's own routes (/v0/workspaces/{ws_id}/…): a workspace admin can manage that workspace's memberships, not another's. On every other route a membership grants at most writer — enough for the data plane (the capsule store is not partitioned by workspace), never server-wide admin: a workspace admin cannot mint API keys, create orgs or assign global roles.
A credential that carries its own role set — an API key, or an offline token issued with --roles — gets exactly those roles and never its owner's bindings, so a least-privilege key stays least-privilege. A role-less service-account token takes its roles from its bindings.
SCIM: a user who is active: false gets no role from any mapped group, even when the IdP adds them to one; reactivation restores the roles of their groups.

4. Tenancy and isolation

Status: experimental (Postgres backend only).

With the Postgres metadata store, tenant isolation is enforced in the database, not in application code: FORCE ROW LEVEL SECURITY with a tenant_isolation policy on runs, capsules, signatures, and retention_policies, a per-transaction SET LOCAL app.current_tenant_id (safe under pgBouncer transaction pooling), and a split between the novafabric_app role (no BYPASSRLS) and novafabric_migrator (ADR-0040, ADR-0052). The policy holds only when the server connects as novafabric_app (or another role without BYPASSRLS). A Postgres superuser bypasses row-level security even under FORCE, so never point NOVAFABRIC_METADATA_DSN at a superuser. The app role is granted SELECT, INSERT, UPDATE on the tenant tables. Capsule deletion (DELETE /v0/capsules/{run_id}, POST /v0/capsules/bulk-delete) also needs DELETE on runs, capsules and signatures, which migration v005 grants — run nova db upgrade --backend postgres after upgrading. A CI gate (metadata_store_security_gate) re-proves cross-tenant isolation on every change.

Tenancy today is a single flat tenant_id per deployment-defined scope. There is no organization/workspace/team hierarchy yet — that is ADR-0178 (future design), which keeps tenant_id as the sole RLS key.

5. Audit trails

Status: works today (local + server).

Hash-chained append-only audit log; domain trails for SCIM (scim_audit_events), tokens (token_audit), SAML (redacted records), and retention decisions ("deletion is evidence", ADR-0134).
nova audit … maps capsule evidence to regulatory controls; nova ledger anchor|verify|status runs the adversary-anchored accountability ledger (ADR-0094); nova seal log … operates the Merkle log.
Give compliance staff the auditor role — read-only by construction.

6. API behavior an admin should know

Versioning: all resource routes sit under /v0; the canonical contract is api/openapi.yaml. A formal deprecation/sunset policy is proposed in ADR-0188 (future design).
Ids in requests are names, never paths (works today): a run_id in a request (POST /v0/evidence, POST /v0/replays, GET|DELETE /v0/capsules/{run_id}) and a run_id/parent_run_id in an uploaded capsule.yaml must be one path segment: no /, \, .., NUL or other control character, not absolute, at most 255 bytes, else **400**. An entry of the capsule store that is a symbolic link is refused with **400** by the read, evidence and replay routes, not followed. POST /v0/evidence writes the bundle only to <evidence dir>/<run_id>.zip (NOVAFABRIC_EVIDENCE_DIR, default $NOVAFABRIC_HOME/evidence), written to a new file and renamed into place, and returns that path as bundle_path. A request that names an output_path is refused with **400**: before v0.107.0 a writer used it to overwrite any file the server could write (review server-01). The signing key the server creates on first export ($NOVAFABRIC_HOME/keys/signing_key.pem) is written mode 0600.
Pagination: cursor-based, default 50 / max 500 per page (works today). GET /v0/capsules and GET /v0/assets now serve keyset cursors — see 6a (experimental).
Rate limiting/quotas: experimental, default off — in-process rate limiting + storage quotas (ADR-0179), plus per-workspace usage metering, GET /v0/usage reporting, per-workspace budgets, and the nova server usage reconcile|export drift-reconciliation / chargeback-export CLI (ADR-0208). See Quotas & rate limits; with the master switch off (the default), plan capacity accordingly.
Health and self-observability (experimental, ADR-0182): GET /health and GET /livez (unauthenticated liveness), GET /readyz (database, migration-head and object-store checks), GET /v0/version (reader role) and Prometheus GET /metrics (reader role unless observability.metrics_exempt: true; needs prometheus-client from the [server] extra). All four are exempt from rate limiting. For deeper diagnostics use nova doctor (add --check-storage for schema/migration state).

6a. Bulk capsule operations + keyset pagination (experimental, ADR-0206)

Status: experimental — shipped by ADR-0206. The normative contract is specified in the maintainers' private design/ tree and is not published.

Keyset pagination on GET /v0/capsules:

List order is pinned to created_at DESC, run_id DESC; next_cursor is an opaque v1 keyset cursor — pass it back verbatim. Page requests are O(page) against a derived runs-cache index (lazily backfilled from the capsule directory, which stays the source of truth) instead of re-parsing every capsule.yaml.
total appears on first-page (and legacy-cursor) responses only; cursor pages omit it by design.
A corrupted/tampered cursor is now a 400 invalid_cursor — it no longer silently restarts from the first page.
Legacy {"offset": N} cursors still work for one deprecation cycle (ADR-0188); those responses carry Deprecation: true. Set pagination.legacy_offset_cursors: false (a top-level pagination: block in server.yaml) (NOVAFABRIC_SERVER_PAGINATION_LEGACY_OFFSET_CURSORS=false) to refuse them early.

Keyset pagination on GET /v0/assets (experimental, ADR-0206 P2) — same cursor format and rules as above: order created_at DESC, id DESC, seek in the registry SQL (spec_json is never read for a list page), total on the first page only, 400 invalid_cursor on garbage (previously a silent restart at page one), legacy {"offset": N} cursors served with Deprecation: true until legacy_offset_cursors: false. Cursors are not bound to the asset_type/status filters — reuse a cursor only with the filters it was issued under.

Keyset pagination on GET /v0/lineage/nodes (experimental, ADR-0206 P2) — order node_id ASC (the primary key, so the seek needs no new index), the same v1 cursor with k = [null, node_id]; a cursor from another listing is a 400 invalid_cursor. total on the first page only (and on legacy pages), legacy {"offset": N} cursors served with Deprecation: true until legacy_offset_cursors: false. Behaviour change: items used to come back in insertion order and a garbage cursor silently restarted at page one. Cursors are not bound to the kind filter — reuse a cursor only with the filter it was issued under. With kind=, SQLite walks the primary key and filters; a (kind, node_id) index would make that seek O(page) (planned — registry schema change).

Governed deletion (admin role only — writer/reader/auditor get 403):

DELETE /v0/capsules/{run_id} — removes the capsule directory plus its derived index rows (runs-cache, content-search, and the MetadataStore row when the caller carries a tenant) and appends a capsule_delete audit entry. Re-deleting is a 404. The capsule is first renamed into a sibling .deleting/ tombstone, then the index rows go, then the tombstone is purged; if an index step fails the capsule is restored (500 delete_failed) so files and index never silently disagree, and a purge failure leaves hidden residue that is reaped after an hour. Also served by nova serve's DELETE /api/runs/{id}. If the rollback also fails the response is 500 delete_inconsistent: the tombstone then holds the capsule's only copy. It is renamed to <data>/.deleting/<run_id>.<epoch>.<hex>.inconsistent (the exact path is in the error text, the details.tombstone field and the audit entry) and is never reaped, whatever its age. Restore it with mv <data>/.deleting/<run_id>.<epoch>.<hex>.inconsistent <data>/capsules/<run_id>, then the next list request re-syncs the runs-cache; or remove the directory yourself once you are sure the capsule is meant to be gone. Until then a retry of the same delete is refused (409 / delete_inconsistent_pending, per-item error in bulk) instead of returning a bare 404 or creating a second tombstone. nova serve's DELETE /api/runs/{id} gives the same 409. If even the marking rename fails, the reported path is a plain tombstone; the reaper still keeps it (whatever its age) while the run has a runs-cache or MetadataStore row or an unfinished-delete audit entry (capsule_delete_failed with no later capsule_delete), or if either source cannot be read. Only orphans whose index rows are gone are reaped after an hour.
POST /v0/capsules/bulk-delete — body {"run_ids": [...], "dry_run": false}; per-item outcomes (deleted | held | not_found | invalid_id | duplicate | error) plus summary counts; no rollback — partial progress is reported, not undone. Batch size is capped by bulk.max_items (top-level bulk: block) (NOVAFABRIC_SERVER_BULK_MAX_ITEMS, default 100, ceiling 1000); oversized batches are a 422 before any work. dry_run: true returns the identical report and deletes nothing.
Legal holds always win: any unreleased hold in any registry's holds.jsonl — the one hold store nova hold and POST /api/holds both write ($NOVAFABRIC_HOME/registries/), plus the pre-unification locations <capsule-dir>/../registries/ and ./.novafabric/registries/ — refuses deletion (409 legal_hold_active / per-item held) and there is no force override. Honest limit: holds are registry-global today — one active hold freezes the whole delete surface. Unexpired WORM locks refuse with 409 worm_hold. NovaSeal-sealed capsules (.seal/ present) refuse with 409 sealed_capsule unless NOVAFABRIC_ALLOW_SEALED_DELETE=1 (experimental; never overrides a hold or WORM). If a configured worm.db (any registries/*/worm.db under $NOVAFABRIC_HOME, ./.novafabric or the capsule store's parent) exists but cannot be read, deletion refuses with 409 worm_state_unavailable instead of assuming the capsule is unlocked; single, bulk, dashboard and metadata-index deletion all apply it, with no override. Repair the file first (runbook). No worm.db anywhere means no local WORM is configured. The LocalWormAdapter is dev/test-only and is not compliance-grade immutability. Refusals append run.index_delete_refused to the chained audit log; removed MetadataStore rows append run.index_delete. Since v0.98.0 a corrupt line in holds.jsonl fails closed: an unparseable line is treated as an active, blocking hold (and logged as a warning) rather than skipped, because a truncated or damaged line may encode the very hold that must block the delete. Symptom: deletion refuses with legal_hold_active and nova hold list shows an active __corrupt__:<registry> hold — repair or remove the damaged line before retrying.
Order of checks and a runnable probe (experimental, ADR-0206 P2): hold, then WORM, then seal. The same function backs DELETE /v0/capsules/{id}, bulk delete and nova serve's DELETE /api/runs/{id}. Against a throwaway directory (no server needed):
```
from novafabric.server.capsule_delete import check_deletable, DeleteBlockedError
# base/run-1/.seal/ exists, NOVAFABRIC_ALLOW_SEALED_DELETE unset
try:
    check_deletable(base, "run-1")
except DeleteBlockedError as e:
    print(e.code, "|", e)
```
```
sealed_capsule | Deletion blocked: 'run-1' is NovaSeal-sealed.
```
With NOVAFABRIC_ALLOW_SEALED_DELETE=1 (exactly 1; any other value is off) the same call returns without raising. Behaviour change: before this release a sealed capsule could be deleted through these routes; it is now refused unless you opt out. The refusal details name the env var.
Audit actions: capsule_delete (per deletion, via: api|bulk), capsule_delete_refused (single-delete 409s), capsule_bulk_delete (one summary per bulk request, dry runs included).
Bulk export is separate and already shipped — ADR-0141 (nova export-blob).

7. Backup and restore

See the dedicated Backup & Restore Runbook — manual procedures that work today, the nova backup tooling (ADR-0181, experimental), and the automated Postgres restore (nova restore — pg profile auto-detected, ADR-0217, ADR-0211, experimental).

7a. Startup schema-skew guard (experimental, ADR-0211)

Before the server touches its database at startup, the lifespan compares the DB's Alembic stamp against the installed build's migration head (registry track; both sqlite and postgres backends):

schema behind the build → the server refuses to start (E-SKEW-BEHIND), naming both revisions and the fix: nova db upgrade --track registry --backend <backend>;
stamp unknown to this build (migrated by a newer/different NovaFabric) → refuses (E-SKEW-AHEAD): upgrade the package, do not downgrade the schema;
unstamped DB (any init_schema()-bootstrapped deployment) or an unknown state → starts with one structured warning — never a refusal based on ignorance, never a fake ok.

Break-glass: NOVAFABRIC_ALLOW_SCHEMA_SKEW=1 downgrades refusals to a structured warning (event=schema_skew_overridden) and starts — for emergency read-mostly access only; unset after the incident. Refusal happens before init_schema() and the org bootstrap, so a refused server mutates nothing. Error/warning text carries backend name and revisions only — never DSNs or hostnames. /readyz's migrations check uses the same comparator and reports real ok/fail for Postgres too.

8. Enterprise hardening at a glance

First slices shipped experimental 2026-07-16; tracked in the enterprise-readiness plan:

Feature	ADR	Status
Secure-by-default local auth (no anonymous admin)	0184	experimental
Workspaces, organizations, service accounts (`/v0/orgs`, `/v0/workspaces`, `/v0/service-accounts`)	0178	experimental
Rate limiting + storage quotas (default off; quota enforcement is warn-then-reject)	0179	experimental
`/metrics`, `/livez`, `/readyz`, `/v0/version`	0182	experimental
Support bundle (`nova support-bundle`)	0187	experimental
Backup sets (`nova backup create/verify`, `nova restore` local profile)	0181	experimental
Automated pg restore (`nova restore`, manifest-driven — 0217) + startup schema-skew guard (0211 Part B)	0217 / 0211	experimental

9. Webhook subscriptions (experimental)

Status: experimental (ADR-0205, spec webhook-registry-v0.md) — API shapes may change. Server mode only; the env-configured NOVA_EVENTS_* / NOVA_ALERTS_* sinks are unchanged.

API-managed outbound event subscriptions: /v0/webhooks CRUD + test ping, a persisted per-attempt delivery log with explicit redelivery, and HMAC-signed delivery through the ADR-0137 sink core.

# ~/.config/novafabric/server.yaml
webhooks:
  enabled: true                  # default false — off ⇒ no dispatch worker at all
  queue_max: 1000                # bounded dispatch queue (overflow = drop-with-audit)
  max_attempts: 5                # POSTs per delivery chain (1–10); backoff 0s/30s/2m/10m/1h
  timeout_s: 5.0                 # per-POST timeout
  delivery_retention_days: 30    # delivery-log age cap
  delivery_retention_rows: 10000 # delivery-log per-webhook row cap
  allow_insecure_url: false      # permit non-loopback http:// endpoints (audited opt-out)
  allow_internal_targets: false  # v0.98.0 SSRF guard opt-out — see below

Env overrides follow NOVAFABRIC_SERVER_WEBHOOKS_* (plus the spec's NOVAFABRIC_WEBHOOKS_QUEUE_MAX alias for the queue bound).

Operating notes:

RBAC: mutations (create/update/delete/ping/redeliver) are admin-only; list/get and the delivery log are admin or auditor; reader/writer have no access.
SSRF guard (v0.98.0, works today). A webhook URL whose host resolves to a private, link-local, reserved, or unspecified address is rejected at create/update time — the server refuses to be used as a probe of your internal network. The host is resolved and checked again before every delivery attempt, so a name that later resolves to an internal address (DNS rebinding) receives nothing: the attempt is recorded as failed, with the blocked address in last_error, and retried on the normal schedule. A rebinding faster than the gap between that check and the connection itself is not caught. Loopback is always allowed, so local webhooks keep working unchanged. A self-hosted deployment that legitimately posts to an internal receiver (http://alertmanager.internal/...) sets webhooks.allow_internal_targets: true — a documented, auditable opt-out, exactly like allow_insecure_url. The rejection message names the blocked address and the setting that overrides it. Config-file only: unlike the other webhook settings there is no NOVAFABRIC_SERVER_WEBHOOKS_* env override for this one today, so it must be set in server.yaml.
Signing secret (nvwh_…) is returned exactly once by POST /v0/webhooks and is never retrievable afterwards. At rest it is wrapped via the ADR-0185 key-wrapping path when NOVAFABRIC_WEBHOOKS_KEK_PATH names a local 256-bit KEK file; otherwise it is stored as-is in the 0600 registry DB (a documented fallback — secret_at_rest: plaintext|wrapped on every GET shows your posture). Secret rotation is planned (P2); until then, delete + recreate.
Verify deliveries with the X-NovaFabric-Signature: t=<unix>,v1=<hex> header: v1 = HMAC-SHA256(secret, "{t}." + raw_body); reject when |now − t| > 300 s. A reference verifier ships as novafabric.server.webhooks.verify_delivery_signature.
Delivery contract: best-effort notification, not evidence transport — 5 bounded attempts, then the row is terminal failed and visible in GET /v0/webhooks/{id}/deliveries; recover with POST /v0/webhooks/{id}/deliveries/{delivery_id}/redeliver. A dead endpoint never delays or fails ingest (bounded queue, drop-with-audit on overflow). Scheduled retries are in-memory (P1): rows left pending/retrying across a restart are not auto-resumed — redeliver them manually.
Audit: every lifecycle change, delivery attempt, and queue overflow appends to the hash-chained audit log (and flows to SIEM export as webhook.* OCSF rows). The secret never appears in any audit entry.

Server Administration Guide (multi-user nova server)