Server Administration Guide (multi-user nova server)

This guide is for the administrator of a shared, multi-user NovaFabric deployment — the nova server REST API (/v0), not the single-operator nova serve dashboard. It covers identity, roles, tenancy, and audit administration. For installation and deployment topologies see the Server Deployment Guide; this guide does not repeat it.

Everything below is labelled per the docs honesty rule: works today, experimental, planned, or future design.

The two servers, in one line: nova server is the team/API surface (OIDC, RBAC, /v0); nova serve is the localhost dashboard (single shared token). See docs/api-reference.md for the full disambiguation table.

1. Enabling server mode

Status: experimental (v0.7+, ADR-0017, ADR-0029).

pip install 'novafabric[server]'
nova server start                       # 127.0.0.1:7433 by default
nova server start --host 0.0.0.0 --port 8080

Configuration lives in nova-server.yaml ($NOVA_SERVER_CONFIG); secrets are environment-only and stripped from YAML (NOVA_DSN, NOVAFABRIC_SCIM_TOKEN, …). See Server Deployment Guide — Scenario 2 for the Postgres backend and migration from SQLite.

Local auth (ADR-0184, experimental — shipped 2026-07-16). With OIDC disabled, the server requires an auto-generated local bearer token (printed at startup, stored at ~/.novafabric/.server-token mode 0600, pinnable via NOVAFABRIC_SERVER_TOKEN). The old anonymous-admin behavior needs the explicit --insecure-no-auth opt-out and refuses non-loopback binds without --i-know-this-is-public (ADR-0184).

2. Identity backends

Backend Status Notes
OIDC (JWT Bearer + JWKS) experimental The primary team backend. Setup: Server Deployment Guide — Scenario 3. JWKS cache flush: nova server flush-jwks-cache. ADR-0018
Offline ed25519 tokens experimental Air-gapped/CI/SLURM machine identity without an IdP. nova server issue-token --subject worker-01 --roles reader,writer --expires-in 30d; revoke with nova server revoke-token <jti>. Revocations are recorded in the token_audit table.
SCIM 2.0 provisioning experimental Off by default (endpoints 404). Enable with NOVAFABRIC_SERVER_SCIM_ENABLED=1 and NOVAFABRIC_SCIM_TOKEN. /scim/v2/* per RFC 7644; all provisioning actions land in the append-only scim_audit_events table. ADR-0139
SAML 2.0 SSO experimental, license-gated SP metadata via nova server saml-metadata. The ACS endpoint deliberately returns 501 until an XML-DSIG verification library clears the dependency-license gate (ADR-0138 §D5) — NovaFabric never skips assertion signature verification.
Device-grant demo flow off by default (endpoints 404) The RFC 8628 /v0/auth/device/code|token|approve flow is local/testing scaffolding whose HS256 tokens the real verifier never honours; /approve is unauthenticated. It stays unmounted unless you set NOVAFABRIC_SERVER_DEMO_DEVICE_GRANT=1. Never enable it in production — use OIDC or offline tokens instead. ADR-0198

3. Roles and authorization

Status: experimental.

ADR-0018 defines six built-in roles. Four are enforced as route-level checks:

Role Grants
reader Read access to runs, capsules, lineage
writer reader + create/ingest
admin Everything, including role management
auditor Orthogonal — audit trails and evidence only; cannot write

The hierarchy is reader < writer < admin; admin satisfies every check; auditor satisfies only auditor checks. The remaining two roles — promoter and approver — implement separation of duties in the maker-checker promotion and NovaSeal approval flows (ADR-0058); the same identity can never serve as both maker and checker for one proposal.

Assigning roles (either surface writes the same role_assignments table):

nova server assign-role --subject alice@example.com --role writer
# or over HTTP (admin-gated):
#   GET/POST /v0/admin/roles, DELETE /v0/admin/roles/{subject}/{role}

Planned (future design): org/workspace-scoped role bindings and named service accounts — ADR-0178.

4. Tenancy and isolation

Status: experimental (Postgres backend only).

With the Postgres metadata store, tenant isolation is enforced in the database, not in application code: FORCE ROW LEVEL SECURITY with a tenant_isolation policy on runs, capsules, signatures, and retention_policies, a per-transaction SET LOCAL app.current_tenant_id (safe under pgBouncer transaction pooling), and a split between the novafabric_app role (no BYPASSRLS) and novafabric_migrator (ADR-0040, ADR-0052). A CI gate (metadata_store_security_gate) re-proves cross-tenant isolation on every change.

Tenancy today is a single flat tenant_id per deployment-defined scope. There is no organization/workspace/team hierarchy yet — that is ADR-0178 (future design), which keeps tenant_id as the sole RLS key.

5. Audit trails

Status: works today (local + server).

6. API behavior an admin should know

6a. Bulk capsule operations + keyset pagination (experimental, ADR-0206)

Status: experimental — shipped by ADR-0206; normative contract in design/spec/bulk-ops-pagination-v0.md.

Keyset pagination on GET /v0/capsules:

Governed deletion (admin role only — writer/reader/auditor get 403):

7. Backup and restore

See the dedicated Backup & Restore Runbook — manual procedures that work today, the nova backup tooling (ADR-0181, experimental), and the automated Postgres restore (nova restore — pg profile auto-detected, ADR-0217, ADR-0211, experimental).

7a. Startup schema-skew guard (experimental, ADR-0211)

Before the server touches its database at startup, the lifespan compares the DB's Alembic stamp against the installed build's migration head (registry track; both sqlite and postgres backends):

Break-glass: NOVAFABRIC_ALLOW_SCHEMA_SKEW=1 downgrades refusals to a structured warning (event=schema_skew_overridden) and starts — for emergency read-mostly access only; unset after the incident. Refusal happens before init_schema() and the org bootstrap, so a refused server mutates nothing. Error/warning text carries backend name and revisions only — never DSNs or hostnames. /readyz's migrations check uses the same comparator and reports real ok/fail for Postgres too.

8. Enterprise hardening at a glance

First slices shipped experimental 2026-07-16; tracked in the enterprise-readiness plan:

Feature ADR Status
Secure-by-default local auth (no anonymous admin) 0184 experimental
Workspaces, organizations, service accounts (/v0/orgs, /v0/workspaces, /v0/service-accounts) 0178 experimental
Rate limiting + storage quotas (default off; quota enforcement is warn-then-reject) 0179 experimental
/metrics, /livez, /readyz, /v0/version 0182 experimental
Support bundle (nova support-bundle) 0187 experimental
Backup sets (nova backup create/verify, nova restore local profile) 0181 experimental
Automated pg restore (nova restore, manifest-driven — 0217) + startup schema-skew guard (0211 Part B) 0217 / 0211 experimental
Backup sets (nova backup create/verify, local profile; restore planned) 0181 experimental

9. Webhook subscriptions (experimental)

Status: experimental (ADR-0205, spec webhook-registry-v0.md) — API shapes may change. Server mode only; the env-configured NOVA_EVENTS_* / NOVA_ALERTS_* sinks are unchanged.

API-managed outbound event subscriptions: /v0/webhooks CRUD + test ping, a persisted per-attempt delivery log with explicit redelivery, and HMAC-signed delivery through the ADR-0137 sink core.

# ~/.config/novafabric/server.yaml
webhooks:
  enabled: true                  # default false — off ⇒ no dispatch worker at all
  queue_max: 1000                # bounded dispatch queue (overflow = drop-with-audit)
  max_attempts: 5                # POSTs per delivery chain (1–10); backoff 0s/30s/2m/10m/1h
  timeout_s: 5.0                 # per-POST timeout
  delivery_retention_days: 30    # delivery-log age cap
  delivery_retention_rows: 10000 # delivery-log per-webhook row cap
  allow_insecure_url: false      # permit non-loopback http:// endpoints (audited opt-out)
  allow_internal_targets: false  # v0.98.0 SSRF guard opt-out — see below

Env overrides follow NOVAFABRIC_SERVER_WEBHOOKS_* (plus the spec's NOVAFABRIC_WEBHOOKS_QUEUE_MAX alias for the queue bound).

Operating notes: