Server Deployment Guide

This guide covers six deployment scenarios. Each scenario is labelled with its current implementation status per the docs honesty rule: works today, experimental, planned, or future design.

Sizing the host first? The capacity planning & sizing guide covers storage, memory and CPU from a single laptop to a shared Postgres server.

Scenario 1 — Default local install (SQLite, no server)

Status: experimental.

The default NovaFabric install requires no server, no database setup, and no network connectivity. All core commands work out of the box:

pip install novafabric

nova capture python agent.py
nova validate .novafabric/runs/01HX.../
nova replay .novafabric/runs/01HX.../ --mode forensic
nova lineage provenance <run-id>
nova export-evidence .novafabric/runs/01HX.../ --key key.pem --output evidence.zip

The local SQLite registry is created automatically at ~/.novafabric/registry.db on first use.

Changed in v0.99.0. The default install is 113 MB / 42 packages instead of 412 MB / 50 — duckdb, pyarrow, python-louvain and clickhouse-connect moved to extras (ADR-0222). Every command above is unaffected. Server and container deployments are unaffected too: [server]/[serve] and the shipped Dockerfile pull in what they need. If you relied on importing those packages after a plain install, use pip install 'novafabric[all]'.

Optional local dashboard

The experimental dashboard is a separate optional extra:

pip install 'novafabric[serve]'
nova serve --experimental        # opens http://127.0.0.1:4321

Status: experimental. See ADR-0027 and dashboard.md for limitations.

Scenario 2 — Postgres opt-in

Status: experimental.

Operators running a shared team deployment can use Postgres as the storage backend. SQLite remains the default; Postgres is opt-in.

Install the server extra

pip install 'novafabric[server]'

Configure the server

Create ~/.config/novafabric/server.yaml, or pass --config <path> to nova server start. There is no other search path: when neither exists, the built-in defaults plus the NOVAFABRIC_SERVER_* environment overrides apply.

# ~/.config/novafabric/server.yaml — host and port are TOP-LEVEL keys.
# (A nested `server:` block is silently ignored.)
backend: postgres
host: "0.0.0.0"
port: 7433

Provide the DSN as an environment variable. Secrets are never read from the YAML file — postgres_dsn, offline_key_path, scim_token and local_token are dropped from it on load:

export NOVAFABRIC_POSTGRES_DSN="postgresql://nova_user:s3cr3t@db.example.com:5432/novafabric"

See ADR-0029 for the config design and the config file reference below for every key.

Start the server

nova server start --backend postgres
# or rely on the config file:
nova server start

The startup banner prints the resolved backend and port:

Starting NovaFabric server on 0.0.0.0:7433 [backend=postgres, workers=1]
API docs: http://0.0.0.0:7433/docs
Press Ctrl+C to stop.

Migrate existing SQLite data

To carry existing local capsules into the new Postgres backend:

# Dry run first — see what would be migrated
nova migrate-to-postgres --dry-run

# Full migration
nova migrate-to-postgres \
  --source ~/.novafabric/registry.db \
  --target "$NOVAFABRIC_POSTGRES_DSN" \
  --log migration.jsonl

# Verify the result
nova doctor --check-storage --backend postgres

The migration is idempotent — safe to re-run after a partial failure. It uses upsert semantics so no rows are duplicated. The SQLite file is never modified or deleted. See ADR-0016 for the migration design.

Exit codes: 0 = success, 1 = row-count mismatch, 2 = connection error.

Storage health check

nova doctor --check-storage
nova doctor --check-storage --backend postgres --postgres-dsn "$NOVAFABRIC_POSTGRES_DSN"

Output includes backend name, schema version (Alembic), migration status, and per-table row counts.

Scenario 3 — OIDC setup

Status: experimental. The OIDC integration is implemented and tested against mock OIDC servers. It has not been validated in production against Keycloak, Auth0, or Okta. The config keys are stable (ADR-0029) but the behaviour under edge cases (clock skew, key rotation races) may change.

Config

Add the oidc block to your server.yaml. OIDC has exactly two keys and is on when both are set — there is no enabled flag:

backend: postgres
host: "0.0.0.0"
port: 7433
oidc:
  issuer_url: "https://keycloak.example.com/realms/nova"
  audience: "nova-server"

• issuer_url is the IdP's issuer, written as it appears in the tokens' iss claim (Auth0's ends in /); a configured trailing slash also matches an iss without one. The server reads the key-set location (jwks_uri) from <issuer_url>/.well-known/openid-configuration, which Keycloak, Okta and Auth0 all publish, and falls back to <issuer_url>/.well-known/jwks.json when there is no discovery document.
• Roles are read from the nova_roles claim (falling back to roles); the claim name is not configurable.
• The JWKS is cached for 3600 s; the interval is not configurable. A token whose kid is not in the cache triggers a re-fetch, but the server fetches at most once every 30 s (failed fetches included), so a key the IdP has just rotated in is picked up within 30 s; flush the cache (below) to pick it up at once.
• OIDC is configured in YAML only — there are no OIDC environment variables.

Login from the CLI

Not usable against a production server today. nova login speaks only to the server's own demo device-grant endpoints (/v0/auth/device/code), which are mounted only when NOVAFABRIC_SERVER_DEMO_DEVICE_GRANT=1 (off by default). That demo flow is local-testing scaffolding: the HS256 tokens it mints are never accepted by the real verifier, and it does not contact your IdP.

For a real OIDC deployment, obtain an access token from your identity provider and pass it as Authorization: Bearer <token>; for automation use an API key or an offline token.

nova login --server http://nova.example.com:7433   # demo device grant only

Against a server without the demo grant, nova login exits 1 with that explanation (not a bare 404); after a demo login it warns on stderr that the stored token is not accepted by the server's API.

Credentials are stored in ~/.config/novafabric/credentials.json at mode 0600.

Log out

nova logout --server http://nova.example.com:7433   # remove one server
nova logout                                           # remove all servers

JWKS cache management

The server caches the OIDC provider's JWKS for 3600 seconds (fixed), and re-fetches it for an unknown kid at most once every 30 seconds, so a random kid cannot make the server call the IdP on every request. To pick up a rotated key immediately, flush the cache without restarting the server:

nova server flush-jwks-cache \
  --server http://nova.example.com:7433 \
  --token "$ADMIN_TOKEN"

Or use the stored credential (if logged in as admin):

nova server flush-jwks-cache --server http://nova.example.com:7433

Scenario 4 — Airgapped / offline-token mode

Status: experimental. Offline tokens require no OIDC provider. Use this mode for SLURM batch jobs, air-gapped clusters, or any environment where reaching an external identity provider is not possible. They also work beside OIDC: with both configured, a token whose iss is nova-offline (every offline and service-account token) is verified against the offline key, and every other token against the IdP.

How it works

nova server issue-token generates an ed25519 keypair on first use when --key-path (default ~/.novafabric/keys/offline-key.pem) does not exist yet, writing the private key and a .pub public key beside it. The private key signs JWTs locally; the server verifies them with the public key, without contacting any external endpoint.

The server accepts offline tokens only when it is started with NOVAFABRIC_OFFLINE_KEY_PATH pointing at the public key (environment only — the YAML key is ignored). Without it, every offline JWT is rejected with 401; issue-token prints this reminder on stderr.

export NOVAFABRIC_OFFLINE_KEY_PATH=/etc/novafabric/keys/offline-key.pub
nova server start --backend postgres

Generate an offline token

nova server issue-token \
  --subject slurm-job-launcher@hpc.example.com \
  --roles writer \
  --expires-in 90d \
  --key-path /etc/novafabric/keys/offline-key.pem

The command prints the raw JWT — and nothing else — on stdout; every status line (including the keypair-generation messages on the very first call) goes to stderr, so $(…) captures only the token. Store it securely:

TOKEN=$(nova server issue-token --subject ci-runner --roles writer --expires-in 30d)

For SLURM jobs, pass the token via a job environment variable. sbatch copies its own environment into the job by default (--export=ALL), so submit from a minimal environment that carries the token and nothing else from your shell:

env -i PATH="$PATH" HOME="$HOME" NOVA_TOKEN="$TOKEN" sbatch job.sh

issue-token also reads its key path from NOVAFABRIC_OFFLINE_KEY_PATH when --key-path is not given. The variable names one keypair by either of its files: given offline-key.pub, issue-token signs with the private key next to it (offline-key.pem, else offline-key.key, else offline-key), so the same export works for the server and for issuing (works today):

NOVAFABRIC_OFFLINE_KEY_PATH=/etc/novafabric/keys/offline-key.pub \
  nova server issue-token --subject user@example.com --roles reader

On a verify-only host that holds just the .pub, issue-token refuses with exit 2 and names the missing private key; it never generates a new keypair over a public key the server trusts. (Before 2026-10-23 it read the .pub as a private key and failed, or — when the .pub did not exist yet — wrote the public key over the private one.) Service-account creation on the server follows the same rule: it needs the private key and answers 400 when the server was given only the .pub.

Revoke a token

Each issued token has a jti (token ID) claim. Revoke it by ID:

nova server revoke-token <token-id>

Revocation is recorded in a SQLite file next to the key, <key-dir>/offline-key-tokens.db, whatever the server backend. The server sees a revocation only if it reads that same file, so revoke on the host whose key directory the server uses. Revoked tokens return HTTP 401 on the next API call.

Config for offline mode

There is no YAML for the offline key: offline_key_path in the config file is ignored. Set NOVAFABRIC_OFFLINE_KEY_PATH in the server's environment (above). Keep the private key owned by the issuing user at mode 0600 (it is generated that way); the server does not check the key file's permissions itself.

Native listener TLS (ADR-0241, experimental)

Both nova server start and nova serve can terminate TLS themselves — for deployments with no reverse proxy in front (HPC login/compute nodes, plain VMs):

# server.yaml
tls:
  enabled: true
  cert_path: /etc/novafabric/tls/server.crt
  key_path: /etc/novafabric/tls/server.key   # must be chmod 600

or NOVA_TLS_ENABLED=1 + NOVA_TLS_CERT_PATH + NOVA_TLS_KEY_PATH (nova serve takes --tls-cert / --tls-key). Works with --workers N. Validation is fail-closed at launch: a missing file or a group/world-readable key refuses to start rather than silently serving plaintext. Certificate issuance and rotation stay with your PKI (rotation = replace files, restart); NovaFabric never operates a CA. The reverse-proxy posture remains fully supported — pick one termination point, not both.

Scenario 5 — Kubernetes (Helm)

Status: experimental. Deploys nova server start (the multi-user REST API, with OIDC/RBAC) by default since v0.102.0, backed by Postgres.

⚠ The default changed in v0.102.0 (ADR-0230), and an upgrade will refuse until you say which side you want. The chart previously shipped mode: dashboard with serve.insecure: true — the nova serve dashboard bound beyond loopback over plain HTTP, behind a single shared token that carries full, irreversible power over signed evidence. That pair matched none of ADR-0042's four named, tested and supported deployment tiers.

The dashboard is not read-only and has not been since v0.8 — it exposes DELETE /api/runs/{id}, POST /api/compliance/pii/erase, POST /api/seal/{id}/bypass and POST /api/admin/roles. Since ADR-0228 every route carries a scope, so a narrower credential can be minted, but the shared token still holds admin. Harden the access model (TLS, auth) before exposing publicly.

Install distribution channels

NovaFabric ships to three channels from the public repository, each on a v* tag:

Images are multi-arch (linux/amd64, linux/arm64). Docker Hub is an optional mirror, populated only when a DOCKERHUB_TOKEN secret is configured on the repo.

Verifying release artifacts (supply chain)

Published images and wheels are signed and carry an SBOM + SLSA build provenance — the provenance platform attesting its own supply chain.

# Image: keyless cosign signature (Sigstore/Fulcio + Rekor transparency log)
cosign verify ghcr.io/mskazemi/novafabric:<X.Y.Z> \
  --certificate-identity-regexp '^https://github.com/MSKazemi/novafabric/' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com

# Image: SLSA build provenance + SBOM (attached as OCI referrers)
gh attestation verify oci://ghcr.io/mskazemi/novafabric:<X.Y.Z> \
  --repo MSKazemi/novafabric
cosign download sbom ghcr.io/mskazemi/novafabric:<X.Y.Z>

# Wheel: SLSA provenance for the PyPI distribution
gh attestation verify novafabric-<X.Y.Z>-py3-none-any.whl --repo MSKazemi/novafabric

Quick start (bundled Postgres, evaluation only)

helm install nova oci://ghcr.io/mskazemi/charts/novafabric --version <X.Y.Z>
kubectl rollout status deploy/nova-novafabric
kubectl port-forward svc/nova-novafabric 4321:4321
# The default mode is `server` (REST API, no /dashboard). The local bearer token
# is NOT printed to the logs in a container (only a fingerprint); read it from
# $NOVAFABRIC_HOME/.server-token inside the pod:
TOKEN=$(kubectl exec deploy/nova-novafabric -- sh -c 'cat "$NOVAFABRIC_HOME/.server-token"')
curl -H "Authorization: Bearer $TOKEN" http://localhost:4321/v0/version

Production (external managed Postgres)

helm install nova oci://ghcr.io/mskazemi/charts/novafabric --version <X.Y.Z> \
  --set postgres.enabled=false \
  --set externalDatabase.host=my-pg.example.com \
  --set externalDatabase.existingSecret=nova-db \
  --set ingress.enabled=true \
  --set ingress.hosts[0].host=novafabric.example.com

The chart runs non-root by default (uid/gid/fsGroup 1000, all capabilities dropped) and applies schema migrations via an init container. See deploy/helm/novafabric/README.md for the full values reference.

Dashboard vs. server mode. The chart defaults to server mode (mode: server, since v0.102.0). The container image on its own still defaults to nova serve (NOVA_MODE=dashboard) — the experimental dashboard. It is not read-only: it exposes irreversible operations including run deletion, PII erasure, and seal bypass, all behind a single token. For the multi-user REST API with OIDC/RBAC over Postgres, use server mode (Scenario 6).

Scenario 6 — Hardened server mode (container / Helm)

Runs the multi-user REST API (nova server start, OIDC/RBAC over Postgres) instead of the dashboard. Auth is on by default: configure OIDC through the oidc: block of a mounted server.yaml (there are no OIDC environment variables), or the server generates a local bearer token (--insecure-no-auth is never used). Migrations always upgrade to the packaged schema head — the server refuses to start when stamped behind head, so a pinned-below revision breaks the deploy.

Container (Docker / Compose)

Set NOVA_MODE=server on the image. It listens on NOVA_PORT (default 7433) and, with NOVA_WORKERS>1, runs that many uvicorn workers (requires Postgres):

The entrypoint waits for Postgres with pg_isready against PGHOST (default postgres) / PGPORT / PGUSER, then runs nova db upgrade --backend postgres, which reads NOVAFABRIC_METADATA_DSN — so set all of them:

docker run -e NOVA_MODE=server \
  -e PGHOST=my-pg -e PGUSER=nova \
  -e NOVAFABRIC_POSTGRES_DSN=postgresql://nova:***@my-pg:5432/nova \
  -e NOVAFABRIC_METADATA_DSN=postgresql://nova:***@my-pg:5432/nova \
  -e NOVA_WORKERS=4 \
  -p 7433:7433 ghcr.io/mskazemi/novafabric:<X.Y.Z>

For OIDC, mount a server.yaml with an oidc: block at ~/.config/novafabric/server.yaml of the container user (see Scenario 3).

Helm

helm install nova oci://ghcr.io/mskazemi/charts/novafabric --version <X.Y.Z> \
  --set mode=server \
  --set server.port=7433 \
  --set server.workers=4 \
  --set postgres.enabled=false \
  --set externalDatabase.host=my-pg.example.com \
  --set externalDatabase.existingSecret=nova-db

In server mode the chart's readiness/liveness probes hit /readyz and /livez (both modes expose them); /readyz gates traffic on database reachability and schema-skew. Provide OIDC config as a mounted server.yaml (the oidc: block) — extraEnv cannot enable OIDC, because no OIDC environment variables exist.

How the server authenticates a request

Every API request carries one bearer credential. The server tries the credential types in this order and stops at the first that applies; a credential that applies but fails verification is rejected with 401 rather than passed on:

flowchart TD
    R["Request with<br/>Authorization: Bearer …"] --> K{"API key<br/>(nvfk_…)?"}
    K -- valid --> OK["authenticated"]
    K -- "not an API key" --> O{"OIDC configured?<br/>oidc.issuer_url + audience"}
    O -- yes --> J["verify JWT against<br/>the IdP's JWKS"]
    J --> OK
    O -- no --> F{"JWT-shaped and server started with<br/>NOVAFABRIC_OFFLINE_KEY_PATH?"}
    F -- yes --> T["verify offline ed25519 token<br/>+ revocation record"]
    T --> OK
    F -- no --> L["compare with the local token<br/>$NOVAFABRIC_HOME/.server-token"]
    L --> OK

insecure_no_auth: true (never in production) replaces the last step with anonymous admin access. Roles then come from the credential, as described below.

Role assignment

Status: experimental. Roles are enforced on all server API endpoints per ADR-0018.

The four built-in roles are:

Assign a role

nova server assign-role user@example.com writer
nova server assign-role ci-runner@cluster admin --assigned-by ops-team

Roles are stored in the role_assignments table. When using OIDC, roles can also be carried in the nova_roles (or roles) JWT claim. The token's own roles are checked first; role_assignments rows are only a fallback — never for an API key or an offline token issued with --roles, whose roles are their whole grant — and that fallback is consulted only once the workspace/org model is in use (at least one scoped membership or service account exists, ADR-0178). ⚠ In a stock deployment that has neither, a recorded row has no effect on OIDC or offline-token principals. assign-role still writes the row (the last-admin guard counts it, and it takes effect the moment the workspace/org model is in use), but it then prints Role '<role>' recorded for '<subject>' and a warning: this assignment is not in force on stderr instead of claiming success. GET /v0/admin/roles reports the same thing per row as effective_now. To grant a role in a stock deployment today, carry it in the token: the OIDC nova_roles/roles claim, or nova server issue-token --roles …. promoter and approver also exist for separation-of-duties and SCIM group mapping (nova server scim-map-group).

SCIM provisioning (experimental)

Status: experimental (ADR-0139, first slice: discovery + Users). Lets an enterprise IdP (Okta, Entra ID, OneLogin, Keycloak) push the user lifecycle — create, deactivate, delete — into the server over standard SCIM 2.0 (RFC 7643/7644). Server-mode only; local mode is structurally unaffected.

Enable it

SCIM is doubly opt-in and disabled by default. Both of these must be set before the /scim/v2/* endpoints exist (otherwise they return plain 404):

# server config (ADR-0029)
scim:
  enabled: true

# The dedicated provisioning bearer token — env var only, never YAML.
export NOVAFABRIC_SCIM_TOKEN="<long random secret>"
# (flag can also be set via env: NOVAFABRIC_SERVER_SCIM_ENABLED=true)

The IdP authenticates with Authorization: Bearer $NOVAFABRIC_SCIM_TOKEN. This token is provisioning-scoped: it is not a JWT and grants no access to the /v0/ API, capsules, assets, lineage, or evidence.

Endpoints (first slice)

Responses use application/scim+json and the RFC 7644 error envelope, not the NovaFabric /v0/ error model.

De-provisioning semantics

• active: false (or DELETE) revokes all of the subject's role_assignments rows immediately — the same table nova server assign-role writes; the userName is the auth subject.
• The ADR-0060 last-admin lockout guard applies: a SCIM deprovision that would remove the last admin (with no OIDC configured in the server YAML) is refused with a SCIM 409 and nothing is mutated — the user stays active and keeps its roles. The same holds for a Group PATCH, PUT or DELETE whose membership change would revoke the last admin: it is refused with 409 before the group changes. A PATCH applies all of its operations or none.
• Every mutation appends an event (user.create / user.update / user.deactivate / user.delete, with roles_before/roles_after) to the append-only scim_audit_events table.
• PII minimization: only userName, active, externalId, displayName and emails are stored. Other wire attributes (enterprise extension, addresses, phone numbers, …) are accepted but dropped, never persisted.

Group → role mapping (shipped, experimental)

Groups → role mapping (/scim/v2/Groups, ADR-0139 P3 / ADR-0190) is implemented: the Groups resource supports POST/GET/list/PATCH/PUT (full-replace)/DELETE, and group membership maps to RBAC roles via nova server scim-map-group <group> <role> (--list / --remove supported). Provisioning events are inspectable with nova server list-scim-events. Roles may still also be assigned via nova server assign-role or OIDC claims; SCIM manages the user lifecycle and revokes roles on deprovision.

Not implemented yet (planned): the nova server issue-scim-token CLI command — provision the SCIM bearer token via $NOVAFABRIC_SCIM_TOKEN for now.

Config file reference

The config file is ~/.config/novafabric/server.yaml, or the file passed with nova server start --config <path> — there is no other search path. (NOVAFABRIC_SERVER_CONFIG is set internally to hand the path to --workers child processes; do not set it yourself — nova server start overwrites it, or clears an inherited value when --config is absent, so every worker loads the same file as the parent.) Keys are top-level; the main ones, with their defaults:

backend: sqlite          # "sqlite" (default) or "postgres"
db_path: null            # SQLite registry for every server table (orgs, workspaces, roles,
                         # API keys, SCIM, usage, webhooks); null = ~/.novafabric/registry.db
host: "127.0.0.1"
port: 7433

oidc:                    # on when BOTH are set
  issuer_url: ""
  audience: ""

tls:                     # first-party listener TLS (ADR-0241)
  enabled: false
  cert_path: null
  key_path: null         # must be mode 0600 — the server refuses otherwise

scim:
  enabled: false         # SCIM 2.0 provisioning (ADR-0139, experimental).
  group_role_map: {}     # Also requires $NOVAFABRIC_SCIM_TOKEN (env only).

rate_limits:             # API rate limiting + storage quotas (ADR-0179, experimental)
  enabled: false         # disabled by default — limiter is fully inert when false
  ingest: { rate: 100, burst: 200 }   # tokens/sec + burst capacity per class
  read:   { rate: 50,  burst: 100 }
  admin:  { rate: 10,  burst: 20 }
  audit_threshold_rejections: 100     # rejections/window that raise an audit event
  audit_window_seconds: 60
  quota:                 # storage quotas (warn-then-reject); absent block ⇒ no checks
    max_capsules_soft: 0 # 0 = unlimited
    max_capsules_hard: 0
    max_bytes_soft: 0
    max_bytes_hard: 0

observability:           # self-observability surface (ADR-0182, experimental)
  metrics_exempt: false  # true serves /metrics without auth (loopback/isolated only)
  self_tracing: false    # opt-in; one OTel span per request, never leaves the deployment
  self_tracing_endpoint: null  # OTLP/HTTP traces URL; null = local serve OTLP ingest

Further blocks, each documented with its feature: ingest (below), saml, bulk, usage, api_keys, pagination, webhooks, step_up; and the acknowledgement flags insecure_no_auth, i_know_this_is_public, i_accept_shared_capsule_store, demo_device_grant. There are no workers, base_url, log_level or otel_endpoint keys — worker count is the --workers flag. Unknown keys are ignored silently, so check spelling.

Environment overrides are a fixed list, not one per key. The main ones: NOVAFABRIC_SERVER_HOST, NOVAFABRIC_SERVER_PORT, NOVAFABRIC_SERVER_BACKEND, NOVAFABRIC_SERVER_DB_PATH, NOVAFABRIC_SERVER_SCIM_ENABLED, NOVAFABRIC_SERVER_METRICS_EXEMPT, NOVAFABRIC_SERVER_RATE_LIMITS_ENABLED, the NOVAFABRIC_SERVER_INGEST_*, NOVAFABRIC_SERVER_USAGE_* and NOVAFABRIC_SERVER_WEBHOOKS_* families, NOVAFABRIC_SERVER_STEP_UP_ENABLED, NOVAFABRIC_SERVER_INSECURE_NO_AUTH, NOVAFABRIC_SERVER_I_KNOW_THIS_IS_PUBLIC, NOVAFABRIC_SERVER_I_ACCEPT_SHARED_CAPSULE_STORE, NOVAFABRIC_SERVER_DEMO_DEVICE_GRANT (full list: _apply_env_overrides in src/novafabric/server/config.py).

Secrets come from the environment only: NOVAFABRIC_POSTGRES_DSN, NOVAFABRIC_OFFLINE_KEY_PATH, NOVAFABRIC_SCIM_TOKEN, NOVAFABRIC_SERVER_TOKEN (pins the local bearer token). If you put postgres_dsn or offline_key_path in the YAML file it is dropped silently — no warning is logged.

Ingest limits (experimental)

Status: experimental — ADR-0203 P1 (spec). Applies to POST /v0/capsules only; local-first commands are untouched.

The capsule-upload route is bounded: the request body is size-capped, spooled to disk in 1 MiB chunks (peak memory per upload is O(chunk), not O(archive)), extracted atomically via a temp directory + rename, and guarded against zip bombs and ..-traversal member names.

Config block (server.ingest.*, ADR-0029 conventions; all keys additive — an absent block means the defaults below):

ingest:
  max_upload_bytes: 268435456          # 256 MiB body cap → 413 payload_too_large
  spool_chunk_bytes: 1048576           # 1 MiB read/decompress chunk (min 65536)
  zip_max_entries: 10000               # archive member cap → 422 zip_guard_violation
  zip_max_uncompressed_bytes: 2147483648  # 2 GiB decompressed-total cap
  zip_max_ratio: 100.0                 # compression-ratio cap (per member + total)
  max_concurrent_unpack: 16            # concurrent unpacks per worker (1-256)

Env overrides: NOVAFABRIC_SERVER_INGEST_MAX_UPLOAD_BYTES, NOVAFABRIC_SERVER_INGEST_SPOOL_CHUNK_BYTES, NOVAFABRIC_SERVER_INGEST_ZIP_MAX_ENTRIES, NOVAFABRIC_SERVER_INGEST_ZIP_MAX_UNCOMPRESSED_BYTES, NOVAFABRIC_SERVER_INGEST_ZIP_MAX_RATIO, NOVAFABRIC_SERVER_INGEST_MAX_CONCURRENT_UNPACK.

• max_concurrent_unpack (ADR-0262, works today) bounds how many capsules one worker unpacks at the same time. Unpacking is blocking CPU + disk work; until ADR-0262 it ran inline on the worker's event loop, so each worker served exactly one upload at a time and ingest capacity was workers / service_time. A 314-VM fleet measured the result: throughput pinned at 61.6 req/s, flat across a 9x span of offered load, p99 26.8 s — on a host at ~20% CPU. Nothing was lost (41,774 uploads, 41,774 stored capsules) — it was slow, not lossy. The work now runs in the threadpool, and this key is the bound that keeps a higher concurrency from becoming unbounded concurrent extraction. Raising it raises peak transient disk and memory proportionally (see the transient-disk note below); lower it on a small host.

• Rejections use the standard error envelope: HTTP 413 payload_too_large (body over the cap; details carries limit_bytes/received_bytes) and HTTP 422 zip_guard_violation (details.reason is one of entry_count, total_uncompressed, compression_ratio, unsafe_member_name, no_capsule_root).

• The manifest the upload checks (run_id, parent, duplicate) is the one it stores: the member that lands at the capsule root, either capsule.yaml or <dir>/capsule.yaml when every member shares that one top-level directory. An archive with no such member is 422 no_capsule_root, one where two members extract to capsule.yaml is 422 unsafe_member_name, and one with no capsule.yaml anywhere is 400.

• Uploads larger than 256 MiB now require explicit operator config — raise max_upload_bytes, or set any key to 0 to disable that limit (escape hatch, discouraged).

• Transient disk: up to max_upload_bytes (spool) plus the decompressed size (temp extract dir) per in-flight upload, under <capsule_dir>/.ingest-tmp/; both are removed on every exit path. Since ADR-0262 a worker can have up to max_concurrent_unpack extractions in flight, so size the volume for that multiple.

Recommended (normative, ADR-0203 D4): for any non-loopback deployment, also enable the ADR-0179 request-rate limiter — it stays opt-in and off by default:

rate_limits:
  enabled: true    # shipped ingest budget: 100 req/s, burst 200

A reverse-proxy body cap (e.g. nginx client_max_body_size) is a sensible additional layer, but NovaFabric is safe as shipped without one.

Migration from local to shared Postgres

Full walkthrough:

# 1. Install the server extra
pip install 'novafabric[server]'

# 2. Provision Postgres and create the novafabric database
createdb novafabric

# 3. Create the schemas (registry track + MetadataStore track)
export NOVAFABRIC_POSTGRES_DSN="postgresql://user:pass@localhost/novafabric"
export NOVAFABRIC_METADATA_DSN="$NOVAFABRIC_POSTGRES_DSN"
nova db upgrade --track registry --backend postgres
nova db upgrade --backend postgres

# 4. Migrate existing SQLite data (idempotent)
nova migrate-to-postgres \
  --source ~/.novafabric/registry.db \
  --target "postgresql://user:pass@localhost/novafabric" \
  --log ~/migration.jsonl

# 5. Verify row counts match
nova doctor --check-storage --backend postgres \
  --postgres-dsn "postgresql://user:pass@localhost/novafabric"

# 6. Switch the server to Postgres (reads NOVAFABRIC_POSTGRES_DSN from step 3)
nova server start --backend postgres

# 7. Confirm the server is healthy
nova doctor --check-storage --backend postgres

The SQLite file is never deleted or modified. Keep it as a backup until the team has confirmed that Postgres is stable.

Postgres-to-SQLite downgrade migration is explicitly not supported per ADR-0016.