NovaFabric Operator Guide

This guide is for HPC system administrators and platform engineers who deploy NovaFabric on shared infrastructure — from a single workstation to a multi-node SLURM cluster or a Kubernetes namespace. NovaFabric captures, replays, diffs, and audits AI-agent and model runs, turning any command into a portable, schema-valid, secret-scanned Run Capsule you own. This document covers the operational mechanics: installation, runner configuration, wire-level capture, site-specific URL-registry customization, cryptographic sealing, and a full Docker Compose deployment.

What you will learn

The mental model

Every runner does exactly one job: guarantee that the capsule directory is writable, that sitecustomize.py is reachable on PYTHONPATH, and that all capsule artifacts are on the local filesystem before nova capture returns. Everything else — Local subprocess, Docker container, SLURM batch job, Kubernetes Job — is a variation on that theme. The capsule is the source of truth; nothing leaves the node unless you explicitly push it.

For developer onboarding (contributing code, running tests), see CONTRIBUTING.md. For the complete CLI usage reference (every command, flag, and default), see docs/cli-reference.md. For the full NovaSeal configuration reference, see docs/novaseal-configuration.md.

Table of contents

  1. Deployment model overview
  2. Prerequisites
  3. Deployment scenarios
  4. URL registry configuration
  5. Troubleshooting
  1. What is not supported yet
  2. Docker Compose deployment (nova serve + dashboard)
  3. Summary and next steps

1. Deployment model overview

NovaFabric is self-contained: there is no server, no agent daemon, and no persistent network service required to capture a run. Capture happens in-process inside the workload's own Python interpreter.

The mechanism is a sitecustomize.py hook loader. When nova capture launches a workload, it materializes this file to a directory that is prepended to PYTHONPATH. Python's import machinery auto-loads sitecustomize at interpreter startup, before any user code runs. The hooks patch httpx, requests, aiohttp, and urllib3 transports to intercept outbound HTTP calls and write structured records to a capsule directory.

Every runner — local subprocess, Docker container, SLURM batch job, Kubernetes Job — uses this same mechanism. The runner's only job is to ensure:

The capsule directory holds the complete record of a run: capsule.yaml (metadata), model-calls.jsonl (wire-level AI traffic), and outputs/ (stdout, stderr). No data leaves the node unless you explicitly push a capsule somewhere.

2. Prerequisites

Python version

Python 3.12 or later is required on every node that runs nova capture or any NovaFabric CLI command. This applies to both the submit node and compute nodes on SLURM clusters.

Package installation

Install with pip or uv. Both work. uv is recommended for reproducible environments:

# pip
pip install novafabric

# uv (recommended for shared or HPC venvs)
uv pip install novafabric

The optional serve extra adds the experimental local dashboard (see section 6):

pip install "novafabric[serve]"

A few other narrow extras exist for optional integrations: clickhouse (ClickHouse sink for cost attribution), nats (NATS JetStream lineage consumer), avro (Avro capsule-event serialization), query (DuckDB accelerator for nova query), and energy-gpu (NVML GPU energy readings). Install everything at once with pip install "novafabric[all]" — this excludes cloud-vendor extras (worm-*, seal-aws/seal-azure/seal-gcp) and agent-framework adapter extras, which you should install individually for the one vendor/framework you actually use.

To see the full list against your own install rather than this prose, run nova doctor --check-extras. It reports every declared extra as complete or incomplete, names the distributions missing from each, and prints the exact pip install 'novafabric[<extra>]' command. The list comes from the installed distribution metadata, so it stays correct as extras are added or renamed — this section can go stale, that output cannot. Exit code stays 0: an omitted extra is a normal choice, not a failure.

Changed in v0.99.0 — a leaner default install

A plain pip install novafabric now pulls 113 MB / 42 packages instead of 412 MB / 50. Four dependencies moved out of the default install into the extras that actually import them (ADR-0222), because every one of their import sites was already behind an extra:

Moved out of core Now comes from
duckdb scale, serve, query
pyarrow scale, serve, lineage-migration
python-louvain serve
clickhouse-connect scale, clickhouse

numpy also disappears from a default install — it only ever arrived transitively via python-louvain.

Migration. If you relied on duckdb, pyarrow, clickhouse_connect or community being importable after a plain pip install novafabric, use pip install 'novafabric[all]' to restore the previous importability, or install the narrower extra from the table above. Prefer the narrow extra: [all] is a superset of the old default install, not an equivalent of it — it also pulls compliance, spkg, scale, sigstore, janusgraph and the rest, so it ends up considerably larger than the 412 MB you had before. These were never part of NovaFabric's public API, but they did work before.

No CLI flag, capsule schema, evidence-bundle format or REST endpoint changed, and networkx deliberately stays in core (the CLI imports it at start-up, and nova insights depends on it). Every default command behaves identically on a lean install. Where a moved dependency is still reachable, the behaviour is explicit rather than silent: nova query falls back to stdlib sqlite3 with identical results and reports index.engine: "sqlite"; nova backup skips the derived DuckDB topology cache with a stated reason and still succeeds; nova restore fails loudly rather than reporting an unopened .duckdb store as verified; ClickHouse paths raise an ImportError naming pip install novafabric[clickhouse].

Container and server deployments are unaffected — deploy/docker/Dockerfile installs [server,serve,clickhouse,lineage-kuzu,lineage-migration], which restores the full surface inside the image.

What compute nodes need

For any runner that executes on a remote node (SLURM, Kubernetes):

NovaFabric does not require internet access for capture, validation, replay, diff, or local lineage queries. The only outbound traffic is the workload's own AI API calls, and only if the workload makes them.

Scheduler tools

Runner Required binaries
local none beyond Python
docker docker (daemon must be reachable)
slurm sbatch, scontrol, sacct on PATH
kubernetes kubectl configured with a reachable cluster

3. Deployment scenarios

All four runners share the same capture mechanism (§1) and the same URL registry (§4). They differ only in where the workload executes and how capsule artifacts return to the submit node. Pick the runner that matches where your workload runs:

Runner --runner Where the workload runs Key operational constraint
Local local (default) Submit node, as a subprocess None — self-contained on disk
Docker docker A container on the submit node Image must have NovaFabric installed
SLURM slurm A batch job on an allocated compute node Capsule dir must be on shared filesystem
Kubernetes kubernetes A Kubernetes Job Namespace RBAC; image must have NovaFabric

Runner defaults can be set per project in .novafabric/runners.yaml and overridden per invocation with --runner-option key=value. The subsections below document each runner in increasing order of operational constraint.

3.1 Laptop or single workstation (LocalRunner)

Works today. This is the default runner and requires no configuration beyond installing the package.

nova capture python my_agent.py

nova capture launches my_agent.py as a subprocess, injects sitecustomize.py into a temporary directory on PYTHONPATH, and writes the capsule to $NOVAFABRIC_HOME/capsules/<run-id>/ (override with NOVAFABRIC_CAPSULE_DIR).

To validate and inspect a run:

# Validate the capsule against its schema
nova validate ~/.novafabric/capsules/<run-id>/

# Inspect the manifest directly (capsule.yaml is human-readable)
cat ~/.novafabric/capsules/<run-id>/capsule.yaml

nova validate accepts a spec file, a capsule directory, or a replay directory. Because the capsule is a plain directory on disk, every other artifact (model-calls.jsonl, trace.jsonl, outputs/) is directly readable without any NovaFabric runtime.

No shared filesystem, no scheduler, no network service needed. The capsule is self-contained on disk.

Explicit runner flag (same as default):

nova capture --runner local python my_agent.py

Timeout (default 600 s):

nova capture --runner local --timeout 120 python my_agent.py

3.2 Docker or OCI container (DockerRunner)

Works today. The workload runs inside a Docker container. The capsule directory is mounted as a volume so the orchestrator can read artifacts after the container exits.

Requirements:

FROM python:3.12-slim
RUN pip install novafabric
COPY my_agent.py /app/my_agent.py
WORKDIR /app

Basic invocation:

nova capture \
  --runner docker \
  --runner-option image=myorg/agent-runtime:abc1234 \
  python my_agent.py

All runner_options for DockerRunner:

Key Required Description
image yes Fully-qualified image reference
network no --network value (default: Docker default / bridge)
workdir no Container working directory
user no --user value (e.g. 1000:1000); recommended to avoid root-owned capsule files
extra_volumes no List of host:container[:opts] mount strings
extra_env no Dict of additional env vars for the container. Passed as docker run -e NAME, never -e NAME=VALUE — see below

Project-level defaults in .novafabric/runners.yaml:

schema_version: "0.1.0"
default_runner: docker
runners:
  docker:
    image: myorg/agent-runtime:latest
    user: "1000:1000"
    network: bridge

Security posture: The DockerRunner never passes --privileged, does not mount the Docker socket, and does not enable host namespaces (hostNetwork, hostPID, hostIPC). These are hard restrictions in the runner code.

The runner only forwards NOVAFABRIC_* environment variables and any keys specified in extra_env into the container. It does not pass through the full host environment, which would risk leaking credentials into the container at runtime.

Values never go on docker run's command line (works today). Each variable is passed as -e NAME, and its value is placed in the environment of the docker client process, from which Docker copies it into the container. A process's argv is readable by every user on the host (ps, /proc/<pid>/cmdline) for as long as the container runs; its environment is readable only by the same user and root. Before this change the runner ran docker run -e OPENAI_API_KEY=sk-…, so an extra_env API key was visible to every local user. The same applies to eval containers (run_eval_container).

What is still visible:

3.3 SLURM cluster (SlurmRunner)

Works today. This section is the most detailed because SLURM introduces constraints that do not exist for local or Docker runners.

Overview

The SlurmRunner submits a generated batch script on sbatch's stdin (never --wrap; see How the job is submitted), polls for completion using scontrol show job (primary) with sacct as a fallback, and reads stdout/stderr from per-job output files. Capsule artifacts accumulate on a shared filesystem, then the orchestrator reads them on the submit node after the job finishes.

Three failure modes were caught and fixed during live-cluster validation (v0.6.10, v0.6.11, v0.6.12). This section reflects that validated behavior.

Shared filesystem requirement

This is the most important operational constraint for SLURM deployments:

The capsule directory must be on a filesystem shared between the submit node and every compute node SLURM might allocate.

If the capsule directory is on node-local storage (e.g. /tmp or any path that is not the cluster's shared scratch/home), the job will fail the moment the scheduler places it on any node other than the submit node. SLURM gives no guarantee about which node a job runs on.

Acceptable shared filesystems: NFS, Lustre, GPFS, BeeGFS, any POSIX-compatible network filesystem that is mounted at the same path on all nodes.

Set the shared base path via the environment variable:

export NOVAFABRIC_SLURM_SHARED_DIR=/lustre/scratch/myproject

When this variable is set, the live integration tests also use it. When unset, tests fall back to node-local /tmp, which only works on a single-node cluster.

Python environment on compute nodes

The compute nodes must be able to import novafabric. The recommended approach is a shared-filesystem venv:

# On the submit node, with the shared FS mounted:
python3 -m venv /lustre/scratch/myproject/nova-venv
/lustre/scratch/myproject/nova-venv/bin/pip install novafabric

Then tell the runner which Python interpreter to use:

export NOVAFABRIC_SLURM_VENV_PYTHON=/lustre/scratch/myproject/nova-venv/bin/python3

If the venv's site-packages are not self-contained (e.g. you have editable installs or a non-standard layout), you can also specify the site-packages path to prepend to PYTHONPATH on the compute node:

export NOVAFABRIC_SLURM_VENV_SITE=/lustre/scratch/myproject/nova-venv/lib/python3.12/site-packages

Alternatively, install NovaFabric at the same absolute path on every compute node and ensure that path is on the default PYTHONPATH. Shared-FS venv is simpler and is the validated approach.

sitecustomize injection

This is how wire-level capture fires on compute nodes.

Before submitting the job, SlurmRunner writes sitecustomize.py to the capsule directory. The capsule directory is on shared filesystem, so it is visible to the compute node. The batch script (sent on sbatch's stdin) prepends the capsule directory to PYTHONPATH. When the compute node's Python starts, it auto-loads sitecustomize, which installs NovaFabric's HTTP hooks before any user code runs.

Without this injection, the workload runs bare on the compute node: no hooks install, model-calls.jsonl is created but never written to, and the capsule appears empty. This was the bug fixed in v0.6.11.

If SlurmRunner cannot write sitecustomize.py to the capsule directory (e.g. permissions error, filesystem not mounted), the runner returns runner_status="failed_setup" with a diagnostic message before submitting the job. It does not silently submit a job that will produce empty capsules.

Environment variables reference

Variable Required Description
NOVAFABRIC_SLURM_SHARED_DIR strongly recommended Base path on shared filesystem for capsule directories and live test isolation
NOVAFABRIC_SLURM_VENV_PYTHON for live tests Path to Python interpreter in the compute-node-accessible venv; used by the test_sitecustomize_fires_on_compute integration test
NOVAFABRIC_SLURM_VENV_SITE optional Site-packages path to prepend to compute-node PYTHONPATH
NOVAFABRIC_SLURM_PARTITION per-site Default partition for the site env template
NOVAFABRIC_URL_REGISTRY optional Path to a site-local URL registry file (see section 4)

NOVAFABRIC_CAPSULE_DIR and NOVAFABRIC_SPAN_ID are set automatically by the orchestrator and exported by the batch script the runner pipes to sbatch. Do not set them manually when using nova capture.

runner_options for SlurmRunner

Pass via --runner-option key=value on the CLI or via .novafabric/runners.yaml:

Key Required Description
partition yes SLURM partition (queue) to submit to
account no SLURM account / project (--account)
qos no Quality of service (--qos)
time no Wall-clock limit, e.g. "01:00:00" (--time). If unset, derived from --timeout
nodes no Number of nodes (--nodes). Default 1
gres no Generic resources, e.g. "gpu:a100:1" (--gres)
mem no Memory request, e.g. "16G" (--mem)
constraint no Node feature constraint (--constraint)
poll_interval_s no How often to poll for job state, in seconds. Default 5.0
output_dir no Where to write slurm-<jobid>.out/.err. Default: capsule directory
extra_env no Variables to export in the job, e.g. extra_env=LOG_LEVEL=debug,HF_HOME=/scratch/hf, or as a JSON object of strings, extra_env='{"LOG_LEVEL":"debug"}' (needed when a value contains a comma; a malformed one fails setup). The job receives only these, NOVAFABRIC_* (minus credentials) and PATH (ADR-0270, ADR-0311). Names must be shell identifiers

How the job is submitted

The runner generates a /bin/sh batch script — the export lines, then exec of your command — and hands it to sbatch on stdin. It is never passed as sbatch --wrap <script> and never written to the capsule directory: a command line is readable by every user on the submit host (ps, /proc/<pid>/cmdline), and the capsule directory is on a shared filesystem and is sealed into the capsule. SLURM stores the script with the job, where you and the cluster's administrators can read it (scontrol write batch_script <jobid>).

Every value is single-quoted in the script (shlex.quote), so $, backticks, ; and & in an extra_env value or a command argument reach the job verbatim; they are not expanded or executed on the compute node.

Credentials in extra_env. The value is visible to the cluster's administrators (above), and if you type it on the command line it is in nova capture's own argv for the whole run. Prefer a credential the job reads from a file in your home directory with mode 0600.

State polling: scontrol and sacct

The runner queries job state using scontrol show job <jobid> as the primary source and falls back to sacct if scontrol returns nothing (which happens approximately 5 minutes after a job completes, when slurmctld forgets it).

This two-path design matters because many development and minimal SLURM clusters do not run slurmdbd. Without slurmdbd, sacct times out or returns no data. scontrol reads directly from slurmctld memory and works without slurmdbd. The sacct fallback handles long-running post-completion queries where scontrol is no longer useful.

If your cluster runs slurmdbd, both paths work. If it does not, the primary scontrol path handles the common case.

Basic invocation

nova capture \
  --runner slurm \
  --runner-option partition=gpu \
  --runner-option gres=gpu:a100:1 \
  --runner-option time=01:00:00 \
  --runner-option mem=16G \
  python my_agent.py

Site environment template

A ready-made environment template is at:

site-config/examples/slurm-hpc/nova-env.sh

Copy it to site-config/local/slurm-hpc/nova-env.sh (the local/ directory is gitignored), fill in your site values, and source it before running nova capture:

cp site-config/examples/slurm-hpc/nova-env.sh site-config/local/slurm-hpc/nova-env.sh
# Edit the file — set NOVAFABRIC_SLURM_SHARED_DIR, NOVAFABRIC_SLURM_VENV_PYTHON, partition
source site-config/local/slurm-hpc/nova-env.sh
nova capture --runner slurm --runner-option partition=$NOVAFABRIC_SLURM_PARTITION python my_agent.py

Never commit the filled-in local/ copy — it contains site-specific paths.

Project-level SLURM defaults in runners.yaml

schema_version: "0.1.0"
default_runner: slurm
runners:
  slurm:
    partition: gpu
    qos: high
    constraint: a100
    gres: "gpu:a100:1"
    mem: "32G"
    time: "02:00:00"

Running live integration tests against a real cluster

The test suite includes opt-in live tests that require a reachable SLURM cluster:

# Minimal: echo test (no wire capture)
export NOVAFABRIC_SLURM_SHARED_DIR=/home/vagrant
uv run pytest tests/test_runners_slurm.py::TestSlurmRunnerLiveSmoke::test_echo_succeeds -v

# Full wire-capture e2e (requires venv on shared FS)
export NOVAFABRIC_SLURM_SHARED_DIR=/home/vagrant
export NOVAFABRIC_SLURM_VENV_PYTHON=/home/vagrant/novafabric-venv/bin/python3
uv run pytest tests/test_runners_slurm.py::TestSlurmRunnerLiveSmoke -k sitecustomize -v

Both tests were validated against a 3-node cluster running Vagrant + libvirt. See v0.6.12 release notes for the full validation summary.

3.4 Kubernetes (KubernetesRunner)

Works today. The workload runs as a Kubernetes Job. The runner submits the manifest via kubectl apply, polls for Job completion, retrieves logs via kubectl logs, copies capsule artifacts back to the submit node via kubectl cp, and then deletes the Job.

Requirements:

Basic invocation:

nova capture \
  --runner kubernetes \
  --runner-option image=myorg/agent-runtime:abc1234 \
  --runner-option namespace=novafabric-runs \
  python my_agent.py

All runner_options for KubernetesRunner:

Key Required Description
image yes Fully-qualified image reference
namespace yes Target namespace (must exist with RBAC)
service_account no Pod serviceAccountName (recommended for least-privilege)
node_selector no Dict of node selector labels
resources no Container resources block (requests and limits)
poll_interval_s no Job status poll interval in seconds. Default 2.0
extra_env no Dict of env vars to place in the pod. The submitting shell's environment is not inherited — see below

Example resources block:

runners:
  kubernetes:
    namespace: novafabric-runs
    service_account: novafabric-runner
    image: myorg/agent-runtime:latest
    resources:
      requests:
        cpu: "2"
        memory: "4Gi"
      limits:
        cpu: "4"
        memory: "8Gi"

⚠ Wire-level capture does not run under the Kubernetes runner

--runner kubernetes records stdout, exit code and the environment lock, but not model calls or tool calls: model-calls.jsonl and tool-calls.jsonl come back empty regardless of what the workload did, and the capsule still reports status: success.

The runner does not inject NovaFabric's capture hook loader into the pod — its capsule is an emptyDir, so neither the Docker bind-mount nor the SLURM shared-filesystem mechanism transfers (defect B3, ADR-0272 — see the decisions index). As of this release the runner says so on stderr and sets runner_metadata.wire_capture = "unavailable", so the gap is visible rather than silent — but it is not yet fixed.

If you need model-call evidence today, use --runner docker (fixed in the same release), --runner slurm, or --runner local.

Environment variables (changed in v0.102.0 — ADR-0270): the pod receives only NOVAFABRIC_* variables plus anything you name in extra_env. The environment of the shell you ran nova capture in is not forwarded.

Previously every variable of the submitting shell was written into the Job object as a literal value: — readable by anyone with get job in the namespace and persisted in etcd. If your workload used to pick up a variable implicitly, name it explicitly now:

nova capture --runner kubernetes \
  --runner-option image=myorg/agent-runtime:abc1234 \
  --runner-option namespace=novafabric-runs \
  --runner-option extra_env=LOG_LEVEL=debug \
  python my_agent.py

Do not put credentials in extra_env. Its values become literal value: entries in the Job object, which is exactly the exposure ADR-0270 closed. Give the pod a service_account bound to a Kubernetes Secret, or bake the credential into the image's own secret mount, so the value never enters the manifest.

Security posture: The KubernetesRunner sets allowPrivilegeEscalation: false, privileged: false, hostNetwork: false, hostPID: false, hostIPC: false on every pod it submits. These are hard-coded in the job manifest builder and cannot be overridden via runner_options.

Capsule artifact sync: After the workload exits, the runner runs kubectl cp to copy the in-pod capsule directory (/novafabric/capsule) back to the submit node. If kubectl cp fails (e.g. the pod exited before the copy ran), the runner records a warning in stderr but still returns the workload's exit code. Inspect the capsule directory to determine whether artifacts were recovered.

Known limitation (v0.6): The KubernetesRunner does not surface the workload's exact exit code when the Job fails. It reports 0 on succeeded and 1 on failed. Exact exit codes are queued for a v0.6.x follow-up.

3.5 LSF and PBS/Torque clusters (LSFRunner, PBSRunner) — experimental

Status: experimental. Both are registered, selectable as --runner lsf / --runner pbs, and covered by unit tests, but they have not been exercised on a live cluster the way slurm has. They were previously documented nowhere, which is not the same as not existing — nova capture --runner lsf has been accepted by the CLI for some time.

Both follow the SLURM model: build a job script, submit it, poll for completion, and read stdout/stderr back. Wire-level capture works — each uses a fixed job-script template that puts the hook loader on PYTHONPATH. lsf pipes its script to bsub on stdin and never writes it to disk, because it carries the extra_env values (as for slurm, above); pbs takes no extra_env and writes its script, which holds no values from your environment, to nova-<run>.pbs in the capsule directory for qsub. Arguments are single-quoted in both scripts, so the job sees them verbatim.

nova capture --runner lsf \
  --runner-option queue=normal \
  --runner-option walltime=2:00 \
  python train.py

nova capture --runner pbs \
  --runner-option queue=default \
  --runner-option nodes=2 --runner-option ppn=8 \
  --runner-option walltime=02:00:00 \
  python train.py
Option LSF (bsub) PBS (qsub)
queue queue → -q, default normal queue → -q, default default
size n_cores → -n, default 1 nodes / ppn → -l nodes=N:ppn=M, default 1/1
wall clock walltime → -W, H:MM walltime → -l walltime=, HH:MM:SS
job name job_name → -J job_name → -N
resources resource_requirement → -R —
poll poll_interval_s, default 30.0 poll_interval_s, default 30.0
logs output_dir, default capsule dir output_dir, default capsule dir
environment extra_env, exported in the stdin script (ADR-0311) — (#PBS -v names only)

Preconditions — not verified by the runner; the job fails loudly if wrong:

4. URL registry configuration

The URL registry is a YAML file that maps outbound URL substrings to AI provider labels. Wire-level hooks consult this registry on every outbound HTTP call to decide whether to record it and how to classify the gen_ai.system field in model-calls.jsonl.

Bundled defaults

The vendored registry (src/novafabric/capture/hooks/url_registry.yaml) covers the providers below out of the box:

Match pattern Provider label
api.openai.com openai
api.anthropic.com anthropic
api.cohere.ai cohere
api.together.xyz together
api.mistral.ai mistral
api.replicate.com replicate
bedrock-runtime. aws.bedrock (all regions)
bedrock. aws.bedrock (control plane)
localhost:11434 ollama
127.0.0.1:11434 ollama

Ollama at its default port (11434) is captured with no configuration. AWS Bedrock is captured across all regions (the match string covers any hostname containing bedrock-runtime.).

Override hierarchy

The registry is loaded using this precedence (first match wins):

  1. $NOVAFABRIC_URL_REGISTRY — absolute path to a YAML file; explicit per-run override.
  2. ~/.novafabric/url_registry.yaml — per-user override; applies to all runs by this user.
  3. Vendored file inside the package — applies when neither override is present.

An override file replaces the vendored default entirely. It does not merge. If your override omits api.openai.com, calls to that endpoint will not be captured. Always copy in the entries you still need from the vendored file, or include them explicitly in your override.

Adding a private inference server

Case 1: Ollama on a non-default port

Set OLLAMA_BASE_URL (used by langchain_ollama) or OLLAMA_HOST (used by the raw ollama SDK) to the actual server address. The URL registry reads these environment variables at call time, so they work even when loaded by python-dotenv inside the workload — no manual registry file needed:

# Tunnel on port 11437:
export OLLAMA_BASE_URL=http://localhost:11437
nova capture python agent.py

# Or in a .env file (loaded by python-dotenv in the workload):
echo 'OLLAMA_BASE_URL=http://localhost:11437' >> .env
nova capture python agent.py

If you prefer an explicit registry entry (e.g. for a shared site config), copy site-config/examples/local-ollama/url_registry.yaml to ~/.novafabric/url_registry.yaml and add an entry for the port:

schema_version: "0.1.0"
patterns:
  - match: "localhost:11437"
    gen_ai_system: "ollama"
    transport: "http"
  - match: "localhost:11434"
    gen_ai_system: "ollama"
    transport: "http"
  - match: "127.0.0.1:11434"
    gen_ai_system: "ollama"
    transport: "http"
  - match: "api.openai.com"
    gen_ai_system: "openai"
    transport: "http"

Case 2: vLLM, TGI, llama.cpp, or any OpenAI-compatible server on a remote host

Use site-config/examples/remote-inference/url_registry.yaml as a starting point. Set match to host:port of the inference server:

schema_version: "0.1.0"
patterns:
  - match: "gpu-node-01.cluster.local:8000"
    gen_ai_system: "vllm"
    transport: "http"
    notes: "vLLM inference server on gpu-node-01"
  - match: "api.openai.com"
    gen_ai_system: "openai"
    transport: "http"

Case 3: SSH port-forward or kubectl port-forward to a remote server

If the server is reachable locally via a port forward, match the local address:

  - match: "localhost:8001"
    gen_ai_system: "vllm"
    transport: "http"
    notes: "port-forwarded from gpu-node-01:8000"

Case 4: Site-wide override for a SLURM cluster

Put the registry file on shared filesystem and reference it in the site env file:

# In site-config/local/slurm-hpc/nova-env.sh:
export NOVAFABRIC_URL_REGISTRY=/lustre/scratch/myproject/nova-url-registry.yaml

Source the env file before submitting jobs. The SlurmRunner forwards NOVAFABRIC_URL_REGISTRY to the compute node's environment via the batch script (the job receives NOVAFABRIC_* minus credentials), so the compute node's hooks load the site registry.

Registry file format

The match field is a substring check against the full URL string (scheme, host, port, and path). The first matching entry wins. Order entries from most specific to least specific.

schema_version: "0.1.0"
patterns:
  - match: "host:port/v1"          # more specific path prefix first
    gen_ai_system: "myprovider"
    transport: "http"
    notes: "Optional human-readable note; not emitted in model-calls.jsonl"
  - match: "host:port"             # broader fallback
    gen_ai_system: "myprovider"
    transport: "http"

Calls to URLs that match no registry entry are not recorded. This is intentional — it avoids capturing unrelated HTTP traffic (health checks, metrics endpoints, object storage) that happens to come from the same process.

5. Troubleshooting

model-calls.jsonl is empty or missing

Cause: sitecustomize.py was not on PYTHONPATH when the workload's Python started, so the hooks never installed.

For LocalRunner: This should not happen — the runner creates a temporary directory with sitecustomize.py and injects it via PYTHONPATH before launching the subprocess. If the file is missing, check whether the workload spawns its own subprocess that does not inherit PYTHONPATH.

For SlurmRunner: The capsule directory must be on shared filesystem. If the compute node cannot see capsule_dir, it cannot load sitecustomize.py. Check:

# On the compute node (or via srun):
ls -la /path/to/capsule_dir/sitecustomize.py

If the file is missing, the capsule directory is not on shared filesystem. Move it to a path that all nodes can reach.

For KubernetesRunner: The image must have NovaFabric installed. The runner injects PYTHONPATH pointing at /novafabric/capsule inside the pod. If the image's Python cannot import novafabric, the hook loader will print an error to stderr and skip capture. Check the pod logs:

kubectl logs <pod-name> -n <namespace>

Look for [novafabric] hook install failed: in stderr output.

Capsule directory not visible from compute nodes

Symptom: Job exits with a non-zero code immediately, or runner_status="failed_setup" with a message about the capsule directory.

Cause: The capsule directory is on node-local storage.

Fix: Set NOVAFABRIC_SLURM_SHARED_DIR to a path on shared filesystem before running nova capture. The orchestrator will create capsule directories under that base path.

sbatch submission fails

Check the exit code and stderr returned in RunnerJobResult. Common causes:

which sbatch && sbatch --version

scontrol shows no job after completion

scontrol show job <jobid> returns "Invalid job id" once slurmctld has forgotten the job (typically 5 minutes after completion). This is normal. The SlurmRunner uses sacct as a fallback in this window. If sacct also fails, confirm that slurmdbd is running on the cluster:

systemctl status slurmdbd
sacct -j <jobid> --format=JobID,State,ExitCode --noheader

If neither tool returns data, the runner will time out polling and return runner_status="timeout". Capsule artifacts may still be present on shared filesystem; check the capsule directory directly.

Docker image pull fails

docker run exit code 125 indicates a Docker daemon-level failure, which includes image pull errors. Check:

docker pull myorg/agent-runtime:abc1234

Make sure the image reference is correct and the daemon has pull access (credentials, registry reachability). The DockerRunner only forwards NOVAFABRIC_* env vars into the container, not DOCKER_AUTH_CONFIG or registry credentials. Pre-pull the image before invoking nova capture if you cannot configure pull credentials at the daemon level.

Kubernetes kubectl cp fails

The kubectl cp step runs after the workload exits. If the pod's container has already terminated and been garbage-collected before kubectl cp starts, the copy will fail. This is more likely when:

The capsule does not record the pod name. host.runner.image in capsule.yaml tells you whether the pod was found: an unresolved_reason of the workload pod was not found means it was not. The Job is named nf-<run-id, lower-cased>, so while the pod still exists you can find it and re-run the copy by hand:

kubectl get pods -n <namespace> -l job-name=nf-<run-id-lowercase>
kubectl cp <namespace>/<pod>:/novafabric/capsule/. ~/.novafabric/capsules/<run-id>/

Wire calls to an unlisted endpoint are not recorded

The URL being called does not match any entry in the active URL registry, so the wire hooks skip it: no record is written at all (there is no gen_ai.system=unknown record — each hook records a request only when UrlRegistry.is_known() is true; capture/hooks/_httpx.py, _requests.py, _urllib3.py, _aiohttp.py). The symptom is a model call that is missing from model-calls.jsonl.

Add an entry to your URL registry override (see section 4). Then check the active registry path:

# Which registry file is being loaded?
python3 -c "
from novafabric.capture.hooks._url_registry import load_url_registry, _VENDORED_DEFAULT_PATH
import os
env = os.environ.get('NOVAFABRIC_URL_REGISTRY', '')
user = os.path.expanduser('~/.novafabric/url_registry.yaml')
print('NOVAFABRIC_URL_REGISTRY:', env or '(not set)')
print('user override exists:', os.path.exists(user))
print('vendored default:', _VENDORED_DEFAULT_PATH)
"

5a. Server environment variables

Every value below is read from the environment by nova server. All defaults are the ones the code actually applies, read from a default ServerConfig rather than transcribed. Safety-off switches are not here — they are in SECURITY.md §Escape hatches, deliberately in one place.

Outbound webhooks

Disabled by default. This is the only path that leaves the trust boundary.

Variable Default Effect
NOVAFABRIC_SERVER_WEBHOOKS_ENABLED false Master switch
NOVAFABRIC_SERVER_WEBHOOKS_QUEUE_MAX 1000 Bounded queue; beyond it deliveries are shed rather than buffered without limit
NOVAFABRIC_SERVER_WEBHOOKS_MAX_ATTEMPTS 5 Retries before a delivery is abandoned
NOVAFABRIC_SERVER_WEBHOOKS_TIMEOUT_S 5.0 Per-attempt HTTP timeout
NOVAFABRIC_SERVER_WEBHOOKS_DELIVERY_RETENTION_DAYS 30 Delivery-record age cap
NOVAFABRIC_SERVER_WEBHOOKS_DELIVERY_RETENTION_ROWS 10000 Delivery-record row cap — whichever bites first

Usage metering

Variable Default Effect
NOVAFABRIC_SERVER_USAGE_METERING_ENABLED true Master switch
NOVAFABRIC_SERVER_API_KEYS_ENFORCE_WORKSPACE_BINDING false Experimental (ADR-0294): refuse (403) a workspace-bound API key whose workspace does not exist or whose request names another workspace (quotas §per-workspace)
NOVAFABRIC_SERVER_USAGE_FLUSH_INTERVAL_S 60.0 How often the in-memory accumulator is flushed
NOVAFABRIC_SERVER_USAGE_ACCUMULATOR_MAX_ENTRIES 10000 Bound on the accumulator between flushes
NOVAFABRIC_SERVER_USAGE_ROLLUP_RETENTION_MONTHS 24 Aggregated rollups
NOVAFABRIC_SERVER_USAGE_LEDGER_RETENTION_MONTHS 3 Raw ledger rows — deliberately shorter than the rollups

Step-up re-authentication

Variable Default Effect
NOVAFABRIC_SERVER_STEP_UP_ENABLED false Require re-auth for privileged actions
NOVAFABRIC_SERVER_STEP_UP_MAX_AGE_SECONDS 300 How long a step-up stays valid

Storage and identity

Variable Default Effect
NOVAFABRIC_DB_URL — Not read by nova serve (its /api/health backend_type reports the SQLite registry it actually opens; before 2026-10-23 this variable flipped that label to postgres). It is not a migration target: nova migrate-to-postgres --target and nova db read NOVAFABRIC_POSTGRES_DSN
NOVAFABRIC_API_WORKERS 1 ⚠ Read by SQLiteMetadataStore, which refuses to construct when this is > 1. It is the guard behind "SQLite cannot be shared across writer processes" — see §3 on --workers
NOVAFABRIC_AUDIT_LOG_PATH $XDG_DATA_HOME/novafabric/audit.jsonl when XDG_DATA_HOME is set, else ~/.local/share/novafabric/audit.jsonl (NOVAFABRIC_HOME deliberately does not move it) Hash-chained audit log path. Every writer and reader (exports, promotions, holds, nova audit-log, nova policy explain, nova serve, backup) resolves it the same way, at the moment it is used

Rate limiting

Off by default. Per-class token buckets; the audit keys fire when rejections cross a threshold inside a window, so a burst is logged once rather than per request.

Variable Default Effect
NOVAFABRIC_SERVER_RATE_LIMITS_ENABLED false Master switch. Class defaults when on: ingest 100/s burst 200, read 50/s burst 100, admin 10/s burst 20
NOVAFABRIC_SERVER_RATE_LIMITS_AUDIT_THRESHOLD_REJECTIONS 100 Rejections before an audit event
NOVAFABRIC_SERVER_RATE_LIMITS_AUDIT_WINDOW_SECONDS 60 Window the threshold is counted over

Observability

Variable Default Effect
NOVAFABRIC_SERVER_SELF_TRACING false Emit NovaFabric's own OTel spans
NOVAFABRIC_SERVER_SELF_TRACING_ENDPOINT — OTLP destination for those spans
NOVAFABRIC_SERVER_METRICS_EXEMPT false Exempt /metrics from auth
NOVA_BUILD_SHA — Build SHA reported in observability output
NOVA_COLLECTOR_METRICS_URL http://localhost:9464/metrics Where the dashboard scrapes collector metrics

Cluster collector receiver (collector_app)

Read by novafabric.collector_app:create_collector_app (the Python POST /capsule receiver), not by nova server. Full walkthrough: cluster-scale tutorial.

Variable Default Effect
NOVA_COLLECTOR_TOKEN unset (local mode) Every POST /capsule must send X-Nova-Token: <token>, from any address. Unset: only loopback callers with a localhost Host and no proxy header are served, everyone else gets 403
NOVA_COLLECTOR_TENANT default The tenant the caller is authenticated as; every event is stored under it

Each event in a batch needs a string event_type; a malformed event (no event_type, a field of the wrong type, a token count out of range) refuses the whole batch with 422 naming the field. The full event contract is in the cluster-scale tutorial.

⚠ Set NOVA_COLLECTOR_TOKEN whenever anything proxies to the collector. Local mode refuses a request carrying Forwarded, X-Forwarded-*, X-Real-IP or Via, but it cannot see a same-host reverse proxy that adds none of them — nginx's proxy_pass http://127.0.0.1:8090; with its defaults connects from loopback, sends Host: 127.0.0.1:8090 and no forwarding header. Behind such a proxy every remote caller looks local and is served without authentication. No setting other than a token closes this; the startup warning says so too.

API shape

Variable Default Effect
NOVAFABRIC_SERVER_BULK_MAX_ITEMS 100 Cap on items per bulk request
NOVAFABRIC_SERVER_PAGINATION_LEGACY_OFFSET_CURSORS true Keep accepting legacy offset cursors alongside the current form
NOVAFABRIC_ALLOW_SEALED_DELETE off Experimental (ADR-0206 P2): exactly 1 lets DELETE /v0/capsules/{id}, bulk delete and nova serve's DELETE /api/runs/{id} remove a NovaSeal-sealed capsule instead of refusing with 409 sealed_capsule; never overrides a legal hold or WORM lock. Refusals and removals are audited (run.index_delete_refused, run.index_delete)
NOVA_SERVE_MAX_FILE_BYTES 5000000 nova serve per-file read cap

Object store and paths

Home resolution (works today). Every home-relative path resolves, in order: the path's own override variable (e.g. NOVAFABRIC_DB_PATH, NOVAFABRIC_CAPSULE_DIR, NOVAFABRIC_EVIDENCE_DIR) > $NOVAFABRIC_HOME/<name> > ~/.novafabric/<name>. This covers the registry, capsules, evidence, the keystore (keys/, which the health panel's keystore_ok checks), tokens, the object-store WAL, the dashboard audit log and the NovaSeal config and nonce store. Unset NOVAFABRIC_HOME and the defaults are exactly the old ~/.novafabric locations. Deliberately not under the home: the per-user keyring (~/.config/novafabric; move it with NOVAFABRIC_KEYRING_DIR), the legacy seal stores (~/.local/share/novafabric), and project-local ./.novafabric/ directories — and the hash-chained audit log, which stays at $XDG_DATA_HOME/novafabric/audit.jsonl, or ~/.local/share/novafabric/audit.jsonl when XDG_DATA_HOME is unset, whatever NOVAFABRIC_HOME says: pointing a shell at another data home must not hide the existing trail or start a second hash chain beside it. Move it explicitly with NOVAFABRIC_AUDIT_LOG_PATH (deployments put it on the data volume that way).

Variable Default Effect
NOVA_S3_ACCESS_KEY / NOVA_S3_SECRET_KEY — ⚠ Credentials. Prefer an instance role or a mounted secret; a value here is visible to anything that can read the process environment
NOVA_S3_GOVERNANCE_BUCKET / NOVA_S3_COMPLIANCE_BUCKET — Buckets for the dual object store
NOVA_OBJECT_STORE_ENCRYPTION off unless 1 Enables backup object encryption
NOVA_OBJECT_STORE_ALLOW_PLAINTEXT_READS off Legacy-migration opt-in: an encrypted object store returns non-envelope (pre-encryption) objects instead of refusing them with PlaintextObjectRefusedError (encryption-at-rest §2.1)
NOVA_OBJECT_STORE_LEGACY_INVENTORY — Experimental (ADR-0295): path to a digest-pinned legacy-object inventory; only listed pre-encryption objects whose stored bytes are unchanged are returned (encryption-at-rest §2.1.1)
NOVA_OBJECT_STORE_LEGACY_INVENTORY_SHA256 — Optional SHA-256 pin of the inventory file; a mismatch refuses startup
NOVA_OBJECT_STORE_REFUSE_V1_ENVELOPES off Experimental strict mode (ADR-0295): refuse unbound v1 envelopes not pinned in the inventory (LegacyEnvelopeRefusedError)
NOVAFABRIC_BACKEND sqlite Storage backend selector
NOVAFABRIC_JOBS_DB — Jobs database path override
NOVAFABRIC_KEYRING_DIR ~/.config/novafabric/keyring Promote keyring directory (maker-checker signing keys); resolved when used, never at import
NOVAFABRIC_LOG_DIR — Log directory the support bundle collects from
NOVA_EVIDENCE_DUCKDB_PATH — Evidence DuckDB path

Capture

Variable Default Effect
NOVAFABRIC_CAPTURE_SOCKET — Capture daemon socket path override
NOVAFABRIC_FAST_EMIT off unless 1 Fast-emit mode inside the captured process
NOVAFABRIC_CAPTURE_STRICT off Truthy (1/true/yes/on): refuse a capture that would overlap another in the same process (ConcurrentCaptureRefused) instead of producing a contended capsule (ADR-0224 OQ-2, works today, opt-in)
NOVAFABRIC_OTLP_LOG_DIR $NOVAFABRIC_HOME/otlp-logs Experimental (ADR-0293): sidecar store for POST /api/otlp/v1/logs; outside every capsule, not sealed evidence — back it up separately
NOVAFABRIC_OTLP_LOGS_STORE_BODY off Experimental: also store OTLP log body text and attribute values (secret-redacted, truncated); default stores metadata and a body SHA-256 only

Other

Variable Default Effect
NOVAFABRIC_SERVER_TOKEN — Pin the local auth token instead of letting the server generate one (see §5d)
NOVAFABRIC_WEBHOOKS_QUEUE_MAX 1000 Accepted as an alias of NOVAFABRIC_SERVER_WEBHOOKS_QUEUE_MAX
NOVA_NATS_DUPLICATE_WINDOW_S 120 Lineage consumer de-duplication window
NOVAFABRIC_TRUTHFUL_QA_OCI_IMAGE — TruthfulQA eval suite image reference. Required, no default (no eval image is published; ADR-0308)
NOVAFABRIC_TRUTHFUL_QA_OCI_DIGEST — Pins that image to an exact digest. Required, no default (ADR-0308)

5b. NovaSeal configuration (cryptographic signing)

NovaSeal is the sealing layer beneath the Evidence Bundle: it signs a capsule with an ECDSA P-256 DSSE signature, optionally timestamps it via RFC 3161 (opt-in: set tsa_url, there is no default TSA, ADR-0292), and records an inclusion proof in an append-only Merkle log (the proof is also carried in the capsule's log-entry.json, so nova verify works on an auditor's machine with no config and no log). nova verify checks every layer present and exits 0 only if the capsule is unmodified since signing; a layer that was not requested prints NOT PRESENT rather than failing. This is what turns "here is a capsule" into "here is a capsule I can prove was not tampered with."

NovaSeal is opt-in — if $NOVAFABRIC_HOME/novaseal.yaml (~/.novafabric/novaseal.yaml when NOVAFABRIC_HOME is unset) and the NOVAFABRIC_SEAL_CONFIG environment variable are both absent, capture behaves exactly as before and no sealing occurs. This keeps the default path zero-dependency and air-gap friendly.

For the full configuration reference (all profiles, env vars, Docker/SLURM patterns, path resolution order, troubleshooting) see: docs/novaseal-configuration.md

Quick start — one command (experimental, ADR-0301): nova seal init creates a dedicated P-256 key, a local seal CA and a certificate under $NOVAFABRIC_HOME/keys/novaseal/ and writes a managed novaseal.yaml with no tsa_url (offline). Its identity is self-asserted; nova verify reports it as such. Rotate with nova seal init --force. It never replaces a novaseal.yaml it did not write, so the manual and KMS configurations below are unaffected.

Manual — your own key + minimal config:

mkdir -p ~/.novafabric
openssl ecparam -name prime256v1 -genkey -noout | \
  openssl pkcs8 -topk8 -nocrypt -out ~/.novafabric/seal.key
openssl req -new -x509 -key ~/.novafabric/seal.key -days 3650 \
  -out ~/.novafabric/seal.crt -subj "/CN=NovaSeal-$(hostname)"
# ~/.novafabric/novaseal.yaml
profile: local
key_path: ~/.novafabric/seal.key
cert_path: ~/.novafabric/seal.crt
tsa_url: https://freetsa.org/tsr   # opt-in (ADR-0292): omit for no timestamp / no network call
merkle_db: ~/.novafabric/novaseal-merkle.db

SLURM / shared-FS deployments: set merkle_db: to a path on the shared filesystem. See novaseal-configuration.md §3.3.

Docker deployments: set NOVAFABRIC_SEAL_CONFIG to the mounted config path and configure merkle_db: inside it. See novaseal-configuration.md §3.2.

Verify a sealed capsule:

nova verify /path/to/capsule/
# Five checks over the capsule and its .seal/ directory:
#   • ECDSA P-256 DSSE signature   → signature_ok=True
#   • RFC 3161 timestamp           → timestamp_ok=True (None when no token: absent, not failed);
#                                    timestamp_trust=anchored | unanchored | absent | invalid
#   • Merkle log inclusion proof   → log_integrity_ok=True (plus the sealer's own log when configured)
#   • manifest binding             → capsule.yaml agrees with the signed payload
#   • evidence binding             → every file's sha256 matches evidence_digests
# It also reports the signer trust level (identity_trust=self-asserted | local-ca-pinned |
# ca-anchored) and fails on recorded workload tampering or a link/special file in the capsule.
# Exits 0 if all checks pass, 1 otherwise, 2 on a usage error (an incompatible flag
# combination). --json prints one JSON object.

# Explicit config path (instead of $NOVAFABRIC_HOME/novaseal.yaml):
nova verify --seal-config ~/configs/novaseal.yaml /path/to/capsule/

Network note: The TSA request (tsa_url) requires outbound HTTPS on port 443. On air-gapped clusters, set tsa_url: "" to skip timestamping, or run a private TSA inside the cluster and point tsa_url at it.

5d. Server credentials: which token, and what the server must be told

nova server accepts three credential classes. Picking the wrong one, or configuring only half of one, is the most common first-multi-node-deploy failure (defect B6) — so this section states both halves of each.

Credential Shape Server needs Use for
Local token (ADR-0184) opaque string nothing — generated at startup single-node, loopback
Offline JWT (ADR-0018/0178) three dot-separated segments NOVAFABRIC_OFFLINE_KEY_PATH multi-node, airgapped, Slurm
API key opaque string key store long-lived integrations

The local token — single node

Generated on first start, stored at $NOVAFABRIC_HOME/.server-token (mode 0600).

It is printed only when stderr is a terminal. Under nohup, systemd, Docker, Kubernetes or CI, stderr is a durable, routinely-collected file, so a non-interactive start prints a fingerprint and the path instead — read the token from the file. That is deliberate (ADR-0263): the token used to land in plaintext on line 1 of every server log.

Offline JWTs — multi-node

Issue one on the machine holding the signing key:

nova server issue-token --subject bench@example.com --roles writer,reader --expires-in 30d

--key-path defaults to NOVAFABRIC_OFFLINE_KEY_PATH, else ~/.novafabric/keys/offline-key.pem; a keypair is generated if absent. The token goes to stdout, so TOK=$(nova server issue-token …) captures only the token.

The half that is easy to miss. The server ignores offline JWTs entirely unless it is started with NOVAFABRIC_OFFLINE_KEY_PATH set. Without it a perfectly valid token is rejected — and until this was fixed the 401 said "Invalid local token", sending operators to check the wrong file for a credential they never presented.

NOVAFABRIC_OFFLINE_KEY_PATH=~/.novafabric/keys/offline-key.pub nova server start

Give the server the .pub, not the private key. Verification accepts either — it derives the public key from a private one if no .pub sits beside it — but shipping the signing key to every server defeats the point of offline signing. Issue tokens on one trusted machine; distribute only the public half.

Revoke by token id with nova server revoke-token.

Precedence

API key first, then offline JWT (only when the key path is configured and the bearer is JWT-shaped), then the local token. A JWT-shaped credential is never compared against the local token — that is what makes the 401 able to name the class you actually presented.

5c. SAML 2.0 SSO (server mode — experimental, partial)

Status: experimental (ADR-0138, spec). Config, SP metadata, role mapping, and the assertion validation policy work today. Live SAML login (assertion consumption at the ACS) shipped in v0.73.0 but is off by default and requires an explicit opt-in — see the honest status below. SAML is a server-mode-only concern: local-first mode (nova capture|validate|replay|diff|lineage) never touches it, and pip install novafabric installs zero SAML dependencies.

Enabling the backend

Add an optional saml: block to the server YAML config (~/.config/novafabric/server.yaml or --config). Absent block ⇒ backend disabled ⇒ behavior identical to an OIDC-only deployment.

saml:
  enabled: true
  sp_entity_id: "https://nova.example.org/saml/metadata"
  acs_url: "https://nova.example.org/v0/auth/saml/acs"
  sp_cert_path: /etc/novafabric/saml/sp.crt
  sp_key_path: /etc/novafabric/saml/sp.key        # mode 0600, never logged
  subject_attribute: email                         # optional; default: NameID
  allow_idp_initiated: false                       # default false (more replay-exposed)
  clock_skew_seconds: 120                          # hard cap 300
  experimental_acs_enabled: false                  # default false; see honest status below
  idp:
    entity_id: "http://www.okta.com/exk1abcXYZ"
    sso_url: "https://example.okta.com/app/nova/exk1abcXYZ/sso/saml"
    x509_cert_path: /etc/novafabric/saml/okta.crt
  attribute_role_map:
    attribute: groups
    mapping:
      nova-admins: [admin]
      nova-writers: [writer]
      nova-auditors: [auditor, reader]
    default_roles: [reader]                        # never defaults to admin

The config is a closed schema: unknown keys, roles outside the six existing ADR-0018/ADR-0058 roles (reader,writer,admin,auditor,promoter,approver), and clock_skew_seconds > 300 are rejected at load time. Unmapped attribute values fail closed to default_roles (default []) — never to admin. If attribute_role_map is absent, roles come from the existing local role-assignment table (nova server assign-role), same as the OIDC fallback — which is consulted only once the workspace/org model is in use (ADR-0178); assign-role warns when a row is not in force.

Registering NovaFabric with the IdP

nova server saml-metadata --config /etc/novafabric/server.yaml > sp-metadata.xml
# or, from a running server:
curl https://nova.example.org/v0/auth/saml/metadata

Hand sp-metadata.xml to the IdP administrator. When sp_cert_path points at a readable PEM certificate, the metadata embeds it as the SP signing key.

Honest status — what works and what refuses

Piece Status
server.saml config block (closed schema, role allow-list, skew cap) works today
nova server saml-metadata + GET /v0/auth/saml/metadata works today
Attribute→role mapping and assertion validation policy (issuer, audience, time bounds, recipient, replay store, InResponseTo, status — spec rules V3–V9, V11) implemented and tested against synthetic assertions
GET /v0/auth/saml/login, POST /v0/auth/saml/acs — assertion verification experimental, opt-in. Refuses with HTTP 501 (saml_not_available) unless saml.experimental_acs_enabled: true and the novafabric[saml] extra is installed. Opted in, the ACS verifies the assertion (401 on any failed rule)
SAML login session planned — no credential exists yet: a verified assertion is answered HTTP 501 saml_session_unavailable (details.assertion_verified: true), not a token. Until then authenticate with OIDC, an API key or an offline token
Single Logout (SLO) future design (ADR-0138 P4)

Why it is off by default: XML-DSIG signature verification (spec rules V1/V2) and the XXE-hardened parser (V10) require a SAML library, and ADR-0138 §D5 held that library as a pre-adoption gate. The gate closed in v0.73.0: signxml (Apache-2.0) + lxml (BSD) passed the ADR-0024 license audit as Tier A, with no native xmlsec/libxml2 bundle to audit. Hand-rolling XML signature validation remains forbidden, and NovaFabric still refuses to consume assertions rather than skip signature validation — without the opt-in (or with the library missing) the ACS never parses the posted XML.

Enabling it is a deliberate act:

pip install 'novafabric[saml]'    # signxml + lxml
# then set saml.experimental_acs_enabled: true in the server YAML

Security-Architect review is a pre-production blocking condition for this path. Setting experimental_acs_enabled: true acknowledges the pre-review status; do not enable it in production before that review.

If your IdP estate can speak OIDC, use the shipped OIDC backend (ADR-0018) today; an OIDC bridge (Keycloak/Dex fronting the SAML IdP) remains a documented interim option.

6. What is not supported yet

Corrected 2026-07-30 — this table had gone stale. Several rows below had shipped since this section was last updated (some as far back as the v0.7x/v0.9x releases) but the table still said "planned" / "future design" against a "v0.2" target. Re-verified against the code in this checkout; still-genuine gaps are listed below the shipped-items note.

Shipped since this table was last accurate (do not re-plan these):

Capability Real status Where
NovaSeal: Sigstore keyless signing experimental, works today --backend sigstore (§5, §6 troubleshooting above); needs novafabric[sigstore] extra + network to Fulcio/Rekor — not usable air-gapped (air-gapped guide)
NovaSeal: Cloud KMS (AWS KMS, Azure KV, GCP KMS) experimental, works today Signing backends AwsKmsSigningBackend/AzureKvSigningBackend/GcpKmsSigningBackend (novafabric[seal-aws|seal-azure|seal-gcp]); the parallel envelope-encryption wrapping backends (AwsKmsWrappingBackend etc., ADR-0185) are separately shipped — see encryption-at-rest.md §4. Verified against unit tests and in-memory SDK fakes; end-to-end verification against live cloud credentials is the one piece still outstanding
NovaSeal: Postgres Merkle log experimental, works today PostgresMerkleLog (trust/novaseal/merkle.py, novafabric[seal-postgres]); pass a postgresql:// DSN as merkle_db: (a libpq host=… password=… keyword string is refused, never opened as a file name). verify_consistency() on this backend is a sampled check, not full re-hash
Parent/child capsule relationships implemented, tested src/novafabric/capsule/{tree_assembler,env_contract,orphan,edge_typer,writer,schema}.py + schemas/parent_child_capsule_v1.schema.json (257 passing tests in tests/capsule/)
SAML SSO: live login (assertion consumption at the ACS) experimental, opt-in The ADR-0138 §D5 library gate closed in v0.73.0 (signxml + lxml, Tier A). POST /v0/auth/saml/acs consumes assertions when saml.experimental_acs_enabled: true and novafabric[saml] is installed; otherwise it still refuses with 501. Security-Architect review remains a pre-production blocking condition — see §5c. A verified assertion is answered 501 saml_session_unavailable: no SAML login session exists yet

Genuinely still not implemented:

Capability Status Target
NovaSeal: X.509 HSM / PKCS#11 (hardware-backed key custody) planned unscheduled. Note this is distinct from the already-shipped x509 profile (ADR-0055, trust/novaseal/x509_identity.py) — that ships a certificate-pinned signing identity with the key as a local PEM file, not hardware-backed custody
SAML SSO: Single Logout (SLO) future design (ADR-0138 P4) — the slo: config block validates, but no SLO endpoint exists unscheduled
Federation across clusters future design (ADR-0021 §federation-topology) unscheduled (the "v0.9+" target in earlier drafts of this table has passed without this shipping)
interLink K8s-to-SLURM integration proposed (ADR-0028, no code) unscheduled
Third-party runner plugins (AWS Batch, Modal, etc.) planned (ADR-0025 §B) unscheduled
Registry merge semantics (override + vendored combined) planned unscheduled

Do not configure production workflows that depend on any of the above. If you need one of these capabilities, open an issue or follow the RFC process described in the RFC process.

7. Docker Compose deployment (nova serve + dashboard)

The bundled deploy/docker/docker-compose.yml runs the full dev stack (Postgres + nova-serve) in a single command. This section documents every data-path decision you must get right for a correct deployment; mistakes here result in silent empty tabs in the dashboard rather than error messages.

7.1 Stack profiles

Profile Services started Command
(none — default) postgres + nova make dev-up
prod + ClickHouse + NATS + Kafka + PgBouncer + JanusGraph make prod-up

Both profiles are defined in deploy/docker/docker-compose.yml. The nova service depends on postgres being healthy before it starts.

make update note: rebuilds the nova image and does a rolling restart with --no-deps, so it intentionally does not restart Postgres. If Postgres is not already running (e.g. first deploy after a reboot), run make dev-up instead, or the nova container will wait forever for the DB.

7.2 Volume mounts

The nova service uses three mounts. All host paths default to ~/novafabric-data/ and can be overridden with NOVA_DATA_DIR.

Container path Default host path Purpose
/data/capsules $NOVA_DATA_DIR/capsules Run capsule directories (written by nova capture, read by all tabs)
/data/nova $NOVA_DATA_DIR/nova NOVAFABRIC_HOME inside the container — registry DB, serve token, audit log

Ownership (non-root image): the container runs as nova (uid/gid 1000 — the same uid the Helm chart pins in podSecurityContext). Docker creates a missing bind-mount host path as root, and a root-owned /data/* surfaces as a loud permission error on the container's first write. Pre-create $NOVA_DATA_DIR as a user-owned directory (the default ~/novafabric-data already is), or chown -R 1000 a path that was created by an older, root deployment. | /data/kuzu | Docker named volume kuzu-data | KuzuDB knowledge graph store |

All testbench workloads must write capsules to the host path that maps to /data/capsules. Do not use a relative capsules/ path or a symlink inside the testbench; set NOVAFABRIC_HOME (or NOVAFABRIC_CAPSULE_DIR) explicitly. See deploy/docker/docker-compose.yml for the exact mount syntax.

7.3 Data-path environment variables

The following variables control where the nova process inside the container reads and writes data. Wrong values cause dashboard tabs to show empty results without any error.

Variable Value in Docker Default outside Docker Effect
NOVAFABRIC_HOME /data/nova ~/.novafabric Registry DB, serve token, audit log root
NOVAFABRIC_EVIDENCE_DIR /data/capsules/evidence $NOVAFABRIC_HOME/evidence (~/.novafabric/evidence) Where nova export-evidence writes ZIPs; where the Evidence tab reads them
NOVAFABRIC_SERVE_PATH_ROOTS (unset) (unset) Extra absolute directories (:-separated) that nova serve may read and write when a request names a path. Without it, writes are confined to $NOVAFABRIC_HOME/exports, $NOVAFABRIC_HOME/evidence and $NOVAFABRIC_EVIDENCE_DIR, and reads also to $NOVAFABRIC_HOME and the capsule dir. Reads never reach the server's secrets (token stores, .env, dek.db, ratchet state, config.yaml, audit logs, and for source paths key material), even under a listed root. Replaces NOVAFABRIC_SERVE_ALLOW_ANY_PATH, which is no longer honoured
NOVAFABRIC_METADATA_BACKEND postgres sqlite MetadataStore backend
NOVAFABRIC_METADATA_DSN postgresql://nova:nova@postgres:5432/nova (SQLite file) DB connection string
NOVAFABRIC_SERVER_BACKEND postgres sqlite Server-mode (nova server start) backend selector: sqlite or postgres (overrides backend: in server.yaml)
NOVAFABRIC_POSTGRES_DSN postgresql://nova:nova@postgres:5432/nova — The Postgres DSN for server mode (when the backend is postgres), nova db and nova migrate-to-postgres, and the fallback DSN for nova backup create --profile pg and nova restore of a pg set when NOVA_DSN is unset
NOVA_WATCHER_BACKEND auto auto CapsuleWatcher backend: auto (watchdog if installed, else polling), polling, watchdog. Any other value stops nova serve at startup (exit 2). An explicit nova ingest-capsule --backend wins.
NOVA_WATCHER_INTERVAL 2.0 2.0 Poll interval in seconds for PollingBackend and nova ingest-capsule --watch; must be a finite number > 0. An explicit --interval wins.

Why NOVAFABRIC_EVIDENCE_DIR matters: Without it the serve app uses $NOVAFABRIC_HOME/evidence (_paths.evidence_dir) — in the compose stack /data/nova/evidence, which is on the persistent volume, but is not where the testbench export stage writes. The compose file sets /data/capsules/evidence (inside the capsule volume) so the Evidence tab and the testbench export stage share one directory.

7.4 Fresh deployment checklist

When deploying to a new host for the first time:

# 1. Clone and pull
git clone https://github.com/MSKazemi/novafabric.git
cd novafabric
git pull

# 2. Create the data directories on the host
mkdir -p ~/novafabric-data/capsules ~/novafabric-data/nova

# 3. Start the full stack (builds image, starts postgres + nova-serve)
make dev-up
# prints: Dashboard: http://localhost:4321/dashboard?token=<token>

# 4. (Optional) SSH tunnel if running on a remote server
#    Run on your laptop:
ssh -N -L 4321:localhost:4321 <user>@<server> &
open http://localhost:4321/dashboard?token=<token>

If the dashboard opens but tabs are empty, check:

  1. Runs tab empty — no capsules in ~/novafabric-data/capsules/ on the host, or the runs_cache index is stale. Force a full re-index:
    docker compose exec nova uv run nova ingest-capsule --all
# or from the host:
nova ingest-capsule --all --capsule-dir ~/novafabric-data/capsules
    nova serve also re-indexes on startup and every 2 s via CapsuleWatcher (NOVA_WATCHER_BACKEND=auto, NOVA_WATCHER_INTERVAL=2.0).
  2. Evidence tab empty — NOVAFABRIC_EVIDENCE_DIR not set or pointing at the wrong path. Verify with: docker compose exec nova sh -c 'echo $NOVAFABRIC_EVIDENCE_DIR'
  3. Container stuck at "waiting for postgres" — Postgres not running. Run make dev-up (not make update) to bring up the full stack.

7.5 Remote access via SSH tunnel (make nova-dashboard)

The lpt/ orchestrator project includes a nova-dashboard Makefile target that automates the SSH tunnel + token fetch in one command:

# From ~/scratch/lpt/ on your laptop:
make nova-dashboard
# → ensures postgres + nova are up on the server
# → fetches the live token
# → opens an SSH tunnel on localhost:4321
# → prints the dashboard URL
# → keeps tunnel open until Ctrl-C

The target uses docker compose exec nova (service name, not container name) to read the token — this is robust to container renames across deploys.

7.6 Deployment mode: dashboard or server (NOVA_MODE)

Status: works today. The bundled entrypoint runs one of two processes, selected by NOVA_MODE:

NOVA_MODE Runs Auth
dashboard (default) nova serve --experimental --insecure --topology --tv5 — the experimental dashboard over HTTP with a printed token Session token only; put TLS in front of it and never expose it directly
server nova server start --backend postgres — the multi-user REST API On by default (OIDC via NOVAFABRIC_SERVER_*, else a generated local bearer token). The entrypoint never passes --insecure-no-auth

Both modes run the Alembic migration first (nova db upgrade --backend postgres --revision ${NOVA_DB_REVISION:-head}) and both expose /livez, which is what the Compose healthcheck probes.

Variable Default Effect
NOVA_MODE dashboard Selects the process above
NOVA_PORT 4321 (dashboard) / 7433 (server) Listen port. The bundled docker-compose.yml publishes only 127.0.0.1:4321, so with NOVA_MODE=server also set NOVA_PORT=4321 or add your own port mapping
NOVA_WORKERS 1 nova server start --workers N — uvicorn worker processes (server mode only; needs Postgres)
NOVA_DB_REVISION head Alembic revision to upgrade to. The server refuses to start when stamped behind head, so pinning below head breaks a server-mode deploy

The Helm chart exposes the same switch as mode: dashboard | server with server.workers (see deploy/helm/novafabric/values.yaml). Unlike the Compose entrypoint, the chart never adds --insecure on its own: mode: dashboard renders only with serve.insecure: true plus serve.acknowledgeInsecureExposure (ADR-0230), and helm template refuses otherwise rather than produce a pod that nova serve would refuse to start.

8. Summary and next steps

You now have the operational picture for deploying NovaFabric across every runner:

Suggested first-run checklist

  1. pip install novafabric (or uv pip install novafabric) on Python 3.12+.
  2. Capture a local run: nova capture python my_agent.py.
  3. Validate it: nova validate ~/.novafabric/capsules/<run-id>/.
  4. Confirm wire capture fired: check that model-calls.jsonl is non-empty; if not, see §5 "model-calls.jsonl is empty or missing".
  5. For SLURM/K8s, verify the compute-node environment can import novafabric and that the capsule directory is on shared storage before scaling out.

Where to go next

If you want to… See
Look up any command, flag, or default docs/cli-reference.md
Configure signing in depth (profiles, KMS, air-gap) docs/novaseal-configuration.md
Understand SLURM validation history v0.6.12 release notes
Contribute code or run the test suite CONTRIBUTING.md
Request an unsupported capability (§6) RFC process

For anything listed in §6 "What is not supported yet," do not build production workflows against it — open an issue or follow the RFC process instead.