NovaFabric User Guide

This guide is the working reference for the shipped nova command surface: what each command does, how to invoke it, and when to reach for it. It picks up where the getting started guide leaves off — that guide walks you from install to your first capsule; this one is what you return to once you are capturing, replaying, diffing, and auditing real runs.

What you will learn

NovaFabric is local-first: every command below runs entirely inside your own infrastructure — laptop to cluster, online or air-gapped, with no accounts and no telemetry. The five primitives it is built on — Asset Registry, Run Capsule, Replay, Lineage, and Evidence Bundle — map directly onto the workflow areas in this guide.

The guide is organized by workflow area, not alphabetically. Use the table of contents to jump to what you need.

Both nova and novafabric refer to the same binary. All examples use nova.

Maturity. Nearly all commands documented here ship today but carry experimental maturity: they work, but on-disk formats and interface details may change before the v1.0 schema freeze. Anything labeled planned or future design is documented intent only and is not implemented — do not build on it. See ROADMAP.md for sequencing, and What shipped experimental in v0.59 for the newest command groups.

Table of contents

  1. Capture
  2. Inspect and validate
  3. Replay and diff
  4. Lineage graph
  5. Trust layer
  6. Non-Python client capture
  7. Asset registry
  8. Local dashboard
  9. Environment variables
  10. What shipped experimental in v0.59
  11. What shipped in v0.75–v0.94
  12. Summary and next steps

Capture

Capture is the entry point to everything else. It turns a command — a script, an agent, an HPC training run, a notebook cell — into a Run Capsule: a directory identified by a time-sortable ULID that holds every observable fact of one execution. A capsule is the unit you later replay, diff, trace, and export.

nova capture

The core capture command. Wraps any command and records its execution as a replayable Run Capsule. No changes to your code are required.

nova capture python train.py --lr 0.001
nova capture python my_agent.py
nova capture -- python -c "import sys; sys.exit(1)"

Capsules are written to $NOVAFABRIC_HOME/capsules/<ulid>/ by default (~/.novafabric/capsules/<ulid>/ when NOVAFABRIC_HOME is unset; $NOVAFABRIC_CAPSULE_DIR overrides the base). Use --output-dir to change the base directory for one run:

nova capture --output-dir /mnt/experiment-runs python train.py

Use -- to separate nova capture flags from commands that contain their own flags:

nova capture -- python script.py -v --config cfg.yaml

NovaFabric's exit code mirrors the wrapped command's exit code. A command that exits 1 will cause nova capture to exit 1 — but the capsule is still written on failure, with status: failure, an error block, and the full artifacts. Capture never silently drops a run: success and failure are both evidence.

On success:

✓ Capsule written: /home/me/.novafabric/capsules/01HXAY7M5JZ8R7K4P9DPBYK2WX  (run_id=01HXAY7M5JZ8R7K4P9DPBYK2WX)

What gets captured. The following transports are hooked automatically via a sitecustomize.py injected over PYTHONPATH — no SDK decorator, no import in your code. If the library is not installed, its hook is silently skipped, and a capsule is still written even when no AI SDK is present at all:

Transport What is recorded
openai SDK chat.completions.create calls → model-calls.jsonl; since v0.105.0 also responses.create, async and streamed calls
anthropic SDK messages.create calls → model-calls.jsonl; since v0.105.0 also async and streamed calls
httpx Requests to URL-registry-classified hosts → model-calls.jsonl and network_events.jsonl
httpx2 Same as httpx, for the client openai 3.x uses by default; patched only when httpx2 is installed (works today)
requests Same classification as httpx; covers LangChain, LlamaIndex, boto3
aiohttp Async wire-level capture; covers async LangChain, FastAPI agents
urllib3 Lowest-tier wire-level capture; covers boto3/Bedrock
mcp SDK ClientSession.call_tool invocations → tool-calls.jsonl
Plugins Any class in the novafabric.hooks entry-point group (experimental)

The requests and urllib3 hooks use a shared layering guard so a single requests.post() produces exactly one record even though requests calls urllib3 internally.

OTel GenAI semantic conventions. Every model-call record follows the OpenTelemetry GenAI semantic conventions. All "Required when applicable" fields are extracted: temperature, top_p, top_k, max_tokens, stop_sequences, seed, frequency_penalty, presence_penalty, response.id, and finish_reasons. The temperature, top_p, and seed fields are what make a later exact replay eligibility check possible (see nova replay).

Export the run as portable OTel spans (experimental, NF-032/033). Because those semconv attributes are already stored, --emit-otel-genai maps the finished capsule outward to OTel GenAI gen_ai.* spans (a root invoke_agent span, a chat client span per model call, an execute_tool span per tool call) written to <capsule>/otel-genai-spans.json. Message content is off by default (ADR-0021); add --capture-content to include request messages, routed through the same secret-redaction gate as capture and size-bounded. A model/tool call that recorded a log_level (ADR-0127) also carries its OTel SeverityNumber projection as novafabric.severity_number / novafabric.severity_text (debug→5, info→9, warn→13, error→17; experimental); a call without a level gets neither attribute — no severity is fabricated. The OTLP ingest endpoint (POST /api/otlp/v1/traces) reads the pair back (experimental, ADR-0127 P4 inbound): an exported span re-imported into a capsule keeps its log_level (recorded with log_level_source: adapter; an ERROR span status still wins when more severe). A span event's OTel SeverityNumber is consumed the same way (TRACE collapses to debug, FATAL to error); malformed values are ignored, never guessed. Emit with:

nova capture --emit-otel-genai python my_agent.py

Standalone OTLP logs (resourceLogs) are ingested by POST /api/otlp/v1/logs (experimental, ADR-0293; OTLP/JSON, or OTLP/protobuf with the otlp extra). They are not written into any capsule — a sealed capsule is never amended — but appended to a separate sidecar store, $NOVAFABRIC_OTLP_LOG_DIR (default $NOVAFABRIC_HOME/otlp-logs), one JSONL stream per run (novafabric.run_id attribute), else per trace id, else per UTC day. Each record notes whether its run's capsule was sealed, unsealed or absent when the log arrived. The sidecar is correlation data, not sealed evidence. By default only metadata is kept: times, severity (mapped to log_level exactly as above), ids, service.name, attribute keys, and the body's type, length and SHA-256. Set NOVAFABRIC_OTLP_LOGS_STORE_BODY=1 to also keep body text and attribute values, redacted with the capture secret rules and truncated. Bounds: 16 MiB per request, 10 000 records per request, 64 MiB per stream file; records over the file cap come back as OTLP partialSuccess.rejectedLogRecords.

Example (the ingest function the route calls, run against a temporary directory; a record carrying a novafabric.run_id resource attribute lands in that run's stream, metadata only):

runs/01RUNEXAMPLE00000000000001.jsonl
{"schema": "novafabric/otlp-log-record/v0", "run_id": "01RUNEXAMPLE00000000000001",
 "service_name": "agent-svc", "severity_number": 17, "log_level": "error",
 "time": "2026-09-21T14:13:20.000000Z", "attribute_keys": [],
 "body": {"type": "string", "bytes": 17, "sha256": "b84afc0d..."}, ...}

The body text itself is absent (only its length and SHA-256) until you set NOVAFABRIC_OTLP_LOGS_STORE_BODY=1. Operator settings are in the operator guide; the route is in the API reference.

Secrets are redacted before the capsule is finalized. A secret scanner rewrites matches in place as [REDACTED:rule-id] and writes redaction-proof.json before the capsule closes. It scans every file in the capsule: the call and event streams, env.lock, assets.jsonl, lineage.jsonl, everything under inputs/ and outputs/, and the manifest. A final pass rescans the finished capsule before it is digested and sealed. The rule pack matches known key formats (LLM providers, vector stores, AWS, GitHub, NovaFabric's own keys); a secret in a format it does not know is not detected — for example PEM private keys, JWTs, passwords and connection-string credentials. The full list is in what the scanner does not detect, so read a clean proof as "scanned", not "secret-free". A capsule that lacks this proof is invalid to nova validate and cannot be exported as an Evidence Bundle — redaction is a precondition of trust, not an afterthought.

SDK decorator (in-process capture)

Use @agent when you own the entry point and want to capture without a subprocess wrapper. The decorator installs capture hooks before calling the function and removes them afterward — even if the function raises.

from novafabric.sdk.agent import agent

@agent(name="research-agent", version="0.1.0", capsule_dir="capsules/")
def run():
    # openai, anthropic, httpx calls made here are auto-captured
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": "Hello"}],
    )
    return response

run()

The capsule structure is the same as nova capture's, with one difference: the decorator runs inside your process, so the capsule records capture_mode: sdk-decorator and a label (@agent:<name>@<version>) instead of a command. nova replay --mode mocked therefore refuses it (CapsuleNotReplayable, exit 1); forensic and semantic replay, nova diff and nova validate work as usual. capsule_dir is a capsule store, like nova capture --output-dir: every call writes its own capsule to capsules/<run_id>/. Without capsule_dir, the decorator emits OTel spans only — no capsule is written. This is the original v0.1 observability mode and is still useful if you only need traces.

Same finalization as nova capture (since v0.105.0). The capsule goes through the same secret scan, manifest redaction, residual pass, lineage.jsonl and evidence_digests, and it is sealed when a signing profile exists (nova seal init), so nova verify checks it like any other capsule. Sealing stays opt-in. If finalization fails, your function's return value or exception is unchanged: the capsule is left unsealed and metadata.finalization_error says why. The same holds when an earlier capture step fails (for example, a full disk while writing env.lock): the reason reads capture incomplete: env.lock: OSError. The framework adapters behave the same way when the capsule cannot even be opened (an unwritable data_dir): the call runs uncaptured and the failure is logged.

Optional record-only tags (experimental, v0.59). The decorator also accepts deployment_environment= (ADR-0126), variant= (ADR-0116, A/B attribution), and session_id=/session_sequence= (ADR-0122, multi-turn sessions). Each is recorded verbatim as an additive optional manifest field, never inferred; the matching CLI flags and NOVAFABRIC_* env vars take precedence. See Python API: SDK decorator.

See examples/minimal-agent-run/agent.py for a complete working example.

When to use which. Prefer nova capture for anything you can invoke as a command — it needs zero code changes and works for non-Python subprocesses. Reach for the @agent decorator only when you own the Python entry point and want capture to live inside the process (for example, inside a long-running service where spawning a subprocess is awkward).

Runners

nova capture delegates subprocess execution to a runner. The runner abstraction is what lets the same capture command target a laptop, a container, a Kubernetes cluster, or a SLURM batch queue without changing your workload. Choose one with --runner:

nova capture --runner local python agent.py    # default: runs on this machine
nova capture --runner docker \
  --runner-option image=myorg/agent-runtime:latest \
  python agent.py
nova capture --runner kubernetes \
  --runner-option image=myorg/agent-runtime:latest \
  --runner-option namespace=ml-jobs \
  python agent.py
nova capture --runner slurm \
  --runner-option partition=gpu \
  python train.py

One thing does change with the workload: its environment. --runner local runs as you, on your machine, and keeps your full environment. Every other runner sends the workload somewhere else, so it receives only NOVAFABRIC_* variables — plus PATH for slurm, and anything you name explicitly in extra_env for docker, kubernetes, slurm and lsf:

nova capture --runner kubernetes \
  --runner-option image=myorg/agent-runtime:latest \
  --runner-option namespace=ml-jobs \
  --runner-option extra_env=LOG_LEVEL=debug \
  python agent.py

Credentials do not belong in extra_env. A value typed on the command line is in nova capture's argv, readable with ps by every user on the host for the whole run; it is then stored in the Kubernetes Job object, in docker inspect, or with the SLURM/LSF job script (which slurm and lsf pipe to the scheduler on stdin, never on a command line). On Kubernetes use a Secret bound to the pod's service_account. See operator-guide.md §3.3–3.4, ADR-0270 and ADR-0311.

Pass runner-specific options with --runner-option key=value (repeatable). All runners are non-privileged by design — see the enforcement notes per runner.

What the capsule records about where the run happened (experimental, since v0.105.0, ADR-0307). capsule.yaml names the runner under host.runner.name. For docker and kubernetes it also records the image you passed and the digest the runtime actually resolved it to:

host:
  runner:
    name: docker
    image:
      reference: python:3.12-slim
      image_id: sha256:…
      repo_digests: [docker.io/library/python@sha256:…]
      resolved_by: docker-image-inspect

When no digest can be resolved, image carries unresolved_reason instead; a digest is never inferred from the tag. When nova capture runs inside a Slurm job, or you use --runner slurm, host.slurm records the job id (plus array ids, partition, cluster and node count when Slurm sets them). Node names are recorded only as a hash, the same way the hostname is. The environment lock still describes the machine that ran nova, not the container. Every field, and what is not recorded: run-capsule.md.

Capturing a framework agent without nova capture

A runner captures a command. If your agent is a framework object inside a Python process, wrap it instead — each invocation writes its own Run Capsule, and the wire-level hooks record the model calls the framework makes.

from novafabric.adapters import wrap_llamaindex, wrap_pydantic_ai, wrap_haystack

engine = wrap_llamaindex(index.as_query_engine())   # LlamaIndex
agent  = wrap_pydantic_ai(agent, run_name="support-bot")  # Pydantic AI
pipe   = wrap_haystack(pipe, run_name="rag-qa")     # Haystack

Each wrapper patches the entry point in place and returns the same object, so existing references keep working (LangGraph's wrap_langgraph is the exception: it returns a wrapper, so use the returned object). Streaming calls are captured too (experimental): LangGraph stream / astream, LlamaIndex stream_chat / astream_chat and streaming=True query engines, Pydantic AI run_stream / iter. The capsule stays open until the stream ends and records status: partial (with metadata.partial_reason) when a stream is abandoned or cancelled before it finishes — see cli-reference.md §Framework Adapters for exactly what each case records. Install the framework itself (pip install llama-index / pydantic-ai / haystack-ai) — these are not NovaFabric extras, and the wrap_* call raises ImportError naming the command if the framework is absent.

Adapters ship for eleven frameworks. Each is importable by name from novafabric.adapters:

Framework Entry point Wraps
LangGraph wrap_langgraph(graph) a compiled graph
AutoGen wrap_autogen(agent) an agent (initiate_chat, a_initiate_chat)
CrewAI wrap_crewai(crew) a crew (kickoff, kickoff_async)
DSPy wrap_dspy(program) a program (forward / __call__, aforward / acall)
LlamaIndex wrap_llamaindex(engine) a query/chat engine or agent
Pydantic AI wrap_pydantic_ai(agent) an Agent
Haystack wrap_haystack(pipeline) a Pipeline
OpenAI Agents SDK register_openai_agents() registers a trace processor
Google ADK make_google_adk_plugin() returns a plugin
AWS Bedrock AgentCore wrap_bedrock_agentcore(client) a boto3 bedrock-agent-runtime client (returns a wrapped client)
A2A SDK make_a2a_interceptor() returns an interceptor

The OpenAI Agents SDK, Google ADK and A2A entry points are not wrappers — they hook the framework's own extension point (ADR-0078) rather than patching a method. A wrapper patches the listed methods on the object you pass and returns the same object; the patched methods take exactly the arguments the framework's do, positional ones included, and each call (sync or async) is one capsule — a captured call that makes another, such as kickoff_async running kickoff, records into the outer capsule.

Evidence and sealing (since v0.105.0). Every adapter finalizes its capsule through the same path as nova capture: secret scan, manifest redaction, residual pass, lineage.jsonl, evidence_digests, and a seal when a signing profile exists (opt-in), so nova verify works on an adapter capsule. Configured maskers (nova capture --masker) do not run in-process. A finalization failure never breaks the framework call: the capsule is left unsealed and metadata.finalization_error records why.

Where adapter capsules land, and how to refer to them. An adapter writes to the data_dir you pass it; with none, to $NOVAFABRIC_HOME/runs/<run-id>/ when NOVAFABRIC_HOME is set, otherwise to ./.novafabric/runs/<run-id>/ under the working directory. That is not where nova capture writes ($NOVAFABRIC_CAPSULE_DIR, default ~/.novafabric/capsules/). A capsule path always works (nova replay .novafabric/runs/<run-id>/ --mode forensic). Since v0.105.0 (v0.104.0 resolved a bare id in the capture store only) the bare run id works too: nova replay, nova diff, nova validate and every other command that takes a run id also look in those two adapter defaults, print a note: on stderr naming the directory the capsule was found in, and fail with an error listing both paths if the same id exists in two of them. Passing --capsule-dir to a command that has it searches only that directory.

Replaying an adapter capsule. An adapter capsule is captured inside a framework call, so it records capture_mode: sdk-decorator and a @framework:name label, not a command. nova replay --mode forensic and --mode semantic work on it; --mode mocked refuses it up front (CapsuleNotReplayable, exit 1) because there is nothing to re-run; --mode exact reports it not eligible. Re-running a framework agent from its capsule is future design (ADR-0306, open question 7). To get a mocked-replayable capsule, capture the script that runs the agent with nova capture -- python agent.py. Full reference, including each adapter's entry-point and capture-mode details, is in cli-reference.md §Framework Adapters.

Google ADK tools in a mocked replay (experimental, since v0.106.0). For an ADK agent captured with nova capture -- python agent.py, add novafabric.adapters.google_adk.make_tool_plugin() as the first plugin of the Runner. Under capture each ADK tool call is recorded (payloads at NOVA_CAPTURE_LEVEL=forensic); under nova replay the recorded result is returned through ADK's documented before_tool_callback short-circuit and the tool body never runs. Unmatched or unservable calls are refused (fail closed), --permissive and replay.yaml tool_overrides work as for MCP and record.tool, and the mutation class that gates them is declared in make_tool_plugin({...}). It does not make capsules from make_google_adk_plugin() replayable — those keep a pseudo-command (ADR-0306 open question 7). Details: cli-reference.md §Google ADK adapter and architecture/replay-modes.md.

OpenAI Agents SDK tools in a mocked replay (experimental, since v0.106.0). For an Agents SDK workload captured with nova capture -- python agent.py, pass the agent's tools through novafabric.adapters.openai_agents.wrap_tools([...], {"tool": "read-only"}). Under capture each function-tool call is recorded (payloads at NOVA_CAPTURE_LEVEL=forensic); under nova replay the recorded result is returned and the tool body never runs. Unmatched or unservable calls are refused (fail closed), and the mutation class that gates --permissive and replay.yaml allow: true is declared in wrap_tools. Limits: the Agents SDK has no documented way to skip a tool, so this replaces FunctionTool.on_invoke_tool — an undocumented seam, checked against openai-agents 0.17.x only; tools you do not pass through wrap_tools still run live; and capsules from register() stay non-replayable (pseudo-command). Details: cli-reference.md §Agents SDK tool seam.

local (default). Runs the workload as a local subprocess. Equivalent to the pre-v0.6 behavior. No additional setup required.

docker. Runs the workload inside a container. Required option: image. The image must already have NovaFabric installed. The capsule directory is mounted as a volume at /novafabric/capsule; artifacts land on the host filesystem.

nova capture --runner docker \
  --runner-option image=myorg/agent-runtime:abc1234 \
  --runner-option network=bridge \
  --runner-option user=1000:1000 \
  python my_agent.py

Optional options: network, workdir, user, extra_volumes, extra_env. The runner enforces no --privileged, no docker-socket mount, no host namespace defaults.

kubernetes. Runs the workload as a Kubernetes Job via kubectl. Required options: image, namespace. Optional: service_account, node_selector, resources. Capsule artifacts are pulled back via kubectl cp after completion. The job manifest enforces privileged: false, hostNetwork: false, backoffLimit: 0 — no root, no host namespace, no silent retries. A failed status poll (an API-server blip, a throttled kubectl, an expired token) is retried until the timeout; the count and last error are recorded as runner_metadata.poll_errors and last_poll_error. Releases up to v0.107.0 treated one failed poll as the end: they deleted the still-running Job and reported a timeout.

slurm. Runs the workload as a SLURM batch job via sbatch. Required option: partition. Optional: account, qos, time, nodes, gres, mem, constraint. The capsule directory must be on a shared filesystem (NFS, Lustre, GPFS) visible from every compute node — the runner trusts the shared FS rather than rsyncing artifacts. This is what makes the shared-filesystem registry usable as a laptop↔cluster handoff protocol.

export NOVAFABRIC_SLURM_SHARED_DIR=/home/vagrant  # shared path
nova capture --runner slurm \
  --runner-option partition=gpu \
  --output-dir $NOVAFABRIC_SLURM_SHARED_DIR/runs \
  python train.py

Prototype. Multi-node distributed runs recorded as a parent/child capsule tree under a single global_run_id shipped as a prototype in v0.10+ (ADR-0039; nova run new-run-id / validate-distributed / show / lineage — see the CLI reference). It is implemented and tested but not yet validated at target scale. Each single-node nova capture still produces one capsule with one writer; a one-node run is the degenerate case of the distributed model.

Customizing the URL registry

The wire-level hooks (httpx, requests, aiohttp, urllib3) classify outbound URLs against a vendored YAML registry. Default coverage includes OpenAI, Anthropic, Cohere, Together, Mistral, Replicate, AWS Bedrock, and Ollama on localhost:11434.

To add a private endpoint or a provider not in the default list, create ~/.novafabric/url_registry.yaml. A user-override file replaces the vendored default entirely — copy any entries you still want from the default before editing.

schema_version: "0.1.0"
patterns:
  - match: "internal-llm.example.com"
    gen_ai_system: "internal-prod"
    transport: "http"
  - match: "api.openai.com"
    gen_ai_system: "openai"
    transport: "http"
  - match: "api.anthropic.com"
    gen_ai_system: "anthropic"
    transport: "http"

To override per-invocation:

NOVAFABRIC_URL_REGISTRY=/path/to/registry.yaml nova capture python agent.py

Because the registry is a local, editable file and never hardcodes a hosted endpoint, you can point every classified call at a private mirror or proxy — NovaFabric requires no connection to any NovaFabric-operated service.

Inspect and validate

Once you have a capsule, the first thing to do is confirm it is well-formed and free of leaked secrets. Both checks are CI-friendly and exit non-zero on failure, so you can wire them into a pipeline.

nova validate

Validates a capsule directory, an asset YAML spec, or a replay result directory against their JSON schemas. The command routes automatically based on what path contains.

Capsule validation:

nova validate ~/.novafabric/capsules/01HXAY7M5JZ8R7K4P9DPBYK2WX/
# ✓ Valid capsule: 01HXAY7M5JZ8R7K4P9DPBYK2WX  status=success

Checks capsule.yaml, env.lock, redaction-proof.json, lineage.jsonl (if present), and the presence of all required files and directories.

Asset spec validation (see Asset registry):

nova validate my-model.yaml
# ✓ Valid model spec: my-model@1.0.0

Replay result validation:

nova validate ~/.novafabric/replays/01HXB.../
# ✓ Valid replay result: 01HXB...  mode=forensic  status=success

Use in CI to gate on capsule integrity:

nova capture python agent.py
nova validate ~/.novafabric/capsules/$(ls -t ~/.novafabric/capsules/ | head -1)

nova scan-secrets

Read-only inspection of a capsule's redaction-proof.json. Reports what the secret scanner found during capture. Does not modify the capsule — to re-run the scanner with different strategies, use nova redact.

nova scan-secrets ~/.novafabric/capsules/01HX.../

Clean output:

✓ 01HXAY7M5JZ8R7K4P9DPBYK2WX: no findings

With findings:

! 01HXAY7M5JZ8R7K4P9DPBYK2WX: 2 findings (critical=1, high=1, medium=0, low=0)
rule_id          severity  target           strategy
openai-api-key   critical  model-calls.jsonl  mask
pii-email        high      outputs/stdout.txt  mask

Use --fail-on <severity> to gate CI on severity level:

nova scan-secrets <capsule> --fail-on critical    # exits 2 on critical findings
nova scan-secrets <capsule> --fail-on high        # exits 2 on critical or high

Exit codes: 0 (clean or below threshold), 1 (missing proof / bad input), 2 (threshold exceeded).

Use --json to emit the full redaction-proof.json as parseable JSON to stdout — useful for piping into jq or a CI reporting tool.

Searching run content (experimental)

Status: experimental (ADR-0204, first slice). nova search finds runs by what the agent actually said or did — not just by run id or command:

nova search "invoice INV-2291"        # which run mentioned this invoice?
nova search "rm -rf"                  # where did an agent run this? (literal)
nova search "429" --status failure    # failed runs that saw a rate limit
nova search --reindex                 # one-time backfill of older capsules

The index lives in the local registry DB (SQLite FTS5, zero new dependencies) and only ever contains post-redaction capsule text — the same four files the secret scanner redacts before a capsule is complete. New capsules are indexed automatically at ingest (nova serve / nova ingest-capsule); run nova search --reindex once to backfill capsules captured before this feature. A capsule missing one of those files simply contributes fewer (or no) content rows; the minimal capture level does not drop any of them today (it is planned, not implemented — see nova policy capture-level). All query terms must match (AND); a trailing * does prefix search; FTS5 operators like OR/NEAR/- are matched literally. The same search is available in the dashboard API via GET /api/runs/search?scope=content. Set NOVA_CONTENT_INDEX=off to disable indexing. Postgres server-mode parity is planned; indexing the extended event streams is planned (ADR-0204 P2 — the ADR-0209 scanner-coverage prerequisite has shipped; the corpus stays pinned to the four v0 streams until the P2 slice bumps it). See the CLI reference.

Replay and diff

This is the core of "replayable AI infrastructure." Tracing tells you what happened; replay tells you whether a past run can be re-executed under controlled conditions, and diff tells you exactly what changed between two runs. Every replay writes a replay_result.yaml report; only an intervention replay also writes a new (counterfactual) capsule that you can diff against its original.

nova replay

Replay a captured run from its capsule, in one of five honest, falsifiable modes (the fifth, intervention, is experimental and described in Concepts). NovaFabric deliberately does not claim byte-exact replay of remote LLM calls; the mode you choose reflects how much determinism the run actually supports.

Forensic mode (recommended as a first step):

nova replay ~/.novafabric/capsules/01HX.../ --mode forensic

No subprocess is launched, no network calls are made. NovaFabric reads the manifest, traces, and model calls directly from the capsule. Safe to run in any context — including on capsules received from colleagues or downloaded from an artifact store. This is the mode for audit and post-incident review.

Semantic mode (read-only similarity analysis):

nova replay ~/.novafabric/capsules/01HX.../ --mode semantic

Computes pairwise text similarity across model call responses using difflib.SequenceMatcher. Returns a similarity_score (0.0–1.0). No subprocess, no network. Useful for checking whether a run's responses are internally consistent or vary significantly across calls — the pragmatic answer when the upstream LLM is remote and non-deterministic.

Exact mode (eligibility check):

nova replay ~/.novafabric/capsules/01HX.../ --mode exact

Checks whether the capsule meets the requirements for byte-exact replay: env.lock mode: deterministic (the legacy lock_mode key is still read), a gen_ai.request.seed on every model call, no tool-schema drift, and a command to re-run (a framework-adapter capsule is never eligible). Returns exact_eligible (bool) and a list of exact_reasons if not eligible. No subprocess, no network. Remote LLMs are almost never exact-eligible; this mode is for local / on-prem / compliance runs where determinism is controllable.

Mocked mode:

nova replay ~/.novafabric/capsules/01HX.../ --mode mocked

The original command is re-spawned as a subprocess (Python workloads; what v0.104.0 served versus v0.105.0 is listed under Release scope). A capsule that records no command — written by a framework adapter or @agent (capture_mode: sdk-decorator), or imported from OpenTelemetry spans — is refused before anything is spawned (CapsuleNotReplayable, exit 1, also under --dry-run). Recorded responses are served, in recorded order, for OpenAI chat.completions.create, OpenAI responses.create and Anthropic messages.create calls — sync or async, streamed or not (ADR-0304) — and recorded results for MCP ClientSession.call_tool calls (one recorded result per call). Every other tool runs live — HTTP requests, shell commands, file writes, framework-native tools, functions you did not declare — so run replays of such agents in a sandbox or against test credentials.

Declaring your own Python tools (experimental, since v0.105.0 — ADR-0306 slice 1). Decorate a tool function with record.tool and a mocked replay returns its recorded result instead of running it:

from novafabric.capture import record

@record.tool(mutation_class="external-side-effect", ignore=("ctx",))
def send_invoice(ctx, customer_id: str, amount_cents: int) -> dict: ...
NOVA_CAPTURE_LEVEL=forensic nova capture python agent.py   # keeps arguments and results
nova replay <run-id>                                       # send_invoice's body does not run

The arguments and result are stored only at the forensic (or air_gapped) capture level; at the default level the capsule holds digests and the replay fails with a "re-capture with payload capture enabled" reason. Those digests are taken after secret redaction, but a secret the rules miss (a short password, a PIN) still feeds them, so do not decorate tools that take credentials at the default level. Results must be JSON-native (no tuples or objects) and at most 1 MiB. See Python API: record.tool. The result lists the outbound connections the replay made (replay_contract.network_connections_live); they are reported, not blocked. Since v0.105.0 (experimental), a recorded call that failed (a rate limit, a 4xx, a 5xx after the SDK's retries, a timeout) is replayed by raising the same SDK exception class at the same position, so the workload's error handling runs again.

The replay is fail-closed (ADR-0300): an extra model call, a call on an unsupported surface (parse, legacy completions, with_raw_response), an MCP or record.tool call with no recorded result (the live tool is not run), a record.tool call whose record cannot be served (tool_result_not_servable), or a recorded response that is never requested makes the replay failure with a divergence_reason. --permissive keeps the older warn-and-continue behaviour and only reports — except that an unmatched MCP or record.tool call runs live only if a safety-ladder flag permits its mutation_class (the decorator's declared class; every MCP call counts as unknown, so it needs --allow-unknown-mutation). Changed in v0.105.0 (ADR-0306): --permissive alone used to run every unmatched MCP call live.

The safety-ladder flags classify the capsule's recorded tool calls for the --dry-run report, and any write-permitting rung (--allow-mutating, --allow-external-side-effects or --allow-unknown-mutation) adds an audited replay_mutating policy gate before the replay starts; a deny refuses the replay. Inside the replayed process they gate only the intercepted tool surfaces (MCP, record.tool and, experimental, Google ADK tools through the tool plugin and OpenAI Agents tools through wrap_tools) — an unmatched call under --permissive, and a replay.yaml allow: true override — and never stop a tool replay does not intercept:

nova replay ~/.novafabric/capsules/01HX.../ --mode mocked \
  --allow-readonly             # rung: read-only tool calls
  --allow-mutating             # rung: idempotent / non-idempotent writes (+ policy gate)
  --allow-external-side-effects  # rung: external-side-effect tools (+ policy gate)
  --allow-unknown-mutation     # rung: unclassified tools (+ policy gate)

Per-tool overrides (experimental, since v0.105.0, ADR-0306). A capsule's replay.yaml may carry tool_overrides; in mocked mode they are enforced inside the replayed process, and --dry-run prints the same decisions:

tool_overrides:
  - tool_name: send_email
    allow: false   # never run live, even under --permissive
  - tool_name: lookup_order
    allow: true    # re-execute -- only if you also pass the ladder flag for its class

replay.yaml travels with the capsule, so its restrictions hold and its permissions need your consent: allow: true without the matching --allow-* flag serves the recorded result and reports override_not_honoured. An allow: false on a tool recorded on a surface replay cannot intercept (HTTP, shell, an undeclared function) cannot be enforced: a strict replay refuses to start (ToolOverrideUnenforceable, exit 3), and --permissive runs it and reports override_unenforceable. Each override's outcome is listed under replay_contract.tool_overrides in replay_result.yaml.

This is the mode for CI and regression testing: re-run the recorded command against its recorded responses and diff the result.

Dry-run (see what would happen without running):

nova replay ~/.novafabric/capsules/01HX.../ --dry-run

Results land in $NOVAFABRIC_HOME/replays/<replay-ulid>/replay_result.yaml. Use --output-dir to change the base directory.

Mode comparison:

Mode Re-executes command LLM calls Tool calls Output Primary use
forensic No From capsule (read-only) Not re-executed Inspection report Audit / post-incident
semantic No Read-only analysis Not re-executed similarity_score (0–1.0) Drift detection
exact No Read-only analysis Not re-executed exact_eligible + exact_reasons[] Determinism / compliance check
mocked Yes (refused for a capsule with no command) OpenAI Chat Completions / Responses API and Anthropic Messages, sync or async, streamed or not: from the capsule, recorded errors raised again; unsupported surfaces refused MCP call_tool: from the capsule or refused; other tools live Replay result + divergence report CI / regression
intervention (experimental) Yes, under mocked semantics From the capsule, with one substituted event Other tools live; a substituted tool result is not delivered Replay result + counterfactual capsule Counterfactual root cause

nova diff

Structurally compare two Run Capsules. Aligned field-by-field: model calls (a span id unique on both sides, then sequence position anchored on identical requests), tool calls (by tool name + argument hash, then position), environment, and every file under outputs/, nested ones included (by capsule-relative path and content hash). This is what turns "it worked yesterday, fails today" into a precise, mechanical answer.

nova diff ~/.novafabric/capsules/01HX.../ ~/.novafabric/capsules/01HY.../

Sample output:

Diff: 01HX... → 01HY...
  changed=1  added=0  removed=0
Outputs:
  ~ outputs/stdout.txt

Use as a CI gate. --assert-no-regressions exits 1 if any changes are detected — a changed, added or removed entry in any section — and, since v0.105.0 (ADR-0303), exits 2 when the comparison could not be made (a capsule ref that does not resolve, a capsule.yaml or env.lock that cannot be read or parsed, or a record line that could not be read, which is counted in skipped_malformed_lines and warned about on stderr). Since v0.105.0 a model call whose recorded request parameters (temperature, seed, …) changed also counts as a change (ADR-0303 Amendment 2). Wire it into CI to catch behavioral regressions before they reach production:

nova diff cap-a/ cap-b/ --assert-no-regressions

Output formats:

nova diff cap-a/ cap-b/ --output-format json               # machine-readable
nova diff cap-a/ cap-b/ --output-format github-annotation  # for PR checks

github-annotation emits each difference as an error line — an added or removed call included — and a single notice when there is none: an error appears exactly when --assert-no-regressions would exit 1.

Asset spec diff. If both arguments contain @, the command falls through to the asset registry diff (see nova inspect):

nova diff my-model@1.0.0 my-model@1.1.0

Group by recorded A/B variant (ADR-0116). --group-by variant labels the diff as cross-arm or within-arm using each capsule's recorded (experiment_id, variant_id) — read-only, capsule paths only, text/json output only:

nova diff --group-by variant runs/arm-a/ runs/arm-b/

Statistical regression gate (experimental, NF-007). --significance runs a Wald SPRT test over stored boolean-outcome scores instead of a structural diff — a distinct mode with no positional capsule refs. It reports Wilson confidence intervals for both sides and exits non-zero on a statistically significant regression:

nova diff --significance --baseline base/ --candidate cand/ --metric task_pass

A common pattern is capture → replay → diff: capture a baseline; after a code change, replay it in mocked mode (which fails closed if the new code makes a model or MCP call the recording does not hold), then capture the changed run and diff that capture against the baseline under --assert-no-regressions. A mocked replay writes a report, not a capsule, so it is not itself a diff input; an intervention replay's counterfactual capsule is.

Lineage graph

Replay and diff answer questions about individual runs; lineage answers questions across runs — what a run consumed, what it produced, and what would be affected if an upstream artifact changed.

Every nova capture run automatically writes lineage.jsonl into the capsule. Three edge types are inferred:

Edge type Source Meaning
consumed assets.jsonl entries The run used this asset
produced_by Output files in capsule.yaml This artifact was produced by the run
replayed_from Replay metadata This run is a replay of another

The edges are also indexed into a local SQLite graph at ~/.novafabric/registry.db (same file as the asset registry). The database is a rebuildable cache — if you lose it, run nova lineage import <runs-dir>. The lineage.jsonl files inside the capsules are the source of truth; the graph is a derived index.

nova lineage provenance

What did this run (or asset, or artifact) depend on? Walks ancestors.

nova lineage provenance <run-id>
nova lineage provenance registry:my-dataset@1.0.0
nova lineage provenance <run-id> --depth 3
nova lineage provenance <run-id> --output json

Options:

nova lineage blast-radius

If this asset or artifact changes, which runs would need to be re-evaluated? Walks descendants — the inverse of provenance.

nova lineage blast-radius local:datasets/training-set@1.0.0
nova lineage blast-radius <run-id> --depth 2

This is the primary "impact analysis" query. Run it before updating a shared dataset to see what downstream runs depend on it, or during incident response to scope what a bad artifact touched.

Options: --depth, --kind, --output (same as provenance).

nova lineage replay-chain

Show the replay ancestry of a run, back to the original captured run.

nova lineage replay-chain <replay-run-id>

Useful when you have several generations of replays and want to trace which original capture they all stem from.

Options: --output text|json.

nova lineage time-travel

Show the lineage state of a node as of a specific timestamp. Useful for reconstructing what was known at a point in time, for audits or incident reviews ("what did the dependency graph look like on the day of the incident?").

nova lineage time-travel <run-id> --asof 2026-05-01T00:00:00Z
nova lineage time-travel registry:my-dataset@1.0.0 --asof 2026-04-15T12:00:00Z

Options: --kind, --output (same as provenance).

nova lineage import

(Re-)index lineage from capsule directories into the local SQLite graph. Use this to backfill runs captured before the lineage graph was set up, or to rebuild the index after moving the database.

nova lineage import ~/.novafabric/capsules/01HX.../    # single capsule
nova lineage import ~/.novafabric/capsules/            # all capsules in the directory

The graph is fully derivable from lineage.jsonl files in the capsule directories — losing the SQLite index is not data loss.

Scale note. The SQLite lineage graph is the local-mode default and is designed for graphs below roughly 1M edges. A KuzuDB-backed v2 tier for larger graphs shipped as experimental in v0.10+ (nova lineage-store migrate / profile; see the lineage migration guide). Postgres (recursive CTE), Apache AGE (openCypher) and JanusGraph (Gremlin) backends also ship as experimental library classes in novafabric.lineage.backends (no CLI flag selects them yet). Cross-site lineage federation remains future design (a protocol prototype lives in novafabric.lineage.federation; nothing in the CLI or server uses it).

nova lineage emit-openlineage

Emit capsule runs as OpenLineage 2.0.2 events (START / COMPLETE / FAIL). Enables integration with data catalog tools such as Marquez, Atlan, and OpenMetadata.

# Print to stdout
nova lineage emit-openlineage ~/.novafabric/capsules/01HX.../ --output -

# Post to an HTTP endpoint
nova lineage emit-openlineage ~/.novafabric/capsules/ --output http://marquez:5000/api/v1/lineage

# Write to a file
nova lineage emit-openlineage ~/.novafabric/capsules/ --output events.jsonl

If --output is omitted, the target is resolved from OPENLINEAGE_URL → OPENLINEAGE_FILE → stdout.

You can also trigger automatic emission at capture time by setting OPENLINEAGE_URL before running nova capture — the orchestrator will post events without an explicit emit-openlineage call.

Custom run facets (experimental, NF-036/037). Add --with-facets to attach NovaFabric's custom facets to the COMPLETE event — novafabric_capsule (capsule/run id and hash), novafabric_eval (verdict + suite + metrics), novafabric_policy (promotion gate decision), and the standard executionParameters facet. Add --otel-correlation (implies --with-facets) to also attach novafabric_otel_correlation (trace_id / span_id) so a lineage node links to its OTel GenAI spans. Facets are additive and schema-validated before emission, so a consumer that ignores them still sees unchanged core OpenLineage events:

nova lineage emit-openlineage ~/.novafabric/capsules/01HX.../ --with-facets --otel-correlation

Note: these OpenLineage run facets are a different thing from the capsule facets container (ADR-0196, experimental) — the additive facets.<domain> evidence objects that record-only modules such as settlement, embodied, and science attach to a capsule. Same word, two unrelated places; see Evidence facets in Concepts.

Graph analytics (experimental — ADRs 0212–0215)

Beyond traversal, NovaFabric can turn the whole lineage graph into insight — all read-only, local-first, and deterministic:

# Structurally critical nodes: hubs + single points of failure
nova lineage metrics --top 10

# Rank the upstream root-cause suspects for a failed run
nova lineage root-cause 01HX...

# Export the graph for Gephi / yEd / Neo4j
nova lineage export-graph --format cypher -o lineage.cypher

# One synthesized report (hubs, communities, orphans, health, cost)
nova insights --output markdown -o insights.md

metrics ranks by degree / PageRank / betweenness and flags articulation points; root-cause combines error signals, recency, edge confidence, and cross-run failure correlation and never fabricates a culprit when there is no signal; export-graph emits byte-stable GraphML/GEXF/Cypher; insights composes all of it plus seeded Louvain communities and best-effort cost, and reports any unavailable data source as unavailable rather than inventing it. See the CLI reference for full flags.

nova lineage consume (experimental)

Runs the JetStream LineageConsumer as a foreground daemon: it pulls NovaFabric lineage events off a NATS subject and bulk-COPYs derived edges into a KuzuDB graph on a size-or-time flush trigger. This is the cluster-scale ingestion path alongside the local nova lineage import — local-mode capture never requires it; use it only when events already flow through a NATS deployment rather than sitting in capsule directories on a shared filesystem. Requires pip install 'novafabric[scale,scale-kg]' (nats-py + kuzu) and a running NATS JetStream server.

nova lineage consume --nats-url nats://nats:4222 --kuzu-path .nova/kg/lineage.kuzu

Options include --subject (default novafabric.lineage.>), --batch-size, --fetch-timeout, --flush-batch-size, and --flush-interval-s (max seconds between flushes even below the batch-size threshold). Not exactly-once: a NATS message is acked once its edges are extracted, before the flush actually persists them — see the CLI reference for the full flag list and delivery-semantics detail.

Trust layer

The trust layer is what makes a capsule shareable and auditable: verifiable redaction of secrets, and a signed Evidence Bundle a reviewer can verify offline with no NovaFabric runtime. This is the compliance-supporting surface of the product — it produces evidence that supports compliance workflows; it does not certify or guarantee compliance.

nova redact

Re-scan a capsule and update its redaction-proof.json. Use this if you want to re-run the scanner with a different strategy, or if you received a capsule with potentially unscanned secrets.

Re-scan with default strategies (mask in-place):

nova redact ~/.novafabric/capsules/01HX.../

Re-scan with strategy overrides. Choose mask (default), hash (SHA-256 first-8 hex chars), or drop (remove the field entirely):

nova redact <capsule> --strategy-override openai-api-key:hash
nova redact <capsule> --strategy-override pii-email:drop

Mark a finding as an acknowledged false positive:

nova redact <capsule> --mark-unsafe-skip <finding-id> --rationale "test fixture — not a real key"

False positives recorded in unsafe_skips are preserved across future re-scans unless you clear them:

nova redact <capsule> --clear-unsafe-skips

Capsules with unsafe_skips entries block nova export-evidence unless you pass --allow-unsafe-skips — so an acknowledged false positive is always a deliberate, recorded decision, never a silent one.

nova export-evidence

Build a signed Evidence Bundle ZIP from a capsule. The bundle is a self-contained, verifiable archive: a copy of the capsule, its lineage edges, the JSON schemas, Ed25519-signed in-toto DSSE attestations, and a manifest.json with a SHA-256 for every file plus the verification recipe (full list). A reviewer can verify the bundle with nothing but sha256sum and an ed25519 verifier — no NovaFabric runtime required. This is the primitive that makes a capsule portable audit evidence you own.

Generate a signing key (one time): nova init writes an Ed25519 pair to $NOVAFABRIC_HOME/keys/signing_key.pem (private, mode 600) and signing_key.pub.pem (public). Re-running nova init keeps an existing pair.

nova init

Build the bundle:

nova export-evidence <capsule> \
  --key ~/.novafabric/keys/signing_key.pem \
  --output evidence.zip

Options:

Bundle layout:

evidence.zip
├── manifest.json                       # index + manifest_hash
├── run-capsule/                        # byte-identical capsule copy
├── lineage-subgraph/edges.jsonl        # lineage for this run
├── attestations/                       # in-toto Statement v1, DSSE-enveloped
│   ├── run.intoto.json
│   ├── redaction.intoto.json
│   ├── lineage.intoto.json
│   └── bundle.intoto.json
├── signatures/                         # raw signature + public key per envelope
│   ├── *.sig
│   └── *.cert
├── schemas/                            # all JSON schemas vendored
└── README.md                           # human-readable verification recipe

Note. --sigstore is Rekor publication, not keyless signing: when NOVA_REKOR_URL is set it uploads the run attestation's DSSE envelope to that Rekor log and prints the entry id; when it is unset it prints a warning and the export still exits 0. The bundle is always signed by --key. Sigstore keyless signing is planned.

Cryptographic sealing (experimental)

Status: experimental — shipped in v0.10+, interfaces may change before the v1.0 schema freeze.

The NovaSeal in-process signing core is implemented and tested: DSSE signing (ECDSA P-256), an opt-in RFC 3161 trusted timestamp (since v0.103, ADR-0292: with no tsa_url in novaseal.yaml no timestamp is requested and no network call is made — earlier releases silently defaulted to FreeTSA), and an append-only SQLite Merkle log, verified by nova verify <capsule> — which reports signature_ok, timestamp_ok, and log_integrity_ok. Verification is self-contained: new seals carry their Merkle inclusion proof, so an auditor needs only the capsule — no novaseal.yaml and no copy of your log; a check that cannot run prints NOT CHECKED / NOT PRESENT and does not fail the command. Sealing is opt-in via $NOVAFABRIC_HOME/novaseal.yaml (~/.novafabric/novaseal.yaml by default); maker-checker seal flows are nova seal propose / approve / verify (ADR-0059). See the NovaSeal configuration guide and the CLI reference.

Still planned (ADR-0041 design intent, not shipped): the dedicated, network-hardened NovaSeal signing service, qualified timestamps, Sigstore-keyless signing as the default path, and WORM-backed retention at the seal layer. For the most stable portable proof today, use the Evidence Bundle above — ed25519-signed bundles with in-toto DSSE attestations and verifiable redaction proofs, verifiable offline.

Non-Python client capture

For agents that do not run in Python — Claude Code, Cursor, Continue.dev, Node.js, Go, Rust — NovaFabric provides two transparent proxy commands that capture LLM API calls without modifying the client. Both write into the same capsule schemas as the in-process hooks.

nova mcp-proxy (experimental)

A transparent proxy that sits between an MCP client and an upstream MCP server, recording every tools/call request/response pair into a capsule.

Two transport modes are supported.

Stdio mode (default): wraps the upstream MCP server process.

NOVAFABRIC_CAPSULE_DIR=/path/to/.novafabric/runs/01HX...
nova mcp-proxy -- npx -y @modelcontextprotocol/server-filesystem /tmp

Or with an explicit flag:

nova mcp-proxy \
  --capsule-dir .novafabric/runs/01HX.../ \
  -- /usr/local/bin/my-mcp-server --flag

If --capsule-dir is omitted and NOVAFABRIC_CAPSULE_DIR is not set, a fresh capsule directory is allocated automatically under .novafabric/runs/.

Claude Desktop integration. Replace the upstream server entry in claude_desktop_config.json:

{
  "mcpServers": {
    "filesystem": {
      "command": "nova",
      "args": [
        "mcp-proxy",
        "--capsule-dir", "/Users/me/.novafabric/runs/01HX...",
        "--",
        "npx", "-y", "@modelcontextprotocol/server-filesystem", "/tmp"
      ]
    }
  }
}

HTTP mode (v0.6.3): listens on a TCP port and forwards POST/SSE to an upstream HTTP MCP server.

nova mcp-proxy \
  --listen 127.0.0.1:8765 \
  --upstream-url http://upstream-mcp.internal:9000/mcp

Stdio and HTTP modes are mutually exclusive per invocation. One proxy invocation handles one upstream server.

Records use the same tool-calls.jsonl schema as the SDK hook, plus an extension field extensions["io.novafabric.capture_method"] = "proxy".

Status: experimental. The API may change.

nova api-proxy (experimental)

A transparent HTTP proxy for non-Python LLM clients. The client points its *_BASE_URL environment variable at the proxy; the proxy forwards to the upstream LLM API and records every call into model-calls.jsonl.

# Terminal 1: start the proxy
nova api-proxy \
  --upstream-url https://api.openai.com \
  --listen 127.0.0.1:8765

# Terminal 2: point your LLM client at the proxy
export OPENAI_BASE_URL=http://127.0.0.1:8765
claude ...     # Claude Code
# or: cursor, continue, any Node/Go/Rust LLM client

For Anthropic:

nova api-proxy \
  --upstream-url https://api.anthropic.com \
  --listen 127.0.0.1:8765

export ANTHROPIC_BASE_URL=http://127.0.0.1:8765

Options:

Flag Default Description
--upstream-url (required) The LLM API base URL to forward to
--listen 127.0.0.1:8765 Proxy listener address as host:port
--capsule-dir auto-allocated Where to write model-calls.jsonl; falls back to NOVAFABRIC_CAPSULE_DIR
--upstream-timeout 60.0 Timeout per upstream read, in seconds (httpx applies it to each connect/read, not to the whole call)

The proxy buffers every response. It relays the upstream status, body and headers (including retry-after, x-should-retry, x-ratelimit-* and request-id, so client SDK retries see them) only after the whole upstream response has arrived. A stream: true client therefore receives nothing until the model finishes — time to first token equals total latency (planned: relaying chunks as they arrive). For the record, an SSE stream is merged into a synthesized non-streaming envelope.

The proxy is base-URL override only. No TLS MITM, no local CA.

Status: experimental. ADR-0026.

Asset registry

The Asset Registry is NovaFabric's identity layer: a local SQLite database at ~/.novafabric/registry.db (override: NOVAFABRIC_DB_PATH) that tracks AI assets — models, agents, prompts, tools, datasets, evaluation suites, and deployment endpoints — with lifecycle statuses and promotion history. Each asset is addressed as name@version and pinned to a git SHA at registration.

Promotion is governance metadata only. Promoting an asset updates one DB record. It does not restart, deploy, or redeploy anything — it records a lifecycle decision, and (for agents) gates on a passing eval.

nova register

Register an asset from a YAML spec file.

nova register my-model.yaml
nova register research-agent.yaml

Model spec example:

novafabric_spec_version: "1"
asset_type: model
name: fraud-detection-model
version: 1.2.0
status: development
description: "XGBoost fraud detection model trained on transactions dataset"
spec:
  framework: xgboost
  artifact_path: s3://ml-artifacts/fraud-detection/v1.2.0

Agent spec example:

novafabric_spec_version: "1"
asset_type: agent
name: support-triage-agent
version: v1.0.0
status: development
description: "LLM agent that triages incoming support tickets"
spec:
  model:
    provider: anthropic
    name: claude-haiku-4-5-20251001
    temperature: 0.1
  tools:
    - search_knowledge_base
    - create_ticket
  prompts:
    system: "You are a support triage assistant. Classify incoming tickets by priority."
  policies: []
  evals:
    - triage_accuracy_suite

Spec files are validated strictly: a key NovaFabric does not define — at the top level or inside spec: — is a validation error that names the key, so a typo such as suite_name: for suite: is caught instead of silently dropped. Put vendor or custom data under the optional top-level extensions: mapping (reverse-DNS keys, kept verbatim), or in tags: / external_refs: (string values). (Before this change, unknown keys were accepted and discarded.)

extensions:
  com.example.owner: fraud-team
  com.example.cost-centre: {id: 4711}

Supported asset types and their spec fields (src/novafabric/spec/models.py):

asset_type Required spec fields Optional spec fields
model framework, artifact_path —
agent model (provider, name), tools, prompts, policies (may be []), evals (at least one) memory, model.temperature
prompt — (spec: itself is optional) template
tool — (spec: itself is optional) entrypoint
dataset — (spec: itself is optional) path
evaluation — (spec: itself is optional) suite
deployment — (spec: itself is optional) endpoint

Exits 0 on success, 1 on validation error or duplicate. Use nova validate <spec.yaml> to validate without registering.

nova suggest-register

Analyze captured Run Capsules and suggest assets to register. This inverts the onboarding workflow: capture first, then let NovaFabric propose what to register from observed evidence (models seen, tools called, agent command).

# Interactive: scan 10 most recent capsules, prompt per suggestion
nova suggest-register

# Write draft YAML files without registering
nova suggest-register --draft-only --output-dir ./drafts/

# Auto-register high-confidence suggestions (models + tools, skip agents)
nova suggest-register --auto --min-confidence 0.8 --skip-types agent

# Analyze a specific run by ID
nova suggest-register 01HXAY7M5JZ8R7K4P9DPBYK2WX

Confidence levels: models = 100% (every observed call), tools = 80% (min 2 calls), agents = 70%. Post-capture, a one-line hint is printed when unregistered assets are detected. Disable with NOVAFABRIC_SUGGEST=0.

nova list

List registered assets. Filter by type or lifecycle status. Use --stale to surface assets with no promotion, consumption, or eval activity in the last N days.

nova list
nova list --type agent
nova list --type model --status staging
nova list --status production

# Stale asset detection
nova list --stale                          # assets inactive > 30 days
nova list --stale --stale-days 60          # custom threshold
nova list --stale --status production      # stale production assets only

--stale output adds last_activity_at and days_stale columns. Activity includes any promotion, capsule consumption (assets.jsonl), or eval run.

nova inspect

Show full metadata for an asset. If version is omitted, shows the latest.

nova inspect fraud-detection-model@1.2.0
nova inspect support-triage-agent          # latest version

Output includes the full spec YAML, lifecycle status, git commit at registration, and eval results for agent assets.

nova promote

nova promote is a sub-group with four commands (direct, keygen, propose, approve). Valid lifecycle transitions:

development → staging → production → archived

Direct transitions (e.g. development → production) are permitted. Once archived, no further promotion is possible.

nova promote direct

Single-actor promotion (original behaviour):

nova promote direct fraud-detection-model@1.2.0 --to staging
nova promote direct fraud-detection-model@1.2.0 --to archived

Protected stages (ADR-0312). production is protected. nova promote direct --to production is refused unless an OPA policy explicitly allows the promote_direct action, and the shipped default policy denies it. With no OPA installed there is no explicit allow, so it is refused. --force does not bypass this. Use propose + approve below, or see the CLI reference for loading an allow policy. The dashboard's Promote button and the HTTP promote endpoints go through the same gate and answer 403.

Eval gate. Agent assets require a passing nova eval result before promotion to staging or production. Use --force to override (requires interactive confirmation; recorded as forced in the audit trail):

nova promote direct support-triage-agent@v1.0.0 --to staging --force

SLSA provenance on promotion (experimental). Add --slsa-provenance to emit a DSSE-signed slsa.dev/provenance/v1 attestation for the promotion, and --slsa-ml-profile to emit the SLSA-for-ML profile instead — its buildDefinition captures dataset versions/hashes and seeds, and its byproducts bind the promoted model to the digest of its recorded eval results (omitted when none is recorded) (NF-057). Both verify with nova verify-envelope or stock cosign:

nova promote direct my-model@1.0.0 --to staging \
    --slsa-provenance --slsa-ml-profile --slsa-out my-model.slsa.json

nova promote propose + nova promote approve

Maker-checker two-step flow for regulated deployments (ADR-0058, verified since ADR-0312). Two different people, each with their own Ed25519 key, and each key bound to that person by an operator-held signer trust list.

One-time setup.

# Each signer creates a key; this prints the trust-list entry for it.
nova promote keygen --identity alice
nova promote keygen --identity bob

An operator copies the printed entries into the trust list. It is read from NOVAFABRIC_PROMOTE_SIGNERS if set, otherwise $NOVAFABRIC_HOME/promote-signers.yaml. Each entry should keep only the roles that person holds:

signers:
  - identity: alice
    public_key: <from keygen>
    roles: [proposer]
  - identity: bob
    public_key: <from keygen>
    roles: [approver]

Each promotion.

# Step 1 — maker proposes (signature verified before the proposal is stored)
nova promote propose fraud-detection-model@1.2.0 --to production --identity alice

# Step 2 — checker approves (both signatures verified, then the asset is promoted)
nova promote approve fraud-detection-model@1.2.0 --identity bob

What is enforced today:

Every single-actor route into production is gated the same way. nova promote direct, nova prompt promote, the dashboard and nova serve promote endpoints, the nova server route PUT /v0/assets/{id}/promote (403 promote_direct_denied), and a nova rollback to a version that was never in production all need an explicit promote_direct policy allow; --force does not bypass it. PUT /v0/assets/{id}/promote records the authenticated principal (server:<subject>) as promoted_by, as registration does; a request-body actor is ignored.

Limit. The trust list is the trust anchor. Anyone who can edit it — or who can set NOVAFABRIC_PROMOTE_SIGNERS / NOVAFABRIC_HOME for the process doing the approving — can bind a second key to a second name. Keep it owned by an operator and read-only to signers, and do not let signers choose the environment of the process that verifies them (ADR-0312 Limits).

nova rollback

Roll back an asset to its most recent previous production version in one command. Atomically archives the current production version and restores the previous one. Designed for incident response — no manual registry queries needed.

nova rollback my-agent --actor on-call-eng
nova rollback my-agent --to v1.8.0 --actor on-call-eng   # explicit target

Steps performed in a single DB transaction:

  1. Find the current production version (error if none).
  2. Find the most recent prior production version (auto, or use --to).
  3. Archive the current production version.
  4. Promote the prior version back to production.
  5. Write an audit log entry with a rollback_reason and a gate field.

"Prior production version" comes from the registry's append-only status history. A version that was in production before is restored at once, even if it is archived — that is what incident response needs. A --to target that was never in production is refused unless the promote_direct policy action explicitly allows it (ADR-0312), the same gate as nova promote direct; otherwise promote --to staging followed by a rollback would be a single-person path into production. --actor is recorded in the audit trail.

nova capture — asset status gate

Before spawning the subprocess, nova capture can verify that a named asset is in the required lifecycle status. Nothing is written to disk if the check fails.

# Hard block — exits non-zero if my-agent@v1 is not in staging or production
nova capture --asset my-agent@v1 --require-asset-status staging,production -- python agent.py

# Soft warn — logs a warning but does not block
nova capture --asset my-agent@v1 --warn-if-asset-status development -- python agent.py

# Block if the asset is not registered at all (default: warn only)
nova capture --asset my-agent@v1 --require-asset-status production --require-registered -- python agent.py

This is an opt-in gate. Without --asset, capture behaviour is unchanged.

nova eval

Run evaluation suites declared in an agent's spec and store the results in the registry. Suites are resolved via the novafabric.evals Python entry-point group.

nova eval agent support-triage-agent@v1.0.0

After a passing eval, nova promote direct <name@version> --to staging will succeed without --force. production is a protected stage: reach it with nova promote propose + nova promote approve, unless a policy explicitly allows promote_direct (ADR-0312).

The bundled standard eval suites (GAIA, SWE-bench, AgentBench, MMLU, Smoke) run in OCI-pinned containers, and promotion can be Rego-gated to block on a regression against a recorded baseline.

Guard against benchmark contamination (experimental, NF-028). Contamination silently inflates scores, so NovaFabric records the dataset + split content hashes an eval ran against. Check a capsule against a configurable registry of known-bad hashes:

nova eval contamination-check ./my-capsule --registry known-bad.json --json

It reports a status per dataset (current / superseded / contaminated / unknown) and exits 4 when any dataset is contaminated or superseded, so CI can gate on it. It fails closed: a facet it cannot read or validate is reported as invalid, with a reason, and also exits 4. Detection only — no remediation. See the feature tour for the dataset provenance and SLSA-for-ML surfaces that pair with it.

nova report

Generate an asset inventory report. Defaults to Markdown on stdout.

nova report                        # Markdown to stdout
nova report --format json          # JSON to stdout
nova report --output report.md     # write to file
nova report --format html --output inventory.html   # self-contained HTML + by-type chart (ADR-0201)
nova report --format pdf --output inventory.pdf      # requires the optional WeasyPrint extra

--format pdf requires --output and the optional pip install 'novafabric[compliance]' extra (WeasyPrint); without it the command exits 1 with an install hint.

Sample Markdown output:

# NovaFabric Asset Report

## Summary
| Type | Count |
|---|---|
| model | 3 |
| agent | 2 |
...

Local dashboard

nova serve (experimental)

An opt-in local HTTP dashboard for browsing capsules, registry assets, and the lineage graph. The CLI remains the canonical interface; the dashboard is a read-oriented satellite. Every dashboard mutation displays the equivalent nova command and is logged to ~/.novafabric/dashboard-audit.jsonl, so nothing the dashboard does is invisible to the CLI-first audit trail.

Install the optional extra (one time):

pip install 'novafabric[serve]'

This adds FastAPI and uvicorn (both Apache-2.0 / MIT licensed).

Start the server:

nova serve --experimental

The --experimental flag is mandatory and acknowledges that the dashboard API may change.

The terminal prints a URL with an embedded session token:

Dashboard: http://127.0.0.1:4321/?token=<token>
API docs:  http://127.0.0.1:4321/api/docs?token=<token>

Click the URL to open the dashboard in your browser. The token is also written to $NOVAFABRIC_HOME/.serve-token (mode 0600) — the start-up panel prints the resolved path, which is ~/.novafabric/.serve-token only when NOVAFABRIC_HOME is unset.

The token is not rotated on every restart. nova serve reuses an existing .serve-token file so that a restart does not break open browser sessions, so a URL you were given earlier keeps working after a restart — and the token stays readable on disk (mode 0600) after the server exits. Delete the file if you want the next start to mint a fresh token.

Flags:

Flag Default Description
--experimental (required) Acknowledges the experimental gate
--port 4321 TCP port
--host 127.0.0.1 Bind address; localhost only without --insecure
--capsule-dir $NOVAFABRIC_CAPSULE_DIR, else $NOVAFABRIC_HOME/capsules Where to look for capsules (where nova capture writes)
--db-path $NOVAFABRIC_DB_PATH, else $NOVAFABRIC_HOME/registry.db Registry/lineage SQLite path
--no-browser off Do not auto-open a browser tab

Capsule index. nova serve keeps a runs_cache SQLite index so the Runs tab stays fast regardless of how many capsule directories are on disk. The index is built on startup and refreshed every 2 s by CapsuleWatcher. If capsules were added while the server was stopped, force a full re-index:

nova ingest-capsule --all                       # re-index everything
nova ingest-capsule <run_id>                    # index a single capsule
nova ingest-capsule --watch                     # foreground watcher loop

Two watcher backends: WatchdogBackend (pip install novafabric[watch]; uses inotify/FSEvents) and PollingBackend (zero extra deps). The default, auto, uses watchdog when it is installed and falls back to polling otherwise. Force one with NOVA_WATCHER_BACKEND=polling|watchdog and set the poll period with NOVA_WATCHER_INTERVAL=<seconds> (default 2); an unknown value is refused at start-up.

Navigating. The 30 tabs are grouped in the sidebar under seven headings: Overview · Runs & Debug · Govern & Promote · Provenance & Trust · Compliance · Platform · Reports & Export. Since v0.98.3 the groups start collapsed, each showing its tab count — click a heading to expand it. The group holding the tab you are on is always expanded, so your current location is never hidden, and whatever you expand or collapse is remembered in that browser. Every tab is deep-linkable as ?tab=<id>, and the Compliance tab is itself a hub whose five panel groups (Frameworks · Audits · Privacy · Exports · Assurance) are deep-linkable as ?sub=<group>. Navigation shortcuts are mnemonic two-key sequences — press g then the tab's letter (g h Home, g r Runs, g c Compliance); press ? for the full map, and ⌘K / Ctrl-K for the command palette. (v0.97.0 replaced the earlier positional 1–9 keys with these sequences; tab ids and ?tab= links are unchanged.)

What the dashboard covers:

The dashboard also surfaces a component-status view covering the cluster-scale subsystems (NovaSeal, collector, object store, metadata DB, parent/child capsules). These shipped in v0.10+ as experimental / prototype tiers — implemented and tested, but not validated at target scale. Check each component's maturity label in ROADMAP.md rather than treating presence in the status view as production readiness.

What requires the CLI. Some operations are intentionally CLI-only: nova report, lineage time-travel, OpenLineage emission, nova mcp-proxy, nova api-proxy, cluster runners, mocked replay, semantic/exact replay, nova promote propose/approve (maker-checker). See docs/dashboard.md for the full capability matrix.

Security model:

Stop the server with Ctrl+C. The session token is invalidated on stop.

Status: experimental. The dashboard API (endpoint shapes, field names) may change between minor versions.

Environment variables

Variable Default Purpose
NOVAFABRIC_HOME ~/.novafabric Root directory for all internal NovaFabric data. Set this once to redirect all files (registry.db, .serve-token, dashboard audit log) to a shared location.
NOVAFABRIC_DB_PATH $NOVAFABRIC_HOME/registry.db SQLite path for asset registry and lineage graph. Overrides NOVAFABRIC_HOME for this path only.
NOVAFABRIC_METADATA_DB_PATH $NOVAFABRIC_HOME/metadata.db SQLite path for the MetadataStore (runtime and nova db upgrade). NOVAFABRIC_DB_PATH does not select it.
NOVAFABRIC_URL_REGISTRY (bundled) Override path for the wire-level URL classification registry
NOVAFABRIC_CAPSULE_DIR — Capsule storage directory. Set by nova capture in the subprocess; used by hook loader and proxy commands. Set explicitly to redirect captures to a shared path (e.g. a shared NFS mount).
NOVAFABRIC_SPAN_ID — Root OTel span id injected into the subprocess by the orchestrator
NOVAFABRIC_SLURM_SHARED_DIR — Shared filesystem path for SLURM live tests (ops / CI use)
NOVAFABRIC_SUGGEST — Set to 0 to disable the post-capture suggest-register hint
OPENLINEAGE_URL — HTTP endpoint for automatic OpenLineage emission at capture time
OPENLINEAGE_FILE — File path for OpenLineage emission (fallback if OPENLINEAGE_URL not set)
NOVAFABRIC_DASHBOARD_AUDIT_FILE $NOVAFABRIC_HOME/dashboard-audit.jsonl Audit log destination for dashboard mutations. Overrides NOVAFABRIC_HOME for this path only.
NOVA_WATCHER_BACKEND auto Capsule-index watcher backend (auto, polling or watchdog); an unknown value is refused. auto uses watchdog when installed, else polling. nova ingest-capsule --watch --backend overrides it; nova serve has no such flag.
NOVA_WATCHER_INTERVAL 2.0 Poll interval in seconds for the capsule-index watcher; an explicit --interval flag wins.
NOVA_CONTENT_INDEX on Set to off to disable content indexing at ingest (experimental, ADR-0204)
NOVA_CONTENT_INDEX_MAX_DOC_KB 64 Per-doc text cap for the content index (operator tuning)
NOVA_CONTENT_INDEX_MAX_DOCS 10000 Per-capsule doc cap for the content index (operator tuning)

Cost analytics (experimental)

Beyond the offline nova cost estimate and the ClickHouse-backed nova cost report (both covered in the CLI reference), four additional nova cost subcommands report descriptive cost evidence over records you already hold. Each is experimental, read-only, and never a verdict — none applies a threshold, quota, or pass/fail; whether a figure is acceptable is the operator's call. Each also takes --json.

See the CLI reference for flags, input shapes, and exit codes.

What shipped experimental in v0.59

v0.59.0 added a large cohort of experimental command groups — first slices of the Langfuse-parity ADRs (0112–0141) plus interop and forensics surfaces. All are additive, off unless you opt in, and their interfaces may change. This guide does not duplicate them; each is fully documented in the CLI reference, and the grouped overview lives in the v0.59.0 release notes:

Area Commands / surfaces (all experimental)
Prompt lifecycle nova prompt register/get/list/history/diff/compose/tree, nova label (deployment labels + protected maker-checker moves)
Evaluation & annotation nova eval score config, nova annotate, nova score submit (+ novafabric.scores.submit), nova comment, nova experiment run/compare
Capture completeness nova session new/add/list/reindex/show/replay/export/verify-bundle/import, nova graph agent, nova capture --capture-media + nova media list, --environment, variant attribution (--experiment/--variant), observation log levels, nova validate --schemas
Offline analytics nova query, nova view, nova trend, per-usage-type token accounting, nova pricing + nova cost estimate
Governance nova retention plan/apply/status/explain, PII masking plugins (--masker), the budget promotion gate (Rego), nova events webhooks, SCIM 2.0 provisioning, partial SAML SSO (metadata + assertion-validation policy; live assertion consumption shipped opt-in in v0.73.0 and is off by default)
Portability & interop nova export --html, nova export-blob + manifest nova verify, OTLP GenAI-span ingest (POST /api/otlp/v1/traces), nova eval import-inspect/export-inspect, nova diagnose --intervene, nova pii status

What shipped in v0.75–v0.94

A second large cohort landed after v0.59, mostly ADR-0101 (attribution/root-cause) and a set of standalone trust primitives. All are experimental; some are CLI-exposed today, others are Python-API only — no nova subcommand wraps them yet, so build against the module, not a shipped command:

Capability Status Where
Counterfactual root-cause search (ADR-0101 NF-018/020) CLI, experimental nova diagnose <run-id> --search-root-cause [--max-interventions N] — sweeps ranked causal-root candidates with bounded intervention replays until one confirms
Causal-graph back-trace (ADR-0101 NF-019/022) Python API only diagnose.causal_root_candidates() — used internally by --search-root-cause; no direct CLI verb
Span-level claim-grounding audit (ADR-0101 NF-021) Python API only diagnose.audit_claims() — flags a model-span claim ungrounded when no tool-span evidence precedes it; not wired into nova diagnose output yet
NATS lineage consumer (cap-006, ADR-0061/0066/0219) CLI, experimental nova lineage consume
EU AI Act Art. 12 record-keeping export CLI, experimental nova euaiact export / nova euaiact status
Compliance cohort (ISO 42001/42005, Art.72 PMM, GPAI Art.53, NIST GenAI/CSA) CLI, experimental nova export-compliance {iso42001,pmm,gpai53,genai-profile}
Portable agent-passport projection (ADR-0149) CLI, experimental nova passport issue / nova passport verify
SAML SSO (ADR-0138) CLI, experimental, partial nova server saml-metadata emits SP metadata only; assertion-consumption is opt-in server config, not a CLI verb
x509 certificate-pinned signing identity (ADR-0055) Python API only trust/novaseal/x509_identity.py
x509 CA-bundle signer chain validation (ADR-0055 trust step 2) CLI, experimental nova verify --ca-bundle <pem> (or ca_bundle: in novaseal.yaml); library: validate_certificate_chain() in trust/novaseal/x509_identity.py
Crypto-agility hybrid-signature envelope (ADR-0072) Python API only trust/novaseal/hybrid_signature.py
did:key + Verifiable Credentials (ADR-0075) Python API only trust/did.py
"Acted-as" delegation chains (ADR-0106) Python API only trust/delegation.py
Transparency-log witness cosigning (ADR-0097) Python API only trust/novaseal/witness.py
Jurisdiction sovereignty site-seals (ADR-0077) Python API only compliance/sovereignty.py

The Python-API-only rows above have no nova command and no dashboard panel — they are real, tested modules (see the corresponding ADR for design intent), but there is no shipped end-user workflow around them yet. Do not follow a tutorial that shows a nova command for them; none exists. See Python API and docs/developer-guide.md for the module-level detail.

Summary and next steps

You now have the full shipped nova command surface, organized around the five primitives:

Primitive Commands
Run Capsule nova capture, @agent decorator, runners, nova validate, nova scan-secrets
Replay nova replay (forensic / mocked / semantic / exact; intervention experimental), nova diff
Lineage nova lineage provenance / blast-radius / replay-chain / time-travel / import / emit-openlineage
Evidence Bundle nova redact, nova export-evidence
Asset Registry nova register, nova suggest-register, nova list, nova inspect, nova promote, nova rollback, nova eval, nova report

Two journeys branch from the Run Capsule: Capture → Replay → Diff (developer: what happened, what changed?) and Capture → Seal → Verify → Audit (trust: can I preserve and verify it later?). Sealing is opt-in and experimental — see Cryptographic sealing. Everything above runs locally today, offline, with no accounts and no telemetry.

A good next move, depending on your goal:

Where to go from here: