Run Capsule anatomy

Architecture, as built › Run Capsule

A Run Capsule is a plain directory named by a ULID. It holds every observable fact about one execution. You can copy it, archive it, attach it to a ticket or verify it without a server, an account or a database.

Maturity: works today. nova validate checks capsule.yaml against the packaged schema src/novafabric/schemas/run-capsule.schema.json (schema_version 0.1.0). The repo-root schemas/run-capsule.schema.json is the v1.0 target schema and is not yet in force (ADR-0223). The format is not frozen until that v1.0 freeze, but changes are additive and optional, so older capsules stay valid.

How a Run Capsule is assembled, file by file

Where capsules live

Setting Default Resolved by
NOVAFABRIC_CAPSULE_DIR — _paths.py:default_capsule_dir
NOVAFABRIC_HOME ~/.novafabric _paths.py:nova_home
Effective capsule root $NOVAFABRIC_HOME/capsules/
Per-run override nova capture -o <dir> cli/capture.py

The run ID is a 26-character ULID (capture/_ulid.py; schema $defs/Ulid). ULIDs sort by creation time, so a directory listing is already in chronological order.

The files, in write order

CaptureOrchestrator.run in capture/orchestrator.py writes the files in this order. A failed run gets the same set of files. Rows 2 to 10 run after the workload, so they treat the capsule as untrusted input (see After the workload runs).

# Path Holds Written by
1 inputs/, outputs/ inputs/ is created empty (capture does not populate it today; the manifest's inputs list is []). outputs/ holds process output and media blobs — the workload's own output files are not copied in capture/capsule.py:CapsuleWriter.open
1 model-calls.jsonl LLM call records, keyed with OTel GenAI attributes (gen_ai.request.*, gen_ai.response.*); a call made through the OpenAI/Anthropic SDK also leaves one transport record per HTTP attempt (see Model-call record roles) hooks, through CapsuleWriter
1 tool-calls.jsonl One record per tool invocation hooks, through CapsuleWriter
1 trace.jsonl Execution spans, including the root span hooks and orchestrator
1 assets.jsonl References to registry assets the run consumed hooks, through CapsuleWriter
2 outputs/stdout.txt, outputs/stderr.txt Process output — each file is written only when its stream is non-empty orchestrator
2 outputs/<sha>.<ext> Media blobs, content-addressed capture/media.py
3 env.lock Environment snapshot: mode (deterministic or best-effort), host, Python interpreter, installed packages capture/env.py:capture_environment
4 replay.yaml Replay policy (schemas/replay-policy.schema.json); capture writes a minimal policy, which you can extend capture/replay.py:minimal_replay_policy
5 capsule.yaml The run manifest, redacted as a data structure before it is written orchestrator
6 lineage.jsonl Typed lineage edges for this run lineage/_writer.py:LineageWriter
7 capture-health.json Dropped-event counts, only when the recorder dropped events — the orchestrator's own and those the workload process handed off at exit — or when the workload tampered with a capsule path (tampered_entries). Written before the residual pass, so it is scanned and listed in evidence_digests capture/event_recorder.py:EventRecorder.finalize_health, capture/workload_health.py
8 redaction-proof.json The secret-scan proof (see below), written once after the residual pass capture/secrets.py:SecretScannerV0
9 capsule.yaml (rewritten) The manifest again, now with evidence_digests orchestrator, _evidence_digests
10 .seal/manifest.dsse, .seal/manifest.dsse.tsr, .seal/log-entry.json The seal, only when NovaSeal is configured orchestrator, _seal_capsule

These files appear only in some runs:

Path When
network_events.jsonl, file_events.jsonl, human_approvals.jsonl When the event stream has at least one record (capture/event_recorder.py). The manifest then carries the matching optional network_events_ref, file_events_ref or human_approvals_ref (each a fixed file name; capture/orchestrator.py:_event_stream_refs). Before this release those three keys were missing from the closed capsule schema, so nova validate rejected any capsule that had one — for example a run with a single instrumented HTTP call to a model endpoint
capture-health.json When the recorder had to drop events. Its absence means nothing was dropped up to the residual pass. Written before that pass (since v0.105.0; v0.104.0 wrote it after sealing, unbound), so it is scanned and listed in evidence_digests; an event dropped after the digest map was computed cannot be added without breaking the seal, so capture logs a warning for it instead. dropped_events covers both processes of a nova capture run: the hooks run in the workload's own Python process, which writes its counts to a hidden .capture-health.workload.<pid>.json at exit; nova capture merges and deletes those files just before writing capture-health.json. When the workload dropped anything, the additive optional sources object splits the total into orchestrator and workload, and sources_note says the workload counts are self-reported and unverified. A hand-off is written by the recorded process, so it is untrusted: only the capsule's own stream names (capture/streams.py:CAPSULE_STREAMS) and counts up to 10⁹ are accepted (any other hand-off file is ignored and logged), each merged total is capped at 10⁹, and a hand-off can only add to the orchestrator's counts, never lower them. A workload process that ends with os._exit or is killed hands off nothing.
c2pa-manifest.json nova capture --mark-provenance
otel-genai-spans.json nova capture --emit-otel-genai
redaction_manifest.json nova capture --pii-subject (experimental, ADR-0314): one entry per protected span

After the workload runs: the capsule is untrusted input

The workload writes into its own capsule, and with --runner docker the capsule directory is bind-mounted read-write into the container (runners/_docker.py). Anything the host-side finalisation then follows is a container-to-host write, so from the moment the runner returns, capture treats the capsule as untrusted (works today):

  1. Integrity check, before any host-side read or write (capture/tamper.py:inspect_capsule_after_workload). Each capsule-managed path (every JSONL stream, capsule.yaml, env.lock, replay.yaml, redaction-proof.json, capture-health.json, lineage.jsonl, c2pa-manifest.json, redaction_manifest.json, outputs/stdout.txt, outputs/stderr.txt, the inputs/ and outputs/ directories, and .seal/) is lstat-ed and never opened. A symlink, FIFO, socket or device there, a directory where a file belongs (or the reverse), or any .seal/ at all is removed without being followed, the empty stream or directory CapsuleWriter.open made is recreated, and a finding is recorded. Then the whole tree is walked, one openat(2) + O_NOFOLLOW directory at a time (re-opened from the root, so a wide tree never exhausts descriptors), and every symlink, FIFO, socket and device at any depth is removed and recorded: NovaFabric never writes one, so each was planted. A directory nested more than 64 levels deep (MAX_SWEEP_DEPTH) is not descended and is recorded as not-swept. A hard link (a regular file with st_nlink > 1) is kept and is not a finding: see step 2. If the capsule directory itself is no longer the one CapsuleWriter.open created (moved and replaced by a link, which only the local runner allows), capture writes nothing more.
  2. Every later write goes through novafabric._capsule_fs: CapsuleWriter, the secret scanner, the masking pipeline, the --pii-subject stage (including redaction_manifest.json), the lineage writer, the event recorder, the C2PA marker and the seal writer replace a whole file by an atomic rename (replace_capsule_file, which replaces a planted link, FIFO or hard link instead of writing through it) or append with O_NOFOLLOW | O_NONBLOCK (open_capsule_file_for_write, which refuses a link or special file and gives a hard-linked file a private copy before appending). The capsule directory is pinned after step 1 (pin_capsule_root, by the (st_dev, st_ino) CapsuleWriter.open recorded): every later open under it opens the root O_NOFOLLOW and raises CapsuleRootChangedError if it is a link or a different directory, and nova capture then prints ✗ capture integrity: ... and exits 1.
  3. Every later read (the scanner, evidence_digests, the call counters, the usage roll-up, the spool re-emission) opens with O_NOFOLLOW | O_NONBLOCK and a regular-file check, so an entry planted after step 1 is never followed and never blocks.
  4. At the digest step (capture/finalize.py:finalize_capsule), the tree is walked once more without following or opening anything; a link or special file found there was planted after step 1 and is appended to metadata.capture_tampered (planted after the sweep: ...) before the manifest is gated and sealed.

A finding marks the capsule tampered in three places:

Where What
capsule.yaml metadata.capture_tampered, e.g. model-calls.jsonl (symlink); inputs/pipe (fifo): the first 20 entries, then and N more (M in total). Inside the signed manifest, so a seal binds it. nova verify and nova validate fail on it (exit 1).
capture-health.json tampered_entries: path, kind (symlink, fifo, socket, char-device, block-device, directory, regular-file), action (removed, removed-and-recreated, not-removed, not-swept) and, for a link, link_target (redacted, never resolved). At most 1,000 entries. The file is written even when nothing was dropped.
redaction-proof.json A target <path>#tampered per finding: skipped: true, skip_reason: "stream_tampered: ...". The # suffix keeps it apart from the recreated file's own target.

Capture is fail-open (ADR-0021): it still exits with the workload's exit code, and prints ✗ capture integrity: ... on stderr. A link the workload leaves anywhere in the capsule, even one that stays inside it (say outputs/latest -> model.bin), is tampering: no export format can represent it, and before 2026-10-23 such a link at an unmanaged path survived capture. nova verify and nova validate also fail any link or special file found in the capsule later (planted after sealing), naming each one. Residual limits: with the local runner a process the workload leaves running can still race step 1. Steps 2–4 then refuse, replace or record what it planted, never following it, but the capture may stop with an error instead of finishing. And kubectl cp (the Kubernetes runner) runs before step 1, so its own handling of links in the pod's capsule directory applies.

Diagram source (Mermaid)
flowchart TB
    root["01J…ULID/"]
    root --> m["capsule.yaml<br/><i>manifest + evidence_digests</i>"]
    root --> s1["model-calls.jsonl"]
    root --> s2["tool-calls.jsonl"]
    root --> s3["trace.jsonl"]
    root --> s4["assets.jsonl"]
    root --> e["env.lock"]
    root --> r["redaction-proof.json"]
    root --> rp["replay.yaml"]
    root --> l["lineage.jsonl"]
    root --> io["inputs/ · outputs/"]
    root --> seal[".seal/ (optional)"]
    seal --> d["manifest.dsse"]
    seal --> t["manifest.dsse.tsr"]
    seal --> le["log-entry.json"]
    m -. "sha256 of every file except<br/>capsule.yaml and .seal/" .-> s1 & s2 & s3 & s4 & e & r & rp & l & io
    d -. "signs" .-> m

The manifest: capsule.yaml

The schema requires schema_version, run_id, created_at, finished_at, duration_ms, status, command, capture_mode, novafabric_version, working_directory, host, the *_ref pointers to the files above, trace_root_span_id, inputs, outputs, and the call counters (model_call_count, tool_call_count, mutating_tool_count). model_call_count counts logical model calls, not records (see below). Optional refs — lineage_ref, attestations_ref, content_provenance_ref, and the event-stream refs network_events_ref, file_events_ref, human_approvals_ref — appear only when their file exists. The top level is closed (additionalProperties: false), so every key a writer adds must be registered in all three schema copies (src/novafabric/schemas/, schemas/, ui/dashboard/src/data/schemas/); tests/capture/test_event_stream_refs_schema.py checks the event-stream refs against the orchestrator and validates a real capture end to end.

The host block

Works today (since v0.105.0; v0.104.0 adapter and decorator capsules still carry the hardcoded values below). capsule.yaml:host is built by one function, capture/env.py:host_info, for every capsule writer: nova capture (CaptureOrchestrator), the framework adapters (adapters/_capsule.py) and the @agent decorator (novafabric.sdk.agent, in sdk/agent.py). Before this was unified, adapter and decorator capsules wrote arch: x86_64, cpu_count: 1 and memory_bytes: 0 whatever the machine; tests/capture/test_host_arch_is_never_hardcoded.py now fails on any literal for a measured host field.

Field Source Limits today
arch capture/env.py:host_arch, which env.lock and the OTLP importer also use: platform.machine() normalised (aarch64 → arm64, amd64 → x86_64) an unmapped machine string is recorded as reported, never guessed
os platform.system(), lower-cased the schema allows only linux, darwin and windows; any other OS is recorded as linux
cpu_count os.cpu_count() 1 if it cannot be read
memory_bytes MemTotal from /proc/meminfo on Linux, sysctl hw.memsize on macOS, GlobalMemoryStatusEx on Windows (capture/env.py) 0 when the reader fails: read 0 as "not measured"
python platform.python_version()
gpu — always [] today: no GPU inventory is collected (env.lock:hardware.gpus is empty too)
hostname_redacted — always true; the hostname is never written to the manifest (env.lock keeps a SHA-256 of it). Slurm node names follow the same rule: slurm.node_list_hash
runner capture/orchestrator.py:_host_block experimental (ADR-0307, since v0.105.0). nova capture only; see below
slurm capture/env.py:slurm_context_from_env experimental (ADR-0307, since v0.105.0). Present only inside a Slurm job; see below
kubernetes — in the schema, never written

These host fields describe the machine that ran nova. With --runner docker, kubernetes or slurm the workload ran somewhere else, so python, cpu_count and memory_bytes (and env.lock) are still the capturing host's, not the container's or the compute node's.

host.runner and host.slurm (experimental, ADR-0307)

Since v0.105.0. Which runner ran the workload, which image, and which Slurm job.

Field Recorded when Value
runner.name every nova capture run local, docker, kubernetes, slurm, pbs or lsf. Absent on adapter, @agent and OTLP-imported capsules, and on capsules from earlier versions
runner.image.reference docker, kubernetes the image= you passed, verbatim
runner.image.image_id docker (and old dockershim clusters) sha256:<64 hex>: what docker image inspect resolved the reference to
runner.image.repo_digests when the runtime reports them <name>@sha256:<64 hex>, the pullable pinned form. Docker: from docker image inspect; Kubernetes: the pod's status.containerStatuses[0].imageID
runner.image.resolved_by with either digest docker-image-inspect or kubernetes-pod-status
runner.image.unresolved_reason instead of all three above why no digest was recorded: the image could not be inspected, the pod was not found, or the tag pointed at a different image after the run than before it
slurm.job_id SLURM_JOB_ID is set, or --runner slurm the job id
slurm.array_job_id, slurm.array_task_id array jobs SLURM_ARRAY_JOB_ID, SLURM_ARRAY_TASK_ID
slurm.partition, slurm.cluster when Slurm set them SLURM_JOB_PARTITION, SLURM_CLUSTER_NAME. For --runner slurm: the partition only if you requested one, the cluster only on a multi-cluster site
slurm.node_count when Slurm set it SLURM_JOB_NUM_NODES
slurm.node_list_hash when Slurm set it sha256: + 16 hex of SLURM_JOB_NODELIST. Node names are hostnames, so they are hashed like env.lock:host.hostname. To check a node list, hash the same string
slurm.source always environment (read from the variables above) or runner (--runner slurm: the job sbatch created)

The digest is never taken from the tag. Docker is asked before and after docker run: if the tag resolved to different images at those two moments, neither can be attributed to the run and no digest is recorded. Slurm context comes from those variables only, never from SLURM_* by prefix, so accounts, submit directories and anything else a site exports under the prefix stay out. Both blocks are inside capsule.yaml and pass the secret scanner like every other manifest field.

Not recorded: Slurm node names in clear, step ids, accounts and QOS; the node list for --runner slurm; PBS and LSF job ids (only runner.name); the Kubernetes namespace, pod and node (host.kubernetes exists in the schema and is never written); the container's own environment.

Model-call record roles

Works today (ADR-0305, since v0.105.0). A call made through the OpenAI or Anthropic SDK is recorded by two hooks: the SDK hook writes the logical record (the parsed response, io.novafabric.api_surface), and the wire hook (httpx) writes one transport record per HTTP attempt underneath it, with no response. Both are kept: the transport records are the only evidence of SDK-internal retries (a 429 followed by a 200) and of what went over the wire. Two optional extensions keys say which is which:

Key Value
io.novafabric.record_role logical or transport
io.novafabric.logical_call_id The model_call_id of the logical call: its own id on a logical record, the covering SDK record's id on a transport record (every retry of one SDK call links to the same id)

A wire record with no SDK call around it (raw httpx, a non-SDK client) is logical. An unmarked record is logical (adapter, API-proxy and OTel-ingest records carry no marker).

Everything that counts or iterates model calls reads logical records only: model_call_count, nova cost, nova diff, mocked replay and its counters, nova query, the dashboard (/api/runs/{id} returns transport records separately as transport_model_calls), the knowledge graph, and the exporters. Evidence surfaces that hash, scan or bundle the file read every record. A transport record whose logical record is missing (the call was cancelled before the SDK hook wrote it) counts once, so a call is never dropped from a count. The rule lives in one place, novafabric.capture.record_roles (logical_model_calls, is_transport_record).

Capsules captured before ADR-0305 carry no marker. Readers recognise the old duplicate shape: a response-less wire record followed by an SDK record for the same request (model + messages) that encloses it in time, or, without timestamps, follows it with only retries in between. Anything less certain is counted, so the fallback never under-counts. The model_call_count sealed into an old capsule.yaml is not rewritten: it keeps the doubled value it was signed with.

Some optional fields matter for the architecture:

Field Purpose
evidence_digests {path: {sha256, size_bytes}} for every file except capsule.yaml and .seal/. This field is what binds the whole directory to the seal.
parent_run_id, replay_of_run_id, replay_mode Relationships to other capsules. replay_of_run_id becomes a replayed_from lineage edge.
session_id, sequence Membership in a multi-turn session (experimental)
facets, extensions Additive extension points (slurm, kubernetes, …) that let the format grow without a new top-level format
exit_code, error The failure record. A failed run is still a complete capsule. error.type is NonZeroExit when the workload ran and exited non-zero, and WorkloadNotStarted when the runner could not start it (runner_status: failed_setup, e.g. command not found: <argv0> from runners/_local.py), so the record never claims that something exited (WorkloadNotStarted is new in v0.105.0).

The redaction proof: redaction-proof.json

The secret scanner runs on every capture. It scans and redacts, in place, the call and event streams (capture/secrets.py:SCAN_TARGETS), env.lock, assets.jsonl, lineage.jsonl and every file under inputs/ and outputs/ (ARTIFACT_SCAN_TARGETS, ARTIFACT_SCAN_DIRS). The manifest is redacted before capsule.yaml is written. Binaries are scanned by string extraction, and a binary that carries a key-shaped match is dropped. A final residual pass rescans every file in the finished capsule before evidence_digests is computed, and the final manifest is checked again before it is sealed. The rule pack is gitleaks-core-v0 0.7.0, which matches known key formats only. Its proof file records:

redaction-proof.json is listed in evidence_digests, so a seal covers the proof along with everything else.

What the scanner detects, and what it does not

The proof records what was scanned and what was redacted. It is not proof that a capsule contains no secret: detection is by pattern, so a secret in a format no rule knows passes through unchanged and the proof shows zero findings for it.

Detected by gitleaks-core-v0 0.7.0 (18 rules, capture/secrets.py:_RULES): API keys with a known prefix for OpenAI, Anthropic, Hugging Face, Replicate, LangFuse, LangSmith, Weaviate, Qdrant and Pinecone (pckey_/pcsk_); Cohere, Together and Mistral keys by shape alone (a bare 40-character alphanumeric, 64-hex or 32-character alphanumeric token), which is also why those three rules never cause a binary file to be dropped; AWS access key IDs (AKIA, ASIA, ABIA, ACCA, A3T…); an AWS secret access key when its key name is next to it (AWS_SECRET_ACCESS_KEY=…, "SecretAccessKey": "…"); GitHub tokens (ghp_, gho_, ghu_, ghs_, ghr_, github_pat_); NovaFabric's own API keys and webhook secrets.

Not detected by the default pack (checked against pack 0.7.0):

Treat a capsule as secret-scanned, not as secret-free. If your workload can print credentials in a format above, add a masker for it or keep it out of the captured output.

Capsules written inside a framework call

Works today (the streaming behaviour, the measured host block, the replay refusal and the shared finalization described here shipped in v0.105.0). The framework adapters (src/novafabric/adapters/*.py; LlamaIndex, Pydantic AI and Haystack share adapters/_capsule.py:AdapterCapture, the others write their own manifest) and the @agent decorator (novafabric.sdk.agent, in sdk/agent.py) write a capsule from inside the Python process:

OTLP-ingested capsules (capture_mode: otel-import, metadata.capture_level: ingested-otlp, written by otel/genai_ingest.py for POST /api/otlp/v1/traces) finalize through the same finalize_in_process_capsule call (since v0.105.0): main scan, manifest redaction (an agent name from a span lands in the manifest), residual pass, evidence_digests, gate, and a seal when a signing profile exists. A failure keeps every ingested record and leaves the capsule unsealed with metadata.finalization_error. The ingested-otlp label is permanent; a seal does not upgrade it. See OTLP ingest.

Parent and child capsules (prototype)

Distributed runs, such as a SLURM job across several nodes, use a separate writer, capsule/writer.py:CapsuleWriter with ParentCapsuleTracker. It records one PARENT plus N WORKER capsules under a shared global_run_id, in a capsule.json that follows schemas/parent_child_capsule_v1.schema.json. capsule/orphan.py handles workers whose parent never finalised. The commands are nova run new-run-id | validate-distributed | show | lineage (capsule/cli.py).

Two classes named CapsuleWriter exist: capture/capsule.py writes ordinary run capsules, and capsule/writer.py writes the distributed parent/child layout.