Capturing at cluster scale — 1,000,000 agents

Who this is for: platform architects planning NovaFabric deployments across HPC clusters, AI factories, or large-scale distributed agent systems.

Status of this document (corrected 2026-07-30): the single-cluster design (v0.7) is shipped, and — contrary to an earlier draft of this page — the collector hierarchy, hot/cold storage split, and lineage-at-scale tiers below it are also shipped, as experimental, not still "design + build." See design/architecture/cluster-scale.md (private) (the authoritative, actively-maintained version of this material) for full per-phase evidence. What genuinely remains design intent only is cross-cluster federation (Phase 6) and full cross-org identity — those are called out explicitly below.

Why the current design doesn't reach 1M agents

v0.7 works for one team on one cluster:

nova capture writes capsules to local disk
nova serve reads them into Postgres on a single server
The lineage graph lives in that Postgres instance

At 1,000,000 concurrent agents this breaks in three places:

Problem	What breaks
Storage	1M agents × ~1,000 events each = 1 billion events per run-cycle. Postgres cannot ingest this.
Ingestion	A single server receiving writes from 1M simultaneous processes becomes a bottleneck that stalls the entire fleet.
Lineage graph	Blast-radius queries over billions of edges need a graph store with proper traversal indexes, not `SELECT WHERE` over a Postgres table.
Identity	One OIDC server issuing tokens to 1M agents is a single point of failure. Each agent needs a credential verifiable without calling home.

The one rule that makes scale possible

Capture is always local. Collection is always async.

A compute node running agent #847,291 must never make a network call to a central server during the agent's execution. If it does, one slow server stalls 1M jobs.

Instead:

[agent #847,291 on compute node]
        │
        ▼
  nova capture  →  writes to LOCAL buffer (tmpfs or NVMe on the node)
                   agent finishes, capsule is sealed
                   collection happens AFTER, in the background
        │
        ▼  (async, after agent exits)
  [rack-level collector]
        │
        ▼
  [cluster-level aggregator]
        │
        ▼
  [central lineage store]

The agent never waits for any central system. This is already baked into the current design — nova capture writes locally. The collector hierarchy above it is also built now (experimental, not yet proven at 1M-agent scale): a Go node-collector (collector/, Lustre-safe spool + NovaSeal batch signing, 295K events/sec measured) and a lighter Python FastAPI receiver (collector_app/, POST /capsule) both exist. They do different jobs: the Go spool forwarder publishes events over NATS (the event bus) to a hub that batch-signs them, while collector_app accumulates the events it receives into DuckDB (in-memory by default) or ClickHouse (when NOVA_CLICKHOUSE_URL is set). Neither writes capsules to the object store; that tier (ObjectCapsuleStore) is a separate library and server component.

Running the Python collector receiver securely

collector_app is an ASGI factory; launch it with any ASGI server, e.g. uvicorn --factory novafabric.collector_app:create_collector_app. It reads two variables at startup (works today, experimental):

Variable	Effect
`NOVA_COLLECTOR_TOKEN`	Every `POST /capsule` must send `X-Nova-Token: <token>` — from any address, loopback included. Compared in constant time.
`NOVA_COLLECTOR_TENANT`	The tenant that caller is authenticated as (default `default`). Every event is stored under it.

export NOVA_COLLECTOR_TOKEN="$(python -c 'import secrets; print(secrets.token_urlsafe(32))')"
export NOVA_COLLECTOR_TENANT=team-a
uvicorn --factory novafabric.collector_app:create_collector_app --host 0.0.0.0 --port 8090

No token = local mode. The collector then serves only a caller that is provably on the same machine: a loopback peer address, a localhost/127.0.0.1/[::1] Host header, and no proxy header (Forwarded, X-Forwarded-*, X-Real-IP, Via). Everyone else gets 403, and a warning is logged at startup. This keeps the zero-setup local workflow working while making a collector started on 0.0.0.0 without a token refuse remote evidence instead of accepting it from anyone. (The factory cannot see the address it is bound to — the ASGI server owns the socket — so the check is made on every request rather than at startup.)
⚠ Set NOVA_COLLECTOR_TOKEN whenever anything proxies to the collector. A proxy on the same host connects from loopback. Local mode refuses the requests of a proxy that marks itself with a forwarding header — but it cannot detect one that adds none. nginx's proxy_pass http://127.0.0.1:8090; with its defaults sends Host: 127.0.0.1:8090 and no X-Forwarded-*, so every remote caller behind it looks local and is served without authentication. Only a token closes this, which is why there is no separate "allow unauthenticated local" switch: a token already turns local mode off entirely.
The tenant comes from the token, not the request. A body tenant_id or an event tenant that differs from NOVA_COLLECTOR_TENANT is refused with 403 and nothing in the batch is ingested. tenant_id is optional. Serving several tenants means one collector, with its own token, per tenant.
A malformed event is refused with 422. Every event needs a non-empty string event_type. run_id, model, model_id and tool_name must be strings when present; tokens_in/tokens_out must be integers in 0–2³¹−1 and tokens_total in 0–2⁶³−1. One event that breaks this refuses the whole batch; nothing in it is ingested, and the response's detail names the event index and the field. Other keys pass through unchanged. (Before, such an event crashed the request with 500.)
Either spelling of the model and token fields is counted by either backend. The model is model_id, else model. ClickHouse stores tokens_total when given, else tokens_in + tokens_out, and NULL when the event reports no count (not reported is not zero). DuckDB stores tokens_in and tokens_out; it does not split a bare tokens_total, so send tokens_in/tokens_out when DuckDB backs the collector.

The full architecture

                   ┌────────────────────────────────┐
                   │        Federation Layer         │
                   │  (cross-org, cross-site policy) │
                   └───────────────┬────────────────┘
                                   │
           ┌───────────────────────┼────────────────────────┐
           ▼                       ▼                        ▼
  [Cluster A — nova serve]  [Cluster B — nova serve]   [Cluster C]
    Postgres + REST API        Postgres + REST API        ...
           │                       │
  ┌────────┴───────┐      ┌────────┴───────┐
  ▼                ▼      ▼                ▼
[Rack collector] [Rack collector]        ...
  │    │    │
  ▼    ▼    ▼
[node][node][node]   ←  each runs nova capture locally

Layer 1 — Compute node (per agent)

Responsibility: capture only, no network during execution.

nova capture writes events to a local buffer (tmpfs, NVMe)
Capsule is sealed when the agent exits
Local buffer is bounded — if the agent crashes, the partial capsule is still valid and readable

Must not do:

Synchronous writes to remote databases
Heavy redaction or graph construction
Block the agent workload for any capture operation

Layer 2 — Rack collector (per rack or per node group)

Responsibility: pull, validate, and forward.

Polls compute nodes for completed capsules
Validates capsule schema
Deduplicates content and repeated deliveries — future design: the shipped Go forwarder delivers at least once, and nothing downstream deduplicates by event_id yet (prompt-level content dedup: see below)
Forwards metadata to cluster server, raw events to object storage
Runs as a daemon process on a dedicated rack-level node

Layer 3 — Cluster server (per cluster)

The current nova serve (v0.7) with Postgres. Responsibilities:

Stores run metadata (not raw events — those live in object storage)
Maintains the lineage graph for this cluster
Serves the REST API and dashboard
Issues short-lived credentials to agents

Layer 4 — Federation (cross-cluster, cross-org)

Not designed yet. Responsibilities will include:

Cross-cluster lineage queries ("show me every run across all clusters that consumed this dataset")
Cross-org identity (an agent at facility A needs a credential verifiable at facility B)
Policy propagation (a data governance rule defined at org level applies to all clusters)

The three hard problems at 1M scale

1. Storage: separate hot and cold paths

Storing 1 billion raw events in Postgres is not feasible. The design separates:

Hot path — Postgres stores only metadata:

run_id | workload | status | model | created_at | asset_refs | ...

Small rows, fast queries, manageable volume.

Cold path — object storage (any S3-compatible system) stores raw capsules:

s3://<bucket>/
    capsules/<tenant>/<sha256[0:4]>/<sha256>/data.zst   ← one whole capsule blob, immutable

The key is the SHA-256 of the whole capsule payload (object_capsule_store/cas.py, experimental), so blobs are write-once and an identical capsule is stored once.

Content deduplication below the capsule level — storing each distinct prompt or response once, with capsules holding only hash pointers — is future design. Today two capsules that share a system prompt each carry their own copy.

2. Lineage graph at billions of edges

A SELECT WHERE target_id = X query over a billion-row Postgres table is unusable. At scale, the lineage graph needs:

Proper graph traversal indexes — blast-radius and provenance queries follow edges, not scan rows
Pre-computed materialized subgraphs for common patterns (frequently-queried assets, recent runs)
Sharding by namespace — HPC facility A's lineage doesn't need to be in the same shard as facility B's

Corrected 2026-07-30 — this is no longer undecided. All four tiered backends are implemented and testcontainers-verified: SQLite (<1M edges, local default), Postgres recursive-CTE (v0.68.0), embedded KuzuDB (v0.17.0, benchmark-cleared at 45.5ms p99 blast-radius @10M edges), Apache AGE openCypher (v0.69.0), and JanusGraph Gremlin for billion-edge scale (v0.70.0, needs the GraphSON serializer and .emit() on traversals — see lineage/backends/). Sharding by namespace and pre-computed materialized subgraphs remain future work; picking which backend to run at a given scale is now a config choice, not an open research question.

3. Agent identity at scale

Currently: one nova serve instance issues tokens. One server = one failure point.

At 1M agents, each agent needs a short-lived, scoped credential:

Valid only for this job's duration
Scoped to this agent's namespace (can't write to another agent's capsule)
Verifiable by any collector without calling the issuing server (offline verification)

This maps to a standard JWT / OIDC design with short TTLs and an offline verification key distributed to collectors. The issuing server can be replicated. The verifying collectors need only the public key, not network access to the issuer.

What exists today vs what's needed

Capability	Status
Local capture with zero network overhead	✓ shipped (v0.1)
Portable capsule format (self-contained directory)	✓ shipped (v0.1)
Single-cluster server with Postgres	✓ shipped (v0.7)
`runs_cache` index — O(1) `/api/runs` at 100K capsules (Scale-S1)	✓ shipped (v0.32.0)
Background capsule watcher + `nova ingest-capsule` (Scale-S3)	✓ shipped (v0.36.0)
Postgres partition DDL for 10K-tenant MetadataStore (Scale-S2)	✓ shipped (v0.32.0)
Content-addressed capsule storage (an identical whole-capsule blob stored once, `cas.py`)	✓ shipped experimental (Phase 4, v0.14.5) — SHA-256 CAS key `capsules/<tenant>/<sha[0:4]>/<sha>/data.zst`. Prompt-level dedup and `event_id` dedup of collector deliveries are future design
Hot/cold storage split (Postgres metadata + object storage events)	✓ shipped experimental (Phase 4, `object_capsule_store/`, v0.14.5)
Rack/node-level collector daemon	✓ shipped experimental (Phase 2, v0.14.3 — Go `collector/`, 295K events/sec p99 4.7ms; plus a lighter Python `collector_app/` FastAPI receiver, v0.29.0)
Graph traversal indexes for lineage at scale	✓ shipped experimental — all four at-scale backends (KuzuDB v0.17.0/Phase 6, Postgres recursive-CTE v0.68.0, Apache AGE v0.69.0, JanusGraph v0.70.0), all testcontainers-verified; zero remaining stub backends
Short-lived agent credentials (offline-verifiable, scoped per-agent-namespace)	— needs design + build. (A related but narrower piece exists: `nova server` issues Ed25519 offline service-account tokens for CI, not per-agent scoped compute-node identity at 1M scale.)
Multi-cluster federation	— not yet designed (Phase 6; depends on 2–5 being stable in production, per `design/architecture/cluster-scale.md` (private))
Cross-org lineage and identity	— not yet designed

Using `nova ingest-capsule` (Scale-S3, shipped v0.36.0)

nova ingest-capsule populates the runs_cache SQLite index from capsule files on disk. It does not require nova serve to be running — useful for batch jobs, cron tasks, or CI pipelines that capture capsules and want to index them without starting the full server.

Quick start

# Index a single capsule by run ID
nova ingest-capsule abc123

# Re-index all capsules in the default capsule directory
nova ingest-capsule --all

# Foreground watcher loop — prints each new capsule as it appears
nova ingest-capsule --watch

# Stop with Ctrl+C

Faster indexing with inotify / FSEvents

The default backend (PollingBackend) scans the capsule directory every 2 s. For environments where watchdog is available, the WatchdogBackend reacts to OS-level filesystem events instead of polling:

# Install the optional extras
pip install 'novafabric[watch]'

# Use the event-driven backend
nova ingest-capsule --watch --backend watchdog

# Or set it globally
export NOVA_WATCHER_BACKEND=watchdog
nova ingest-capsule --watch

Override capsule directory and DB path

nova ingest-capsule --all \
  --capsule-dir /shared/nfs/capsules \
  --db-path /var/lib/novafabric/registry.db

In `nova serve`

When you run nova serve, it automatically creates a CapsuleWatcher internally and delegates all startup indexing and incremental polls to it. The backend is selected by NOVA_WATCHER_BACKEND (default: auto, which uses WatchdogBackend if novafabric[watch] is installed, else PollingBackend).

You do not need to run nova ingest-capsule --watch alongside nova serve — the server already does this in its background stats loop every 2 s.

Environment variables

Variable	Default	Effect
`NOVA_WATCHER_BACKEND`	`auto`	`auto` \| `polling` \| `watchdog`
`NOVA_WATCHER_INTERVAL`	`2.0`	Poll interval in seconds

Why we built the single-cluster case first

The collector hierarchy, storage split, and lineage-at-scale tiers described above are now built (experimental) — but they were built in this order deliberately, and the reasoning still matters for what's genuinely left (federation): you need to know what the data actually looks like under load before designing a cross-cluster protocol on top of it.

How large is a typical capsule?
How many events does a typical agent run produce?
What does the lineage graph topology look like in practice — deep chains or wide fans?
What are the hot query patterns — blast-radius, provenance, or replay-chain?

The nova-testbench continuous loop exists to answer these questions with real data, not assumptions. 1,000 capsules from the testbench tells you more about storage requirements and graph shape than any design document can.

The sequence:

Make one cluster solid and measure it — done
Build the collector, storage-split, and lineage-at-scale layers on measured data — done (all shipped experimental; see the status table above)
Design federation based on measured multi-cluster patterns — still open (Phase 6; genuinely not yet designed)

Building a federation protocol before you know the single-cluster behavior was speculation. That's why it was measured first and built second — federation is the one piece still waiting on real multi-cluster patterns.

References

Each decision below is summarised — title, status, and rationale — in the public ADR index, docs/decisions.md. The full ADR texts and the formal design principles live in the maintainers' private design/ tree and are not part of this repository, so the index is the reference a reader here can actually open.

ADR-0020 — low-overhead capture on compute nodes
ADR-0021 — AI-factory design intent
ADR-0022 — polyglot persistence and object storage
ADR-0023 — cache architecture
ADR-0032 — parent/child capsule hierarchy (prerequisite for multi-node agent linking)
ROADMAP.md — release sequencing