API Rate Limiting & Storage Quotas

This guide covers protecting a shared nova server deployment from noisy clients: in-process API rate limiting and warn-then-reject storage quotas (ADR-0179; normative budgets in design/spec/rate-limiting-quotas-v0.md (private)).

Status: experimental (v0.61, shipped 2026-07-16) — and disabled by default. With the feature off (the default), no middleware is installed at all: no headers, no timing side effects, zero behavior change on upgrade.

Both tracks apply to the nova server app only; the nova serve dashboard is not rate-limited.

1. Rate limiting

Design in one paragraph

A stdlib-only token bucket per key and limit class, on a monotonic clock, held in a bounded in-process map (LRU-evicted above 10,000 buckets). It is deliberately not distributed — the supported topology is a single-writer server, so an in-process limiter is the honest fit (ADR-0179).

Limit classes and defaults

Every request is classified, in order: exemptions → admin prefixes → OTLP ingest paths → write verbs → reads.

Class Matches Default rate (tokens/s) Default burst
ingest write verbs (POST/PUT/…) + any OTLP ingest path, regardless of verb 100 200
read GET / HEAD 50 100
admin /v0/admin/*, /v0/roles/* 10 20

Always exempt — never limited, never counted, no headers: /health, /livez, /readyz, /metrics. Probes must not brown out with the API, and /metrics is how you see the limiting.

Bucket keying

The bucket key is resolved per request in strict fallback order — first available wins:

  1. principal — the SHA-256 digest of the presented bearer token (the raw token is never used as a key value), once a request carrying that token has authenticated;
  2. tenant — a resolved tenant id (the slot exists for the ADR-0178 workspace model; no server surface populates it before routing today);
  3. client IP — unauthenticated surfaces, and every request whose bearer token has not yet authenticated.

A token's first request therefore draws from its client's IP bucket; once it authenticates, the token has its own bucket. A client that sends a different invalid token on every request is limited by its IP and creates no buckets. With insecure_no_auth no token names a principal, so all requests are keyed by client IP.

Classes never share buckets, so an ingest storm cannot starve reads for the same client.

The 429 contract

A limited request receives 429 with the standard ADR-0017 error envelope (error.code = "rate_limited") plus:

Successful responses on classed routes carry the three X-RateLimit-* headers too, so well-behaved clients can pace themselves before hitting 429.

Enabling and tuning

In ~/.config/novafabric/server.yaml (or the --config file), as a top-level rate_limits: block:

rate_limits:
  enabled: true
  ingest: { rate: 100, burst: 200 }
  read:   { rate: 50,  burst: 100 }
  admin:  { rate: 10,  burst: 20 }
  audit_threshold_rejections: 100   # rejections per key per window that trigger an audit event
  audit_window_seconds: 60

or via environment overrides: NOVAFABRIC_SERVER_RATE_LIMITS_ENABLED=1, NOVAFABRIC_SERVER_RATE_LIMITS_{INGEST,READ,ADMIN}_{RATE,BURST}, NOVAFABRIC_SERVER_RATE_LIMITS_AUDIT_THRESHOLD_REJECTIONS, NOVAFABRIC_SERVER_RATE_LIMITS_AUDIT_WINDOW_SECONDS.

Watch the effect in Prometheus: rate(nova_http_requests_total{status="429"}[5m]) — 429 rejections are counted by the metrics middleware (see monitoring).

Audit trail

Sustained limiting of one key — at least audit_threshold_rejections rejections within one audit_window_seconds window — emits one audit record per key per window (rate_limit_sustained), carrying the key digest (sha256:…), the limit class, the rejection count, and the window start. The raw key value (token, IP) is never stored. A structured warning is always logged for operator visibility.

Honest note: these records go to the same append-only JSONL audit log the server's role-management routes already use. That log is append-only but not hash-chained; the spec's hash-chaining requirement lands when the hash-chained AuditLog is wired into the server app (see the ADR-0179 status note).

2. Storage quotas

Semantics

Quotas bound what the capsule store holds — capsule count and total stored bytes — with warn-then-reject enforcement, checked at capsule ingest only (the capsule write routes):

Usage is derived from the existing capsule store (directories holding a capsule.yaml, plus every file's lstat size; the ingest spool .ingest-tmp/ is excluded, so in-flight uploads never count) — no new ledger to reconcile — and cached with a short monotonic TTL (5 s) so hot ingest paths don't re-count the store per request. The recount runs on a worker thread, one at a time (requests arriving mid-recount use the previous value), and examines at most 1 000 000 entries, past which it enforces against a logged lower bound. Quota audit events carry no key hash: this global track is per-deployment; per-workspace budgets are a separate additive track (ADR-0208, §3 below, experimental) that reads metered counters instead of the store walk.

Enabling

Quotas require the same master switch as rate limiting (rate_limits.enabled: true) plus a quota block with at least one non-zero limit. 0 means unlimited; all-zero limits keep the feature fully inert.

rate_limits:
  enabled: true
  quota:
    max_capsules_soft: 900
    max_capsules_hard: 1000
    max_bytes_soft:  8_000_000_000    # 8 GB
    max_bytes_hard: 10_000_000_000    # 10 GB

Hard limits must be ≥ their soft counterpart (validated at config load).

3. Per-workspace budgets & usage metering

Status: experimental (ADR-0208; normative contract in design/spec/usage-metering-v0.md (private)). Behind the same master switch (rate_limits.enabled); with the switch off there is no accounting, no middleware, and no new tables — zero behavior change.

Usage metering

Every accepted capsule upload is attributed to a workspace at ingest and recorded in an append-only usage_ledger + transactional usage_counters in the registry DB (metrics: capsules_created, bytes_stored = unpacked size, api_requests via a bounded accumulator flushed at most once per usage.flush_interval_s). Attribution order: API-key workspace binding → the principal's single workspace membership → the default workspace — recorded on each row (attribution = key|membership|default), so unattributed usage is visible, not laundered. Counting is idempotent by construction: a retried upload hits the 409-duplicate path, and a replayed ledger insert is a no-op under the (metric, ref) unique index. Deleting a capsule (single or bulk, ADR-0206) appends negative ledger rows so budgets reflect reclaimed storage; capsules uploaded before metering shipped appear only in the derived global/drift figures. A capsule uploaded again after a delete is metered again: each stored lifetime of a run has its own ledger ref (<run_id>, then <run_id>/2, <run_id>/3…), and a delete reverses the latest one.

Monthly rollups freeze each finished period lazily at the first write of the next one (no cron) and are retention-bounded (usage.rollup_retention_months, default 24; raw ledger rows usage.ledger_retention_months, default 3).

Reporting — GET /v0/usage (?period=YYYY-MM, default current): per-workspace metric totals, org rollups, and — for admin/auditor only — the global store-derived figures plus a drift block (derived minus metered). Other principals get a filtered response (their membership workspaces only) — filtering, not 403. Accounting is best-effort relative to the upload: a metering failure is logged + audited and never fails the write.

Reconciliation and chargeback export (ADR-0208 P2/P3)

Status: experimental. Two admin CLI commands read the same registry DB the server meters into (--db-path, else the server config's db_path).

nova server usage reconcile compares the metered lifetime sums with measure_capsule_store and prints the drift (derived minus metered). Both sides cover the whole history: totals of rollups pruned past usage.rollup_retention_months are carried forward (not forgotten), so a retention prune never shows up as drift; the drift block of GET /v0/usage uses the same whole-history figure, so the two agree. Rollups pruned before this carry existed are not recoverable: the first reconcile after upgrading such a deployment may report that history once as drift — inspect before --apply. It is report-only by default. With --apply and a non-zero drift it appends one signed (positive or negative) adjustment row per drifting metric (capsules_created, bytes_stored) to the default workspace only, with attribution = 'reconciliation' and ref = 'recon:<timestamp>'. Existing ledger rows are never rewritten; the counters move only through the appended, visible row, and a second --apply finds zero drift and writes nothing. Concurrent --apply runs are serialized by the SQLite write lock (the metered side is read and the rows appended in one BEGIN IMMEDIATE transaction), so the same drift is never booked twice; an upload in flight during the store scan can still leave a ±1-capsule drift for the next run to report. Two refusals by design: there is no workspace selector (attributing unmetered bytes to a real workspace would be a guess), and a negative adjustment that would take the default workspace below zero is refused (exit 1) — the over-count belongs to another workspace, typically after an out-of-band deletion. Every run appends a usage.reconcile entry to the hash-chained audit log (an unkeyed SHA-256 chain — tamper-evident, not a keyed signature). Note: adjustment rows count toward the default workspace's budget, if one is configured.

nova server usage reconcile                     # report only
nova server usage reconcile --apply --actor alice@example.com

nova server usage export writes one row per (period, org, workspace, metric) for a --from/--to range of YYYY-MM periods, sorted in that order (deterministic: the same DB state gives byte-identical output). Finalized periods come from the monthly rollups (status = final); periods not yet finalized — always the current one — come from the live counters (status = provisional) and can still grow. --format csv (default) is RFC 4180 (CRLF, header row, quoted fields) with formula-injection-safe text cells (a leading =, +, -, @, TAB, CR or LF — checked after NFKC normalization, so full-width ＝＋－＠ count, and also after leading whitespace — gets a ' prefix; integer totals are never rewritten); --format ndjson emits one sorted-key JSON object per line. Read-only. Periods older than usage.rollup_retention_months have been pruned and export empty.

nova server usage export --from 2026-07 --to 2026-09 -o chargeback-q3.csv
nova server usage export --from 2026-09 --format ndjson --workspace ml-platform

HTTP export — GET /v0/usage/export (experimental, ADR-0208 P3) serves the same rows and bytes over the nova server API, as an attachment:

curl -H "Authorization: Bearer $TOKEN" -OJ \
  "https://nova.example.com/v0/usage/export?from=2026-07&to=2026-09&format=csv"
# -> nova-usage-2026-07_2026-09.csv   (text/csv; charset=utf-8; header=present)
curl -H "Authorization: Bearer $TOKEN" \
  "https://nova.example.com/v0/usage/export?from=2026-09&format=ndjson&workspace=ml-platform"
# -> application/x-ndjson

Query parameters: from/to (YYYY-MM; from defaults to the current UTC period, to to from; at most 120 periods), format (csv default, or ndjson), optional exact-match workspace / org. Access follows GET /v0/usage: admin and auditor see every workspace; any other authenticated principal only its membership workspaces — filtering, not 403, and a workspace/org filter never widens that view. A malformed period, inverted or over-wide range, or unknown format is a 400 error envelope (invalid_period, invalid_period_range, invalid_format). The registry is opened read-only (mode=ro): the request never creates the DB or the usage tables, and the membership lookup for non-privileged callers is read-only too. A missing DB or missing usage tables export as zero rows; any other registry failure (locked, I/O error) is a 503 error envelope (usage_store_unavailable) — never an empty billing file. Exports are not audited (neither is the CLI export nor GET /v0/usage); with metering on, the request itself still counts toward api_requests like any other. The rows are materialized and sorted in memory before streaming — bounded by 120 periods × workspaces × metrics.

Per-org budgets and request-time enforcement of the API-key workspace binding shipped as experimental (ADR-0294, below).

Per-workspace budgets

An additive workspaces map inside the existing quota block; the global limits keep their exact meaning and both checks run (strictest wins). Enforcement reads the metered counters, not the store walk:

rate_limits:
  enabled: true
  quota:
    workspaces:              # ADR-0208; absent => byte-identical behavior
      ml-platform:
        max_capsules_soft: 1000
        max_capsules_hard: 2000
        max_bytes_soft: 0    # 0 = unlimited, per existing convention
        max_bytes_hard: 0

Same ladder and contract: soft ⇒ write succeeds + X-NovaFabric-Quota-Warning: <workspace>/<kind> <usage>/<limit> (comma- joined after any global warning) + one audit event per (workspace, kind) per window; hard ⇒ 429 quota_exceeded with an additive workspace field in details, no Retry-After. A config that names a workspace slug that does not exist is refused at startup. Alerting (ADR-0192): a hard rejection fires ops.quota.breached (critical, subject quota:<workspace>:<kind> — per-workspace dedup); the first soft crossing per window fires the same event at warning severity (subject suffixed :soft). Both are no-ops unless NOVA_ALERTS_* is configured.

Per-org budgets (experimental, ADR-0294)

An additive orgs map beside workspaces, with the same four keys:

rate_limits:
  enabled: true
  quota:
    orgs:                    # ADR-0294; absent => byte-identical behavior
      acme:
        max_capsules_soft: 5000
        max_capsules_hard: 8000
        max_bytes_hard: 0    # 0 = unlimited

Org usage is the sum of the all-time metered counters of every workspace in the org (workspace → org resolved the same way attribution resolves it; unknown workspaces count toward default). Global, workspace and org checks all run and the strictest wins. Soft ⇒ the warning header gains an org:<org>/<kind> <usage>/<limit> part; hard ⇒ 429 quota_exceeded with an additive org field in details. Alerts use the subjects quota:org:<org>:<kind> (critical) and …:soft (warning), so an org breach never deduplicates against a workspace with the same slug. An unknown org slug in config is refused at startup. Limit: counters are keyed by workspace slug and slugs are unique only within an org, so two orgs that share a workspace slug share one counter (assigned to the first org by slug).

API-key workspace-binding enforcement (experimental, ADR-0294)

api_keys:
  enforce_workspace_binding: true   # env NOVAFABRIC_SERVER_API_KEYS_ENFORCE_WORKSPACE_BINDING

Off by default. When on, a request authenticated by an API key with a workspace binding gets 403:

Unbound keys and non-key credentials are unaffected. Each refusal is written to the audit log as api_key.binding_refused (refs only, never the key), at most once per (subject, reason, requested workspace) per 60 s. List results are not narrowed to the bound workspace — the capsule store is not partitioned (ADR-0178).

Env overrides: NOVAFABRIC_SERVER_USAGE_{METERING_ENABLED,FLUSH_INTERVAL_S, ACCUMULATOR_MAX_ENTRIES,ROLLUP_RETENTION_MONTHS,LEDGER_RETENTION_MONTHS}. usage.metering_enabled: false keeps rate limits on while switching accounting off.

Honest limits: attribution consumes the ADR-0193 API-key workspace binding, which is enforced at request time only when api_keys.enforce_workspace_binding is on — with it off, metering inherits that gap; this feature meters and scopes, it does not isolate the capsule store (ADR-0178 gap unchanged); api_requests counts are deliberately coarse (bounded accumulator, crash loses at most one interval).

4. What this feature is not

See also