---
title: "API Rate Limiting & Storage Quotas"
description: "This guide covers protecting a shared nova server deployment from noisy clients: in-process API rate limiting and warn-then-reject storage quotas…"
canonical: https://novafabric.ai/docs/ops/quotas-and-rate-limits/
last_updated: 2026-10-24
source: https://github.com/MSKazemi/novafabric/blob/main/docs/ops/quotas-and-rate-limits.md
---

# API Rate Limiting & Storage Quotas

This guide covers protecting a shared `nova server` deployment from noisy
clients: in-process API rate limiting and warn-then-reject storage quotas
([ADR-0179](https://novafabric.ai/docs/decisions/); normative
budgets in `design/spec/rate-limiting-quotas-v0.md` (private)).

**Status: experimental** (v0.61, shipped 2026-07-16) — and **disabled by
default**. With the feature off (the default), no middleware is installed at
all: no headers, no timing side effects, zero behavior change on upgrade.

Both tracks apply to the `nova server` app only; the `nova serve` dashboard
is not rate-limited.

---

## 1. Rate limiting

### Design in one paragraph

A stdlib-only **token bucket** per key and limit class, on a monotonic clock,
held in a bounded in-process map (LRU-evicted above 10,000 buckets). It is
deliberately **not distributed** — the supported topology is a single-writer
server, so an in-process limiter is the honest fit (ADR-0179).

### Limit classes and defaults

Every request is classified, in order: exemptions → admin prefixes → OTLP
ingest paths → write verbs → reads.

| Class | Matches | Default rate (tokens/s) | Default burst |
|---|---|---|---|
| `ingest` | write verbs (POST/PUT/…) + any OTLP ingest path, regardless of verb | 100 | 200 |
| `read` | GET / HEAD | 50 | 100 |
| `admin` | `/v0/admin/*`, `/v0/roles/*` | 10 | 20 |

**Always exempt — never limited, never counted, no headers:** `/health`,
`/livez`, `/readyz`, `/metrics`. Probes must not brown out with the API, and
`/metrics` is how you *see* the limiting.

### Bucket keying

The bucket key is resolved per request in strict fallback order — first
available wins:

1. **principal** — the SHA-256 digest of the presented bearer token (the raw
   token is never used as a key value), once a request carrying that token has
   authenticated;
2. **tenant** — a resolved tenant id (the slot exists for the ADR-0178
   workspace model; no server surface populates it before routing today);
3. **client IP** — unauthenticated surfaces, and every request whose bearer
   token has not yet authenticated.

A token's first request therefore draws from its client's IP bucket; once it
authenticates, the token has its own bucket. A client that sends a different
invalid token on every request is limited by its IP and creates no buckets.
With `insecure_no_auth` no token names a principal, so all requests are keyed
by client IP.

Classes never share buckets, so an ingest storm cannot starve reads for the
same client.

### The 429 contract

A limited request receives `429` with the standard ADR-0017 error envelope
(`error.code = "rate_limited"`) plus:

- `Retry-After` — whole seconds until one token is available (ceiling);
- `X-RateLimit-Limit` — the class burst (bucket capacity);
- `X-RateLimit-Remaining` — whole tokens still available;
- `X-RateLimit-Reset` — delta-seconds until the bucket is full again.

Successful responses on classed routes carry the three `X-RateLimit-*`
headers too, so well-behaved clients can pace themselves before hitting 429.

### Enabling and tuning

In `~/.config/novafabric/server.yaml` (or the `--config` file), as a **top-level** `rate_limits:` block:

```yaml
rate_limits:
  enabled: true
  ingest: { rate: 100, burst: 200 }
  read:   { rate: 50,  burst: 100 }
  admin:  { rate: 10,  burst: 20 }
  audit_threshold_rejections: 100   # rejections per key per window that trigger an audit event
  audit_window_seconds: 60
```

or via environment overrides: `NOVAFABRIC_SERVER_RATE_LIMITS_ENABLED=1`,
`NOVAFABRIC_SERVER_RATE_LIMITS_{INGEST,READ,ADMIN}_{RATE,BURST}`,
`NOVAFABRIC_SERVER_RATE_LIMITS_AUDIT_THRESHOLD_REJECTIONS`,
`NOVAFABRIC_SERVER_RATE_LIMITS_AUDIT_WINDOW_SECONDS`.

Watch the effect in Prometheus:
`rate(nova_http_requests_total{status="429"}[5m])` — 429 rejections are
counted by the metrics middleware (see [monitoring](https://novafabric.ai/docs/ops/monitoring/)).

### Audit trail

Sustained limiting of one key — at least `audit_threshold_rejections`
rejections within one `audit_window_seconds` window — emits **one audit
record per key per window** (`rate_limit_sustained`), carrying the key
*digest* (`sha256:…`), the limit class, the rejection count, and the window
start. The raw key value (token, IP) is never stored. A structured warning is
always logged for operator visibility.

**Honest note:** these records go to the same append-only JSONL audit log the
server's role-management routes already use. That log is append-only but
**not hash-chained**; the spec's hash-chaining requirement lands when the
hash-chained `AuditLog` is wired into the server app (see the ADR-0179
status note).

## 2. Storage quotas

### Semantics

Quotas bound what the capsule store holds — capsule **count** and total
stored **bytes** — with **warn-then-reject** enforcement, checked at capsule
ingest only (the capsule write routes):

- **Soft limit reached** (`usage >= soft`) — the write **succeeds**, and the
  response carries a warning header:
  `X-NovaFabric-Quota-Warning: capsules 950/1000` (comma-joined when both
  kinds warn). One audit event per kind per audit window
  (`quota_soft_exceeded`).
- **Hard limit reached** (`usage >= hard`) — the write is **rejected** with
  `429`, `error.code = "quota_exceeded"`, and `details` carrying
  `kind` / `usage` / `limit`. There is deliberately **no `Retry-After`
  header** — a quota does not decay on a clock; the remedy is deleting or
  archiving capsules, or raising the limit. Audit event:
  `quota_hard_exceeded`.

Usage is **derived from the existing capsule store** (directories holding a
`capsule.yaml`, plus every file's `lstat` size; the ingest spool
`.ingest-tmp/` is excluded, so in-flight uploads never count) — no new ledger
to reconcile — and cached with a short monotonic TTL (5 s) so hot ingest paths
don't re-count the store per request. The recount runs on a worker thread, one
at a time (requests arriving mid-recount use the previous value), and examines
at most 1 000 000 entries, past which it enforces against a logged lower bound.
Quota audit events carry no key hash: this global
track is **per-deployment**; per-workspace budgets are a separate additive
track (ADR-0208, §3 below, **experimental**) that reads metered counters
instead of the store walk.

### Enabling

Quotas require the same master switch as rate limiting
(`rate_limits.enabled: true`) plus a `quota` block with at least one non-zero
limit. `0` means unlimited; all-zero limits keep the feature fully inert.

```yaml
rate_limits:
  enabled: true
  quota:
    max_capsules_soft: 900
    max_capsules_hard: 1000
    max_bytes_soft:  8_000_000_000    # 8 GB
    max_bytes_hard: 10_000_000_000    # 10 GB
```

Hard limits must be ≥ their soft counterpart (validated at config load).

## 3. Per-workspace budgets & usage metering

**Status: experimental** ([ADR-0208](https://novafabric.ai/docs/decisions/);
normative contract in `design/spec/usage-metering-v0.md` (private)). Behind the same
master switch (`rate_limits.enabled`); with the switch off there is no
accounting, no middleware, and no new tables — zero behavior change.

### Usage metering

Every accepted capsule upload is attributed to a workspace at ingest and
recorded in an append-only `usage_ledger` + transactional `usage_counters`
in the registry DB (metrics: `capsules_created`, `bytes_stored` = unpacked
size, `api_requests` via a bounded accumulator flushed at most once per
`usage.flush_interval_s`). Attribution order: API-key workspace binding →
the principal's single workspace membership → the `default` workspace —
recorded on each row (`attribution = key|membership|default`), so
unattributed usage is visible, not laundered. Counting is idempotent by
construction: a retried upload hits the 409-duplicate path, and a replayed
ledger insert is a no-op under the `(metric, ref)` unique index. Deleting a
capsule (single or bulk, ADR-0206) appends **negative** ledger rows so
budgets reflect reclaimed storage; capsules uploaded before metering shipped
appear only in the derived `global`/`drift` figures. A capsule uploaded again
after a delete is metered again: each stored lifetime of a run has its own ledger
`ref` (`<run_id>`, then `<run_id>/2`, `<run_id>/3`…), and a delete reverses the
latest one.

Monthly rollups freeze each finished period lazily at the first write of the
next one (no cron) and are retention-bounded
(`usage.rollup_retention_months`, default 24; raw ledger rows
`usage.ledger_retention_months`, default 3).

**Reporting — `GET /v0/usage`** (`?period=YYYY-MM`, default current):
per-workspace metric totals, org rollups, and — for admin/auditor only — the
`global` store-derived figures plus a `drift` block (derived minus metered).
Other principals get a **filtered** response (their membership workspaces
only) — filtering, not 403. Accounting is best-effort relative to the
upload: a metering failure is logged + audited and never fails the write.

### Reconciliation and chargeback export (ADR-0208 P2/P3)

**Status: experimental.** Two admin CLI commands read the same registry DB
the server meters into (`--db-path`, else the server config's `db_path`).

**`nova server usage reconcile`** compares the metered **lifetime** sums
with `measure_capsule_store` and prints the drift (derived minus metered).
Both sides cover the whole history: totals of rollups pruned past
`usage.rollup_retention_months` are carried forward (not forgotten), so a
retention prune never shows up as drift; the `drift` block of
`GET /v0/usage` uses the same whole-history figure, so the two agree. Rollups pruned before this carry existed
are not recoverable: the first reconcile after upgrading such a deployment
may report that history once as drift — inspect before `--apply`. It is
**report-only by default**. With `--apply` and a non-zero drift it **appends** one signed
(positive or negative) adjustment row per drifting metric
(`capsules_created`, `bytes_stored`) to the **`default` workspace only**,
with `attribution = 'reconciliation'` and `ref = 'recon:<timestamp>'`.
Existing ledger rows are never rewritten; the counters move only through the
appended, visible row, and a second `--apply` finds zero drift and writes
nothing. Concurrent `--apply` runs are serialized by the SQLite write
lock (the metered side is read and the rows appended in one `BEGIN
IMMEDIATE` transaction), so the same drift is never booked twice; an upload
in flight during the store scan can still leave a ±1-capsule drift for the
next run to report. Two refusals by design: there is no workspace selector (attributing
unmetered bytes to a real workspace would be a guess), and a negative
adjustment that would take the `default` workspace below zero is refused
(exit `1`) — the over-count belongs to another workspace, typically after an
out-of-band deletion. Every run appends a `usage.reconcile` entry to the
hash-chained audit log (an *unkeyed* SHA-256 chain — tamper-evident, not a
keyed signature). Note: adjustment rows count toward the `default`
workspace's budget, if one is configured.

```bash
nova server usage reconcile                     # report only
nova server usage reconcile --apply --actor alice@example.com
```

**`nova server usage export`** writes one row per
`(period, org, workspace, metric)` for a `--from`/`--to` range of
`YYYY-MM` periods, sorted in that order (deterministic: the same DB state
gives byte-identical output). Finalized periods come from the monthly
rollups (`status = final`); periods not yet finalized — always the current
one — come from the live counters (`status = provisional`) and can still
grow. `--format csv` (default) is RFC 4180 (CRLF, header row, quoted
fields) with **formula-injection-safe** text cells (a leading `=`, `+`, `-`,
`@`, TAB, CR or LF — checked after NFKC normalization, so full-width
`＝＋－＠` count, and also after leading whitespace — gets a `'` prefix;
integer totals are never rewritten);
`--format ndjson` emits one sorted-key JSON object per line. Read-only.
Periods older than `usage.rollup_retention_months` have been pruned and
export empty.

```bash
nova server usage export --from 2026-07 --to 2026-09 -o chargeback-q3.csv
nova server usage export --from 2026-09 --format ndjson --workspace ml-platform
```

**HTTP export — `GET /v0/usage/export`** (**experimental**, ADR-0208 P3)
serves the same rows and bytes over the `nova server` API, as an attachment:

```bash
curl -H "Authorization: Bearer $TOKEN" -OJ \
  "https://nova.example.com/v0/usage/export?from=2026-07&to=2026-09&format=csv"
# -> nova-usage-2026-07_2026-09.csv   (text/csv; charset=utf-8; header=present)
curl -H "Authorization: Bearer $TOKEN" \
  "https://nova.example.com/v0/usage/export?from=2026-09&format=ndjson&workspace=ml-platform"
# -> application/x-ndjson
```

Query parameters: `from`/`to` (`YYYY-MM`; `from` defaults to the current UTC
period, `to` to `from`; at most 120 periods), `format` (`csv` default, or
`ndjson`), optional exact-match `workspace` / `org`. Access follows
`GET /v0/usage`: admin and auditor see every workspace; any other
authenticated principal only its membership workspaces — filtering, not
403, and a `workspace`/`org` filter never widens that view. A malformed
period, inverted or over-wide range, or unknown format is a `400` error
envelope (`invalid_period`, `invalid_period_range`, `invalid_format`). The
registry is opened **read-only** (`mode=ro`): the request never creates the
DB or the usage tables, and the membership lookup for non-privileged callers
is read-only too. A missing DB or missing usage tables export as zero rows;
any other registry failure (locked, I/O error) is a `503` error envelope
(`usage_store_unavailable`) — never an empty billing file. Exports are not audited (neither is the CLI export
nor `GET /v0/usage`); with metering on, the request itself still counts
toward `api_requests` like any other. The rows are materialized and sorted
in memory before streaming — bounded by 120 periods × workspaces × metrics.

Per-org budgets and request-time enforcement of the API-key workspace
binding shipped as **experimental** (ADR-0294, below).

### Per-workspace budgets

An additive `workspaces` map inside the existing quota block; the global
limits keep their exact meaning and both checks run (strictest wins).
Enforcement reads the **metered** counters, not the store walk:

```yaml
rate_limits:
  enabled: true
  quota:
    workspaces:              # ADR-0208; absent => byte-identical behavior
      ml-platform:
        max_capsules_soft: 1000
        max_capsules_hard: 2000
        max_bytes_soft: 0    # 0 = unlimited, per existing convention
        max_bytes_hard: 0
```

Same ladder and contract: soft ⇒ write succeeds +
`X-NovaFabric-Quota-Warning: <workspace>/<kind> <usage>/<limit>` (comma-
joined after any global warning) + one audit event per (workspace, kind)
per window; hard ⇒ `429 quota_exceeded` with an additive `workspace` field
in `details`, no `Retry-After`. A config that names a workspace slug that
does not exist is refused at startup. Alerting (ADR-0192): a hard rejection
fires `ops.quota.breached` (critical, subject `quota:<workspace>:<kind>` —
per-workspace dedup); the first soft crossing per window fires the same
event at `warning` severity (subject suffixed `:soft`). Both are no-ops
unless `NOVA_ALERTS_*` is configured.

### Per-org budgets (experimental, ADR-0294)

An additive `orgs` map beside `workspaces`, with the same four keys:

```yaml
rate_limits:
  enabled: true
  quota:
    orgs:                    # ADR-0294; absent => byte-identical behavior
      acme:
        max_capsules_soft: 5000
        max_capsules_hard: 8000
        max_bytes_hard: 0    # 0 = unlimited
```

Org usage is the sum of the all-time metered counters of every workspace in
the org (workspace → org resolved the same way attribution resolves it;
unknown workspaces count toward `default`). Global, workspace and org checks
all run and the strictest wins. Soft ⇒ the warning header gains an
`org:<org>/<kind> <usage>/<limit>` part; hard ⇒ `429 quota_exceeded` with
an additive `org` field in `details`. Alerts use the subjects
`quota:org:<org>:<kind>` (critical) and `…:soft` (warning), so an org breach
never deduplicates against a workspace with the same slug. An unknown org
slug in config is refused at startup. Limit: counters are keyed by workspace
slug and slugs are unique only within an org, so two orgs that share a
workspace slug share one counter (assigned to the first org by slug).

### API-key workspace-binding enforcement (experimental, ADR-0294)

```yaml
api_keys:
  enforce_workspace_binding: true   # env NOVAFABRIC_SERVER_API_KEYS_ENFORCE_WORKSPACE_BINDING
```

Off by default. When on, a request authenticated by an API key **with** a
workspace binding gets `403`:

- `workspace_binding_invalid` — the bound workspace does not exist (or the
  workspace store cannot be read: fail closed);
- `workspace_binding_mismatch` — the request declares another workspace via
  the `X-NovaFabric-Workspace` header or the `workspace` query parameter
  (`details: {binding, requested, source}`).

Unbound keys and non-key credentials are unaffected. Each refusal is written
to the audit log as `api_key.binding_refused` (refs only, never the key),
at most once per (subject, reason, requested workspace) per 60 s. List
results are **not** narrowed to the bound workspace — the capsule store is
not partitioned (ADR-0178).

Env overrides: `NOVAFABRIC_SERVER_USAGE_{METERING_ENABLED,FLUSH_INTERVAL_S,`
`ACCUMULATOR_MAX_ENTRIES,ROLLUP_RETENTION_MONTHS,LEDGER_RETENTION_MONTHS}`.
`usage.metering_enabled: false` keeps rate limits on while switching
accounting off.

**Honest limits:** attribution consumes the ADR-0193 API-key workspace
binding, which is enforced at request time only when
`api_keys.enforce_workspace_binding` is on — with it off, metering inherits
that gap; this feature meters and scopes, it does **not** isolate
the capsule store (ADR-0178 gap unchanged); `api_requests` counts are
deliberately coarse (bounded accumulator, crash loses at most one interval).

## 4. What this feature is not

- **Not distributed.** Buckets and quota caches are in-process. If you run
  multiple server replicas (not the supported single-writer topology),
  limits apply per replica.
- **Not applied to `nova serve`.** The dashboard app has no limiter.
- **Not tenant isolation.** Per-workspace budgets and metering (§3) are
  scoping and accounting on top of the shared capsule store — the ADR-0178
  isolation gap is unchanged and Security-Architect gated.

---

## See also

- [Monitoring & self-observability](https://novafabric.ai/docs/ops/monitoring/) — how to see 429s and
  request rates
- [Server admin guide](https://novafabric.ai/docs/ops/server-admin-guide/) — roles, tokens, and audit
  trails
- [ADR-0179](https://novafabric.ai/docs/decisions/) — the
  decision record; `design/spec/rate-limiting-quotas-v0.md` (private) — normative
  defaults and the 429/quota contracts