Hugging Face

Status: experimental. Implemented and tested offline against the real smolagents 1.26.0 and huggingface_hub 2.3.0 libraries, but not yet in a release — the entry points below exist on main only. Every feature on this page is labelled with exactly one of works today, experimental, planned or future design; where a label is not stated, it is experimental.

NovaFabric is an independent open-source project. It is not affiliated with, certified by or endorsed by Hugging Face.

NovaFabric records and verifies evidence about runs that use Hugging Face models, datasets, inference and smolagents. It does not train, serve or orchestrate anything, and it does not decide whether a model is safe or was lawfully trained. None of this needs a Hugging Face account, and none of it is required: a run that never touches these entry points is captured exactly as before.

What is recorded, and what is not

Surface Recorded Replay Status
smolagents agent run (wrap_agent) one capsule per run(); a span per step; one tool-call record per tool call named in a step (name, argument digest, step outcome) not intercepted — tools run live experimental
InferenceClient.chat_completion through the Inference Providers router (https://router.huggingface.co/…, auto-routed or an explicit provider) the request, requested model and selector, status, timing not intercepted — runs live experimental
the same with stream=True, and AsyncInferenceClient the request at send time; the stream is never read not intercepted experimental
Dedicated Inference Endpoint (https://*.endpoints.huggingface.cloud/…) as above, with system huggingface.endpoint not intercepted experimental
A custom base_url not captured unless you add it through a URL-registry override — —
Hub metadata, dataset and weight downloads never recorded — —
Hub model, dataset or space revision references an opt-in receipt you create yourself (below) — experimental
Evaluation receipts and .eval_results export an opt-in receipt over an existing experiment (below) — experimental

Mocked replay never serves Hugging Face model calls or smolagents tools. Both are listed in the replay support matrix as capture only; a replay runs them live and reports their connections.

smolagents

pip install novafabric smolagents    # smolagents is not a NovaFabric extra
from novafabric.adapters.smolagents import wrap_agent   # or: from novafabric.adapters import wrap_smolagents

agent = wrap_agent(agent, run_name="support-bot")
answer = agent.run("Is SKU-1042 in stock?")

wrap_agent patches run in place and returns the same object (wrapping twice is a no-op). run keeps its arguments, return value (including RunResult), exceptions and the laziness of stream=True. Without smolagents installed the module still imports and wrap_agent raises ImportError naming the install command. The run name defaults to the agent's name, then its class.

Each run() writes one Run Capsule to data_dir, or to $NOVAFABRIC_HOME/runs/<run-id>/ / ./.novafabric/runs/<run-id>/ when none is given (the same defaults as the other framework adapters, see CLI reference § Framework Adapters). It holds:

Model calls are not written by the adapter. The wire-level hooks record them (see below), so a call is never counted twice.

Managed (sub-)agents in agent.managed_agents are wrapped recursively and appear as smolagents.managed_agent.run spans inside the parent's capsule, not as capsules of their own, including when smolagents runs them from a worker thread.

What this adapter cannot see

Try it without a model or a token

The script below needs no network, no Hugging Face token and no model: a scripted model answers a toy stock question in two steps. It was run against smolagents 1.26.0 and produced one capsule with three spans under the run span (two action steps and the final-answer marker) and two tool-call records (lookup, final_answer).

import tempfile
from pathlib import Path

from smolagents import Tool, ToolCallingAgent, models

from novafabric.adapters.smolagents import wrap_agent


class Lookup(Tool):
    name = "lookup"
    description = "Look up the stock level of a SKU."
    inputs = {"sku": {"type": "string", "description": "the SKU"}}
    output_type = "string"

    def forward(self, sku: str) -> str:
        return f"{sku}: 0 on hand"  # invented data


def _reply(name: str, arguments: dict) -> models.ChatMessage:
    call = models.ChatMessageToolCall(
        id=f"call-{name}", type="function",
        function=models.ChatMessageToolCallFunction(name=name, arguments=arguments),
    )
    return models.ChatMessage(role="assistant", content="", tool_calls=[call])


class ScriptedModel(models.Model):
    def __init__(self) -> None:
        super().__init__(model_id="scripted")
        self._queue = [
            _reply("lookup", {"sku": "SKU-1042"}),
            _reply("final_answer", {"answer": "SKU-1042 is out of stock."}),
        ]

    def generate(self, messages, stop_sequences=None, response_format=None,
                 tools_to_call_from=None, **kwargs):
        return self._queue.pop(0)


out = Path(tempfile.mkdtemp(prefix="smolagents-capture-"))
agent = wrap_agent(
    ToolCallingAgent(tools=[Lookup()], model=ScriptedModel(), verbosity_level=-1),
    run_name="stock-bot",
    data_dir=out,
)
print(agent.run("Is SKU-1042 in stock?"))
print("capsule:", next(out.iterdir()))

Read trace.jsonl and tool-calls.jsonl in the printed directory. The capsule is unsealed, as every default capture is (sealing is opt-in); nova verify <capsule> says so rather than passing it.

Inference capture (InferenceClient, router and Endpoints)

Nothing to wrap: when NovaFabric's wire hooks are active (nova capture, or an adapter capsule), a request to either of two HTTPS hosts is recorded as a model call:

These two registry entries use match_kind: regex, which is anchored at the start of the URL: it pins the scheme and the whole host. A URL that merely contains one of these in its path, query or fragment, a host that continues the name (router.huggingface.co.example), and plain http:// are not recorded. The match is deliberately not a bare huggingface.co: the Hub itself (metadata, datasets, weight files) lives there and is not a model call. match_kind is an optional registry field; every existing entry stays a substring match.

The wire layer reads the request, not the response. A Hugging Face model-call record therefore carries an io.novafabric.model_identity block that says exactly what is and is not known:

Field Value Meaning
requested_model the model you sent, minus a routing selector (router only) observed
requested_selector / selector_kind fastest, cheapest, preferred (a policy) or a known provider name (an explicit provider); any other :tag such as :latest stays part of the model; none for a dedicated endpoint what you asked for — not what happened
reported_model null not observed — the response is never read
served_provider null not observed — the router selects a provider server-side, and the hook does not read the response
identity_source request_only

gen_ai.response.model on every wire record mirrors the request; for Hugging Face records read this block instead. Token counts are not recorded either. A Hub commit is a repository revision, not proof of which weights a provider served.

Limits: every classified route, including embeddings, is labelled operation: chat; the Responses API (beta) has no route-specific handling. Replay never serves these records (capture only).

Privacy: what a capsule holds by default

This is a project-wide default, not something specific to Hugging Face. Wire-level model-call capture (model-calls.jsonl) stores request messages — prompt text — at every capture level, for every provider including Hugging Face; NOVA_CAPTURE_LEVEL does not change that (CLI reference § nova policy capture-level). For Hugging Face only the request is recorded, so no completion text is stored from the inference hook. If prompts are sensitive, do not share the capsule unreviewed.

The Authorization header is not written to a capsule. The secret scanner includes a rule for Hugging Face tokens, so a token pasted into a prompt is redacted at finalisation — scanning is rule-based and is not proof that no secret remains.

Hub revision receipts

Experimental. Python API only — there is no nova command for this, and capture does not create receipts for you. You declare a reference, optionally resolve it, and attach the result to a manifest yourself.

from novafabric.interop.hf_hub import (
    attach_hub_references, declare_reference, resolve_reference,
)

ref = declare_reference("dataset", "owner/name", revision="main", config="default", split="test")
ref = resolve_reference(ref)          # the ONLY call here that touches the network
manifest = attach_hub_references(manifest, [ref])   # returns a copy under extensions["dev.novafabric.huggingface-hub"]

resolve_reference is anonymous by default (token=False is passed to the Hub client, so an ambient HF_TOKEN is not sent; pass token= for a private or gated repo) and needs the optional huggingface_hub package. It never raises for a Hub condition: the outcome and a reason from a closed vocabulary (offline, not_found, unauthorized_or_gated, rate_limited, hub_unreachable, …) are recorded in the receipt, and HF_HUB_OFFLINE makes it return unavailable/offline without importing the client. No text supplied by the Hub or by an exception is copied into the receipt.

Each receipt carries one of three strengths, with the claim it is allowed to make stamped beside it:

Strength Meaning
declared_only You stated it. Nothing was checked.
hub_resolved The Hub returned a commit for the revision at resolved_at. A branch or tag can move later; only a revision you gave as a full 40-hex commit is pinned.
local_bytes_verified Files you hold hash to the SHA-256 the Hub published (Git LFS) for that revision; covers only those files (verify_local_file). A digest you supply yourself is recorded as declared and can never raise the strength.

check_revision_drift(previous, current) answers same, moved or unknown for two resolutions of the same reference. Offline nova verify never needs the Hub, and a capsule without the extension is valid and unchanged.

Evaluation receipts and .eval_results

Experimental. Python API only. build_eval_receipt binds an existing, finalized experiment (the one the eval subsystem already produced) to a pinned dataset reference, and optionally a model reference. Every number in the receipt is the experiment's own — counts and values are copied, the 95% Wilson interval is computed from them — and the receipt lists what it does not establish, such as a moving dataset revision, items that produced no score, a small sample, an unpinned harness, or a model declared rather than resolved.

from novafabric.eval.experiment import load_experiment
from novafabric.eval.hf_receipt import build_eval_receipt, to_eval_results, write_eval_results

experiment = load_experiment("<experiment-id>")            # must be finalized
receipt = build_eval_receipt(experiment, dataset_ref)      # dataset_ref: a resolved "dataset" reference
entries = to_eval_results(receipt, metric="exact_match", task_id="qa_test", on="2026-10-28")
write_eval_results(entries, "out/.eval_results/acme-run.yaml")

A sample of what to_eval_results produced against a small offline experiment (the dataset id and commit are synthetic):

- dataset:
    id: acme/qa
    task_id: qa_test
    revision: cccccccccccccccccccccccccccccccccccccccc
  value: 1.0
  notes: Recorded by NovaFabric, not a Hugging Face verified score. exact_match pass_rate
    (fraction in [0, 1]), n=3 of 3 items. 95% Wilson interval [0.439, 1.000]. experiment
    sha256:a1cc960dbcdf344efc585a2a0eb6d2dfeeae3b6d2c52dab00008a316f79e315b
  date: '2026-10-28'

A receipt is evidence recorded by NovaFabric. It is not a Hugging Face verified score, and it does not show that the benchmark is uncontaminated.

Known limits