--- title: "How NovaFabric compares to Langfuse, LangSmith and MLflow" description: "An honest comparison of NovaFabric with Langfuse, LangSmith, MLflow, Weights & Biases and OpenTelemetry, including where NovaFabric is the wrong choice." canonical: https://novafabric.ai/docs/comparison/ last_updated: 2026-10-22 source: https://github.com/MSKazemi/novafabric/blob/main/docs/comparison.md --- # How NovaFabric compares An honest comparison, **including where NovaFabric loses**. If a tool below is a better fit for what you are doing, use it — a wrong recommendation costs you days and costs this project the only currency it has. The short version: most tools in this space answer *"how is my system performing right now?"* NovaFabric answers *"can I prove, replay, and compare a run that happened three months ago?"* Those are different questions, and it is completely reasonable to need both. --- ## At a glance | | **NovaFabric** | Self-hosted observability (Langfuse, Arize Phoenix) | Hosted SaaS (LangSmith, W&B Weave, Helicone) | Experiment tracking (MLflow, DVC) | |---|---|---|---|---| | Deployment | Self-hosted CLI, server optional, **no account** | Self-hosted server + database (required) | Managed cloud | Self-hosted or managed | | Primary artifact | Portable evidence **capsule** — a folder | Trace row in a database | Trace row in a vendor cloud | Run record + artifacts | | Where your data lives | **Your machine** | Your server | Vendor cloud | Your store or vendor | | Replay a past run | **✓** recorded model replies (tools run live) + 3 inspection modes | ✗ | ✗ | partial — re-run a script | | Run-to-run structural diff | **✓** | partial (eval) | partial (eval) | metric comparison only | | Cryptographic signing / provenance | **✓** opt-in: DSSE seal, Sigstore, optional RFC 3161 | ✗ | ✗ | ✗ | | Capture with no code changes | **✓** hooks, wire-level, proxy | SDK instrumentation | SDK or proxy | explicit logging calls | | Works fully offline / air-gapped | **✓** | self-host only | ✗ | partial | | Real-time dashboards & alerting | ✗ **weak** | ✓ strong | ✓ strong | partial | | Hosted multi-user convenience | ✗ **none** | partial | ✓ strong | ✓ | | Prompt playground / A-B testing | ✗ no playground; local, record-only A/B attribution (`nova experiment`, `--variant`; experimental) | ✓ | ✓ | ✗ | | Maturity | **beta, pre-1.0, small** | mature | mature | very mature | --- ## Where NovaFabric is genuinely the wrong choice Use something else if: - **You want live monitoring and alerting.** NovaFabric is not an APM and does not try to be. Langfuse, Phoenix, LangSmith, and Helicone all do this properly. - **You want a hosted service with a team UI and zero operations.** There is no SaaS, no account system, and no plan to build one. - **You want a prompt playground or a hosted A/B testing platform.** Not built, not planned. (Local, record-only variant attribution and dataset-experiment comparison exist — `nova capture --experiment/--variant`, `nova experiment run/compare`, `nova diff --group-by variant` — experimental.) - **You need a mature, battle-tested tool right now.** NovaFabric is pre-1.0. The capsule format is not frozen and will change before v1.0. - **Your workload is a classic ML training sweep and you want metric comparison.** MLflow and W&B are better at exactly that, and have been for years. ## Where NovaFabric is genuinely better - **Showing that a recorded run has not been altered, months later, to someone who does not trust you.** A sealed capsule (sealing is opt-in: a `novaseal.yaml` and your own key) is signed, can carry an RFC 3161 timestamp, and verifies with no server and no network. A seal proves the record is unchanged since signing; it does not prove the record is complete. - **Replaying a run against its recorded model responses** — no live model call and no model tokens spent. OpenAI and Anthropic model calls (sync or async, streamed or not) and MCP tool results are served from the capsule; every other tool runs live. - **Structural diff between two runs** — not "metric A went up", but *what changed in the execution*. - **Air-gapped and regulated environments.** No telemetry, no update checks, no internet requirement for core features. - **HPC and SLURM workloads.** Capture works on a login node with no daemon and no privileged access — a genuinely under-served case. - **Regulatory evidence.** EU AI Act, ISO 42001, CycloneDX AI-BOM exporters run off the same captured facts. --- ## Specific comparisons ### NovaFabric vs Langfuse **Use Langfuse if** you want self-hosted LLM observability with strong dashboards, cost and token analytics, prompt management, and a team UI. It is mature, widely deployed, and better than NovaFabric at all of it. **Use NovaFabric if** you need to replay a run, diff two runs structurally, or hand an auditor a signed Evidence Bundle that verifies without your infrastructure being up. **Use both:** NovaFabric emits OpenTelemetry GenAI spans and OpenLineage events, so it can feed a Langfuse deployment while keeping portable capsules for replay and audit. They are complementary; this is not a migration story. ### NovaFabric vs LangSmith LangSmith is a hosted service tightly integrated with LangChain, with a mature evaluation and tracing product. Its data lives in LangSmith's cloud. The distinction is ownership: with NovaFabric the artifact is a folder on your filesystem that you can `tar` and archive for a decade. If your constraint is *"the data cannot leave our infrastructure"* — regulated industry, national lab, air-gapped cluster — that constraint alone decides it. ### NovaFabric vs MLflow / Weights & Biases Different eras of the same instinct. MLflow and W&B track *experiments*: parameters, metrics, artifacts, model versions, and they are excellent at it. NovaFabric captures *executions*: the full call graph of an agent or job, the environment lock, the secret-scan proof, and, when you seal the capsule, a signature over all of it. If you want to compare learning curves, use MLflow. If you want to keep a recorded run you can replay (tools still run live) and later verify has not been altered since it was sealed, use NovaFabric. Many teams will reasonably run both. ### NovaFabric vs OpenTelemetry alone OpenTelemetry is a standard, not a product, and NovaFabric emits it rather than competing with it. Raw OTel gives you spans in a backend; it does not give you a portable folder, a replay engine, an environment lock, a redaction proof, or a signature. If a span in Jaeger answers your question, you do not need NovaFabric. ### NovaFabric vs "just save the logs" The honest baseline, and it works longer than people expect. It stops working when you need to prove the logs were not edited, re-run the workload without the provider, know exactly which package versions were installed, or show that no secret leaked into the record. Those are the four things a capsule adds. --- ## Frequently asked **Is NovaFabric production-ready?** It is **beta**. Local capture, replay, diff, lineage, and the trust layer are stable and used daily. Server mode, the cluster-scale collector, and the dashboard are `experimental`. The capsule format is **not frozen** — expect schema changes before v1.0. Read [the roadmap](https://github.com/MSKazemi/novafabric/blob/main/ROADMAP.md) before betting a production pipeline on it. **Does it send any data anywhere?** No. No telemetry, no update checks, no analytics. Core local-mode features need no internet at all. **Does it work without a server or database?** Yes — that is the default mode. A server is optional and exists for teams sharing capsules. **Does it capture my prompts and responses?** Yes, inside the capsule. When a Python workload's model calls are captured, the request messages and response text are written to `model-calls.jsonl` in your capsule, on your machine, after the built-in secret scan (18 API-key/token rules; PII masking is a separate opt-in). What is opt-in is sending content anywhere else: `--emit-otel-genai` exports spans without content unless you also pass `--capture-content`. **Which frameworks does it support?** Capture is framework-agnostic — it wraps a command. Eleven experimental adapters add richer detail: LangGraph, CrewAI, AutoGen, DSPy, LlamaIndex, Haystack, Pydantic AI, the OpenAI Agents SDK, Google ADK, Bedrock AgentCore and A2A. MCP tool calls are captured by a hook and by the experimental `nova mcp-proxy`. See [the architecture map](https://github.com/MSKazemi/novafabric/blob/main/docs/architecture.md). **Can I use it with my existing observability stack?** Yes, and that is the intended shape. Emit OTel GenAI to your backend, keep capsules for replay and evidence. --- ## See also - [Benchmarks](https://novafabric.ai/docs/benchmarks/) — reproducible numbers, with the commands - [Concepts](https://novafabric.ai/docs/concepts/) — the five primitives and the replay modes - [Architecture](https://github.com/MSKazemi/novafabric/blob/main/docs/architecture.md) — the subsystem map and design invariants - [Roadmap](https://github.com/MSKazemi/novafabric/blob/main/ROADMAP.md) — what is shipped, experimental, and planned - [Getting started](https://novafabric.ai/docs/getting-started/) *Found something unfair or out of date here? That is a bug — [open an issue](https://github.com/MSKazemi/novafabric/issues/new?template=documentation.yml). Comparisons rot, and a comparison that flatters us is worse than useless.*