why i built another evals package

In early 2025, at Thoughtworks, we had to take an LLM application to production. This wasn't a demo, nor a spike. We (for the first time) were looking at a real deployment that would take in real traffic, real money moving through it, and real engineers who'd get woken up when it went wrong.

The moment we started planning for go-live, we hit an uncomfortable question: how do we analyse whether this application will perform as good as we think, in front of users, who, at their best, would not use the app to form grammatically correct english sentences, and use code-mixed sentences? And crucially, how do we actually observe and measure that once it's live?

the crowded ecosystem had answers in pieces:

• deepeval answered evaluation. Custom metrics since Mar 2024, synthetic testset generation since Mar 2024. But its observability lived behind Confident AI, a separate paid SaaS. Not something you wire into a self-hosted deployment :)
• Grafana/Phoenix did tracing and dashboards, beautifully. But traces alone don't tell you whether your evals are holding up in production. Dashboards will show you metrics, but do not tell you what to measure.

What didn't exist at the time was the thing we actually needed: a self-hosted, open-source package that did observability and evaluation together on custom datasets on prod data (real questions with no ground truth) and measured custom metrics. merit-ai solved a lot of problems for us, allowing us to generate testsets directly from incoming production data, pull production traffic in (Kafka, log files, API), run it against the test cases and estimate metrics(without ground truth for the actual user questions), watch cost/latency/error rates, and generate reports. All of this without sending data to a cloud you don't control.

So I built it. That's merit-ai.

The timeline, for context

What we shipped

merit-ai was a small, focused solution:

• Monitoring & observability: real-time LLM monitoring, Kafka/log/API collectors, REST API, live dashboard, latency/throughput/error/cost tracking, multi-backend storage (SQLite/MongoDB/file).
• Evaluation — custom bring-your-own RAG metrics (also included correctness, faithfulness, relevance, context precision etc.), LLM-as-a-judge, testset generation.
• A 3-line integration so a working app could be wired into all of it without a rewrite.

It went live for that deployment. It did the job.

Since then, merit-ai has supported evals of a few more client deployments in Thoughtworks. It's been a while, and the industry did move on.

Where did the industry go to

Within weeks of our go-live, the industry caught up. All at once.

• May 2025: deepeval v3.0 shipped open-source tracing and "production-ready observability." The exact layer we'd built because nobody had it open, opened up. We'd read the gap right; the gap just didn't plan on staying a gap.
• 2025–2026: evals and observability stopped being separate categories. The platforms consolidated: evaluation + tracing + datasets + red-teaming + agentic metrics, all in one.
• The "standard metrics" idea mostly disappeared and custom evals became the default.

In under eighteen months, what we'd had to build by hand for one deployment became the default architecture of the ecosystem. merit-ai's job was done.