This is the second essay in the Evaluation pillar. The first one, Why benchmarks lie, argued that public benchmark scores tell you less than you think. This one is about what to do instead.

The thing teams actually do

Every team that has shipped an LLM-powered feature for more than a few months has gone through the same arc:

  1. Pick the model with the best public benchmarks.
  2. Ship a prototype. It feels great in the demo.
  3. Real users hit it. Something is subtly wrong — maybe one prompt in twenty produces nonsense, maybe the tone drifts, maybe a regression creeps in after a model update.
  4. Someone on the team starts maintaining a spreadsheet of “prompts that broke.” That spreadsheet, over time, is the eval.

This is not a failure of discipline. It is the correct response to the fact that no public benchmark measures the thing your users actually do. The literature has a name for the spreadsheet: a task-specific evaluation set.

Three kinds of eval, in order of cost

Programmatic checks. The cheapest evals are deterministic. Does the output parse as JSON? Does it match a regex? Did it call the right tool with the right arguments? Tools like EleutherAI’s lm-evaluation-harness exist precisely to make this kind of check easy to run across many models.1 If you can express a quality criterion as code, do it that way — it scales for free.

LLM-as-a-judge. When the criterion is fuzzier — “is this response helpful?”, “is this summary faithful?” — you can use a stronger model to grade the outputs of a weaker one. The MT-Bench paper laid out how this is done and, importantly, where it breaks: judges have position bias, length bias, and a tendency to agree with themselves.2 LLM judges are useful, not infallible. FActScore is a related technique that decomposes long-form answers into atomic claims and checks each one against a knowledge source — a more rigorous form of automated judging for factual tasks.3

Human preference. The gold standard, and the most expensive. Show two outputs to a human and ask which is better. Aggregate over enough comparisons and you get an Elo score. The public version of this is Chatbot Arena, which has collected millions of pairwise judgments and produces the leaderboard most practitioners actually trust.4 The private version of this is hiring annotators (or using your own team) on data that looks like your real workload.

What a good eval set looks like

A useful internal eval has three properties:

  • Drawn from real traffic. Not synthesized, not adapted from a benchmark. Real prompts your users typed, with PII stripped.
  • Stratified by failure mode. If 5% of your traffic is in Spanish, 5% of your eval should be in Spanish.
  • Versioned. When you change the prompt template or swap models, you re-run the same fixed set. The diff is the point.

If you only ever do one of these three, do the first. A hundred real prompts that you grade by hand will tell you more about whether to ship a new model than every public leaderboard combined.

Footnotes

  1. EleutherAI, lm-evaluation-harness repository. ↩

  2. Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, 2023. ↩

  3. Min et al., FActScore, 2023. ↩

  4. Chiang et al., Chatbot Arena, 2024. ↩

Citations

  1. lm-evaluation-harness — EleutherAI · accessed May 17, 2026
  2. Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (Chiang et al., 2024) · accessed May 17, 2026
  3. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., 2023) · accessed May 17, 2026
  4. FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation (Min et al., 2023) · accessed May 17, 2026

More in Evaluation

  • Why benchmarks lie

    Contamination, saturation, and the incentive to teach to the test — a field guide to reading model launch charts skeptically.