Back to all writing in Evaluation
Every model launch comes with a chart. The chart almost always shows the new model on top. The question this pillar takes seriously is: does the chart mean anything?
Sometimes yes. Often no. The gap between “scored well on a benchmark” and “is actually useful for what you want to do” is the single most expensive misunderstanding in applied AI today.
Read these in order
- Why benchmarks lie — contamination, saturation, and the incentive to teach to the test.
- Evals vs. vibes — why you still need to write your own evaluation set, and what a good one looks like.
Where this pillar is going
Future essays will cover LLM-as-a-judge (using one model to grade another, and where that breaks), red-teaming (adversarial evaluation of safety), and human preference data (what RLHF and DPO actually measure).
Citations
- Measuring Massive Multitask Language Understanding (MMLU) — Hendrycks et al., 2020 · accessed May 17, 2026 · view archived copy
- Beyond the Imitation Game (BIG-bench) — Srivastava et al., 2022 · accessed May 17, 2026 · view archived copy
- Chatbot Arena Leaderboard — LMSYS · accessed May 17, 2026 · view archived copy
Essays in this pillar
- Why benchmarks lie
Contamination, saturation, and the incentive to teach to the test — a field guide to reading model launch charts skeptically.
~2 min read
- Evals vs. vibes
Why every serious AI project ends up writing its own evaluation set, and what a good one looks like.
~3 min read