Two years ago you could win enterprise deals with a demo. Today the demo is table stakes, every vendor has one, and buyers have been burned by leaderboard scores that evaporated in production. The differentiator that survived contact with reality is boring: measurement. Whoever can prove their system works on your actual task, with your actual data, wins the contract. Evals went from QA chore to strategic asset, and the labs treating them as such are pulling ahead.
Leaderboards stopped meaning anything
MMLU got saturated. HumanEval got gamed. SWE-bench scores cluster so tightly at the top that sub-point differences are noise wearing a lab coat. Public benchmarks have three terminal diseases: contamination (test items leak into training data), saturation (everyone trains to pass them), and mismatch (they measure what was measurable in 2023, not what your product does in 2026). A lab can post a spectacular number on a public suite and ship a model that flubs your document-review pipeline. Everyone in procurement learned this the expensive way.
There's a fourth disease nobody names politely: Goodhart's law with a budget. Once a suite is famous, entire training runs get aimed at it. The model doesn't get smarter, it gets more benchmark-shaped, and the gap between paper numbers and your Slack channel widens. Some suites have been so thoroughly optimized that labs report "hint-resistant" or "contamination-audited" variants just to publish credible scores. When the score needs an asterisk to be publishable, the suite is done being a yardstick.
This doesn't mean public benchmarks are useless. They're a coarse filter. But nobody with sense makes a purchase decision or a release decision on them anymore, at least nobody serious.
What a real eval looks like
A serious eval is a small, representative slice of the actual task: real documents, real user queries, real failure costs attached. It has three properties most public suites lack. It's private, so it can't leak into training. It's graded, ideally by execution or by a rubric, not by vibes. And it's versioned, because your task drifts and a static test silently rots. The teams with the best evals iterate weekly: new failure modes from production go in as test cases, and every model swap gets run against the whole suite before it ships.
One anti-pattern worth naming: the eval that only ever gets run on release day. That's a gate, not a compass. The useful evals run continuously in the background, wired into dashboards and diff reports, so a slow quality slide shows up weeks before it becomes a churn event. Treat flaky grading and stale cases as incidents, because they are: silent failures of the one system whose entire job is to notice silent failures.
There's a subtle art to sampling that slice. If you just grab the last 200 tickets, you'll over-represent loud users and under-represent the silent majority. Good teams stratify: by task type, by difficulty, by customer segment, by blast radius if the model errs. They also include negative controls, inputs where the right answer is "I don't know" or "escalate to a human," because models that always produce an answer are all risk, no help. That's a surprisingly common eval blind spot.
- Task fidelity: samples from the real distribution, not a cleaned-up toy version.
- Grading rigor: deterministic checks where possible (unit tests, exact match, schema validation), rubric-based LLM judging with spot human audits where not.
- Regression coverage: every past production bug becomes a permanent test case.
- Cost and latency budgets: a model that passes at 10x the latency may still fail your eval in any meaningful sense.
- Adversarial coverage: prompt injections, formatting traps, and hostile inputs your production logs will eventually serve up anyway.
- Human baselines: measure the human alternative, because "beats GPT-4-whatever" matters less than "beats our tired reviewers on Tuesday afternoon."
- Slice-based reporting: aggregate scores hide regressions in small but critical segments; always break results down by language, document type and customer tier.
Why evals compound into a moat
Data and model weights are leaky moats; your fine-tune gets distilled, your open weights get forked. Evals are different. A great eval suite encodes thousands of hours of operational knowledge: which edge cases matter, what "good" means for your domain, where the model bites back. That knowledge lives in your test harness, your graders, your failure taxonomy, and the muscle memory of the team maintaining it. A competitor can copy your model in a weekend. They cannot copy your understanding of what quality means for your customers without doing the same painful homework.
There's also a flywheel effect. Production failures feed the eval. The eval gates releases. Releases fail less often in production. The feedback loop tightens every quarter, and each pass widens the gap between you and teams still eyeballing outputs. This is why the best AI product teams now talk about their eval harness the way they talk about their core infrastructure.
Second-order effect: evals de-risk vendor selection. When every contender must run the same private suite on your data under blind conditions, macho benchmark charts become irrelevant and the better artifact wins quietly. Procurement cycles shorten, and "trust us, it's frontier" stops being a pitch. A few enterprise teams now require eval sign-off before any AI can touch customer data, which is exactly the governance gravity this shift deserves.
The LLM-as-judge trap
Since grading by hand doesn't scale, everyone reaches for an LLM judge. Fine, but know the failure modes: judges favor confident, verbose answers; they share the biases of their own training; they can be prompt-injected by the outputs they're grading. Mitigate with pairwise comparisons instead of absolute scores, rubrics with concrete anchors, randomized order, and regular human spot-checks to calibrate. And never let the judge be the same model family you're evaluating if you can help it. Self-preference bias is real and measurable.
There's also a deeper question: what does your eval optimize for? Plenty of teams discover their suite rewards fast, shallow answers because the graders were written by the same impatient humans who wrote the spec. Ask what behavior each test case actually reinforces, and you'll find a few that reward the wrong thing entirely. Pruning or fixing those is often worth more than adding 50 new cases. An eval is a objective function, and the model will duly optimize it, including its blind spots.
Who owns the evals?
The uncomfortable org-chart question underneath all of this is ownership. Evals sit awkwardly between engineering, product and QA, and things that sit between three groups get owned by nobody. The teams doing this well assign a named owner, often a staff engineer or an applied AI lead, with a real backlog, a weekly failure-review ritual, and authority to block a release. Without that, the suite rots in a quarter, the dashboards go green forever, and you're back to vibes.
There's also a budget question nobody asks out loud. Evals cost real money: human audits, judge inference, engineering time. That spend looks unproductive on a spreadsheet because it produces no features. The rebuttal is arithmetic: one silent production failure in a customer-facing workflow costs more in churn and support load than a year of suite upkeep. Frame it as insurance with a deductible, not as overhead. Plenty of CFOs get this immediately once someone shows them the incident cost from a model swap that nobody evaled.
And a social note: eval results are most useful when they're uncomfortable. A suite that never fails anything is furniture. Celebrate the engineer who adds the case that breaks the release candidate, the same way you'd celebrate a red test that caught a regression. If nobody wants to be the bearer of bad scores, your suite is measuring your courage, not your model. Good evals make a team braver and calmer at the same time, which is a rare and valuable combination.
How to start without boiling the ocean
You don't need a research team. You need one engineer, one afternoon, and your last 50 production incidents. Extract the task inputs and the correct outcomes, grade them mechanically where possible, and call it v0. Then run every model candidate against it before deploying. Add cases weekly. Within a quarter you'll have something no vendor's sales deck can fake: evidence. If you buy models rather than build, demand eval results on your data from every vendor, and walk away from any who won't run the test.
One more trap to avoid: don't confuse eval size with eval quality. Fifty ruthless cases from your worst incidents will outscore five hundred synthetic happy-path questions every time. Small and sharp beats big and bland. And whatever you do, keep the raw transcripts. Numbers tell you a model regressed; the traces tell you why, and "why" is where the fixes live.
Also: timestamp your eval results. A passing score from eight months ago proves nothing about the model serving you today, especially if a vendor has silently swapped versions, which happens more often than the contracts would suggest. Continuous evaluation isn't paranoia; it's the only honest bookkeeping you have.
Start small, keep it honest, and resist the urge to grade your own homework with a model that shares the student's textbook. Your evals are the one place in this industry where rigor is still scarce and therefore genuinely differentiating.
And don't forget migration safety: every model switch should be treated like a database migration, with a dry run, a rollback plan and a blast-radius assessment, because the failure mode is silent wrong answers, not loud crashes.
And share the results, carefully. A one-page internal scorecard that every engineer sees does more for release discipline than any policy memo, because nothing sharpens a team's attention like a visible, dated number that everyone agrees to defend together.
The frontier labs get this too. The quiet secret of the big releases is that the jump you see in quality usually traces to better internal evals enabling more aggressive training, not a 10x architecture breakthrough. Measurement drives optimization. Optimization drives capability. Capability drives revenue. Evals sit at the very top of that stack, and right now it's the cheapest place in AI to build a durable advantage.
If you take one thing from this: treat your eval harness as a product, with a backlog, a changelog and a dedicated owner. Version it. Review regressions in release meetings. Pay it down like tech debt. And when a model launches that beats your numbers, celebrate, then immediately find the failure cases it introduced, because there are always some. The team that sees the cracks first is the team that fixes them first, and that, more than any single model choice, is what "moat" means in AI products this year.