Every enterprise AI decision eventually collapses into one uncomfortable question: which model do we actually bet on? The vendor decks say GPT-class X, the benchmark aggregators say Claude-class Y, and your data-science lead says "it depends," which is honest and useless. Public benchmarks are saturated, contaminated and increasingly gamed. The only eval that matters is your workload. Here's how to build one, what to measure, and how to keep your optionality while the frontier keeps moving.
Stop shopping by leaderboard
LMArena, MMLU, GPQA, HumanEval, SWE-bench — these are useful for lab-vs-lab marketing, and nearly useless for procurement. Benchmarks leak into training data, labs over-index on them, and scores compress at the top so that a 2-point gap is statistically meaningless. Worse, none of them measure what you actually care about: whether the model completes your tasks correctly, safely, at acceptable latency and cost, inside your tooling. Anthropic, OpenAI, Google and the open-weights shops all publish impressive tables. Your mileage will differ, mostly.
The right frame: evaluation is a product decision disguised as a research task. You're choosing a component for a system, and components are judged in-system, not in vitro.
Build an internal task suite
This is the single highest-leverage thing you can do, and it takes a week, not a quarter. Collect 50 to 200 representative tasks from your real workload: real tickets, real documents, real code reviews, real agent trajectories with tool calls. Label them with success criteria your team actually agrees on — exact match where possible, rubric-scored judging where necessary, or tests-that-run where you're doing code. Then run candidate models against this suite on every release.
A few practical rules:
- Golden tasks, frozen. Once in the suite, don't tweak them to favor a model. Freeze the set and version it like code.
- Weight by business value. Ten tasks pulled from your highest-volume automated flow matter more than a hundred rare edge cases. Score weighted.
- Include failure-shaped tasks. Prompt-injection attempts, ambiguous instructions, missing context. Any model that fails these confidently is a liability.
- Judge with a rubric, not vibes. If you use an LLM judge, calibrate it against human labels on a subsample. Judge drift is real.
Measure what the invoice and the SLO care about
Capability is table stakes; economics decide deployments. For every candidate, capture:
- Cost per successful task, not cost per token. A model twice the price that finishes in one shot beats a cheaper model that needs three retries and a human rescue.
- Latency at p50 and p95, end to end, through your actual gateway and tool-call loop. Providers' headline numbers are single-request, best-case.
- Throughput under concurrency. Can it sustain your Monday-morning peak? Rate limits are part of the product.
- Context behavior. Long-context claims degrade silently. Test recall at 128k, not at 8k.
- Tool-use reliability. Function-calling correctness across your schema set, including malformed-argument recovery.
Two models with identical benchmark scores routinely differ by 3-5x on cost-per-task once retries, verbosity and tool-loop failures are priced in. That's the number your CFO wants, and nobody publishes it.
Evaluate the provider, not just the model
An enterprise model choice is a vendor commitment wearing a technical costume. Check the things that bite at renewal time: data retention and training-on-your-data terms, regional availability and data residency, SLA credits that mean nothing but still signal seriousness, deprecation policies (how much notice before a snapshot sunsets), and enterprise support that answers on a Tuesday at 3am. Also check the boring one everyone skips: does the fine-tuning story actually work for your data volumes and does the vendor let you export or replicate it elsewhere if you leave?
Keep switching cheap
The frontier moves every few months. A rational strategy assumes you'll switch at least once a year, so make switching a config change:
- Put a gateway between your app and the model APIs — your own, or a managed one (OpenRouter, LiteLLM, Bedrock, Portkey, take your pick).
- Standardize on a tool-calling interface and structured outputs; avoid provider-specific prompt incantations baked into business logic.
- Version prompts next to the model ID in config, so "model + prompt" is one deployable unit.
- Re-run the internal suite on every new candidate or model version. Regression on your suite is a hard stop, even if the public numbers improved.
Teams that do this ship model upgrades in days. Teams that don't treat every switch as a quarter-long migration and freeze in place while the ground moves under them.
A common failure pattern: pilot purgatory
The most expensive mistake enterprises make isn't picking the wrong model; it's running endless pilots that never harden into production. A three-week spike proves a model can answer a demo prompt, leadership gets excited, and then the project stalls in security review, integration work and eval ambiguity. The fix is procedural, not technical. Define the acceptance bar before the pilot starts, score against your internal suite on week one, and make the go/no-go a meeting with a number on the table. If your suite is any good, "the model fails tasks 14 and 31" beats "the vibe is off" in every budget conversation that follows.
Related: keep humans in the loop exactly as long as the eval data says they're needed, then take them out. Many teams leave a human reviewer sitting on every model output forever because it feels prudent. Sometimes it is. Often it's a staffing decision hiding inside a quality decision, and it silently erases the ROI you were promised.
Governance and review: the unglamorous multiplier
Evaluation isn't a one-time gate; it's an operating loop. Assign an owner for the eval suite the way you'd assign an owner for uptime. Log production failures back into the task set every sprint. Review cost and latency dashboards monthly, not quarterly, because a provider's pricing or capacity change can silently blow up a unit-economics model that looked fine at launch. And keep a written record of why you chose what you chose — when the CFO or the board asks in eighteen months, "we ran the suite and here's the delta" is a much better answer than "the demos were impressive."
One more governance note: involve legal and security before the pilot, not after. Data-flow review, retention terms, and audit logging for model outputs are cheap up front and ruinously expensive retrofitted. The enterprises moving fastest on frontier models are the ones who made the boring approvals routine, not the ones who skipped them.
A worked example: evaluating a coding agent
Suppose you're picking between two frontier models for an internal code-review agent. Don't ask which is smarter in the abstract; build the suite. Pull 80 real pull requests from your repo, half with subtle bugs injected, half clean. Success criteria: did the model flag the injected bug, did it avoid crying wolf on the clean PR, and was its suggested fix actually applicable? Run both models, three seeds each, temperature fixed. Score precision and recall on bug detection, note the latency at p95 through your real tooling, and price the whole run. One provider will usually win on accuracy-at-cost by an uncomfortable margin, and that provider may not be the one with the louder launch event. This exact pattern — real artifacts, frozen set, weighted scoring — works for support-ticket triage, contract extraction, analyst summarization, and RAG over your internal docs. The shape of the eval stays constant; only the artifacts change.
The questions to ask every vendor
- Where does inference physically run, and in which regions?
- What is your data-retention and training-use policy, in the contract, not the FAQ?
- How much notice do we get before a model snapshot is deprecated?
- What are the rate limits at our tier, and do they scale with our spend?
- Can we export our fine-tuned weights or replicate the pipeline elsewhere?
- What does support look like on a Sunday during an incident?
If a vendor can't answer all six in writing, you don't have a vendor, you have a science project with an invoice.
Open-weights changes the math
DeepSeek, Qwen, gpt-oss and the Llama line mean "frontier-adjacent self-hosted" is a real option now for workloads that tolerate slightly-lower-ceiling quality in exchange for full data control, flat infrastructure cost, and no per-token surprises. The evaluation bar there shifts: you own the ops, the quantization choices, the serving stack (vLLM, SGLang, TensorRT-LLM), and the safety layer your provider used to handle. For many extraction and classification pipelines, an 8B or 70B open model is so much cheaper it isn't close. For agentic multi-step work, the closed frontier usually still wins on reliability. Test, don't assume.
A sane procurement cadence
My recommended rhythm: quarterly lightweight re-eval of the shortlist (your two incumbents plus whatever launched recently), a full suite run on any major provider release, and one annual deep dive where you renegotiate and re-baseline everything. Keep the suite alive with fresh failure cases from production. The teams that get the most value from frontier models aren't the ones with the smartest researchers; they're the ones whose eval harness lets them adopt a better model the week it ships.
One last thing: nobody's eval suite is finished. If yours is more than six months old, it's measuring last era's failure modes. The frontier isn't static, so your evaluation can't be either.