For a decade, the recipe for a better AI model was boringly simple: more data, more parameters, more training compute. Then o1 came along and broke the script. Turns out you can also just let the model think longer. Inference-time compute, the practice of spending more FLOPs per query at generation time, has become the second axis of scaling, and it's arguably the more interesting one.
The old law and the new one
Pre-training scaling laws, popularized by OpenAI's and DeepMind's work in 2020 through 2022, told us that loss drops predictably as you scale parameters, data and compute together. Chinchilla refined the ratios. Everyone internalised the lesson: bigger training run, better model. That held for years and it still holds, sort of. But the curve is flattening at the frontier. Doubling pre-training compute no longer buys what it used to, and the data supply is finite. You can't scrape the internet twice.
Inference-time scaling is the other knob. Instead of making the model smarter in the weights, you let it spend more tokens thinking: generating long chains of reasoning, exploring branches, checking its own work, backtracking when a path dead-ends. OpenAI's o1 was the proof of concept. DeepSeek-R1 showed the recipe could be replicated cheaply and openly. By 2025 and into 2026, every serious lab had a reasoning model, and the benchmark numbers on math and code jumped in ways that pre-training gains hadn't managed in two years.
Worth being precise about what "scaling law" means here. It's not a clean power law like pre-training. It's more like a dial with diminishing returns and high variance per problem. Some prompts plateau after a few hundred thinking tokens; others genuinely improve with tens of thousands. The research community is still working out the exact functional forms, and anyone quoting you a single exponent for test-time scaling is selling something. The honest statement is empirical: on verifiable domains, more thinking reliably helps up to a task-dependent ceiling.
Why test-time thinking works
The intuition is almost embarrassingly simple. A hard problem, say a competition math proof or a gnarly debugging session, has a much larger effective search space than what one forward pass can cover. When a model generates 10,000 tokens of reasoning instead of 500, it's effectively searching that space: trying approaches, noticing contradictions, correcting course. Empirically, accuracy on hard reasoning tasks climbs smoothly with the number of thinking tokens, at least up to a point, and the gains show up on problems with verifiable answers far more than on open-ended ones.
There are a few mechanisms under the hood. Long chain-of-thought lets the model decompose a problem into steps it couldn't fit in a single pass. Self-correction lets it catch its own errors mid-stream. Some setups go further and sample many candidate solutions, then vote or verify, which is where techniques like majority voting and process reward models come in. The unifying idea is that compute spent at inference is compute spent on this specific problem, not amortised across all queries like pre-training.
One useful mental model is the difference between a sprinter and a chess player. A base model plays chess by intuition: fast, plausible, occasionally brilliant, unforgivably blunder-prone on move thirty. A reasoning model sits there and calculates lines, evaluates them, discards most, and picks the least-bad one. Same player, same opening book, completely different process. The blunder rate drops for exactly the class of problems humans solve slowly and carefully, which is why the biggest gains show up in math, code and multi-step planning rather than in trivia.
The economics: when does thinking pay off?
Here's the part that gets glossed over in the hype. Inference-time compute is not free scaling. It's a per-query tax. If every query takes 30x more tokens, your serving bill takes roughly 30x more per query too. The bet only makes sense when the value of a correct answer exceeds the marginal cost of thinking, and when the task actually benefits from reasoning.
- Great fit: competition math, code generation with tests, formal proofs, multi-step agentic planning, scientific hypothesis checking. Anything with a verifiable outcome.
- Poor fit: simple classification, sentiment tagging, short factual lookups, high-volume chat where latency matters more than depth.
- Gray zone: long-horizon agent tasks, where a thinking model may save you money by failing less often, but latency and token costs can balloon in ways that are hard to predict.
- Emerging fit: vision-language reasoning and scientific document analysis, where error costs are high and verifiable structure exists. The gains are real but still lumpier than in pure text math and code.
The smart play, and most production teams have landed here, is routing. Send easy queries to the fast model, hard ones to the thinking model, and use a cheap classifier or the model's own confidence to decide. Reasoning models have also gotten better at deciding how long to think; some can produce short answers when the problem is trivial and long ones when it isn't.
What this means for model economics
Inference-time scaling quietly rewrites the economics of building AI products. Pre-training is a fixed cost you pay once; inference is a variable cost you pay forever. If models get smarter mostly by thinking longer, then the cost structure of every AI application shifts toward serving. Gross margins for API providers depend on how efficiently they can run thinking tokens, which is partly why inference optimization (speculative decoding, distillation of reasoning traces, KV-cache tricks) has become such a hot area.
It also changes how you budget. With pre-training, finance asks "how many dollars did the run cost?" With inference-time compute, finance asks "what did this ticket cost to resolve?" Those are different conversations. Teams that instrument per-request thinking tokens, accuracy, and dollar cost end up routing far more aggressively than teams that just look at the monthly invoice. The number you want on your dashboard is quality per dollar per task, not tokens per second.
It also changes who can compete. Training a frontier base model costs, by reported figures, hundreds of millions of dollars. Teaching an existing open model to reason with RL is reportedly much cheaper, and DeepSeek demonstrated that a small team with good taste and access to GPUs could do it. The moat moved from "who has the biggest cluster" to "who has the best training recipe and the best evals." More on that in our evals piece.
The limits and the open questions
Don't let the benchmark curves fool you. Inference-time scaling has real limits. Gains concentrate on verifiable domains; on fuzzy, open-ended tasks like writing or strategy advice, longer thinking helps less and sometimes produces confident nonsense with more steps. Models can also overthink: burning thousands of tokens on problems a human would solve in one line, or talking themselves out of a correct first instinct. There's active research on calibrating when to stop thinking, and nobody has a clean answer yet.
There's also a safety angle. Long reasoning traces are a new surface for reward hacking, where the model learns to produce reasoning that looks convincing rather than reasoning that is correct. Process reward models and verifier-based training try to patch this, but it's genuinely hard. And if chain-of-thought is faithful, monitoring it is a useful safety tool; if it isn't, we're staring at a very articulate post-hoc rationalisation. The evidence so far suggests the traces are partially faithful at best.
Benchmarks vs. vibes
One trap worth flagging: eval scores for reasoning models are noisy and easy to misread. Pass@1 with a long thinking budget can swing several points run to run, and contamination debates around AIME-style suites never die. If you're comparing models for a real workload, build your own small private suite, exactly the discipline we argue for in our evals piece. Synthetic-think tokens can flatter public numbers while doing little for your actual failure modes, like tool misuse or silently malformed JSON.
Reasoning models are also poorly understood in their failure geography. They're superb on linear chains and mediocre on tasks requiring broad commonsense leaps, they hoard tokens on trivial inputs, and they can produce beautifully structured wrong answers. Public charts will not tell you this. A day of playing with the model on your own gnarliest tickets will tell you everything.
There's also the human factors side. Reasoning models are slower to first token, which breaks UX patterns designed for snappy chat. Products that hide this well (progressive streaming of the thinking trace, explicit "working on it" states, background agents) feel calm. Products that don't feel broken. Latency budgeting is now a design problem, not just an infra problem.
A field guide to thinking budgets
Most teams I talk to manage thinking tokens the way they used to manage prompt length: crude dials, lots of regret. The setup that works: measure, per task type, the quality-versus-thinking-tokens curve on your own workload. You'll find a knee where accuracy flattens and sometimes dips as the model overthinks. Set your default budget just before that knee. The model's maximum is almost never value-optimal.
Second, cap retries. A failing reasoning model often fails by burning its whole budget, then an impatient wrapper re-rolls it. Decide in advance how many attempts a task gets, and what happens after: escalate, degrade gracefully, or hand back to a human. Third, log sampled traces. When quality drops after a model swap, traces show whether the reasoning style changed, not just whether answers got worse. That distinction separates a targeted fix from a week of flailing.
One contrarian opinion I'll defend: unlimited thinking budgets are usually a product smell, not a feature. They signal nobody has measured where thinking stops helping. If a vendor brags that their model thinks for 100,000 tokens, ask what accuracy that buys over 10,000, and at what latency. Usually the answer is a billing page.
And watch your "regret tokens," the tokens spent on attempts that got discarded or re-run. In healthy pipelines that number stays small. When it creeps up, something upstream changed, a prompt, a router, a model version, and your per-task economics are quietly rotting while every dashboard still looks green.
The practical takeaway
If you're shipping AI products in late 2026, treat inference-time compute as a budget line, not just a feature checkbox. Measure whether thinking tokens actually improve outcomes on your workload, route queries by difficulty, and cap thinking budgets for latency-sensitive paths. The teams winning right now aren't the ones with the biggest models; they're the ones who know exactly when a longer think is worth paying for.
The old scaling law isn't dead. It's just been joined by a second one, and this one is billed per token.
A final word of caution from the trenches: watch your tails. Average latency will look acceptable while the 99th percentile, the queries where the model spirals into 40,000 tokens of circular reasoning, quietly blows up your p99 and your bill. Cap thinking budgets, monitor retry behaviour, and alert on pathological traces. Inference-time compute is a wonderful tool. It is also a very easy thing to spend money on without noticing.