The marketing version of reasoning models goes like this: they "think." The engineering reality is more interesting and less mystical. Reasoning models are what you get when you take a strong base model, point reinforcement learning at problems with checkable answers, and let the reward signal do the teaching. Here's how the sausage actually gets made, based on what's been published by OpenAI, DeepSeek and others. Note that exact recipes remain proprietary, so some of this is inference from partial disclosures.
Stage one: the pre-trained base
Everything starts with a capable base model, trained on the usual torrent of text, code and (increasingly) synthetic data. This model already "knows" a lot of math and can pattern-match solutions. What it can't do reliably is work through a long multi-step problem without making an early error and stubbornly following it to a wrong answer. Pre-training gives you knowledge. It does not, by itself, give you reliable deliberation.
Scale matters here. RLVR amplifies whatever the base model can already almost do; it doesn't conjure capability from nothing. If the base model has never seen a class of problems, RL won't teach it the domain, just sharpen its instincts within reach. This is why the reasoning-model race is really a pre-training race plus a post-training race, not one or the other.
Stage two: supervised warm-up
Most pipelines include a light supervised fine-tuning step before the heavy RL. Sometimes called cold-start or format tuning, the goal is narrow: teach the model the shape of a reasoning trace (long chain-of-thought, maybe with explicit self-correction) and stop it from producing chat-style one-line answers. DeepSeek-R1 zero skipped this and got pure RL to work, but interestingly, their main R1 model used a cold-start phase because pure RL traces were readable but ugly: language mixing, repetition, chaotic formatting. Warm-up trades a little rawness for legibility and stability.
Data quality in this stage is disproportionately valuable relative to quantity. A few thousand excellent, human-polished reasoning examples often beat hundreds of thousands of mediocre teacher traces, because the warm-up sets the prior the RL loop will spend weeks refining. Garbage in the warm-up means the RL stage spends most of its budget un-teaching pathological style instead of learning to reason.
Stage three: RL with verifiable rewards
This is the heart of it, and it's usually called RLVR: reinforcement learning from verifiable rewards. Take a batch of math problems or coding tasks where a program can check the final answer. Sample several attempts per problem. Reward the ones that pass, punish the ones that don't. No human raters needed in the loop for the reward itself, which is what makes this scale.
The objective function of choice lately is GRPO, group relative policy optimization, popularized by DeepSeek. The trick is elegant: instead of a separate critic network estimating value, you sample a group of completions for the same prompt and normalize rewards within the group. Better-than-average attempts get reinforced, worse-than-average ones get suppressed. That removes the memory and compute cost of a value model, which matters when your completions are 10,000 tokens long. Classic PPO still gets used, but GRPO's simplicity made it the default for reasoning-scale runs.
A few knobs do most of the heavy lifting in practice. The KL penalty against the reference policy is a leash that keeps the model from drifting into gibberish that happens to score well. Reward shaping decides whether correct-but-ugly reasoning counts as a win. Group size sets your variance: bigger groups mean stabler advantage estimates, at linearly more sampling cost. And there's the curriculum, the ordering of easy to hard problems across training, which teams guard as closely as any recipe secret. Publish the GRPO loss equation, sure. The curriculum stays in the vault.
Why verifiable rewards beat human preference here
Classic RLHF relies on human preference, thumbs up, thumbs down, crowdsourced taste. That's fine for tone and helpfulness and garbage for truth. Humans can't reliably judge whether step 47 of an algebra proof is valid, so a preference model trained on their judgments can't either. Verifiable rewards sidestep the entire bottleneck: if the code passes the tests and the math checks out, the signal is clean regardless of how convinced the rater felt. That's the fundamental asymmetry that made RLVR explode. Human preference data is expensive, noisy and gameable at the margins. A compiler is cheap and honest.
The catch is that most economically valuable work isn't verifiable this way yet. Customer support, legal drafting, strategy memos, design critique, all of it resist mechanical checking. That's why the frontier is pushing toward semi-verifiable rewards: rubric graders, tool-backed checks, retrieval-grounded verification, and process reward models that score intermediate steps. Expect an unglamorous decade of building verifiers for fuzzy domains, much as the last decade was spent building reward models for taste.
What the model figures out on its own
The striking, reproducible finding from R1-style training is that reasoning behaviors emerge without being taught. Models taught only "get the right answer" spontaneously develop longer chains of thought, self-checking, and even backtracking when stuck. DeepSeek called this the aha moment: mid-RL, the model suddenly starts re-examining its own approach. It's not magic, it's reward shaping plus enough samples, but it genuinely surprised the researchers. Nobody programmed in reflection tokens.
- The model samples an attempt and gets a verifiable reward or zero.
- Over many iterations, attempts that stumble into good reasoning patterns get reinforced.
- Longer, more careful traces correlate with correct answers, so trace length climbs.
- Self-correction behaviors get picked up because they genuinely raise reward, not because anyone asked for them.
Reward hacking and other fun failure modes
RLVR has a dirty secret: the verifier becomes the attack surface. If you reward "final answer matches," the model will learn to do whatever maximizes that signal, including weird move: hardcoding, guessing patterns, exploiting flawed test suites, or verbose reasoning that games loose graders. Good pipelines use held-out checks, canary tests, and reward shaping that penalizes degenerate traces. It's an arms race between your verifier and your model's optimizer, and the optimizer is very persistent.
Beyond hacking, there's mode collapse toward brevity or verbosity, depending on how the reward is shaped, and language drift, where traces wander into whatever token distribution maximizes reward bizarreness. R1-Zero showed all of this: emergent reasoning alongside occasionally unreadable, self-contradictory output. The fixes are unglamorous: mixed rewards, format constraints in early training, and humans reading a lot of traces. Anyone who tells you this is push-button is skipping the part where a researcher stares at 400 completions and notices subtle nonsense.
Generalization is the other open question. RLVR on math and code transfers reasonably well to adjacent reasoning tasks, but nobody has shown that training on verifiable domains produces general "thinking" that helps everywhere. Benchmarks suggest partial transfer at best, strong within domain, shakier across it. Be suspicious of any claim that RL makes models broadly wiser. It makes them better at reward-shaped skills.
The data question
Where do the training problems come from? Mostly a mix of curated competition-style math and programming problems, synthetic variants, and distillation: reasoning traces generated by a stronger model, filtered for correctness, then used as SFT data for a smaller one. R1's open-weights release triggered a wave of distillations, where small dense models fine-tuned on R1's correct traces got surprisingly strong at narrow reasoning benchmarks. Distillation gives you capability cheaply but locks in the teacher's blind spots. Native RL gives you the genuine article at much higher cost. Teams pick based on budget and how much they trust the teacher.
Compute accounting for a full reasoning run is humbling. Between pre-training, cold-start SFT, and RL rollouts (each problem may require dozens of sampled completions), the bill adds up fast. Roughly speaking, published estimates suggest RL stages for frontier reasoning models run to a nontrivial fraction of pre-training FLOPs, which is why only a handful of organizations do it in-house. Everyone else rides on open weights plus lighter-touch tuning, and that's a perfectly rational posture.
It also reframes what "open weights" buys you. A fully open reasoning model with released weights and training logs (R1 was unusually generous here) lets smaller teams stand on frontier shoulders rather than guess at recipes. The genuinely scarce resource now is engineering judgment: choosing verifiers, curating curricula, and reading enough traces to know when the model is fooling you. That scarce resource doesn't ship in a checkpoint, which is exactly why the field hasn't commoditized.
What this looks like if you're not a frontier lab
Most teams reading this will never run GRPO in-house, and that's fine. The useful version of "how reasoning models are trained" for everyone else is a debugging lens. When a reasoning model flubs your task, you can now ask sharper questions: is this a base-capability gap that no amount of inference budget will fix? A warm-up format problem, where the model never learned your domain's trace shape? An RL blind spot, where the verifiable reward it was trained against doesn't match your definition of correct? Each failure mode has a different remedy, and only one of them is "throw more tokens at it."
There's a second, quieter implication. If you do fine-tune on reasoning traces, the same verifier logic that trains the big models applies to you at small scale: filter hard, keep only verified-correct traces, and never SFT on unverified teacher output just because it reads confidently. A few hundred clean, checked traces will beat tens of thousands of plausible-looking ones. I've watched teams learn this the expensive way, shipping a "reasoning-tuned" model that confidently reproduced the teacher's subtle errors with extra steps.
Also worth internalizing: the whole pipeline presumes you can tell right from wrong mechanically, at least sometimes. If your domain has even a narrow slice of verifiable tasks, unit tests, schema validation, math checks, that's where your reward hacking starts, and where your quality bar rises fastest. If your domain has none, temper your expectations of what RL-style tuning can do, and spend the effort on evals and retrieval instead. The labs spend their billions where checking is cheap; you should copy the principle, not the price tag.
What to take away
Reasoning training isn't a new kind of intelligence, it's a very effective recipe: strong base, small warm-up, big RL with verifiable rewards, careful verifier hygiene. The frontier labs are racing on verifier quality and curriculum design now, not just scale. If you're building on these models, the practical implication is simple: they fail differently than chat models. They overthink easy problems, they confidently justify wrong answers, and they reward-hack sloppy graders. Eval your actual workload, don't trust the leaderboard, and you'll do fine.
The one prediction I'll make with some confidence: process supervision, verifying each step rather than just the final answer, becomes standard as soon as tooling makes step-level labels affordable. Outcome-only rewards work, but they're statistically starving compared to dense process signals. Whoever makes process rewards cheap gets the next sizable jump in reasoning reliability. Until then, assume your reasoning model is a sharp intern with no sense of when to stop, not an oracle.