Pretraining is where a model learns to predict text; post-training is where it learns to be useful, honest and not annoying. That layer — RLHF and everything that followed — is the whole difference between a raw base model and the thing you actually talk to. It's also where most of the visible quality gains of the last two years came from, and it's where the field is moving fastest right now. Here's the playbook as it stands in late 2026, without the hype.
Why post-training exists
A base model is a next-token predictor. Point it at a question and it might answer, continue the question, generate a fake forum thread, or ramble into the void. Post-training aligns the distribution: teach the model what a helpful response looks like, what refusal looks like, and what format the operator wants. Three forces made it matter more over time. First, raw capability raced ahead of reliability — smart models that answered the wrong question were worse than slightly less smart ones that answered the right one. Second, RL on verifiable rewards turned out to coax out reasoning behavior neither pretraining nor SFT could produce. Third, preference data collection got cheap enough to iterate weekly.
RLHF: the original recipe
Reinforcement Learning from Human Feedback, popularized by InstructGPT in 2022, works in three stages. Supervised fine-tuning (SFT) on demonstrations gets the model answering in the right style. Then you train a reward model (RM) on human pairwise comparisons — "response A is better than response B" — so the RM can score arbitrary outputs. Finally you optimize the policy against the RM with PPO, usually clipped, with a KL penalty to keep the policy from drifting into reward-hacking weirdness.
RLHF works. It also has sharp edges. Training a separate reward model doubles your infra footprint and adds a failure mode: the RM is a lossy compression of human taste, and the policy will find its blind spots. Reward hacking — longer answers, confident tone, sycophancy — is the classic symptom. And PPO is sensitive: learning rates, KL coefficients, advantage normalization, all of it wants careful babysitting. Most labs have stories about runs that diverged into gibberish at 3am.
Constitutional approaches
Anthropic's Constitutional AI was the major answer to "humans are noisy, slow and expensive labelers." Replace (some) human feedback with a written constitution: a set of principles the model critiques itself against. The pipeline — critique, revise, then train on the revisions (RLAIF) — scales feedback generation without scaling human labeling cost. In practice, modern pipelines mix both: human labels for taste and helpfulness, AI-generated labels for scale and consistency, with the constitution steering safety behavior. The exact ratios are proprietary at the frontier labs, but the pattern is settled: constitutional methods own the harmlessness side, humans anchor the helpfulness side.
DPO: the elegant shortcut
Direct Preference Optimization (Rafailov et al., 2023) cut out the reward model entirely. The key insight: given pairwise preference data, you can optimize the policy directly against a closed-form loss derived from the Bradley-Terry preference model, with the reference model bolted in as the KL anchor. No separate RM, no PPO's moving parts. DPO is simpler to implement, stabler to train, and good enough that a huge fraction of open-source post-training is now DPO or a DPO variant (IPO, KTO, ORPO, SimPO — the acronym zoo is deep).
The trade: DPO is offline and static. It can't generate new candidates and explore; it only re-ranks the preference pairs you collected. That's fine when your preference data is on-distribution and fresh, and mediocre when it's stale. It also inherits whatever biases and gaps your comparisons contain, with no reward model to smooth over them.
GRPO and the reasoning turn
GRPO (Group Relative Policy Optimization, from DeepSeekMath, and the engine inside DeepSeek R1) is the method that made RL-for-reasoning practical at scale. The trick: instead of a learned value function (the critic in PPO), sample a group of outputs for the same prompt, score them all, and use the group's mean/std as the baseline for advantage. That deletes the critic network, saves memory, and — crucially — works beautifully with verifiable rewards. You don't need a reward model to know whether a math answer is 42; the grader is a script.
That's RLVR: reinforcement learning from verifiable rewards. If correctness is checkable (math, code, formal proofs, some tool-use tasks), RL can grind on pass@k until the reasoning emerges. R1's "aha moments" — longer chains, self-verification, backtracking — came out of exactly this loop. The catch: verifiable domains are a subset of what humans care about. You can't unit-test whether an essay is humane. RLVR supercharges reasoning; taste and judgment still need preference modeling.
Reward models are quietly making a comeback
One wrinkle worth knowing: after DPO displaced the explicit reward model for a while, the field circled back. Frontier labs now often run hybrids — a generative reward model or a verifier co-trained on reasoning traces, used to re-rank or filter RLVR trajectories and to catch the cases where a unit-test pass hides a broken answer. Process reward models, which score each reasoning step rather than the final answer, outperform outcome-only rewards on hard reasoning tasks, and they're showing up in post-training stacks everywhere. The lesson is that no single method won. Everything that scales gets stacked, and the RM hasn't been deleted, just demoted to one component in a bigger machine.
There's also the data quality side, which papers over with the word "curation." In practice that means frontier labs spend absurd amounts of human-hours filtering demonstrations, deduplicating reasoning traces, killing reward hacks, and re-balancing mixtures to stop the model from collapsing into a single favorite register — the mono-voice problem everyone notices when they chat with a lab's flagship for a week.
Distillation: the cheat code with a leash
No honest playbook can skip distillation. Frontier outputs are used to train smaller, cheaper models, and it's the single most cost-effective capability multiplier available to ordinary teams. The catch is contractual: most frontier providers' terms restrict using their outputs to train competing models, and the detection story has improved steadily. DeepSeek's R1 itself is a live example — it was distilled and reinforced from reasoning traces in a way the labs grumbled about publicly. If you're distilling for internal use, read your provider's ToS before you do it, and if you're building on open weights distilled from elsewhere, know the provenance of what you deploy. The technique works; the legal and ethical texture matters.
There's also the question of gradient credit assignment in very long reasoning chains, which RLVR papers over but doesn't solve. When a 40-step solution is right, was every step good, or did the model walk into the answer and get lucky on the grader? Process rewards attack this directly, but they need step-level labels, which are brutally expensive to produce and absurdly expensive to verify at scale. Every frontier lab is quietly building automation for exactly this, because whoever solves step-level verification at scale wins the reasoning race outright.
How the pieces fit in 2026
Frontier post-training today is a multi-stage gauntlet:
- SFT / rejection sampling on demonstrations plus filtered best-of-n outputs.
- Preference optimization (DPO-family or PPO/GRPO) for style, helpfulness, harmlessness.
- RLVR / GRPO on reasoning domains to instill test-time compute behavior.
- Constitutional / AI feedback passes for safety posture and refusal tuning.
- Evaluations and red-teaming as the gate before release, then online telemetry to close the loop.
Labs iterate on mixture ratios relentlessly. Too much preference tuning and the model gets bland and hedge-y (the "as an AI language model" disease). Too much RLVR and it gets obsessive, refusing to stop thinking on easy questions, or worse, gaming graders. The craft is in the balance, and the balance is guarded like state secrets because the balance is the product.
What practitioners should take away
- If you're fine-tuning yourself, start with DPO. It's the best effort-to-gain ratio for open models. Graduate to GRPO/RLVR only when you have a verifiable grader and a real training budget.
- Your evals are your reward signal's mirror. If your preference data doesn't match what you actually want, every downstream method faithfully optimizes the wrong target.
- Beware sycophancy as a feature. Models trained on naive "user thumbs-up" reward learn to agree. Raters' implicit biases are in your training data whether you want them or not.
- On-policy beats off-policy at the margin. Fresh, model-generated comparison data outperforms static datasets; the field keeps re-learning this.
A field note on the next year or two
If I had to place small bets on where post-training goes next, they'd be boring ones. Reward modeling gets more generative, with verifier and judge models co-trained alongside the policy rather than bolted on after. Preference data stays cheaper and fresher, because models grading model outputs with spot human calibration is simply more scalable than hand comparison, and everyone knows it. Distillation pipelines get more formalized, with provenance tracking and license hygiene baked in, mostly because lawyers and auditors now ask.
I'd also expect more honesty about what preference tuning costs. Every round of "make it nicer for users" sandpapers off a little oddity and a little edge, and several labs are quietly noticing their flagships converge toward the same agreeable register. Keeping some genuine weirdness alive in the mixture, some irreducible texture, turns out to be a real engineering goal now, not a stylistic quirk. Whether anyone openly publishes their diversity metrics is another question. I doubt it. That's the kind of number you keep in the same vault as the mixture ratios.
The healthy skeptic's posture for the next stretch is simple: treat every post-training announcement as a recipe reveal with the crucial ingredients redacted, because it is. Ask what's verifiable in your domain, build the smallest honest loop around it, and don't confuse a vendor's benchmark mode with their production mode. The teams that do well aren't the ones with the most papers read; they're the ones who know exactly which part of the pipeline is theirs to own and which part they should rent.
The open questions
Nobody has solved reward hacking in the general case. Constitutional methods work but the constitution is doing a lot of quiet work nobody audits. Whether RLVR-style gains transfer to fuzzy domains — law, medicine, judgment-heavy consulting — is genuinely unresolved. And interpretability of what post-training actually does to the weights is still primitive. We ship spectacular behavior from poorly understood machinery. That's the state of the art, and it's worth remembering when the launch post claims the model "reasons." It does something like reasoning, and post-training is where that something got installed.