Open any frontier model card from the last two years and you'll find the same confession: the headline parameter count is a lie. Sort of. Underneath sits a mixture of experts architecture, where only a fraction of the network fires per token. MoE didn't just squeak by. It won. Here's why the sparse approach beat the dense one, and what it costs you.
What a mixture of experts actually is
A dense transformer activates every parameter for every token. A mixture of experts model swaps the feed-forward layers for a collection of parallel "expert" networks and a small router, sometimes called a gating network, that picks which experts handle each token. Typical designs route each token to the top 1, 2 or 8 experts out of dozens or hundreds. The result: total parameters can be enormous while active parameters per token stay modest.
DeepSeek V3 is the canonical example people cite, with reportedly around 671 billion total parameters and roughly 37 billion active per token. Mixtral, the model that made the pattern legible to the open-source world, paired eight experts with two active per token. The exact numbers vary by lab and are usually self-reported, but the shape is consistent: sparse activation, dense-level quality.
It's worth remembering this idea is old. Mixture of experts dates back to the early 1990s, and Google's Switch Transformer scaled it to pre-trained language models years before the current wave. What changed was fit. Once transformers ate the world and compute became the binding constraint, the math of sparsity went from elegant curiosity to budget spreadsheet. The labs that shipped MoEs first simply noticed that you could keep 90% of a giant model asleep on any given token and nobody downstream could tell.
The economics are brutal and persuasive
Training and serving a dense model scales roughly linearly with parameter count. MoE breaks that link. You pay for a big model's knowledge capacity, but you pay inference cost closer to a much smaller dense model. For API providers, that's the whole game: quality that looks frontier, cost that doesn't. It's no coincidence that the cheapest high-quality open models are MoEs, and that every lab chasing price-performance has quietly adopted the pattern.
There's a subtlety here, though. The savings are real for compute but not for memory. All those experts have to live somewhere, which means MoE models demand large aggregate VRAM footprints. Serving DeepSeek V3 locally isn't a weekend hobby project; you need either a serious multi-GPU rig or heavy quantization plus offloading. Sparse in FLOPs, dense in bytes. Plan accordingly.
Do the napkin math on an approximation: a frontier dense model running at, say, 8-bit weights might need a terabyte of memory to serve naively, while a comparably capable sparse model needs a similar footprint but does a small fraction of the multiply-accumulates per token. That difference in active compute is the entire speed and cost story. Memory is the tax you pay to get it.
Routing is where the bodies are buried
The router sounds trivial and absolutely isn't. Get it wrong and you get expert collapse: a few popular experts hog all the tokens while the rest starve and learn nothing. Get it subtly wrong and you get load imbalance, where half your cluster idles while one expert melts. Labs fight this with auxiliary load-balancing losses, capacity factors, token dropping, and increasingly, aux-loss-free balancing tricks (DeepSeek reportedly went this route to avoid the quality drag from balancing penalties).
- Load balancing: keep every expert fed roughly equally so capacity doesn't go to waste.
- Routing stability: small changes in router logits can flip expert assignments, which makes training touchy.
- Specialization: experts tend to differentiate (some lean toward code, some toward languages, some toward math), though the boundaries are fuzzy and nobody fully controls them.
Expert specialization is the most oversold part of the story. Pick any token-routing visualization floating around and you'll see clusters, but in practice experts overlap heavily. Treat "specialist experts" as a useful mental model, not a guarantee.
There's also the question of fine-grained vs. coarse experts. DeepSeek went with hundreds of small experts plus a couple of shared ones that fire on every token, which smooths knowledge gaps and stabilizes training. Other designs use fewer, chunkier experts and accept coarser routing. Neither is clearly right; both beat dense at the same budget. The shared-expert trick in particular is underrated: it acts as a floor of general competence so that routing mistakes hurt less.
Why MoE and reasoning models pair so well
Here's the connection people miss. Reasoning models (o1-style, R1-style) generate very long chains of thought. Training and serving them is token-hungry. A sparse model lets you afford that: you get a model with a lot of stored capability whose per-token cost is low enough that 10,000-token reasoning traces don't bankrupt the serving budget. If inference-time compute is the new scaling axis, MoE is the enabler that makes it economical. The two trends are the same trend wearing different hats.
The training side matters too. RL on reasoning traces means sampling many completions per problem and, increasingly, running verifiers or tool calls inside the loop. That multiplies token throughput requirements. A dense model at equivalent quality would make RLVR runs prohibitively expensive for most labs. Sparsity is what lets a team with a few thousand GPUs play the same game as a hyperscaler, at least in the reasoning post-training stage where much of the leap now happens.
The real costs and failure modes
MoE isn't free lunch, it's a trade. Training instability is meaningfully higher, and fine-tuning a sparse model can behave oddly: if your fine-tune data is narrow, the router may overfit and degrade general capability. Quantizing MoEs is harder too, because different experts have different sensitivity to low-bit weights, and naive uniform quantization leaves quality on the table. Distributed serving adds communication overhead, since tokens have to be shuffled to whichever expert lives on whichever GPU.
Capacity planning is its own special nightmare. During bursty traffic, hot experts can saturate while cold ones sit idle, and the router has no concept of your SLO. Good serving stacks now do expert-aware batching and sometimes replicate hot experts across GPUs. Naive setups just eat the tail latency. If you're benchmarking a vendor, ask how they handle expert hotspots; the answer tells you whether they actually run MoEs in production or just pass through to someone who does.
Debugging is also weirder. When a dense model fails, at least the failure is legible. When an MoE fails, it might be a routing problem, an expert that never trained, or a capacity bottleneck. Repro issues abound. If you're running these models in production, invest in evals that can localize regressions by domain (more on that in our evals post).
What this means for you
If you're choosing a model to build on right now, default to the open MoE family: they're cheap, capable, and well supported by every major inference stack. If you're self-hosting, budget for the memory footprint, not the FLOP count, and don't expect to quantize your way to a laptop. If you're training, the message is blunter: dense models at the frontier are effectively extinct, and your compute budget goes roughly twice as far with sparsity, if you can stomach the engineering complexity.
One practical caveat on fine-tuning: LoRA on a sparse model rarely moves the experts themselves unless you target the right modules. Many teams fine-tune attention and shared layers and leave the routed experts frozen, which preserves general capability while teaching style and domain. It works, but it means MoE fine-tuning results depend heavily on which weights you touch. If your fine-tune "did nothing," that's often why.
A quick checklist before you commit to a sparse model
If you're evaluating an MoE for production, most of the risk sits outside the model card, so interrogate the boring stuff. First, ask for tokens-per-second and p95 latency on your actual context lengths, not theirs. Sparse models shine in aggregate throughput but can stumble on single-request latency when expert shuffling crosses nodes. Second, ask how they handle expert hotspots and whether hot experts get replicated; vague answers here usually mean a pass-through proxy rather than real serving infrastructure. Third, ask about quantization: which experts took the low-bit hit and how much quality that cost on a held-out set. Uniform "4-bit everything" is a red flag dressed up as a feature.
Fourth, check the fine-tuning story. Which modules are safe to attach LoRA to, and what happened when they tried it? If nobody on their team tried it, you will be the beta tester, on your own dime and your own incident reviews. Fifth, look at the failure telemetry they'll give you: per-domain pass rates, not just aggregate scores. Aggregate numbers hide the slice where your workload actually lives.
And one opinion: don't self-host an MoE just to save money on your API bill unless you're already running a serious GPU operation. The memory footprint alone will eat the savings, and your on-call rotation will eat the rest. The economics work at scale, with batching and expert-aware routing. At one small team scale, paying per token to someone else who solved the topology problem is often the honest choice. Do your own napkin math before your heroics become a line item.
The open questions
Nobody has fully solved dynamic routing at inference flexibly, or expert deletion for deployment slimming, or robust low-bit quantization across heterogeneous experts. There's also live research on routing-aware distillation, compressing a big MoE into a smaller dense model for edge deployment, with mixed results. Expect 2026 and 2027 to be busy here, because the prize is large: all the capability, none of the memory headache.
There's also the philosophical question of whether MoE is where we want to be. Sparse activation is a pragmatic compromise, not a destination. Some researchers argue dense models remain more predictable for alignment and interpretability, since every parameter participates in every token and post-hoc analysis is at least coherent. That's a legitimate concern. For now the pragmatists are winning because, frankly, the invoices are winning.
MoE won because it changed the unit economics of intelligence, not because it's elegant. The labs that figured out routing first got the best price-performance in the market, and everyone else is catching up. That's about as close to a settled question as this field ever gets.
If you want a closing heuristic: treat parameter count as a marketing number and active parameters plus memory bandwidth as the engineering number. Any model card that flaunts total parameters without the active count and the deployment footprint is hiding something, usually because the honest story is "cheap to run, expensive to host." Ask both questions every time. You'll quickly learn which vendors understand their own architecture and which ones are reading from a press release.