Dense models activate every parameter for every token. MoE models split parameters into experts and route each token to the best-fitting ones — often 2 out of 8.
Result: you get the breadth of a huge model with the compute cost of a fraction of it.
Mixtral 8x7B was the wake-up call. DeepSeek V3 at 671B with 37B active raised the bar again. Kimi K2 does the same trick at trillion scale.
Practical implications: MoE models dominate open-weight frontier. They need more memory than the active parameter count suggests (all experts sit in memory), but inference compute stays cheap.
Fine-tuning angle: routing is sparse, which makes LoRA fine-tuning tricky — you're tuning the shared layers and router, not just experts. Plan accordingly.
Where it's going: finer-grained experts, better routing, models that specialize per task automatically. It's the architecture of the rest of the decade.