Skip to main content

Mixture of Experts

Conditional computation’s long road from an obscure 1991 paper to the architecture powering frontier models. Routing, load-balancing pathologies, and the GShard-to-Mixtral-to-DeepSeek lineage that made sparse models the default.

The core idea behind Mixture of Experts (MoE) is almost embarrassingly simple, and I think its simplicity is exactly why it took the field so long to take it seriously. Not every input needs every parameter. A question about organic chemistry probably does not need the same neural circuitry as a request to write a limerick. So instead of pushing every token through the full network, you train a collection of specialized subnetworks, call them “experts,” together with a small learned routing function that decides which experts handle which inputs. You end up with a model that has far more total parameters than it actually uses on any single forward pass. Capacity goes up; compute per token does not.

That trade is the whole game, and for most of deep learning’s history nobody could make it pay. The idea is genuinely old. Jacobs, Jordan, Nowlan & Hinton (199135ya) proposed adaptive mixtures of local experts three decades before it became fashionable, and Jordan & Jacobs (199432ya) extended it to hierarchical mixtures trained with the EM algorithm. One of those four authors was Geoffrey Hinton, which tells you the pedigree was never the problem.1 The problem was that the thing did not want to train. Experts collapsed, routing was unstable, and the discrete “pick an expert” decision sat awkwardly inside a gradient-based optimizer that wants everything smooth and differentiable.2

Geoffrey Hinton in 2024. One of the four authors of the original 199135ya mixture-of-experts paper, three decades before the architecture became the default for frontier-scale language models. Photo: Arthur Petron, CC BY-SA 4.0, via Wikimedia Commons.

Geoffrey Hinton in 2024. One of the four authors of the original 199135ya mixture-of-experts paper, three decades before the architecture became the default for frontier-scale language models. Photo: Arthur Petron, CC BY-SA 4.0, via Wikimedia Commons.

So what changed? Scale changed. When your dense model costs tens of millions of dollars to train and hundreds of thousands of dollars a day to serve, a 3x cut in per-token compute stops being a nice-to-have and becomes the line between a viable product and a science project. The modern MoE renaissance starts at Google around 2017 and runs in a fairly clean line through GShard, Switch Transformer, Mixtral, and DeepSeek-V3. That lineage tells you most of what the field figured out, and a fair amount of what it is still arguing about.

The Routing Problem

The gating function is where all the difficulty lives. You have \(N\) expert networks and you need to pick \(k\) of them per token, where \(k\) is usually 1 or 2 in the early work and 8 in the current frontier designs. The obvious approach is a learned linear layer followed by a softmax, then take the top-\(k\) scores. It works, sort of. Shazeer et al. (2017) demonstrated it at real scale with their “Sparsely-Gated Mixture-of-Experts” layer, dropping MoE blocks between the layers of an LSTM language model and reaching up to 137 billion parameters, which was an absurd number in 2017.

But that paper also turned up the first of many load-balancing headaches, and they have haunted every MoE system since. Left alone, the router degenerates. A handful of experts grab most of the traffic early, get more training signal, get better, attract more traffic, and the rich-get-richer dynamic leaves the rest of the experts undertrained and effectively dead weight.3 You paid for \(N\) experts and you are running maybe five of them.

Shazeer’s fix was noisy top-\(k\) gating, adding tunable Gaussian noise before the selection to encourage exploration, plus an auxiliary “importance loss” that punishes imbalanced utilization. It worked well enough to prove the concept. But the underlying tension never goes away, and it is worth stating plainly because it explains the next eight years of research: you want the router to make good decisions, sending each token to the expert that will handle it best, and you also want it to make balanced decisions, spreading load roughly evenly so no expert starves. Those two objectives fight each other. Every major MoE system since 2017 is, at heart, a different proposed truce between them.4

|[^dbrx-details]: DBRX was one of the first fine-grained MoE models to be open-sourced under a permissive license, which made it a popular choice for fine-tuning and research. The 16-expert design with top-4 gating represents a middle ground between the 8-expert/2-active design of Mixtral and the 256-expert design of DeepSeek-V3. Databricks subsequently integrated DBRX into their Mosaic AI platform, and the model became a demonstration of what fine-grained MoE could accomplish in the open-source space without requiring specialized hardware. The model was trained on the RedPajama dataset, and its performance showed that open-source MoE could compete with Google’s proprietary systems when trained on carefully curated corpora.

There is also a piece of folklore worth naming. The architect of that 2017 layer, Noam Shazeer, is the same person who co-authored the Transformer paper, co-founded Character.AI, and was later reportedly hired back to Google in a deal valued in the billions. When he joined OpenAI, an HN commenter (HarHarVeryFunny) recounted the lineage of the Transformer itself: “The architecture was Shazeer’s, but the rough idea came from Jakob Uszkoreit who initiated the project.” The point being that the same small cluster of people who built the dense Transformer also built the sparse alternative to it. MoE is not a rival tradition to the Transformer; it is the same tradition asking what happens when you stop running the whole feed-forward block on every token.

GShard and the Scaling Breakthrough

GShard (Lepikhin et al., 2020) is where MoE crossed from “interesting research direction” to “this demonstrably works at scale.” The paper trained a 600-billion-parameter MoE Transformer for machine translation, sharding the experts across as many as 2,048 TPU cores. Two ideas made it tractable:

  1. Top-2 routing with a hard expert capacity. Each expert gets a fixed buffer size per batch. Once an expert’s buffer fills, extra tokens routed to it are dropped or passed straight through via the residual connection. It is a blunt instrument, and dropping tokens feels like throwing away compute you already paid for, but it converts load balancing from a soft optimization objective into a hard constraint the system simply cannot violate.5

  2. Random second-expert selection. The first expert is picked by top-1 gating; the second is sampled in proportion to the remaining gating weights rather than taken deterministically. That bit of stochasticity discourages expert collapse without bolting on an explicit noise term.

GShard kicked off a lineage. Google’s Switch Transformer (Fedus et al., 2022) went the other way on \(k\) and simplified all the way down to top-1 routing, one expert per token, and showed you could push to 1.6 trillion parameters that way. The result is counterintuitive the first time you see it: routing to fewer experts per token improved training stability, because the gradient signal reaching each expert is cleaner.6 You give up some per-token capacity and you get back training efficiency and far less engineering pain. ST-MoE (Zoph et al., 2022) then folded the Switch stability tricks together with lessons from the Vision Transformer line and produced what was, for a while, the most reliable recipe anyone had published for actually getting an MoE to converge.

I want to flag something about this stretch of history that is easy to miss in hindsight. None of these were product models. They were translation systems and research artifacts, trillion-parameter curiosities that lived on enormous TPU pods and never touched a consumer. The architecture was proven well before it was useful to anyone outside Google’s research org. That gap, between “demonstrated at scale in a paper” and “something you can actually run,” is the part of the story that the open-weight community closed, and it is where the next chapter starts.

Mixtral: MoE Goes Mainstream

For all of Google’s foundational work, the model that actually put MoE in front of the open-source world was Mixtral 8x7B from Mistral AI, released in December 2023. The design was almost aggressively unfancy: take a Mistral 7B-class dense model, replace each feed-forward block with eight expert feed-forward blocks plus a router, and activate two of the eight per token. Total parameters land around 46.7 billion; active parameters per token sit near 12.9 billion. The payoff was a model that matched or beat LLaMA 2 70B while costing a fraction as much to run.

The importance of Mixtral was less architectural than economic. It proved an open-weight MoE could actually be deployed by normal people. The community immediately started quantizing it, fine-tuning it, and squeezing it onto consumer hardware. If you had 48GB of VRAM, two RTX 3090s, you could run something in the neighborhood of a 70B-class model at home. The economics of local inference shifted overnight.

You can feel that shift in how practitioners still talk about Mixtral years later. One Hacker News commenter (sbinnee) put the historical weight plainly: “When it comes to MoE, to me, I remember Mixtral model that showed the viability of MoE for the first time. I was impressed by their technical report. To be clear, MoE idea was already out there.” That captures the dual nature of the thing well, an old idea, but the first time the broader public watched it work in a model they could download. The debate over quantization quality vs. size reduction was equally vigorous, with early adopters noting that 4-bit quantized Mixtral-8x7B lost quality on arithmetic tasks but gained the ability to run on consumer hardware.

The trade-offs got debated just as quickly, and this is where the community commentary is more honest than any paper. The defining limitation of the cheap-to-run MoE is the low ratio of active to total parameters, and people who run these locally feel it. As freehorse described it on HN: “I wish we had some MoE models with a higher ratio of active parameters per total. My experience is that the newer MoE models that can run in a 64[GB] laptop have too few active parameters to be useful outside narrower, specific tasks. Mixtral 8x7b was a 14b active parameter (56b total) MoE model a few years ago and was probably the best model one could run in that range for some time, but it is too old now.” That is the real tension a benchmark table hides: total parameters buy you knowledge, active parameters buy you reasoning, and the cheap MoE configurations skew hard toward the former.7

The economics also flow straight through to the people renting GPUs. Another HN commenter (aurareturn) worked the napkin math for serving Mixtral on a single H100: “Assuming 80GB H100 and you inference a model that is MoE and close to the size of the 80GB VRAM, you’re going to see around 10k tokens/second fully batched and saturated… You’re generating about 36 million tokens/hour.” Whether the exact figures hold, the structural point is the one that matters, the sparse model’s small active footprint is what makes that batched throughput possible at all.

The Mixtral router also taught the field a humbling lesson about what experts actually learn. The intuitive story, that you would get a “chemistry expert” and a “poetry expert,” turned out to be mostly wrong. Mistral’s own analysis showed expert assignment correlating far more with token position and syntax than with topic. Experts behaved less like “the history specialist” and more like “the expert that tends to fire at clause boundaries.” That fits the superposition picture from interpretability research: the model smears knowledge across all experts and uses routing mostly to allocate computation, not to carve up the world into tidy domains.[^specialization-myth]

|[^specialization-myth]: This matters for how you reason about MoE. If experts were clean topical specialists, you could imagine pruning the ones you do not need for a given deployment, keeping only “code” and “math” for a coding assistant, say. Because the specialization is mostly positional and distributed instead, that kind of surgical pruning does not work nearly as well as the naive picture suggests. The router is doing load allocation, not librarianship.

|## GLaM: scaling to the trillion-parameter regime {.collapse}

Between Switch Transformer and the open-source MoE wave, there is an essential bridge: Google’s GLaM (Du et al., 2022). Where Switch was a research demonstration of top-1 MoE at scale, GLaM was the first model to show that MoE could genuinely scale model capacity beyond what dense models could economically afford. GLaM is a 1.2 trillion parameter model with 64 experts per MoE layer and top-2 gating across 32 MoE layers, activated only 96.6 billion parameters per token during inference. The energy savings were dramatic: Google reported GLaM achieved better zero-shot performance than GPT-3 (175B) using roughly half the inference FLOPs and a third of the training energy.[^glaM-energy]

GLaM is interesting for several reasons beyond the headline parameter count. First, the expert count of 64 per layer turned out to be an inflection point: the Google team found that going beyond 64 experts started to pay diminishing returns on zero-shot performance because the communication overhead between experts began to outweigh the capacity gains. This is an important architectural signal. When you are routing across 2,048 TPU cores in GShard, the communication topology is different from what you face in a modern GPU cluster, but the fundamental tension between expert count and communication latency is universal.[^glaM-experts]

Second, GLaM demonstrated that MoE could work well in the decoder-only language modeling regime, not just in the encoder-decoder translation setting that GShard and Switch were optimized for. Each expert in GLaM was itself a large feed-forward network; the team scaled the hidden dimensions of experts rather than the number of experts beyond a certain point, because they found that a 64-expert setup with top-2 gating “struck an optimal balance between execution efficiency and performance on zero-shot and one-shot tasks.” This is the same kind of engineering trade-off that would echo through every subsequent large MoE: how many experts is enough before the routing complexity starts to dominate?8

There is something quietly remarkable about GLaM’s trajectory. The 1.2T model was never deployed to a consumer product. It was a research model that proved MoE could stretch model capacity well beyond the dense limit. It was also the first model to show that you could have a trillion parameters in total while routing only a fraction of them per token. That distinction, between total capacity and active compute, is what the entire modern MoE industry has built on. GLaM did not become the default architecture, but it showed the path.[^glaM-path]

|[^glaM-energy]: The energy numbers come from the Google Research blog, which reported that GLaM 1.2T used about one third of the energy of GPT-3 while achieving better average performance. The exact figures are: GLaM 64B (64 experts) used 96.6B active parameters and achieved better zero-shot results than GPT-3’s 175B with roughly half the inference FLOPs and a third of the training energy. These numbers set the bar for what MoE could accomplish in the efficiency domain. The zero-shot capability was measured on tasks like MMLU, SuperGLUE, and common sense reasoning benchmarks — tasks that require broad knowledge rather than specific training signal.

|[^glaM-experts]: The team also explored scaling beyond 64 experts, and found that the communication overhead started to dominate. With more experts, the all-to-all communication during the MoE forward and backward passes becomes the bottleneck, not the computation itself. This is why modern MoE systems tend to cap expert counts between 8 and 64, with the sweet spot depending heavily on the communication topology between devices. NVIDIA’s Blackwell NVLink bandwidth improvements in 2025 specifically targeted this MoE communication pattern, reducing the cost per token for MoE reasoning models by roughly 10x compared to the Hopper generation.

|[^glaM-path]: GLaM demonstrated MoE’s path-dependence in a way that few models do. The architecture was proven at the 1.2T scale, but the economics of deployment were what made it a research artifact rather than a product model. It took another three years for the open-weight movement and the inference optimization stack (vLLM, TensorRT-LLM) to mature enough that MoE became economically viable for companies outside of Google and DeepMind to seriously consider.

The 2024 Frontier: DeepSeek, DBRX, Grok-1

If Mixtral proved that MoE could work in open-source, the models released in 2024 showed that the architecture had matured into a serious competitive advantage for companies that mastered it. Three releases stand out as turning points.

DeepSeek-V2 and V3. The DeepSeek team published what was arguably the most architecturally ambitious MoE design in the LLM era. DeepSeek-V2 (2024) introduced a dual-chunk MoE architecture where each token is routed to a pair of experts: a shared expert that processes every token and a set of fine-grained experts that handle specific routing decisions. The architecture used fine-grained experts (256 experts in V3 versus the 8 that Mixtral used) with only 2 active per token, achieving a 671 billion parameter total with 37 billion active per token. The key insight was that making experts finer-grained (more of them, each smaller) improved routing precision because the gating network could make more targeted decisions. The 2024 arXiv technical report[^deepseek-v3] describes a system where the routing is assisted by a dynamic bias term that eliminates the need for auxiliary loss entirely, a simplification that the DeepSeek team found made training more stable.

MoE with Mixture-of-Attention (MoA) architecture from the DeepSeek paper.

MoE with Mixture-of-Attention (MoA) architecture from the DeepSeek paper.

|[^deepseek-v3]: DeepSeek-V3 was presented at the 2024/2025 research cycle, with the technical report describing a 671B MoE with 256 experts per layer, top-2 gating, and a novel “auxiliary-loss-free” routing design using dynamic expert bias. The active-to-total parameter ratio is 37B/671B, meaning each token activates roughly 5.5% of the model. DeepSeek also published detailed analysis of MoE in reasoning models, documenting how the routing behavior changes when the model is in chain-of-thought mode versus direct answer mode — a finding that has practical implications for deployment.

The community reaction to DeepSeek’s MoE design was notably practical. On Hacker News, one user (logicchains) calculated that the base model training cost was $5.5 million, and others (rockemsockem) pointed out that the MoE routing strategy was explicitly documented in the paper. The pricing discussion was the dominant theme: DeepSeek’s MoE architecture made their API pricing dramatically undercut the competition, because each token only activates 37B worth of computation. This is what happens when MoE shifts from “research architecture” to “cost advantage.” One Alifatisk put it plainly on the DeepSeek-R1 thread: “DeepSeek V3 came in the perfect time, precisely when Claude Sonnet turned into crap and barely allows me to complete something without me hitting some unexpected constraints… their pricing are this cheap! I’ve fully switched to DeepSeek on Aider and Cursor.” That is the real-world consequence of MoE economics.

DBRX. Databricks’ Mosaic AI team released DBRX in March 2024 as a demonstration of what fine-grained MoE could accomplish in the open-source space. DBRX is a 132 billion parameter model with 16 experts and top-4 gating. The choice of 16 experts with 4 active was deliberate: the Databricks team found that this configuration provides 65x more possible routing combinations than Mixtral’s 8-expert/2-active setup, which they argued “improves model quality” without the communication overhead that would come from pushing the expert count higher. DBRX consistently achieved 2x or higher throughput compared to a similarly sized dense model, while performing at a comparable quality level to models roughly twice its size. The model was open-sourced under Apache 2.0 and became one of the more widely deployed fine-grained MoE architectures in the industry.[^dbrx-details]

|[^dbrx-details]: The DBRX paper from Databricks/Mosaic details a decoder-only transformer with 48 layers, each containing a fine-grained MoE block with 16 experts and top-4 gating. The router uses a noisy top-4 selection similar to GShard’s approach, and the model achieves 2x throughput vs. a dense 132B model of comparable quality. The choice of 16 experts with 4 active per token (giving 65x routing combinations vs. Mixtral’s 56) was specifically cited as improving model quality through a more expressive routing function.

Grok-1. xAI’s Grok-1, released in early 2024, was one of the first high-profile closed-to-open MoE models. Grok-1 is a 314 billion parameter model with 8 experts and top-2 gating, activating roughly 70-80 billion parameters per token (25% active). The architecture is conceptually straightforward for a model of its scale: a large dense base with MoE layers distributed throughout the transformer stack. The 25% active ratio (top-2 of 8) is the same ratio used by Mixtral, suggesting that by 2024 this had become the industry consensus for the active-to-total ratio that balances quality and efficiency. Grok-1 was open-sourced under Apache 2.0 and became a reference point for how large a MoE model could be while still being deployable on a single GPU cluster. The Vector Institute benchmarking analysis noted that “the MoE architecture implements eight experts with two of them used per token. This means that only ~25% (or ~79B) parameters are active for any forward pass.” The open release of Grok-1 was significant because it showed that even a company without Google-scale TPUs could produce a competitive frontier MoE model.[^grok-details]

|[^grok-details]: Grok-1’s 314B parameters with 8 experts and top-2 gating puts it in the same MoE configuration family as Mixtral but at a much larger scale. The 25% active ratio (70-80B active parameters) means the model’s per-token compute is roughly comparable to a 70B-class dense model, while its total capacity is closer to a 300B model. This is the MoE scaling trade-off: you get the capacity of a very large model with the compute of a medium one. The model was later followed by Grok-2 and Grok-3, which appear to have moved toward denser architectures or undisclosed hybrid designs. The xAI community noted that the Grok lineage showed the field’s gradual shift from sparse to dense as reasoning capabilities became the primary bottleneck, but the MoE designs remain the default for cost-effective scaling.

Qwen3-MoE. In 2025, Qwen’s Qwen3-MoE emerged as a major milestone for open-weight MoE deployment. Qwen3-MoE uses a 30 billion parameter base with expert counts scaled to 256 per layer (2 active), demonstrating that the fine-grained expert approach DeepSeek pioneered could be replicated by open teams without Google-scale TPU clusters. The Qwen team’s key contribution was not just the architecture but the open release of weights, router, and inference code, which allowed the community to study fine-grained MoE routing behavior at a scale that would have been impractical in the pre-2024 era. Qwen3-Next-80B-A3B took the approach even further with 512 experts per layer (top-10 active), pushing the fine-grained design to its current extreme. The Qwen community response on Hacker News was notably technical, with practitioners debating the trade-off between fine-grained routing precision and the all-to-all communication cost of 512 experts. One commenter noted that “the expert count is so high now that the communication overhead might actually dominate the compute for short sequences” — a valid concern for production workloads that process short prompts.

What all three of these 2024 models have in common is a clear understanding of what MoE gives you and what it does not. The architecture is a capacity multiplier, not a quality multiplier. You get more parameters in the model for the same per-token compute, and that translates to better few-shot performance and broader knowledge coverage. But MoE does not magically make a model smarter. The routing must work, the experts must be well-balanced, and the communication fabric must be fast enough to handle the all-to-all patterns. DeepSeek, DBRX, and Grok-1 each showed that the infrastructure to support large MoE had matured beyond Google’s walled garden. The open-source community could now build and deploy competitive MoE models.

|## MoE in production: the infrastructure bottleneck

There is a gap between an MoE architecture that works in a paper and an MoE model that runs well in production, and it is wider than most people realize. The fundamental tension is this: all experts must reside in memory to be available for routing, even though most are not used for any given input. This means the memory footprint of an MoE model is dominated by its total parameter count, not its active parameter count. A 671 billion parameter model like DeepSeek-V3 requires roughly the same VRAM tier as a 671B dense model to serve, even though the forward pass only activates 37B parameters. The compute is cheap. The memory is expensive.

This creates a deployment problem that dense models do not face. With a dense model, your VRAM requirements scale linearly with your active parameter count. With an MoE model, they scale with total parameters, while your compute savings scale with the active-to-total ratio. As one danielhanchen noted on the HN DeepSeek-V3.1 thread: “For local runs, I made some GGUFs! You need around RAM + VRAM >= 250GB for good perf for dynamic 2bit (2bit MoE, 6-8bit rest) - can also do SSD offloading but it’ll be slow.” That 250GB figure is the MoE tax: the total model weights, not the active weights, must be loaded.

Microsoft’s DeepSpeed team published a comprehensive analysis of this challenge. Their research blog notes explicitly that “MoE models’ large size makes inference difficult and costly” because the routing requires all-to-all communication between GPU devices, and the memory bandwidth for sharding expert weights across devices becomes a bottleneck. The solutions they explored include expert clustering (grouping frequently co-used experts on the same hardware to reduce communication), caching active experts for a batch of requests, and routing at a higher level where the input is first classified into a domain and only the relevant experts are loaded. These are all clever engineering solutions to a fundamental architectural problem.

A new approach to the memory problem is emerging. A 2025 paper from researchers at Tsinghua University introduced FluxMoE, which decouples expert weights from persistent GPU residency through a technique called “expert paging.” Instead of keeping all experts in VRAM, FluxMoE streams expert weights on demand, compressing only the routed-expert weights (using Huffman encoding of exponent bits) to reduce GPU memory usage by 32%. In testing on Mixtral-8x7B on 2xL40S GPUs where standard vLLM could not even fit the model, FluxMoE delivered 4.3x the throughput of KTransformers and 29% lower time-per-output-token. On GLM-4.5 with 160 experts per layer, it achieved up to 7.2x vLLM’s throughput at batch size 256 by freeing KV cache capacity from the weight footprint. This is a different paradigm: instead of trying to fit all experts permanently, treat experts like pages in an operating system.

The hardware industry is responding. NVIDIA’s Blackwell GB200 NVL72 platform connects 72 GPUs with 1,800 GB/s bidirectional NVLink bandwidth, specifically optimized for the all-to-all communication patterns that MoE requires. NVIDIA reports a 10x reduction in cost per token for MoE reasoning models compared to the previous Hopper generation. Software optimizations in frameworks like vLLM and TensorRT-LLM have further improved MoE inference throughput by up to 2.8x, including multi-token prediction and NVFP4 quantization for MoE experts. These advances are rapidly closing the deployment gap between MoE and dense models, but the gap has not closed completely.

The practical upshot for the field is that MoE shifts the bottleneck from compute to memory and bandwidth. This matters for which organizations can afford to deploy frontier MoE models at scale. Google and DeepSeek have the infrastructure. The rest of the industry is racing to catch up. And as the active-to-total ratios of frontier MoEs continue to fall (DeepSeek-V3 at 5.5% active, some newer designs pushing even lower), the memory cost of MoE will become an increasingly significant fraction of the total deployment cost. The architecture’s promise and its paradox are the same thing: enormous capacity at small compute cost, but enormous memory cost at small compute fraction. You get more for less, until you hit the memory wall.

|## What comes next

MoE has gone from an obscure 199135ya idea to the dominant architecture for frontier-scale language models in three decades. The arc is not a straight line. There were long periods where the field forgot about MoE, then remembered it when dense scaling became too expensive, then rediscovered it when MoE itself became too expensive. The routing problem that Jacobs, Jordan, and Hinton identified in 199135ya is still the same problem: how do you split computation across specialized subnetworks without starving any of them? Every major MoE system since 2017 has been a different proposed truce between the competing objectives of quality (routing each token to its best expert) and balance (spreading load evenly across all experts).9

The architecture has settled into a pattern that seems robust. Dense models still win when you need every parameter for every token. MoE wins when the token distribution is uneven and you can afford the memory overhead. The sweet spot for MoE is roughly 5-25% active-to-total ratio, with expert counts in the range of 8 to 64 per layer. Beyond 64, communication dominates. Below 8, you lose the capacity multiplier. These are not hard limits, but they are strong attractors that all current systems cluster around.

One question that will define the next phase of MoE is whether the active-to-total ratio can improve without sacrificing quality. DeepSeek-V3’s 5.5% ratio is impressive but risky: with only 37B active parameters per token, the model is operating with genuinely thin computation. If the active parameters are not doing enough work per token, the model can feel qualitatively “dumber” than a dense model of the same size, even though it has more total knowledge. The tension between capacity (total parameters) and reasoning (active parameters) that freehorse identified on HN about Mixtral is real, and it is the constraint that will drive the next generation of MoE designs.

Another question is whether MoE will become the default architecture for smaller models, not just frontier ones. If the communication and memory overhead can be reduced further, and if the expertise that MoE provides (specialized knowledge in different experts) can be made useful for narrower domains, then MoE could become the standard architecture for any model larger than a few billion parameters. We are not there yet. But the trajectory is clear.

The most important thing about MoE, from my perspective, is that it shows a principle of machine learning that keeps coming back: specialization at scale is powerful, but only if you can manage the coordination cost. The gating network is a coordination mechanism. The auxiliary loss is a coordination subsidy. The capacity factor is a coordination buffer. Every MoE design since Shazeer’s2017 paper has been about a different way of solving the same problem that Jacobs, Jordan, and Hinton posed in 199135ya: how do you get a collection of specialists to work together without any of them going hungry? It is a problem that has survived three decades, three paradigm shifts, and one Turing Award. It is probably not solved yet. But we are much further along than we were in 199135ya. The tension between specialization and coordination that MoE makes visible applies equally to the transformer architecture itself: more parameters help until the attention mechanism’s quadratic scaling becomes the bottleneck, at which point MoE’s linear-in-tokens scaling becomes the natural next step.

|## References

  1. Du, N., Huang, Y., Dai, A. M., et al. “GLaM: Efficient Scaling of Language Models with Mixture-of-Experts.” arXiv, 2021. https://arxiv.org/abs/2112.06905

  2. Lepikhin, D., Lee, H., Xu, Y., et al. “GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding.” ICLR, 2021. https://arxiv.org/abs/200620ya.16668

  3. Fedus, W., Zoph, B., Shazeer, N. “Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity.” JMLR, 2022. https://arxiv.org/abs/2101.03961

  4. Jiang, A. Q., Sablayrolles, A., Roux, A., et al. “Mixtral of Experts.” arXiv, 2024. https://arxiv.org/abs/2401.04088

  5. DeepSeek-AI. “DeepSeek-V3 Technical Report.” arXiv, 2024. https://arxiv.org/abs/2412.19437

  6. Databricks/Mosaic. “Introducing DBRX: A New State-of-the-Art Open LLM.” https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm

  7. xAI. “Open Release of Grok-1.” https://x.ai/blog/grok-os

  8. Shazeer, N., Mirhoseini, A., Maziarz, K., et al. “Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer.” arXiv, 2017. https://arxiv.org/abs/1701325ya.06538

  9. Jacobs, R. A., Jordan, M. I., Nowlan, S. J., Hinton, G. E. “Adaptive Mixtures of Local Experts.” Neural Computation, 199135ya. https://doi.org/10.1162/neco.199135ya.3.1.79

  10. Cai, W., Jiang, J., Wang, F., et al. “A Survey on Mixture of Experts in Large Language Models.” IEEE TKDE, 2025. https://arxiv.org/abs/2407.06204