Mixture of Experts Explained: How Sparse AI Models Scale

When a modern language model claims hundreds of billions of parameters yet runs at the speed of a far smaller one, the trick is almost always the same architecture. Mixture of experts is the design that lets a network hold an enormous number of parameters while using only a small slice of them for any given word. The payoff is a model that learns the capacity of a giant but pays the inference bill of a dwarf. This piece explains how the idea works inside a transformer, traces its path from a 2017 research paper to the open models of 2026, and is honest about the costs the technique quietly carries.

The core idea: sparse activation

A standard transformer runs every input through every parameter. Each token passes through the attention layers and then a feed-forward network, and all of those weights participate in the computation. That is dense activation, and it means compute scales directly with parameter count. Double the parameters and you roughly double the work for every token, which puts a hard ceiling on how large a model you can afford to run.

Sparse activation breaks that link. Instead of one big feed-forward network, the layer holds many smaller ones, called experts, and a small gating network decides which experts handle each token. If a layer has eight experts but only two run for a given token, the model carries the knowledge of all eight while doing the work of two. Parameter count and compute, once chained together, are now free to move independently. That separation is the whole reason the technique exists, and it is why a sparse model can be vastly larger than a dense one running on the same hardware.

How mixture of experts works inside a transformer

The mechanics are more concrete than the abstraction suggests. In a transformer, a mixture of experts layer modifies only the feed-forward block. The attention mechanism stays shared across all tokens exactly as before. What changes is that the single feed-forward network becomes a set of parallel experts, each one its own small feed-forward network, sitting behind a gate that routes traffic. Every transformer layer can have its own independent set of experts, so a token might use one combination of experts early in the network and a different combination later.

The router

The router, sometimes called the gating network, is surprisingly simple. It multiplies the incoming token representation by a weight matrix to produce one score per expert, then applies a softmax to turn those scores into probabilities. A top-k operation keeps only the highest-scoring experts, typically the top one or two, and discards the rest. The selected experts each process the token, and their outputs are combined as a weighted sum using the router's normalized probabilities. The community has found that even sending each token to just two experts is enough for strong performance, which keeps the per-token cost low.

Routing strategy is an active area of design. Top-k gating is the workhorse, but variants exist, including expert-choice routing where experts pick tokens rather than the reverse, along with soft and differentiable gating schemes that smooth the otherwise hard selection. Each tries to solve the same underlying tension: route tokens to the experts that serve them best while keeping the work balanced across the whole set.

The experts

An expert is not a separate model with its own specialty assigned by hand. It is simply one of the parallel feed-forward networks, and what each one comes to represent emerges during training rather than being designed. Over many batches, the router and the experts co-adapt, and individual experts drift toward handling particular kinds of tokens, though the specialization is rarely as clean or interpretable as the name implies. A useful refinement, introduced in some recent models including DeepSeek's, adds a shared expert that every token passes through unconditionally, alongside the routed experts. The shared expert captures common patterns so the routed ones can specialize, and it does not need the router at all.

A short history, from 2017 to today

The modern lineage starts with a 2017 paper from Noam Shazeer and colleagues, including Geoffrey Hinton and Jeff Dean, titled Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. It introduced the sparsely-gated layer and scaled a recurrent translation model to 137 billion parameters, an enormous figure for the time, while keeping inference fast through sparsity. The work also named the problems that would haunt the field for years, including heavy communication costs between experts and instability during training.

Google's GShard, published in June 2020, carried the idea into transformers at scale, pushing past 600 billion parameters by replacing every other feed-forward layer with a mixture of experts layer using top-2 gating. GShard introduced the concept of expert capacity and the routing tricks needed to keep training balanced across thousands of distributed cores. It proved the technique could work in the transformer architecture that everything else was converging on.

The turning point for visibility was Switch Transformers, whose paper appeared in 2021 and was published in early 2022. As the Hugging Face overview documents, it simplified routing to a single expert per token, showed a roughly 4x pretraining speedup over the dense T5-XXL model, and released an open model with 1.6 trillion parameters spread across 2048 experts. Reporting around the architecture noted pretraining up to 7 times faster than a dense model of equivalent parameter count. Switch Transformers made the case that single-expert routing, long thought too unstable, could be both stable and effective.

Then in December 2023, Mistral released Mixtral 8x7B and brought the architecture into the open-source mainstream. Its design was clean and fully documented: eight experts per layer, two activated per token. The model holds 47 billion total parameters but uses only about 13 billion per token, and it matched or beat the dense Llama 2 70B on most benchmarks while using roughly a fifth of that model's inference compute. Mixtral, more than any paper, is what made mixture of experts a mainstream tool rather than a research curiosity.

What the experts actually learn

It is tempting to imagine that one expert handles grammar, another handles math, and a third handles code, each a tidy specialist. Reality is messier. Because the specialization emerges from training rather than being assigned, the patterns experts pick up are often hard to name and do not line up with human categories. Studies that probe trained routers tend to find that experts specialize at the level of tokens and syntax more than at the level of topics, so an expert might activate on certain punctuation or word types across many subjects rather than on a clean domain.

This fuzziness is not a defect. The point of the architecture is not interpretable specialists but a larger pool of parameters the model can draw on selectively, and the router learns whatever division of labor lowers the training loss. The shared-expert refinement leans into this understanding. By guaranteeing that every token passes through one common expert, the design lets the routed experts drift toward rarer patterns without having to relearn the basics each time, which tends to use the capacity more efficiently than routing alone. The takeaway is that an expert is a unit of capacity, not a labeled module, and reading too much intent into which expert fires for which token usually overstates how organized the system really is.

The load balancing problem

Sparse routing has a failure mode built into it. Early in training, some experts happen to perform slightly better, so the router sends them more tokens, so they train faster and get even better, so they receive still more tokens. Left alone, this feedback loop collapses the model onto a handful of overworked experts while the rest sit idle, wasting most of the capacity the architecture was supposed to provide.

The standard fix is an auxiliary loss added during training, sometimes called a load balancing loss. It penalizes the model when token assignments grow lopsided, nudging the router toward spreading work so that all experts receive a roughly equal share of examples. This extra term does not directly improve the task the model is learning. Its only job is to keep the experts evenly used, which preserves the effective capacity of the layer and stops the self-reinforcing collapse before it starts.

Getting the strength of this auxiliary loss right is a balancing act of its own. Too weak and the experts collapse; too strong and the router is forced to spread tokens so evenly that it can no longer send each one to the expert that would actually serve it best. Much of the practical craft of training sparse models lives in tuning this single term.

Expert capacity and the cost of routing

Distributed training adds a hard constraint the math has to respect. When experts are spread across many devices, each can only process so many tokens per batch before it runs out of room, so the architecture sets an expert capacity. The formula is straightforward: capacity equals the number of tokens per batch divided by the number of experts, multiplied by a capacity factor that gives some headroom.

That capacity factor is a real dial. The Switch Transformers team found their models performed well at low capacity factors of about 1 to 1.25, meaning only modest headroom above the perfectly balanced case. The reason to keep it low is communication cost. Every token routed to an expert on another device has to travel there and back, and that network traffic, not the arithmetic, is often the real bottleneck in training a large sparse model. When a token's chosen expert is already at capacity, the token is dropped from that layer or passed through unchanged, which is the price of keeping the system within its memory and bandwidth budget.

The memory catch: total versus active parameters

Here is the cost that marketing numbers tend to bury. A sparse model is cheap in compute but not in memory. Even though only two of Mixtral 8x7B's experts run for any token, all of them must be loaded and resident, because the router might call any expert for the next token. As the Hugging Face writeup spells out, serving Mixtral requires enough memory to hold a model of roughly 47 billion parameters, even though the active compute per token resembles a model closer to 13 billion.

This is why the naming can mislead. Mixtral 8x7B is not eight independent 7-billion-parameter models, and its total is not 56 billion, because the attention layers and other components are shared rather than duplicated, which is how the total lands near 47 billion. The lesson for anyone planning hardware is to budget VRAM against the total parameter count and budget speed against the active count. A sparse model gives you the inference latency of a small model only if you can first afford to load a large one into memory. That asymmetry is the central trade of the whole approach, and ignoring it leads to a model that benchmarks beautifully on a machine you cannot actually run it on.

Comparing the cost: a dense model and a sparse one

A concrete comparison makes the bargain vivid. Picture a dense model with 13 billion parameters. Every token it processes touches all 13 billion, so its quality is bounded by that capacity and its speed is set by it too. Now picture Mixtral 8x7B. On any token it also does roughly 13 billion parameters of work, so its per-token speed is in the same neighborhood as the dense 13B model. The difference is that across different tokens it can reach into a 47-billion-parameter pool, so the knowledge it can apply is far larger than the dense model's, even though the work per token is comparable.

That is the entire pitch of the architecture in one image. You get the inference speed of a small model with access to the capacity of a large one, which is why Mixtral could rival a dense 70-billion-parameter model while costing far less to run. The bill comes due in memory, since the full 47 billion parameters have to be loaded even though only a fraction fire per token, and in training complexity, since the router and the load balancing have to be tuned carefully. For workloads where memory is available but compute is the binding constraint, which describes most large-scale serving, the trade strongly favors the sparse design. For a hobbyist with a single consumer card and limited memory, the same trade can be the wrong one, because the memory floor is set by the total parameters rather than the active ones.

Fine-tuning sparse models

The advantages of sparse models do not transfer cleanly to every downstream setting. The Switch Transformers work observed that at a fixed pretraining perplexity, a sparse model tended to do worse than its dense counterpart when fine-tuned on reasoning-heavy benchmarks such as SuperGLUE. The extra parameters that helped during pretraining did not always pay off on hard, narrow tasks, and sparse models showed a greater tendency to overfit during fine-tuning.

The picture brightened with instruction tuning. Later research found that mixture of experts models benefit disproportionately from instruction tuning, with the improvement larger than the equivalent gain a dense model saw from the same treatment. In other words, the way you adapt a sparse model matters a great deal, and the techniques that work best on dense models are not always the right ones here. This is part of why the open sparse models that caught on, Mixtral among them, shipped instruction-tuned variants rather than leaving users to fine-tune raw weights.

Where mixture of experts is heading

The trajectory is clear even if the destination is not. Sparse architectures have moved from a niche scaling trick into the default way to build very large models, precisely because they let parameter count grow without dragging inference cost up in lockstep. The open releases of 2026 lean on the design heavily, and refinements like shared experts and smarter routing keep chipping at its weak spots. The memory cost remains the stubborn limit, and a good deal of current work aims at running the experts more cheaply, through quantization and through serving systems that stream experts in and out of memory rather than holding all of them resident.

For anyone trying to understand why a model can be both huge and fast, the answer almost always comes back to this architecture. Mixture of experts decouples what a model knows from what it computes per token, and that single move is what makes trillion-parameter models thinkable at all. The technique is not magic, and the memory bill is real, but the core bargain it strikes, vast capacity for modest per-token compute, has proven durable enough that it now sits under a large share of the systems people use every day. Understanding it is the difference between treating those parameter counts as marketing and reading them for what they actually say about a model's cost and capability.