MoE

A 671-billion-parameter AI model does not necessarily use 671 billion parameters to generate your next token.

一个拥有6710亿参数的AI模型并不一定使用这6710亿个参数来生成你的下一个令牌。

That sounds ridiculous until you understand Mixture of Experts.

这听起来很荒谬，直到你了解专家混合模型的概念。

Imagine a normal dense Transformer. Every token enters a layer and goes through the same set of parameters. Whether the token is Python, mitochondria, or bonjour, the same feed-forward network does the work.

想象一个普通的密集型Transformer。每个令牌进入一个层并经过相同的参数集。无论是Python、线粒体还是Bonjour，相同的前馈网络都会执行相同的任务。

Now make one change.

现在做一个改变。

Instead of one feed-forward network, put many of them inside the layer. Then add a tiny router whose job is to look at the current token representation and decide which few should handle it.

与其使用单一的前馈网络，不如在层中嵌入许多这样的网络。然后加入一个小型路由器，其任务就是查看当前的令牌表示，并决定哪几个网络来处理它。

Those networks are the "experts."

这些网络就是“专家网络”。

The name makes them sound like someone manually created a coding expert, a French expert, a mathematics expert, etc. That isn't generally how it works. They are learned subnetworks, and whatever specialization emerges is learned during training.

这个名字让人联想到有人手动创建了一个编码专家、法语专家、数学专家等。但实际上并非如此。它们是学习得到的子网络，任何特定的专长都是在训练过程中习得的。

This gives you a strange machine.

这让你得到了一台奇怪的机器。

The model can contain an enormous number of parameters, while only activating a fraction of them for each token.

该模型可以包含大量参数，但每个令牌只激活其中的一小部分。

DeepSeek-V3 is the cleanest example => 671B total parameters, about 37B activated for each token.

DeepSeek-V3 是最清晰的例子 => 总共 671B 参数，每个令牌约 37B 参数被激活。

So when it generates one token, hundreds of billions of parameters can effectively sit that computation out.

因此，当它生成一个令牌时，数百亿参数实际上可以不必参与计算。

That is the trick behind sparse MoE => increase the model's parameter capacity without paying the compute cost of running every parameter for every token.

这就是稀疏 MoE 背后的技巧 => 增加模型的参数容量，而不必为每个令牌运行所有参数。

But there is a catch that gets missed constantly.

但有一个经常被忽视的难点。

Inactive does not mean nonexistent.

“不活跃”并不意味着不存在。

Those weights still have to be stored somewhere. On large deployments they are spread across accelerators, and now the router has to send tokens to the right experts. Some experts can receive more traffic than others. Moving those token representations between devices creates communication overhead. Keeping expert utilization balanced becomes an engineering problem of its own.

这些权重仍然需要存储在某个地方。在大规模部署中，它们分散在多个加速器上，现在路由器必须将令牌发送给正确的专家。有些专家可能接收的流量比其他专家多。在设备之间传输这些令牌表示会引入通信开销。平衡专家的使用率本身就是一个工程难题。

So a 671B MoE is not secretly a 37B model.

因此，671B的MoE模型并非真正的37B模型。

It is a 671B model that has learned to choose which parts of itself are worth waking up for each token.

这是一个拥有671B参数的模型，它已经学会了选择每个令牌应激活的自身部分。

And once you understand that, some of the strangest parameter counts in modern AI suddenly make a lot more sense.

一旦你理解了这一点，现代AI中一些最奇怪的参数数量就会变得合理多了。