Knowledge Maxxing← Back to topic library
AI MaxxingAdvanced90 sec
Mixture-of-Experts
A Mixture-of-Experts model contains several specialised neural-network components but activates only selected experts for each input.
Prerequisite: Transformer
What matters
A router scores the available experts and sends each token or example to a small subset. The model can therefore contain many parameters without using every parameter for every computation.
Example
Different experts may become more useful for code-like inputs, dialogue or other learned patterns, although their specialisation is learned rather than manually assigned.
Common misconception
The experts are not normally separate human-designed subject bots debating one another.
Recall question
Why does a Mixture-of-Experts model not activate all its parameters for every token?
Keywords and related concepts
sparse modelexpertsroutergatingneural network