SAE Reveals Fine-Grained Specialization of MoE Experts
Research published on Hugging Face Papers demonstrates a method using Sparse AutoEncoders (SAE) to reveal fine-grained specialization of experts in MoE models.
Mixture-of-Experts (MoE) models route each token to a few of many experts, but what an individual expert is responsible for remains unsettled. Existing analyses count how often an expert is selected on domain-labeled corpora, an instrument that fixes the taxonomy in advance, stays coarse, and records selection rather than contribution.
This study investigates specialization in feature space by training a sparse autoencoder (SAE) on the output of an MoE layer. Since this output is a router-weighted sum of expert outputs and the SAE encoder is affine, the pre-activation of every latent decomposes exactly over the experts. The paper received 6 upvotes on Hugging Face Daily Papers.
This item is a summary. The full, checkable version is at the original source:
Read the original at Hugging Face Papers ↗