EMO: Pretraining mixture of experts for emergent modularity

Researchers propose EMO, a pretraining method using mixture of experts to achieve emergent modularity in neural networks, enhancing specialization and efficiency.