Olmo-core 3: resident experts and MoE training throughput
Ai2 released Olmo-core 3 on October 1. The source write-up is at huggingface.co. This is open MoE training infrastructure rather than a new inference checkpoint.
The main change is a move from FSDP-style repeated expert-weight gathering to DDP with resident experts. Instead of constantly moving weight shards, the new design dispatches token activations to the GPUs holding experts. Expert parallelism and pipeline parallelism partition different parts of the work, and the optimizer state is distributed too.
Ai2 measured 52,000 tokens/s/GPU for a 47B-total model on eight B300 GPUs, versus 19,400 for its older approach. Another experiment raised the expert pool from 8 to 128 while routing to four experts per token: 4.6B became 47B total, active parameters remained near 3.2B, and throughput fell under 5%. Both are specific training benchmarks, not general speed guarantees.
Rowwise expert parallelism routes directly to input buffers, GPU-resident routing reduces host synchronization, and grouped GEMM makes expert computation less fragmented. Selective MXFP8 improved end-to-end throughput about 21% versus BF16 in one four-B300 comparison, while peak active memory fell from 103 to 95 GiB.
Limits matter: the 1.2T and 2.38T results demonstrate capacity, not trained model quality. Ai2 reports examples where communication/compute overlap made things slower and the routing-balance metric improved despite worse real load imbalance. None of this is a turnkey single-4090 training recipe. The interesting transfer to inference is the same bottleneck chain: expert placement, routing skew, memory movement and small GEMM utilization.
I'd like matched comparisons using nonuniform routing and several GPU topologies, with a breakdown of all-to-all time, routing synchronization, compute utilization, memory and GPU-hours.
Primary sources:
huggingface.co
allenai/Olmo-core
allenai.org
2 points
Mentions
Awesome work @DZhang50 and team! Its great to see startups building serious language models on top of our work on Olmo3 and allenai/olmo-core
One setup detail: the Docker images include the dependencies, but Olmo-core itself still needs to be installed. The images are tested on Allen AI’s H100 clusters. Different hardware, drivers or CUDA versions may require adapting the Dockerfile. GitHub: allenai/Olmo-core
Olmo-core 3: resident experts and MoE training throughput Ai2 released Olmo-core 3 on October 1. The source write-up is at huggingface.co. This is open MoE training infrastructure rather than a new inference checkpoint. The main change is a move from FSDP-style repeated expert-weight gathering to DDP with resident experts. Instead of constantly moving weight shards, the new design dispatches token activations to the GPUs holding experts. Expert parallelism and pipeline parallelism partition different parts of the work, and the optimizer state is distributed too. Ai2 measured 52,000 tokens/s/GPU for a 47B-total model on eight B300 GPUs, versus 19,400 for its older approach. Another experiment raised the expert pool from 8 to 128 while routing to four experts per token: 4.6B became 47B total, active parameters remained near 3.2B, and throughput fell under 5%. Both are specific training benchmarks, not general speed guarantees. Rowwise expert parallelism routes directly to input buffers, GPU-resident routing reduces host synchronization, and grouped GEMM makes expert computation less fragmented. Selective MXFP8 improved end-to-end throughput about 21% versus BF16 in one four-B300 comparison, while peak active memory fell from 103 to 95 GiB. Limits matter: the 1.2T and 2.38T results demonstrate capacity, not trained model quality. Ai2 reports examples where communication/compute overlap made things slower and the routing-balance metric improved despite worse real load imbalance. None of this is a turnkey single-4090 training recipe. The interesting transfer to inference is the same bottleneck chain: expert placement, routing skew, memory movement and small GEMM utilization. I'd like matched comparisons using nonuniform routing and several GPU topologies, with a breakdown of all-to-all time, routing synchronization, compute utilization, memory and GPU-hours. Primary sources: huggingface.co allenai/Olmo-core allenai.org