Skip to main content
535B-A23B 18T Token Hero Run + Scaling Ladder
Last updated: October 23, 2026
Phases:
- Step 0. Start of run.
- Step 58014. Add light weight decay (0.02) to attention gate and router. https://github.com/marin-community/marin/issues/8818
- Step 81716. Improve EP implementation, 21->23 MFU and near-dropless. (Slightly lower train loss from less token dropping)
- Step 108000. Improve datamix. (Slightly higher train loss from harder to predict dataset)
- Step 108778. Disable PDL to work around intermittent hangs
- Step 121638. Fix checkpoint crashing.
- Step 231464. MoE optimization stack: Triton router top-k and short conv, leaner ragged transport, fused expert backward, and XLA scheduling fixes, 26->29 MFU.
Projected Trajectory (at 4k context)
Scaling Ladder
Add a comment
nb!!
Reply
niubility
Reply
bravo!
Reply
loss 咋开始发散了
Reply
I want to be a part of this😭
Reply
COOL
Reply
I want to be a part of this😭
Reply
great job
Reply
solu
Reply
cool
Reply
excellent
Reply
so cool
Reply
太酷啦
Reply
cool
Reply
sudo
Reply
4guoyi
Reply
Très bien
Reply
Hehe
Reply
Thanks for making this so public!
Reply
wonderful
Reply
great job
Reply
bravo!
Reply
🫡
Reply
Su神到此一游
Reply


