Forge
Skip to main content

535B-A23B 18T Token Hero Run + Scaling Ladder

Last updated: October 23, 2026
Phases:
  1. Step 0. Start of run.
  2. Step 58014. Add light weight decay (0.02) to attention gate and router. https://github.com/marin-community/marin/issues/8818
  3. Step 81716. Improve EP implementation, 21->23 MFU and near-dropless. (Slightly lower train loss from less token dropping)
  4. Step 108000. Improve datamix. (Slightly higher train loss from harder to predict dataset)
  5. Step 108778. Disable PDL to work around intermittent hangs
  6. Step 121638. Fix checkpoint crashing.
  7. Step 146139. Native SM100 FA4 attention kernels and mask-free ragged expert MLP, 24->26 MFU.
  8. Step 231464. MoE optimization stack: Triton router top-k and short conv, leaner ragged transport, fused expert backward, and XLA scheduling fixes, 26->29 MFU.
  9. Step 257553. 4k->16k seqlen

9 selected runs




9 selected runs

Projected Trajectory (at 4k context)

13 selected runs

Scaling Ladder

22 selected runs

Yu Hu
Yu Hu •  
nb!!
Reply
Freeman Long
Freeman Long •  
niubility
Reply
LU ZHANG
LU ZHANG •  
bravo!
Reply
Xinlong Zhang
Xinlong Zhang •  
loss 咋开始发散了
Reply
ShawingZ
ShawingZ •  
I want to be a part of this😭
Reply
Xiaoliang Shi
Xiaoliang Shi •  
COOL
Reply
Chandler Alexander
Chandler Alexander •  
I want to be a part of this😭
Reply
QIANGGE QIANG
QIANGGE QIANG •  
great job
Reply
Trangle Heshvp
Trangle Heshvp •  
solu
Reply
frank
frank •  
cool
Reply
Fangwei Xiang
Fangwei Xiang •  
excellent
Reply
YongHua Hei
YongHua Hei •  
so cool
Reply
92Gexin
92Gexin •  
太酷啦
Reply
孔咲梨
孔咲梨 •  
cool
Reply
Yuchen Wang
Yuchen Wang •  
sudo
Reply
Chuiyun Wu
Chuiyun Wu •  
4guoyi
Reply
Houyi
Houyi •  
Très bien
Reply
Daniel Tobi
Daniel Tobi •  
Hehe
Reply
Julián Núñez
Julián Núñez •  
Thanks for making this so public!
Reply
Sangwu Lee
Sangwu Lee •  
wonderful
Reply
mathsfan
mathsfan •  
great job
Reply
Yuan Ji
Yuan Ji •  
bravo!
Reply
Ruxue Ji
Ruxue Ji •  
🫡
Reply
Mi kasi
Mi kasi •  *
Su神到此一游
Reply