Forge
Skip to main content

hero handoff to the MoE optimization stack: 200-step trial

GO: hero-mfu30-step231k vs hero-fa4sm100-nomask-step146k on the same batches, steps 231464-231663: MFU +11.9%, loss within noise.
Last updated: October 9, 2026
Status: GO at step 231664 on 2026-10-09; the new run stays on as the hero. hero-mfu30-step231k continues hero-fa4sm100-nomask-step146k from checkpoint 231464 on main with the MoE optimization stack (#9829-#9834) and the launch memory settings from #9958. The old run trained past step 231464 on the same batches, so steps 231464 to 231664 pair the two runs step for step.

Result: GO

hero-mfu30-step231k passed the 200-step paired trial (steps 231464-231663, same batches as the old run).
Training loss: mean difference -8.5e-5, largest 4.2e-4; the new run was lower on 75% of steps. Median MFU 29.44% against 26.32% (+11.9%); median step time 13.33 s against 14.91 s; iteration time 13.43 s against 15.14 s; tokens/s +11.9%. Peak HBM 123.4 GiB against 120.8 GiB (allocator limit 143.8 GiB).
Router z-loss, load-balancing loss, and routing entropy are within 0.5% of the old run. Mean drop fraction 7.66e-5 against 7.25e-5 (the old run's preceding 464 steps averaged 7.47e-5). Total gradient norm ratios over 20 watch steps range 0.72-1.44 (median 1.04), dominated by output_proj; the 09-23 cutover ranged 0.66-1.25. No restart, retry, or error. Not exercised in the window: the run's first eval (step 234000) and a resume from its own checkpoint.
Gate: training loss tracks the old run within bf16 noise (the two previous cutovers drifted one-signed positive by up to about 6e-4 over the window after a restore); MFU about 9% higher (one rack: 28.1% to 30.9%); drop fraction, routing entropy, and router losses unchanged; peak memory within the 143.8 GiB allocator limit; no crash, retry, hang, or watchdog. This is the first run of the stack on 11 racks.

Loss and gradients (steps 231464-231664)


2 selected runs


Routing (steps 231464-231664)


2 selected runs


Throughput and memory (steps 231464-231664)


2 selected runs


Context (steps 228000 onward)


2 selected runs