Forge
Skip to main content

Hero cutover: native SM100 FA4 and mask-free ragged MLP, 200-step overlap at checkpoint 146139

hero-main-step121638 versus hero-fa4sm100-nomask-step146k: loss, performance, drops and router diagnostics over steps 146139-146338.
Last updated: September 23, 2026
Production d6144 cutover: hero-main-step121638 (e8460b35bd) versus hero-fa4sm100-nomask-step146k (ad754c6d67). The new run deploys the native SM100 FA4 kernels (#9332), ragged MLP tail-mask removal (#9333), coordinated GC every 100 steps (#9266), and a data-loader thread-hang fix (#9376). Resume checkpoint: step-146139; W&B fork boundary: _step=146138. Paired trial: 200 updates on the same batches, steps 146139-146338. Blue is the old run; orange is the new run. History before the fork is inherited.
Gate: training loss tracks the old run step for step within bf16 noise; MFU at least 5% higher after startup (one-rack measurement +11.8%); drops, routing entropy, load-balancing and router z-loss, gradient norm unchanged; peak memory equal or lower; no crash, retry, watchdog, hang, or new alert. Not exercised by the window: evaluation (next at step 147000), the new run's own temporary save and resume, an Iris retry, and the first permanent save (step 150000).

Result: GO

Over all 200 paired steps: training loss +3.6e-4 on average (maximum 8.8e-4), rising over the window, matching the accepted step-121638 cutover (+3.1e-4, maximum 8.2e-4); median MFU 26.75% against 24.10% (+11.0%); median step time 14.67 s against 16.29 s; iteration time 15.53 s against 18.62 s; tokens/s 3.14M against 2.83M; drop fraction 0.95e-4 against 1.01e-4; router z-loss 15.13 against 15.09; load balancing 7.998 against 8.000; gradient norm 0.692 against 0.719; peak memory 103.6 GiB against 120.9 GiB. One execution, no restarts or errors. Coordinated GC pauses rank 0 for about 1.2 s per collection.
Validation: single-rack d768 run validate9378-d768-20260923 tracked the previous handoff's validation within bf16 noise over 11,420 steps (mean loss difference -1.6e-5), with MFU +10.7%.

2 selected runs

Wider context.

2 selected runs

Per-layer router z-loss, all 48 layers.

2 selected runs