Skip to main content
d768 validation: hero MoE optimization stack (#9829–#9834)
Passed: validate9885-d768-20261008 (#9829-#9834, main 9fa23ea1a1) tracks validate9378-d768-20260923 over 11,420 steps: training loss -6.3e-4, final held-out -0.0009, routing unchanged, MFU +5.4%.
Last updated: October 8, 2026
Status: finished on 2026-10-08 at about 23:00 UTC; the result section follows the introduction.
Two from-scratch trainings of the d768 rung of the hero MoE scaling ladder on one GB200 NVL72 rack (64 GPUs). The treatment, validate9885-d768-20261008, runs the hero MoE optimization stack: main at 9fa23ea1a1 (tree-identical to 8be8d83267, the #9834 head it was launched from). The control, validate9378-d768-20260923, ran the code the hero runs today, the #9378 tip 6f1ed72755. Both runs use the same model (8 layers, width 768, 384 routed experts with top-8 routing, ragged all-to-all expert parallelism over the 64 GPUs), data order, optimizer, seed, and batch (1,024 sequences of 4,096 tokens) for 11,420 steps, with a dropless held-out evaluation every 571 steps (20 evaluations).
The stack changes kernels and compiler settings. #9829 runs the depthwise short convolution in a Triton kernel. #9830 selects each token's top-9 router logits with a Triton kernel. #9831 runs all collectives synchronously in a ragged run that recomputes its layers, which is this run's configuration, and raises the memory budget for the hero's layer-carry offload. #9832 leaves transport rows that nothing reads unwritten and builds the dispatch buffer with one scatter. #9833 computes the routing-weight gradient from the QuACK expert MLP's own backward. #9834 applies the SwiGLU backward in the epilogue of the expert dh GEMM and skips the gate/up pre-activation store in a forward pass that the backward recomputes. Each PR reports bitwise-equal outputs or differences at bf16 rounding.
Configuration differences from the control: run identifiers and output paths, W&B code capture off, and a startup watchdog deadline of 80 minutes (control: 30 minutes). Main between the two commits also adds #9882 (a shared sharding check in the FA4 and short-convolution wrappers) and #9777 (an NVIDIA device check before the SM100 QuACK gate); the other main changes do not reach this training path. Both runs pass XLA_PYTHON_CLIENT_MEM_FRACTION=0.75 and memory-limit slop factor 85, which are also the launcher defaults for this rung at both commits.
MFU caveat: at d768, #9831 adds --xla_gpu_disable_async_collectives=ALLCOLLECTIVES, so every collective runs synchronously; the control ran them asynchronously. The hero never runs in this mode, because its layer-carry offload keeps the latency-hiding scheduler on. The d768 MFU change therefore combines the kernel gains with the cost of synchronous collectives, and understates what the stack gives the hero.
Coverage gap: this rung recomputes every layer in the backward (remat mode recompute_all). The hero offloads the layer carry to pinned host memory (offload_carry). This run therefore does not exercise the offload-carry paths: #9831's memory fraction 0.78, slop factor 105, and host-memory-offloading flag, and #9833's saved routed MoE output under that remat policy. One rack also excludes the 11-rack inter-rack collectives. The hero's trial window is the first test of those paths.
Pass criteria: training loss tracks the control within bf16 noise (the two previous d768 validations differed from their controls by a mean of about 1e-4 and a maximum single-step difference of about 0.01); held-out loss within about 0.001 at each evaluation; routing and drop metrics unchanged; MFU above the control's 7.75% (mean step time 0.444 s), subject to the caveat above; no crash and no non-finite value.
Control run | Treatment run | Stack #9885 | Treatment source | Main merge of #9834 | Control source
Result: pass
Both runs completed all 11,420 steps with no non-finite loss and no instability beyond one isolated step (below). Final held-out loss is within 0.001 of the control, routing is unchanged, and throughput is higher. The treatment's training loss is slightly lower than the control's throughout most of the run, by more than earlier validation pairs drifted.
Training loss: the treatment minus control difference averaged -6.3e-4 nats over the full run, with 80% of steps below zero. Mean loss over the last 100 steps was 1.9974 against 1.9980. Step 0 is bitwise equal (11.80409), so the forward pass is unchanged. Per 1,000-step window the mean difference was +4e-6 and -3e-5 over the first 2,000 steps, then between -5.1e-4 and -9.2e-4 in every later window, and -6.9e-4 from the cooldown data switch at step 9,120 to the end, so the shift persists through the learning-rate decay. The shift is in the treatment's favor, but it is larger than the drift between earlier d768 validation runs: the same windows for #9378 against #9215, the 2026-09-22 replay against #9215, and #9378 against that replay stay between -2.9e-4 and +3.3e-4. Each of those pairs also changed code. Here #9834 runs the SwiGLU backward on the unrounded fp32 accumulator and #9833 takes the routing-weight gradient from the expert MLP's backward; this run cannot attribute the shift to either.
Held-out loss: the final dropless evaluation was 2.6783 against 2.6791 (-0.0009; bits per byte 0.9088 against 0.9091). Across the 20 evaluations the mean difference was -0.0005 and the largest 0.0036, at step 2,854; the earlier pairs had mean differences between -0.0008 and +0.0009 and maxima up to 0.0099. The held-out difference is in the same direction as the training shift and within the range of those pairs. Paloma and uncheatable-eval macro losses ended 0.0009 and 0.0011 lower.
Step 4,250: the one step whose training-loss difference exceeds 0.01 (the earlier pairs' maximum was 0.0099). The treatment's loss was 2.2243 against 2.2063 (+0.018) and its total gradient norm 0.582 against 0.354. The excursion is isolated. The 15 steps on each side differ by at most 0.0031; the next step is 2.0898 against 2.0904. Gradient norms at the watched steps 10 before and after are below the control's (0.443 against 0.486, 0.367 against 0.407). The median per-parameter gradient-norm ratio is 1.86 at step 4,250 and 0.95 to 1.02 at steps 4,230 to 4,270 otherwise; the elevation spans 7 of the 8 layers and expert, shared-expert, attention, and norm weights, with no dominant layer or router. Routing at that step is normal: drop fraction 0.0003 against 0.0004, router z-loss 9.5 against 10.0, and per-layer margins differ no more than at neighbouring steps. Steps with a difference above 0.005 number 46, against 45 to 48 in the earlier pairs.
Routing matched: token-drop fraction 7.17e-4 against 7.08e-4, routing entropy 5.944 for both, load-balancing loss 8.047 for both, and router z-loss 9.63 against 9.72 (99th percentile 31.4 against 31.3). The QB margin maximum averaged 22.2 against 21.1; the earlier validation runs averaged 21.1 to 23.5. Total gradient norm averaged 0.351 against 0.350.
Throughput, over steps 100 onward: MFU 8.18% against 7.75% (+5.4%), mean step time 0.426 s against 0.444 s (-4.2%), and 11.43M against 10.84M tokens per second. Peak device memory was 5.495 against 5.503 GiB. The gain includes the cost of the synchronous collectives that #9831 imposes at this rung and the hero does not use, so it does not predict the hero's gain; the hero trial window measures that.
After training finished, Iris retried the coordinator when its pod was deleted, and the retry restarted the training gang. It was cancelled after about two minutes, before it logged any step; the run's history and final checkpoint (step 11,420) are from the original attempt.
Loss
Routing
Throughput, memory, and gradients
Add a comment