Forge
Skip to main content

150M concept memory vs earlier 100M runs

The running 150M leaderboard model next to the 100M runs of 9-14 Nov, at equal tokens.
Last updated: November 15, 2026

What this compares. The 150M concept-memory model now training on AWS (stage 1 of 3, Muon optimizer, new data mix with Nemotron web, Q&A and maths, SmolLM2 tokenizer) against the earlier 100M runs.

How to read it. Lower held-out loss is better. Compare at the same step: every run here takes 524,288 tokens per step, so equal steps = equal data seen.

Two caveats. (1) The old-mix runs are scored on the old mix's held-out text and the 150M run on the new mix's, so the first chart compares shapes and gaps, not an exact winner. (2) The November 9-13 runs used the GPT-2 tokenizer and different data, so their loss (about 3.1) cannot be set against the new runs' (about 2.2); they are shown for the race between architectures, which ended in a near tie.



Same tokenizer: the 150M run against the 100M runs of 14 Nov


3 selected runs


The 100M architecture race, 9-13 Nov (GPT-2 tokenizer, 3.0B tokens)

These are the four designs now queued at 150M, plus plain full attention. They ended within 0.006 of each other.




5 selected runs