Skip to main content

OLMo-7B-Twin-2T

Training metrics
Last updated: February 6, 2024
This is a subset of the metrics collected during the OLMo-7B-Twin-2T training run. You can explore the full set of metrics on the runs page here.

Train perplexity



Notes: unlike the other OLMo-7B run, this run had a series of instabilities in the form of "fast spikes," i.e. spikes that recovered quickly. Despite the spikes we decided to keep the run going since it was always able to recover. We believe now that the instabilities were a result of a bad random initialization of the model weights.
💡

In-loop downstream evaluations




In-loop perplexity evaluations





Optimizer metrics




Throughput metrics