Update: reproducing GPT-2-1.5B cost $672, running on one 8XH100 GPU node for 24 hours. https://x.com/karpathy/status/1811467135279104217

---

And reproducing GPT-2-1.5B should cost 100x less than in 2019.

Reproducing GPT-2 (124M) in llm.c in 90 minutes for $20 · karpathy/llm.c · Discussion #481

It was a 124M GPT-2 architecture Transformer, on 10B tokens of FineWeb. The parameter count and the dataset token count matches the original 124M GPT-2.

With llm.c, which is quite efficient at up to ~60% model flops utilization, reproducing this model on one 8X A100 80GB SXM node takes ~90 minutes. For example, on Lambda this node goes for ~$14/hr, so the total cost of reproducing this model today is about $20. You can train the model with a single GPU too, it would just take proportionally longer (e.g. ~4-24 hours depending on the GPU).

For reference, training of the GPT-2 (1.5B) on 10B tokens in 2019 cost $50,000. If we assume Compute is 6 * Parameter * Token count (C = 6ND), then it means training GPT-2 1.5B today would cost $250.

Surely a lower bound since parallelizing would have overhead, but I think reproducing the entire GPT-2 1.5B today would cost less than $500, because the overhead shouldn't be that high (see below).

---

Reproducing GPT-2 in llm.c | Hacker News

The 350M model I trained last night was 30B tokens, 14 hours, ~$200. Conveniently, 300B is exactly 10X the tokens so ~$2K would be the estimate. You'd have to wait 140 hours on one box though. Getting an H100 box instead of A100 will already cut the time latency down probably by a factor of 2-3X, for free, even without going to fp8 (which we do plan to support).

Assuming the C = 6ND formula, training a 350M model with 30B tokens would cost 350/124 * 30/10 * 20 = $170, which is only a 20% overhead.