LLM Time Travel Visualization Proposal

It is difficult to explain to people who have not been using them from the start how remarkable the steady, rapid, broad increase of LLM capabilities has been. If you weren’t using GPT-2 in 2019, how could you possibly feel the success of scaling in your gut? (Especially now that the capability improvements are increasingly beyond the ability of people to see in even their own specialties.)

There are few valid comparisons between davinci and GPT-6 Astra Pro, because scarcely any benchmark has nontrivial scores for both, and what comparisons do exist (like perplexity) are uninterpretable.

Worse, there is a certain reflexive skepticism of any retrospective benchmarking, like the OpenAI “Progress” page, which shows 4 fixed prompts through GPT-1, GPT-2, text-davinci-001, gpt-4-0314 and GPT-5, and yet convinces no one.

I suspect that deep down, people assume that the comparisons are cherrypicked or narrow or misleading somehow. They weren’t paying attention, so they may not know how exactly they are being tricked by something like the METR time horizon chart, but they feel a (usually rational) “epistemic learned helplessness” around these ‘breathless sci-fi claims’.

People who were using LLMs that far back have an advantage: they made their own predictions, however informally, and lived to see themselves refuted. Many of these predictions were on idiosyncratic tasks not represented on any benchmarks. I am struck, when I ask people “how were you convinced of DL scaling, and LLM scaling in particular?”, at the variety of answers. One person told me that they were convinced when a ChatGPT upgrade could write a pornographic short story; another when it analyzed a legal brief correctly; and a major AI policymaker told me he began to believe in scaling when GPT-4 o1 was able to solve tricky unpublished crossword puzzle clues made by his grandma that no previous LLM could solve. And Simon Willison has his famous pelican SVG (and dramatizing the problem, every time the pelican improves, naysayers immediately claim that the frontier labs must have trained specifically on pelican data—no matter how little evidence there ever is for this or how broad capability gains are).

I have become convinced that just as there is no such thing as an AI “warning shot”, there is also no such thing as an AI “Sputnik moment” for people in general. What convinces each person is idiosyncratic and unpredictable—even by them!

We should encourage people to try lots of weird and unusual things with chatbots! It is critical to expose people to as many diverse task outputs as possible, and regularly, over time, so they can see the progress with their own eyes—on tasks they chose or which were at least “pre-registered” in some sense, and much more diverse than standard benchmarks, allowing for surprising outputs. (This is one reason it is concerning that frontier lab employees often use LLMs in narrow, thoughtless ways, such as exclusively for coding.)

Unfortunately, for most people, September 2026 is already too late in the game to be convinced usefully soon. By the time they have seen two or 3 years of undeniable progress, and no longer believe that DL scaling has “hit the wall” or is “just stochastic parrots”, a dangerous amount of time will have passed and many actions will no longer be possible.

How can we help them more quickly?, Ideally, before 2027?

We can simulate the experience of traveling back into the past and then living through the past decade as they try out their private idiosyncratic prompts year after year, and see the continual progress and how eventually some LLM release solved it. Then they may understand things better than some dry graphs or anecdotes.

And how do we do that? Well, the nice thing about LLMs is that time travel is easy: just fire up an old checkpoint and talk to it!

So, a useful AI public outreach project would be to gather up all the old models one can, run them on a server or through an API, and then provide a simple web page GUI around them to let people sample from an old 2015 char-RNN (possibly retrained, see NanoGPT), 2016 Google LM1B, a 2018 GPT-1-117M, a 2019 GPT-2-1.5b, a 2020 GPT-3 (OpenAI has some odd retrained GPT-3s it could still provide), a 2021 GPT-J-6B, a 2022 GLM-130B or FLAN-T5-XXL, a 2023 Qwen-72B-Chat, a 2024 DeepSeek-V3, a 2025 GLM-4.7 or GPT-5.2 Pro or Kimi K2 Thinking, a late-2026 Astra or Fable or GLM-5.3, etc. Where to start is unclear because early models were so weak or so specialized; for the ending, the best available model should be used to represent the status quo. (Resources and latency permitting, one could even try to create a ‘future’ model which extrapolates out a few months by doing best-of-n sampling over responses from multiple frontier models, although it’s unclear how well this works in general.)

Depending on whether the service is abused, the prompts and responses could be made public. This would discourage abuse of the project, and may be an interesting corpus in its own right, in showing how people test out LLMs or what ‘weird’ prompts they try, although it would also unfortunately discourage people from trying very personal or private prompts, of the sort which may be most convincing.

This could be extended further. Correctness or quality may not be obvious, especially for early (low quality) and later (high quality) models. So the best available model (preferably not the same one used to represent the status quo ‘today’) could be used to grade each response 1–10, and a simple graph over time displayed. (And then overlaid on this graph could be a density plot of all estimated quality curves, for context.)

We could also imagine updates: perhaps users could register a prompt and email address, and be emailed an updated output/score for every new model semi-annually?

This would be an excellent project for Anthropic and OpenAI, who have emphasized the need for public education and outreach, and have access to extensive archives of model snapshots along with the engineering talent to resurrect old ones. It could also be done by small nonprofits like the just-formed “Old Models Foundation”, or possibly LM Arena.