Seems like a very brittle "benchmark" but still a fun observation :)

Agreed! Though it's perhaps more fair than it seems due to every model except Mistral here using the same tokenizer (NeoX) and being trained on 300B tokens from The Pile.

One attempt or repeated queries?

is the sequence correct though?

Yup, the y-axis here shows how many digits each model got right before making an error. All models were queried with top_k=1