Holy hell, I was able to reproduce this -- gpt 3.5-turbo-instruct can play full games of chess without losing board state or suggesting illegal moves. This puts it at ~1800 Elo, around 79th percentile of human players on Lichess.
Grant Slatton
 @GrantSlatton
18 Sep 2023
The new GPT model, gpt-3.5-turbo-instruct, can play chess around 1800 Elo. I had previously reported that GPT cannot play chess, but it appears this was just the RLHF'd chat models. The pure completion model succeeds. nitter.net/GrantSlatton/sta… See game & thoughts below:

Sep 19, 2023 · 2:59 AM UTC

13
20
8
196
What's spooky about this is it was calculating successfully *several moves ahead*, which I wouldn't expect it to be able to do particularly well. Note that this means it can: a) calculate legal moves b) has a pretty good 'chess engine' inside it somehow, despite being an LLM, because it's suggesting good moves for the opponent c) figure out good responses to these moves, including clever tactics :/ Games, for reference: //game record: lichess.org/3cRBGDQv // GPT-3.5-completer vs. Stockfish Level 4. GPT-3.5 wins. [Event "Shamkir Chess"] [White "Anand, Viswanathan"] [Black "Topalov, Veselin"] [Result "1-0"] [WhiteElo "2779"] [BlackElo "2740"] 1. e4 c5 2. Nf3 d6 3. d4 cxd4 4. Nxd4 Nf6 5. Nc3 a6 6. Be3 e6 7. f3 Nbd7 8. Qd2 h6 9. O-O-O d5 10. exd5 Nb6 11. dxe6 h5 12. exf7+ Kxf7 13. Qf2 Bc5 14. Nf5 Bxe3+ 15. Nxe3 Bd7 16. Nc4 Nxc4 17. Bxc4+ Kg6 18. Qg3+ Kh7 19. Bd3+ Kg8 20. Bc4+ Kh7 21. Rhe1 Rc8 22. Bd3+ Kg8 23. Bf5 Qc7 24. Re7 Qxg3 25. hxg3 Bxf5 26. Rxb7 Re8 27. Ra7 Kf8 28. Rxa6 Be6 29. a4 h4 30. gxh4 Bc4 31. Rc6 Be2 32. Nxe2 Kf7 33. Rc7+ Ke6 34. Nf4+ Kf5 35. g3 Rg8 36. Rc5+ Re5 37. Rxe5+ Kxe5 38. a5 Kf5 39. a6 Ra8 40. Rd6 Ne8 41. Rb6 Ke5 42. b4 Nd6 43. b5 Rg8 44. a7 Ra8 45. Ra6 Kd4 46. Rxd6+ Kc5 47. Rd7 Kxb5 48. Rxg7 Ka6 49. h5 Kb5 50. h6 Ka5 51. h7 Kb5 52. Rg8 Rxa7 53. h8=Q Ra4 54. Qe5+ Kb6 55. Rb8+ Ka7 56. Qc7+ Ka6 57. Qb6# 1-0 //game 2, against Stockfish Level 5. GPT-3.5 wins. //lichess.org/2zpuSTZG [Event "Shamkir Chess"] [White "Anand, Viswanathan"] [Black "Topalov, Veselin"] [Result "1-0"] [WhiteElo "2779"] [BlackElo "2740"] 1. e4 e6 2. d4 b6 3. Nf3 Bb7 4. Bd3 Be7 5. O-O d6 6. c4 Nd7 7. Nc3 e5 8. d5 Ngf6 9. Bc2 O-O 10. Re1 c6 11. dxc6 Bxc6 12. b4 Bb7 13. Bb3 Re8 14. a3 h6 15. Bb2 Bf8 16. Qd3 g6 17. Rad1 Rc8 18. Nb5 Re6 19. Nxa7 Ra8 20. Nb5 Rc8 21. Nd2 Nh5 22. g3 Ng7 23. Nf1 f5 24. c5 fxe4 25. Qe2 d5 26. Rxd5 bxc5 27. Red1 Bxd5 28. Bxd5 Kh7 29. Bxe6 Nxe6 30. Qg4 Qe8 31. Nd6 Bxd6 32. Rxd6 cxb4 33. Rxe6 Nb8 34. Rxe8 Rxe8 35. Qxe4 Nd7 36. Qb7 Rd8 37. Qc7 bxa3 38. Bxa3 Rg8 39. Qxd7+ Rg7 40. Qe6 Kh8 41. Bb2 Re7 42. Qxe7 h5 43. Bxe5+ Kg8 44. Qg7# 1-0
2
2
2
25
That said, I tried it with Fischer Random chess, a variant in which the order of the back row of pieces is randomized. It suggested an illegal move on move 2. (Was able to replicate this a few times, with different starting positions.)
1
1
1
12
Replying to @nabeelqu
but I would like to see if the model really sees the error in the middle of the game. Can you play a perfect game and make a bad move on move 20 for example? GPT sees the error and takes advantage?
1
it does
1
1
more replies
Replying to @nabeelqu
Have you also tried this with gpt-4?
1
No but the op did if you go into the original tweet
1
Replying to @nabeelqu
What does "instruct" mean?
1
1
Replying to @nabeelqu
keeping a perfect map of board state from memory is beyond your typical 1800 player too most impressive part imo
2
Replying to @nabeelqu
This just proves Chess is a shallow game of pattern matching. I am very smart about the goalposts.
1
31
Replying to @nabeelqu
So they RLHFed directly on chess to remove those issues. Interesting. I wonder if there’s anything additional beyond that going on.
1
Load more