[see also “Assessing Human Error Against a Benchmark of Perfection”, et al 2016] How does artificial intelligence (AI) improve human decision-making? Answering this question is challenging because it is difficult to assess the quality of each decision and to disentangle AI’s influence on decisions. We study professional Go games, which provide an unique opportunity to overcome such challenges.
In 2016 an AI-powered Go program (APG) unexpectedly beat the best human player, surpassing the best human knowledge and skills accumulated over thousands of years. To investigate the impact of APGs, we compare human moves to AI’s superior solutions, before and after the initial public release of an APG [Leela Zero, KataGo, and NHN’s Handol]. Our analysis of 750,990 moves in 25,033 games by 1,242 professional players reveals that:
APGs noticeably improved the quality of the players’ moves as measured by the changes in winning probability with each move.
We also show that the key mechanisms are reductions in the number of human errors and in the magnitude of the most critical mistake during the game. Interestingly, the improvement is most prominent in the early stage of a game when uncertainty is higher. Further, young players—who are more open to and better able to use APG—benefit more than senior players, suggesting generational inequality in AI adoption and usage.
[Keywords: artificial intelligence (AI), technology adoption, decision-making, human capital, professional Go players, AI adoption inequality]
…The historic Go match (AlphaGo vs. Sedol Lee) was held in March 2016; in this game, AI beat the best human professional player for the first time and by a large margin. Shortly after this event, the first open APG, Leela, became available to players in February 2017. Our quantitative and qualitative investigation indicates that professional Go players have used APGs heavily in their training since its release.
The great advantage of this context is that it allows us to observe every single decision of professional Go players before and after the public release of APGs; a game’s entire move history is well archived and maintained for all major games. Furthermore, using the APG’s best solution as a benchmark, we can calculate the probability of winning for every move (ie. 750,990 decisions) by 1,242 professional Go players in 25,033 major games held 2015–42019; note that this can be done even for the games played before APG’s release. We then compare the move-level probability of winning to that of APG’s best solution.
The results show that the quality of moves by professional Go players improved substantially following the release of APG. Before the release, the winning probability of each move by professional Go players averaged 2.47 percentage points lower than the moves of APG. This gap decreased by about 0.756 percentage points (or 30.5%) after the release of APG. Additional analyses indicate that the improvement in move quality eventually leads to the final win of the game. Interestingly, this effect is most prominent in the early stage of a game where higher uncertainty is exhibited and there is more opportunity for players to learn from AI. Furthermore, quality improvement is more prominent among young players who are open to and capable of using APGs; this has important implications for digital literacy and inequality in AI usage.
[Example of an absolute human error rate: from the AI’s perspective, each move a human Go pro makes costs them ~1.2% chance of winning!]
We also explore the mechanisms through which professional players achieve a higher probability of winning. Our mediation analysis reveals that improvements in the quality of moves are driven mainly by reducing the number of errors (moves where the winning probability drops by 10 or more percentage points compared to the immediately preceding move by a focal player) and by reducing the magnitude of the most critical mistake (the biggest drop in winning probability during the game). Specifically, the number of errors per game decreased by 0.15–0.50 and the magnitude of the most critical mistake decreased by 4–7 percentage points.
…3.3.1. Go Games and Players: We collect data on professional Go games held from 2015–42019 from the Go4Go database, which has been widely used in studies of Go (eg. et al 2018, 2003, et al 2018). The data contains detailed information on the game, its players, Komi (the number of bonus points given to the second mover), the sequence of all moves, and the game outcome. From Go Ratings we gather additional data on the ages, nationalities (eg. China, Japan, South Korea, Taiwan, and others), genders, and annual rankings of professional players. We multiplied negative one by the ranking and divide it by 1,000 to ease the interpretation of the result; the higher the value, the better the player. To control for the difference in players’ capabilities for each game, we create a variable, Rank difference, as the difference between the raw rankings of 2 players; we divide this difference by 1,000 such that a positive value indicates that the focal player’s ranking is lower than the opponent’s ranking.
…Using 2–8 Nvidia TitanX GPUs running in parallel, the computational analysis of games took about 3 months.
…This analysis is motivated by the norm that, after Go games, players spend substantial time and effort analyzing and evaluating each move—especially if the move was an error or a mistake. In an interview with news media, Jin-seo Shin (who was ranked first in the world in 2020) stated:
Before APG, players and their peers replayed the game and discussed which move was an error and which was a critical mistake. After the public release of APG, this replay and discussion by players became almost meaningless.
APG teaches us by showing the accurate winning probability with each move. If the winning probability drops 60% → 40% after a move, that is an error. If it drops 80% → 20%, that is a critical mistake…I have to admit that the APG-based training provides limitless help in developing my Go skills (2021).