Human-level play in the game of Diplomacy by combining language models with strategic reasoning

We took R2C2 (22) as our base model – a 2.7B parameter Transformer-based (23) encoder-decoder model pre-trained on text from the Internet using a BART de-noising objective (24). The base pre-trained model was then further trained on WebDiplomacy (§2.1) via standard Maximum Likelihood Estimation. Specifically, with a dataset D={[x(i),y(i)]}$\mathcal{D}=\left.\left.x^{\left(i\right)},y^{\left(i\right)}\right]\right\}$, the model was trained to predict a dialogue message y⁽ⁱ⁾ from player A$\mathcal{A}$ to player B$\mathcal{B}$ at time t, given all of the following represented as text x⁽ⁱ⁾: dialogue history (all messages exchanged between player A and the six other players up to time t); game state and action history (current game state and recent action history); player rating (rating for A corresponding to Elo rating computed from games in WebDiplomacy); game and message metadata (additional info about game settings and the current message, e.g., time since the last message, current turn, etc.). Additionally, the model conditions on intents (a set of proposed actions for players A and B for the current turn and future turns, representing the intent for message y⁽ⁱ⁾, to be described in §2.2.2). Further details on the training data, training procedure, relevant hyperparameters, sampling procedures and other inference-time methods are provided in SM, §D.1.

During play, we use additional modules governing when to speak and to whom, described in SM, §D.4.

Standard language modeling approaches would train our dialogue model only to imitate the messages from our dataset, but not to outperform them. To go beyond imitation learning, we made the dialogue model controllable by generating messages conditioned on a plan specified by the strategic reasoning module (intents), resulting in higher quality messages. More specifically, a message is defined to have intent z if z is the most likely set of actions that the sender and recipient will take – for both the current turn and several future turns – if no further dialogue occurs after the message is received. To establish this control, we developed techniques to automatically annotate every message in the training set with a set of actions corresponding to the message content. During training, the dialogue model learned the distribution pθ[y(i)∣∣x(i),z(i)]$p_{\theta }\left.y^{\left(i\right)}{x}^{\left(i\right)},z^{\left(i\right)}\right]$ where z⁽ⁱ⁾ represents the intent for datapoint [x⁽ⁱ⁾, y⁽ⁱ⁾]; as a result, at inference time z provides a point of control over generation (25). The effect of the intents on the generated dialogue is demonstrated in Fig. 3: observe how conditioning on different planned actions results in different messages. We will describe the training and inference process, which is also illustrated in the pipeline in Fig. 2.

We considered other notions of intent during development, such as controlling messages to focus on specific subsets of actions, third party actions, or to have a particular tone. Richer intents are harder to annotate on human messages, harder to select with the planning module, and create greater risk of taking the language model out of distribution.

When annotating messages in the training data with corresponding intents, our goal was for the proposed actions z⁽ⁱ⁾ to closely reflect the content of a message y⁽ⁱ⁾, such that at training time the model learned to exploit the information in z⁽ⁱ⁾.

Naïvely, we could have used the actual actions played by the sender and recipient at the end of each turn in the span of the intent. However, these actions may not reflect the content of a message if (i) a message is not honest or (ii) subsequent messages change the sender's plans. To resolve (i), we predicted the most likely action according to a dialogue-conditional action prediction model trained on a “truthful” subset of the dataset, in which we predicted that a player's dialogue was not deceptive to others (see SM, §D.2 for details). This is showcased in Fig. 2A – we refer to this model as the intent model. To resolve (ii), we restricted the dialogue history that this intent model saw up to the message in question, which signaled to the model to predict actions as if the dialogue had ended at that point in time. We additionally added messages to the dialogue history suggesting a conclusive agreement between the two parties, Fig. 2B. As a result, we obtained a high degree of correspondence between the action annotated as the intent of a message and the content, achieving a score of 97% on a small test set designed to measure this correspondence (compared to 77% for a simpler baseline) (table S2). Then, the dialogue model could be trained in the manner described in sec:methods:imitation_dialogue and Fig. 2C. See SM, §D.2 for more details.

During play, Cicero uses the strategic reasoning module to select intent actions for the current turn, Fig. 2D, while intent actions for future turns are generated via a human-imitation model.

Cicero conditions its dialogue on the action that it intends to play for the current turn. This choice maximizes Cicero 's honesty and its ability to coordinate, but risks leaking information that the recipient could use to exploit it (e.g., telling them which of their territories Cicero plans to attack) and sometimes led to out-of-distribution intents when the intended action was hostile, as in adversarial situations humans may rarely communicate their intent honestly. We describe approaches for mitigating these risks in §2.4.

Cicero considers the subset of recipient actions with high likelihood under its beliefs about their policy. High likelihood requires that either an action is deemed beneficial for the recipient and/or that they are believed to be likely to play it given the dialogue. Among this restricted set, it selects the recipient action with the highest expected value for itself.

See SM, §D.2.4 for further details.

We compared the performance of our dialogue model to a baseline without intent grounding and one without intent or game state grounding (a “language model”). We report both perplexity on the validation set and dialogue quality rating scores, which were calculated based on expert annotation of messages generated in 126$126$Diplomacy game situations. Experts were asked to label whether a message was (i) consistent with the game state, (ii) consistent with the agent's plan, and (iii) notably high quality, compared to that of an average human. Results are shown in Fig. 4 and more details regarding this evaluation are provided in SM, §D2.3. Our model outperformed the baselines on all metrics. The improvement in validation perplexity demonstrated that the model can use additional grounding information to better predict human messages. Expert annotations showed that the grounding information provided by the intents and game state led to higher quality messages that were highly consistent with the agent's intended action.

piKL is an iterative algorithm that predicts player policies. A complete description of the algorithm can be found in SM, §E.1. piKL treats each turn in Diplomacy as its own subgame in which each player i simultaneously chooses an action a_i resulting in joint action a = (a₁, ..., a_n), and then each player i receives a reward u_i(a) determined by a value function u_i. We discuss the training of this value function in §2.3.3.

piKL assumes player i seeks a policy π_i maximizing the modified utility functionwhere π_–i represents the policies of all players other than i and u_i(π_i, π_–i) is the expected value of π_i given that other players play π_–i. Specifically, let Qt−1i(ai)=ui(ai,πt−i−i)$Q_i^{t-1}\left(a_i\right)=u_i\left(a_i,{\pi }_{-i}^{t-i}\right)$ and letOn each iteration t, piKL updates its prediction of the players' joint policies to bepiKL provably converges to an equilibrium in the modified utility space (26). When the anchor strength λ is set to a large value, piKL predicts that player i's policy will be close to the anchor policy τ_i. When λ is small, piKL predicts that player i's policy will have high expected value and may deviate significantly from τ_i.

A generalization of piKL referred to as Distributional Lambda piKL (DiL-piKL) replaces the single λ parameter in piKL with a probability distribution over λ values (see SM, §E.1.3). On each iteration, each player samples a λ value from their distribution. In practice we found this led to better performance (17).

Since dialogue influences the BC policy (the anchor policy τ_i), piKL provides a mechanism for dialogue to influence policy predictions. As observed in Fig. 5, different possible messages between Cicero and another player may produce different anchor policies, which ultimately gives different final predictions about what that player will do.

Other players of course may be deceptive about their plans. Cicero does not explicitly predict whether a message is deceptive or not, but rather relies on piKL to directly predict the policies of other players based on both the BC policy (which conditions on the message) and on whether deviating from the BC policy would benefit that player.

Because dialogue in Diplomacy occurs privately between pairs of players, Cicero must reason about what information players have access to when making predictions. For example, if Cicero is coordinating an attack with an ally against an adversary, Cicero 's prediction of the adversary's policy must account for the fact that the adversary is not aware of the intended coordination. Cicero accomplished this by predicting via pairwise piKL what every other player's policy will be.

Specifically, during strategic planning, for each player j, Cicero computed an anchor policy for both itself and player j based only on their shared conversation, the board state, and the recent action history. Cicero then ran DiL-piKL for the two players in order to predict player j's policy. On each iteration, Cicero assumed the remaining five players would play according to a policy computed via RL (described in §$\text{§}$2.3.3), conditional on the policies of Cicero and player j. This process gave an independent prediction of each player's policy.

Next, Cicero accounted for the fact that the players' policies were not independent due to their ability to correlate their actions via private dialogue that Cicero did not observe. Cicero accomplished this by constructing an approximate joint policy for all other players via self-normalized importance sampling: we sampled N=1000 joint actions a from the independent piKL policies of the other players and reweighted them by the likelihood ratio of a under the correlated and independent RL policies, respectively.

Finally, Cicero chose the action a_i that best responds to the predicted joint policy π_–i of the other players, while still being as consistent as possible with its dialogue. Specifically, Cicero chose the actionargmaxaiui(ai,π−i)+λlogτi(ai)$\arg {\max }_{a_i}{u}_i\left(a_i,{\pi }_{-i}\right)+\lambda \log {\tau }_i\left(a_i\right)$ where u_i is the RL value function, τ_i(a_i) is the probability of the action under the dialogue-conditional imitation policy and λ = 3 × 10⁻³. Cicero used a smaller λ for regularizing its best response than for its computation of other players' policies; thus, the dialogue more strongly informed Cicero's expectations of how other players would coordinate while still allowing Cicero more leeway to deviate when the action it predicted humans would most likely choose in its situation was suboptimal.

Applying piKL requires a state value function. Self-play provides an avenue for training such a value function, but risks becoming incompatible with human play (16, 17). To address this, we used piKL during self-play to keep the policies human-compatible.

One challenge in doing self-play in Diplomacy is that players may adapt their actions significantly based on dialogue with other players, including coordinating joint actions. Explicitly simulating conversations would be extremely expensive in RL. However, a key insight is that a joint, shared BC policy trained on the joint action distribution of the human data already implicitly captures the effects of dialogue on the action distribution of human players, by modeling that action distribution directly.

We therefore developed Correlated and Shared (CoShar) piKL, which allowed for regularization toward a joint, correlated anchor policy τ shared by all players rather than toward per-player policies. In this way, we relied on the joint anchor policy to capture the correlation between all players' policies. Specifically, CoShar piKL differs from standard piKL in that the probability of joint action a = (a₁, ..., a_n) in policy π^Δt becomesWe found that CoShar piKL retained much of the correlation present in the joint anchor policy τ while also modeling strong human play better than imitation alone.

Our final self-play algorithm operated similarly to AlphaZero (27) and ReBeL (28), by applying planning “in the loop” as the improvement operator for RL. In our case, planning was via an approximated version of CoShar piKL. We generated self-play trajectories where on each turn we computed the CoShar piKL policy using a learned state-value model. We regressed the joint policy model toward that policy and regressed the value model toward the expected values of all players under that policy. We then sampled a joint action from that policy to generate the next state in the trajectory. The anchor policy was fixed throughout training in order to anchor the RL near human play. See SM, §E.4 for details.

Much work has used adversarial or counterfactual examples to improve the robustness of natural language systems (30, 31). Following this approach, we generated many kinds of counterfactual messages that contained mistakes language models are prone to, including heuristically-corrupted text as well as model-generated negatives. We trained a suite of 16 classifiers to discriminate between the ground truth human message and different kinds of counterfactual messages (sometimes varying the random seed or context information available), and used these classifiers in an ensemble to filter messages. This approach risked overly filtering complex messages containing precise plans and accepting bland messages, such as “ok”, which are unlikely to contain mistakes. However, we found that carefully designing our ensemble allowed us to filter most nonsensical messages with minimal impact on message complexity: on a small evaluation set with 362 expert-annotated examples, we found that we could detect 83% of nonsense messages, without significant impact to message diversity as measured via the proxy of message length and the number of references to Diplomacy-specific entities. See SM, §D.3.1 for details.

As noted in sec:methods:controllable_dialogue, controlling dialogue generation via intents has the two-fold benefit of improving the strategic value of a message and reducing discussion of impossible moves or other hallucinations. However, this control is imperfect and the dialogue model may generate messages that contradict the intents it conditions on. To address this, we filtered messages that would reduce the likelihood of the actions in the intent. Evaluating this method on a small test set of 1013 expert-annotated messages, we achieved a recall of 65%, filtering 24% of all messages. See SM, §D.3.2 for details.

Conditioning on intents can lead to “information leakage,” where the agent reveals compromising information about its plan to an adversary (see §2.2.4). To mitigate this, we developed a method to score potential messages based on their estimated value impact. We computed the piKL policies for all agents after each candidate message, and filtered those that led to a lower EV for Cicero playing its intended action. Expert evaluation on a set of 127 dialogue scenarios demonstrated that accepted messages were preferred over filtered messages 62% of the time (p<0.05). See SM, §D.3.3 for details.

We additionally deployed other filters, e.g., to detect toxic language (SM, §D.3.4), and heuristics to curb bad behaviors including repetition and off topic messages (SM, §D.3.5).