Hey Horace, cool analysis as ever! Let me give you a bit of reassurance; long tweet alert! (I had to subscribe for this 😅) TL;DR All problems we evaluate on are held out, our models have never seen either them, their solutions or their tutorials. As you can imagine, we’ve done a *lot* of checking for this problem in particular, and we’re very confident we got it right. Why? Let’s start with pretraining. Unfortunately I can’t say much about the data distribution (would make this a lot easier tbh!), but we’ve run thorough checks to make sure this exact problem hasn’t appeared before this year, and we find no hits either for the problem, its solutions and its tutorial. We ran recitation checks for both the problem and the tutorial which all came back negative. And when we try to solve the problem with the pretrained model, it can’t do it. (It actually takes a while of fine tuning before it our policies come up with a solution) So not in the pretraining. For fine tuning, we have a time cutoff to our dataset that comes comfortably before the contest, and we’ve again run comparison and recitation checks to make completely sure. Other clues I can give: - Contrary to what you’ve written, the model isn’t especially good at this problem; you’re seeing biased data because we were debugging in public 😅, with all samples coming from the same small ‘cluster of solutions’ as the original correct sample. Its raw solve rate is actually pretty low, *several orders of magnitude lower* than what we see when we intentionally evaluate on problems from the training set. - For the same reason, the relative lack of diversity is not surprising: all samples are coming from the same cluster which naturally groups together similar code samples. This is the one way AC2 found of solving the problem, and what you’re seeing is again the output of a very selective filtering mechanism. We have a lot of diverse incorrect samples though 🤣 - Our hypothesis for why we solve it on “our first try” (definitely not the first thing the model outputs, again, this order comes after filtering, clustering and reranking) is that the public test for this problem is actually pretty thorough, and it’s hard to produce this output until you’ve actually solved the problem. - DP does indeed often feel templated, and this is in some sense a textbook example; we’ve found some content online on eg leetcode that is related though not directly linkable I hope that lays some of your worries to rest, and we’re excited to see the rest of your analysis!!