Meta-Learning Information-Maximizing Personality Surveys
Proposal to teach LLM efficient interviewing by using psychometric tests to construct optimal question-and-answer sequences to train on. Would enable better personalization and imitation, and eventually, uploading.
LLMs are good at inferring preference variables from text because this is so useful for modeling text written by specific authors. They likely represent things such as personality, identity, esthetics, morality etc as a latent, low-dimensional, embedding or manifold (like many other things, such as truthfulness or intelligence or nationality or geographic location etc).
This embedding could be useful for research purposes like in silico psychology, customization/personalization (especially for art or psychotherapy), understanding & preservation of human value, uploading which preserves personal identity etc, and we would like as high-quality ‘LLM personalities’ as possible.
Efficient Personality Testing
But we would like to get personality inference from less than ‘tens of millions of words per person’ (which is usually infeasible), and we would also like the LLM to target questions to reduce its uncertainty as much as possible, to minimize cold-starts.
Can we simply ask users who they are? No, because most users will be unknown—authors like myself are often well-known to LLMs, because we write so much online, but no one else.
Can we ask users to take the standard personality tests and just input those as context? Only partially: that would work, but it would be inefficient and limited: personality tests are designed using standard psychometric principles to avoid wasting questions, that is true, but only on the assumption of a blank slate where nothing is known about the current respondent, and because tests must be static & non-adaptive for ease of administration as pencil-and-paper tests. There is no way at present to ‘boost’ a personality inventory with the latent LLM knowledge, and skip to just the few useful questions (much less for the LLM to synthesize an optimally-informative question). They are also confined to a relatively small set of curated items: we might want to go into great detail, cumulatively, over the years a user is using a system, far beyond the few items on a validated published test. (Such tests often drew on databanks of tens of thousands of candidate items—but we don’t have access to those, and most of those would be bad or redundant, which is why they weren’t used in the first place.) So, just using existing tests would be both wasteful and ultimately plateau at a low level of detail.
Can we simply ask a LLM to ask questions? Probably not. LLM meta-cognition is still limited, and there is not much precedent in the training datasets for “LLM optimally interviewing users for psychological profiling”. Simply prompting a LLM to ask questions may not be optimal in any sense: it might instead resort to imitating the questions some generic stereotypical psychoanalyst would ask, but there is no reason to expect that to be optimal in any sense (given how poorly psychotherapy works on average—while the best therapists couldn’t be imitated due to lack of data).
This is a hard exploration problem, where it isn’t clear what success would even mean.
VoI Learning
Here is one possible approach, which treats it in Bayesian meta-learning terms. In the Bayesian meta-reinforcement learning perspective, an agent (the LLM) is confronted with an endless series of problems, each varying in some hidden way, which it attempts to maximize its performance (predicting the next token) in as quickly as possible, by inferring those hidden variables. We want to ‘learn to learn’ a general strategy which adapts on the fly to infer the state of some latent, hidden, variable, and act Bayes-optimally.
From this perspective, a sequence of Q&A can be considered a sequence of actions (questions) and observed states (answers). At the end of a long sequence, we have ‘located’ the respondent in ‘personality space’ and can predict their responses to all future personality-related questions. (This is then a sufficient statistic for any downstream tasks which benefit from knowledge of personality, and we can stop there.) So the goal is to narrow down to the target as quickly as possible.
We can train a LLM to infer this in general by directly training on this scenario: training on a large ‘raw’ psychometric dataset (such as YourMorals.org) with rater/respondent IDs, aggregating everything into (Q, A, ID) triplets. (For this, we want a dataset which links respondents across as many tests as possible. It would be much more useful to have a few people who take every test, than have thousands take only 1 test each.) By learning in the usual way to predict each respondent’s responses to arbitrary questions, this trains a personality embedding, simply because that helps represent the generative process (ie. the similar-yet-different humans).
This doesn’t get us the Bayes-optimality of asking questions, however, because it’s just a large haphazardly-sampled dataset. Nothing in it represents ‘an optimally informative Q&A sequence’. (If we had that, we wouldn’t be asking this question! We’d just train on those Q&A sequences.)
However, after we have trained our base LLM on such a dataset (cf. Centaur), of a big pile of Q/A/ID triplets, we can quantify progress towards the desired final inference: we look at the predicted log-likelihood of the actual rater ID!
The more informative about the rater ID (ie. a specific personality) a Q is, the higher the log-likelihood of the true ID will be after the A is seen. If some question has a unique answer, then the rater ID will be immediately inferred nearly perfectly. If it instead takes a particular set of 50 questions, then the log-likelihood will increase gradually over all sequences including those 50 questions.
So, we now have a way to construct synthetic optimal Q&A sequences. We can ‘plan’ over the corpus of questions to find sequences which increase the true ID log-likelihood as fast as possible. We can apply any sort of planning or search procedure we want, but a simple way would be just to bruteforce it: there are not that many personality questions in the tests (perhaps a few thousand if we pool several), so we can just, for each rater ID, loop through all the questions they’ve answered, and pick the one which maximizes the gain in their ID; then loop through the remainder, etc. We can construct many of these for each user, too, picking different subsets, and terminating when the ID becomes too predictable.
Once we have synthesized an adequate set of nearly-optimal Q&A transcripts, we can simply train our LLM again on them as the highest-quality ‘interviews’. Now we can deploy this LLM to talk to users, and it will (in theory) have learned all about the unique properties of each question, and users, and adapt on the fly to their answers. (The resulting rating ID prediction will be meaningless, but we don’t care about that, as we are simply using the interview to feed into other models.)
We can of course iterate this: each new Q&A interview is assigned a new rater ID, and we can retrain to update the LLM, and try to construct shorter interview sequences to help bootstrap interviewing skills. (Overall, this can be seen as a kind of expert iteration where we use the data as a non-parametric model to plan over1, and then iterate the expert with each node incurring a cost, the reward being the likelihood of the true rater ID.)
So, this in theory lets us create an extremely flexible, sample-efficient LLM interviewer, using only straightforward LLM training & search methods and reformatting data, offline, with existing datasets, which can be improved online. The main limitation is that our final LLM is still asking only pre-existing Qs.
We might hope that if we don’t incur catastrophic forgetting, a sufficiently intelligent LLM might be able to generalize from this narrow Q&A training process to be able to ask questions in general, questions which appear in no personality test, but which will be informative anyway, and spontaneously breaks out of the pre-existing set to ask ever deeper novel questions. That would be nice, but it probably wouldn’t happen by default; hope is not a plan.
Freeform Interviews
So, how do we remove this limitation? Our limitation is due to the fact that we construct synthetic interviews from real data by real people. We cannot reorganize them in any way which would handle this ‘open interview’ demand, because there are no tests which ask a bunch of people about “how did it feel to grow up on 10 Privet Drive in 199530ya?”, even in principle. But if we have learned to model real data from real people so well, we can learn to ask such questions by constructing synthetic interviews from ‘real’ data by synthetic people!
We learn to ask free-form interviews by repeating the previous process, but simulating the answers to arbitrary questions instead.
How do we construct a synthetic person? We simply sample a possible interview from the LLM, ignore the implied rater ID, assign a novel ID (as if they were a real new user), retrain, and now we can sample any questions we please from the novel ID. So, we create a virtual population of, say, 10,000 simulacra or persona, who are available to interview at any time in any way.
Now we can construct our own ‘raw’ dataset, which does include arbitrary novel questions. We can make our LLM generate a very large set of questions, or we can get a list of questions from any other source. (For example, we could download a large Internet-sourced dataset like The Pile, and simply parse it for anything which might be a question, ending in ‘?’.) Then we ask each persona to answer each question, and now we have a new synthetic ‘raw’ dataset.
We then simply do what we did with the original raw dataset: construct new optimal interviews, and retrain. If we have enough candidate questions, we can avoid any repetition—every question will be novel and never-before-seen, which avoids the ‘closed world’ failure mode of the first phase. When every question is novel, the LLM has no choice but to learn to never repeat itself and always come up with novel informative questions.
Now we have the desired product: a LLM interviewer which exploits all available knowledge about a user, such as answers from static personality tests, but also knows how to ask customized informative questions which still learn as fast as possible about the current user.
We can imagine further extensions of this, like dropping questions into regular conversations, or prompting LLMs to ask questions before executing a task which might depend on personal preferences & taste.
Other Domains
We can also apply this to any domain we want to ‘personalize’, where we have a relatively few ‘questions’ and ‘answers’, and we can induce an embedding in which we want to quickly interactively locate points. Most obviously, generative media with esthetics: images, audio, video, text, etc. This would be a potential way to, say, implement personalization for a Midjourney-like image generator where artists will have highly distinct preferences, and there are large preference datasets like yes/no (thumbs up/down) or forced-choice comparisons. (The pre-existing images are the ‘questions’, the yes/no or win/lose are the ‘answers’, and we predict the rater ID as before; simply with image models instead of text LLMs.)
Morality, religion, and political preferences would also be valuable domains to apply this technique in; and since all of these are related, ideally we would have a single interview model which handles them all jointly. (We can use a multimodal vision-language model to do this: media can be handled by encoding into tokens with associated URLs/IDs, and then the LLM interface can natively interpret outputs, like Markdown, and the LLM can simply ask questions like Which do you think is prettier,  or ?.)
Preferences, Not Capabilities
We probably cannot apply this trick of constructing synthetic transcripts to more general capabilities, like science and math. I cannot quite see how you might apply this to, say, theorem-proving or chemical retrosynthesis or protein engineering—what would be the embedding, exactly, or equivalent to ‘rater ID’? (Or at least, any successful generalization would be so broad that it would simply be a general meta-RL solution, already handled by methods such as domain randomization inducing a solution of the POMDP when trained at adequate scale.)
medical interviewing for clinical case studies and diagnoses? This might be a great use-case for the o1 RL finetuning feature: there are lots of clinical case summaries + diagnoses, but few available transcribed dialogues; so you can synthesize optimal dialogues to yield the known summary+diagnosis, and train on those to learn to interview. https://gwern.net/doc/ai/nn/transformer/gpt/4/nonfiction/2025-johri.pdf
cf. /free-play
This would be highly questionable in many environments—you can’t simply shuffle states around, order matters!—but is reasonable in this context, because psychology tests are usually constructed to minimize ordering effects and to be valid when shuffled. A Big Five personality test shouldn’t care which order you answer questions in…↩︎