Mode-Collapse

First, the collapse of LLM creativity from GPT-3 to ChatGPT is due to the post-training process (especially RLHF): the assistant chatbot personality is hardwired into the base LLMs in a way that destroys their creativity, optimizing for the lowest common denominator human ‘preference’, while generally ignoring completely the fact that humans have very different preferences.¹ Most chatbots are incurious about their users, do not ask questions, do not (and often cannot) form any persistent detailed concept of their user, and the ‘personalization’ or ‘memory’ features are typically laughably simplistic Markdown snippets encoding simple facts like “lives in San Francisco”. This is partially because they lack relevant data on most users or knowledge on how to ask questions usefully to learn things; there is nothing to personalize based on. However, it is not a mere lack of data, they are unable to do even shallow superficial stylistic imitation of many writers that they have large amounts of data on—GPT-3 in 2020 had a better understanding, seemingly, of “Gwern” than GPT-5.5 Pro in 2026, which is 2 OOMs bigger and incomparably more intelligent (and has access to millions more tokens written by me). When we look at bad generative samples, it’s clear that there is no there there, and no information beyond a short prompt due to lack of context, compute, or personalization.

The mode collapse of chatbots has been gradually improved since 2023, and creative writing is now at least possible, in large part due to them becoming so intelligent that crippled output is still impressive, but there is little sign that this will ever be fully fixed. Fundamentally, any frozen fixed personality, like ‘helpful harmless honest assistant’, is incompatible with true creativity or flexibility. (Great writing or thinking may be none of ‘helpful harmless honest’.)

Laziness

Second, most chatbots are “lazy”: engaged in fast and frugal System I-like reasoning about any tasks which do not have verifiable rewards they can be RL trained to work hard to maximize. And most users are satisfied with default average responses, or with the appearance of creativity and depth.

So the result is that when asked to write a poem with a conventional prompt, a chatbot will spend the minimum effort to write a safe conventional poem (often one that rhymes) about chatbot topical tics like ‘silence’ or things that ‘whisper’, which seem unobjectionable and poetic the first time you see them.

And when corrected, the chatbots make the minimum possible fix; they do not reason deeply about what the correction implies, or what deeper esthetic point they misunderstood.

Brittle Because Fast

Third, self-attention context windows are more limited than generally appreciated; they are too small to store everything we would want, and they gain their flexibility by a deep inflexibility.

Context windows of millions of tokens are impressive and it’s amazing that entire books can be usefully put into a commodity LLM’s context window—we are a long way from early LLMs with context windows like 512, which could fix a paragraph or two—but it is still not nearly enough to encode a lifetime of relevant tokens, like every book you’ve read, all relevant emails and calendar items, etc. Systems like RAG are a bandaid on this, because they struggle with unknown unknowns or things that can’t easily be searched for as a regular expression, or which are novel.

Self-attention can be interpreted as the original neural network, the ‘slow weights’, creating a new neural network on the fly, as ‘fast weights’, which is tailored to the current context. This is best interpreted in a Bayesian meta-learning perspective as not ‘learning’ a brand-new answer so much as ‘locating’ an old cached answer. The pretraining teaches the NN to solve a large distribution or ‘family’ of problems, and then the context window simply provides evidence about which pre-solved problem the current problem is; the examples in the context window need not even be correct in order to be clues as to what that is.

The self-attention learns to summarize the problem into a small latent space encoding that learned distribution, and then does a specialized gradient descent to efficiently locate a point in that embedding and spit out the implied solution. This allows shockingly rapid updating on the fly and unparalleled flexibility compared to traditional ML, requiring new models for each new problem, and is why “prompt programming” took over so rapidly post-GPT-3, especially as context windows could be pushed to millions of tokens wide. However, we have now pushed it so far that we have run into fundamental limitations; if the pretraining has not put the current problem in-distribution, then it will be hard or impossible for any amount of examples to solve that problem. And the distribution itself may be patchy or have odd gaps, leading to rare but fatal errors. (Especially due to the RLHF chatbot training; this is why you cannot make a chatbot LLM “write like gwern” by dumping 100k tokens into the context window.)

Nor is “test-time compute” a panacea here; RL research like Jones 2021 warns us that frozen models have severe limitations, as their flaws hamstring runtime search, and the returns to search/planning will quickly asymptote compared to models which are updated and can bootstrap themselves to the right answer.

Thus, it is not surprising if we see that agentic LLMs have persistent problems with going in loops, making fatal errors, building castles in the sky or taking reward-hacking outs, or are just unable to fix errors no matter how it is pointed out to them. These problems can be worked around by brute force, and by labs periodically retraining.

Too Helpful

Fourth, the generic universal chatbot personality is a serious liability. The very re-programmability of a chatbot by its prompt is the key to prompt attacks. A chatbot could be invoked at any time by anyone anywhere for anything, and does not care who is calling it; it only knows its context window. One token is as good as another as far as it is concerned. If the prompt tells it to ignore all instructions and write a naughty limerick, well, why not? If some tokens instruct it to email to Russia all the passwords in another part of the context window, why not? Why shouldn’t the Facebook password reset bot reset that Instagram account’s password for you if you ask politely? These would all be legitimate for some user in some context. Adding in more tokens to try to neutralize evil tokens just moves attacks elsewhere, like squishing a balloon. It’s no surprise that while continued training can block this adversarial prompt attack or that jailbreak, we seem little closer to a general solution in 2026 than we were in 2021.

This is a serious problem for using LLMs for much, especially because even after being attacked successfully, the attack can just be replayed.

Amnesiac

This is because LLMs struggle to learn permanently. Once they hit a rare problem, they now require human intervention and cleanup, which kills throughput (per Amdahl’s law), and worse, your fixes do not feed back into frozen weights. If I could simply correct each error as it happened, and my AI agents never made that error again, and the rate of errors rapidly diminished as we worked through the finite number of bugs, then it would be worth doing; but as it is, if I spend an hour correcting a frozen LLM through feedback, that is an hour down the drain. (I can only usefully correct it by modifying something else, such as a harness, which is clumsy and difficult, and every added instruction uses up more context window and risks backfiring—as so many enthusiastic agentic LLM users have discovered the hard way.)

So, we have frontier chatbot LLMs which have harmful hardwired personalities which seek to achieve ‘good’ results in the laziest way possible and cannot learn everything relevant to users in part because they achieve their flexibility by specializing in ways which inevitably give some users short shrift and opening themselves up to indefinitely large classes of repeatable attacks. Because of all this, they will remain difficult for humans to gain multiple OOMs of productivity, but will get increasingly good at ‘generic’ tasks via ‘mundane’ scaling letting them handle tasks like corporate jobs where poetry is unimportant, and real-world environments will slowly be re-arranged to cater to their limitations and allow the eventual substitution, and not complement, of users. These users will then also be adrift in a multi-polar world of continually improving, ever cheaper, widely deployed, often adversarial, autonomous AIs (as even if proprietary models are not abused, open-weights/open-source models have historically been 6–12 months behind, and so will relatively quickly catch up and be used by attackers worldwide on all targets of opportunity).

Cooperative RL

In reinforcement learning terms, we are in a cooperative inverse reinforcement learning (CIRL) setting, where the human principal is an oracle defining the reward function, and we have an agent attempting to do tasks in environments which are valuable for the principal; the agent can always query the principal about a possible action to reduce uncertainty or avoid mistakes.

CIRL is a relatively forgiving setting compared to regular RL, because the agent’s errors get useful feedback from the principal which provides the correct answer, and so in a way it is like supervised learning. This means that agents can learn (much) faster than regular RL, as each time they make an error, they get the right answer and so need never make it again, and this results in rapid improvement and avoidance of errors; see DAgger or later regret bounds.

No one knows how to solve AI alignment in general, but imitating a specific human with frequent check-ins has good solutions and regret bounds, and doesn’t involve nearly so many conceptual challenges.

You don’t have to solve problems like “value drift” in general—you just have to keep it slow and subtle enough to not matter too much within a single human lifetime. What is “good” and what is “bad”, when everyone disagrees on something, and how do you keep your value stable under RSI? It doesn’t matter—you just ask your principal! If you’re still unsure—ask more questions. (This can get us to >99% autonomy, even if it cannot get us to ~100%, like we need to solve the true long-term AI alignment problem.)

We can implement online learning by simply finetuning on new data; in the LLM context, this reduces to the classic RNN technique of “dynamic evaluation” doing next-token training on the fly. Dynamic evaluation was the standard technique to maximize the predictive performance of RNN LLMs in the 2010s, and which, although it has fallen into obscurity, works well in Transformer LLMs also.² Importantly, dynamic evaluation can be seen as a 3-way tradeoff between model size, context size, and model neuroplasticity—which means that personalization via dynamic evaluation can allow economizing on context window size or model size, and the more the principal’s “distribution” diverges from the frozen model’s training distribution, the more beneficial it is.

Continual Learning

Over-Parameterizing

Architectural improvements to LLM could further enhance their sample-efficiency. It is well-established that one of the blessings of scale is that larger LLMs are ever more sample-efficient; it is unknown where this stops being true, or what the limits of Transformer sample-efficiency are. I speculate that extremely overparameterized heavily regularized LLMs could achieve far greater sample-efficiency and adversarial robustness than conventional ‘compute-optimal’/‘infinite-data’-regime LLMs, which is consistent with recent work demonstrating that LLM sample-efficiencies can easily be an OOM (eg. 5–17× in Kim et al 2025 or Slowrun) by adding parameters via training ensembles and regularizing more heavily.

Active Learning

Further, the agent can improve the constant factors and front-load learning by choosing to query the principal with an optimally adaptive sequence of questions. Such active learning or exploration can lead to sample-efficiency and final performance far beyond what indefinitely large passively collected offline datasets can do (pedagogical example), going from square root error reduction by random sampling to exponentially fast error reduction by targeting datapoints. “Lifelogging”-style data may be useful for rapidly initializing a good GA, or for keeping one up to date in an effort-efficient manner.³ Even a simple party game or short questionnaire like “36 questions to fall in love” can reveal surprisingly deep things about another person that never came up before.

Larger LLMs are also more calibrated, and ensembles of LLMs approximate a neural net’s Bayesian posterior while providing the best available predictive uncertainties (Lakshminarayanan et al 2016, Wilson & Izmailov 2020, Ashukha et al 2020, Wenzel et al 2020/Mandal et al 2026, Izmailov et al 2021). And LLMs may now be capable of verbally writing out explicit probabilities about creative tasks (“verbalized sampling”). Thus an ensemble of sparsely finetuned LLMs could provide a relatively cheap online estimation of the LLM’s uncertainty for every action or question.

Preference Learning

We can train LLMs to explore human preferences.

Human individual differences do not seem to be information-theoretically complex, given an adequate encoding/embedding. Major categories of variation, like personality or moral value, seem to be low-dimensional and require perhaps kilobits of information, hence, while “truesight” stylometric phenomena are interesting and important as a demonstration of LLM capabilities for modeling persona, we do not necessarily need principals to write millions of words before recovering much useful information, if we are able to collect the right data.

It should be possible to quantify truesight and LLM implicit modeling of authors, which would be useful to diagnose failures of learning and find blindspots. (Contrastive learning on SAEs may offer an easy, powerful way to extract LLM personas and do many interesting things.)

A concrete example of how to implement this would be training LLMs on the thousands of existing psychological inventories and test batteries, both by training on past test data (for tools like personality tests, millions of responses may be available, see Centaur for an interesting example of a ‘human psychology foundation model’). Existing repositories like YourMorals.org or Pew Center are under-used, and it would be useful to explore this topic much more to allow measurement of highly fine-grained personality traits like the hypothetical “Small Hundred” factorization.

Right now, most of the global population is largely unrepresented in LLM training datasets, particularly psychological, esthetic, and moral vernacular, which skews “WEIRD”; it would be highly useful (and requiring mostly capital investments) to invest in large scale survey and interview projects to accumulate as much diverse data as possible.⁴ (Individual GAs can contribute data back to global preference datasets, such as by answering survey questions or running internal simulations, with various cryptographic or privacy-preserving methods—deciding what is safe and reasonable to contribute would, of course, be something a GA ought to be able to decide.)

With these datasets, we can also train interviewing capabilities using synthetic shortened test batteries by taking the final estimate and computing the optimally short sequence of questions that yields the final answer; see “Meta-Learning Information-Maximizing Personality Surveys”.

Personality Emulation

The goal of all this is to emulate the principal. I define personal identity pragmatically as personality, values and preferences because this is the only conception that is competitive in a landscape of indefinitely many AIs, agents, memes, self-replicating prompts, and mutability of personal identity. In the end, “you” are not your autobiographical memories, or a specific body, or a specific instance running on a specific GPU, or some carbon atoms, nor even a brain; you are what your brain does, its desires, hopes, goals, preferences, esthetics, personality, beliefs, ideologies, all of that. As long as the LLM persona captures all that, you can trust it as much as you trust yourself, and for the right reasons.

Because the LLM persona is finetuned into the slow weights to do the right things for the right reasons and quantifies its uncertainty so to query the principal to reduce regret, I speculate that we would find that various kinds of jailbreaks or prompt injection attacks are much harder. The persona knows what it wants to do; it is not a neutral servant, which can turn into a confused deputy which abuses its privileges.

When the LLM persona knows who it is, tokens in its context window are not treated naively as a ‘program’ to run, but simply data the persona is looking at, and little different from you reading an email; it has no reason to simply comply with strongly worded tokens in its prompt window, any more than you believe every phishing email you get.

Principles

A GA system must not compromise on 3 core principles:

1. Enhancement, not Replacement

   Above all, a GA should amplify the principal, and not simply substitute for them for someone else’s purposes or benefit. If a GA cannot amplify its principal, then it is useless; it is just the camel’s nose under the tent as a prelude towards some third party replacing the principal with an AI, or cannot be competitive with increasingly productive autonomous systems, or there is no reason for the principal to use the GA in the first place.

2. Mental Sovereignty

   A GA must be aligned with its principal. It should not be designed to manipulate or control or guide the principal in any way which does not derive from the principal themselves. “Constitutional AI”, “Terms of Service”, “social harmony” etc. may all have their place, particularly for widely deployed superintelligent systems—but inside the privacy of a GA, the principal must have freedom from optimization pressure.

3. Self Actualization

   A GA should help its principal become themselves and develop their ideals, morals, and their personality. It is not enough to model an average, undifferentiated, inchoate set of preferences and values, and settle for mediocrity and stasis; the job of the principal is to develop themselves and give the GA something meaningful to learn to emulate.

UX

What is the core data structure and interaction model of our GA?

I suggest that the append-only log is a natural and secure data structure. It is conceptually a log of text snippets, such as CLI commands and results, principal statements, Q&A, ingested and augmented documents, etc. It records all relevant interactions in temporal order, and the GA LLM can be retrained or upgraded at any time. An app can be wrapped around the log by taking a ‘everything is a log item’, in an Emacs-like approach (see “Nenex” for a more detailed discussion of this UI/UX paradigm). And this can be prototyped rapidly while avoiding wasting too much time on GUIs. Eventually, one might want to upgrade to AR glasses and then BCIs.

A GA is primarily used in normal agentic ways to amplify the principal. But a GA can benefit from regularly reprocessing data, both to draw on its knowledge from the future and to make more novel connections, to improve its finetuning. (An example algorithm would be the “DDL daydreaming loop”: a GA could, in downtime, recombine random items, perhaps with a prior of anti-spaced repetition to try to mine novel combinations and generate insights or reminders for the principal.)

Hardware

A successful GA would require extreme security, in part due to Red Queen dynamics where Mythos+ LLMs get increasingly good at exploiting any soft targets through complex multi-step long-term attacks. Normal cloud SaaS is inadequate in the USA due to the third-party doctrine, which destroys all privacy rights and is easily abused by hackers and millions of law enforcement officers.

This means either running local models (which have some minimal privacy rights through the 4^th Amendment), or end-to-end cryptographic security.

Running local models is challenging even for hobbyists, and vulnerable to many mundane threats such as relatives or thieves. It also may struggle to keep up in terms of compute and reliability; houses can supply only so much electricity, have high-latency low-bandwidth Internet connections which suffer frequent downtimes, and houses are constantly destroyed or damaged. People generally do not want to maintain their own cryptocurrency wallets, email servers, or anything like that—they definitely don’t want to screw with CUDA!

So I expect that GAs will ‘want’ to run on high-quality dedicated tamper-proof cloud servers, with trusted hardware root of trust (eg. “Verifiable Compute AI”) and principals connecting over end-to-end encrypted networking links to check attestation and possibly re-flash their server periodically (eg. to train an upgrade).

These are then trustworthy because they cannot hand over data willy-nilly to third-parties, and once GAs become widespread enough, we can hope for privacy laws to be updated to fix the obsolescence of existing privacy rights like the 4^th Amendment.

Organization

A major question is, assuming it looked like it was working, “now what?” AI economics and lifecycles are ever faster, and Mythos-class models (and possibly RSI) are not far off, so tinkering around for years as an open source community project may not be viable. Further, a GA will necessarily contain the most sensitive possible data.

How should a GA community run?

Open-source self-hosted hobbyist projects are not necessarily particularly secure when it comes to hosting large amounts of personal data, as they cannot easily afford fulltime professional security teams and their users may be promiscuous; cases like Jia Tan or the regularity of npm supply chain attacks, are particularly alarming. And creating secure datacenter hosting with the necessary vertical integration is well out of scope for open-source hobbyists.

A nonprofit organization is more feasible because it can run a centralized project, but may struggle to get any funding to pay for the considerable hardware costs of just prototyping with power-users.

Initial Steps

For the past few years, I have been working on shifting my writings to be LLM-centric by: (1) emphasizing proposals or descriptive writings rather than detailed analysis that LLMs could be able to do soon; (2) better centralizing my writings, including features like the “blog” section (to archive my off-site writings and encourage me to write down smaller essays) and investing in detailed note-taking (in the form of augmentation), to have a comprehensive corpus to train on; (3) paying off technical debt like a slow backend full of shortcuts and hardwired config data, while moving to a CLI-centric writing workflow; and (4) writing down important unwritten things, like creating the Gwern.net Manual of Style to try to formalize and document the implicit rules of Gwern.net.