Sample Inefficiency

Why do NNs require Chinchilla-style scaling of data and compute, when humans appear to learn from multiple orders of magnitude less data, and it is increasingly plausible (given various estimates of human-brain equivalents) that they learn from less total compute? Why, as so many connectionist pioneers like Alan Turing expected, do we not train AI like children, with a curriculum and clear developmental stages?

There are many answers offered, none satisfactory. (And what should we make of theoretical results like Rosenfeld 2021’s “Nyquist learners”?)

• Multi-modality: while useful, multi-modality has failed to yield any major change of scaling law exponents; unimodal models work shockingly well, and language models turn out to already encode a large amount of visual knowledge and can easily be plugged into vision models (eg. Flamingo, Tsimpoukelli et al 2021).

• Human sensory input is actually large: Another common explanation is to deny that humans learn from less data, and argue from raw sensory bandwidth: if vision+sound+touch is such-and-such bits per second and you accumulate over an adult’s lifetime, it can look much more comparable to the trillions of tokens we train an LLM on.

  This is unconvincing because the raw sensory bitrate is meaningless: the input is extremely redundant & predictable for the most part. (Imagine sitting in a room staring at a computer screen.) Attempts at quantifying the information content of images, video, or sound, usually indicate that they boil down to the equivalent of a few hundred or thousand tokens and those modalities are easily learned by small models (eg. iGPT/DALL·E 1). The asymmetry is particularly striking in text-to-image generative models, where the text encoder (usually an afterthought) is often far bigger than the image generator itself.

  And on the human side, disabled people are not much less intelligent than normal humans: deaf/blind people are much worse at language tasks, but their fluid intelligence often remains normal. If the sensory bandwidth were so critical, this would be impossible.

• Active Learning: human children, unlike models confined to offline imitation learning, can choose what to learn about by exploring their environment or asking questions. In theory, active learning & optimal exploration can be far more sample-efficient than indiscriminate training (exponential rather than power law, at a minimum), and this could account for the entire gap.

  However, if we look at the things children actually choose, the data in question doesn’t appear all that amazing. Further, in stark violation of any notion of optimal Bayesian exploration, children often choose to learn on the same data point—eg. watching the same YouTube video hundreds of times.¹ Or if we watch them ‘explore’ a game or computer, it looks like it is by acting largely at random, and an adult would learn far faster by more carefully thought-out exploration.²

• Embodiment: a closely-related topic is the idea of “embodied cognition”, which used to be quite popular as an explanation for the weaknesses of AI—AI models simply lacked commonsense & generalization for lack of a body and an appropriate environment.

  But thus far, ‘embodiment’ like training on robotics data (eg. Gato) has exhibited zero transfer to other tasks, never mind massive scaling law gains, and ironically, it is, in fact, embodied tasks like robotics models which have been greatly benefiting from non-embodied pretrained models (including LLMs!).

• Architecture Magic: Perhaps in some way, Homo sapiens-style biological neurons are just some near-perfect architecture, and this explains most of the gap; someday we will understand how all artificial neurons are severely hobbled by mistakes that will seem as tragically obvious in hindsight as earlier mistakes like not using backpropagation or using sigmoid activation functions now seem to us, but they remain a mystery for now.

  This view was highly plausible until recently, but has been running into many problems.

  For starters, we simply have not found any architecture magic. The most obvious place to find magic would be the learning rule for biological NNs, whatever they use in place of backpropagation… But while people have proposed many biologically-plausible learning rules since Hebb proposed the first learning rule in 1949_77ya, which respect the requirements like locality, in every case, those learning rules perform worse than, or at best similar to, backprop! To quote Geoff Hinton:

  So maybe it’s [GPT-4] actually got a much better learning algorithm than us.

  And if biological NNs are not so good but there is something special about humans which does make them much better, then why do Homo sapiens not appear to have any major neuroscientific breakthroughs compared to our primate relatives? Why are we so genetically similar, and we have failed in the search for major novel mutations that create humans, and human brains seem increasingly like nothing but “a scaled-up primate brain”? If human (or primate) brains are so uniquely efficiently tuned by evolution, why are bird brains so much more efficient in size & thermodynamics than primate brains, with clear genetic changes, and better scaling to the point where small bird brains like ravens or parrots or vultures exhibit eerie levels of intelligence & behavioral complexity almost on par with dolphins, chimpanzees—or humans? Why does human intelligence exhibit so many bizarre drawbacks or anomalies if it is so optimized, like childhood amnesia or lurching through developmental phases over decades?

  If the story of NNs is one of us gradually recapitulating evolution’s perfect neural networks, why does neuroscience provide so little useful inspiration (as is emphasized constantly by neuroscientists & AI researchers, even the “neuroscientific” inspiration for things like self-attention is a very loose inspiration)? Why don’t all major improvements come from success in reverse-engineering the human brain to ever greater biologically-realistic detail? Why is the rapid progress in neuroscience, like scanning entire connectomes, completely irrelevant to cutting-edge NN models? Why do all the scaling laws for CNNs & Transformers look so similar in the exponent, and durable improvement so difficult, to the point where 7 years after Vaswani et al 2017, it is still a relevant baseline? And why, if we have such fundamentally inferior architectures, are the scaling laws so smooth & reliable, instead of breaking frequently or predicted to asymptote at levels far below human?³

  • Conversely, why do NNs provide little insight into biological brains? In the other direction, neuroscience & individual psychology has hardly benefited from DL; DL has provided tools, and has provided good predictive models of brains, but that is about it. One could open an issue of a psychometrics or neuroscience journal, and note that, over a decade into the deep learning revolution, if it had never happened, that issue would look about the same and would be completely intelligible to a researcher from 2010_16ya.

    It is impressive that we can now turn fMRI scans into crude visualizations of what a person is looking at, or we can use LLMs to generate possible questions for surveys, but DL has provided essentially no major insights into such fundamental questions as “what is fluid intelligence? why is the g factor so general? how do neuroanatomic traits like neuron count or network properties cause intelligence or other cognitive abilities?”

    Isn’t this astounding? We can now create models of enormous generality like Gato or GPT-4o from scratch to match or exceed humans without the hand-engineering of GOFAI, which seem so eerily human-like in many ways, and which recapitulate so many aspects of human cognition down to heuristics & biases, and our ability to create these artificial intelligences tells us… nothing important about the human brain? Really?

    So what, all the DL is just a dead-end and coincidentally capable of all that, and sometime in the future we’ll discover the real route to brain-like intelligence…?

Sample Efficiency

Why do NNs require so much data to pretrain, when NNs are as sample-efficient in narrow comparisons?

Despite the huge amount of indiscriminate data used in NN pretraining, we are puzzled because if we examine how well models do in apples-to-apples comparisons, NNs appear to be as roughly as good as humans or biological neural networks at learning from small data.

The simplest answer to the exorbitant data-scaling would be that NNs do a bad job of learning each datapoint—but that doesn’t seem to be true when we compare things like in-context learning (even GPT-3), transfer learning smoothly scaling, child-sized datasets (eg. SAYCam, BabyLM, TinyStories), compare total AlphaZero Go/chess games to total games played by human pros over history, or disable human priors to compare learning speed (cf. human vs Transformer meta-learning), and remarkable experiments like learning an unknown language from a book or raising chickens in virtual reality & comparing their visual capabilities with a Transformer (or vice-versa, have profoundly mistaken beliefs about the world when raised passively in a bubble).

How can we reconcile this apparent contradiction?

Smallness

Why are NNs so intelligent, and yet so small?

A NN like GPT-3 is ~0.1t parameters (which are absurdly simplified caricatures of biological neurons), while the human brain is extremely loosely estimated at 100t ‘parameters’, and many neuroscience results are taken as implying that each of those ‘parameters’ translates to thousands of equivalent parameters, at a minimum; an LLM is further disadvantaged by lack of recurrency & online learning, and is generally unable to engage in adaptive computation or ‘ponder’, the way that a brain can continually learn and ruminate. And yet, despite all these severe disadvantages, LLMs seem too good. It does not feel like a GPT-3 knows or does less than 1⧸100,000^th of what the human brain does; but if we argue that a more plausible number like 1⧸1,000^th is true, that appears to commit us to the position that human brains are highly wasteful of parameter-equivalents (despite the touted superiority of biological brain architecture & evolutionary pressure for efficiency). And if a GPT-3 really does know that little, then what are those >99,999 things missing from a GPT-3—which each require an entire GPT-3 to handle?

This observation becomes even more extreme as a NN like GPT-3 is well-known to be highly overparameterized and can be shrunk down by orders of magnitude, and it is also known that small NNs can be trained for a long time to achieve performance of large models (albeit compute-inefficiently). We appear to be between Scylla & Charybdis.

Alternate version: why are human brains so overparameterized? Many humans get by with much smaller brains than others; there is a real and causal correlation with brain volume, but even allowing for how crude a proxy that is for the underlying neural nets’ capacity & speed, the effect size seems surprisingly small. Biological brains can survive shocking amounts of physical damage or brain loss, as long as it occurs early in development (eg. hemispherectomy; see also hydrocephalus), and it’s surprisingly hard to find brain lesions which damage IQ instead of specific cognitive abilities. Cross-species, there is clear allometric scaling and the rankings make sense, but the slope is shallow: chimpanzees with less than a third of the brain often seem competitive with some humans in terms of intelligence and ingenuity. And remarkable instances like Portia spiders or the ability of the dragonfly to home in on prey using barely 16 neurons further raise the question of why so many neurons are necessary (as does, indeed, power law scaling in general).

Superhuman Prediction

In particular, why do even tiny LLMs like GPT-2 appear to already be superhuman at next-token prediction in terms of perplexity/bits-per-character, and yet still greatly underperform humans on benchmarks & real-world and benefit in every way from scaling up next-token prediction pretraining?

Next-token prediction pretraining clearly works (and continues to work), and yet, also something is clearly missing from the simple story we tell about it. Humans appear to predict better the more important tokens; why and how?

Persistent Adversarial Examples

Why are NN adversarial examples still unsolved over a decade later, while the best efforts at human adversarial examples are debatable?

Scarcely any problems from 2013–2014_12ya DL remain, and yet, in 2024, adversarial examples are not merely still around, but as powerful as ever on standard NN models. Countless solutions have been proposed—and failed. The best defenses are weak & compromise performance, like adversarial training damaging generalization (but oddly, seeming more human-like), or unable to scale, like robustness certification. SOTA NNs rarely even attempt to reduce adversarial examples, LLMs are highly susceptible (and have rough decision boundaries), and DL practitioners usually accept them as a fact of life & work around them.

Meanwhile, attempts to transfer to humans, or create new ones… work to some degree, but not much, and pushing them to the point of meaningful human error requires such large distortions that they no longer seem meaningful.

Human Amnesia

Why are humans so forgetful compared to NNs?

If forgetting is critical to flexible decision-making in biological brains like rodent studies, as has been argued, in part by analogy to successful machine learning regularization techniques like weight decay or dropout, why do the most successful NNs still remember so much compared to humans? For example, an LLM like GPT-3 can casually memorize entire pages of text after training on it once. This is not an extraordinary capability, but happens routinely with LLMs. And the bigger and better they get, the more they are able to memorize.

Whereas with a human, even ones who pride themselves on their memory for text (like myself), memorizing any page of text is effectively impossible without extensive practice in mnemonic techniques, and we tell awed stories of those few human beings who appear to have photographic memories (and demonstrate it is possible for human brains to do such things, they just don’t usually). The usual human’s lack of a photographic memory, and the shallowness of our understanding, can be demonstrated with hilariously simple examples, like “how is the letter ‘g’ written?” or “which way does Lincoln face on the American penny?” Our memories are so bad, in fact, that in addition to struggling to remember what we ate for breakfast yesterday or learning basic general knowledge like “the capital of Russia”, we forget almost our entire childhood, and treat this childhood amnesia as completely normal and may even deny that fact.

This is particularly puzzling given that: we never have more neural connections than we do in childhood, as they continually die or are pruned away; it is not universal cross-species; the amnesia doesn’t seem to damage more core knowledge like motor ability or language; important traumatic memories can be retained; and that those lost episodic memories are, to a considerable degree, still there—detectable by implicit measures like lower reaction times or faster recognition of blurred images or needing fewer hints/cues.

Nor is this because memorizing facts is useless to learning; memorization is critical to learning, and that is true even for eg. spaced repetition of abstract topics.

But then, if memorizing is so useful and more knowledge is better, why do the most successful human memorizers seem to underperform, and it is a mixed blessing at best? Why are examples of photographic memory anecdotally associated with odd limitations in creativity or generalization, like John von Neumann or Luria’s Solomon Shereshevsky, when experimentally in spaced-repetition memory research, better memorization appears to help generalization & understanding without such drawbacks? Why are normal humans capable of memorizing some things, like the Koran or Paradise Lost or the Homeric epics, given enough time & motivation & tricks like memory palaces, but not routinely?

The most extreme cases are idiot savants, whose overall performance profile and pre-DL descriptions bear an uncanny description to LLMs. The feats of savants are too well-documented to doubt, but are impossible to fit comfortably into standard frameworks: if any humans could defeat LLMs in their strong points, it is savants, but then why does savantism seem to only come with such severe costs?

Human Ignorance

And why are humans so ignorant compared to NNs, overall?

Not only do we not recall much easily, we don’t know much either: the breadth of knowledge of even a low-quality LLM in 2024 vastly surpasses that of any individual human. Individual expert humans still usually beat the best LLMs in their narrow niche of expertise, but if we quizzed humans with the same breadth of questions that we benchmark LLMs, I don’t think there is a human alive who would come anywhere near them.

Human Intelligence

So why are humans so smart in general, ie. why do humans generalize more robustly (eschewing the use of non-robust features/‘dimpled’ manifolds like GAN steganography⁴), are more creative, and largely immune to adversarial examples?

One of the most counterintuitive aspects of NNs is that for all the incredible fluency & genuine capabilities, they nevertheless still occasionally make simple blatant mistakes. LLMs in particular—even excluding technical issues caused by BPEs, RLHF & other safety tuning, greedy sampling, and so on, it remains true that they will make baffling errors which are fatal and cause them to go in circles and fail to generalize for things where there seems to be ample training data.

Another version: given that NNs seem to so easily memorize & overfit (and famously can memorize all training data), and operating by “direct fitting” to ‘interpolate’ between the “underspecified” training data, why do they generalize at all?

Need to Sleep

Why do humans need to sleep—so much so that we usually die if sleep-deprived harshly enough?

In contrast, there is no known equivalent of sleep in current DL, nor any apparent need for it.

Even for continual learning or reinforcement learning replay, ordinary minibatch SGD suffices. There is no need for extended periods of offline inactivity or non-interactive computation simply to train on some new data.

Stage-Wise Development

Why does human development spurt through stages to a much greater degree than NNs do?

Children seem to often develop in phases, with sudden spurts (Van Der Maas et al 2006; van Geert 1991; cf. phase transition neural dynamics). Further, children say, believe, and do all sorts of crazy or stupid or evil things—while people make fun of the bizarre things LLMs sometimes say, LLMs have nothing on the things little kids routinely say. (It makes one wonder how human brains reach the adult stage when they apparently must all go through the little kid stages, where if an adult said such things, they would be committed to a mental hospital or feared as a future serial killer.)

That is strikingly different from NNs. NNs demonstrate various kinds of phases within-training and emergence across-scaling, but many of the identified phase transitions, like ‘induction heads’, barely even change the training loss or more than a few benchmarks. In general, NN training appears much smoother than humans do, aside from changes in learning rates, particularly cyclical learning rates.

Slow Development

Why does intelligent development take so long?

Since larger models are more sample-efficient, and many intelligent animals (and definitely humans) are massively larger, why does it take so long?

One of the most universal observations in NN scaling is that, whatever the task, “larger parameter-count models are more sample-efficient”: more precisely, they reduce the loss more per data point than a smaller model does, and so, when the training is plotted comparing loss with data processed, larger models show a distinctly ‘steeper’ plunge than small models do.⁵ (This raises important questions that I’m not sure we know the answers to: “How sample-efficient are extremely, extremely large models—multiple orders of magnitude larger than we now train? How large do they have to become before they stop becoming more sample-efficient? If they do stop at some point, can better regularization fix them? Why do we not train these if we are data-limited in some regimes?”)

Since many animal brains appear to be so much vastly more parameterized than any DL model ever, and to have a large compute-budget and be bottlenecked by processing samples sequentially, we would naively expect an even steeper plunge. The expense of intelligence would be running the brain, not wallclock time.

But everywhere we look, like primates or cetaceans or elephants or birds, we see that intelligence tends to be associated with prolonged childhoods & longevity & low reproductive rates. These long developmental periods don’t seem explicable in purely metabolic terms—you cannot produce a college-level human intelligence by force-feeding toddlers, and while we do see changes like average of puberty dropping like a stone from contemporary caloric abundance, it doesn’t seem to accelerate brain development much.

And why does the development stop? Why don’t we go on being neuroplastic our whole lives but seem to coast after young adulthood, appearing fully functional even if we’ve almost stopped learning like many old people?

Qualitative Differences in Training Dynamics

Where are the missing equivalents for humans of major NN dynamics like deep double descent (true even with 1,000-degree polynomial splines) & “epoch-wise double descent”, super-convergence, and grokking/catapult or patient-teachers?

And if there are none, why not?

Sleep

Tononi’s SHY is one of the most interesting theories attempting to explain sleep in general (rather than dreaming). In SHY, ‘sleep’ is forced by neural net local learning, where the neural net ‘weights’ increase in size over time as they learn; this is bad because it directly increases the energetic demands of firing a synapse, and because (just like an artificial NN), it makes it easier to memorize rather than generalize. The ‘weights’ need to all be multiplied by a fraction, to shrink them back down to normal energy consumption, and helping regularize the brain; but it can’t be done one by one, because they are all in use.

Sleep, then, is when the entire brain is taken offline, to shrink them all simultaneously. So, you can skip sleep for a while, and your brain keeps chugging along, weights growing slowly and becoming more and more inefficient, but it causes increasing problems, and eventually something diverges.

(Dreams, then are not essential; they are probably more of an enhancement for additional sample-efficiency, by doing a more extreme form of experience replay than can be done during waking.)

Savantism

Savants are often capable of remarkable feats of memory or perception, like making near-photorealistic drawings from a single glance at a photograph or memorizing books.

How is it possible for the (non-autistic) Kim Peek to “at age 16–20 months…memorize every book that was read to him. His parents moved Kim’s finger along each sentence being read.”? He could hardly have developed mnemonic palace techniques as an infant or other adult-like memory tricks like hours of repetition; instead, he seems to simply memorize books in a single epoch, so to speak, to parrot them back like an LLM.

Further, why do Peek & other savants seem to avoid childhood amnesia entirely (accounts emphasize that their memory tends to be lifelong, and Peek seems to remember books read as a young child), while also appearing to not go through childhood development stages?

Why do they usually suffer from severe neurological (associated with left hemisphere trauma) & physical problems and demonstrating remarkable deficits in abstract understanding and general intelligence? For example, “he [Peek] cannot, for example, explain many commonplace proverbs”—and even this level of abstraction understanding some proverbs is considered unusual for a savant! As far back as Down 1887, observers were struck by the contrast between the lifelong “verbal adhesion” of many savants and their understanding, or general intelligence. (Treffert & Wallace 2002 note that of the ~100 well-described cases, the highest IQ was 114.) Strikingly for LLM parallels, savants can also be capable of confabulation: Down 1887_139ya describes an illiterate child who “take up a book…and improvise stories of all kinds with a great deal of skill, and in any variety, to suit the supposed tastes of his auditors”.⁶

In cases of mathematical calculation prodigies like calendar dates or taking roots, the underlying algorithm can be inferred to some degree, and appear to mix huge amounts of memorization & distilled intuition with explicit algorithms. These all appear to benefit from disabling higher levels of the brain.

Savants are often intellectually-disabled or otherwise abnormal. An important example in this context is Luria’s famous mnemonist, Solomon Shereshevsky (The Mind of a Mnemonist), who simply remembered everything he saw or heard: in part by using extensive visualization & mnemonic approaches like the memory palace, but his level of synesthesia also meant that everything was inextricably associated for him.⁷

He struggled with non-literal interpretations, reading, or even recognizing faces—he could remember exactly how a person’s face looked previously, of course, but then he could not generalize to how their face looked now. Why did he not benefit from the ability to recall every version of a person’s face, and gain uncanny super-recognizer powers? (See also hyperthymesia: people with HSAM do not have an intrinsically superior memory for learning, but for not forgetting—they seem to struggle to focus on daily life and cope with anxiety or memories of bad days.) He has what sounds like maladaptive daydreaming, and ultimately died of alcoholism.

On a similar note, one of John von Neumann’s famous party tricks was memorizing pages of books, and being able to recite books on command—just one of his many remarkable feats of cognition which have entered legend. But von Neumann was criticized for pedanticism, explicitness, inelegance, brute-force calculation, comfort with following complicated lines of reasoning by reliance on his photographic memory. Eugene Wigner, who knew Von Neumann so well, while describing his feats, struck a cautionary note: none of them were truly original, and Von Neumann himself reportedly said that “[I] will be forgotten while Kurt Gödel is remembered with Pythagoras”. Was von Neumann ever original & creative enough to be truly great like Pythagoras or Gödel?[^Wigner] Was he held back by his great knowledge & calculating ability (if not exactly “library work”), and unable to drill to the essence of matters like a Grothendieck?

If we combine the ideas of lack of abstraction or later stages of childhood cognitive development, memorization, synesthesia (failing to separate modalities), injuries to the left hemisphere (particularly the temporal lobe), ability to confabulate, and general extreme imbalances in cognitive capabilities compared to normal humans, I am left with the impression that savants are the “LLM version” of humans. Savants are what can sometimes happen when a key node in the brain is knocked out and higher level processes become less effective, exposing more of the brain’s low-level raw intelligence, which is satisfied by simpler forms of learning like massive memorization.

Prototyping With Arithmetic

It might be feasible to test LLM catapulting on small-scale tasks where current LLMs clearly generalize poorly, like arithmetic. Arithmetic is about the smallest, simplest, easiest-to-generate problem that LLMs still fail in oddly brittle ways on, so it’s a great testbed.

Arithmetic is learnable with appropriate formatting by small cheap LLMs, but standard LLMs (trained on natural arithmetic text data) continue to not implement true arithmetic, even at the capability frontier like GPT-4, and arithmetic problems are easy to generate, benchmark, understand, and even do neural net interpretability on; so one could pilot catapulting on a pretrained LLM by looking for training schedules which make it find true arithmetic much faster than standard finetuning does.

Prototyping With Image Classification

For non-LLMs like CNNs/MLPs trained on CIFAR-10/CIFAR-100 or ImageNet-1k (MNIST being too trivial to be convincing), we would similarly expect more human-like images. In image classification, it’s harder to define ‘hard’ than arithmetic, and the standard NN accuracy is so high that simply filtering for errors might not yield enough ‘reasonable’ errors at this point.

So we would instead use one of the many ‘post-ImageNet’ image datasets designed to stress-test classifiers, like ImageNet-A, ImageNet-C, ImageNet-Hard, ImageNet-R, & ImageNet-Sketch. These would be used as the target in scaling law sweeps as described previously.

Prior Art

There is essentially no research on training >10t-parameter LLMs, or cyclical LRs on large LLMs (as opposed to small GPT-2-scale ones).

Historically, this is due to a mix of academic fashion/prejudices and capability gains by smaller models. Work on training highly-overparameterized LLMs, or on the equiparameterization regime¹⁴, was largely killed by the release of the Chinchilla paper, which provided the perfect excuse for everyone to immediately halt parameter-scaling (since it no longer led immediately to SOTA results), as they have always wanted to, for a mix of good and bad reasons. Similarly, the excuse of “optimizing for inference-optimality” by overtraining small models has become popular, by optimizing scaling laws which assume that the trained model will be naively deployed as-is, without the actual pruning, quantizing, & distilling everyone does anyway.

This means that the field of high-energy DL is wide-open: this proposal will be highly unpopular, it is much less likely than usual that anyone would independently investigate this direction, and they will be discouraged by poor preliminary results when the training runs appear to have simply failed (because of subtle bugs, poor hyperparameters, or simply inadequate training time to catapult into a region where better results can be benchmarked—assuming the right benchmarks are being used to begin with).

Benchmarking

This is such a different training regime that previous scaling law sweeps are inapplicable. Further, the goal here may be one that existing benchmarks actively mislead on. They test mostly common easy questions—the sort where ‘direct fit’-like thinking does best on, by definition.

So a major question here would be whether scaling laws should target perplexity as usual, or if they need to target a custom benchmark which tries to test human-like generalization rather than memorization.

Given the difficulty we have in constructing non-trivia-heavy benchmarks which existing LLMs can’t beat, this might be one of the hardest parts! But I suspect that after training a reasonable HLLM, possibly via trial-and-error, and interacting with it for a while to get an idea of how it acts qualitatively, the right metric might become more obvious.

The benchmarks might include adversarial robustness, hard negative mining (ie. the hardest problems that the best LLMs still get wrong), meta-learning, checking how much sets of models add to an ensemble’s accuracy¹⁵, or use a metric which rewards performance while penalizing memorization of training data.

Capabilities

The final model should generalize much better—possibly achieving the Nyquist learner limit of perfectly modeling the true (non-dimpled?) latent manifold, and thereby constructively answering Rosenfeld’s question about how a band-limited Nyquist learner could be implemented in current NN architectures.

Per the lottery ticket hypothesis, once the true generalizing algorithm has been found, and has been further trained as desired (perhaps on some large trivia-heavy corpus), we can prune it down to a much smaller, faster, more feasible-to-use model, in an example of “train large, then compress”.

Given the capacity of even small models, it may be possible to finetune these smaller models on arbitrarily large amounts of data to exploit the scaling laws of transfer. Because they start with the human-like generalizing prior, they will learn & memorize the data as appropriate, but without the pathologies of ‘direct fit’ to the data as when trained from scratch on the same data—thereby achieving the best of both worlds.

Alignment

Speculating further, on the premise of the above: most capability improvements do not help AI alignment as much. This is because they either are specific to a capability in a narrow domain, or they enhance broad capabilities but only in a ‘brittle’ non-generalizing way, which can be highly economically valuable but doesn’t help ‘alignment’ much because we are interested in alignment not for its current narrow in-domain but generalizing. We need the kind of alignment which doesn’t help a chatbot say socially-acceptable things today (or do the right things for the wrong reasons), but which would make chatbots in charge of civilization do socially-acceptable things in the future (and do the right things, for the right reasons, so they will keep on doing the right things indefinitely).

But this catapult or Nyquist learner may be the exception, because what it helps with is true generalization. A catapulted LLM trained on large amounts of morality-related text has not learned purely an assemblage of memorized fragments, heuristics, tricks, and statistical associations, with any underlying algorithm begrudgingly forced by scaling, or learned deception & situated reasoning to maximize a reward, or been adversarially selected by use of ‘interpretability’ techniques to learn to think in nonlinear opaque ways which do not raise any red flags; instead, it has learned the underlying value-manifold the hard way, like a highly-intelligent, grown, moral adult human has.

Even if the capability improvement turns out to be beaten by standard LLM scaling approaches (like simply brute-force annotating every error), true generalization would be invaluable to alignment.

It does not solve the alignment problem in generality—but it might provide a way to create a genuinely moral AI, and that is a good starting point.

Interpretability

Further, because the native neural net way of thinking is a large complicated pastiche of memorization & heuristics, while the overparameterized grokked LLM has distilled out an algorithmic core for the key tasks, such a catapulted LLM ought to be much more useful for interpretability work. We can more easily validate that they are genuinely moral AI.

Once the overparameterization has been removed, what is left should be natively much closer to simple, verifiable, interpretable, and extractable algorithms. These can then be formally analyzed & verified (possibly with the assistance of the genuinely-moral AI LLMs, which are still risky but not too risky if they are probably moral to begin with, and we confine them to carefully-checked formal outputs and tasks like algorithmic equivalence of a sparsified neural network and extracted sub-algorithms.)