Skip to main content

WBE and DRL: a Middle Way of Imitation Learning

N/A

by: Gwern 2018-10-20–7y2026-05-27 finished certainty: log

Description of emerging machine learning paradigm identified by commentator starspawn0: discussions of building artificial brains typically presume either learning a brain architecture & parameters from scratch (AGI) or laboriously ‘scanning’ and reverse-engineering a biological brain in its entirety to get a functioning artificial brain.

However, the rise of deep learning’s transfer learning & meta-learning shows a wide variety of intermediate approaches, where ‘side data’ from natural brains can be used as scaffolding to guide & constrain standard deep learning methods. Such approaches do not seek to ‘upload’ or ‘emulate’ any specific brain, they merely seek to imitate an average brain. A simple example would be training a CNN to imitate eye tracking saliency data: what a human looks at while playing a video game or driving is the important part of a scene, and the CNN doesn’t have to learn importance from scratch. A more complex example would be using EEG as a ‘description’ of music in addition to the music itself. fMRI data could be used to guide a NN to have a similar modularized architecture with similar activation patterns given a particular stimulus as a human brain, which presumably is related to human abilities to zero-shot/few-shot learn and generalize.

While a highly marginal approach at the moment compared to standard approaches like scaling up models & datasets, it is largely untapped, and progress in VR headsets with eye tracking capabilities (intended for foveated rendering but usable for many other purposes), brain imaging methods & BCIs has been more rapid than generally appreciated—in part thanks to breakthroughs using DL itself, suggesting the potential for a positive feedback loop where a BCI breakthrough enables a better NN for BCIs and so on.

Most deep learning methods attempt to learn artificial neural networks from scratch, using architectures or neurons or approaches often only very loosely inspired by biological brains; on the other hand, most discussions of ‘whole brain emulation’ assume that one will have to learn every or almost every neuron in large regions of or the entire brain from a specific person, and the debate is mostly about how realistic (and computationally demanding) those neurons must be before it yields a useful AGI or an ‘upload’ of that person.

This is a false dichotomy: there’s a lot of approaches in between. WBE is, in some ways, the worst and least efficient way of approaching AGI. What sorts of less-than-whole brain emulation are possible and useful?

Highlighted by /u/starspawn0 a year ago (“A possible unexpected path to strong AI (AGI)”), there’s an interesting vein of research which takes the middle way of treating DL/biological brains as a kind of imitation learning (or knowledge distillation), where human brain activity such as fMRI, EEG, or Eyetracking, is taken as being itself as being some kind of rich dataset or oracle to learn better algorithms from, to learn to imitate, or meta-learn new architectures which then train to something similar to the human brain:

With enough data from enough people, a model could generalize flexibly and quickly to any new people. In brain imitation learning, I’m not thinking of imitating a specific brain, necessarily. I was thinking more of meta-learning: each brain is drawn from the distribution of brains, you don’t care much about each specific brain or the ‘average’ brain, you want to capture the generic algorithm which each brain is instantiating in a somewhat different way.

So for example, you might treat it MAML-style. Take a set of images, expose the set to a set of humans while recording their brain; now sample each human’s recording set and train the seed NN to try to match its activations; take a gradient step in the seed NN to minimize the loss; repeat. Let’s make better use of human brains than just single-labeling cat images or drawing some bounding boxes!

Does this get you a much more human-like, generalizable, sample-efficient CNN which you can then apply to any other dataset which performs better than your standard resnet?

Human preferences/brain activations are themselves the reward (especially useful for things where explicit labeling is quite hard, such as, say, moral judgments or feelings of safety or fairness, or adaptive computation like eye tracking where humans can’t explain what they do), or the distance between neural activations for a pair of images represents their semantic distance and a classification CNN is penalized accordingly, or the activation statistics become a target in hyperparameter optimization/neural architecture search (‘look for a CNN architecture which when trained in this dataset produces activations with similar distributions as that set of human brain recordings looking at said dataset’), and so on. (Eye-tracking+fMRI activations = super-semantic segmentation?)

Given steady progress in brain imaging technology, the extent of recorded human brain activity will escalate and more and more data will become available to imitate/optimize based on. (The next generation of consumer desktop VR is expected to include eye tracking, which could be interesting for DRL as people are already moving to 3D environments and so you could get thousands of hours of eye tracking/saliency data for free from an installed base of hundreds of thousands or millions of players; and starspawn0 often references the work of Mary Lou Jepsen, among other brain imaging trends.) As human brain architecture must be fairly generic, learning to imitate data from many brains may usefully reverse-engineer architectures.

These are not necessarily SOTA on any tasks yet (I suspect usually there’s some more straightforward approach using way more unlabeled/labeled data which works), so I’m not claiming you should run out and try to use this right away. But this seems like a potentially very useful in the long run paradigm which has not been explored nearly as much as other topics and is a bit of a blind spot, so I’m raising awareness a little here.

Looking to the long-term and taking an AI risk angle: given the already demonstrated power & efficiency of DL without any such help, and the compute-budget requirement of even optimistic WBE estimates, it seems quite plausible that a DL learning to imitate (but not actually copying or ‘emulating’ in any sense) a human brain could, a fortiori, achieve AGI long before any WBE does (which must struggle with the major logistics challenge of scanning a brain in any way and then computing it), and it might be worth thinking about this kind of approach more.

What is interesting is that these prototype approaches work at all. If you had asked me, ‘can you use EEG signals to meaningfully improve music or image classification’ I would have been amused at the suggestion and said of course not. What could EEG signals possibly convey that the NN or SVM or random forest or other algorithm couldn’t learn much more easily on its own?

Brain imaging approaches have been increasing exponentially in precision and resolution for decades now, so the trend there is good, independent of the specific lines of research. Plus VR headsets will come online soon. Once you can capture eye tracking with a $657.02$5002018 headset you bought for Serious Research Purposes, and a few lines of code in Unity, why wouldn’t you?

So, I think this is an untapped paradigm that very few people even know is a thing, much less are thinking about what hybrid approaches are possible, or running serious large-scale research projects on

And on the flip side, one possibility is that the BCI will allow powerful interaction of the sort simply not possible now based on using brain activations as supervision for understanding material in a way which transports those extremely complex abstractions into the computer in a software-understandable way

To give a quick random example: imagine the BCI records your global activations as you read that Reddit post about deep learning augmented by EEG/MRI/etc data; a year later while reading HN about something AI, you want to leave a comment & think to yourself ‘what was that thing about using labels from brain imaging’ which produces similar activations and the BCI immediately pulls up 10 hits in a sidebar, and you glance over and realize the top one is the post you were thinking of and you can immediately start rereading it. And then of course your brain activations could be decoded into a text summary for the comment which you can slightly edit and then post…

One of the things I find frustrating about BCIs is that everyone is working hard on them without a good idea of what exactly one would do with them (aside from the most obvious things like ‘robot hand’).

I can barely remember how to use the software I have already; to quote an old Tweet of mine:

Me: “I don’t understand this Mary Lou Jepsen BCI stuff but it sounds very cool. If only Engelbart were here to see!”

Also me: “I forgot how Emacs registers & bookmarks work. I should take 20s to look it up but I won’t.”

also also me: “which order does ln -s go in, again”

It’s very hand-wavy: “it’ll be a memory prosthetic increasing IQ 20 points!” ‘yeah but how’ ‘uh…’. (And why don’t tools for thought work?) I don’t need a detailed prototype laying out every step, even just a generic description would do. What’s the VisiCalc or visual text editor of BCI? You can describe them, the way Engelbart or Alan Kay could describe their systems on paper, without needing to actually make them or know all the details. But no one’s done so for BCIs that I’ve seen. As enormous as Waitbutwhy discussion of Neuralink is, the examples kinda boil down to ‘maybe you could have a little TV in your mind’.

Taking a brain-imitation approach seems to help me imagine more concretely what could be done with a BCI.

So even with just this surface recording data you can imagine doing a lot. You could use the embedding as an annotation for all input streams, like lifelogging. There are probably tons of specific applications you can imagine just on the paradigm of associating mental embeddings with screenshots/text/emails/documents/video timestamps: it’s automatic semantic tagging of persons, places, times, subjects, emotions…

It could be used as feedback too. Perhaps there’s an embedding which corresponds to coding or deep thought, in which case all notifications are automatically disabled, except for notifications about emails where the RNN predictor predicts high importance based on alertness/excitedness embeddings of earlier emails. Or neurofeedback: the simplest version being to make you calm down. (I remember Gmail had a ‘beer’ feature, I think, where it would offer to delay email if you sent them late at night or make you solve arithmetic puzzles to be sure you wanted to send it.)

[]