“Twitching in Sensorimotor Development from Sleeping Rats to Robots”, 2023-06-17 (; backlinks):
It is still not known how the ‘rudimentary’ movements of fetuses and infants are transformed into the coordinated, flexible and adaptive movements of adults. In addressing this important issue, we consider a behavior that has been perennially viewed as a functionless by-product of a dreaming brain: the jerky limb movements called myoclonic twitches.
Recent work has identified the neural mechanisms that produce twitching as well as those that convey sensory feedback from twitching limbs to the spinal cord and brain. In turn, these mechanistic insights have helped inspire new ideas about the functional roles that twitching might play in the self-organization of spinal and supraspinal sensorimotor circuits. [offline reinforcement learning]
Striking support for these ideas is coming from the field of developmental robotics: when twitches are mimicked in robot models of the musculoskeletal system, the basic neural circuitry undergoes self-organization. Mutually inspired biological and synthetic approaches promise not only to produce better robots, but also to solve fundamental problems concerning the developmental origins of sensorimotor maps in the spinal cord and brain.
[an argument people make for biological neural networks being extremely sample-efficient is to point out incredible feats of motor learning that newborn animals engage in, like being born and then able to walk or run within minutes, which seems to far surpass the sample-efficiency of any DRL robotics learning from scratch (ie. without pretraining); this is taken to imply either that genetics has been able to encode priors into animal brains or that biological brains are doing superior RL to DRL.
However, if they are spending months twitching for hours to do offline RL of motor control before they are born, then they are actually collecting many samples before birth, and the post-birth sample efficiency must be correspondingly much less than one would expect, so their priors and/or brain algorithm look that much less impressive.
It’s not learning from scratch, it’s closer to sim2real or meta-learning, where NNs like Dactyl can do quite well given a few seconds/minutes of experience too, and look much closer to animal-like efficiency in light of the twitching.]