“From Reinforcement Learning to Agency: Frameworks for Understanding Basal Cognition”, 2024 ():
Organisms play, explore, and mimic those around them. Is there a purpose to this behavior? Are organisms just behaving, or are they trying to achieve goals? We believe this is a false dichotomy.
To that end, to understand organisms, we attempt to unify two approaches for understanding complex agents, whether evolved or engineered. We argue that formalisms describing multiscale competencies and goal-directedness in biology (eg. TAME), and reinforcement learning (RL), can be combined in a symbiotic framework.
While RL has been largely focused on higher-level organisms and robots of high complexity, TAME is naturally capable of describing lower-level organisms and minimal agents as well.
We propose several novel questions that come from using RL/TAME to understand biology as well as ones that come from using biology to formulate new theory in AI. We hope that the research programs proposed in this piece shape future efforts to understand biological organisms and also future efforts to build artificial agents.
[Keywords: AI, reinforcement learning, machine learning, agency, goal-directedness, Teleonomy]
…The ability of biological systems to respond to novel conditions goes even deeper than subtractive injury or abnormal starting states. When cells of the newt are artificially increased in size, the resulting animals are normal, showing adjustment and rescaling of organs to a smaller number of cells per structure. The most amazing aspect is the kidney tubule, which in cross-section normally consists of 8 cells working together. When the cells are made bigger in experiments, fewer and fewer cells cooperate to make the same diameter tubules, until the cells are made extremely huge, at which point just one cell wraps around itself to make a lumen of the correct size (1945). This example shows that diverse molecular mechanisms (cell-cell communication vs. cytoskeletal bending) can be called up in the service of a large-scale anatomical goal. But even the large-scale goals of living forms can be altered on-the-fly, and it does not require changes of the genome. Planarian flatworms can be turned into animals that always produce two heads upon damage ( et al 2017; Oviedo et al 201014ya), or indeed produce heads belonging to other species of worms (Emmons- et al 2015), by a transient modification to the bioelectric memory pattern that encodes their target morphology ( et al 2016; et al 2019), without transgenes or mutation. Similarly, wild-type skin cells liberated from the instructive influence of their neighbors reboot their multicellularity toward a new motile form: Xenobots ( et al 2020): proto-organisms which exhibit novel behaviors (including kinematic self-replication ( et al 2021)) and healing after damage to their new Xenobot form.