August 2021 Gwern.net newsletter with links on molecular recording, TODO
August 2021’s Gwern.net newsletter is now out; previous, July 2021 (archives). This is a collation of links and summary of major changes, overlapping with my Changelog; brought to you by my donors on Patreon.
Writings
Links
AI
- “EfficientZero: Mastering Atari Games with Limited Data”, et al 2021 (beating humans on ALE-100k/2h by adding self-supervised learning to MuZero-Reanalyze); “Procedural Generalization by Planning with Self-Supervised World Models”, et al 2021 (generalization capabilities of MuZero-Reanalyze: self-supervised learning also leads to new SOTA on ProcGen, w/implicit meta-learning on MetaWorld)
- “DORA: No-Press Diplomacy from Scratch”, et al 2021 (previously: et al 2020 )
- “Monkey Plays Pac-Man with Compositional Strategies and Hierarchical Decision-making”, et al 2021 ( videos)
- “Fake It Till You Make It: Face analysis in the wild using synthetic data alone”, et al 2021 (modern ML-aided CGI pipelines are amazing)
- “Acquisition of Chess Knowledge in AlphaZero”, et al 2021
- “KaraSinger: Score-Free Singing Voice Synthesis with VQ-VAE using Mel-spectrograms”, et al 2021 (I am easily amused by AI papers with sung abstracts; similar to Jukebox)
-
“Open-Ended Learning Leads to Generally Capable Agents”, Open Ended Learning et al 2021 ( blog)
-
“Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model”, et al 2022 ( blog; 3× GPT-3 & using The Pile, but undertrained, so only modestly better); “Yuan 1.0: Large-Scale Pre-trained Language Model in Zero-Shot and Few-Shot Learning”, et al 2021 ( media; 245.7b-parameter GPT-3-like Chinese text model)
-
“China Has Already Reached Exascale—On Two Separate Systems” (anonymous source claims 4.4 exaflops FP16; but kept secret? Concerning arms race dynamics if true)
-
“Recursively Summarizing Books with Human Feedback”, et al 2021
-
“Exploring the Limits of Large Scale Pre-training”, et al 2021
-
“FLAN: Finetuned Language Models Are Zero-Shot Learners”, et al 2021 ( blog)
-
“Unsupervised Neural Machine Translation with Generative Language Models Only”, et al 2021; “Scaling Laws for Neural Machine Translation”, et al 2021; “Data and Parameter Scaling Laws for Neural Machine Translation”, et al 2021
- “Training Verifiers to Solve Math Word Problems”, et al 2021 ( blog; boosting GPT-3 on math word problems from ~15% to ~60% by self-distilling a critic & best-of = 100 sampling)
-
“Program Synthesis with Large Language Models”, et al 2021 (smooth scaling like Codex continues to 137b-parameter LaMDA; enables dialogue, solving math problems); “Show Your Work: Scratchpads for Intermediate Computation with Language Models”, et al 2021 ( AQUA-RAT might be worth revisiting)
-
“On the Opportunities and Risks of Foundation Models”, et al 2021
-
“A Universal Law of Robustness via Isoperimetry”, 2021 (Twitter, talk; blessings of scale—scaling is also magic pixie dust for adversarial attacks, bitter-lessoning an entire academic field of ever more elaborate (yet failed) defenses…?); “Why Robust Generalization in Deep Learning is Difficult: Perspective of Expressive Power”, et al 2022
-
“On the Predictability of Pruning Across Scales”, et al 2020 (scaling laws for sparsity: initially free large size reductions, then power-law worsening, then plateau at tiny but bad models)
Genetics
Everything Is Heritable:
-
new GIANT n = 5.4m height GWAS explains full SNP heritability of height: “A Saturated Map of Common Genetic Variants Associated with Human Height from 5.4 Million Individuals of Diverse Ancestries”, et al 2022 ( media; height becomes the first human GWAS to ‘finish’ by reaching the ceiling of SNP heritability; so much for ‘missing heritability’! It took barely a decade to go from the first height GWASes of n = 4k / 0.3% variance to complete prediction. Exponentials are a helluva drug.)
-
“Rare variant contribution to human disease in 281,104 UK Biobank exomes”, et al 2021; “Protein-coding repeat polymorphisms strongly shape diverse human phenotypes”, et al 2021; “The sequences of 150,119 genomes in the UK Biobank”, et al 2021
-
“Discovery of 42 Genome-Wide Statistically-Significant Loci Associated with Dyslexia”, et al 2021
-
“Somatic mutation rates scale with lifespan across mammals”, et al 2021
-
“100,000 Genomes Pilot on Rare-Disease Diagnosis in Health Care—Preliminary Report”, The 100k Genomes Project Pilot 2021 (reminder: we should WGS sequence everyone); “Ultrarapid Nanopore Genome Sequencing in a Critical Care Setting”, et al 2022
Recent Evolution:
- “A selection pressure landscape for 870 human polygenic traits”, et al 2021
- “The genetic basis of tail-loss evolution in humans and apes”, et al 2021 (progress towards catgirls, or, Return To Monke? Rudimentary human tails still appear sometimes)
Engineering:
- “How Silicon Valley hatched a plan to turn blood into human eggs: A well-connected startup company [Conception] is trying to rewrite the rules of reproduction”
- “CRISPR Gene-Editing Experiment Partly Restores Vision In Legally Blind Patients”
- “In a First, Surgeons Attached a Pig Kidney to a Human—and It Worked” (and then into the abdomen as well); “Man Receives a Heart From a Genetically Altered Pig” (unsurprisingly, ultimately failed—but progress)
- “A Cure for Type 1 Diabetes? For One Man, It Seems to Have Worked. A new treatment using stem cells that produce insulin has surprised experts and given them hope for the 1.5 million Americans living with the disease”
Statistics/Meta-Science
- “Nuclear Fission: Reaction to the Discovery in 1939”, et al 1986 (what does it look & feel like to be in the early stages of creating an x-risk like nuclear bombs? eg. Nobel physicist Millikan’s assertion that nuclear energy is a “hobgoblin” & men should “sleep in peace with the consciousness that the Creator has put some foolproof elements into his handiwork and that man is powerless to do it any titanic physical damage”—one striking flaw is the repeated reliance on equilibrium/anthropic Outside View arguments, that ‘nuclear risks must be impossible because the Earth and natural ore deposits still exist’, which is irrelevant to the question of the dangers of creating new technologies such as with highly-concentrated ores)
- “Serendipity: Towards a taxonomy and a theory”, 2018
- “Truncating Bar Graphs Persistently Misleads Viewers”, et al 2021
Politics/Religion
- “Good news on climate change” (emissions progress; tail risks overestimated; cost/experience curves underestimated); “Advice to Young People, as You Face Annihilation”, Roger Hallam (co-founder of Extinction Rebellion); “Stop Telling Kids They’ll Die From Climate Change: Many young people feel like their future is in peril. To make progress on climate change, we must move past doomsday scenarios”, Hannah Ritchie (Our World in Data)
- “Genes, Ideology, and Sophistication”, 2021 (many peoples’ politics are too inchoate to have stable causes like genes or upbringing)
- “Long-term Health and Social Outcomes in Children and Adolescents Placed in Out-of-Home Care”, et al 2021 (foster homes may be bad)
Psychology/Biology
- “A Multisite Preregistered Paradigmatic Test of the Ego-Depletion Effect”, et al 2021 (large failure to replicate ego-depletion, strong support for null; as predicted)
- “Physical principles for scalable neural recording”, et al 2013; “Can One Concurrently Record Electrical Spikes from Every Neuron in a Mammalian Brain?”, et al 2019; “Tracking neural activity from the same cells during the entire adult life of mice”, et al 2021; “Imaging intact human organs with local resolution of cellular structures using hierarchical phase-contrast tomography”, et al 2021
- “Computation in the human cerebral cortex uses less than 0.2 watts yet this great expense is optimal when considering communication costs”, 2020 (communication 20× energy of computation; 7 bits/sec/neuron; similar efficiency to rats—humans aren’t special, we just have big brains)
- “Energy compensation and adiposity in humans”, et al 2021; “The future of weight loss”, Stephan J. Guyenet (on semaglutide); “Anti-obesity drug discovery: advances and challenges”, et al 2021
- Molecular recording: “Time-tagged ticker tapes for intracellular recordings”, et al 2021 ( Twitter); “Recording of cellular physiological histories along optically readable self-assembling protein chains”, et al 2021; “Infinite re-reading of single proteins at single-amino-acid resolution using nanopore sequencing”, et al 2021; “Molecular recording of sequential cellular events into DNA”, et al 2021; “Multiplex genomic recording of enhancer and signal transduction activity in mammalian cells”, et al 2021; “A temporally resolved, multiplex molecular recorder based on sequential genome editing”, et al 2021
Technology
- “Preliminary Study Of The Nuclear Subterrene”, et al 1971 ( background, and outcome)
- “A Brief History of the Index”, I Love Typography
- “Better eats: The kitchen of 2020 looks mostly the same as that of 1960. But what we do in it has changed dramatically, almost entirely for the better—due to a culture of culinary innovation”
- “The YouTube Revolution in Knowledge Transfer”, Samo Burja
Economics
- “Experienced well-being rises with income, even above $75,000 per year”, 2021; “The impact of cash transfers on subjective wellbeing and mental health in low-income and middle-income countries: A systematic review and meta-analysis”, et al 2020
- “Empirical audit and review and an assessment of evidentiary value in research on the psychological consequences of scarcity”, et al 2021
- “Only the Bad Die Young: Restaurant Mortality in the Western US”, 2014 (not as bad as I thought: 17% first-year mortality, median 4.5 years)
Philosophy
- “Why Philosophers Should Care About Computational Complexity”, 2011
- “Language and thought are not the same thing: evidence from neuroimaging and neurological patients”, 2016
- “Fundamentally Misunderstanding Visual Perception: Adults’ Belief in Visual Emissions”, et al 2002 (most children & many college-educated adults believe you see by shooting beams from your eyes & can feel stares; it is difficult to durably correct their intuition)
Fiction
- “Shovel Knight Drop Caps: Bitmap Graphics Reimagined as Medieval Woodblock Prints”, Jamie Clarke (custom initials for a game; handsome, and perhaps an under-represented idea? Robert Jordan’s Wheel Of Time was memorable in part for the chapter icons which hinted at the topic & helped the reader keep their place in such long novels; wouldn’t web serials like A Practical Guide To Evil benefit from a set of drop caps, which could also depict character progression? A few hundred dollars in commissioned art could add a lot of visual flavor!)
Miscellaneous
- New London School explosion (the worst school disaster in American history)
- The Pink Panthers (New Yorker)