August 2021 News
August 2021 Gwern.net newsletter with links on molecular recording, TODO
August 2021’s Gwern.net newsletter is now out; previous, July 2021 (archives). This is a collation of links and summary of major changes, overlapping with my Changelog; brought to you by my donors on Patreon.
Writings
Links
AI
-
“EfficientZero: Mastering Atari Games with Limited Data”, et al 2021 (beating humans on ALE-100k/2h by adding self-supervised learning to MuZero-Reanalyze); “Procedural Generalization by Planning with Self-Supervised World Models”, et al 2021 (generalization capabilities of MuZero-Reanalyze: self-supervised learning also leads to new SOTA on ProcGen, w/implicit meta-learning on MetaWorld)
-
“DORA: No-Press Diplomacy from Scratch”, et al 2021 (previously: et al 2020 )
-
“Monkey Plays Pac-Man with Compositional Strategies and Hierarchical Decision-making”, et al 2021 ( videos)
-
“Fake It Till You Make It: Face analysis in the wild using synthetic data alone”, et al 2021 (modern ML-aided CGI pipelines are amazing)
-
“Acquisition of Chess Knowledge in AlphaZero”, et al 2021
-
“RASP: Thinking Like Transformers”, et al 2021 (programming languages in Transformers); “𝑖λ, a family of imaginary programming libraries” (programming languages calling Transformers)
-
“KaraSinger: Score-Free Singing Voice Synthesis with VQ-VAE using Mel-spectrograms”, et al 2021 (I am easily amused by AI papers with sung abstracts; similar to Jukebox)
-
“Percival: Making In-Browser Perceptual Ad Blocking Practical With Deep Learning”, et al 2019
-
“Open-Ended Learning Leads to Generally Capable Agents”, Open Ended Learning et al 2021 ( blog)
-
“Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model”, et al 2022 ( blog; 3× GPT-3 & using The Pile, but undertrained, so only modestly better); “Yuan 1.0: Large-Scale Pre-trained Language Model in Zero-Shot and Few-Shot Learning”, et al 2021 ( media; 245.7b-parameter GPT-3-like Chinese text model)
-
“China Has Already Reached Exascale—On Two Separate Systems” (anonymous source claims 4.4 exaflops FP16; but kept secret? Concerning arms race dynamics if true)
-
“A General Language Assistant as a Laboratory for Alignment”, et al 2021 (when scaling to 52b, larger models get friendlier faster & learn from rich human preference data); “Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned”, et al 2022 (scaling helps RL preference learning, but not other safety)
-
“Recursively Summarizing Books with Human Feedback”, et al 2021
-
“Exploring the Limits of Large Scale Pre-training”, et al 2021
-
“FLAN: Finetuned Language Models Are Zero-Shot Learners”, et al 2021 ( blog)
-
“Unsupervised Neural Machine Translation with Generative Language Models Only”, et al 2021; “Scaling Laws for Neural Machine Translation”, et al 2021; “Data and Parameter Scaling Laws for Neural Machine Translation”, et al 2021
-
“Training Verifiers to Solve Math Word Problems”, et al 2021 ( blog; boosting GPT-3 on math word problems from ~15% to ~60% by self-distilling a critic & best-of = 100 sampling)
-
-
“ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation”, et al 2021 ( demo; #1 SuperGLUE, exceeding human baseline & Microsoft Research performance)
-
“Structured Denoising Diffusion Models (D3PMs) in Discrete State-Spaces”, et al 2021; “Beyond In-Place Corruption: Insertion and Deletion In Denoising Probabilistic Models”, et al 2021; “Argmax Flows and Multinomial Diffusion: Learning Categorical Distributions”, et al 2021a; “ARDMs: Autoregressive Diffusion Models”, et al 2021b; “SUNDAE: Step-unrolled Denoising Autoencoders for Text Generation”, et al 2021 “Zero-Shot Translation using Diffusion Models”, 2021; “Diffusion-LM Improves Controllable Text Generation”, et al 2022 (DDPMs for text; potentially better global coherency as the entire text is updated repeatedly, in a much more natural way allowing for turning compute into performance & exploring alternatives… I suspect a diffusion text model will be largely immune to the repetition trap. Good for top-down story generation? Or useful for anagram/palindrome generation? Anagram generators are useless past 1–2 words due to combinatorial explosion; but a DDPM could handle almost arbitrary sized inputs to reconstruct meaningful sentences. Training noise = shuffling letters. To create: take a target string, random shuffle, and denoise. Do this repeatedly for multiple anagrams of the original target. Can use embeddings to guide denoising towards semantically different or identical anagrams of original. Applications in CS, genomics?)
Genetics
Everything Is Heritable:
-
new GIANT n = 5.4m height GWAS explains full SNP heritability of height: “A Saturated Map of Common Genetic Variants Associated with Human Height from 5.4 Million Individuals of Diverse Ancestries”, et al 2022 ( media; height becomes the first human GWAS to ‘finish’ by reaching the ceiling of SNP heritability; so much for ‘missing heritability’! It took barely a decade to go from the first height GWASes of n = 4k / 0.3% variance to complete prediction. Exponentials are a helluva drug.)
-
“Rare variant contribution to human disease in 281,104 UK Biobank exomes”, et al 2021; “Protein-coding repeat polymorphisms strongly shape diverse human phenotypes”, et al 2021; “The sequences of 150,119 genomes in the UK Biobank”, et al 2021
-
“Discovery of 42 Genome-Wide Statistically-Significant Loci Associated with Dyslexia”, et al 2021
-
“Somatic mutation rates scale with lifespan across mammals”, et al 2021
-
“100,000 Genomes Pilot on Rare-Disease Diagnosis in Health Care—Preliminary Report”, The 100k Genomes Project Pilot 2021 (reminder: we should WGS sequence everyone); “Ultrarapid Nanopore Genome Sequencing in a Critical Care Setting”, et al 2022
-
“Predicting skeletal stature using ancient DNA”, et al 2021 (sequencing the graveyard)
-
“Molecular Support for Heterogonesis Resulting in Sesquizygotic Twinning”, et al 2019 (“maternally identical but chimerically shared 78% of their paternal genome”)
Recent Evolution:
-
“A selection pressure landscape for 870 human polygenic traits”, et al 2021
-
“Clonal genome evolution and rapid invasive spread of the marbled crayfish”, et al 2018
-
“The genetic basis of tail-loss evolution in humans and apes”, et al 2021 (progress towards catgirls, or, Return To Monke? Rudimentary human tails still appear sometimes)
Engineering:
-
“CRISPR Gene-Editing Experiment Partly Restores Vision In Legally Blind Patients”
-
“In a First, Surgeons Attached a Pig Kidney to a Human—and It Worked” (and then into the abdomen as well); “Man Receives a Heart From a Genetically Altered Pig” (unsurprisingly, ultimately failed—but progress); “Surgeons Transplant Pig Kidney Into a Patient, a Medical Milestone: The man continues to improve, doctors said. Organs from genetically engineered pigs one day may make dialysis obsolete”; “Doctors Combined a Heart Pump and Pig Kidney Transplant in Breakthrough Surgery”
Statistics/Meta-Science
-
“Nuclear Fission: Reaction to the Discovery in 1939”, Badash et al 198638ya (what does it look & feel like to be in the early stages of creating an x-risk like nuclear bombs? eg. Nobel physicist Millikan’s assertion that nuclear energy is a “hobgoblin” & men should “sleep in peace with the consciousness that the Creator has put some foolproof elements into his handiwork and that man is powerless to do it any titanic physical damage”—one striking flaw is the repeated reliance on quasi-religious equilibrium/anthropic Outside View arguments, that ‘nuclear risks must be impossible because the Earth and natural ore deposits still exist’, which is irrelevant to the question of the dangers of creating new technologies such as with highly-concentrated ores)
-
“Generalizability of heterogeneous treatment effect estimates across samples”, et al 2018 (rare good news)
-
“An Excess of Positive Results: Comparing the Standard Psychology Literature With Registered Reports”, et al 2021
-
“Truncating Bar Graphs Persistently Misleads Viewers”, et al 2021
Politics/Religion
-
“Good news on climate change” (emissions progress; tail risks overestimated; cost/experience curves underestimated); “Advice to Young People, as You Face Annihilation”, Roger Hallam (co-founder of Extinction Rebellion); “Stop Telling Kids They’ll Die From Climate Change: Many young people feel like their future is in peril. To make progress on climate change, we must move past doomsday scenarios”, Hannah Ritchie (Our World in Data)
-
“Genes, Ideology, and Sophistication”, 2021 (many peoples’ politics are too inchoate to have stable causes like genes or upbringing)
-
“Long-term Health and Social Outcomes in Children and Adolescents Placed in Out-of-Home Care”, et al 2021 (foster homes may be bad)
-
“Moral disciplining: the cognitive and evolutionary foundations of puritanical morality”, et al 2021
Psychology/Biology
-
Molecular recording: “Time-tagged ticker tapes for intracellular recordings”, et al 2021 ( Twitter); “Recording of cellular physiological histories along optically readable self-assembling protein chains”, et al 2021; “Infinite re-reading of single proteins at single-amino-acid resolution using nanopore sequencing”, et al 2021; “Molecular recording of sequential cellular events into DNA”, et al 2021; “Multiplex genomic recording of enhancer and signal transduction activity in mammalian cells”, et al 2021; “A temporally resolved, multiplex molecular recorder based on sequential genome editing”, et al 2021
-
“A Multisite Preregistered Paradigmatic Test of the Ego-Depletion Effect”, et al 2021 (large failure to replicate ego-depletion, strong support for null; as predicted)
-
“Physical principles for scalable neural recording”, Marblestone et al 201311ya; “Can One Concurrently Record Electrical Spikes from Every Neuron in a Mammalian Brain?”, et al 2019; “Tracking neural activity from the same cells during the entire adult life of mice”, et al 2021; “Imaging intact human organs with local resolution of cellular structures using hierarchical phase-contrast tomography”, et al 2021
-
“Computation in the human cerebral cortex uses less than 0.2 watts yet this great expense is optimal when considering communication costs”, 2020 (communication 20
-
“Energy compensation and adiposity in humans”, et al 2021; “The future of weight loss”, Stephan J. Guyenet (on semaglutide); “Anti-obesity drug discovery: advances and challenges”, et al 2021
-
“The dress” cause is probably individual differences in indoor lighting exposure: Lafer-et al 2015 , et al 2015 , et al 2015 , 2017
-
“A pathogenic fungus uses volatiles to entice male flies into fatal matings with infected female cadavers”, et al 2022
-
“Assessing the calorific importance of episodes of human cannibalism in the Paleolithic”, 2017; “How long could the human race survive on only cannibalism?”; “Assuming you’re stranded with no food, but access to advanced surgical equipment, how long could you survive just eating your own severed limbs?”
-
“Blood, sweat, and tears: extraterrestrial regolith biocomposites with in vivo binders”, et al 2021
Technology
-
“Preliminary Study Of The Nuclear Subterrene”, Robinson et al 197153ya (background, and outcome)
-
“The YouTube Revolution in Knowledge Transfer”, Samo Burja
-
“The Halo Drive: Fuel-Free Relativistic Propulsion of Large Masses via Recycled Boomerang Photons”, 2019
-
“Neutron tomography of Van Leeuwenhoek’s microscopes”, et al 2021 (why Van Leeuwenhoek saw ‘animalcules’ before everyone else: a radically-improved microscope kept secret for prestige)
-
“A Brief History of the Index”, I Love Typography
Economics
-
“Experienced well-being rises with income, even above $75,000 per year”, 2021; “The impact of cash transfers on subjective wellbeing and mental health in low-income and middle-income countries: A systematic review and meta-analysis”, et al 2020
-
“Only the Bad Die Young: Restaurant Mortality in the Western US”, 2014 (not as bad as I thought: 17% first-year mortality, median 4.5 years)
Philosophy
-
“Why Philosophers Should Care About Computational Complexity”, 2011
-
“The Psychology of Existential Risk: Moral Judgments about Human Extinction”, et al 2019
-
“Language and thought are not the same thing: evidence from neuroimaging and neurological patients”, 2016
-
“Fundamentally Misunderstanding Visual Perception: Adults’ Belief in Visual Emissions”, Winer et al 200222ya (most children & many college-educated adults believe you see by shooting beams from your eyes & can feel stares; it is difficult to durably correct their intuition)
-
“Laser ablation of human guilt”, et al 2021
Fiction
-
“Shovel Knight Dropcaps: Bitmap Graphics Reimagined as Medieval Woodblock Prints”, Jamie Clarke (custom initials for a game; handsome, and perhaps an under-represented idea? Robert Jordan’s Wheel Of Time was memorable in part for the chapter icons which hinted at the topic & helped the reader keep their place in such long novels; wouldn’t web serials like A Practical Guide To Evil benefit from a set of dropcaps, which could also depict character progression? A few hundred dollars in commissioned art could add a lot of visual flavor!)
Miscellaneous
-
New London School explosion (the worst school disaster in American history)