AI

• “EfficientZero: Mastering Atari Games with Limited Data”, Ye et al 2021 (beating humans on ALE-100k/2h by adding self-supervised learning to MuZero-Reanalyze); “Procedural Generalization by Planning with Self-Supervised World Models”, Anand et al 2021 (generalization capabilities of MuZero-Reanalyze: self-supervised learning also leads to new SOTA on ProcGen, w/implicit meta-learning on MetaWorld)

• “DORA: No-Press Diplomacy from Scratch”, Bakhtin et al 2021 (previously: Gray et al 2020)

• “Monkey Plays Pac-Man with Compositional Strategies and Hierarchical Decision-making”, Yang et al 2021 (videos)

• “Fake It Till You Make It: Face analysis in the wild using synthetic data alone”, Wood et al 2021 (modern ML-aided CGI pipelines are amazing)

• “Acquisition of Chess Knowledge in AlphaZero”, McGrath et al 2021

• “RASP: Thinking Like Transformers”, Weiss et al 2021 (programming languages in Transformers); “𝑖λ, a family of imaginary programming libraries” (programming languages calling Transformers)

• “KaraSinger: Score-Free Singing Voice Synthesis with VQ-VAE using Mel-spectrograms”, Liao et al 2021 (I am easily amused by AI papers with sung abstracts; similar to Jukebox)

• “Percival: Making In-Browser Perceptual Ad Blocking Practical With Deep Learning”, Din et al 2019

• “Optimizing color for camouflage and visibility using deep learning: the effects of the environment and the observer’s visual system”, Fennell et al 2019

Matters Of Scale:

• “Open-Ended Learning Leads to Generally Capable Agents”, Open Ended Learning Team et al 2021 (blog)

• “Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model”, Smith et al 2022 (blog; 3× GPT-3 & using The Pile, but undertrained, so only modestly better); “Yuan 1.0: Large-Scale Pre-trained Language Model in Zero-Shot and Few-Shot Learning”, Wu et al 2021 (media; 245.7b-parameter GPT-3-like Chinese text model)

• “China Has Already Reached Exascale—On Two Separate Systems” (anonymous source claims 4.4 exaflops FP16; but kept secret? Concerning arms race dynamics if true)

• “A General Language Assistant as a Laboratory for Alignment”, Askell et al 2021 (when scaling to 52b, larger models get friendlier faster & learn from rich human preference data); “Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned”, Ganguli et al 2022 (scaling helps RL preference learning, but not other safety)

• “Recursively Summarizing Books with Human Feedback”, Wu et al 2021

• “Exploring the Limits of Large Scale Pre-training”, Abnar et al 2021

• “FLAN: Finetuned Language Models Are Zero-Shot Learners”, Wei et al 2021 (blog)

• “Unsupervised Neural Machine Translation with Generative Language Models Only”, Han et al 2021; “Scaling Laws for Neural Machine Translation”, Ghorbani et al 2021; “Data and Parameter Scaling Laws for Neural Machine Translation”, Gordon et al 2021

  • “Training Verifiers to Solve Math Word Problems”, Cobbe et al 2021 (blog; boosting GPT-3 on math word problems from ~15% to ~60% by self-distilling a critic & best-of = 100 sampling)

• “ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation”, Sun et al 2021 (demo; #1 SuperGLUE, exceeding human baseline & Microsoft Research performance)

• “Structured Denoising Diffusion Models (D3PMs) in Discrete State-Spaces”, Austin et al 2021; “Beyond In-Place Corruption: Insertion and Deletion In Denoising Probabilistic Models”, Johnson et al 2021; “Argmax Flows and Multinomial Diffusion: Learning Categorical Distributions”, Hoogeboom et al 2021a; “ARDMs: Autoregressive Diffusion Models”, Hoogeboom et al 2021b; “SUNDAE: Step-unrolled Denoising Autoencoders for Text Generation”, Savinov et al 2021 “Zero-Shot Translation using Diffusion Models”, Nachmani & Dovrat2021; “Diffusion-LM Improves Controllable Text Generation”, Li et al 2022 (DDPMs for text; potentially better global coherency as the entire text is updated repeatedly, in a much more natural way allowing for turning compute into performance & exploring alternatives… I suspect a diffusion text model will be largely immune to the repetition trap (as they indeed turn out to be). Good for top-down story generation? Or useful for anagram/palindrome generation? Anagram generators are useless past 1–2 words due to combinatorial explosion; but a DDPM could handle almost arbitrary sized inputs to reconstruct meaningful sentences. Training noise = shuffling letters. To create: take a target string, random shuffle, and denoise. Do this repeatedly for multiple anagrams of the original target. Can use embeddings to guide denoising towards semantically different or identical anagrams of original. Applications in CS, genomics?)

Genetics

Everything Is Heritable:

• new GIANT n = 5.4m height GWAS explains full SNP heritability of height: “A Saturated Map of Common Genetic Variants Associated with Human Height from 5.4 Million Individuals of Diverse Ancestries”, Yengo et al 2022 (media; height becomes the first human GWAS to ‘finish’ by reaching the ceiling of SNP heritability; so much for ‘missing heritability’! It took barely a decade to go from the first height GWASes of n = 4k / 0.3% variance to complete prediction. Exponentials are a helluva drug.)

  • “Improving GWAS discovery and genomic prediction accuracy in Biobank data”, Orliac et al 2021

• “Rare variant contribution to human disease in 281,104 UK Biobank exomes”, Wang et al 2021; “Protein-coding repeat polymorphisms strongly shape diverse human phenotypes”, Mukamel et al 2021; “The sequences of 150,119 genomes in the UK Biobank”, Halldorsson et al 2021

• “Discovery of 42 Genome-Wide Statistically-Significant Loci Associated with Dyslexia”, Doust et al 2021

• “Somatic mutation rates scale with lifespan across mammals”, Cagan et al 2021

• “100,000 Genomes Pilot on Rare-Disease Diagnosis in Health Care—Preliminary Report”, The 100k Genomes Project Pilot Investigators2021 (reminder: we should WGS sequence everyone); “Ultrarapid Nanopore Genome Sequencing in a Critical Care Setting”, Gorzynski et al 2022

• “Predicting skeletal stature using ancient DNA”, Cox et al 2021 (sequencing the graveyard)

• “Molecular Support for Heterogonesis Resulting in Sesquizygotic Twinning”, Gabett et al 2019 (“maternally identical but chimerically shared 78% of their paternal genome”)

Recent Evolution:

• “A selection pressure landscape for 870 human polygenic traits”, Song et al 2021

• “Clonal genome evolution and rapid invasive spread of the marbled crayfish”, Gutekunst et al 2018

• “The genetic basis of tail-loss evolution in humans and apes”, Xia et al 2021 (progress towards catgirls, or, Return To Monke? Rudimentary human tails still appear sometimes)

Engineering:

• “How Silicon Valley hatched a plan to turn blood into human eggs: A well-connected startup company [Conception] is trying to rewrite the rules of reproduction”

• “CRISPR Gene-Editing Experiment Partly Restores Vision In Legally Blind Patients”

• “In a First, Surgeons Attached a Pig Kidney to a Human—and It Worked” (and then into the abdomen as well); “Man Receives a Heart From a Genetically Altered Pig” (unsurprisingly, ultimately failed—but progress); “Surgeons Transplant Pig Kidney Into a Patient, a Medical Milestone: The man continues to improve, doctors said. Organs from genetically engineered pigs one day may make dialysis obsolete”; “Doctors Combined a Heart Pump and Pig Kidney Transplant in Breakthrough Surgery”

• “A Cure for Type 1 Diabetes? For One Man, It Seems to Have Worked. A new treatment using stem cells that produce insulin has surprised experts and given them hope for the 1.5 million Americans living with the disease”

Statistics/Meta-Science

• “Nuclear Fission: Reaction to the Discovery in 1939”, Badash et al 1986 (what does it look & feel like to be in the early stages of creating an x-risk like nuclear bombs? eg. Nobel physicist Millikan’s assertion that nuclear energy is a “hobgoblin” & men should “sleep in peace with the consciousness that the Creator has put some foolproof elements into his handiwork and that man is powerless to do it any titanic physical damage”—one striking flaw is the repeated reliance on quasi-religious equilibrium/anthropic Outside View arguments, that ‘nuclear risks must be impossible because the Earth and natural ore deposits still exist’, which is irrelevant to the question of the dangers of creating new technologies such as with highly-concentrated ores)

• “Generalizability of heterogeneous treatment effect estimates across samples”, Coppock et al 2018 (rare good news)

• “An Excess of Positive Results: Comparing the Standard Psychology Literature With Registered Reports”, Scheel et al 2021

• “Serendipity: Towards a taxonomy and a theory”, Yaqub2018

• “Truncating Bar Graphs Persistently Misleads Viewers”, Yang et al 2021

Politics/Religion

• “Good news on climate change” (emissions progress; tail risks overestimated; cost/experience curves underestimated); “Advice to Young People, as You Face Annihilation”, Roger Hallam (co-founder of Extinction Rebellion); “Stop Telling Kids They’ll Die From Climate Change: Many young people feel like their future is in peril. To make progress on climate change, we must move past doomsday scenarios”, Hannah Ritchie (Our World in Data)

• “Genes, Ideology, and Sophistication”, Kalmoe & Johnson2021 (many peoples’ politics are too inchoate to have stable causes like genes or upbringing)

• “Long-term Health and Social Outcomes in Children and Adolescents Placed in Out-of-Home Care”, Sariaslan et al 2021 (foster homes may be bad)

• “Moral disciplining: the cognitive and evolutionary foundations of puritanical morality”, Fitouchi et al 2021

Psychology/Biology

• Molecular recording: “Time-tagged ticker tapes for intracellular recordings”, Lin et al 2021 (Twitter); “Recording of cellular physiological histories along optically readable self-assembling protein chains”, Linghu et al 2021; “Infinite re-reading of single proteins at single-amino-acid resolution using nanopore sequencing”, Brinkerhoff et al 2021; “Molecular recording of sequential cellular events into DNA”, Loveless et al 2021; “Multiplex genomic recording of enhancer and signal transduction activity in mammalian cells”, Chen et al 2021; “A temporally resolved, multiplex molecular recorder based on sequential genome editing”, Choi et al 2021

• “A Multisite Preregistered Paradigmatic Test of the Ego-Depletion Effect”, Vohs et al 2021 (large failure to replicate ego-depletion, strong support for null; as predicted)

• “Physical principles for scalable neural recording”, Marblestone et al 2013; “Can One Concurrently Record Electrical Spikes from Every Neuron in a Mammalian Brain?”, Kleinfeld et al 2019; “Tracking neural activity from the same cells during the entire adult life of mice”, Zhao et al 2021; “Imaging intact human organs with local resolution of cellular structures using hierarchical phase-contrast tomography”, Walsh et al 2021

• “Computation in the human cerebral cortex uses less than 0.2 watts yet this great expense is optimal when considering communication costs”, Levy & Calvert2020 (communication 20× energy of computation; 7 bits/sec/neuron; similar efficiency to rats—humans aren’t special, we just have big brains)

• “Energy compensation and adiposity in humans”, Careau et al 2021; “The future of weight loss”, Stephan J. Guyenet (on semaglutide); “Anti-obesity drug discovery: advances and challenges”, Müller et al 2021

• “The dress” cause is probably individual differences in indoor lighting exposure: Lafer-Sousa et al 2015, Winkler et al 2015, Gegenfurtner et al 2015, Wallisch2017

• “A pathogenic fungus uses volatiles to entice male flies into fatal matings with infected female cadavers”, Naundrup et al 2022

• “Assessing the calorific importance of episodes of human cannibalism in the Paleolithic”, Cole2017; “How long could the human race survive on only cannibalism?”; “Assuming you’re stranded with no food, but access to advanced surgical equipment, how long could you survive just eating your own severed limbs?”

• “Blood, sweat, and tears: extraterrestrial regolith biocomposites with in vivo binders”, Roberts et al 2021

Technology

• “Preliminary Study Of The Nuclear Subterrene”, Robinson et al 1971 (background, and outcome)

• “The YouTube Revolution in Knowledge Transfer”, Samo Burja

• “The Halo Drive: Fuel-Free Relativistic Propulsion of Large Masses via Recycled Boomerang Photons”, Kipping2019

• “Neutron tomography of Van Leeuwenhoek’s microscopes”, Cocquyt et al 2021 (why Van Leeuwenhoek saw ‘animalcules’ before everyone else: a radically-improved microscope kept secret for prestige)

• “A Brief History of the Index”, I Love Typography

• “Better eats: The kitchen of 2020 looks mostly the same as that of 1960_65ya. But what we do in it has changed dramatically, almost entirely for the better—due to a culture of culinary innovation”

• “The Paper of Record Meets an Ephemeral Web: An Examination of Linkrot and Content Drift within The New York Times”, Zittrain et al 2021

• The APL programming philosophy

Economics

• “Experienced well-being rises with income, even above $75,000 per year”, Killingsworth2021; “The impact of cash transfers on subjective wellbeing and mental health in low-income and middle-income countries: A systematic review and meta-analysis”, McGuire et al 2020

• “Empirical audit and review and an assessment of evidentiary value in research on the psychological consequences of scarcity”, O’Donnell et al 2021

• “Only the Bad Die Young: Restaurant Mortality in the Western US”, Luo & Stark2014 (not as bad as I thought: 17% first-year mortality, median 4.5 years)

Philosophy

• “Why Philosophers Should Care About Computational Complexity”, Aaronson2011

• “The Psychology of Existential Risk: Moral Judgments about Human Extinction”, Schubert et al 2019

• “Language and thought are not the same thing: evidence from neuroimaging and neurological patients”, Fedorenko & Varley2016

• “Fundamentally Misunderstanding Visual Perception: Adults’ Belief in Visual Emissions”, Winer et al 2002 (most children & many college-educated adults believe you see by shooting beams from your eyes & can feel stares; it is difficult to durably correct their intuition)

• “Laser ablation of human guilt”, Fried et al 2021

Fiction

• “Shovel Knight Dropcaps: Bitmap Graphics Reimagined as Medieval Woodblock Prints”, Jamie Clarke (custom initials for a game; handsome, and perhaps an under-represented idea? Robert Jordan’s Wheel Of Time was memorable in part for the chapter icons which hinted at the topic & helped the reader keep their place in such long novels; wouldn’t web serials like A Practical Guide To Evil benefit from a set of dropcaps, which could also depict character progression? A few hundred dollars in commissioned art could add a lot of visual flavor!)

Miscellaneous

• New London School explosion (the worst school disaster in American history)

• The Pink Panthers (New Yorker)