AI

• “EfficientZero: Mastering Atari Games with Limited Data”⁠, Yeet al2021 (beating humans on ALE-100k/2h by adding self-supervised learning⁠ to MuZero-Reanalyze⁠); “Procedural Generalization by Planning with Self-Supervised World Models”⁠, Anandet al2021 (generalization capabilities of MuZero-Reanalyze: self-supervised learning also leads to new SOTA on ProcGen⁠, w/implicit meta-learning on MetaWorld⁠)

• “DORA: No-Press Diplomacy from Scratch”⁠, Bakhtinet al2021 (previously: Grayet al2020⁠)

• “Monkey Plays Pac-Man with Compositional Strategies and Hierarchical Decision-making”⁠, Yanget al2021 (videos⁠)

• “Fake It Till You Make It: Face analysis in the wild using synthetic data alone”⁠, Woodet al2021 (modern ML-aided CGI pipelines are amazing)

• “Acquisition of Chess Knowledge in AlphaZero”⁠, McGrathet al2021

• “RASP: Thinking Like Transformers”⁠, Weisset al2021 (programming languages in Transformers⁠); “𝑖λ, a family of imaginary programming libraries”⁠ (programming languages calling Transformers)

• ⁠“KaraSinger: Score-Free Singing Voice Synthesis with VQ-VAE using Mel-spectrograms”, Liaoet al2021 (I am easily amused by AI papers with sung abstracts; similar to Jukebox⁠)

• “Percival: Making In-Browser Perceptual Ad Blocking Practical With Deep Learning”⁠, Dinet al2019

• “Optimizing color for camouflage and visibility using deep learning: the effects of the environment and the observer’s visual system”⁠, Fennellet al2019

Matters Of Scale⁠:

• “Open-Ended Learning Leads to Generally Capable Agents”⁠, Open Ended Learning Teamet al2021 (⁠blog⁠)

• “Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model”⁠, Smithet al2022 (blog⁠; 3× GPT-3⁠ & using The Pile⁠, but undertrained, so only modestly better); “Yuan 1.0: Large-Scale Pre-trained Language Model in Zero-Shot and Few-Shot Learning”⁠, Wuet al2021 (⁠media; 245.7b-parameter GPT-3-like Chinese text model)

• “China Has Already Reached Exascale—On Two Separate Systems”⁠ (anonymous source claims 4.4 exaflops FP16; but kept secret? Concerning arms race dynamics if true)

• “A General Language Assistant as a Laboratory for Alignment”⁠, Askellet al2021 (when scaling to 52b, larger models get friendlier faster & learn from rich human preference⁠ data); “Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned”⁠, Ganguliet al2022 (scaling helps RL preference learning, but not other safety)

• “Recursively Summarizing Books with Human Feedback”⁠, Wuet al2021

• “Exploring the Limits of Large Scale Pre-training”⁠, Abnaret al2021

• “FLAN: Finetuned Language Models Are Zero-Shot Learners”⁠, Weiet al2021 (blog⁠)

• “Unsupervised Neural Machine Translation with Generative Language Models Only”⁠, Hanet al2021; “Scaling Laws for Neural Machine Translation”⁠, Ghorbaniet al2021; “Data and Parameter Scaling Laws for Neural Machine Translation”⁠, Gordonet al2021

  • “Training Verifiers to Solve Math Word Problems”⁠, Cobbeet al2021 (blog⁠; boosting GPT-3 on math word problems from ~15% to ~60% by self-distilling a critic & best-of = 100 sampling)

• “ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation”⁠, Sunet al2021 (⁠demo; #1 SuperGLUE⁠, exceeding human baseline & Microsoft Research performance)

• “Structured Denoising Diffusion Models (D3PMs) in Discrete State-Spaces”⁠, Austinet al2021; “Beyond In-Place Corruption: Insertion and Deletion In Denoising Probabilistic Models”⁠, Johnsonet al2021; “Argmax Flows and Multinomial Diffusion: Learning Categorical Distributions”⁠, Hoogeboomet al2021a; “ARDMs: Autoregressive Diffusion Models”⁠, Hoogeboomet al2021b; “SUNDAE: Step-unrolled Denoising Autoencoders for Text Generation”⁠, Savinovet al2021 “Zero-Shot Translation using Diffusion Models”⁠, Nachmani & Dovrat2021; “Diffusion-LM Improves Controllable Text Generation”⁠, Liet al2022 (DDPMs⁠ for text; potentially better global coherency as the entire text is updated repeatedly, in a much more natural way allowing for turning compute into performance & exploring alternatives… I suspect a diffusion text model will be largely immune to the repetition trap (as they indeed turn out to be⁠). Good for top-down story generation? Or useful for anagram/palindrome generation? Anagram generators are useless past 1–2 words due to combinatorial explosion; but a DDPM could handle almost arbitrary sized inputs to reconstruct meaningful sentences. Training noise = shuffling letters. To create: take a target string, random shuffle, and denoise. Do this repeatedly for multiple anagrams of the original target. Can use embeddings to guide denoising towards semantically different or identical anagrams of original. Applications in CS, genomics?)

Genetics

Everything Is Heritable:

• new GIANT n = 5.4m height GWAS explains full SNP heritability of height⁠: “A Saturated Map of Common Genetic Variants Associated with Human Height from 5.4 Million Individuals of Diverse Ancestries”⁠, Yengoet al2022 (media⁠; height becomes the first human GWAS to ‘finish’ by reaching the ceiling of SNP heritability; so much for ‘missing heritability’! It took barely a decade to go from the first height GWASes of n = 4k / 0.3% variance⁠ to complete prediction. Exponentials are a helluva drug.)

  • “Improving GWAS discovery and genomic prediction accuracy in Biobank data”⁠, Orliacet al2021

• “Rare variant contribution to human disease in 281,104 UK Biobank exomes”⁠, Wanget al2021; “Protein-coding repeat polymorphisms strongly shape diverse human phenotypes”⁠, Mukamelet al2021; “The sequences of 150,119 genomes in the UK Biobank”⁠, Halldorssonet al2021

• “Discovery of 42 Genome-Wide Statistically-Significant Loci Associated with Dyslexia”⁠, Doustet al2021

• “Somatic mutation rates scale with lifespan across mammals”⁠, Caganet al2021

• “100,000 Genomes Pilot on Rare-Disease Diagnosis in Health Care—Preliminary Report”⁠, The 100k Genomes Project Pilot Investigators2021 (reminder: we should WGS sequence everyone); “Ultrarapid Nanopore Genome Sequencing in a Critical Care Setting”⁠, Gorzynskiet al2022

• “Predicting skeletal stature using ancient DNA”⁠, Coxet al2021 (sequencing the graveyard)

• “Molecular Support for Heterogonesis Resulting in Sesquizygotic Twinning”⁠, Gabettet al2019 (“maternally identical but chimerically shared 78% of their paternal genome”)

Recent Evolution:

• “A selection pressure landscape for 870 human polygenic traits”⁠, Songet al2021

• “Clonal genome evolution and rapid invasive spread of the marbled crayfish”⁠, Gutekunstet al2018

• “The genetic basis of tail-loss evolution in humans and apes”⁠, Xiaet al2021 (progress towards catgirls, or, Return To Monke? Rudimentary human tails⁠ still appear⁠ sometimes⁠)

Engineering:

• ⁠“How Silicon Valley hatched a plan to turn blood into human eggs: A well-connected startup company [Conception] is trying to rewrite the rules of reproduction”⁠

• ⁠“CRISPR Gene-Editing Experiment Partly Restores Vision In Legally Blind Patients”⁠

• “In a First, Surgeons Attached a Pig Kidney to a Human—and It Worked”⁠ (and then into the abdomen⁠ as well); “Man Receives a Heart From a Genetically Altered Pig”⁠ (unsurprisingly, ultimately failed⁠—but progress); “Surgeons Transplant Pig Kidney Into a Patient, a Medical Milestone: The man continues to improve, doctors said. Organs from genetically engineered pigs one day may make dialysis obsolete”⁠; ⁠“Doctors Combined a Heart Pump and Pig Kidney Transplant in Breakthrough Surgery”⁠

• “A Cure for Type 1 Diabetes? For One Man, It Seems to Have Worked. A new treatment using stem cells that produce insulin has surprised experts and given them hope for the 1.5 million Americans living with the disease”⁠

Statistics/Meta-Science

• “Nuclear Fission: Reaction to the Discovery in 1939”⁠, Badashet al1986 (what does it look & feel like to be in the early stages of creating an x-risk⁠ like nuclear bombs? eg. Nobel physicist Millikan’s⁠ ⁠assertion⁠ that nuclear energy is a “hobgoblin” & men should “sleep in peace⁠ with the consciousness that the Creator has put some foolproof elements into his handiwork and that man is powerless to do it any titanic physical damage”—one striking flaw is the repeated reliance on ⁠quasi-religious⁠ equilibrium/anthropic ⁠Outside View⁠ arguments, that ‘nuclear risks must be impossible because the Earth and natural ore deposits still exist’, which is irrelevant to the question of the dangers of creating new technologies such as with highly-concentrated ores)

• “Generalizability of heterogeneous treatment effect estimates across samples”⁠, Coppocket al2018 (rare good news)

• “An Excess of Positive Results: Comparing the Standard Psychology Literature With Registered Reports”⁠, Scheelet al2021

• “Serendipity: Towards a taxonomy and a theory”⁠, Yaqub2018

• “Truncating Bar Graphs Persistently Misleads Viewers”⁠, Yanget al2021

Politics/Religion

• ⁠“Good news on climate change”⁠ (emissions progress; tail risks overestimated; cost/experience curves⁠ underestimated); ⁠“Advice to Young People, as You Face Annihilation”⁠, Roger Hallam⁠ (co-founder of Extinction Rebellion⁠); ⁠“Stop Telling Kids They’ll Die From Climate Change: Many young people feel like their future is in peril. To make progress on climate change, we must move past doomsday scenarios”⁠, Hannah Ritchie (Our World in Data⁠)

• “Genes, Ideology, and Sophistication”⁠, Kalmoe & Johnson2021 (many peoples’ politics are too inchoate to have stable causes like genes or upbringing)

• “Long-term Health and Social Outcomes in Children and Adolescents Placed in Out-of-Home Care”⁠, Sariaslanet al2021 (foster homes may be bad)

• “Moral disciplining: the cognitive and evolutionary foundations of puritanical morality”⁠, Fitouchiet al2021

Psychology/Biology

• Molecular recording: “Time-tagged ticker tapes for intracellular recordings”⁠, Linet al2021 (Twitter⁠); “Recording of cellular physiological histories along optically readable self-assembling protein chains”⁠, Linghuet al2021; “Infinite re-reading of single proteins at single-amino-acid resolution using nanopore sequencing”⁠, Brinkerhoffet al2021; “Molecular recording of sequential cellular events into DNA”⁠, Lovelesset al2021; “Multiplex genomic recording of enhancer and signal transduction activity in mammalian cells”⁠, Chenet al2021; “A temporally resolved, multiplex molecular recorder based on sequential genome editing”⁠, Choiet al2021

• “A Multisite Preregistered Paradigmatic Test of the Ego-Depletion Effect”⁠, Vohset al2021 (large failure to replicate ego-depletion, strong support for null; as predicted⁠)

• “Physical principles for scalable neural recording”⁠, Marblestoneet al2013; “Can One Concurrently Record Electrical Spikes from Every Neuron in a Mammalian Brain?”⁠, Kleinfeldet al2019; “Tracking neural activity from the same cells during the entire adult life of mice”⁠, Zhaoet al2021; “Imaging intact human organs with local resolution of cellular structures using hierarchical phase-contrast tomography”⁠, Walshet al2021

• “Computation in the human cerebral cortex uses less than 0.2 watts yet this great expense is optimal when considering communication costs”⁠, Levy & Calvert2020 (communication 20× energy of computation; 7 bits/sec/neuron; similar efficiency to rats—humans aren’t special, we just have big brains)

• “Energy compensation and adiposity in humans”⁠, Careauet al2021; ⁠“The future of weight loss”⁠, Stephan J. Guyenet (on semaglutide⁠); “Anti-obesity drug discovery: advances and challenges”⁠, Mülleret al2021

• “The dress”⁠ cause is probably individual differences in indoor lighting exposure: Lafer-Sousaet al2015⁠, Winkleret al2015⁠, Gegenfurtneret al2015⁠, Wallisch2017

• “A pathogenic fungus uses volatiles to entice male flies into fatal matings with infected female cadavers”⁠, Naundrupet al2022

• “Assessing the calorific importance of episodes of human cannibalism in the Paleolithic”⁠, Cole2017; ⁠“How long could the human race survive on only cannibalism?”⁠; “Assuming you’re stranded with no food, but access to advanced surgical equipment, how long could you survive just eating your own severed limbs?”

• “Blood, sweat, and tears: extraterrestrial regolith biocomposites with in vivo binders”⁠, Robertset al2021

Technology

• “Preliminary Study Of The Nuclear Subterrene”⁠, Robinsonet al1971 (⁠background, and outcome)

• ⁠“The YouTube Revolution in Knowledge Transfer”, Samo Burja

• “The Halo Drive: Fuel-Free Relativistic Propulsion of Large Masses via Recycled Boomerang Photons”⁠, Kipping2019

• “Neutron tomography of Van Leeuwenhoek’s microscopes”⁠, Cocquytet al2021 (why Van Leeuwenhoek⁠ saw ‘animalcules’ before everyone else: a radically-improved microscope kept secret for prestige)

• ⁠“A Brief History of the Index”, I Love Typography

• “Better eats: The kitchen of 2020 looks mostly the same as that of 1960_65ya. But what we do in it has changed dramatically, almost entirely for the better—due to a culture of culinary innovation”⁠

• “The Paper of Record Meets an Ephemeral Web: An Examination of Linkrot and Content Drift within The New York Times”⁠, Zittrainet al2021

• The APL programming philosophy⁠

Economics

• “Experienced well-being rises with income, even above $75,000 per year”⁠, Killingsworth2021; ⁠“The impact of cash transfers on subjective wellbeing and mental health in low-income and middle-income countries: A systematic review and meta-analysis”⁠, McGuireet al2020

• “Empirical audit and review and an assessment of evidentiary value in research on the psychological consequences of scarcity”⁠, O’Donnellet al2021

• “Only the Bad Die Young: Restaurant Mortality in the Western US”⁠, Luo & Stark2014 (not as bad as I thought: 17% first-year mortality, median 4.5 years)

Philosophy

• “Why Philosophers Should Care About Computational Complexity”⁠, Aaronson2011

• “The Psychology of Existential Risk: Moral Judgments about Human Extinction”⁠, Schubertet al2019

• “Language and thought are not the same thing: evidence from neuroimaging and neurological patients”⁠, Fedorenko & Varley2016

• “Fundamentally Misunderstanding Visual Perception: Adults’ Belief in Visual Emissions”⁠, Wineret al2002 (most children & many college-educated adults believe you see by shooting beams from your eyes & can feel stares; it is difficult to durably correct their intuition)

• “Laser ablation of human guilt”, Friedet al2021

Fiction

• ⁠“Shovel Knight Dropcaps: Bitmap Graphics Reimagined as Medieval Woodblock Prints”⁠, Jamie Clarke (custom initials⁠ for a game⁠; handsome, and perhaps an under-represented idea? Robert Jordan’s⁠ Wheel Of Time⁠ was memorable in part for the ⁠chapter ⁠icons⁠ which hinted at the topic & helped the reader keep their place in such long novels; wouldn’t web serials like ⁠A Practical Guide To Evil benefit from a set of dropcaps, which could also depict character progression? A few hundred dollars in commissioned art⁠ could add a lot of visual flavor!)

Miscellaneous

• New London School explosion⁠ (the worst school disaster in American history)

• The Pink Panthers⁠ (New Yorker⁠)