AI

• “Human-Level Performance in No-Press Diplomacy via Equilibrium Search”⁠, Grayet al2020

• “DreamerV2: Mastering Atari with Discrete World Models”⁠, Hafneret al2020

• “Taxonomy of Real Faults in Deep Learning Systems”⁠, Humbatovaet al2019 (on the painfully primitive & error-prone tooling of AI research (such a contrast to Lisp machines⁠, where the tooling was superior to today’s DL ecosystem but the AI research was useless)—“neural networks want to work”, so how many serious bugs do we never realize? Every researcher has a story, like R2D2⁠, of how their code had a fatal bug but worked anyway… Just not nearly as well as it should have. Recently in Tensorfork we learned our BigGAN samples didn’t work—because the Batch Norm statistics weren’t being used at runtime like we thought!)

• “Artificial Intelligence versus Maya Angelou: Experimental evidence that people cannot differentiate AI-generated from human-written poetry”⁠, Köbis & Mossink2020 (humans preferred screened GPT-2⁠ poem completions 43% of the time)

• “Tensorflow Neural Network Playground”⁠ (explorable: train small NNs in-browser to watch how their decision boundaries evolve)

Matters Of Scale⁠:

• “Scaling Laws for Autoregressive Generative Modeling”⁠, Henighanet al2020 (⁠Kaplan talk⁠; GPT-3⁠ was not a fluke nor language-specific: all modalities tested—math, video, images, text, combined—scale cleanly and in the same way where bigger models = better; the unsupervised/pretrained models then transfer to supervised learning, like image classification. GPT-3 all the things!)

• “Vision Transformer (ViT): An Image is Worth 16×16 Words: Transformers for Image Recognition at Scale”⁠, Anonymouset al2020 (as expected, pure Transformers surpass CNN⁠ SOTA when scaled appropriately: as fast, better, simpler with fewer parameters and cheaper to train, with improvements to come: “Finally, [we plan] to further scale ViT, given that the performance does not seem yet to be saturating with the increased model size…” lucidrains’s implementation⁠)

• “Meta-trained agents implement Bayes-optimal agents”⁠, Mikuliket al2020; “One Epoch Is All You Need”⁠, Komatsuzaki2019 (are bigger deep models better because they ⁠meta-learn Bayes-optimal inference⁠?)

• NMT: “mT5: A massively multilingual pre-trained text-to-text transformer”⁠, Xueet al2020 (13b-parameters); “M2M-100: Beyond English-Centric Multilingual Machine Translation”⁠, Fanet al2020 (blog⁠; 15b-parameters, using ZeRO)

• “Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition”⁠, Zhanget al2020 (pretraining enables scaling to 1b-parameter efficient-attention⁠ Conformers⁠)

Genetics:

Everything Is Heritable:

• “Multivariate genomic analysis of 1.5 million people identifies genes related to addiction, antisocial behavior, and health”⁠, Linneret al2020; “Happiness and Wellbeing; the value and findings from genetic studies”⁠, Van de Weijeret al2020

• “Combined Utility of 25 Disease and Risk Factor Polygenic Risk Scores for Stratifying Risk of All-Cause Mortality”⁠, Meisneret al2020

• “Exome sequencing and characterization of 49,960 individuals in the UK Biobank”⁠, Van Houtet al2020 (“2% of this population has a medically actionable variant”)

• “Exploring the variance in complex traits captured by DNA methylation assays”⁠, Battramet al2020 (epigenome heritability/variance-components⁠: ~0 across 400 traits)

Engineering:

• “The second decade of synthetic biology: 2010^–₁₀2020”⁠, Meng & Ellis2020

• Emmanuelle Charpentier⁠ & Jennifer Doudna⁠ win 2020 Nobel Prize for⁠ CRISPR⁠

Statistics/Meta-Science

• “Publication rate in preclinical research: a plea for preregistration”⁠, van der Naaldet al2020 (media⁠; ~75% of sacrificed animals yield no published result; why do we have IRBs & ‘ethics boards’ supposedly ensuring animal welfare when they cannot even ensure the data animals are killed for gets published?)

• “The Empirical Metamathematics of Euclid and Beyond”, Stephen Wolfram (analyzing Elements⁠ as a DAG); “Singing Euclid: the oral character of Greek geometry”, Viktor Blåsjö (see also “Having Had No Predecessor to Imitate, He Had No Successor Capable of Imitating Him”)

Politics/Religion

• Immanentizing the Eschaton, Anabaptist⁠ edition: Münster rebellion⁠; the wreck of the Batavia⁠

Psychology/Biology

• “Effects of caloric restriction on human physiological, psychological, and behavioral outcomes: highlights from CALERIE phase 2”⁠, Dorlinget al2020

• “Questions About Trees”, Brian Hayes (“the paradox of the plankton”: what maintains diversity in forests?)

• “The hearing aid dilemma: amplification, compression, and distortion of the neural code”⁠, Armstronget al2020 (towards hearing aids designed by blackbox optimization to make neural patterns match patterns of normal hearing?)

• “The role of cat eye narrowing movements in cat-human communication”⁠, Humphreyet al2020

• Encephalitis lethargica⁠ (“A viral infection of the mind? The curious case of encephalitis lethargica”⁠)

• “Brain Training Habits Are Not Associated With Generalized Benefits to Cognition: An Online Study of Over 1000 ‘Brain Trainers’”⁠, Stojanoskiet al2020

• “C₆₀ in olive oil causes light–dependent toxicity and does not extend lifespan in mice”⁠, Grohnet al2020 (Back in 2012_13ya, I found the original result unbelievable for many reasons⁠^⁠1⁠; but in case anyone still takes it seriously…)

Technology

• LibGen/Sci-Hub fulltext search engine: search 5.5m+ downloadable books/77.5m+ papers (great alternative to Google Books/Scholar)

• “Profiling a warehouse-scale computer”⁠, Kanevet al2015 (previously: Renet al2010⁠)

• “Expert judgment on markers to deter inadvertent human intrusion into the Waste Isolation Pilot Plant”⁠, Trauthet al1993 (the Sandia⁠ long-term nuclear waste warning message⁠ expert report for the WIP⁠)

• “The Elusive Peril of Space Junk: Millions of human artifacts circle the Earth. Can we clean them up before they cause a disaster?”⁠ (the Kessler syndrome⁠ never went away)

• “The rise and fall of Adobe Flash: Before Flash Player sunsets this December, we talk its legacy with those who built it”⁠; “How Flash Games Shaped The Video Game Industry: Flash is dead. But the influence of Flash games on modern gameplay is inescapable”, Jonas Richner; ⁠“Reliving Flash Game History”⁠, Alex Irpan (why were Flash⁠ browser games⁠ so creative? The Flash scene benefited from easy entry using an integrated programming environment, small-scale games enabling rapid iteration, many exemplars to imitate, envy and admiration driving imitators, the conviction that improvements were possible, a global talent pool with lots of free time, and game websites like Newgrounds⁠ to curate games to create a sophisticated & demanding userbase which could drive mega-traffic to hits, in a virtuous cycle of recruitment/competition/feedback/improvement. See also Bakewell⁠.)

• “This page is a truly naked, Brutalist HTML quine.”; A self-building website (cf. naked objects⁠)

Fiction

• “My Dad, the Pornographer”⁠ (on Andrew J. Offutt⁠)

• The Monkees⁠ (from passing references, I was always confused if ‘The Monkees’ were a TV show or band—apparently they were both! Best-selling ones, too.)