AI

• “Human-Level Performance in No-Press Diplomacy via Equilibrium Search”, Gray et al 2020
• “DreamerV2: Mastering Atari with Discrete World Models”, Hafner et al 2020
• “Taxonomy of Real Faults in Deep Learning Systems”, Humbatova et al 2019 (on the painfully primitive & error-prone tooling of AI research (such a contrast to Lisp machines⁠, where the tooling was superior to today’s DL ecosystem but the AI research was useless)—“neural networks want to work”, so how many serious bugs do we never realize? Every researcher has a story, like R2D2⁠, of how their code had a fatal bug but worked anyway… Just not nearly as well as it should have. Recently in Tensorfork we learned our BigGAN samples didn’t work—because the Batch Norm statistics weren’t being used at runtime like we thought!)
• “Artificial Intelligence versus Maya Angelou: Experimental evidence that people cannot differentiate AI-generated from human-written poetry”, Köbis & Mossink2020 (humans preferred screened GPT-2 poem completions 43% of the time)
• “Tensorflow Neural Network Playground” (explorable: train small NNs in-browser to watch how their decision boundaries evolve)

Matters Of Scale:

• “Scaling Laws for Autoregressive Generative Modeling”, Henighan et al 2020 (Kaplan talk⁠; GPT-3 was not a fluke nor language-specific: all modalities tested—math, video, images, text, combined—scale cleanly and in the same way where bigger models = better; the unsupervised/​pretrained models then transfer to supervised learning, like image classification. GPT-3 all the things!)
• “Vision Transformer (ViT): An Image is Worth 16×16 Words: Transformers for Image Recognition at Scale”, Anonymous et al 2020 (as expected, pure Transformers surpass CNN SOTA when scaled appropriately: as fast, better, simpler with fewer parameters and cheaper to train, with improvements to come: “Finally, [we plan] to further scale ViT, given that the performance does not seem yet to be saturating with the increased model size…” lucidrains’s implementation)
• “Meta-trained agents implement Bayes-optimal agents”, Mikulik et al 2020; “One Epoch Is All You Need”, Komatsuzaki2019 (are bigger deep models better because they meta-learn Bayes-optimal inference?)
• NMT: “mT5: A massively multilingual pre-trained text-to-text transformer”, Xue et al 2020 (13b-parameters); “M2M-100: Beyond English-Centric Multilingual Machine Translation”, Fan et al 2020 (blog⁠; 15b-parameters, using ZeRO)
• “Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition”, Zhang et al 2020 (pretraining enables scaling to 1b-parameter efficient-attention Conformers)

Genetics:

Everything Is Heritable:

• “Multivariate genomic analysis of 1.5 million people identifies genes related to addiction, antisocial behavior, and health”, Linner et al 2020; “Happiness and Wellbeing; the value and findings from genetic studies”, Van de Weijer et al 2020
• “Combined Utility of 25 Disease and Risk Factor Polygenic Risk Scores for Stratifying Risk of All-Cause Mortality”, Meisner et al 2020
• “Exome sequencing and characterization of 49,960 individuals in the UK Biobank”, Van Hout et al 2020 (“2% of this population has a medically actionable variant”)
• “Exploring the variance in complex traits captured by DNA methylation assays”, Battram et al 2020 (epigenome heritability/​variance-components: ~0 across 400 traits)

Engineering:

• “The second decade of synthetic biology: 2010–2020”, Meng & Ellis2020
• Emmanuelle Charpentier & Jennifer Doudna win 2020 Nobel Prize for CRISPR

Statistics/​Meta-Science

• “Publication rate in preclinical research: a plea for preregistration”, van der Naald et al 2020 (media⁠; ~75% of sacrificed animals yield no published result; why do we have IRBs & ‘ethics boards’ supposedly ensuring animal welfare when they cannot even ensure the data animals are killed for gets published?)
• “The Empirical Metamathematics of Euclid and Beyond”⁠, Stephen Wolfram (analyzing Elements as a DAG); “Singing Euclid: the oral character of Greek geometry”⁠, Viktor Blåsjö (see also “Having Had No Predecessor to Imitate, He Had No Successor Capable of Imitating Him”)

Politics/​Religion

• Immanentizing the Eschaton, Anabaptist edition: Münster rebellion⁠; the wreck of the Batavia

Psychology/​Biology

• “Effects of caloric restriction on human physiological, psychological, and behavioral outcomes: highlights from CALERIE phase 2”, Dorling et al 2020
• “Questions About Trees”⁠, Brian Hayes (“the paradox of the plankton”: what maintains diversity in forests?)
• “The hearing aid dilemma: amplification, compression, and distortion of the neural code”, Armstrong et al 2020 (towards hearing aids designed by blackbox optimization to make neural patterns match patterns of normal hearing?)
• “The role of cat eye narrowing movements in cat-human communication”, Humphrey et al 2020
• Encephalitis lethargica (“A viral infection of the mind? The curious case of encephalitis lethargica”)
• “Brain Training Habits Are Not Associated With Generalized Benefits to Cognition: An Online Study of Over 1000 ‘Brain Trainers’”, Stojanoski et al 2020
• “C₆₀ in olive oil causes light–dependent toxicity and does not extend lifespan in mice”, Grohn et al 2020 (Back in 2012, I found the original result unbelievable for many reasons¹⁠; but in case anyone still takes it seriously…)

Technology

• LibGen / ​Sci-Hub fulltext search engine: search 5.5m+ downloadable books/​77.5m+ papers (great alternative to Google Books/​Scholar)
• “Profiling a warehouse-scale computer”, Kanev et al 2015 (previously: Ren et al 2010)
• “Expert judgment on markers to deter inadvertent human intrusion into the Waste Isolation Pilot Plant”, Trauth et al 1993 (the Sandia long-term nuclear waste warning message expert report for the WIP)
• “The Elusive Peril of Space Junk: Millions of human artifacts circle the Earth. Can we clean them up before they cause a disaster?” (the Kessler syndrome never went away)
• “The rise and fall of Adobe Flash: Before Flash Player sunsets this December, we talk its legacy with those who built it”⁠; “How Flash Games Shaped The Video Game Industry: Flash is dead. But the influence of Flash games on modern gameplay is inescapable”⁠, Jonas Richner; “Reliving Flash Game History”⁠, Alex Irpan (why were Flash browser games so creative? The Flash scene benefited from easy entry using an integrated programming environment, small-scale games enabling rapid iteration, many exemplars to imitate, envy and admiration driving imitators, the conviction that improvements were possible, a global talent pool with lots of free time, and game websites like Newgrounds to curate games to create a sophisticated & demanding userbase which could drive mega-traffic to hits, in a virtuous cycle of recruitment/​competition/​feedback/​improvement. See also Bakewell⁠.)
• “This page is a truly naked, Brutalist HTML quine.”⁠; A self-building website

Fiction

• “My Dad, the Pornographer” (on Andrew J. Offutt)
• The Monkees (from passing references, I was always confused if ‘The Monkees’ were a TV show or band—apparently they were both! Best-selling ones, too.)