AI

• “Autonomous navigation of stratospheric balloons using reinforcement learning”, Bellemare et al 2020

• “Image Generators with Conditionally-Independent Pixel Synthesis”, Anokhin et al 2020 (what a weird architecture—fully-connected neural nets, how do they work‽)

• “A Review of Skeb.jp—An Artwork Commissioning Website” (JA ↔︎ EN deep learning translation DeepL-powered website for Westerners commissioning Japanese artists: >100k requests thus far; example of Brynjolfsson et al 2019; broader discussion: Weber 2020)

• “Object-based attention for spatio-temporal reasoning: Outperforming neuro-symbolic models with flexible distributed architectures”, Ding et al 2020 (“DL can’t do reasoning”; reminds me of Santoro et al 2017; the bitter lesson?—if “neurosymbolic” types want their work to be relevant in even 5 years, they’d do better to focus on creating datasets which will induce symbolic reasoning eg. if you want understanding of common noun weights, scrape Amazon & dump metadata of millions of listings formatted with shipping weight at the end (forcing a flexible understanding of titles/descriptions, quantities, and special-cases like fragility or danger), or convert knowledge graph databases like OpenCyc/ Wikidata to text to distill them into a NN model. The future belongs to those who show up—because their content is still available online for the AI to read…)

Matters Of Scale:

• “Draft report on AI timelines”, Ajeya Cotra

• “How Much Computational Power Does It Take to Match the Human Brain?”, Carlsmith 2020

• “Imitating Interactive Intelligence”, Interactive Agents Group 2020 (DM blog; “With each doubling of the dataset size, performance grew by approximately the same increment…agents trained to imitate human action and language demonstrate powerful combinatorial generalisation capabilities.” DeepMind gets closer to the goal of matching a mouse.¹)

• Exotic training paradigms:

  • On DFA: “Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures”, Launay et al 2020a; “Hardware Beyond Backpropagation: a Photonic Co-Processor for Direct Feedback Alignment”, Launay et al 2020b

  • Graphcore: “Parallel Training of Deep Networks with Local Updates”, Laskin et al 2020

  • Cerebras: “Pipelined Backpropagation at Scale: Training Large Models without Batches”, Kosson et al 2020

• “The Pile: An 800GB Dataset of Diverse Text for Language Modeling”, EleutherAI 2020

• “Extrapolating GPT-N performance”, Lukas Finnveden

• “CPM (Chinese Pre-trained Language Model): A Large-scale Generative Chinese Pre-trained Language Model”, Zhang et al 2020 (GPT-2.6b trained on 100GB; checkpoint released)

Genetics

Everything Is Heritable:

• “Rare Genetic Variation Underlying Human Diseases and Traits: Results from 200,000 Individuals in the UK Biobank”, Jurgens et al 2020 (sequence everyone!)

• “Ten years of enhancing neuro-imaging genetics through meta-analysis: An overview from the ENIGMA Genetics Working Group”, Medland et al 2020

• “High-throughput, low-cost and rapid DNA sequencing using surface-coating techniques”, Qin et al 2020 (BGI claims $18.87^$15₂₀₂₀ WGS reagent cost? big if true, but must also compete with even cheaper alternatives like imputation)

Engineering:

• “Reprogramming to recover youthful epigenetic information and restore vision [in mice]”, Lu et al 2020 (media; thesis)

• “Reconstitution of the oocyte transcriptional network with transcription factors”, Hamazaki et al 2020 (media)

• “Biotechnology Research Viewed With Caution Globally, but Most Support Gene Editing for Babies To Treat Disease”, Pew Research poll (international, Dec 2019–March 2020); “Acceptance of genetic editing and of whole genome sequencing of human embryos by patients with infertility before and after the onset of the COVID-19 pandemic”, Neuhausser et al 2023

Statistics/Meta-Science

• “Is the rate of scientific progress slowing down?”, Cowen & Southwood 2019

• “The statistical properties of RCTs and a proposal for shrinkage”, van Zwet et al 2020

• “The Busy Beaver Frontier”, Aaronson 2019 (media)

• “How To Shuffle A Big Dataset” (implementing the Rao-Sandelius external-memory algorithm, for when data doesn’t fit in-RAM for a Fisher-Yates shuffle)

• “Did people really drink bleach to prevent COVID-19? A tale of problematic respondents and a guide for measuring rare events in survey data”, Litman et al 2020 (lizardman constant)

Psychology/Biology

• “Digital Voicing of Silent Speech”, Gaddy & Klein 2020 (“Our method greatly improves intelligibility of audio generated from silent EMG compared to a baseline that only trains with vocalized data, decreasing transcription word error rate from 64% to 4%”); “Real-time Synthesis of Imagined Speech Processes from Minimally Invasive Recordings of Neural Activity”, Angrick et al 2020; “Thinking ahead: prediction in context as a keystone of language in humans and machines”, Goldstein et al 2020 (GPT-2); “Adversarial images for the primate brain”, Yuan et al 2020 (“Remarkably, the same images fooled monkeys and humans at the behavioral level. These results challenge fundamental assumptions about the similarity between computer and primate vision…”; samples make me slightly queasy; this is better than Bashivan et al 2018 because it manipulates high-level classifications rather than neural population activations, and better than Elsayed et al 2018 because not time-limited)

• “Applying insights from magic to improve deception in research: The Swiss cheese model”, Olson & Raz2020 (see also Macknik et al 2008)

• Common phenomena: Intrusive thoughts (eg. “the call of the void”); trypophobia

• Marbled crayfish

Technology

• “The Physics of Space War: How Orbital Dynamics Constrain Space-to-Space Engagements”, Reesman & Wilson 2020 (media)

• Microtypography: “Breaking paragraphs into lines”, Knuth & Plass 1981_45ya (bonus history of polyglot Bibles); “Micro-typographic extensions to the T_eX typesetting system”, Thành 2000

• “Noise in the landscape: Disputing the visibility of mundane technological objects”, Fukushima 2020 (the esthetics of power-lines & poles in anime; eg. “Wired Sky”)

• “IP over Avian Carriers”

Philosophy

• “The Timing of Evolutionary Transitions Suggests Intelligent Life Is Rare”, Snyder-Beattie et al 2020

Fiction

• “A Secret Vice”, Tolkien 1931_95ya (on conlanging; see also “The Subcreation Theory of J. R. R. Tolkien”, Yohn 2021, and machines as satanic/parasitic mockery of creation)

• “What to Make of Rod McKuen?” (revisiting the now-forgotten pop poet of the masses; another example of childhood abuse spurring an insatiable need for achievement, made-up if need be); more interesting than modern examples like Rupi Kaur

• “Goodreads plans to retire API access, disables existing API keys” (get your data out while you still can; I have stopped posting my reviews to GR in favor of my own page)