December 2020 Gwern.net newsletter with links on AI and technology; major new site feature: fully-generalized recursive popups.
December 2020’s Gwern.net newsletter is now out; previous, November 2020 (archives). This is a collation of links and summary of major changes, overlapping with my Changelog; brought to you by my donors on Patreon.
Writings
Links
AI
-
“Autonomous navigation of stratospheric balloons using reinforcement learning”, et al 2020
-
“Image Generators with Conditionally-Independent Pixel Synthesis”, et al 2020 (what a weird architecture— fully-connected neural nets, how do they work‽)
-
“A Review of Skeb.jp—An Artwork Commissioning Website” (JA ↔︎ EN deep learning translation DeepL-powered website for Westerners commissioning Japanese artists: >100k requests thus far; example of et al 2019 ; broader discussion: 2020)
-
“Object-based attention for spatio-temporal reasoning: Outperforming neuro-symbolic models with flexible distributed architectures”, et al 2020 (“DL can’t do reasoning”; reminds me of et al 2017 ; the bitter lesson?—if “neurosymbolic” types want their work to be relevant in even 5 years, they’d do better to focus on creating datasets which will induce symbolic reasoning eg. if you want understanding of common noun weights, scrape Amazon & dump metadata of millions of listings formatted with shipping weight at the end (forcing a flexible understanding of titles/descriptions, quantities, and special-cases like fragility or danger), or convert knowledge graph databases like OpenCyc/ Wikidata to text to distill them into a NN model. The future belongs to those who show up—because their content is still available online for the AI to read…)
-
“Draft report on AI timelines”, Ajeya Cotra
-
“How Much Computational Power Does It Take to Match the Human Brain?”, 2020
-
“Imitating Interactive Intelligence”, Interactive Agents 2020 (DM blog; “With each doubling of the dataset size, performance grew by approximately the same increment…agents trained to imitate human action and language demonstrate powerful combinatorial generalisation capabilities.” DeepMind gets closer to the goal of matching a mouse.1)
-
Exotic training paradigms:
-
On DFA: “Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures”, et al 2020a; “Hardware Beyond Backpropagation: a Photonic Co-Processor for Direct Feedback Alignment”, et al 2020b
-
Graphcore: “Parallel Training of Deep Networks with Local Updates”, et al 2020
-
Cerebras: “Pipelined Backpropagation at Scale: Training Large Models without Batches”, et al 2020
-
-
“The Pile: An 800GB Dataset of Diverse Text for Language Modeling”, EleutherAI 2020
-
“Extrapolating GPT-N performance”, Lukas Finnveden
-
“CPM (Chinese Pre-trained Language Model): A Large-scale Generative Chinese Pre-trained Language Model”, et al 2020 ( GPT-2.6b trained on 100GB; checkpoint released)
Genetics
Everything Is Heritable:
-
“Rare Genetic Variation Underlying Human Diseases and Traits: Results from 200,000 Individuals in the UK Biobank”, et al 2020 (sequence everyone!)
-
“High-throughput, low-cost and rapid DNA sequencing using surface-coating techniques”, et al 2020 ( BGI claims $15 WGS reagent cost? big if true, but must also compete with even cheaper alternatives like imputation)
Engineering:
-
“Reprogramming to recover youthful epigenetic information and restore vision [in mice]”, et al 2020 ( media; thesis)
-
“Reconstitution of the oocyte transcriptional network with transcription factors”, et al 2020 ( media)
-
“Biotechnology Research Viewed With Caution Globally, but Most Support Gene Editing for Babies To Treat Disease”, Pew Research poll (international, Dec 2019–March 2020); “Acceptance of genetic editing and of whole genome sequencing of human embryos by patients with infertility before and after the onset of the COVID-19 pandemic”, et al 2023
Statistics/Meta-Science
-
“The statistical properties of RCTs and a proposal for shrinkage”, van et al 2020
-
“The Busy Beaver Frontier”, Aaronson 2019 (media)
-
“How To Shuffle A Big Dataset” (implementing the Rao-Sandelius external-memory algorithm, for when data doesn’t fit in-RAM for a Fisher-Yates shuffle)
-
“Did people really drink bleach to prevent COVID-19? A tale of problematic respondents and a guide for measuring rare events in survey data”, et al 2020 ( lizardman constant)
Psychology/Biology
-
“Digital Voicing of Silent Speech”, 2020 (“Our method greatly improves intelligibility of audio generated from silent EMG compared to a baseline that only trains with vocalized data, decreasing transcription word error rate from 64% to 4%”); “Real-time Synthesis of Imagined Speech Processes from Minimally Invasive Recordings of Neural Activity”, et al 2020; “Thinking ahead: prediction in context as a keystone of language in humans and machines”, et al 2020 (GPT-2); “Adversarial images for the primate brain”, et al 2020 ( “Remarkably, the same images fooled monkeys and humans at the behavioral level. These results challenge fundamental assumptions about the similarity between computer and primate vision…”; samples make me slightly queasy; this is better than et al 2018 because it manipulates high-level classifications rather than neural population activations, and better than et al 2018 because not time-limited)
-
“Applying insights from magic to improve deception in research: The Swiss cheese model”, 2020 (see also et al 2008 )
-
Common phenomena: Intrusive thoughts (eg. “the call of the void”); trypophobia
Technology
-
“The Physics of Space War: How Orbital Dynamics Constrain Space-to-Space Engagements”, 2020 (media)
-
Microtypography: “Breaking paragraphs into lines”, 1981 (bonus history of polyglot Bibles); “Micro-typographic extensions to the TeX typesetting system”, 2000
-
“Noise in the landscape: Disputing the visibility of mundane technological objects”, 2020 (the esthetics of power-lines & poles in anime; eg. “Wired Sky”)
Philosophy
-
“The Timing of Evolutionary Transitions Suggests Intelligent Life Is Rare”, Snyder-et al 2020
Fiction
-
“A Secret Vice”, 1931 (on conlanging; see also “The Subcreation Theory of J.R.R. Tolkien”, 2021)
-
“What to Make of Rod McKuen?” (revisiting the now-forgotten pop poet of the masses; another example of childhood abuse spurring an insatiable need for achievement, made-up if need be); more interesting than modern examples like Rupi Kaur
-
“Goodreads plans to retire API access, disables existing API keys” (get your data out while you still can; I have stopped posting my reviews to GR in favor of my own page)
-
The consequences of creating a mouse-scale brain—as opposed to a ‘mouse-in-a-can’—will be weird: even AI experts have a poor imagination and tend to imagine AI minds as either glorified calculators or humans-in-a-robot-suit, rather than trying to imagine a diverse Technium richer than biological ecosystems filled with minds with bizarre patterns of weaknesses & strengths (GPT-3/AI-Dungeon or AlphaGo/AlphaStar/OA5 ‘delusions’ are just the start), evolving at a computer pace, filling & creating new niches in a global economy.
“The street finds its own use for things”—I can’t predict what somebody in China will have done with my anime GANs last year I don’t yet know about, how can anyone hope to predict what everyone can do with a mind decades from now? If your vision of the future is not weird & disturbing (and orthogonal to current culture wars), you aren’t trying.↩︎