May 2020 Gwern.net newsletter: GPT-3 scaling, implications, deep theory; anime GAN updates, and 1 book review.
May 2020’s Gwern.net newsletter is now out; previous, April 2020 (archives). This is a collation of links and summary of major changes, overlapping with my Changelog; brought to you by my donors on Patreon.
Writings
-
Ganbooru prototype: released:
256px BigGAN trained on Danbooru2019 Danbooru2019 Figures dataset
-
Gwern.net:
-
experimental
<srcset>
mobile image optimization (since removed) -
popups.js
: +support for reverse-footnote popups
Mailing List Switch
The newsletter moved this month to Substack due to reaching the TinyLetter 5000-subscriber limit. Please let me know of any issues beyond the known issue of length truncation. (Note that reading the website version on desktop is the recommended way for annotations etc.)
-
On GPT-3: Meta-Learning, Scaling, Implications, And Deep Theory
On “GPT-3: Language Models are Few-Shot Learners”, et al 2020 (poems & my followup GPT-3 Creative Writing, compare my old finetuned GPT-2 poetry; random samples; “OpenAI API” with real-world demos)
Media
Links
AI:
-
Matters Of Scale:
-
GPT-3: see above; for GPT-3 compared to humans on the absolute scale of character prediction, see Scaling Hypothesis, footnote 18
-
“Measuring the Algorithmic Efficiency of Neural Networks”, 2020 (blog/interview; the first prototype is never the best one, but given enough compute & time, you can refine it and figure out how it should have been done all along, and this paper quantifies the neural net hardware overhang just since 201212ya: “it now takes 44× less compute to train…to the level of AlexNet”. Unsurprising—eg the experience curve in linear programming: 2002; see 2013/2013. We don’t know how to train the right kind of neural nets and make huge mistakes with the simplest things, as capability jumps like resnets or EfficientNet or R2D2 occasionally remind us.)
-
“IntelliCode Compose: Code Generation Using [GPT-2] Transformer”, et al 2020 (unclear if application of ZeRO-2; see also the GPT-3 few-shot code completion abilities)
-
“GrokNet: Unified Computer Vision Model Trunk and Embeddings For Commerce” (blog; one model, 7 datasets, 89m images, 83 losses/tasks, and +8% search quality boost worldwide)
-
-
“Deep neuroethology of a virtual rodent”, et al 2019 ( media)
-
“Go-Explore 2: First return then explore”, et al 2021
-
“Learning to Simulate Dynamic Environments with GameGAN”, et al 2020 ( project page, code; an unexpected appearance of a Neural Turing Machine)
-
“Exploring Bayesian Optimization: Breaking Bayesian Optimization into small, sizeable chunks”, 2020
-
“This Word Does Not Exist” (GPT-2); “This Fursona Does Not Exist (TFDNE)” editor (a simple but high-quality StyleGAN 2 face model of furries, also available on Artbreeder; interesting for how the fur flew due to legal fuzziness & some artists acting like animals, howling about ‘theft’ & free fursonas being a wolf in sheep’s clothing upsetting their pecking order1—though the creator has outfoxed the paper tiger threats, these kittlesome questions will dog ML as DL models multiply like rabbits)
Genetics:
-
Everything Is Heritable:
-
“Local genetic correlation analysis reveals heterogeneous etiologic sharing of complex traits”, et al 2020 (“autism/IQ rg…could be explained by 2 etiologically-distinct genetic signatures w/bidirectional local genetic correlations”)
-
“Genome-wide association study of school grades identifies a genetic overlap between language ability, psychopathology and creativity”, et al 2020 (“math performance was severely affected whereas language performance (Danish and English) was relatively unaffected or enhanced in those with psychiatric disorders”)
-
“Insights into the genetic architecture of the human face”, et al 2020
-
-
Recent Evolution:
-
“Sex-biased reduction in reproductive success drives selective constraint on human genes”, et al 2020; “Genome-wide analysis identifies genetic effects on reproductive success and ongoing natural selection at the FADS locus”, et al 2020 (previously: et al 2016 /et al 2016 /et al 2017 )
-
“Disentangling selection on genetically correlated polygenic traits using whole-genome genealogies”, et al 2020
-
-
Engineering:
Statistics/Meta-Science:
-
“Variability in the analysis of a single neuroimaging dataset by many teams”, Botvinik-et al 2020
-
“Remembering John Conway’s FRACTRAN, a ridiculous, yet surprisingly deep language”, Reginald Braithwaite (how does the recently-deceased John Conway’s 1980 esolang lead to the Collatz conjecture?)
-
“Tumbling toast, Murphy’s Law and the fundamental constants”, 1995 (overview; anthropics size argument from 1980; see also et al 2001 /2012)
Politics/religion:
-
“The Voluntariness of Voluntary Consent: Consent Searches and the Psychology of Compliance”, 2019 (people are bad at predicting resistance to police requests; see also et al 2012 )
-
Operation INFEKTION (see also 1997)
-
“Progress Studies for Aspiring Young Scholars” (experimental online summer class for high school students by Jason Crawford on development)
Psychology/biology:
-
“Understanding immunity through the lens of disease ecology”, 20172 (“…for the past few thousand years, we human beings have been the most diseased species on earth”; followup to 2004; commentary)
-
“How sanitation conquered disease long before vaccines or antibiotics”, Jason Crawford
-
“Everyday Life as an Intelligence Test: Effects of Intelligence and Intelligence Context”, 1997
-
“Objective and subjective experiences of child maltreatment and their relationships with psychopathology”, 2020 (nothing in psychology makes sense except in the light of individual-differences)
-
“Brainless but Multi-Headed: Decision Making by the Acellular Slime Mould Physarum polycephalum”, 2015
-
“I’m paid biweekly, just not by leprechauns: Evaluating valid-but-incorrect response rates to attention check items”, 2019 (how do “lizardman constant” responders justify it? Or, ‘free response is the devil’)
Technology:
-
“Reflections on How Designers Design with Data”, et al 2014 (why are data visualizations so bad—superficially pretty but misleading or useless? Because many designers don’t look at the data, avoid automation & create manually so they can focus on pretty shapes/colors & enjoying fiddling with it, and ignore readers)
-
“Do Ads Harm News Consumption?”, et al 2020 (“Users who adopt ad blockers subsequently consume 20% more news articles corresponding to 10% more categories. The effect persists over time…”; see my ad page)
Economics:
-
“In Ohio, the Amish Take On the Coronavirus” (supply and demand: masks can be easily made anywhere if prices are allowed to rise & they are not illegal to sell)
-
“The Story of America’s Most Prolific Counterfeiter” (how Frank Bourassa tricked a Swiss mill into selling him the unique U.S. dollar linen-paper to create $344.05$2502012m in perfect counterfeit money & mostly got away with it)
Fiction:
Misc:
Books
Fiction:
-
The Battle Between the Frogs and the Mice: A Tiny Homeric Epic, translated 2009 (review)
Nonfiction:
Music
-
“Sept Jours sans Elle (Vocal)” (Raven’s Jig; Une Semaine chez les Écarlates {2018}) [classical]
-
“Un Jour Joueur” (Raven’s Jig; Une Semaine chez les Écarlates {2018}) [classical]
-
“Bons et mauvais Jours” (Raven’s Jig; Une Semaine chez les Écarlates {2018}) [classical]
MLP:
-
“Morning in Baltimare” (Mane in Green; II. The Journey [The Quest of the Lost Sapphire—Ep. 2] {2017}) [instrumental rock]
-
“Love and Reflection” (Dionte George; Ignite {2020}) [jazz]
-
“Second Prances (Vocal VIP)” (Etherium Apex ft. Nicole Carino {2020}) [electronic]
-
“Spun” (The Wasteland Wailers feat. Brittany Church & Haymaker; Ignite {2020}) [country]
-
“Equiterian Empire” (Carbon Maestro; Celestial Divide OST) [orchestral]
-
“The Storm Is Coming VIP [Single Purpose Remix]” (UndreamedPanic feat. Metajoker; Ignite {2020}) [rock]
-
“Mare Cognitum” (Idyllia feat. Velvet R. Wings; Ignite {2020}) [orchestral rock]
-
“Fire City (Day & Night)” (Wandering Artist; Ignite {2020}) [orchestral]
-
“What Remains” (Totalspark; Ignite {2020}) [Liquid Drum & Brass]
Doujin:
-
“Come, Sweet Death [Komm, süsser Tod]” (Platina Jazz feat. Niklas Gabrielsson; Anime Standards Vol. 6 {2019}) [jazz]
-
“Hope” (Simpsonill {2017}) [electronic]
-
Don’t worry: we already have short-shorts & ear-TIPS to hedge against fursona inflation. That said, we advise taking a large position in equineties image macro funds to benefit from a flight to quality and herding: it’ll be a bear market for kinky bonds—and that’s no bull.↩︎
-
Some interesting references on viral evolution:
-
Coevolution Of Virulence:
-
Experimental Epidemiology, et al 1936 ( editorial)
-
“Population biology of infectious diseases: Part I”/“Part II”, Anderson & May 1979
-
“Coevolution of hosts and parasites”, Anderson & May 1982
-
-
-
“History of Sabin attenuated poliovirus oral live vaccine strains”, 1973 (making Sabin’s polio vaccine by dozens of passages through monkeys & monkey tissues)
-