October 2020 Gwern.net newsletter with links on AI scaling, Euclid; further site reorganization & improvement.
October 2020’s Gwern.net newsletter is now out; previous, September 2020 (archives). This is a collation of links and summary of major changes, overlapping with my Changelog; brought to you by my donors on Patreon.
Writings
-
more Gwern.net fixes: line-breaking on slashes, hyphenation & justified text on Chrome desktop browsers using Edward Kmett’s
hyphenation
at compile-time; width-full mobile images; rewrote GoodReads conversion for much more readable book review page; browsable directories; popup annotations now render more like body -
compiled all movie & anime reviews
Links
AI
-
“Human-Level Performance in No-Press Diplomacy via Equilibrium Search”, et al 2020
-
“DreamerV2: Mastering Atari with Discrete World Models”, et al 2020
-
“Taxonomy of Real Faults in Deep Learning Systems”, et al 2019 (on the painfully primitive & error-prone tooling of AI research (such a contrast to Lisp machines, where the tooling was superior to today’s DL ecosystem but the AI research was useless)—“neural networks want to work”, so how many serious bugs do we never realize? Every researcher has a story, like R2D2, of how their code had a fatal bug but worked anyway… Just not nearly as well as it should have. Recently in Tensorfork we learned our BigGAN samples didn’t work—because the Batch Norm statistics weren’t being used at runtime like we thought!)
-
“Artificial Intelligence versus Maya Angelou: Experimental evidence that people cannot differentiate AI-generated from human-written poetry”, 2020 (humans preferred screened GPT-2 poem completions 43% of the time)
-
“Tensorflow Neural Network Playground” (explorable: train small NNs in-browser to watch how their decision boundaries evolve)
-
“Scaling Laws for Autoregressive Generative Modeling”, et al 2020 ( Kaplan talk; GPT-3 was not a fluke nor language-specific: all modalities tested—math, video, images, text, combined—scale cleanly and in the same way where bigger models = better; the unsupervised/pretrained models then transfer to supervised learning, like image classification. GPT-3 all the things!)
-
“Vision Transformer (ViT): An Image is Worth 16×16 Words: Transformers for Image Recognition at Scale”, et al 2020 (as expected, pure Transformers surpass CNN SOTA when scaled appropriately: as fast, better, simpler with fewer parameters and cheaper to train, with improvements to come: “Finally, [we plan] to further scale ViT, given that the performance does not seem yet to be saturating with the increased model size…” lucidrains’s implementation)
-
“Meta-trained agents implement Bayes-optimal agents”, et al 2020; “One Epoch Is All You Need”, 2019 (are bigger deep models better because they meta-learn Bayes-optimal inference?)
-
NMT: “mT5: A massively multilingual pre-trained text-to-text transformer”, et al 2020 (13b-parameters); “M2M-100: Beyond English-Centric Multilingual Machine Translation”, et al 2020 ( blog; 15b-parameters, using ZeRO)
-
“Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition”, et al 2020 (pretraining enables scaling to 1b-parameter efficient-attention Conformers)
Genetics:
Everything Is Heritable:
-
“Multivariate genomic analysis of 1.5 million people identifies genes related to addiction, antisocial behavior, and health”, et al 2020; “Happiness and Wellbeing; the value and findings from genetic studies”, Van de et al 2020
-
“Exome sequencing and characterization of 49,960 individuals in the UK Biobank”, Van et al 2020 (“2% of this population has a medically actionable variant”)
-
“Exploring the variance in complex traits captured by DNA methylation assays”, et al 2020 (epigenome heritability/ variance-components: ~0 across 400 traits)
Engineering:
Statistics/Meta-Science
-
“Publication rate in preclinical research: a plea for preregistration”, van der et al 2020 ( media; ~75% of sacrificed animals yield no published result; why do we have IRBs & ‘ethics boards’ supposedly ensuring animal welfare when they cannot even ensure the data animals are killed for gets published?)
-
“The Empirical Metamathematics of Euclid and Beyond”, Stephen Wolfram (analyzing Elements as a DAG); “Singing Euclid: the oral character of Greek geometry”, Viktor Blåsjö (see also “Having Had No Predecessor to Imitate, He Had No Successor Capable of Imitating Him”)
Politics/Religion
-
Immanentizing the Eschaton, Anabaptist edition: Münster rebellion; the wreck of the Batavia
Psychology/Biology
-
“Questions About Trees”, Brian Hayes (“the paradox of the plankton”: what maintains diversity in forests?)
-
“The hearing aid dilemma: amplification, compression, and distortion of the neural code”, et al 2020 (towards hearing aids designed by blackbox optimization to make neural patterns match patterns of normal hearing?)
-
“The role of cat eye narrowing movements in cat-human communication”, et al 2020
-
Encephalitis lethargica (“A viral infection of the mind? The curious case of encephalitis lethargica”)
-
“C60 in olive oil causes light–dependent toxicity and does not extend lifespan in mice”, et al 2020 (Back in 2012, I found the original result unbelievable for many reasons 1; but in case anyone still takes it seriously…)
Technology
-
LibGen/Sci-Hub fulltext search engine: search 5.5m+ downloadable books/77.5m+ papers (great alternative to Google Books/Scholar)
-
“Profiling a warehouse-scale computer”, et al 2015 (previously: et al 2010 )
-
“Expert judgment on markers to deter inadvertent human intrusion into the Waste Isolation Pilot Plant”, et al 1993 (the Sandia long-term nuclear waste warning message expert report for the WIP)
-
“The Elusive Peril of Space Junk: Millions of human artifacts circle the Earth. Can we clean them up before they cause a disaster?” (the Kessler syndrome never went away)
-
“The rise and fall of Adobe Flash: Before Flash Player sunsets this December, we talk its legacy with those who built it”; “How Flash Games Shaped The Video Game Industry: Flash is dead. But the influence of Flash games on modern gameplay is inescapable”, Jonas Richner; “Reliving Flash Game History”, Alex Irpan (why were Flash browser games so creative? The Flash scene benefited from easy entry using an integrated programming environment, small-scale games enabling rapid iteration, many exemplars to imitate, envy and admiration driving imitators, the conviction that improvements were possible, a global talent pool with lots of free time, and game websites like Newgrounds to curate games to create a sophisticated & demanding userbase which could drive mega-traffic to hits, in a virtuous cycle of recruitment/competition/feedback/improvement. See also Bakewell.)
-
“This page is a truly naked, Brutalist HTML quine.”; A self-building website (cf. naked objects)
Fiction
-
The Monkees (from passing references, I was always confused if ‘The Monkees’ were a TV show or band—apparently they were both! Best-selling ones, too.)
-
Aside from extremely strong priors about life extension effects (typically much smaller) and the low quality of animal research, some of the specific problems with the original result included:
-
contradictory median/lifespan figures
-
duplicate image
-
small sample
-
doses of C60 small enough that the direct antioxidant activity can’t be responsible
-
bizarre intervention, whose justifying cites were not published when experiment started
-
the C60 was administered for brief period (‘imagine taking a supplement only during your 40s and doubling your lifespan’)
-
the massive life extension observed in the olive-oil-only rats—not doubling, but still implausible
-