2019 News
Annual summary of 2019 Gwern.net newsletters, selecting my best writings, the best 2019 links by topic, and the best books/movies/anime I saw in 2019, with some general discussion of the year and the 2010s, and an intellectual autobiography of the past decade.
This is the 2019 summary edition of the Gwern.net newsletter (archives), summarizing the best of the monthly 2019 newsletters:
Previous annual newsletters: 2018, 2017, 2016, 2015.
Writings
2019 went well, with much interesting news and several stimulating trips. My 2019 writings included:
-
“Finetuning the GPT-2-117M Transformer for English Poetry Generation”
-
Danbooru2018 released: a dataset of 3.33m anime images (2.5tb) with 92.7m descriptive tags
-
“On Seeing Through ‘On Seeing Through: A Unified Theory’: A Unified Theory”
-
“Dog Cloning For Special Forces: Breed All You Can Breed”/NBA recruiting using height polygenic scores
I’m particularly proud of the technical improvements to the Gwern.net site design this year: along with a host of minor typographic improvements & performance optimizations, Inflation.hs
enables automatic updates of currencies (a feature I’ve long felt would make documents far less misleading), the link annotations/popups (popups.js
) are a major usability enhancement few sites have, sidenotes.js
eliminates the frustration of footnotes by providing sidenotes, collapsible sections help tame long writings by avoiding the need for hiding code or relegating material to appendices, and link icons & dropcaps & epigraphs are just pretty. While changes are never unanimously received, we have received many compliments on the overall design, and are pleased with it.
Site traffic (more detailed breakdown) was again up as compared with the year before: 2019 saw 1,361,195 pageviews by 671,774 unique visitors (lifetime totals: 7,988,362 pageviews by 3,808,776 users). I benefited primarily from TWDNE, although the numbers are somewhat inflated by hosting a number of popular archived pages from DeepDotWeb/OKCupid/Rotten.com, which I put Google Analytics on to keep track of referrals.
Overview
2019 was a fun year.
AI: 2019 was a great year for hobbyists and fun generative projects like mine, thanks to spinoffs and especially pretrained models. How much more boring it would have been without the GPT-2 or StyleGAN models! (There was irritatingly little meaningful news about self-driving cars.) More seriously, the theme of 2019 was scaling. Whether GPT-2 or StyleGAN 1+2, or the scaling papers, or AlphaStar, or MuZero, 2019 demonstrated the blessings of scale in scaling up models, compute, data, and tasks; it is no accident that the most extensively discussed editorial on DL/DRL was Rich Sutton’s “The Bitter Lesson”. For all the critics’ carping and goalpost-moving, scaling is working, especially as we go far past the regimes where they assured us years ago that mere size and compute would break down and we would have to use more elegant and intelligent methods like Bayesian program synthesis. Instead, every year it looks increasingly like the strong connectionist thesis is correct: much like humans & evolution, AGI can be reached by training an extremely large number of relatively simple units end-to-end for a long time on a wide variety of multimodal tasks, and it will recursively self-improve meta-learning efficient internal structures & algorithms optimal for the real world which learns how to generalize, reason, self-modify with internal learned reward proxies & optimization algorithms, and do zero/few-shot learning bootstrapped purely from the ultimate reward signals—without requiring extensive hand-engineering, hardwired specialized modules designed to support symbolic reasoning, completely new paradigms of computing hardware etc. (eg. 2019). Self-driving cars remain a bitter mystery (although it was nice to see in 2019 Elon Musk & Tesla snatch victory from the jaws of snatching-defeat-from-the-jaws-of-victory).
2019 for genetics saw more progress on genetic-engineering topics than GWASes; the GWASes that did come out were largely confirmatory—no one really needed more SES GWASes from Hill et al, or confirmation that the IQ GWASes work and that brain size is in fact causal for intelligence, and while the recovery of full height/BMI trait heritability from WGS is a strong endorsement of the long-term value of WGS, the transition from limited SNP data to WGS is foreordained (especially since WGS costs appear to finally be dropping again after their long stagnation). Even embryo selection saw greater mainstream acceptance, with a paper in Cell concluding (for the crudest possible simple embryo selection methods) that, fortunately, the glass was half-empty and need not be feared overmuch. (I am pleased to see that as of 2019, human geneticists have managed to reinvent 1918/1943’s breeder’s equation; with any luck, in a few years they may progress as far as reinventing 1943.) More interesting were the notable events along all axis of post-simple-embryo-selection strategies: Genomic Prediction claimed to have done the first embryo selection on multiple PGSes, genome synthesis saw E. coli achieved, multiple promising post-CRISPR or mass CRISPR editing methods were announced, gene drive progressed to mammals, gametogenesis saw progress (including at least two human fertility startups I know of), serious proposals for human germline CRISPR editing are being made by a Russian (among others), and while He Jiankui was imprisoned by a secret court there otherwise do not appear to have been serious repercussions such as reports of the 3 CRISPR babies being harmed or an ‘indefinite moratorium’ (ie. ban). Thus, we saw good progress towards the enabling technologies for massive embryo selection (breaking the egg bottleneck by allowing generation of hundreds or thousands of embryos and thus multiple-SD gains from selection), IES (Iterated Embryo Selection), massive embryo editing (CRISPR or derivatives), and genome synthesis.
VR’s 2019 launch of Oculus Quest proved quite successful, selling out occasionally well after launch, and appears to appeal to normal people, with even hardcore VR fans acknowledging how much they appreciate the convenience of a single integrated unit. Unfortunately… it is not successful enough. There is no VR wave. Selling out may have as much to do with Facebook not investing too much into manufacturing. Worse, there is still no killer app beyond Beat Saber. The hardware is adequate to the job, the price is nugatory, the experience unparalleled, but there is no stampede into VR. So it seems VR is doomed to the long slow multi-decade adoption slog like that of PCs: it’s too new, too different, and we’re still not sure what to do with it. One day, it would not be surprising if most people have a VR headset, but that day is a long way away.
Bitcoin: little of note. Darknet markets proved unusually interesting: Dream Market, the longest-lived DNM ever, finally expired; Reddit betrayed its users by wholesale purging of subreddits, including /r/DarkNetMarkets
, causing me a great deal of grief; and most shockingly, DeepDotWeb was raided by the FBI over affiliate commissions it received from DNMs (apparently into the tens of millions of dollars—you’d’ve thought they’d taken down those hideous ads all over DDW if the affiliate links were so profitable…)
Media
Links
Genetics:
-
Everything Is Heritable:
-
“Recovery of [full height/BMI] trait heritability from whole genome sequence data”, et al 2019
-
“Genetic analysis identifies molecular systems and biological pathways associated with household income”, et al 2019a; “Genome-wide analysis identifies molecular systems and 149 genetic loci associated with income”, et al 2019b
-
“Comparing within. and between-family polygenic score prediction”, et al 2019
-
“Large-scale GWAS reveals insights into the genetic architecture of same-sex sexual behavior”, et al 2019
-
“Extreme inbreeding in a European ancestry sample from the contemporary UK population”, et al 2019
-
-
Engineering:
-
full genome synthesis of E. coli: 4 million base-pairs; “Technological challenges and milestones for writing genomes: Synthetic genomics requires improved technologies”, et al 2019
-
“Extensive Mammalian Germline Genome Engineering”, et al 2019; “Enabling large-scale genome editing by reducing DNA nicking”, et al 2019
-
“Functional Oocytes Derived from Granulosa Cells”, et al 2019; “Controlled modeling of human epiblast and amnion development using stem cells”, et al 2019
-
“Synthetic glycolate metabolism pathways stimulate crop growth and productivity in the field”, et al 2019
-
“Super-Mendelian inheritance mediated by CRISPR-Cas9 in the female mouse germline”, et al 2019
-
“Livestock 2.0—genome editing for fitter, healthier, and more productive farmed animals”, Tait-et al 2018
-
Recent Evolution:
-
“Highly Heritable and Functionally Relevant Breed Differences in Dog Behavior”, Macet al 2019; “Absolute brain size predicts dog breed differences in executive function”, et al 2019; “Breed differences of heritable behavior traits in cats”, et al 2019
-
“Evolutionary Perspectives on Genetic and Environmental Risk Factors for Psychiatric Disorders”, 2018
-
“Thinking About the Evolution of Complex Traits in the Era of Genome-Wide Association Studies”, 2019
-
“Genetic contributions to variation in human stature in prehistoric Europe”, et al 2019
AI:
-
“The Bitter Lesson”, 2019
-
Language models: “Better Language Models and Their Implications”, OpenAI 2019 (“Language Models are Unsupervised Multitask Learners”, et al 2019)
-
“EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks”, 2019
-
“Adversarial Examples Are Not Bugs, They Are Features”, et al 2019 ( discussion)
-
MuseNet: large Sparse Transformers for MIDI music generation (et al 2019 )
-
-
“MuZero: Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model”, et al 2019
-
“Grandmaster level in StarCraft II using multi-agent reinforcement learning”, et al 2019 (blog: [1](https://deepmind.google/discover/blog/alphastar-mastering-the-real-time-strategy-game-starcraft-ii/ “AlphaStar: Mastering the Real-Time Strategy Game StarCraft II”/)/ 2)
-
“Dactyl: Solving Rubik’s Cube With A Robot Hand”, et al 2019 ( blog)
-
“Dota 2 with Large Scale Deep Reinforcement Learning”, et al 2019 ( blog)
-
‘Pluribus’: “Superhuman AI for multiplayer poker”, 2019
-
“Meta-learning of Sequential Strategies”, et al 2019; “Reinforcement Learning, Fast and Slow”, et al 2019; “Meta-learners’ learning dynamics are unlike learners’”, 2019; “Ray Interference: a Source of Plateaus in Deep Reinforcement Learning”, et al 2019
-
On “Meta Reinforcement Learning”, Lilian Weng
-
“Learning to Predict Without Looking Ahead: World Models Without Forward Prediction”, et al 2019
-
“ICML 2019 Notes”, David Abel
-
Statistics/meta-science:
-
“Most Rigorous Large-Scale Educational RCTs are Uninformative: Should We Be Concerned?”, Lortie-2019
-
“Meta-Research: A comprehensive review of randomized clinical trials in three medical journals reveals 396 [13%] medical reversals”, Herrera-et al 2019
-
“Comparing meta-analyses and preregistered multiple-laboratory replication projects”, et al 2019
-
“The Big Crunch”, David Goodstein 199430ya; “1960: The Year The Singularity Was Cancelled”, Scott Alexander
-
“We Need a New Science of Progress: Humanity needs to get better at knowing how to get better”, Patrick Collison & Tyler Cowen
-
“Ingredients for creating disruptive research teams”, Stefan Torges
-
“Evidence on good forecasting practices from the Good Judgment Project”
-
“The maddening saga of how an Alzheimer’s ‘cabal’ thwarted progress toward a cure for decades”; amyloid does not even predict cognitive decline
-
“Registered reports: an early example and analysis”, et al 2019
-
“bioRxiv: the preprint server for biology”, et al 2019
Politics/religion:
-
“The Real War 1939–6194579ya”, Paul Fussell 198935ya; “Why Men Love War”, 1984
-
“How Gullible are We? A Review of the Evidence from Psychology and Social Science”, 2017
-
“Aztec Political Thought”; “Nggwal, who travels in structures of fiber and bone atop rivers of blood.”
-
“Anthropology’s Science Wars: Insights from a New Survey”, et al 2019 ( visualizations)
Psychology:
-
“The impact of background music on adult listeners: A meta-analysis”, et al 2011
-
“Debunking the Stanford Prison Experiment”, Le 2019; Did psychologist David Rosenhan fabricate his famous 197351ya “Being Sane in Insane Places” mental hospital exposé?; “A real-life Lord of the Flies: the troubling legacy of the Robbers Cave experiment”
-
“What the voice inside your head says about you: We tend to assume that our internal monologue ‘speaks’ in words—but it turns out that, for many of us, it’s much more complicated” (see also “Toward a phenomenology of inner speaking”, Hurlburt et al 201311ya)
-
“Doing Despite Disliking: Self-regulatory Strategies in Everyday Aversive Activities”, et al 2018 (see also Structured Procrastination)
-
“Enjoy It Again: Repeat Experiences Are Less Repetitive Than People Think”, 2019
Biology:
-
“Restoration of brain circulation and cellular functions hours post-mortem”, et al 2019
-
“A Look Back at 2019: Progress Towards the Treatment of Aging as a Medical Condition”, Reason
-
“Widespread associations between grey matter structure and the human phenome”, Couvy-et al 2019
-
“Invisible Designers: Brain Evolution Through the Lens of Parasite Manipulation”, Del 2019
Technology:
-
“STEPS Toward Expressive Programming Systems: ‘A Science Experiment’”, et al 2012
-
“How can we develop transformative tools for thought?”, Andy Matuschak & Michael Nielsen 2019
-
“Measuring Long-term Impact of Ads on LinkedIn Feed”, et al 2019
Economics:
-
“Roads and Bridges: The Unseen Labor Behind Our Digital Infrastructure”, Nadia 2016
-
“The changing structure of American innovation: Some cautionary remarks for economic growth”, et al 2020
-
“End-To-End Arguments In System Design”, Saltzer et al 198440ya; “Observations on Errors, Corrections, & Trust of Dependent Systems”, James Hamilton (the end-to-end principle)
-
“Labour repression—the Indo-Japanese divergence”, Pseudoerasmus
-
“The Great Shift in Japanese Pop Culture—Part One”/Part Two/Part Three/Part Four/Part Five
-
“Managing an iconic old luxury brand in a new luxury economy: Hermès handbags in the US market”, 2014 (summary; see also “Birkin Demand: A Sage & Stylish Investment”, 2016)
-
“The Price of Nails since 1700324ya: Even Simple Products Experienced Large Price Declines”, 2017
-
“The Corporate Governance of Benedictine Abbeys: What can Stock Corporations Learn from Monasteries?”, et al 2010
Philosophy:
-
“Two Arms and a Head: The Death of a Newly Paraplegic Philosopher”, 2008
-
“Possible Girls”, 2008
-
“Consider the Lobster: For 56 years, the Maine Lobster Festival has been drawing crowds with the promise of sun, fun, and fine food. One visitor would argue that the celebration involves a whole lot more”, David Foster Wallace 2004
Fiction:
-
RIP Gene Wolfe
-
“A Colder War”, Charles 2000
-
“The Proverbial Murder Mystery”, SSC, SSC (see also “The Study of Anglophysics”)
-
“A science fiction writer of the Fifties”, Brad Leithauser 2006
-
“Killing Rabbits”, Miroslav 1963
-
“The First Sally (A), or, Trurl’s Electronic Bard”, Stanisław Lem
-
“Neon Genesis Evangelion: Graphic designer Peiran Tan plumbs the typographic psyche of the celebrated anime franchise”; Typography in Alien & Blade Runner
Books
Nonfiction:
TV/movies
Nonfiction movies:
Fiction:
-
Invasion of the Body Snatchers, 197846ya (review)
-
Rurouni Kenshin 201212ya/Rurouni Kenshin: Kyoto Inferno 2014/Rurouni Kenshin: The Legend Ends 2014 (review)
Anime:
-
Neon Genesis Evangelion Concurrency Project, 201311ya (review)