July 2021 Gwern.net newsletter with links on TODO
July 2021’s Gwern.net newsletter is now out; previous, June 2021 (archives). This is a collation of links and summary of major changes, overlapping with my Changelog; brought to you by my donors on Patreon.
Writings
-
Gwern.net: “link tags” system created
Links
AI
-
“Highly accurate protein structure prediction with AlphaFold 2”, et al 2021 ( source code)
-
“Predictive Coding: a Theoretical and Experimental Review”, et al 2021
-
“ETA Prediction with Graph Neural Networks in Google Maps”, Derrow-et al 2021 (impressive that NNs can produce such gains on such mature systems—where’s your “AI winter” now?)
-
SHRDLU didn’t work (Cyc also never worked—you ever hear of any use for it? cf. 1991)
-
“Fine-tuned Language Models are Continual Learners”, et al 2022
-
“Building Machine Translation Systems for the Next Thousand Languages”, et al 2022
-
“Scaling Vision Transformers”, et al 2021 (testing w/n = 0.03–3b & parameters 0.05–2b using an upgraded JFT-300M dataset, ‘JFT-3B’, confirms the scaling laws: larger models continue to perform much better, more sample-efficient, incredible few-shot performance, & weight decay still useful; ImageNet top-1 classification is approaching saturation, like top-5 before it)
-
“Scaling Laws for Acoustic Models”, 2021 (smooth scaling of Transformer vs LSTM audio models for log-Mel → next-word prediction; Transformers scale better: 33× per halving vs 63×)
-
“NaturalProver: Grounded Mathematical Proof Generation with Language Models”, et al 2022
-
“CPM-2: Large-scale Cost-effective Pre-trained Language Models”, et al 2021 (11b-dense/198b MoE Zh+En; models have been released—largest publicly-available autoregressive English models as of June 2021)
-
Tabular DL: “Regularization is all you Need: Simple Neural Nets can Excel on Tabular Data”, et al 2021 (or if you prefer, et al 2021 , et al 2021 , et al 2021 , et al 2022 —the last ML holdout, tabular, starts to fall to the Bitter Lesson of Transformers & carefully-regularized MLPs—although those heavily invested in tree approaches will doubtless fight to the bitter end, saying this or that hyperparameter optimization or hand-feature-engineering would reestablish some slight superiority, between this and earlier tabular NNs, the writing is on the wall, and reportedly many industry tabular-ML users have moved to DL already for its other benefits)
-
“A Low-latency Communication Design for Brain Simulations”, et al 2022 (benchmarking 4k GPUs simulating 20b spiking neurons on Kunlun)
-
“Jurassic-1: Technical Details And Evaluation”, et al 2021 ( GPT-3-178b model+public API offered by new startup AI21 Labs; demos; this completes the question of GPT-3 time-lags: it took ~11 months for replication by a competitor, and ~14 months for replication+public-API)
-
Big if true: Tesla AI 2021: “Tesla Dojo’s Unique Packaging and Chip Design Allow An Order Magnitude Advantage Over Competing AI Hardware” (“Each tile has an impressive 9 PFlops of BF16/CFP8 and 36 TB/s of off-tile bandwidth. This far surpasses the off-wafer bandwidth of Cerebras, and enables the Tesla system to scale out better than even scale out designs such as the Tenstorrent architecture.”; SemiAnalysis criticism)
Genetics
Everything Is Heritable:
-
“A Global Nucleic Acid Observatory for Biodefense and Planetary Health”, The Nucleic Acid Observatory 2021 (Twitter)
-
“Complex feline disease mapping using a dense genotyping array”, et al 2021 (a rare cat GWAS!)
-
“Using DNA to predict behavior problems from preschool to adulthood”, et al 2021; “Identification of 370 genetic loci for age at first sex and birth linked to externalising behavior”, et al 2021
Recent Evolution:
-
“25 years of selection for improved leg health in purebred broiler lines and underlying genetic parameters”, et al 2012
-
“De novo evolution of macroscopic multicellularity”, et al 2021 ( Twitter; the multicellularity Long Term Evolution Experiment)
-
Virophage (“a flea / Hath smaller fleas that on him prey; / And these have smaller fleas to bite ’em…”)
Engineering:
-
“Gene-drive suppression of mosquito populations in large cages as a bridge between lab and field”, et al 2021
-
“Generation of ovarian follicles from mouse pluripotent stem cells”, et al 2021
-
“Successful Serial Recloning in the Mouse over Multiple Generations”, et al 2013 (“>500 viable offspring from 1 original donor mouse”)
-
“RNA demethylation increases the yield and biomass of rice and potato plants in field trials”, et al 2021
Statistics/Meta-Science/Math
-
“Three Months in Monte Carlo” (interactive explorables of Ising model algorithms & emergent behavior, and the history of Monte Carlo algorithms)
-
“John W. Tukey: His Life and Professional Contributions”, 2002
-
“Causal and Associational Linking Language From Observational Research and Health Evaluation Literature in Practice: A systematic language evaluation”, et al 2021 ( Twitter; researchers interpret most correlational language as implying causality)
-
“MegaLMM: Mega-scale linear mixed models for genomic predictions with thousands of traits”, et al 2021 (cf. variance components)
Politics/Religion
-
“The Dark Side of Private Ordering: An Institutional and Empirical Analysis of Organized Crime”, 2000
-
“‘He who pays the piper calls the tune’: Researcher experiences of funder suppression of health behavior intervention trial findings”, et al 2021 (“Our results, along with those of previous investigations, suggest that government funders interfere with public-good research.”)
Psychology/Biology
-
“High-throughput mapping of a whole rhesus monkey brain at micrometer resolution”, et al 2021
-
“Intelligence, health and death”, et al 2021
-
“Genetic and environmental contributions to IQ in adoptive and biological families with 30-year-old offspring”, et al 2021a; “Parent Contributions to the Development of Political Attitudes in Adoptive and Biological Families”, et al 2021b
-
“Toxoplasmosis: Recent Advances in Understanding the Link Between Infection and Host Behavior”, 2021
-
“Cognitive Behavior Therapy for Depression From an Evolutionary Perspective”, et al 2021
-
“Energetic Aliens”, Stephen Malina
-
“Steps of Reasoning in Children and Adolescents”, 2021 (Piagetian levels in strategy & hypothetical reasoning)
-
“Male Chimpanzees Prefer Mating with Old Females”, et al 2006 (so why don’t humans?)
Technology
-
Space gun (“Bigger NNs will never make AGI, any more than bigger bottle rockets or compressed tubes of gunpowder could ever get you to space—” “In Project HARP, a 1960s joint United States and Canada defence project, a U.S. Navy…gun was used to fire a 180kg projectile at 3,600 m⁄s, reaching an apogee of 180 km, hence performing a suborbital spaceflight.”)
-
The Manhattan Project & the Shinkolobwe mine (on the extraordinary bad luck of humanity)
-
“XY problem” (when asking questions or for help, solve problems end-to-end, don’t yak-shave)
-
“In-place Superscalar Samplesort (IPS4o): Engineering In-place (Shared-memory) Sorting Algorithms”, et al 2020 (what it takes to write a truly fast sorting algorithm for today’s hardware)
-
Forgotten wiki techs: E2’s “soft links” (the E2 wiki used click-tracking to automatically build “see also” sets of links at the bottom of pages, based on the next link readers visited; writers & readers deliberately cultivate these links to create paths through the wiki, poems or short stories, commentary, criticism, voting systems or reactions… I know of no recent wikis with an equivalent.)
Economics
-
“Win-win denial: The psychological underpinnings of zero-sum thinking”, et al 2021
-
“The economic value of targeting aging”, et al 2021
-
“Two Blades of Grass: The Impact of the Green Revolution”, et al 2021 (we still underestimate the Green Revolution)
-
“TV Advertising Effectiveness and Profitability: Generalizable Results From 288 Brands”, et al 2021
-
“Flash Boys 2.0: Frontrunning in Decentralized Exchanges, Miner Extractable Value, and Consensus Instability”, et al 2020 (fun & games with Ethereum; see also Polymarket, the Dark Forest, bot trapping, rugpulling liquidity pools, ponzis, sniping best NFTs, hacking nonce reuses)
Philosophy
Fiction
-
“Emotionally Numb: Expertise Dulls Consumer Experience”, et al 2021 (collecting & hedonic treadmills?)
-
“Avaunt”, Amble (dark-comedy: Terry Pratchett meets Rick+Glen Cook programmer-fantasy)
Miscellaneous
Books
Nonfiction:
-
The Great Successor: The Divinely Perfect Destiny of Brilliant Comrade Kim Jong Un, 2019 (a quick journalistic tour through what is known of Kim Jong-un’s life up until quite recently; because of coronavirus, there hasn’t been much to report about North Korea in the ~2 years since. Stylistically, Fifield tends to the annoyingly-chatty/smarmy journalist type (it’d be improved if most of the attempts at humor were excised, to save the patient), and much of the contents will be familiar to anyone who diligently reads Western media coverage of NK, but the book is good anyway: what Fifield brings to the table is access to obscure relatives & their unpublished autobiography—this clears up a lot about the complicated, not to mention cruel, household dynamics of the Kim dynasty.
Both Kim Il-sung and Kim Jong-il come off as deeply concerned about the succession problem; their solution, in a nominally monogamous Communist society, is maintaining multiple secret households of wives & children, winnowing through the offspring for an at least ‘adequate’ heir. Jong Un emerges as a last-minute victor when his siblings disqualify themselves: an older brother suffers an opaque hormonal issue rendering him unfit for anything but noodling around playing guitar (apparently remarkably well), while rival household’s Kim Jong-nam blows his lead by disqualifying himself as an impulsively incompetent hedonist; sister Kim Yo-Jong is ruthlessly effective (borderline sociopathic), but suffers from the fatal defect of being female in a nominally egalitarian Communist society. Jong-un was not a particularly sterling candidate, but was the best of the bunch that Jong-Il had to work with, and as Jong-il’s health problems accelerated rapidly (due to over-indulgence), he frantically seized on Jong-un.
Fortunately, Jong-un was young enough to be molded: the normal young basketball-loving boy who returned from school in Switzerland was made king of all he surveyed, and coached in megalomania. He was showered with paternal attention, allowed to order around aged generals as his playthings, and told constantly how brilliant he was by everyone around him. If Jong-un seems like a narcissist on par with Donald Trump, then Fifield’s account suggests this was not born, but made. Simultaneously, an even-more-absurd-than-usual personality cult was built up around Jong-un as the eponymous Great Successor—all of what making him so great being made up frantically as Jong-Il’s death loomed. When he died, Jong-un effectively seized and consolidated power, surviving the dangerous window when he was still illegitimate by extreme actions like murdering his uncle. One might think that could spark rebellion, but as much as they resemble a bag of cats, the Kims always agree on one thing: preserving the House of Kim.
His primary challenge is how to ride the tiger of development: by following the Chinese model and turning NK into another Tibet or Xinjiang, he may be able to pull it off. Fifield is permitted to visit Pyongyang, and records her amusement at how the nomenklatura there are gratified by the most trivial improvements: a ‘French restaurant’ here, some small ski slopes there, an aquarium over there, an outdated featurephone for this daughter and jeans for this son etc. But that’s exactly how to do it: if revolutions are triggered by elite disaffection and disappointed expectations, then doling improvements out as slowly and steadily as possible is how to do it, ensuring that each elite has “too much to lose”, never allowing a Schelling point for rival elites, and reining in any elite overproduction.
He has done so. Indeed, Jong-un’s most dangerous enemy now, judging by his obesity and reported health issues, is himself. But even there he appears to’ve produced at least one heir, and slimmed down considerably. Regrettably, under the current circumstances, Jong-un can look forward to a long reign. The regime must fall or change someday, because if something can’t go on forever, it won’t; but as Adam Smith remarked, “there is a great deal of ruin in a nation”, and NK has outlasted almost everyone who forecast that it would have to collapse soon.)
Film/TV
Live-action:
-
The Green Knight (2021; Hollywood has always been in love with telling stories about Hollywood, and The Green Knight is no exception—Sir Gawain becomes the hero of a story like he wanted, and discovers he doesn’t like it one bit, while everyone else (including the viewer) enjoys watching it and speculating about his impending death. The hero of a story, after all, can’t know how it ends. Long review after rewatch.)
Animated:
-
Space Battleship Yamato (197450ya; review)