July 2021 Gwern.net newsletter with links on TODO
July 2021’s Gwern.net newsletter is now out; previous, June 2021 (archives). This is a collation of links and summary of major changes, overlapping with my Changelog; brought to you by my donors on Patreon.
Writings
- CO2 Coin: A Blockchain For Carbon Capture
- Gwern.net: “link tags” system created
Links
AI
- “Highly accurate protein structure prediction with AlphaFold 2”, et al 2021 ( source code)
- “StarCraft Commander (SCC): an efficient deep reinforcement learning agent mastering the game of StarCraft II”, et al 2021
- “Predictive Coding: a Theoretical and Experimental Review”, et al 2021
- “ETA Prediction with Graph Neural Networks in Google Maps”, Derrow-et al 2021 (impressive that NNs can produce such gains on such mature systems—where’s your “AI winter” now?)
- “From talking to OpenAI, GPT-4 will be about 100 trillion parameters. That won’t be ready for several years.” (Cerebras CEO on new clustering & software capabilities; note that Sam Altman at the 2021-09-05 SSC Q&A described this as misleading, and to paraphrase, that it was highly speculative discussion a while ago & GPT-4 might never exist, total compute is much more important than parameter count, and they are more interested in making more efficient use of existing models by better prompting, continuous training, & human feedback)
- “Jurassic-1: Technical Details And Evaluation”, et al 2021 ( GPT-3-178b model+public API offered by new startup AI21 Labs; demos; this completes the question of GPT-3 time-lags: it took ~11 months for replication by a competitor, and ~14 months for replication+public-API)
- Tesla AI 2021: “Tesla Dojo’s Unique Packaging and Chip Design Allow An Order Magnitude Advantage Over Competing AI Hardware” (“Each tile has an impressive 9 PFlops of BF16/CFP8 and 36 TB/s of off-tile bandwidth. This far surpasses the off-wafer bandwidth of Cerebras, and enables the Tesla system to scale out better than even scale out designs such as the Tenstorrent architecture.”; SemiAnalysis criticism)
- “ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation”, et al 2021 ( demo; #1 SuperGLUE, exceeding human baseline & Microsoft Research performance)
- “Structured Denoising Diffusion Models (D3PMs) in Discrete State-Spaces”, et al 2021; “Beyond In-Place Corruption: Insertion and Deletion In Denoising Probabilistic Models”, et al 2021; “Argmax Flows and Multinomial Diffusion: Learning Categorical Distributions”, et al 2021a; “ARDMs: Autoregressive Diffusion Models”, et al 2021b; “SUNDAE: Step-unrolled Denoising Autoencoders for Text Generation”, et al 2021 “Zero-Shot Translation using Diffusion Models”, 2021; “Diffusion-LM Improves Controllable Text Generation”, et al 2022 ( DDPMs for text; potentially better global coherency as the entire text is updated repeatedly, in a much more natural way allowing for turning compute into performance & exploring alternatives… I suspect a diffusion text model will be largely immune to the repetition trap. Good for top-down story generation? Or useful for anagram/palindrome generation? Anagram generators are useless past 1–2 words due to combinatorial explosion; but a DDPM could handle almost arbitrary sized inputs to reconstruct meaningful sentences. Training noise = shuffling letters. To create: take a target string, random shuffle, and denoise. Do this repeatedly for multiple anagrams of the original target. Can use embeddings to guide denoising towards semantically different or identical anagrams of original. Applications in CS, genomics?)
Genetics
Everything Is Heritable:
- “A Global Nucleic Acid Observatory for Biodefense and Planetary Health”, The Nucleic Acid Observatory 2021 (Twitter)
- “Genomic evidence consistent with antagonistic pleiotropy may help explain the evolutionary maintenance of same-sex sexual behaviour in humans”, et al 2021
- “Complex feline disease mapping using a dense genotyping array”, et al 2021 (a rare cat GWAS!)
- “Using DNA to predict behaviour problems from preschool to adulthood”, et al 2021; “Identification of 370 genetic loci for age at first sex and birth linked to externalising behaviour”, et al 2021
Recent Evolution:
- “De novo evolution of macroscopic multicellularity”, et al 2021 ( Twitter; the multicellularity Long Term Evolution Experiment)
- Virophage (“a flea / Hath smaller fleas that on him prey; / And these have smaller fleas to bite ’em…”)
Engineering:
Statistics/Meta-Science/Math
- “Three Months in Monte Carlo” (interactive explorables of Ising model algorithms & emergent behavior, and the history of Monte Carlo algorithms)
- “John W. Tukey: His Life and Professional Contributions”, 2002
- “Causal and Associational Linking Language From Observational Research and Health Evaluation Literature in Practice: A systematic language evaluation”, et al 2021 ( Twitter; researchers interpret most correlational language as implying causality)
- Solving Archimedes’s Cattle Problem (WP)
- “MegaLMM: Mega-scale linear mixed models for genomic predictions with thousands of traits”, et al 2021 (cf. variance components)
- Ulam spiral
Politics/Religion
- “The Dark Side of Private Ordering: An Institutional and Empirical Analysis of Organized Crime”, 2000
- “‘He who pays the piper calls the tune’: Researcher experiences of funder suppression of health behaviour intervention trial findings”, et al 2021 ( “Our results, along with those of previous investigations, suggest that government funders interfere with public-good research.”)
- Tama-Re
Psychology/Biology
- “High-throughput mapping of a whole rhesus monkey brain at micrometer resolution”, et al 2021
- “Principles of Cryopreservation by Vitrification”, 2015
- “Intelligence, health and death”, et al 2021
- “Genetic and environmental contributions to IQ in adoptive and biological families with 30-year-old offspring”, et al 2021a; “Parent Contributions to the Development of Political Attitudes in Adoptive and Biological Families”, et al 2021b
- “Toxoplasmosis: Recent Advances in Understanding the Link Between Infection and Host Behavior”, 2021
- “Cognitive Behavior Therapy for Depression From an Evolutionary Perspective”, et al 2021
- “A Contamination Theory of the Obesity Epidemic”
- “Energetic Aliens”, Stephen Malina
- “Steps of Reasoning in Children and Adolescents”, 2021 (Piagetian levels in strategy & hypothetical reasoning)
Technology
- Space gun (“Bigger NNs will never make AGI, any more than bigger bottle rockets or compressed tubes of gunpowder could ever get you to space—” “In Project HARP, a 1960s joint United States and Canada defence project, a U.S. Navy…gun was used to fire a 180kg projectile at 3,600 m⁄s, reaching an apogee of 180 km, hence performing a suborbital spaceflight.”)
- The Manhattan Project & the Shinkolobwe mine (on the extraordinary bad luck of humanity)
- “XY problem” (when asking questions or for help, solve problems end-to-end, don’t yak-shave)
- Video game speedrunning improvements over time
- “In-place Superscalar Samplesort (IPS4o): Engineering In-place (Shared-memory) Sorting Algorithms”, et al 2020 (what it takes to write a truly fast sorting algorithm for today’s hardware)
- Forgotten wiki techs: E2’s “soft links” (the E2 wiki used click-tracking to automatically build “see also” sets of links at the bottom of pages, based on the next link readers visited; writers & readers deliberately cultivate these links to create paths through the wiki, poems or short stories, commentary, criticism, voting systems or reactions… I know of no recent wikis with an equivalent.)
Economics
- “Win-win denial: The psychological underpinnings of zero-sum thinking”, et al 2021
- “The economic value of targeting aging”, et al 2021
- “Two Blades of Grass: The Impact of the Green Revolution”, et al 2021 (we still underestimate the Green Revolution)
- “TV Advertising Effectiveness and Profitability: Generalizable Results From 288 Brands”, et al 2021
- “Flash Boys 2.0: Frontrunning in Decentralized Exchanges, Miner Extractable Value, and Consensus Instability”, et al 2020 (fun & games with Ethereum; see also Polymarket, the Dark Forest, bot trapping, rugpulling liquidity pools, ponzis, sniping best NFTs, hacking nonce reuses)
Philosophy
- “Buddhist Animal Release Practices: Historic, Environmental, Public Health And Economic Concerns”, 2008
- “Survey of subjective ‘God encounter experiences’: Comparisons among naturally occurring experiences and those occasioned by the classic psychedelics psilocybin, LSD, ayahuasca, or DMT”, et al 2019
Fiction
- “Emotionally Numb: Expertise Dulls Consumer Experience”, et al 2021 (collecting & hedonic treadmills?)
- “Don’t Make Me Think”, Zero HP Lovecraft (Twitter)
- “Avaunt”, Amble (dark-comedy: Terry Pratchett meets Rick+Glen Cook programmer-fantasy)
Miscellaneous
Books
Nonfiction:
-
The Great Successor: The Divinely Perfect Destiny of Brilliant Comrade Kim Jong Un, 2019 (a quick journalistic tour through what is known of Kim Jong-un’s life up until quite recently; because of coronavirus, there hasn’t been much to report about North Korea in the ~2 years since. Stylistically, Fifield tends to the annoyingly-chatty/smarmy journalist type (it’d be improved if most of the attempts at humor were excised, to save the patient), and much of the contents will be familiar to anyone who diligently reads Western media coverage of NK, but the book is good anyway: what Fifield brings to the table is access to obscure relatives & their unpublished autobiography—this clears up a lot about the complicated, not to mention cruel, household dynamics of the Kim dynasty.
Both Kim Il-sung and Kim Jong-il come off as deeply concerned about the succession problem; their solution, in a nominally monogamous Communist society, is maintaining multiple secret households of wives & children, winnowing through the offspring for an at least ‘adequate’ heir. Jong Un emerges as a last-minute victor when his siblings disqualify themselves: an older brother suffers an opaque hormonal issue rendering him unfit for anything but noodling around playing guitar (apparently remarkably well), while rival household’s Kim Jong-nam blows his lead by disqualifying himself as an impulsively incompetent hedonist; sister Kim Yo-Jong is ruthlessly effective (borderline sociopathic), but suffers from the fatal defect of being female in a nominally egalitarian Communist society. Jong-un was not a particularly sterling candidate, but was the best of the bunch that Jong-Il had to work with, and as Jong-il’s health problems accelerated rapidly (due to over-indulgence), he frantically seized on Jong-un.
Fortunately, Jong-un was young enough to be molded: the normal young basketball-loving boy who returned from school in Switzerland was made king of all he surveyed, and coached in megalomania. He was showered with paternal attention, allowed to order around aged generals as his playthings, and told constantly how brilliant he was by everyone around him. If Jong-un seems like a narcissist on par with Donald Trump, then Fifield’s account suggests this was not born, but made. Simultaneously, an even-more-absurd-than-usual personality cult was built up around Jong-un as the eponymous Great Successor—all of what making him so great being made up frantically as Jong-Il’s death loomed. When he died, Jong-un effectively seized and consolidated power, surviving the dangerous window when he was still illegitimate by extreme actions like murdering his uncle. One might think that could spark rebellion, but as much as they resemble a bag of cats, the Kims always agree on one thing: preserving the House of Kim.
His primary challenge is how to ride the tiger of development: by following the Chinese model and turning NK into another Tibet or Xinjiang, he may be able to pull it off. Fifield is permitted to visit Pyongyang, and records her amusement at how the nomenklatura there are gratified by the most trivial improvements: a ‘French restaurant’ here, some small ski slopes there, an aquarium over there, an outdated featurephone for this daughter and jeans for this son etc. But that’s exactly how to do it: if revolutions are triggered by elite disaffection and disappointed expectations, then doling improvements out as slowly and steadily as possible is how to do it, ensuring that each elite has “too much to lose”, never allowing a Schelling point for rival elites, and reining in any elite overproduction.
He has done so. Indeed, Jong-un’s most dangerous enemy now, judging by his obesity and reported health issues, is himself. But even there he appears to’ve produced at least one heir, and slimmed down considerably. Regrettably, under the current circumstances, Jong-un can look forward to a long reign. The regime must fall or change someday, because if something can’t go on forever, it won’t; but as Adam Smith remarked, “there is a great deal of ruin in a nation”, and NK has outlasted almost everyone who forecast that it would have to collapse soon.)
Film/TV
Live-action:
- The Green Knight (2021; Hollywood has always been in love with telling stories about Hollywood, and The Green Knight is no exception—Sir Gawain becomes the hero of a story like he wanted, and discovers he doesn’t like it one bit, while everyone else (including the viewer) enjoys watching it and speculating about his impending death. The hero of a story, after all, can’t know how it ends. Long review after rewatch.)
Animated:
- Space Battleship Yamato (1974; review)