September 2020 Gwern.net newsletter with links on DRL and AI scaling, psychiatric disorders; no reviews.
September 2020’s Gwern.net newsletter is now out; previous, August 2020 (archives). This is a collation of links and summary of major changes, overlapping with my Changelog; brought to you by my donors on Patreon.
Writings
-
Gwern.net: dark mode image handling fixed (images are inverted based on automated heuristic + manual tagging); annotations can now be ‘definitions’ (popup annotations for non-link text); expanded tooltips as fallback for link annotations; bolded abstracts & revised list hierarchy to use bold; changed all dates to YYYY-MM-DD for consistency; added missing link icons for >7 domains, fixed spacing & overlapping, and harmonized opacity of all link icons; Pandoc now uses MathJax directly, enabling colored equations; internal links to a previous or later section now point up or down (respectively) as a navigation aid; experimental use of instant.page prefetching; custom asterism instead of horizontal ruler; simplified mobile appearance; miscellaneous bug fixes (assisted by new test/demo page)
Links
AI
-
“ReBeL: Combining Deep Reinforcement Learning and Search for Imperfect-Information Games”, et al 2020 (heads-up no-limit Texas hold’em poker close to solved with a generalizable AlphaZero-like approach?)
-
“Assessing Game Balance with AlphaZero: Exploring Alternative Rule Sets in Chess”, et al 2020
-
“A digital biomarker of diabetes from smartphone-based vascular signals”, et al 2020 (time-series of blood pulses can diagnose diabetes? weird, but not as weird as predicting gender from retinas)
-
“Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves”, et al 2020; “Gradient Descent: The Ultimate Optimizer”, et al 2019 (deeper—we need to go deeper… now scales to VGG-Net)
-
“X-LXMERT/X-UNITER: Paint, Caption and Answer Questions with Multi-Modal Transformers”, et al 2020 ( online demo; finally, some new work on text → image)
-
“StyleGAN network blending”, Justin Pinkney/“Resolution Dependent GAN Interpolation for Controllable Image Synthesis Between Domains”, 2020 (interpolating between models to transfer images between domains, eg. FFHQ ↔︎ foxes ↔︎ anime ↔︎ MLP ↔︎ furries)
-
“Optimal Peanut Butter and Banana Sandwiches”, Ethan Rosenthal (bin packing banana slices)
Matters Of Scale:
-
“DeepSpeed: Extreme-scale model training for everyone” (demonstrates training of GPT-3-180b & 1t-parameter models (“The trillion-parameter model has 298 layers of Transformers with a hidden dimension of 17,408 and is trained with sequence length 2,048 and batch size 2,048”), w/open-source code; able to use CPU+GPU RAM simultaneously for 13b-parameter models on each node per et al 2020 ; sparse attention for saving RAM; approximated Adam gradients for saving bandwidth)
-
“GPT-f: Generative Language Modeling for Automated Theorem Proving”, 2020b (GPT-2 for Metamath scales & can bootstrap its theorem-proving ability—onward to IMO!)
-
“Measuring Massive Multitask Language Understanding”, et al 2020 (introducing new much harder benchmarks to stress-test models—nevertheless, bigger=better: GPT-3 model size vs Q&A, with GPT-3-173b hitting almost 50% out of 4-choice questions vs baseline models like T5 at the random baseline of only 25%)
-
“DynamicEmbedding: Extending TensorFlow for Colossal-Scale Applications”, et al 2020 (embeddings-as-a-service Google-wide; tool AIs want to be agent AIs…)
-
“A Time Leap Challenge for SAT Solving”, et al 2020 ( your computer is faster than you think, software+hardware overhang: ~2.5× performance increase in SAT solving since 2000, about equally due to software & hardware gains, although slightly more software—new software on old hardware beats old on new. See previously 2013.)
-
“Efficient Transformers: A Survey”, et al 2020 (more on efficient attention approaches)
Genetics
Everything Is Heritable:
-
“A general dimension of genetic sharing across diverse cognitive traits inferred from molecular data”, de la et al 2020
-
“Mapping genomic loci prioritises genes and implicates synaptic biology in schizophrenia”, PGC et al 2020 (PGC SCZ3: 248 genome-wide hits; 7.7% PGS; OR = 44 comparing top vs bottom percentile risk); “Exome sequencing identifies rare coding variants in 10 genes which confer substantial risk for schizophrenia”, et al 2020 ( figure 6a is a striking demonstration of negative/purifying selection: more harmful mutations are rarer)
-
“Genome-wide association study of over 40,000 bipolar disorder cases provides novel biological insights”, et al 2020 (PGC3 BPD; PGS: 4.75%)
-
“Genetic Fortune: Winning or Losing Education, Income, and Health”, et al 2020 (within-family education/income/health PGSes)
-
“Shared heritability of face and brain shape distinct from cognitive traits”, et al 2020 (setting limits to facial prediction of some phenotypes)
Recent Evolution:
Engineering:
-
The first successful cloning of the endangered Przewalski’s horse
-
“Human Embryo Gene Editing Gets a Road Map—Not a Green Light” (National Academy of Science’s Heritable Human Genome Editing 2020 report)
-
“CRISPR-engineered human brown-like adipocytes prevent diet-induced obesity and ameliorate metabolic syndrome in mice”, et al 2020; “CRISPR-enhanced human adipocyte ‘browning’ as cell therapy for metabolic disease”, et al 2020
Statistics/Meta-Science
-
“Blueberry Earth”, Sandberg 2018
Politics/Religion
-
“America in Decay: The Sources of Political Dysfunction”, Fukuyama 2014 (see also: constitutional hardball/“The Empty Chamber”)
-
“‘Dwarf Pride’ Was Hard Won. Will a Growth Drug Undermine It? An experimental medication that increases height in children with the most common form of dwarfism has raised hope that it can help them lead easier lives. But some say the condition is not a problem in need of a cure.”, FDA-approved (et al 2020 ; amusing trivia: the reason why crossing any dog breed with Corgis gives you a Corgified version of that dog breed is because Corgis are homozygous dominant on an FGF4 mutation causing similar achondroplasia as in humans)
-
“Laws of Xmas”, Akiva & Ilene 1998
Psychology/Biology
-
“Self-Supervised Natural Image Reconstruction and Rich Semantic Classification from Brain Activity”, et al 2020 (towards brain imitation learning)
-
“Catching Cowpox: The Early Spread of Smallpox Vaccination, 1798–1810”, 2009
-
“Creativity and Intelligence: An Investigation of the Threshold Hypothesis”, et al 2020
-
Discovery of more water on Mars and first extra-galactic planet
Technology
-
“Unraveling The JPEG”, Omar 2019 (explorable: editing JPEGs to understand their encoding)
-
“The Recursive Universe”, Amanda Ghassaei (how to implement Conway’s Game of Life inside itself as a single giant ‘metapixel’ pattern)
-
“The First Roman Fonts”, John Boardley
Economics
-
“Salary Negotiation: Make More Money, Be More Valued”, patio11
-
“A Deep Dive into K-pop”, Matt Lakeman
-
“Wikipedia Matters”, et al 2019 (randomization of edits to foreign languages’ Wikipedia entries about an obscure Spanish city increases tourism to the city from those foreign countries by 9%)
Miscellaneous
-
1955 Le Mans disaster (the past is a Third World country)