October 2021 News
October 2021 Gwern.net newsletter with links on TODO
October 2021’s Gwern.net newsletter is now out; previous, September 2021 (archives). This is a collation of links and summary of major changes, overlapping with my Changelog; brought to you by my donors on Patreon.
Writings
Gwern.net, interesting new tags:
Links
AI
-
“Program Synthesis with Large Language Models”, et al 2021 (smooth scaling like Codex continues to 137b-parameter LaMDA; enables dialogue, solving math problems); “Show Your Work: Scratchpads for Intermediate Computation with Language Models”, et al 2021 ( AQUA-RAT might be worth revisiting)
-
“The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models”, et al 2022 (“phase transitions: capability thresholds at which the agent’s behavior qualitatively shifts”)
-
“CoCa: Contrastive Captioners are Image-Text Foundation Models”, et al 2022 (86% ImageNet zero-shot)
-
CLIPIT PixelDraw (Colab; high-quality pixel art via CLIP which Just Works™, using the CLIPDraw approach; this is particularly striking given the failure of pixel art generative models to date, like the endless Pokemon sprite attempts—apparently it really did need n = 400m before pixel art levels of visual abstraction could work… See also “Projected GANs Converge Faster”, et al 2021 which gets its good Pokemon via ImageNet/CLIP)
-
“Make Every feature Binary (MEB): A 135b-parameter sparse neural network for massively improved search relevance” (an extremely large embedding for text ranking, now powering 100% of Microsoft Bing search queries)
-
“Partial success in closing the gap between human and machine vision”, et al 2021 (“We have to admit that we view our results concerning the benefits of increasing dataset size by 1–3 orders of magnitude with mixed feelings. On the one hand, ‘simply’ training standard models on (a lot) more data certainly has an intellectually disappointing element—particularly given many rich ideas in the cognitive science and neuroscience literature…”); “Evaluating Machine Accuracy on ImageNet”, et al 2021 ( “the latest models from 2020 are on par with our best human labeler”); “When does dough become a bagel? Analyzing the remaining mistakes on ImageNet”, et al 2022 ( “CoCa-FT gets 42 of the 68 [remaining hard errors] correct”)
-
“What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers”, et al 2021 ( Naver)
-
Deep2020 budget reaches $1b ($1.06b, +$0.085b)
-
“M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretraining”, 2021 (easy GPT-10t training via weight-tying: pretrain a single module/layer that fits in nodes, then ‘unroll’ it to the fullsize model for finetuning)
-
“Persia: A Hybrid System Scaling Deep Learning Based Recommenders up to 100 Trillion Parameters”, et al 2021 ( Kuaisho)
Genetics
Everything Is Heritable:
Recent Evolution:
Engineering:
Statistics/Meta-Science
Politics/Religion
Psychology/Biology
-
“Adolescent cannabis use and adult psychoticism: A longitudinal co-twin control analysis using data from two cohorts”, et al 2021 (“no”)
-
“How Long Does It Take Ordinary People To ‘Get Good’ At Chess?”, Joseph Wong
-
“Did we find a copycat? ‘Do as I Do’ in a domestic cat (Felis catus)”, et al 2020
-
“Speed-accuracy trade-off in plants”, et al 2020
-
“Wet mammals shake at tuned frequencies to dry”, et al 2021 ( Sandberg thread)
-
Impossible colors (“Hello, yes, I would like a blacker black, also, a ‘self-luminous red’ which is simultaneously red & brighter than white”—real colors, from the demented ravings of madmen; THEY HAVE PLAYED US FOR FOOLS!)
Technology
-
“AIR-FI: Generating Covert Wi-Fi Signals from Air-Gapped Computers”, 2020 (some fun earlier side-channels: et al 2015a , et al 2015b )
-
“Ditherpunk—The article I wish I had about monochrome image dithering”
-
Block writing:
-
Anita 2016 “Hangulatin”: a Hangul-inspired font for English/German word blocks ; Sebastian Moser’s “Sori”
-
Square Kufic (eg. ISIS, or SUN?)
Economics
-
“What is the Commons Worth? Estimating the Value of Wikimedia Imagery by Observing Downstream Use”, et al 2018; “Digitization and the Demand for Physical Works: Evidence from the Google Books Project”, 2019 (copyright deadweight losses)
-
“Potterian Economics”, 2017
-
“The Theory of Interstellar Trade”, Krugman 197846ya/2010
-
“Should subscription-based content creators display their earnings on crowdfunding platforms? Evidence from Patreon”, Crosby & 2021 (I’m giving disabling my public display a try… Doesn’t seem to be working thus far.)
Philosophy
Fiction
Miscellaneous
Books
Nonfiction:
Fiction:
Film/TV
Live-action:
Animated:
Music
MLP:
Doujin:
Misc: