March 2021 Gwern.net newsletter; 2 major new site features: ‘popins’ and recursive Wikipedia popups.
March 2021’s Gwern.net newsletter is now out; previous, February 2021 (archives). This is a collation of links and summary of major changes, overlapping with my Changelog; brought to you by my donors on Patreon.
Writings
-
Gwern.net: mobile “popins” are finally enabled! (example); new Wikipedia popups (this 7th implementation enables recursive WP popups)
Links
AI
-
“Multimodal Neurons in Artificial Neural Networks”, et al 2021 (dissecting CLIP concepts, discovering typographical classification ‘attacks’1 and a Stroop effect! Is there anything CLIP can’t do?)
-
“Evolving Reinforcement Learning Algorithms”, Co-et al 2021 (evolving eg. TD-learning)
-
“Waymo Simulated Driving Behavior in Reconstructed Fatal Crashes within an Autonomous Vehicle Operating Domain”, et al 2021 ( blog; hard negative mining—self-driving cars, being inhuman, can learn not just from their mistakes but humans’ mistakes too)
-
“Debugging Reinforcement Learning Systems Without The Agonizing Pain”, Andy L. Jones; “My Reinforcement Learning Learnings”, Clemens Winter; “Systems That Defy Understanding”
-
“SEER: Self-supervised Pretraining of Visual Features in the Wild”, et al 2021 ( blog; near-SOTA by training 1b-param CNN on 1b unfiltered unlabeled Internet images—another reminder that unsupervised learning is really working!); “‘Learning From Videos’ to understand the world” (rapid FB expansion of self-supervised training to millions of photos/videos/hours-of-speech); “Contrasting Contrastive Self-Supervised Representation Learning Models”, et al 2021 (Supervised learning from ImageNet is now obsolete for transfer learning, and ImageNet just a contaminated validation set)
-
“Understanding Robustness of Transformers for Image Classification”, et al 2021 ( Vision Transformers gain robustness faster than CNNs as dataset size increases)
-
“Artificial Intelligence Index 2021”: technical performance and cost (Ding questions whether this shows China catching up on AI at all, as we are incessantly told it is doing; one question to ask: ignoring fast-following, can you, out of the thousands upon thousands of publications flooding out these days, name 3 major novel AI breakthroughs coming out of all pure-Chinese labs combined which could be plausibly equated in importance with, say, just OpenAI’s recent output of GPT-3/DALL·E 1/CLIP?)
-
OA GPT-3 API: >300 apps, >10k developers, >4.5b words per day (Thought: GPT-3 trained on ~300b tokens. Rule of thumb: training costs 1× of forward + backprop is 2× cost of forward = 3×; 500b tokens of training ~ 1500524yab generated. If 1.25 tokens = 1 word then 4.5bw = 5.6b tokens. So the OA API uses 1 full GPT-3 training run of compute every 53 days.)
-
“A mathematical theory of semantic development in deep neural networks”, et al 2019 (are jumps in NN capabilities to be expected when scaling? see also 2021’s discussion of phase transitions & averaging of exponentials giving power-laws)
-
“An early cell shape transition drives evolutionary expansion of the human forebrain”, Benito-et al 2021 ( media; a simple switch for the scaling up of the primate brain)
-
“Crows possess higher intelligence long thought primarily human” (the remarkable, yet not extraordinary, crow/raven brain as scaled-up bird brain); “Behavioral and Neuronal Representation of Numerosity Zero in the Crow”, et al 2021
-
Genetics
Everything Is Heritable:
-
“GWAS in almost 195,000 individuals identifies 50 previously unidentified genetic loci for eye color”, et al 2021
-
“Why Do Wealthy Parents Have Wealthy Children?”, et al 2021 (I’m always impressed just how difficult it is for rich people to pass on wealth—“shirtsleeves to shirtsleeves in 3 generations” etc)
Evolution:
Engineering:
Statistics/Meta-Science
-
“Broad cross-national public support for accelerated COVID-19 vaccine trial designs”, et al 2021 (“we can’t do challenge trials with volunteers in February 2020 to save countless thousands of lives because ordinary people might think it unethical”—have you tried asking them, or was that irrelevant because it was just another noble lie?)
-
“This is the story of how I found what I believe to be scientific misconduct and what happened when I reported it”, Joe Hilgard (on leaving academia)
-
“The Revolution in Classic Tetris: How a younger generation used the Internet to master the falling blocks” (how achieving classic Tetris maximum-scores, first done in 201014ya, became routine thanks to YouTube & online competition for excellence)
Politics/Religion
-
“Magic, Explanations, and Evil: The Origins and Design of Witches and Sorcerers”, 2021 (doubtless even cavemen were all “Og: sus.”)
-
“Self-blinding citizen science to explore psychedelic microdosing”, et al 2021 (related to et al 2021 ; a self-blinding study, similar to my old self-blinding protocols, confirms that microdosing is just placebo effect, as I said in 2012, and I’m reminded of DNB studies like et al 2016 )
-
The 2019–2020 vaping moral panic over adulterated black-market THC products (depressing to see how irresponsibly reported & alarmist this was, and how everyone attempted to frame nicotine for it2. Naturally, no one involved has apologized or admitted fault—after all, their intentions were good, “won’t someone think of the children”‽ The incompetence and/or dishonesty here emphasizes how 2020–2021 was business as usual, and the only unusual part is that reality happened so fast we saw some of the unseen.)
-
Alexandra David-Néel (one of those 1800–1001900124yas biographies)
Psychology/Biology
-
“The pandemic fallacy: Inaccuracy of social scientists’ and lay judgments about COVID-19’s societal consequences in America”, et al 2021 (highly-inaccurate even retrospectively, typically grossly overestimating)
-
“Training Working Memory for 2 Years—No Evidence of Latent Transfer to Intelligence”, et al 2021 (fade-out of expectancy/placebo effects)
-
“Real-time dialogue between experimenters and dreamers during REM sleep”, et al 2021
-
“Leroy’s elusive little people: A systematic review on lilliputian hallucinations”, 2021 (Alice in Wonderland syndrome)
-
“A Group of Orca Outcasts Is Now Dominating an Entire Sea: ‘Transient’ killer whales that feast on seals and hunt in small packs are thriving while their widely beloved ‘Resident’ siblings are dying out” (I wonder how the third orca type, ‘offshore’, are doing?)
-
“Estimation of the total saliva volume produced per day in 5-year-old children”, et al 1995
Technology
-
“The Aesthetic-Usability Effect”, 2017 (“They Might Never Tell You It’s Broken” if it’s pretty enough; see also “The Third User”)
-
“Cameras and Lenses”, Bartosz Ciechanowski (explorable; followup to “Lights and Shadows”)
-
“Large Batch Simulation for Deep Reinforcement Learning”, et al 2021 (your computer is faster than you think)
-
“The incredible boxes of Hock Wah Yeo” (unusual video game packaging design)
Economics
-
“The Use and Misuse of Income Data and Extreme Poverty in the United States”, et al 2021 ( measurement error in non-registry surveys of population extremes—not quite “lizardman” but similar problem)
-
“Is economics performative? Option theory and the construction of derivatives markets”, 2006 (the mechanics of how the Black-Scholes model changed markets: Black ran a service printing “paper” estimating optimal prices for all options which traders could consult & use with simple heuristics to try to arbitrage the market)
-
“Whitewood under Siege: On the front lines of the pallet wars” (the competition between the 2 ecosystems of shipping pallets: ‘whitewood’ & ‘blue pallet’)
Philosophy
Fiction
Miscellaneous
-
America’s top ace, Major Dick Bong
Film/TV
Live-action:
-
North by Northwest (Hitchcock 195965ya; for such an extremely respected movie, it felt oddly formless and like it was bouncing through genres as more of a comedic B-movie romp than a serious auteur’s effort—since James Bond started in 195371ya, with a TV adaptation in 195470ya, NbN comes off as almost a satire. I mean, really, monkeying around in Presidential noses!)
-
While interesting, these are ‘attacks’ only in the most generous interpretation possible (since it does know the difference), and the fact that CLIP can read text in images to note the semantic similarity, is to its credit. As the CLIP authors note, some queries benefit from ensembling, more context than a single word class name such as prefixing “A photograph of a”, and class names can be highly ambiguous: in ImageNet, the class name “crane” could refer to the bird or construction equipment; and the Oxford-IIIT Pet dataset labels one class “boxer”. (CLIP is still vulnerable to regular adversarial examples, of course.)↩︎
-
It couldn’t’ve been nicotine because people had been vaping for a decade and a half without widespread near-instantaneous lung-related fatalities! It had to be a new adulterant, and as soon as the first few black-market THC links surfaced, that meant the problem had to be THC-products-only because how would the same adulterant simultaneously get into the different supply chains? And yet, every article, health official, and activist did their paternalist best to suggest otherwise to pin the blame on regular vaping, no matter how many tests turned up clean, and it was the nicotine vaping products which got summarily banned… One must assume many of those laws are still on the books, inasmuch as the shipping bans keep expanding.↩︎