AI

• “Multimodal Neurons in Artificial Neural Networks”, Goh et al 2021 (dissecting CLIP concepts, discovering typographical classification ‘attacks’¹ and a Stroop effect! Is there anything CLIP can’t do?)

• “Evolving Reinforcement Learning Algorithms”, Co-Reyes et al 2021 (evolving eg. TD-learning)

• “Waymo Simulated Driving Behavior in Reconstructed Fatal Crashes within an Autonomous Vehicle Operating Domain”, Scanlon et al 2021 (blog; hard negative mining—self-driving cars, being inhuman, can learn not just from their mistakes but humans’ mistakes too)

• “Debugging Reinforcement Learning Systems Without The Agonizing Pain”, Andy L. Jones; “My Reinforcement Learning Learnings”, Clemens Winter; “Systems That Defy Understanding”

Matters Of Scale:

• “SEER: Self-supervised Pretraining of Visual Features in the Wild”, Goyal et al 2021 (blog; near-SOTA by training 1b-param CNN on 1b unfiltered unlabeled Internet images—another reminder that unsupervised learning is really working!); “‘Learning From Videos’ to understand the world” (rapid FB expansion of self-supervised training to millions of photos/videos/hours-of-speech); “Contrasting Contrastive Self-Supervised Representation Learning Models”, Kotar et al 2021 (Supervised learning from ImageNet is now obsolete for transfer learning, and ImageNet just a contaminated validation set)

• “Understanding Robustness of Transformers for Image Classification”, Bhojanapalli et al 2021 (Vision Transformers gain robustness faster than CNNs as dataset size increases)

• “Artificial Intelligence Index Report2021”: technical performance and cost (Ding questions whether this shows China catching up on AI at all, as we are incessantly told it is doing; one question to ask: ignoring fast-following, can you, out of the thousands upon thousands of publications flooding out these days, name 3 major novel AI breakthroughs coming out of all pure-Chinese labs combined which could be plausibly equated in importance with, say, just OpenAI’s recent output of GPT-3/DALL·E 1/CLIP?)

• OA GPT-3 API: >300 apps, >10k developers, >4.5b words per day (Thought: GPT-3 trained on ~300b tokens. Rule of thumb: training costs 1× of forward + backprop is 2× cost of forward = 3×; 500b tokens of training ~ 1500b generated. If 1.25 tokens = 1 word then 4.5bw = 5.6b tokens. So the OA API uses 1 full GPT-3 training run of compute every 53 days.)

• “A mathematical theory of semantic development in deep neural networks”, Saxe et al 2019 (are jumps in NN capabilities to be expected when scaling? see also Viering & Loog2021’s discussion of phase transitions & averaging of exponentials giving power-laws)

• “An early cell shape transition drives evolutionary expansion of the human forebrain”, Benito-Kwiecinski et al 2021 (media; a simple switch for the scaling up of the primate brain)

  • “Crows possess higher intelligence long thought primarily human” (the remarkable, yet not extraordinary, crow/raven brain as scaled-up bird brain); “Behavioral and Neuronal Representation of Numerosity Zero in the Crow”, Kirschhock et al 2021

Genetics

Everything Is Heritable:

• “GWAS in almost 195,000 individuals identifies 50 previously unidentified genetic loci for eye color”, Simcoe et al 2021

• “Why Do Wealthy Parents Have Wealthy Children?”, Fagereng et al 2021 (I’m always impressed just how difficult it is for rich people to pass on wealth—“shirtsleeves to shirtsleeves in 3 generations” etc.)

Evolution:

• “Nothing in evolution makes sense except in the light of parasites”, Hickinbotham et al 2021

Engineering:

• “The Demise and Potential Revival of the American Chestnut”

Statistics/Meta-Science

• “Broad cross-national public support for accelerated COVID-19 vaccine trial designs”, Broockman et al 2021 (“we can’t do challenge trials with volunteers in February 2020 to save countless thousands of lives because ordinary people might think it unethical”—have you tried asking them, or was that irrelevant because it was just another noble lie?)

• “This is the story of how I found what I believe to be scientific misconduct and what happened when I reported it”, Joe Hilgard (on leaving academia)

• “The Revolution in Classic Tetris: How a younger generation used the Internet to master the falling blocks” (how achieving classic Tetris maximum-scores, first done in 2010_16ya, became routine thanks to YouTube & online competition for excellence)

Politics/Religion

• “Magic, Explanations, and Evil: The Origins and Design of Witches and Sorcerers”, Singh2021 (doubtless even cavemen were all “Og: sus.”)

• “Self-blinding citizen science to explore psychedelic microdosing”, Szigeti et al 2021 (related to Kaertner et al 2021; a self-blinding study, similar to my old self-blinding protocols, confirms that microdosing is just placebo effect, as I said in 2012, and I’m reminded of DNB studies like Foroughi et al 2016)

• The 2019–2020 vaping moral panic over adulterated black-market THC products (depressing to see how irresponsibly reported & alarmist this was, and how everyone attempted to frame nicotine for it². Naturally, no one involved has apologized or admitted fault—after all, their intentions were good, “won’t someone think of the children”‽ The incompetence and/or dishonesty here emphasizes how 2020–2021 was business as usual, and the only unusual part is that reality happened so fast we saw some of the unseen.)

• Mark Hofmann

• Alexandra David-Néel (one of those 1800^–₁₀₀1900_126yas biographies)

• John Harvey Kellogg

Psychology/Biology

• “Can You Ever Be Too Smart for Your Own Good? Comparing Linear and Nonlinear Effects of Cognitive Ability on Life Outcomes”, Brown et al 2021

• “The pandemic fallacy: Inaccuracy of social scientists’ and lay judgments about COVID-19’s societal consequences in America”, Hutcherson et al 2021 (highly-inaccurate even retrospectively, typically grossly overestimating)

• “Training Working Memory for 2 Years—No Evidence of Latent Transfer to Intelligence”, Watrin et al 2021 (fade-out of expectancy/placebo effects)

• “Real-time dialogue between experimenters and dreamers during REM sleep”, Konkoly et al 2021

• “Leroy’s elusive little people: A systematic review on lilliputian hallucinations”, Blom2021 (Alice in Wonderland syndrome)

• “A Group of Orca Outcasts Is Now Dominating an Entire Sea: ‘Transient’ killer whales that feast on seals and hunt in small packs are thriving while their widely beloved ‘Resident’ siblings are dying out” (I wonder how the third orca type, ‘offshore’, are doing?)

• “Estimation of the total saliva volume produced per day in 5-year-old children”, Watanabe et al 1995

Technology

• “The Aesthetic-Usability Effect”, Moran2017 (“They Might Never Tell You It’s Broken” if it’s pretty enough; see also “The Third User”)

• “Cameras and Lenses”, Bartosz Ciechanowski (explorable; followup to “Lights and Shadows”)

• “Large Batch Simulation for Deep Reinforcement Learning”, Shacklett et al 2021 (your computer is faster than you think)

• “The incredible boxes of Hock Wah Yeo” (unusual video game packaging design)

• “Stone Walls That Stay Built: A master waller shares how to dry-lay stone walls that hold their ground for centuries”, Post2017

• Automated storage and retrieval system

• Visual cryptography

Economics

• “The Use and Misuse of Income Data and Extreme Poverty in the United States”, Meyer et al 2021 (measurement error in non-registry surveys of population extremes—not quite “lizardman” but similar problem)

• “Is economics performative? Option theory and the construction of derivatives markets”, Mackenzie2006 (the mechanics of how the Black-Scholes model changed markets: Black ran a service printing “paper” estimating optimal prices for all options which traders could consult & use with simple heuristics to try to arbitrage the market)

• “Whitewood under Siege: On the front lines of the pallet wars” (the competition between the 2 ecosystems of shipping pallets: ‘whitewood’ & ‘blue pallet’)

• Mautam

Philosophy

• “Coping with mortality: responses of monkeys and great apes to collapsed, inanimate and dead conspecifics”, De Marco et al 2021

• Braitenberg vehicle

Fiction

• “Reply of the Zaporozhian Cossacks”

Miscellaneous

• America’s top ace, Major Dick Bong