February 2021 News
February 2021 Gwern.net newsletter with links on AI scaling, semaglutide, and ethicist ethics.
February 2021’s Gwern.net newsletter is now out; previous, January 2021 (archives). This is a collation of links and summary of major changes, overlapping with my Changelog; brought to you by my donors on Patreon.
Writings
-
Gwern.net: popups: can now be moved, stickied, and full-screened (another step towards our ambition of Windows-95-in-the-browser!)
Links
AI
-
“Controllable Neural Text Generation”, Lilian Weng; “Recent Advances in Language Model Fine-tuning”, Sebastian Ruder (review)
-
“Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm”, Reynolds & 2021 (original 10-shot Fr → En translation can be beaten by the better 0-shot prompt: “French: XYZ / English:…”; this is “true of most worst-performing prompts…”); “Calibrate Before Use: Improving Few-Shot Performance of Language Models”, et al 2021 (huge boost from calibrating unstable prompts; both demonstrate, as always, that “sampling can prove the presence of knowledge but not the absence.”)
-
-
“TransGAN: Two Transformers Can Make One Strong GAN”, et al 2021 (Transformer-only GAN: attention is all you need)
-
“PACT: Proof Artifact Co-training for Theorem Proving with Language Models”, et al 2021 ( GPT-f for Lean)
-
“Towards End-to-End In-Image Neural Machine Translation”, et al 2020 (sure why not)
-
Brains:
-
“Artificial Neural Nets Finally Yield Clues to How Brains Learn”; 2019 (short overview of biologically-plausible backprop: feedback alignment, target propagation, predictive coding, & attentional feedback; also of recent interest, VS-ML; given their increasing success in training while respecting more biological constraints, the increasing power of backprop-trained ANNs and the neurological success of ANNs in predicting & imitating brain signals, it is increasingly clear that brains really do do backprop in some sense)
-
“NSD: A massive 7-tesla fMRI dataset to bridge cognitive and computational neuroscience”, et al 2021 (“…The availability of NSD thus opens the door to using brain activity to directly guide the optimization of deep neural networks.”)
-
“Brain2Pix: Fully convolutional naturalistic video reconstruction from brain activity”, et al 2021 (reconstructing Dr. Who)
-
“High-performance brain-to-text communication via imagined handwriting”, et al 2020
-
“Brain-computer interface for generating personally attractive images”, et al 2021 (simple EEG-based optimization of ProGAN faces; many ways to improve this…)
-
-
“Scaling Laws for Transfer”, et al 2021 (“We find that pre-training effectively multiplies the fine-tuning dataset size”; a shot across the bow of anyone floating on a proprietary-dataset moat: large models can drop data requirements by orders of magnitude overnight, even surpassing you)
-
“ALIGN: Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision”, et al 2021 (see also CC-12M; CLIP-like w/EfficientNet trained on 1.8 billion images on a TPUv3-1024—DM argues that fancier cross-modal Transformers are better, nevertheless, ‘TPUs go brrr’. Given DALL·E 1, CLIP, ALIGN, VDVAE, CW-VAE, AIPO, DCTransformer neural radiance fields et al, are GANs already dead, and just don’t realize it yet? Or at least soon to be relegated to only DRL-like uses as a final finetuning phase to sharpen up a self-supervised model?); “WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Training”, et al 2021
-
“DALL·E 1: Zero-Shot Text-to-Image Generation”, et al 2021 ( original blog); “M6: A Chinese Multimodal Pretrainer”, et al 2021 (Chinese DALL·E 1: 1.9TB images/0.29TB text for 10b-parameter dense/100b-parameter MoE Transformer; shockingly fast Chinese replication of DALL·E 1/CLIP)
-
“Explaining Neural Scaling Laws”, et al 2021/ “Learning Curve Theory”, 2021 (Rohin Shah commentary; more on the manifold hypothesis)
Genetics
Everything Is Heritable:
-
“Phenotypic covariance across the entire spectrum of relatedness for 86 billion pairs of individuals”, et al 2021
-
“Genetic variation, brain, and intelligence differences”, et al 2021
-
“Pathfinder: A gamified measure to integrate general cognitive ability into the biological, medical and behavioral sciences”, et al 2021 (not the focus, but the IQ PGS is a slight improvement over et al 2018 due to less phenotype measurement error?)
Recent Evolution:
Engineering:
Statistics/Meta-Science
-
“Lessons from Gerolamo Cardano’s The Book of My Life” (progress studies; see also Newton’s anthropic argument, Bakewell & inventing progress, The Autobiography of Benvenuto Cellini)
-
“How Many Microcovids Would You Spend on a Burrito?” (on the microCOVID Project Calculator)
-
“Artifact and Recording Concepts in EEG”, Tatum et al 201113ya (on the EEG signals of Jell-O, or, the importance of negative controls)
Politics/Religion
-
Fads: “The Logic of Fashion Cycles”, Acerbi et al 201212ya; “Fashion and art cycles are driven by counter-dominance signals of elite competition: quantitative evidence from music styles”, et al 2019; “The hipster effect: When anti-conformists all look the same”, 2019; “Right Is The New Left”, Scott Alexander (see also et al 2010 , 1972/Gupta & Jenkins-2015, Lorenz-et al 2019 /et al 2019 , 1994)
-
“What can we learn from the lunar pandemic that never was?” (NASA’s lunar quarantine was a sham intended to mollify the public as they covered up repeated major failures & lab leaks both before & after—had there been any dangerous lunar organisms, they would have escaped easily)
-
MrBeast (the new aristocracy of prestige? Borrowed plumage, perhaps, but effective…)
-
“Russia’s new Lysenkoism”, et al 2017
Psychology/Biology
-
“Lessons from the host defences of bats, a unique viral reservoir”, et al 2021 ( bat-borne viruses; previously, Trevor Klee)
-
“Beneficial & Detrimental Effects of Reactive Oxygen Species on Lifespan: A Comprehensive Review of Comparative & Experimental Studies”, et al 2021 (antioxidants still aren’t the fountain of youth, and may be harmful; animal studies still frequently inconsistent)
-
“Positive expectations predict improved mental-health outcomes linked to psychedelic microdosing”, et al 2021 (placebo)
-
“The Effects of Fluoride in Drinking Water”, Aggeborn & Öhman 2021
Semaglutide
-
WP: “Once-Weekly Semaglutide in Adults with Overweight or Obesity”, et al 2021; “Effect of Subcutaneous Semaglutide vs Placebo as an Adjunct to Intensive Behavioral Therapy on Body Weight in Adults With Overweight or Obesity: The STEP 3 Randomized Clinical Trial”, et al 2021
A longer-acting version of the insulin/appetite peptide liraglutide, semaglutide greatly reduces weight, fat, blood sugar, cholesterol etc, with an upcoming oral version; background: et al 2020 , et al 2019 , 2019, et al 2018 , et al 2017 , et al 2016 , et al 2015 .
Quick-fixes like semaglutide may be our only hope, however unvirtuous they seem, because society is fixed but biology mutable.
Technology
-
Wringing gauge blocks (“With their precisely-flat metal faces, gauge blocks can be stuck together non-magnetically via a process calling ‘wringing’, requiring substantial effort to separate. Scientists are still uncertain exactly how wringing works.”)
Economics
-
“Why did renewables become so cheap so fast? And what can we do to use this global opportunity for green growth?”, Max Roser (specifically, why such an extreme experience curve?)
-
“IQ, trading behavior, and performance”, Grinblatt et al 201212ya; “Genetic Endowments and Wealth Inequality”, et al 2020 (why, despite notorious setbacks, did Isaac Newton & LTCM’s founders die wealthy? Why, in general, are more intelligent people so much better investors? ‘The indifference of the indicator’: it’s not one thing, it’s everything—more intelligent people have lower discount rates, save more for longer & are less risk-averse, more accurately predict future growth or inflation, are more likely to participate in +EV opportunities like the stock market, to use low-fee rather than high-fee (and thus, underperforming) mutual funds, succumb less to biases like herding as they trade better & at better times, trade less, and harvest losses more efficiently when trading poorly.)
Philosophy
-
Are ethics experts more ethical? “The Behavior of Ethicists”, 2016 (most recently: “The moral behavior of ethics professors: A replication-extension in German-speaking countries”, et al 2019; given moral licensing & activism, perhaps we should be surprised we don’t hear about more ethicists doing things like trying to dox reviewers, posting enemy lists, or dumping files to leak. “Woe to you Pharisees!”)
-
“Meta-analysis on belief in free will manipulations”, et al 2021 (another noble lie turns out to be ignoble)
Fiction
Miscellaneous
-
“Caesar Lives”, Iggy Pop 199529ya (on Gibbon)