AI

• ⁠“Controllable Neural Text Generation”⁠, Lilian Weng; ⁠“Recent Advances in Language Model Fine-tuning”, Sebastian Ruder (review)

  • “Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm”⁠, Reynolds & McDonell2021 (original 10-shot Fr → En translation can be beaten by the better 0-shot prompt: “French: XYZ / English:…”; this is “true of most worst-performing prompts…”); “Calibrate Before Use: Improving Few-Shot Performance of Language Models”⁠, Zhao et al 2021 (huge boost from calibrating unstable prompts; both demonstrate, as always⁠, that “sampling can prove the presence of knowledge but not the absence.”)

• “TransGAN: Two Transformers Can Make One Strong GAN”⁠, Jiang et al 2021 (Transformer-only GAN⁠: attention is all you need)

• “PACT: Proof Artifact Co-training for Theorem Proving with Language Models”⁠, Han et al 2021 (GPT-f⁠ for Lean⁠)

• “Towards End-to-End In-Image Neural Machine Translation”⁠, Mansimov et al 2020 (sure why not)

• Brains:

  • ⁠“Artificial Neural Nets Finally Yield Clues to How Brains Learn”⁠; Whittington & Bogacz2019⁠ (short overview of biologically-plausible backprop⁠: feedback alignment, target propagation, predictive coding, & attentional feedback; also of recent interest, VS-ML⁠; given their increasing success in training while respecting more biological constraints, the increasing power of backprop-trained ANNs and the neurological success of ANNs in predicting & imitating brain signals, it is increasingly clear that brains really do do backprop in some sense)

  • “NSD: A massive 7-tesla fMRI dataset to bridge cognitive and computational neuroscience”⁠, Jean et al 2021 (“…The availability of NSD thus opens the door to using brain activity to directly guide the optimization of deep neural networks.”)

  • “Brain2Pix: Fully convolutional naturalistic video reconstruction from brain activity”⁠, Le et al 2021 (reconstructing Dr. Who⁠)

  • “High-performance brain-to-text communication via imagined handwriting”⁠, Willett et al 2020

  • “Brain-computer interface for generating personally attractive images”⁠, Spape et al 2021 (simple EEG-based optimization of ProGAN⁠ faces; many ways to improve this…)

Matters Of Scale⁠:

• “Scaling Laws for Transfer”⁠, Hernandez et al 2021 (“We find that pre-training effectively multiplies the fine-tuning dataset size”; a shot across the bow of anyone floating on a proprietary-dataset moat: large models can drop data requirements by orders of magnitude overnight, even surpassing you)

• “ALIGN: Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision”⁠, Jia et al 2021 (see also CC-12M⁠; CLIP⁠-like w/EfficientNet⁠ trained on 1.8 billion images on a TPUv3⁠-1024—DM⁠ argues that fancier cross-modal Transformers are better, nevertheless, ‘TPUs go brrr’. Given DALL·E 1, CLIP, ALIGN, VDVAE⁠, CW-VAE⁠, AIPO⁠, DCTransformer⁠ neural radiance fields⁠ et al, are GANs already dead, and just don’t realize it yet? Or at least soon to be relegated to only DRL-like uses as a final finetuning phase to sharpen up a self-supervised model?); “WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Training”⁠, Huo et al 2021

• “DALL·E 1: Zero-Shot Text-to-Image Generation”⁠, Ramesh et al 2021 (original blog⁠); “M6: A Chinese Multimodal Pretrainer”⁠, Lin et al 2021 (Chinese DALL·E 1: 1.9TB images/0.29TB text for 10b-parameter dense/100b-parameter MoE Transformer; shockingly fast Chinese replication of DALL·E 1/CLIP)

• “Explaining Neural Scaling Laws”⁠, Bahri et al 2021/“Learning Curve Theory”⁠, Hutter2021 (⁠Rohin Shah commentary⁠; more on the manifold hypothesis)

Genetics

Everything Is Heritable:

• “Phenotypic covariance across the entire spectrum of relatedness for 86 billion pairs of individuals”⁠, Kemper et al 2021

• “Genetic variation, brain, and intelligence differences”⁠, Deary et al 2021

• “Pathfinder: A gamified measure to integrate general cognitive ability into the biological, medical and behavioral sciences”⁠, Malanchini et al 2021 (not the focus, but the IQ PGS⁠ is a slight improvement over Allegrini et al 2018⁠ due to less phenotype measurement error⁠?)

• “Polygenic burden has broader impact on health, cognition, and socioeconomic outcomes than most rare and high-risk copy number variants”⁠, Saarentaus et al 2021

• On candidate-genes & COMT

Recent Evolution:

• “Million-Year-Old DNA Rewrites the Mammoth Family Tree: Genomic data—the oldest ever recovered from a fossil—reveals the origin and evolution of the Columbian mammoth”⁠

• “Kin selection explains the evolution of cooperation in the gut microbiota”⁠, Simonet & McNally2021

Engineering:

• First Black-Footed Ferret cloned⁠

Statistics/Meta-Science

• ⁠“Lessons from Gerolamo Cardano’s The Book of My Life”⁠ (progress studies; see also Newton’s anthropic argument⁠, Bakewell & inventing progress⁠, ⁠The Autobiography of Benvenuto Cellini⁠)

• ⁠“How Many Microcovids Would You Spend on a Burrito?”⁠ (on the microCOVID Project Calculator)

• ⁠On Piffles

• “Artifact and Recording Concepts in EEG”⁠, Tatum et al 2011 (on the EEG⁠ signals of Jell-O⁠, or, the importance of negative controls⁠)

Politics/Religion

• ⁠Fads⁠: “The Logic of Fashion Cycles”⁠, Acerbi et al 2012; “Fashion and art cycles are driven by counter-dominance signals of elite competition: quantitative evidence from music styles”⁠, Klimek et al 2019; “The hipster effect: When anti-conformists all look the same”⁠, Touboul2019; ⁠“Right Is The New Left”⁠, ⁠Scott Alexander⁠ (see also Han et al 2010⁠, Downs1972⁠/Gupta & Jenkins-Smith2015⁠, Lorenz-Spreen et al 2019⁠/Candia et al 2019⁠, Loury1994⁠)

• ⁠“What can we learn from the lunar pandemic that never was?” (NASA’s lunar quarantine was a sham intended to mollify the public as they covered up repeated major failures & lab leaks both before & after—had there been any dangerous lunar organisms, they would have escaped easily)

• MrBeast⁠ (the new aristocracy of⁠ prestige⁠? Borrowed plumage, perhaps, but effective…)

• “Russia’s new Lysenkoism”⁠, Kolchinsky et al 2017

Psychology/Biology

• “Lessons from the host defences of bats, a unique viral reservoir”⁠, Irving et al 2021 (bat-borne viruses⁠; previously, Trevor Klee)

• “Beneficial & Detrimental Effects of Reactive Oxygen Species on Lifespan: A Comprehensive Review of Comparative & Experimental Studies”⁠, Shields et al 2021 (antioxidants still aren’t the fountain of youth, and may be harmful; animal studies still frequently inconsistent)

• “Positive expectations predict improved mental-health outcomes linked to psychedelic microdosing”⁠, Kaertner et al 2021 (placebo)

• “The Effects of Fluoride in Drinking Water”⁠, Aggeborn & Öhman 2021

• “Sleep & Sex: What Can Go Wrong? A Review of the Literature on Sleep Related Disorders and Abnormal Sexual Behaviors & Experiences”⁠, Schenck et al 2007

Technology

• New X-Prize: $100m in prizes for Carbon Removal

• Wringing gauge blocks⁠ (“With their precisely-flat metal faces, gauge blocks can be stuck together non-magnetically via a process calling ‘wringing’, requiring substantial effort to separate. Scientists are still uncertain exactly how wringing works.”)

• Armored train⁠

Economics

• ⁠“Why did renewables become so cheap so fast? And what can we do to use this global opportunity for green growth?”⁠, Max Roser (specifically, why such an extreme experience curve⁠?)

• “IQ, trading behavior, and performance”⁠, Grinblatt et al 2012; “Genetic Endowments and Wealth Inequality”⁠, Barth et al 2020 (why, despite notorious setbacks, did Isaac Newton & LTCM’s founders die wealthy? Why, in general, are more intelligent people so much better investors? ‘The indifference of the indicator’: it’s not one thing, it’s everything—more intelligent people have lower discount rates, save more for longer & are less risk-averse, more accurately predict future growth or inflation, are more likely to participate in +EV⁠ opportunities like the stock market, to use low-fee rather than high-fee (and thus, underperforming) mutual funds, succumb less to biases like herding as they trade better & at better times, trade less, and harvest losses more efficiently when trading poorly.)

Philosophy

• Are ethics experts more ethical?⁠ “The Behavior of Ethicists”⁠, Schwitzgebel & Rust2016 (most recently: “The moral behavior of ethics professors: A replication-extension in German-speaking countries”⁠, Schönegger et al 2019; given moral licensing & activism, perhaps we should be surprised we don’t hear about more ethicists doing things like trying to dox reviewers, posting enemy lists, or dumping files to leak. “Woe to you Pharisees!”)

• “Meta-analysis on belief in free will manipulations”⁠, Genschow et al 2021 (another noble lie⁠ turns out to be ignoble)

• Gricean maxims of communication⁠

Fiction

• Bunnies & Burrows⁠

Miscellaneous

• ⁠“Caesar Lives”⁠, Iggy Pop⁠ 1995_30ya (on Gibbon⁠)

• Mad honey⁠

• Imperial Court System⁠