Bibliography (125):

1. Gwern.net newsletter (Substack subscription page)

2. ​ April 2021 News

3. ​ ‘newsletter’ directory

4. ​ Changelog

5. Gwern Branwen Creating Essays on Gwern.net

6. ​ Choose-Your-Own-Adventure AI Dungeon Games

7. ​ GPT-2 Preference Learning for Music Generation § Decision Transformers: Preference Learning As Simple As Possible

8. ML Scaling subreddit

9. Podracer architectures for scalable Reinforcement Learning

10. Google details new AI accelerator chips

11. ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning

12. GSPMD: General and Scalable Parallelization for ML Computation Graphs

13. GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism

14. GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding

15. DeepSpeed: Extreme-scale model training for everyone

16. High-performance, Distributed Training of Large-scale Deep Learning Recommendation Models (DLRMs)

17. DynamicEmbedding: Extending TensorFlow for Colossal-Scale Applications

18. RecPipe: Co-designing Models and Hardware to Jointly Optimize Recommendation Quality and Performance

19. From Motor Control to Team Play in Simulated Humanoid Football

20. Grounded Language Learning Fast and Slow

21. Unsupervised Speech Recognition

22. Anthropic raises $124 million to build more reliable, general AI systems

23. Cooperative AI Foundation (CAIF)

24. MLP-Mixer: An all-MLP Architecture for Vision

25. ​ ‘MLP NN’ directory

26. The Bitter Lesson

27. Data-Efficient Language-Supervised Zero-Shot Learning with Self-Distillation

28. Distilling the Knowledge in a Neural Network

29. Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning

30. Image Super-Resolution via Iterative Refinement

31. Diffusion Models Beat GANs on Image Synthesis

32. Denoising Diffusion Probabilistic Models

33. Generative Modeling by Estimating Gradients of the Data Distribution

34. Large Scale GAN Training for High Fidelity Natural Image Synthesis

35. Improved Denoising Diffusion Probabilistic Models

36. CDM: Cascaded Diffusion Models for High Fidelity Image Generation

37. Learning to summarize from human feedback

38. Grokking: Generalization Beyond Overfitting On Small Algorithmic Datasets

39. https://www.reddit.com/r/mlscaling/comments/n78584/grokking_generalization_beyond_overfitting_on/

40. 2021-power-poster.png#openai

41. Knowledge distillation: A good teacher is patient and consistent

42. Scaling End-to-End Models for Large-Scale Multilingual ASR

43. The Shape of Learning Curves: a Review

44. Reward is enough

45. Inverse-Scaling/prize: A Prize for Finding Tasks That Cause Large Language Models to Show Inverse Scaling

46. Lazy, a tool for running things in idle time

47. GPT-J-6B: 6B JAX-Based Transformer

48. Aggregating Nested Transformers

49. Less is More: Pay Less Attention in Vision Transformers

50. ByT5: Towards a token-free future with pre-trained byte-to-byte models

51. Machine learning attacks against the Asirra CAPTCHA

52. Deep Big Multilayer Perceptrons for Digit Recognition

53. Bi-ancestral depression GWAS in the Million Veteran Program and meta-analysis in >1.2 million individuals highlight new therapeutic directions

54. The complete sequence of a human genome

55. A complete human genome sequence is close: how scientists filled in the gaps. Researchers added 200 million DNA base pairs and 115 protein-coding genes—but they’ve yet to entirely sequence the Y chromosome

56. A complete reference genome improves analysis of human genetic variation

57. Using DNA to predict intelligence

58. Long read sequencing of 3,622 Icelanders provides insight into the role of structural variants in human diseases and other traits

59. Rapid Sequencing-Based Diagnosis of Thiamine Metabolism Dysfunction Syndrome

60. Sense codon reassignment enables viral resistance and encoded polymer synthesis

61. In vivo CRISPR base editing of PCSK9 durably lowers cholesterol in primates

62. Partial recovery of visual function in a blind patient after optogenetic therapy

63. With engineered proteins, scientists use optogenetics for the first time to help a blind patient see again

64. Wireless multilateral devices for optogenetic studies of individual and social behaviors

65. Scientists Drove Mice to Bond by Zapping Their Brains With Light: The study, a tour de force in bioengineering, comes after two decades of research on brain-to-brain synchrony in people

66. High-throughput functional variant screens via in vivo production of single-stranded DNA

67. First genetically modified mosquitoes released in the United States: Biotech firm Oxitec launches controversial field test of its insects in Florida after years of push-back from residents and regulatory complications

68. Genomic characterization of world’s longest selection experiment in mouse reveals the complexity of polygenic traits

69. Surrogate broodstock to enhance biotechnology research and applications in aquaculture

70. Utility of polygenic embryo screening for disease depends on the selection strategy

71. Limit on lab-grown human embryos dropped by stem-cell body: The International Society for Stem Cell Research relaxed the famous 14-day rule on culturing human embryos in its latest research guidelines

72. Useful Mutants, Bred With Radiation

73. Correlated Failures

74. How a Publicity Blitz Created The Myth of Subliminal Advertising

75. Clarifying the Structure and Nature of Left-Wing Authoritarianism (LWA)

76. Book Review: The Decline and Fall of the Roman Empire

77. Highlights from The Decline and Fall of the Roman Empire

78. A connectomic study of a petascale fragment of human cerebral cortex

79. Saturated Reconstruction of a Volume of Neocortex

80. Accelerating progress in brain recording tech

81. Neuroimaging evidence for a network sampling theory of individual differences in human intelligence test performance

82. The neural basis of intelligence in fine-grained cortical topographies

83. Predicting intelligence from brain gray matter volume

84. The Parieto-Frontal Integration Theory (P-FIT) of intelligence: Converging neuroimaging evidence

85. ​ ‘NN sparsity’ directory

86. ​ backstop#deep-bayes

    [Transclude the forward-link's context]

87. Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth

88. MDMA-assisted therapy for severe PTSD: a randomized, double-blind, placebo-controlled phase 3 study

89. Effects of Psilocybin-Assisted Therapy on Major Depressive Disorder: A Randomized Clinical Trial

90. https://www.newyorker.com/magazine/2021/04/05/why-animals-dont-get-lost

91. In The Future Of Collecting, Is Anyone Having Fun?

92. Linking Brain Biology to Intellectual Endowment: A Review on the Associations of Human Intelligence With Neuroimaging Data

93. The Best And The Rest: Revisiting The Norm Of Normality Of Individual Performance

94. ​ Leaky Pipelines

95. A conserved strategy for inducing appendage regeneration

96. https://www.astralcodexten.com/p/know-your-amphetamines

97. Feeling Small: Exploring the Tactile Perception Limits

98. One Writer Enters International Competition to Play the World-Conquering Game That Redefines What It Means to Be a Geek (And a Person)

99. How Counting Neutrons Explains Nuclear Waste

100. Making Is Show Business Now

101. Shop Class As Soulcraft: The Case for the Manual Trades

102. https://www.kickstarter.com/projects/upperstory/spintronics-build-mechanical-circuits

103. RCTs to Scale: Comprehensive Evidence from Two Nudge Units

104. No causal associations between childhood family income and subsequent psychiatric disorders, substance misuse and violent crime arrests: a nationwide Finnish study of >650 000 individuals and their siblings

105. Parental income and mental disorders in children and adolescents: prospective register-based study

106. Everything You Might Want to Know about Whaling

107. ​ The Exploding Nash 2-Of-2 NashX Equilibrium

108. Love Is the Plan the Plan Is Death

109. The Strange Story of Dagobert, the ‘DuckTales’ Bandit

110. Nordic Quack: Sweden’s bizarre tradition of watching Donald Duck cartoons on Christmas Eve

111. Wikipedia Bibliography:

     1. Neural scaling law  :

        https://en.wikipedia.org/wiki/Neural_scaling_law

     2. Optogenetics

     3. Atomic gardening  :

        https://en.wikipedia.org/wiki/Atomic_gardening

     4. Animal navigation  :

        https://en.wikipedia.org/wiki/Animal_navigation

     5. Bobblehead  :

        https://en.wikipedia.org/wiki/Bobblehead

     6. Diplomacy (game)

     7. Turing Tumble  :

        https://en.wikipedia.org/wiki/Turing_Tumble

     8. James Tiptree Jr

     9. Love Is the Plan the Plan Is Death  :

        https://en.wikipedia.org/wiki/Love_Is_the_Plan_the_Plan_Is_Death

     10. Arno Funke  :

         https://en.wikipedia.org/wiki/Arno_Funke

     11. John D. Rockerduck  :

         https://en.wikipedia.org/wiki/John_D._Rockerduck

     12. From All of Us to All of You  :

         https://en.wikipedia.org/wiki/From_All_of_Us_to_All_of_You

     13. Atmospheric optics § List  :

         https://en.wikipedia.org/wiki/Atmospheric_optics#List

     14. Franz Nopcsa von Felső-Szilvás  :

         https://en.wikipedia.org/wiki/Franz_Nopcsa_von_Fels%C5%91-Szilv%C3%A1s

     15. Krishnacore  :

         https://en.wikipedia.org/wiki/Krishnacore