--- title: June 2020 News description: June 2020 Gwern.net newsletter with 3 new pages/essays, and links on CRISPR, population screening, AI scaling, politics, and technological unemployment. created: 2019-12-26 status: finished previous: /newsletter/2020/05 next: /newsletter/2020/07 confidence: log cssExtension: dropcaps-de-zs backlink: False ... June 2020's [Gwern.net](/newsletter/2020/06 "'June 2020 News', Branwen 2019") [newsletter](https://gwern.substack.com/ "'Gwern.net newsletter (Substack subscription page)', Branwen 2013") is now out; previous, [May 2020](/newsletter/2020/05 "'May 2020 News', Branwen 2019") ([archives](/doc/newsletter/index)). This is a collation of links and summary of major changes, overlapping with my [Changelog](/changelog); brought to you by my donors on [Patreon](https://www.patreon.com/gwern). # Writings - [GPT-3 Creative Writing](/gpt-3 "Creative writing by OpenAI's GPT-3 model, demonstrating poetry, dialogue, puns, literary parodies, and storytelling.") - ["PALM: The PALM Anime Location Model And Dataset"](/crop#hands "PALM ('PALM Anime Locator Model') is a dataset of k=5,382 anime-style Danbooru2019 images annotated with the locations of _n_ = 14,394 hands, a YOLOv3 model trained using those annotations to detect hands in anime-style images, and a second dataset of _n_ = 96,534 hands cropped from the Danbooru2019 dataset using the PALM YOLO model and _n_ = 58,536 of them upscaled to ≥512px.") - ["Tech Holy Wars are Coordination Problems"](/holy-war "Flamewars over platforms & upgrades are so bitter not because people are jerks but because the choice will influence entire ecosystems, benefiting one platform through network effects & avoiding 'bitrot' while subtly sabotaging the rest through 'bitcreep'.") # Media ## Links **Genetics**: - [Everything Is Heritable]{.smallcaps}: - ["Efficient polygenic risk scores for biobank scale data by exploiting phenotypes from inferred relatives"](https://www.nature.com/articles/s41467-020-16829-x), Truong et al 2020 (PGSes are not optimal for pure predictive power; [BLUP](https://en.wikipedia.org/wiki/Best_linear_unbiased_prediction)/mixed-models greatly outperform them; see [Gianola & Rosa 2015](/doc/genetics/selection/artificial/2015-gianola.pdf "One Hundred Years of Statistical Developments in Animal Breeding")) - ["Germline mutation rates in young adults predict longevity and reproductive lifespan"](https://www.nature.com/articles/s41598-020-66867-0), Cawthon et al 2020 - *Sequence everyone*: ["Whole-genome sequencing of rare disease patients in a national healthcare system"](https://www.biorxiv.org/content/10.1101/507244.full), Ouwehand et al 2020; ["Genomic analyses implicate noncoding de novo variants in congenital heart disease"](/doc/genetics/heritable/rare/2020-richter.pdf), Richter et al 2020; ["Genetic ancestry analysis on >93,000 individuals undergoing expanded carrier screening reveals limitations of ethnicity-based medical guidelines"](https://www.nature.com/articles/s41436-020-0869-3), Kaseniit 2020; ["An integrated polygenic and clinical risk tool enhances coronary artery disease prediction"](https://www.medrxiv.org/content/10.1101/2020.06.01.20119297.full), Aguilera et al 2020 - [Recent Evolution]{.smallcaps}: - ["Evidence of Polygenic Adaptation in Sardinia at Height-Associated Loci Ascertained from the Biobank Japan"](/doc/genetics/selection/natural/human/2020-chen.pdf){#chen-et-al-2020-heightselection}, Chen et al 2020 - [Engineering]{.smallcaps}: - [Successful treatment of Beta Thalassemia & Sickle Cell Anemia in humans with CRISPR](https://www.newscientist.com/article/2246020-three-people-with-inherited-diseases-successfully-treated-with-crispr/ "Three people with inherited diseases successfully treated with CRISPR") - ["Preimplantation Genetic Testing for Polygenic Disease Relative Risk Reduction: Evaluation of Genomic Index Performance in 11,883 Adult Sibling Pairs"](https://www.mdpi.com/2073-4425/11/6/648), Treff et al 2020 (previously: [Karavani et al 2019](/doc/genetics/selection/artificial/2019-karavani.pdf "Screening Human Embryos for Polygenic Traits Has Limited Utility") & [Treff et al 2019](https://www.frontiersin.org/articles/10.3389/fendo.2019.00845/full "Utility and First Clinical Application of Screening Embryos for Polygenic Disease Risk Reduction"); while [I have concluded](/embryo-selection "'Embryo Selection For Intelligence', Branwen 2016") that transhumanists need not be interested in embryo selection because it will not be causing major societal changes in the next decade or two compared to the much more rapid progress in AI & it increasingly looks like more powerful paradigms will arrive too late to matter, it *is* still happening) - ["A Year In, 1^st^ Patient To Get CRISPR Gene-Editing For Sickle Cell Disease Is Thriving"](https://www.npr.org/sections/health-shots/2020/06/23/877543610/a-year-in-1st-patient-to-get-gene-editing-for-sickle-cell-disease-is-thriving "'A Year In, 1st Patient To Get Gene Editing For Sickle Cell Disease Is Thriving', Stein 2020") **AI**: - ["Image Augmentations for GAN Training"](https://arxiv.org/abs/2006.02595#google), Zhao et al 2020b; [Tran et al 2020](https://arxiv.org/abs/2006.05338 "Towards Good Practices for Data Augmentation in GAN Training"); [Karras et al 2020](https://arxiv.org/abs/2006.06676#nvidia "StyleGAN2-ADA: Training Generative Adversarial Networks with Limited Data"); [Zhao et al 2020c](https://arxiv.org/abs/2006.10738 "Differentiable Augmentation for Data-Efficient GAN Training")^[A different Zhao.] ([Commentary](https://github.com/tensorfork/tensorfork/issues/35)---correct data augmentation for GANs: surprisingly simple & effective, and invented simultaneously 4 times in the past month; see also [Zhao et al 2020a](https://arxiv.org/abs/2002.04724 "Improved Consistency Regularization for GANs").) - ["SBR: Learning to Play No-Press Diplomacy with Best Response Policy Iteration"](https://arxiv.org/abs/2006.04635#deepmind "'Learning to Play No-Press Diplomacy with Best Response Policy Iteration', Anthony et al 2020"), Anthony et al 2020 (natural language [_Diplomacy_](https://en.wikipedia.org/wiki/Diplomacy_\(game\)) agents surely can't be too much more difficult given NLM progress...) - ["Exploration Strategies in Deep Reinforcement Learning"](https://lilianweng.github.io/lil-log/2020/06/07/exploration-strategies-in-deep-reinforcement-learning.html), Lilian Weng - [Matters Of Scale]{.smallcaps}: - ["OpenAI API"](https://openai.com/blog/openai-api/) (GPT-3-as-a-service: [demos](https://platform.openai.com/ "'OpenAI API Beta homepage', OpenAI 2020"); eg. [code writing video](https://player.vimeo.com/video/426819809)) - ["GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding"](https://arxiv.org/abs/2006.16668#google), Lepikhin et al 2020 Training a 600b-parameter NN translation model for 100 languages; +13.5 BLEU; [training 1t-parameter models already tested](https://twitter.com/lepikhin/status/1278174444528132098 "We ran MoE (2048E,60L) with bfloat16 activations with total of 1 trillion model weights. Although trainable with manual diagnostics, with deep 1 trillion model we encountered several trainability issues with numerical stability. Will follow up."). Note that while 600b is impressive, as a [mixture of experts](https://en.wikipedia.org/wiki/Mixture_of_experts) (MoE), it is more like thousands of much smaller highly-redundant models pasted together---this has great performance advantages in training & querying, but means it is more analogous to a regular NMT model with, say, 60b-parameters rather than 600b. - [iGPT](https://openai.com/research/image-gpt "'Image GPT (iGPT): We find that, just as a large transformer model trained on language can generate coherent text, the same exact model trained on pixel sequences can generate coherent image completions and samples. By establishing a correlation between sample quality and image classification accuracy, we show that our best generative model also contains features competitive with top convolutional nets in the unsupervised setting', Chen et al 2020"){#chen-et-al-2020-igpt-blog}: [Chen et al 2020](/doc/ai/nn/transformer/gpt/dall-e/1/2020-chen-2.pdf#openai "Generative Pretraining from Pixels"){#chen-et-al-2020-igpt-paper} Literally GPT-2 but for 64px images---simply train a big Transformer to predict the next pixel, and it catches up with the fancier semi-supervised methods like SimCLR. This eschews any metric learning or bootstrapping or non-quadratic attention mechanisms like [Sparse Transformers](https://openai.com/research/sparse-transformer "Generative Modeling with Sparse Transformers: We’ve developed the Sparse Transformer, a deep neural network which sets new records at predicting what comes next in a sequence—whether text, images, or sound. It uses an algorithmic improvement of the attention mechanism to extract patterns from sequences 30× longer than possible previously."), but with enough compute still works... "Attention is all you need"? The new efficient attentions may break the quadratic bottleneck limiting current uses. - ["Bajic, and other chip teams, are responding to...neural net models that Bajic said he couldn't talk about publicly that will have on the order of one-trillion parameters."](https://www.zdnet.com/article/startup-tenstorrent-and-competitors-show-how-computing-is-changing-ai-and-vice-versa/ "Startup Tenstorrent shows AI is changing computing and vice versa: Tenstorrent is one of the rush of AI chip makers founded in 2016 and finally showing product. The new wave of chips represent a substantial departure from how traditional computer chips work, but also point to ways that neural network design may change in the years to come.") - *2018 AI budgets*: OpenAI: ~[\$51m](/doc/reinforcement-learning/openai/2018-openai-form990.pdf "'Form 990 [OpenAI 2018]', Clark 2019") (\$31m cloud compute); DeepMind: ~[\$700m](/doc/reinforcement-learning/deepmind/2018-deepmind-fullaccounts.pdf "'Deepmind Technologies Limited: Report and Financial Statements Year Ended 31 December 2018 [Full Accounts]', Harris 2019")[^filings] - ["How big should my language model be?"](https://huggingface.co/spaces/teven-projects/calculator "'How Big Should My Language Model Be?', Scao 2020"), Huggingface 2020 [^filings]: Full list of filings: - OpenAI Form 990s: [2016](/doc/reinforcement-learning/openai/2016-openai-form990.pdf "'Form 990 [OpenAI 2016]', Clark 2016"), [2017](/doc/reinforcement-learning/openai/2017-openai-form990.pdf "'Form 990 [OpenAI 2017]', Clark 2019"), [2018](/doc/reinforcement-learning/openai/2018-openai-form990.pdf "'Form 990 [OpenAI 2018]', Clark 2019") - DeepMind 'full accounts' ([Companies House](https://en.wikipedia.org/wiki/Companies_House)): [2014](/doc/reinforcement-learning/deepmind/2014-deepmind-fullaccounts.pdf)/[2015](/doc/reinforcement-learning/deepmind/2015-deepmind-fullaccounts.pdf "'Audited Financial Statements DeepMind Technologies Limited For the year ended 31 December 2015 [Full Accounts]', Harris 2016")/[2016](/doc/reinforcement-learning/deepmind/2016-deepmind-fullaccounts.pdf "‘Audited Financial Statements DeepMind Technologies Limited For the year ended 31 December 2016 [Full Accounts]’, Yi 2016")/[2017](/doc/reinforcement-learning/deepmind/2017-deepmind-fullaccounts.pdf "'Deepmind Technologies Limited: Report and Financial Statements Year Ended 31 December 2017 [Full Accounts]', Harris 2018")/[2018](/doc/reinforcement-learning/deepmind/2018-deepmind-fullaccounts.pdf "'Deepmind Technologies Limited: Report and Financial Statements Year Ended 31 December 2018 [Full Accounts]', Harris 2019")/[2019](/doc/reinforcement-learning/deepmind/2019-deepmind-fullaccounts.pdf "'Deepmind Technologies Limited: Report and Financial Statements Year Ended 31 December 2019 [Full Accounts]', Harris 2020") **Politics/religion**: - [Fads](/note/fashion "'Fashion Cycles', Branwen 2021"): ["Accelerating dynamics of collective attention"](https://www.nature.com/articles/s41467-019-09311-w), Lorenz-Spreen et al 2019 (the [issue-attention cycle](/doc/sociology/1972-downs.pdf "'Up and down with ecology - the `issue-attention cycle`', Downs 1972") speeds up? See also ["The universal decay of collective memory and attention"](/doc/culture/2019-candia.pdf), Candia et al 2019) - ["Radical Chic: That Party at Lenny’s"](https://nymag.com/article/tom-wolfe-radical-chic-that-party-at-lennys.html), [Tom Wolfe](https://en.wikipedia.org/wiki/Tom_Wolfe) [1970](https://en.wikipedia.org/wiki/Radical_Chic_%26_Mau-Mauing_the_Flak_Catchers "Radical Chic & Mau-Mauing the Flak Catchers") (previously: ["Mau-Mauing the Flak Catchers"](https://teageegeepea.tripod.com/maumau.html)) - ["Agenda Seeding: How 1960s Black Protests Moved Elites, Public Opinion and Voting"](/doc/sociology/2020-wasow.pdf), Wasow 2020 - ["The Cost of Torture: Evidence from the Spanish Inquisition"](/doc/history/2020-hassner.pdf), Hassner 2020 - ["Breast-Feeding of Animals by Women: Its Socio-Cultural Context and Geographic Occurrence"](/doc/sociology/1982-simoons.pdf), Simoons & Baldwin 1982 (from ["Did Breast-Feeding Play A Role In the Evolution of Pets?"](https://www.psychologytoday.com/us/blog/animals-and-us/201908/did-breast-feeding-play-role-in-the-evolution-pets "'Like the dolphin who adopted a baby whale, humans have often breast-fed pets', Herzog 2019") via ["Animal Ethics and Evolutionary Psychology---10 ideas"](https://dianaverse.com/2020/06/15/evpsychandanimalethics/ "Diana Fleischman 2020")) - ["Incest Laws and Absent Taboos in Roman Egypt"](/doc/history/2005-strong.pdf), Strong 2005 **Psychology/biology**: - ["The Curse of Konzo: In 1981, an international group of doctors identified the devastating disease behind a perplexing outbreak of paralysis in northern Mozambique"](https://www.damninteresting.com/the-curse-of-konzo/) (the investigation of [cyanide poisoning](https://en.wikipedia.org/wiki/Konzo) by, among others, [Hans Rosling](https://en.wikipedia.org/wiki/Hans_Rosling)) - [Scintillating scotoma](https://en.wikipedia.org/wiki/Scintillating_scotoma) (['typical mind fallacy'](https://www.lesswrong.com/posts/baTWMegR42PAsH9qJ/generalizing-from-one-example "'Generalizing From One Example', Scott Alexander 2009") examples: 1% of the population has occasional hour-long visual blindness/hallucination for no particular reason) - ["Why Humans Totally Freak Out When They Get Lost: People really do circle past the same tree over and over again---it doesn't just happen in movies"](https://www.wired.com/story/why-humans-totally-freak-out-when-they-get-lost/) (why [redirected walking](/doc/technology/2018-nilsson.pdf "'15 Years of Research on Redirected Walking in Immersive Virtual Environments', Nilsson et al 2018") works---we're not good at this) - ["Increased weight loading reduces body weight and body fat in obese subjects---A proof of concept randomized clinical trial"](https://www.thelancet.com/journals/eclinm/article/PIIS2589-5370(20)30082-1/fulltext), Ohlsson et al 2020 (the rodent 'gravitostat' replicates in humans with [weight vests](https://en.wikipedia.org/wiki/Weighted_clothing)? previously: [Jansson et al 2018](https://www.pnas.org/doi/10.1073/pnas.1715687114 "Body weight homeostat that regulates fat mass independently of leptin in rats and mice")/[Ohlsson et al 2018](https://academic.oup.com/endo/article/159/7/2676/5001726 "The Gravitostat Regulates Fat Mass in Obese Male Mice While Leptin Regulates Fat Mass in Lean Male Mice")/[Ohlsson & Jansson 2018](https://www.pnas.org/doi/10.1073/pnas.1800116115 "Where does the gravitostat fit in?")/[Palsdottir et al 2019](https://academic.oup.com/endo/article/160/5/1057/5381910 "Interactions Between the Gravitostat and the Fibroblast Growth Factor System for the Regulation of Body Weight"). This raises so many questions if it's really real: which bones? Can we use springs on our legs instead? Do standing desks help? What about acute stresses like squatting, or just holding, say, 140kg on our shoulders for 5 minutes? How simple is the chemical signaling of bone stress? Are the receptors druggable?) - ["Human postprandial responses to food and potential for precision nutrition"](/doc/genetics/heritable/2020-berry.pdf), Berry et al 2020 - ["Fast food outlets, physical activity facilities, and obesity among adults: a nationwide longitudinal study from Sweden"](/doc/sociology/2020-okuyama.pdf), Okuyama et al 2020; ["Association Between Lottery Prize Size and Self-reported Health Habits in Swedish Lottery Players"](/doc/sociology/2020-ostling.pdf), Östling et al 2020 (population registries: where correlations go to die---note that the lottery gets much more precise & near-zero causal effect estimates than previous work, exemplifying the [iron & stainless steel laws](/doc/sociology/1987-rossi "'The Iron Law Of Evaluation And Other Metallic Rules', Rossi 1987")) - ["Health Recommendations and Selection in Health Behaviors"](/doc/sociology/2020-oster.pdf), Oster 2020 (self-fulfilling prophecies in nutrition/diet) - [Johan Bjorksten's cross-linkage theory of aging research (predecessors to LysoSENS)](/doc/longevity/johan-bjorksten/index) (["one man's modus ponens is another man's modus tollens"](/modus#bjorksten)) - ["On the Potential of Silicon as a Building Block for Life"](https://www.mdpi.com/2075-1729/10/6/84/htm), Petkowski et al 2020 (generally infeasible except possibly in a sulfuric acid environment) **Technology**: - ["A Group Is Its Own Worst Enemy"](/doc/technology/2005-shirky-agroupisitsownworstenemy.pdf "'A Group is Its Own Worst Enemy', Shirky 2005"), Clay Shirky 2003/2005 - [METAFONT](https://en.wikipedia.org/wiki/Metafont): ["The Concept of a Meta-Font"](/doc/design/typography/1982-knuth.pdf), [Knuth](https://en.wikipedia.org/wiki/Donald_Knuth) 1982 - ["The Monotype 4-Line System for Setting Mathematics"](/doc/design/typography/tex/2007-rhatigan.pdf), Rhatigan 2007 (the [Monotype system](https://en.wikipedia.org/wiki/Monotype_system) brought [the black art of boiling lead](https://www.practicallyefficient.com/2017/10/13/from-boiling-lead-and-black-art.html "From boiling lead and black art: An essay on the history of mathematical typography") & mathematical typesetting to perfection just in time to be obsoleted by computers---famously spurring Knuth to invent [TeX](https://en.wikipedia.org/wiki/TeX), eventually giving academia the omnipresent look of [LaTeX](https://en.wikipedia.org/wiki/LaTeX); see also [_The Printing of Mathematics_](#chaundy-et-al-1954-2)) - ["Real-world dynamic programming: seam carving"](https://avikdas.com/2019/05/14/real-world-dynamic-programming-seam-carving.html), Avik Das (explanation of [Avidan & Shamir 2007's](/doc/technology/2007-avidan.pdf "Seam carving for content-aware image resizing") [seam carving](https://en.wikipedia.org/wiki/Seam_carving), a fun & simple image editing trick exploiting [dynamic programming](https://en.wikipedia.org/wiki/Dynamic_programming)) - ["The Incredible Story of the U.S. Army's Earth-Shaking, Off-Road Land Trains"](https://www.thedrive.com/news/33645/the-incredible-story-of-the-us-armys-earth-shaking-off-road-land-trains "'The Incredible Story of the US Army's Earth-Shaking, Off-Road Land Trains: Oh, your pickup has a lift? That's cute', Holderith 2020") (the 174-meter-long TC-497 [Overland Train](https://en.wikipedia.org/wiki/Overland_train) Mark II, designed for building the Arctic [DEW](https://en.wikipedia.org/wiki/Distant_Early_Warning_Line), could haul 150 tons >640km; its predecessor, the VC-22 [Sno-Freighter](https://en.wikipedia.org/wiki/Sno-Freighter) was designed & built in a month) - ["When _SimCity_ got serious: the story of Maxis Business Simulations and _SimRefinery_"](https://obscuritory.com/sim/when-simcity-got-serious/ "'When SimCity got serious: the story of Maxis Business Simulations and SimRefinery', Salvador 2020") **Economics**: -
["Automation As Colonization Wave"](/doc/economics/automation/index)
**Philosophy**: - ["Schrödinger’s Zombie: Adam Brown"](https://forums.fqxi.org/d/3345 "'Schrödinger’s Zombie: Adam Brown at the 6th FQXi Meeting', Musser 2019") (the [quantum bomb tester](https://en.wikipedia.org/wiki/Elitzur%E2%80%93Vaidman_bomb_tester) can ['photograph' objects](https://arxiv.org/abs/quant-ph/9803060 "'`Interaction-Free` Imaging', White et al 1998") & ['run' computers](https://arxiv.org/abs/quant-ph/9805086 "'Quantum Effects in Algorithms', Jozsa 1998") to [compute counterfactually](https://en.wikipedia.org/wiki/Counterfactual_quantum_computation) without [any interaction](https://en.wikipedia.org/wiki/Interaction-free_measurement); what if instead you use a simulation or a real brain? [Keith Bowden](/doc/science/1997-bowden-classicalcomputationcanbecounterfactual.html "Classical Computation can be Counterfactual 9/2/96 V1.1 (or Can Schroedinger's Cat Collapse the Wavefunction?)") suggests it could test whether you have a quantum soul.) ## Books **Nonfiction**: - [_The Printing of Mathematics_](/doc/design/typography/1954-chaundy-theprintingofmathematics.pdf "'The Printing of Mathematics: Aids for Authors and Editors and Rules for Compositors and Readers at the University Press, Oxford', Chaundy et al 1954"), Chaundy et al 1954 ## Music **Misc**: - ["Pumped Up Kicks"](https://www.youtube.com/watch?v=cRIfsFefatg) ([Hildegard](https://en.wikipedia.org/wiki/Hildegard_of_Bingen) Von Blingin' {2020}) [folk]