GPT-3 paper § Figure F.1: Four uncurated completions from a context suggesting the model compose a poem in the style of Wallace Stevens with the title ‘Shadows on the Way’
https://www.reddit.com/r/reinforcementlearning/search/?q=flair%3AMetaRL&include_over_18=on&restrict_sr=on&sort=top
Matt Botvinick on the spontaneous emergence of learning algorithms
AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence
On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models
Jukebox: We’re introducing Jukebox, a neural net that generates music, including rudimentary singing, as raw audio in a variety of genres and artist styles. We’re releasing the model weights and code, along with a tool to explore the generated samples.
GPT-1: Improving Language Understanding with Unsupervised Learning
I Recently Came across Https://arxiv.org/abs/2004.08900, Which ‘Assumes 2-3 Runs’ of T5-11B. In Fact, We Trained T5-11B once. That’s Why We Spend 35 Pages Figuring out How We Should Train Before We Start Training. You Don’t Want to Mess up a Training Run That Big.
CERN makes bold push to build €21-billion supercollider: European particle-physics lab will pursue a 100-kilometre machine to uncover the Higgs boson’s secrets—but it doesn’t yet have the funds
Dota 2 With Large Scale Deep Reinforcement Learning § Pg11
Efficient Attention: Breaking The Quadratic Transformer Bottleneck
WBE and DRL: a Middle Way of imitation learning from the human brain
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
GPT-3: Language Models Are Few-Shot Learners: 5. Limitations
CTRL: A Conditional Transformer Language Model For Controllable Generation
MegatronLM: Training Billion+ Parameter Language Models Using GPU Model Parallelism
T5: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Turing-NLG: A 17-billion-parameter language model by Microsoft
GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
Does Learning Require Memorization? A Short Tale about a Long Tail
Scaling to Very Very Large Corpora for Natural Language Disambiguation
2017-koehn-figure3-bleuscoreswithvaryingamountsoftrainingdata.png
WinoGrande: An Adversarial Winograd Schema Challenge at Scale
People I Know at OpenAI Say V4 Is around the Corner and Easily Doable, And...will Be Here Soon (not Months but Year or So). And They Are Confident It Will Scale and Be around 100--1000×.
Microsoft announces new supercomputer, lays out vision for future AI work
Scaling Laws for Neural Language Models: Figure 1: Language Modeling Performance Improves Smoothly As We Increase the Model Size, Dataset Size, and Amount of Compute Used for Training.
Scaling Laws for Neural Language Models: Figure 15: Far beyond the Model Sizes We Study Empirically, We Find a Contradiction between Our Equations § Pg17
Table 2.2: Datasets Used to Train GPT-3. ‘Weight in Training Mix’ Refers to the Fraction of Examples during Training That Are Drawn from a given Dataset, Which We Intentionally Do Not Make Proportional to the Size of the Dataset. As a Result, When We Train for 300 Billion Tokens, Some Datasets Are Seen up to 3.4 times during Training While Other Datasets Are Seen Less Than Once.
2020-adiwardana-meena-figure1-humanratingsvslikelihood.png
2020-brown-figure313-humanabilitytodetectmodelgeneratednewsstories.jpg
Image GPT (iGPT): We find that, just as a large transformer model trained on language can generate coherent text, the same exact model trained on pixel sequences can generate coherent image completions and samples
Vision Transformer: An Image is Worth 16×16 Words: Transformers for Image Recognition at Scale
Generative Language Modeling for Automated Theorem Proving
The neural architecture of language: Integrative reverse-engineering converges on a model for predictive processing
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch Sizes
Dota 2 With Large Scale Deep Reinforcement Learning: §4.3: Batch Size
BigGAN: Large Scale GAN Training For High Fidelity Natural Image Synthesis § 5.2 Additional Evaluation On JFT-300M
Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images
Big Transfer (BiT): General Visual Representation Learning
On Robustness and Transferability of Convolutional Neural Networks
Self-training with Noisy Student improves ImageNet classification
Measuring Robustness to Natural Distribution Shifts in Image Classification
Understanding Robustness of Transformers for Image Classification
12-in-1: Multi-Task Vision and Language Representation Learning
VideoBERT: A Joint Model for Video and Language Representation Learning
The messy, secretive reality behind OpenAI’s bid to save the world: The AI moonshot was founded in the spirit of transparency. This is the inside story of how competitive pressure eroded that idealism
High Fidelity Video Prediction with Large Stochastic Recurrent Neural Networks
Grandmaster level in StarCraft II using multi-agent reinforcement learning
One-Shot High-Fidelity Imitation: Training Large-Scale Deep Nets with RL
A Style-Based Generator Architecture for Generative Adversarial Networks
Environmental drivers of systematicity and generalization in a situated agent
Gated-Attention Architectures for Task-Oriented Language Grounding
Interactive Grounded Language Acquisition and Generalization in a 2D World
Compositional generalization through meta sequence-to-sequence learning
Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning
DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames
Procgen Benchmark: We’re releasing Procgen Benchmark, 16 simple-to-use procedurally-generated environments which provide a direct measure of how quickly a reinforcement learning agent learns generalizable skills
Understanding RL Vision: With diverse environments, we can analyze, diagnose and edit deep reinforcement learning models using attribution
Muppet: Massive Multi-task Representations with Pre-Finetuning
A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play
MuZero: Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model
Understanding deep learning requires rethinking generalization
Deep Double Descent: We show that the double descent phenomenon occurs in CNNs, ResNets, and transformers: performance first improves, then gets worse, and then improves again with increasing model size, data size, or training time
Understanding the generalization of ‘lottery tickets’ in neural networks
Bayesian Deep Learning and a Probabilistic Perspective of Generalization
Zoom In: An Introduction to Circuits—By studying the connections between neurons, we can find meaningful algorithms in the weights of neural networks
Direct Fit to Nature: An Evolutionary Perspective on Biological and Artificial Neural Networks
The Shape of Learning Curves: a Review: 6. Ill-Behaved Learning Curves: 6.1. Phase Transitions
The remarkable, yet not extraordinary, human brain as a scaled-up primate brain and its associated cost
Jumping NLP Curves: A Review of Natural Language Processing Research [Review Article]
difference#efficient-natural-languages
Data Distributional Properties Drive Emergent Few-Shot Learning in Transformers
Building a Large Annotated Corpus of English: The Penn Treebank
One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling
The LAMBADA dataset: Word prediction requiring a broad discourse context
https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf#page=5
Estimation of Gap Between Current Language Models and Human Performance
If You Want to Solve a Hard Problem in Reinforcement Learning, You Just Scale. It's Just Gonna Work Just like Supervised Learning. It's the Same, the Same Story Exactly. It Was Kind of Hard to Believe That Supervised Learning Can Do All Those Things, but It's Not Just Vision, It's Everything and the Same Thing Seems to Hold for Reinforcement Learning Provided You Have a Lot of Experience.
https://wandb.ai/wandb_fc/gradient-dissent/reports/What-could-make-AI-conscious-with-Wojciech-Zaremba-co-founder-of-OpenAI--Vmlldzo3NDk3MDI
Evolution Strategies as a Scalable Alternative to Reinforcement Learning
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
Google DeepMind founder and leader in artificial intelligence returns to Hamilton
When Will the First Artificial General Intelligence System Be Devised, Tested, and Publicly Known Of?
‘How GPT-3 Is Shaping Our AI Future’ With Sam Altman/Azeem Azhar (The Exponential View), Wednesday 7 October 2020
June 2020 News § Companies House
Trust Algorithms? The Army Doesn’t Even Trust Its Own AI Developers
ZeRO-2 & DeepSpeed: Shattering barriers of deep learning speed & scale
Ilya Sutskever: Deep Learning | AI Podcast #94 With Lex Fridman
What Next? A Dozen Information-Technology Research Goals: 3. Turing’s Vision of Machine Intelligence
Pushing the limit of molecular dynamics with ab initio accuracy to 100 million atoms with machine learning
ProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised Deep Learning and High Performance Computing
Training Kinetics in 15 Minutes: Large-scale Distributed Training on Videos
Peter Norvig, Google’s Director of Research—Singularity Is in the Eye of the Beholder: We'Re Thrilled to Have Peter Norvig Who Join Us to Talk about the Evolution of Deep Learning, His Industry-Defining Book, His Work at Google, and What He Thinks the Future Holds for Machine Learning Research (2020-11-20)
The Deep Learning Revolution and Its Implications for Computer Architecture and Chip Design
OpenAI Built Gaming Bots That Can Work As a Team With Inhuman Precision
Can a Machine Learn to Write for The New Yorker? Extraordinary Advances in Machine Learning in Recent Years Have Resulted in A.I.s That Can Write for You.
A Review of Winograd Schema Challenge Datasets and Approaches
Appendix F: Personal Observations on the Reliability of the Shuttle
Do As I Can, Not As I Say (SayCan): Grounding Language in Robotic Affordances
Here’s Another Stabilized Sky Timelapse, This Time at Crater Lake, Oregon. The Water Was Still for Most of It, Which Created a Nice Mirror for the Stars. I Also Got My Astro-Modified Camera Working, Which Provides More Vibrancy in the Nebulae in the Milky Way. #EppurSiMuove
Wikipedia Bibliography:
