- See Also
-
Links
- “Mamba: Linear-Time Sequence Modeling With Selective State Spaces”, Gu & Dao 2023
- “Diffusion Models Without Attention”, Yan et al 2023
- “Learning Few-shot Imitation As Cultural Transmission”, Bhoopchand et al 2023
- “HGRN: Hierarchically Gated Recurrent Neural Network for Sequence Modeling”, Qin et al 2023
- “ProSG: Using Prompt Synthetic Gradients to Alleviate Prompt Forgetting of RNN-like Language Models”, Luo et al 2023
- “GateLoop: Fully Data-Controlled Linear Recurrence for Sequence Modeling”, Katsch 2023
- “Transformers Learn Higher-Order Optimization Methods for In-Context Learning: A Study With Linear Models”, Fu et al 2023
- “Parallelizing Non-linear Sequential Models over the Sequence Length”, Lim et al 2023
- “A High-performance Neuroprosthesis for Speech Decoding and Avatar Control”, Metzger et al 2023
- “Absolute Unit NNs: Regression-Based MLPs for Everything”, Gwern 2023
- “Retentive Network: A Successor to Transformer for Large Language Models”, Sun et al 2023
- “RWKV: Reinventing RNNs for the Transformer Era”, Peng et al 2023
- “Resurrecting Recurrent Neural Networks for Long Sequences”, Orvieto et al 2023
- “SpikeGPT: Generative Pre-trained Language Model With Spiking Neural Networks”, Zhu et al 2023
- “Organic Reaction Mechanism Classification Using Machine Learning”, Burés & Larrosa 2023
- “A High-performance Speech Neuroprosthesis”, Willett et al 2023
- “Hungry Hungry Hippos: Towards Language Modeling With State Space Models”, Fu et al 2022
- “A 64-core Mixed-signal In-memory Compute Chip Based on Phase-change Memory for Deep Neural Network Inference”, Gallo et al 2022
- “Melting Pot 2.0”, Agapiou et al 2022
- “VeLO: Training Versatile Learned Optimizers by Scaling Up”, Metz et al 2022
- “Legged Locomotion in Challenging Terrains Using Egocentric Vision”, Agarwal et al 2022
- “Massively Multilingual ASR on 70 Languages: Tokenization, Architecture, and Generalization Capabilities”, Tjandra et al 2022
- “Transformers Learn Shortcuts to Automata”, Liu et al 2022
- “Omnigrok: Grokking Beyond Algorithmic Data”, Liu et al 2022
- “Semantic Scene Descriptions As an Objective of Human Vision”, Doerig et al 2022
- “Benchmarking Compositionality With Formal Languages”, Valvoda et al 2022
- “Learning to Generalize With Object-centric Agents in the Open World Survival Game Crafter”, Stanić et al 2022
- “PI-ARS: Accelerating Evolution-Learned Visual-Locomotion With Predictive Information Representations”, Lee et al 2022
- “Spatial Representation by Ramping Activity of Neurons in the Retrohippocampal Cortex”, Tennant et al 2022
- “Neural Networks and the Chomsky Hierarchy”, Delétang et al 2022
- “BYOL-Explore: Exploration by Bootstrapped Prediction”, Guo et al 2022
- “AnimeSR: Learning Real-World Super-Resolution Models for Animation Videos”, Wu et al 2022
- “Task-Agnostic Continual Reinforcement Learning: In Praise of a Simple Baseline (3RL)”, Caccia et al 2022
- “Simple Recurrence Improves Masked Language Models”, Lei et al 2022
- “Sequencer: Deep LSTM for Image Classification”, Tatsunami & Taki 2022
- “Data Distributional Properties Drive Emergent Few-Shot Learning in Transformers”, Chan et al 2022
- “Semantic Projection Recovers Rich Human Knowledge of Multiple Object Features from Word Embeddings”, Grand et al 2022
- “Block-Recurrent Transformers”, Hutchins et al 2022
- “All You Need Is Supervised Learning: From Imitation Learning to Meta-RL With Upside Down RL”, Arulkumaran et al 2022
- “Learning by Directional Gradient Descent”, Silver et al 2022
- “Retrieval-Augmented Reinforcement Learning”, Goyal et al 2022
- “General-purpose, Long-context Autoregressive Modeling With Perceiver AR”, Hawthorne et al 2022
- “End-to-end Algorithm Synthesis With Recurrent Networks: Logical Extrapolation Without Overthinking”, Bansal et al 2022
- “Data Scaling Laws in NMT: The Effect of Noise and Architecture”, Bansal et al 2022
- “Active Predictive Coding Networks: A Neural Solution to the Problem of Learning Reference Frames and Part-Whole Hierarchies”, Gklezakos & Rao 2022
- “Learning Robust Perceptive Locomotion for Quadrupedal Robots in the Wild”, Miki et al 2022
- “Inducing Causal Structure for Interpretable Neural Networks (IIT)”, Geiger et al 2021
- “Evaluating Distributional Distortion in Neural Language Modeling”, Anonymous 2021
- “Gradients Are Not All You Need”, Metz et al 2021
- “An Explanation of In-context Learning As Implicit Bayesian Inference”, Xie et al 2021
- “Minimum Description Length Recurrent Neural Networks”, Lan et al 2021
- “S4: Efficiently Modeling Long Sequences With Structured State Spaces”, Gu et al 2021
- “A Connectome of the Drosophila Central Complex Reveals Network Motifs Suitable for Flexible Navigation and Context-dependent Action Selection”, Hulse et al 2021
- “LSSL: Combining Recurrent, Convolutional, and Continuous-time Models With Linear State-Space Layers”, Gu et al 2021
- “Recurrent Model-Free RL Is a Strong Baseline for Many POMDPs”, Ni et al 2021
- “Photos Are All You Need for Reciprocal Recommendation in Online Dating”, Neve & McConville 2021
- “Perceiver IO: A General Architecture for Structured Inputs & Outputs”, Jaegle et al 2021
- “PES: Unbiased Gradient Estimation in Unrolled Computation Graphs With Persistent Evolution Strategies”, Vicol et al 2021
- “Shelley: A Crowd-sourced Collaborative Horror Writer”, Delul et al 2021
- “Ten Lessons From Three Generations Shaped Google’s TPUv4i”, Jouppi et al 2021
- “RASP: Thinking Like Transformers”, Weiss et al 2021
- “Scaling Laws for Acoustic Models”, Droppo & Elibol 2021
- “Scaling End-to-End Models for Large-Scale Multilingual ASR”, Li et al 2021
- “Efficient Transformers in Reinforcement Learning Using Actor-Learner Distillation”, Parisotto & Salakhutdinov 2021
- “Finetuning Pretrained Transformers into RNNs”, Kasai et al 2021
- “Pretrained Transformers As Universal Computation Engines”, Lu et al 2021
- “Perceiver: General Perception With Iterative Attention”, Jaegle et al 2021
- “When Attention Meets Fast Recurrence: Training SRU++ Language Models With Reduced Compute”, Lei 2021
- “Predictive Coding Is a Consequence of Energy Efficiency in Recurrent Neural Networks”, Ali et al 2021
- “Deep Residual Learning in Spiking Neural Networks”, Fang et al 2021
- “Distilling Large Language Models into Tiny and Effective Students Using PQRNN”, Kaliamoorthi et al 2021
- “Meta Learning Backpropagation And Improving It”, Kirsch & Schmidhuber 2020
- “On the Binding Problem in Artificial Neural Networks”, Greff et al 2020
- “A Recurrent Vision-and-Language BERT for Navigation”, Hong et al 2020
- “Towards Playing Full MOBA Games With Deep Reinforcement Learning”, Ye et al 2020
- “Adversarial Vulnerabilities of Human Decision-making”, Dezfouli et al 2020
- “Learning to Summarize Long Texts With Memory Compression and Transfer”, Park et al 2020
- “Human-centric Dialog Training via Offline Reinforcement Learning”, Jaques et al 2020
- “AFT: An Attention Free Transformer”, Anonymous 2020
- “Deep Reinforcement Learning for Closed-Loop Blood Glucose Control”, Fox et al 2020
- “HiPPO: Recurrent Memory With Optimal Polynomial Projections”, Gu et al 2020
- “Adding Recurrence to Pretrained Transformers for Improved Efficiency and Context Size”, Yoshida et al 2020
- “Matt Botvinick on the Spontaneous Emergence of Learning Algorithms”, Scholl 2020
- “Cultural Influences on Word Meanings Revealed through Large-scale Semantic Alignment”, Thompson et al 2020
- “DeepSinger: Singing Voice Synthesis With Data Mined From the Web”, Ren et al 2020
- “High-performance Brain-to-text Communication via Imagined Handwriting”, Willett et al 2020
- “Transformers Are RNNs: Fast Autoregressive Transformers With Linear Attention”, Katharopoulos et al 2020
- “The Recurrent Neural Tangent Kernel”, Alemohammad et al 2020
- “Untangling Tradeoffs between Recurrence and Self-attention in Neural Networks”, Kerg et al 2020
- “Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing”, Dai et al 2020
- “Learning Music Helps You Read: Using Transfer to Study Linguistic Structure in Language Models”, Papadimitriou & Jurafsky 2020
- “Syntactic Structure from Deep Learning”, Linzen & Baroni 2020
- “Agent57: Outperforming the Human Atari Benchmark”, Puigdomènech et al 2020
- “Machine Translation of Cortical Activity to Text With an Encoder-decoder Framework”, Makin et al 2020
- “Learning-based Memory Allocation for C++ Server Workloads”, Maas et al 2020
- “Accelerating Feedforward Computation via Parallel Nonlinear Equation Solving”, Song et al 2020
- “Direct Fit to Nature: An Evolutionary Perspective on Biological and Artificial Neural Networks”, Hasson et al 2020
- “Scaling Laws for Neural Language Models”, Kaplan et al 2020
- “Placing Language in an Integrated Understanding System: Next Steps toward Human-level Performance in Neural Language Models”, McClelland et al 2020
- “Estimating the Deep Replicability of Scientific Findings Using Human and Artificial Intelligence”, Yang et al 2020
- “Measuring Compositional Generalization: A Comprehensive Method on Realistic Data”, Keysers et al 2019
- “SimpleBooks: Long-term Dependency Book Dataset With Simplified English Vocabulary for Word-level Language Modeling”, Nguyen 2019
- “Single Headed Attention RNN: Stop Thinking With Your Head”, Merity 2019
- “Excavate”, Lynch 2019
- “MuZero: Mastering Atari, Go, Chess and Shogi by Planning With a Learned Model”, Schrittwieser et al 2019
- “CommonGen: A Constrained Text Generation Challenge for Generative Commonsense Reasoning”, Lin et al 2019
- “Legendre Memory Units: Continuous-Time Representation in Recurrent Neural Networks”, Voelker et al 2019
- “High Fidelity Video Prediction With Large Stochastic Recurrent Neural Networks”, Villegas et al 2019
- “SEED RL: Scalable and Efficient Deep-RL With Accelerated Central Inference”, Espeholt et al 2019
- “Restoring Ancient Text Using Deep Learning (Pythia): a Case Study on Greek Epigraphy”, Assael et al 2019
- “Mixed-Signal Neuromorphic Processors: Quo Vadis?”, Bavandpour et al 2019
- “Mogrifier LSTM”, Melis et al 2019
- “R2D3: Making Efficient Use of Demonstrations to Solve Hard Exploration Problems”, Paine et al 2019
- “Language Modelling State-of-the-art Leaderboards”, paperswithcode.com 2019
- “Metalearned Neural Memory”, Munkhdalai et al 2019
- “Generating Text With Recurrent Neural Networks”, Sutskever et al 2019
- “Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank”, Socher et al 2019
- “XLNet: Generalized Autoregressive Pretraining for Language Understanding”, Yang et al 2019
- “Playing the Lottery With Rewards and Multiple Languages: Lottery Tickets in RL and NLP”, Yu et al 2019
- “Reinforcement Learning, Fast and Slow”, Botvinick et al 2019
- “MoGlow: Probabilistic and Controllable Motion Synthesis Using Normalizing Flows”, Henter et al 2019
- “Meta-learners’ Learning Dynamics Are unlike Learners’”, Rabinowitz 2019
- “Speech Synthesis from Neural Decoding of Spoken Sentences”, Anumanchipalli et al 2019
- “Good News, Everyone! Context Driven Entity-aware Captioning for News Images”, Biten et al 2019
- “Surrogate Gradient Learning in Spiking Neural Networks”, Neftci et al 2019
- “On the Turing Completeness of Modern Neural Network Architectures”, Pérez et al 2019
- “Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context”, Dai et al 2019
- “Natural Questions: A Benchmark for Question Answering Research”, Kwiatkowski et al 2019
- “High Fidelity Video Prediction With Large Stochastic Recurrent Neural Networks: Videos”, Villegas et al 2019
- “Bayesian Layers: A Module for Neural Network Uncertainty”, Tran et al 2018
- “Meta-Learning: Learning to Learn Fast”, Weng 2018
- “Piano Genie”, Donahue et al 2018
- “Learning Recurrent Binary/Ternary Weights”, Ardakani et al 2018
- “R2D2: Recurrent Experience Replay in Distributed Reinforcement Learning”, Kapturowski et al 2018
- “HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering”, Yang et al 2018
- “Object Hallucination in Image Captioning”, Rohrbach et al 2018
- “Adversarial Reprogramming of Text Classification Neural Networks”, Neekhara et al 2018
- “This Time With Feeling: Learning Expressive Musical Performance”, Oore et al 2018
- “Character-Level Language Modeling With Deeper Self-Attention”, Al-Rfou et al 2018
- “General Value Function Networks”, Schlegel et al 2018
- “Universal Transformers”, Dehghani et al 2018
- “Deep-speare: A Joint Neural Model of Poetic Language, Meter and Rhyme”, Lau et al 2018
- “Accurate Uncertainties for Deep Learning Using Calibrated Regression”, Kuleshov et al 2018
- “The Natural Language Decathlon: Multitask Learning As Question Answering”, McCann et al 2018
- “Neural Ordinary Differential Equations”, Chen et al 2018
- “Know What You Don’t Know: Unanswerable Questions for SQuAD”, Rajpurkar et al 2018
- “DVRL: Deep Variational Reinforcement Learning for POMDPs”, Igl et al 2018
- “Greedy Attack and Gumbel Attack: Generating Adversarial Examples for Discrete Data”, Yang et al 2018
- “Hierarchical Neural Story Generation”, Fan et al 2018
- “Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context”, Khandelwal et al 2018
- “Newsroom: A Dataset of 1.3 Million Summaries With Diverse Extractive Strategies”, Grusky et al 2018
- “A Tree Search Algorithm for Sequence Labeling”, Lao et al 2018
- “An Analysis of Neural Language Modeling at Multiple Scales”, Merity et al 2018
- “Reviving and Improving Recurrent Back-Propagation”, Liao et al 2018
- “Learning Memory Access Patterns”, Hashemi et al 2018
- “Learning Longer-term Dependencies in RNNs With Auxiliary Losses”, Trinh et al 2018
- “One Big Net For Everything”, Schmidhuber 2018
- “Efficient Neural Audio Synthesis”, Kalchbrenner et al 2018
- “Deep Contextualized Word Representations”, Peters et al 2018
- “M-Walk: Learning to Walk over Graphs Using Monte Carlo Tree Search”, Shen et al 2018
- “ULMFiT: Universal Language Model Fine-tuning for Text Classification”, Howard & Ruder 2018
- “Overcoming the Vanishing Gradient Problem in Plain Recurrent Networks”, Hu et al 2018
- “Large-scale Comparison of Machine Learning Methods for Drug Target Prediction on ChEMBL”, Mayr et al 2018
- “A Flexible Approach to Automated RNN Architecture Generation”, Schrimpf et al 2017
- “Learning Compact Recurrent Neural Networks With Block-Term Tensor Decomposition”, Ye et al 2017
- “Mastering the Dungeon: Grounded Language Learning by Mechanical Turker Descent”, Yang et al 2017
- “Evaluating Prose Style Transfer With the Bible”, Carlson et al 2017
- “Breaking the Softmax Bottleneck: A High-Rank RNN Language Model”, Yang et al 2017
- “Neural Speed Reading via Skim-RNN”, Seo et al 2017
- “Generalization without Systematicity: On the Compositional Skills of Sequence-to-sequence Recurrent Networks”, Lake & Baroni 2017
- “Word Translation Without Parallel Data”, Conneau et al 2017
- “Mixed Precision Training”, Micikevicius et al 2017
- “To Prune, or Not to Prune: Exploring the Efficacy of Pruning for Model Compression”, Zhu & Gupta 2017
- “Dynamic Evaluation of Neural Sequence Models”, Krause et al 2017
- “N2N Learning: Network to Network Compression via Policy Gradient Reinforcement Learning”, Ashok et al 2017
- “Why Pay More When You Can Pay Less: A Joint Learning Framework for Active Feature Acquisition and Classification”, Shim et al 2017
- “SRU: Simple Recurrent Units for Highly Parallelizable Recurrence”, Lei et al 2017
- “Learning to Look Around: Intelligently Exploring Unseen Environments for Unknown Tasks”, Jayaraman & Grauman 2017
- “Skip RNN: Learning to Skip State Updates in Recurrent Neural Networks”, Campos et al 2017
- “Twin Networks: Matching the Future for Sequence Generation”, Serdyuk et al 2017
- “Revisiting Activation Regularization for Language RNNs”, Merity et al 2017
- “Bayesian Sparsification of Recurrent Neural Networks”, Lobacheva et al 2017
- “On the State-of-the-Art of Evaluation in Neural Language Models”, Melis et al 2017
- “Controlling Linguistic Style Aspects in Neural Language Generation”, Ficler & Goldberg 2017
- “Device Placement Optimization With Reinforcement Learning”, Mirhoseini et al 2017
- “Language Generation With Recurrent Generative Adversarial Networks without Pre-training”, Press et al 2017
- “Towards Synthesizing Complex Programs from Input-Output Examples”, Chen et al 2017
- “Biased Importance Sampling for Deep Neural Network Training”, Katharopoulos & Fleuret 2017
- “Deriving Neural Architectures from Sequence and Graph Kernels”, Lei et al 2017
- “A Deep Reinforced Model for Abstractive Summarization”, Paulus et al 2017
- “DeepTingle”, Khalifa et al 2017
- “TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension”, Joshi et al 2017
- “A Neural Network System for Transformation of Regional Cuisine Style”, Kazama et al 2017
- “Sharp Models on Dull Hardware: Fast and Accurate Neural Machine Translation Decoding on the CPU”, Devlin 2017
- “Adversarial Neural Machine Translation”, Wu et al 2017
- “Learning to Reason: End-to-End Module Networks for Visual Question Answering”, Hu et al 2017
- “SearchQA: A New Q&A Dataset Augmented With Context from a Search Engine”, Dunn et al 2017
- “Exploring Sparsity in Recurrent Neural Networks”, Narang et al 2017
- “Get To The Point: Summarization With Pointer-Generator Networks”, See et al 2017
- “DeepAR: Probabilistic Forecasting With Autoregressive Recurrent Networks”, Salinas et al 2017
- “Bayesian Recurrent Neural Networks”, Fortunato et al 2017
- “Recurrent Environment Simulators”, Chiappa et al 2017
- “Learning to Generate Reviews and Discovering Sentiment”, Radford et al 2017
- “Learning Simpler Language Models With the Differential State Framework”, II et al 2017
- “I2T2I: Learning Text to Image Synthesis With Textual Data Augmentation”, Dong et al 2017
- “Improving Neural Machine Translation With Conditional Sequence Generative Adversarial Nets”, Yang et al 2017
- “Learned Optimizers That Scale and Generalize”, Wichrowska et al 2017
- “Parallel Multiscale Autoregressive Density Estimation”, Reed et al 2017
- “Tracking the World State With Recurrent Entity Networks”, Henaff et al 2017
- “Optimization As a Model for Few-Shot Learning”, Ravi & Larochelle 2017
- “Neural Combinatorial Optimization With Reinforcement Learning”, Bello et al 2017
- “Frustratingly Short Attention Spans in Neural Language Modeling”, Daniluk et al 2017
- “Tuning Recurrent Neural Networks With Reinforcement Learning”, Jaques et al 2017
- “Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer”, Shazeer et al 2017
- “Neural Data Filter for Bootstrapping Stochastic Gradient Descent”, Fan et al 2017
- “Your TL;DR by an AI: A Deep Reinforced Model for Abstractive Summarization”, Paulus 2017
- “SampleRNN: An Unconditional End-to-End Neural Audio Generation Model”, Mehri et al 2016
- “Improving Neural Language Models With a Continuous Cache”, Grave et al 2016
- “NewsQA: A Machine Comprehension Dataset”, Trischler et al 2016
- “Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation”, Johnson et al 2016
- “Learning to Learn without Gradient Descent by Gradient Descent”, Chen et al 2016
- “RL2: Fast Reinforcement Learning via Slow Reinforcement Learning”, Duan et al 2016
- “DeepCoder: Learning to Write Programs”, Balog et al 2016
- “Bidirectional Attention Flow for Machine Comprehension”, Seo et al 2016
- “Neural Architecture Search With Reinforcement Learning”, Zoph & Le 2016
- “QRNNs: Quasi-Recurrent Neural Networks”, Bradbury et al 2016
- “Scaling Memory-Augmented Neural Networks With Sparse Reads and Writes”, Rae et al 2016
- “Hybrid Computing Using a Neural Network With Dynamic External Memory”, Graves et al 2016
- “Using Fast Weights to Attend to the Recent Past”, Ba et al 2016
- “Achieving Human Parity in Conversational Speech Recognition”, Xiong et al 2016
- “Video Pixel Networks”, Kalchbrenner et al 2016
- “HyperNetworks”, Ha et al 2016
- “Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation”, Wu et al 2016
- “Multiplicative LSTM for Sequence Modelling”, Krause et al 2016
- “Pointer Sentinel Mixture Models”, Merity et al 2016
- “Image-to-Markup Generation With Coarse-to-Fine Attention”, Deng et al 2016
- “Hierarchical Multiscale Recurrent Neural Networks”, Chung et al 2016
- “Deep Learning Human Mind for Automated Visual Classification”, Spampinato et al 2016
- “Decoupled Neural Interfaces Using Synthetic Gradients”, Jaderberg et al 2016
- “Full Resolution Image Compression With Recurrent Neural Networks”, Toderici et al 2016
- “Clockwork Convnets for Video Semantic Segmentation”, Shelhamer et al 2016
- “Layer Normalization”, Ba et al 2016
- “LSTMVis: A Tool for Visual Analysis of Hidden State Dynamics in Recurrent Neural Networks”, Strobelt et al 2016
- “Learning to Learn by Gradient Descent by Gradient Descent”, Andrychowicz et al 2016
- “Iterative Alternating Neural Attention for Machine Reading”, Sordoni et al 2016
- “Deep Reinforcement Learning for Dialogue Generation”, Li et al 2016
- “Programming With a Differentiable Forth Interpreter”, Bošnjak et al 2016
- “Training Deep Nets With Sublinear Memory Cost”, Chen et al 2016
- “Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex”, Liao & Poggio 2016
- “Improving Sentence Compression by Learning to Predict Gaze”, Klerke et al 2016
- “Adaptive Computation Time for Recurrent Neural Networks”, Graves 2016
- “Dynamic Memory Networks for Visual and Textual Question Answering”, Xiong et al 2016
- “PlaNet—Photo Geolocation With Convolutional Neural Networks”, Weyand et al 2016
- “Learning Distributed Representations of Sentences from Unlabeled Data”, Hill et al 2016
- “Exploring the Limits of Language Modeling”, Jozefowicz et al 2016
- “PixelRNN: Pixel Recurrent Neural Networks”, Oord et al 2016
- “Persistent RNNs: Stashing Recurrent Weights On-Chip”, Diamos et al 2016
- “Deep-Spying: Spying Using Smartwatch and Deep Learning”, Beltramelli & Risi 2015
- “On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models”, Schmidhuber 2015
- “Neural GPUs Learn Algorithms”, Kaiser & Sutskever 2015
- “Sequence Level Training With Recurrent Neural Networks”, Ranzato et al 2015
- “Generating Sentences from a Continuous Space”, Bowman et al 2015
- “Neural Programmer-Interpreters”, Reed & Freitas 2015
- “Generative Concatenative Nets Jointly Learn to Write and Classify Reviews”, Lipton et al 2015
- “Generating Images from Captions With Attention”, Mansimov et al 2015
- “Semi-supervised Sequence Learning”, Dai & Le 2015
- “RNN Metadata for Mimicking Author Style”, Gwern 2015
- “Training Recurrent Networks Online without Backtracking”, Ollivier et al 2015
- “Deep Recurrent Q-Learning for Partially Observable MDPs”, Hausknecht & Stone 2015
- “Teaching Machines to Read and Comprehend”, Hermann et al 2015
- “Scheduled Sampling for Sequence Prediction With Recurrent Neural Networks”, Bengio et al 2015
- “Visualizing and Understanding Recurrent Networks”, Karpathy et al 2015
- “The Unreasonable Effectiveness of Recurrent Neural Networks”, Karpathy 2015
- “Deep Neural Networks for Large Vocabulary Handwritten Text Recognition”, Bluche 2015
- “Reinforcement Learning Neural Turing Machines—Revised”, Zaremba & Sutskever 2015
- “End-To-End Memory Networks”, Sukhbaatar et al 2015
- “Inferring Algorithmic Patterns With Stack-Augmented Recurrent Nets”, Arm et al 2015
- “DRAW: A Recurrent Neural Network For Image Generation”, Gregor et al 2015
- “Neural Turing Machines”, Graves et al 2014
- “Learning to Execute”, Zaremba & Sutskever 2014
- “Neural Machine Translation by Jointly Learning to Align and Translate”, Bahdanau et al 2014
- “Distributed Representations of Sentences and Documents”, Le & Mikolov 2014
- “A Clockwork RNN”, Koutník et al 2014
- “One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling”, Chelba et al 2013
- “Generating Sequences With Recurrent Neural Networks”, Graves 2013
- “On the Difficulty of Training Recurrent Neural Networks”, Pascanu et al 2012
- “Recurrent Neural Network Based Language Model”, Mikolov et al 2010
- “Large Language Models in Machine Translation”, Brants et al 2007
- “Learning to Learn Using Gradient Descent”, Hochreiter et al 2001
- “Long Short-Term Memory”, Hochreiter & Schmidhuber 1997
- “Flat Minima”, Hochreiter & Schmidhuber 1997
- “Gradient-Based Learning Algorithms for Recurrent Networks and Their Computational Complexity”, Williams & Zipser 1995
- “A Focused Backpropagation Algorithm for Temporal Pattern Recognition”, Mozer 1995
- “Learning Complex, Extended Sequences Using the Principle of History Compression”, Schmidhuber 1992
- “Learning to Control Fast-Weight Memories: An Alternative to Dynamic Recurrent Networks”, Schmidhuber 1992
- “Untersuchungen Zu Dynamischen Neuronalen Netzen [Studies of Dynamic Neural Networks]”, Hochreiter 1991
- “Finding Structure In Time”, Elman 1990
- “Connectionist Music Composition Based on Melodic, Stylistic, and Psychophysical Constraints [Technical Report CU-CS–495–90]”, Mozer 1990
- “A Learning Algorithm for Continually Running Fully Recurrent Neural Networks”, Williams & Zipser 1989b
- “A Local Learning Algorithm for Dynamic Feedforward and Recurrent Networks”, Schmidhuber 1989
- “Experimental Analysis of the Real-time Recurrent Learning Algorithm”, Williams & Zipser 1989
- “A Sticky-Bit Approach for Learning to Represent State”, Bachrach 1988
- “The Utility Driven Dynamic Error Propagation Network (RTRL)”, Robinson & Fallside 1987
- “Serial Order: A Parallel Distributed Processing Approach”, Jordan 1986
- “Attention and Augmented Recurrent Neural Networks”
- “Deep Learning for Assisting the Process of Music Composition (part 3)”
- Sort By Magic
- Wikipedia
- Miscellaneous
- Link Bibliography
See Also
Links
“Mamba: Linear-Time Sequence Modeling With Selective State Spaces”, Gu & Dao 2023
“Mamba: Linear-Time Sequence Modeling with Selective State Spaces”
“Diffusion Models Without Attention”, Yan et al 2023
“Learning Few-shot Imitation As Cultural Transmission”, Bhoopchand et al 2023
“HGRN: Hierarchically Gated Recurrent Neural Network for Sequence Modeling”, Qin et al 2023
“HGRN: Hierarchically Gated Recurrent Neural Network for Sequence Modeling”
“ProSG: Using Prompt Synthetic Gradients to Alleviate Prompt Forgetting of RNN-like Language Models”, Luo et al 2023
“ProSG: Using Prompt Synthetic Gradients to Alleviate Prompt Forgetting of RNN-like Language Models”
“GateLoop: Fully Data-Controlled Linear Recurrence for Sequence Modeling”, Katsch 2023
“GateLoop: Fully Data-Controlled Linear Recurrence for Sequence Modeling”
“Transformers Learn Higher-Order Optimization Methods for In-Context Learning: A Study With Linear Models”, Fu et al 2023
“Parallelizing Non-linear Sequential Models over the Sequence Length”, Lim et al 2023
“Parallelizing non-linear sequential models over the sequence length”
“A High-performance Neuroprosthesis for Speech Decoding and Avatar Control”, Metzger et al 2023
“A high-performance neuroprosthesis for speech decoding and avatar control”
“Absolute Unit NNs: Regression-Based MLPs for Everything”, Gwern 2023
“Retentive Network: A Successor to Transformer for Large Language Models”, Sun et al 2023
“Retentive Network: A Successor to Transformer for Large Language Models”
“RWKV: Reinventing RNNs for the Transformer Era”, Peng et al 2023
“Resurrecting Recurrent Neural Networks for Long Sequences”, Orvieto et al 2023
“SpikeGPT: Generative Pre-trained Language Model With Spiking Neural Networks”, Zhu et al 2023
“SpikeGPT: Generative Pre-trained Language Model with Spiking Neural Networks”
“Organic Reaction Mechanism Classification Using Machine Learning”, Burés & Larrosa 2023
“Organic reaction mechanism classification using machine learning”
“A High-performance Speech Neuroprosthesis”, Willett et al 2023
“Hungry Hungry Hippos: Towards Language Modeling With State Space Models”, Fu et al 2022
“Hungry Hungry Hippos: Towards Language Modeling with State Space Models”
“A 64-core Mixed-signal In-memory Compute Chip Based on Phase-change Memory for Deep Neural Network Inference”, Gallo et al 2022
“Melting Pot 2.0”, Agapiou et al 2022
“VeLO: Training Versatile Learned Optimizers by Scaling Up”, Metz et al 2022
“Legged Locomotion in Challenging Terrains Using Egocentric Vision”, Agarwal et al 2022
“Legged Locomotion in Challenging Terrains using Egocentric Vision”
“Massively Multilingual ASR on 70 Languages: Tokenization, Architecture, and Generalization Capabilities”, Tjandra et al 2022
“Transformers Learn Shortcuts to Automata”, Liu et al 2022
“Omnigrok: Grokking Beyond Algorithmic Data”, Liu et al 2022
“Semantic Scene Descriptions As an Objective of Human Vision”, Doerig et al 2022
“Semantic scene descriptions as an objective of human vision”
“Benchmarking Compositionality With Formal Languages”, Valvoda et al 2022
“Learning to Generalize With Object-centric Agents in the Open World Survival Game Crafter”, Stanić et al 2022
“Learning to Generalize with Object-centric Agents in the Open World Survival Game Crafter”
“PI-ARS: Accelerating Evolution-Learned Visual-Locomotion With Predictive Information Representations”, Lee et al 2022
“Spatial Representation by Ramping Activity of Neurons in the Retrohippocampal Cortex”, Tennant et al 2022
“Spatial representation by ramping activity of neurons in the retrohippocampal cortex”
“Neural Networks and the Chomsky Hierarchy”, Delétang et al 2022
“BYOL-Explore: Exploration by Bootstrapped Prediction”, Guo et al 2022
“AnimeSR: Learning Real-World Super-Resolution Models for Animation Videos”, Wu et al 2022
“AnimeSR: Learning Real-World Super-Resolution Models for Animation Videos”
“Task-Agnostic Continual Reinforcement Learning: In Praise of a Simple Baseline (3RL)”, Caccia et al 2022
“Task-Agnostic Continual Reinforcement Learning: In Praise of a Simple Baseline (3RL)”
“Simple Recurrence Improves Masked Language Models”, Lei et al 2022
“Sequencer: Deep LSTM for Image Classification”, Tatsunami & Taki 2022
“Data Distributional Properties Drive Emergent Few-Shot Learning in Transformers”, Chan et al 2022
“Data Distributional Properties Drive Emergent Few-Shot Learning in Transformers”
“Semantic Projection Recovers Rich Human Knowledge of Multiple Object Features from Word Embeddings”, Grand et al 2022
“Semantic projection recovers rich human knowledge of multiple object features from word embeddings”
“Block-Recurrent Transformers”, Hutchins et al 2022
“All You Need Is Supervised Learning: From Imitation Learning to Meta-RL With Upside Down RL”, Arulkumaran et al 2022
“All You Need Is Supervised Learning: From Imitation Learning to Meta-RL With Upside Down RL”
“Learning by Directional Gradient Descent”, Silver et al 2022
“Retrieval-Augmented Reinforcement Learning”, Goyal et al 2022
“General-purpose, Long-context Autoregressive Modeling With Perceiver AR”, Hawthorne et al 2022
“General-purpose, long-context autoregressive modeling with Perceiver AR”
“End-to-end Algorithm Synthesis With Recurrent Networks: Logical Extrapolation Without Overthinking”, Bansal et al 2022
“End-to-end Algorithm Synthesis with Recurrent Networks: Logical Extrapolation Without Overthinking”
“Data Scaling Laws in NMT: The Effect of Noise and Architecture”, Bansal et al 2022
“Data Scaling Laws in NMT: The Effect of Noise and Architecture”
“Active Predictive Coding Networks: A Neural Solution to the Problem of Learning Reference Frames and Part-Whole Hierarchies”, Gklezakos & Rao 2022
“Learning Robust Perceptive Locomotion for Quadrupedal Robots in the Wild”, Miki et al 2022
“Learning robust perceptive locomotion for quadrupedal robots in the wild”
“Inducing Causal Structure for Interpretable Neural Networks (IIT)”, Geiger et al 2021
“Inducing Causal Structure for Interpretable Neural Networks (IIT)”
“Evaluating Distributional Distortion in Neural Language Modeling”, Anonymous 2021
“Evaluating Distributional Distortion in Neural Language Modeling”
“Gradients Are Not All You Need”, Metz et al 2021
“An Explanation of In-context Learning As Implicit Bayesian Inference”, Xie et al 2021
“An Explanation of In-context Learning as Implicit Bayesian Inference”
“Minimum Description Length Recurrent Neural Networks”, Lan et al 2021
“S4: Efficiently Modeling Long Sequences With Structured State Spaces”, Gu et al 2021
“S4: Efficiently Modeling Long Sequences with Structured State Spaces”
“A Connectome of the Drosophila Central Complex Reveals Network Motifs Suitable for Flexible Navigation and Context-dependent Action Selection”, Hulse et al 2021
“LSSL: Combining Recurrent, Convolutional, and Continuous-time Models With Linear State-Space Layers”, Gu et al 2021
“Recurrent Model-Free RL Is a Strong Baseline for Many POMDPs”, Ni et al 2021
“Recurrent Model-Free RL is a Strong Baseline for Many POMDPs”
“Photos Are All You Need for Reciprocal Recommendation in Online Dating”, Neve & McConville 2021
“Photos Are All You Need for Reciprocal Recommendation in Online Dating”
“Perceiver IO: A General Architecture for Structured Inputs & Outputs”, Jaegle et al 2021
“Perceiver IO: A General Architecture for Structured Inputs & Outputs”
“PES: Unbiased Gradient Estimation in Unrolled Computation Graphs With Persistent Evolution Strategies”, Vicol et al 2021
“Shelley: A Crowd-sourced Collaborative Horror Writer”, Delul et al 2021
“Ten Lessons From Three Generations Shaped Google’s TPUv4i”, Jouppi et al 2021
“RASP: Thinking Like Transformers”, Weiss et al 2021
“Scaling Laws for Acoustic Models”, Droppo & Elibol 2021
“Scaling End-to-End Models for Large-Scale Multilingual ASR”, Li et al 2021
“Scaling End-to-End Models for Large-Scale Multilingual ASR”
“Efficient Transformers in Reinforcement Learning Using Actor-Learner Distillation”, Parisotto & Salakhutdinov 2021
“Efficient Transformers in Reinforcement Learning using Actor-Learner Distillation”
“Finetuning Pretrained Transformers into RNNs”, Kasai et al 2021
“Pretrained Transformers As Universal Computation Engines”, Lu et al 2021
“Perceiver: General Perception With Iterative Attention”, Jaegle et al 2021
“When Attention Meets Fast Recurrence: Training SRU++ Language Models With Reduced Compute”, Lei 2021
“When Attention Meets Fast Recurrence: Training SRU++ Language Models with Reduced Compute”
“Predictive Coding Is a Consequence of Energy Efficiency in Recurrent Neural Networks”, Ali et al 2021
“Predictive coding is a consequence of energy efficiency in recurrent neural networks”
“Deep Residual Learning in Spiking Neural Networks”, Fang et al 2021
“Distilling Large Language Models into Tiny and Effective Students Using PQRNN”, Kaliamoorthi et al 2021
“Distilling Large Language Models into Tiny and Effective Students using pQRNN”
“Meta Learning Backpropagation And Improving It”, Kirsch & Schmidhuber 2020
“On the Binding Problem in Artificial Neural Networks”, Greff et al 2020
“A Recurrent Vision-and-Language BERT for Navigation”, Hong et al 2020
“Towards Playing Full MOBA Games With Deep Reinforcement Learning”, Ye et al 2020
“Towards Playing Full MOBA Games with Deep Reinforcement Learning”
“Adversarial Vulnerabilities of Human Decision-making”, Dezfouli et al 2020
“Learning to Summarize Long Texts With Memory Compression and Transfer”, Park et al 2020
“Learning to Summarize Long Texts with Memory Compression and Transfer”
“Human-centric Dialog Training via Offline Reinforcement Learning”, Jaques et al 2020
“Human-centric Dialog Training via Offline Reinforcement Learning”
“AFT: An Attention Free Transformer”, Anonymous 2020
“Deep Reinforcement Learning for Closed-Loop Blood Glucose Control”, Fox et al 2020
“Deep Reinforcement Learning for Closed-Loop Blood Glucose Control”
“HiPPO: Recurrent Memory With Optimal Polynomial Projections”, Gu et al 2020
“HiPPO: Recurrent Memory with Optimal Polynomial Projections”
“Adding Recurrence to Pretrained Transformers for Improved Efficiency and Context Size”, Yoshida et al 2020
“Adding Recurrence to Pretrained Transformers for Improved Efficiency and Context Size”
“Matt Botvinick on the Spontaneous Emergence of Learning Algorithms”, Scholl 2020
“Matt Botvinick on the spontaneous emergence of learning algorithms”
“Cultural Influences on Word Meanings Revealed through Large-scale Semantic Alignment”, Thompson et al 2020
“Cultural influences on word meanings revealed through large-scale semantic alignment”
“DeepSinger: Singing Voice Synthesis With Data Mined From the Web”, Ren et al 2020
“DeepSinger: Singing Voice Synthesis with Data Mined From the Web”
“High-performance Brain-to-text Communication via Imagined Handwriting”, Willett et al 2020
“High-performance brain-to-text communication via imagined handwriting”
“Transformers Are RNNs: Fast Autoregressive Transformers With Linear Attention”, Katharopoulos et al 2020
“Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention”
“The Recurrent Neural Tangent Kernel”, Alemohammad et al 2020
“Untangling Tradeoffs between Recurrence and Self-attention in Neural Networks”, Kerg et al 2020
“Untangling tradeoffs between recurrence and self-attention in neural networks”
“Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing”, Dai et al 2020
“Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing”
“Learning Music Helps You Read: Using Transfer to Study Linguistic Structure in Language Models”, Papadimitriou & Jurafsky 2020
“Learning Music Helps You Read: Using Transfer to Study Linguistic Structure in Language Models”
“Syntactic Structure from Deep Learning”, Linzen & Baroni 2020
“Agent57: Outperforming the Human Atari Benchmark”, Puigdomènech et al 2020
“Machine Translation of Cortical Activity to Text With an Encoder-decoder Framework”, Makin et al 2020
“Machine translation of cortical activity to text with an encoder-decoder framework”
“Learning-based Memory Allocation for C++ Server Workloads”, Maas et al 2020
“Accelerating Feedforward Computation via Parallel Nonlinear Equation Solving”, Song et al 2020
“Accelerating Feedforward Computation via Parallel Nonlinear Equation Solving”
“Direct Fit to Nature: An Evolutionary Perspective on Biological and Artificial Neural Networks”, Hasson et al 2020
“Direct Fit to Nature: An Evolutionary Perspective on Biological and Artificial Neural Networks”
“Scaling Laws for Neural Language Models”, Kaplan et al 2020
“Placing Language in an Integrated Understanding System: Next Steps toward Human-level Performance in Neural Language Models”, McClelland et al 2020
“Estimating the Deep Replicability of Scientific Findings Using Human and Artificial Intelligence”, Yang et al 2020
“Estimating the deep replicability of scientific findings using human and artificial intelligence”
“Measuring Compositional Generalization: A Comprehensive Method on Realistic Data”, Keysers et al 2019
“Measuring Compositional Generalization: A Comprehensive Method on Realistic Data”
“SimpleBooks: Long-term Dependency Book Dataset With Simplified English Vocabulary for Word-level Language Modeling”, Nguyen 2019
“Single Headed Attention RNN: Stop Thinking With Your Head”, Merity 2019
“Excavate”, Lynch 2019
“MuZero: Mastering Atari, Go, Chess and Shogi by Planning With a Learned Model”, Schrittwieser et al 2019
“MuZero: Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model”
“CommonGen: A Constrained Text Generation Challenge for Generative Commonsense Reasoning”, Lin et al 2019
“CommonGen: A Constrained Text Generation Challenge for Generative Commonsense Reasoning”
“Legendre Memory Units: Continuous-Time Representation in Recurrent Neural Networks”, Voelker et al 2019
“Legendre Memory Units: Continuous-Time Representation in Recurrent Neural Networks”
“High Fidelity Video Prediction With Large Stochastic Recurrent Neural Networks”, Villegas et al 2019
“High Fidelity Video Prediction with Large Stochastic Recurrent Neural Networks”
“SEED RL: Scalable and Efficient Deep-RL With Accelerated Central Inference”, Espeholt et al 2019
“SEED RL: Scalable and Efficient Deep-RL with Accelerated Central Inference”
“Restoring Ancient Text Using Deep Learning (Pythia): a Case Study on Greek Epigraphy”, Assael et al 2019
“Restoring ancient text using deep learning (Pythia): a case study on Greek epigraphy”
“Mixed-Signal Neuromorphic Processors: Quo Vadis?”, Bavandpour et al 2019
“Mogrifier LSTM”, Melis et al 2019
“R2D3: Making Efficient Use of Demonstrations to Solve Hard Exploration Problems”, Paine et al 2019
“R2D3: Making Efficient Use of Demonstrations to Solve Hard Exploration Problems”
“Language Modelling State-of-the-art Leaderboards”, paperswithcode.com 2019
“Metalearned Neural Memory”, Munkhdalai et al 2019
“Generating Text With Recurrent Neural Networks”, Sutskever et al 2019
“Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank”, Socher et al 2019
“Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank”
“XLNet: Generalized Autoregressive Pretraining for Language Understanding”, Yang et al 2019
“XLNet: Generalized Autoregressive Pretraining for Language Understanding”
“Playing the Lottery With Rewards and Multiple Languages: Lottery Tickets in RL and NLP”, Yu et al 2019
“Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLP”
“Reinforcement Learning, Fast and Slow”, Botvinick et al 2019
“MoGlow: Probabilistic and Controllable Motion Synthesis Using Normalizing Flows”, Henter et al 2019
“MoGlow: Probabilistic and controllable motion synthesis using normalizing flows”
“Meta-learners’ Learning Dynamics Are unlike Learners’”, Rabinowitz 2019
“Speech Synthesis from Neural Decoding of Spoken Sentences”, Anumanchipalli et al 2019
“Good News, Everyone! Context Driven Entity-aware Captioning for News Images”, Biten et al 2019
“Good News, Everyone! Context driven entity-aware captioning for news images”
“Surrogate Gradient Learning in Spiking Neural Networks”, Neftci et al 2019
“On the Turing Completeness of Modern Neural Network Architectures”, Pérez et al 2019
“On the Turing Completeness of Modern Neural Network Architectures”
“Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context”, Dai et al 2019
“Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context”
“Natural Questions: A Benchmark for Question Answering Research”, Kwiatkowski et al 2019
“Natural Questions: A Benchmark for Question Answering Research”
“High Fidelity Video Prediction With Large Stochastic Recurrent Neural Networks: Videos”, Villegas et al 2019
“High Fidelity Video Prediction with Large Stochastic Recurrent Neural Networks: Videos”
“Bayesian Layers: A Module for Neural Network Uncertainty”, Tran et al 2018
“Meta-Learning: Learning to Learn Fast”, Weng 2018
“Piano Genie”, Donahue et al 2018
“Learning Recurrent Binary/Ternary Weights”, Ardakani et al 2018
“R2D2: Recurrent Experience Replay in Distributed Reinforcement Learning”, Kapturowski et al 2018
“R2D2: Recurrent Experience Replay in Distributed Reinforcement Learning”
“HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering”, Yang et al 2018
“HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering”
“Object Hallucination in Image Captioning”, Rohrbach et al 2018
“Adversarial Reprogramming of Text Classification Neural Networks”, Neekhara et al 2018
“Adversarial Reprogramming of Text Classification Neural Networks”
“This Time With Feeling: Learning Expressive Musical Performance”, Oore et al 2018
“This Time with Feeling: Learning Expressive Musical Performance”
“Character-Level Language Modeling With Deeper Self-Attention”, Al-Rfou et al 2018
“Character-Level Language Modeling with Deeper Self-Attention”
“General Value Function Networks”, Schlegel et al 2018
“Universal Transformers”, Dehghani et al 2018
“Deep-speare: A Joint Neural Model of Poetic Language, Meter and Rhyme”, Lau et al 2018
“Deep-speare: A Joint Neural Model of Poetic Language, Meter and Rhyme”
“Accurate Uncertainties for Deep Learning Using Calibrated Regression”, Kuleshov et al 2018
“Accurate Uncertainties for Deep Learning Using Calibrated Regression”
“The Natural Language Decathlon: Multitask Learning As Question Answering”, McCann et al 2018
“The Natural Language Decathlon: Multitask Learning as Question Answering”
“Neural Ordinary Differential Equations”, Chen et al 2018
“Know What You Don’t Know: Unanswerable Questions for SQuAD”, Rajpurkar et al 2018
“Know What You Don’t Know: Unanswerable Questions for SQuAD”
“DVRL: Deep Variational Reinforcement Learning for POMDPs”, Igl et al 2018
“Greedy Attack and Gumbel Attack: Generating Adversarial Examples for Discrete Data”, Yang et al 2018
“Greedy Attack and Gumbel Attack: Generating Adversarial Examples for Discrete Data”
“Hierarchical Neural Story Generation”, Fan et al 2018
“Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context”, Khandelwal et al 2018
“Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context”
“Newsroom: A Dataset of 1.3 Million Summaries With Diverse Extractive Strategies”, Grusky et al 2018
“Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategies”
“A Tree Search Algorithm for Sequence Labeling”, Lao et al 2018
“An Analysis of Neural Language Modeling at Multiple Scales”, Merity et al 2018
“An Analysis of Neural Language Modeling at Multiple Scales”
“Reviving and Improving Recurrent Back-Propagation”, Liao et al 2018
“Learning Memory Access Patterns”, Hashemi et al 2018
“Learning Longer-term Dependencies in RNNs With Auxiliary Losses”, Trinh et al 2018
“Learning Longer-term Dependencies in RNNs with Auxiliary Losses”
“One Big Net For Everything”, Schmidhuber 2018
“Efficient Neural Audio Synthesis”, Kalchbrenner et al 2018
“Deep Contextualized Word Representations”, Peters et al 2018
“M-Walk: Learning to Walk over Graphs Using Monte Carlo Tree Search”, Shen et al 2018
“M-Walk: Learning to Walk over Graphs using Monte Carlo Tree Search”
“ULMFiT: Universal Language Model Fine-tuning for Text Classification”, Howard & Ruder 2018
“ULMFiT: Universal Language Model Fine-tuning for Text Classification”
“Overcoming the Vanishing Gradient Problem in Plain Recurrent Networks”, Hu et al 2018
“Overcoming the vanishing gradient problem in plain recurrent networks”
“Large-scale Comparison of Machine Learning Methods for Drug Target Prediction on ChEMBL”, Mayr et al 2018
“Large-scale comparison of machine learning methods for drug target prediction on ChEMBL”
“A Flexible Approach to Automated RNN Architecture Generation”, Schrimpf et al 2017
“A Flexible Approach to Automated RNN Architecture Generation”
“Learning Compact Recurrent Neural Networks With Block-Term Tensor Decomposition”, Ye et al 2017
“Learning Compact Recurrent Neural Networks with Block-Term Tensor Decomposition”
“Mastering the Dungeon: Grounded Language Learning by Mechanical Turker Descent”, Yang et al 2017
“Mastering the Dungeon: Grounded Language Learning by Mechanical Turker Descent”
“Evaluating Prose Style Transfer With the Bible”, Carlson et al 2017
“Breaking the Softmax Bottleneck: A High-Rank RNN Language Model”, Yang et al 2017
“Breaking the Softmax Bottleneck: A High-Rank RNN Language Model”
“Neural Speed Reading via Skim-RNN”, Seo et al 2017
“Generalization without Systematicity: On the Compositional Skills of Sequence-to-sequence Recurrent Networks”, Lake & Baroni 2017
“Word Translation Without Parallel Data”, Conneau et al 2017
“Mixed Precision Training”, Micikevicius et al 2017
“To Prune, or Not to Prune: Exploring the Efficacy of Pruning for Model Compression”, Zhu & Gupta 2017
“To prune, or not to prune: exploring the efficacy of pruning for model compression”
“Dynamic Evaluation of Neural Sequence Models”, Krause et al 2017
“N2N Learning: Network to Network Compression via Policy Gradient Reinforcement Learning”, Ashok et al 2017
“N2N Learning: Network to Network Compression via Policy Gradient Reinforcement Learning”
“Why Pay More When You Can Pay Less: A Joint Learning Framework for Active Feature Acquisition and Classification”, Shim et al 2017
“SRU: Simple Recurrent Units for Highly Parallelizable Recurrence”, Lei et al 2017
“SRU: Simple Recurrent Units for Highly Parallelizable Recurrence”
“Learning to Look Around: Intelligently Exploring Unseen Environments for Unknown Tasks”, Jayaraman & Grauman 2017
“Learning to Look Around: Intelligently Exploring Unseen Environments for Unknown Tasks”
“Skip RNN: Learning to Skip State Updates in Recurrent Neural Networks”, Campos et al 2017
“Skip RNN: Learning to Skip State Updates in Recurrent Neural Networks”
“Twin Networks: Matching the Future for Sequence Generation”, Serdyuk et al 2017
“Twin Networks: Matching the Future for Sequence Generation”
“Revisiting Activation Regularization for Language RNNs”, Merity et al 2017
“Bayesian Sparsification of Recurrent Neural Networks”, Lobacheva et al 2017
“On the State-of-the-Art of Evaluation in Neural Language Models”, Melis et al 2017
“On the State-of-the-Art of Evaluation in Neural Language Models”
“Controlling Linguistic Style Aspects in Neural Language Generation”, Ficler & Goldberg 2017
“Controlling Linguistic Style Aspects in Neural Language Generation”
“Device Placement Optimization With Reinforcement Learning”, Mirhoseini et al 2017
“Language Generation With Recurrent Generative Adversarial Networks without Pre-training”, Press et al 2017
“Language Generation with Recurrent Generative Adversarial Networks without Pre-training”
“Towards Synthesizing Complex Programs from Input-Output Examples”, Chen et al 2017
“Towards Synthesizing Complex Programs from Input-Output Examples”
“Biased Importance Sampling for Deep Neural Network Training”, Katharopoulos & Fleuret 2017
“Biased Importance Sampling for Deep Neural Network Training”
“Deriving Neural Architectures from Sequence and Graph Kernels”, Lei et al 2017
“Deriving Neural Architectures from Sequence and Graph Kernels”
“A Deep Reinforced Model for Abstractive Summarization”, Paulus et al 2017
“DeepTingle”, Khalifa et al 2017
“TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension”, Joshi et al 2017
“TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension”
“A Neural Network System for Transformation of Regional Cuisine Style”, Kazama et al 2017
“A neural network system for transformation of regional cuisine style”
“Sharp Models on Dull Hardware: Fast and Accurate Neural Machine Translation Decoding on the CPU”, Devlin 2017
“Sharp Models on Dull Hardware: Fast and Accurate Neural Machine Translation Decoding on the CPU”
“Adversarial Neural Machine Translation”, Wu et al 2017
“Learning to Reason: End-to-End Module Networks for Visual Question Answering”, Hu et al 2017
“Learning to Reason: End-to-End Module Networks for Visual Question Answering”
“SearchQA: A New Q&A Dataset Augmented With Context from a Search Engine”, Dunn et al 2017
“SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine”
“Exploring Sparsity in Recurrent Neural Networks”, Narang et al 2017
“Get To The Point: Summarization With Pointer-Generator Networks”, See et al 2017
“Get To The Point: Summarization with Pointer-Generator Networks”
“DeepAR: Probabilistic Forecasting With Autoregressive Recurrent Networks”, Salinas et al 2017
“DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks”
“Bayesian Recurrent Neural Networks”, Fortunato et al 2017
“Recurrent Environment Simulators”, Chiappa et al 2017
“Learning to Generate Reviews and Discovering Sentiment”, Radford et al 2017
“Learning Simpler Language Models With the Differential State Framework”, II et al 2017
“Learning Simpler Language Models with the Differential State Framework”
“I2T2I: Learning Text to Image Synthesis With Textual Data Augmentation”, Dong et al 2017
“I2T2I: Learning Text to Image Synthesis with Textual Data Augmentation”
“Improving Neural Machine Translation With Conditional Sequence Generative Adversarial Nets”, Yang et al 2017
“Improving Neural Machine Translation with Conditional Sequence Generative Adversarial Nets”
“Learned Optimizers That Scale and Generalize”, Wichrowska et al 2017
“Parallel Multiscale Autoregressive Density Estimation”, Reed et al 2017
“Tracking the World State With Recurrent Entity Networks”, Henaff et al 2017
“Optimization As a Model for Few-Shot Learning”, Ravi & Larochelle 2017
“Neural Combinatorial Optimization With Reinforcement Learning”, Bello et al 2017
“Neural Combinatorial Optimization with Reinforcement Learning”
“Frustratingly Short Attention Spans in Neural Language Modeling”, Daniluk et al 2017
“Frustratingly Short Attention Spans in Neural Language Modeling”
“Tuning Recurrent Neural Networks With Reinforcement Learning”, Jaques et al 2017
“Tuning Recurrent Neural Networks with Reinforcement Learning”
“Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer”, Shazeer et al 2017
“Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer”
“Neural Data Filter for Bootstrapping Stochastic Gradient Descent”, Fan et al 2017
“Neural Data Filter for Bootstrapping Stochastic Gradient Descent”
“Your TL;DR by an AI: A Deep Reinforced Model for Abstractive Summarization”, Paulus 2017
“Your TL;DR by an AI: A Deep Reinforced Model for Abstractive Summarization”
“SampleRNN: An Unconditional End-to-End Neural Audio Generation Model”, Mehri et al 2016
“SampleRNN: An Unconditional End-to-End Neural Audio Generation Model”
“Improving Neural Language Models With a Continuous Cache”, Grave et al 2016
“NewsQA: A Machine Comprehension Dataset”, Trischler et al 2016
“Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation”, Johnson et al 2016
“Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation”
“Learning to Learn without Gradient Descent by Gradient Descent”, Chen et al 2016
“Learning to Learn without Gradient Descent by Gradient Descent”
“RL2: Fast Reinforcement Learning via Slow Reinforcement Learning”, Duan et al 2016
“RL2: Fast Reinforcement Learning via Slow Reinforcement Learning”
“DeepCoder: Learning to Write Programs”, Balog et al 2016
“Bidirectional Attention Flow for Machine Comprehension”, Seo et al 2016
“Neural Architecture Search With Reinforcement Learning”, Zoph & Le 2016
“QRNNs: Quasi-Recurrent Neural Networks”, Bradbury et al 2016
“Scaling Memory-Augmented Neural Networks With Sparse Reads and Writes”, Rae et al 2016
“Scaling Memory-Augmented Neural Networks with Sparse Reads and Writes”
“Hybrid Computing Using a Neural Network With Dynamic External Memory”, Graves et al 2016
“Hybrid computing using a neural network with dynamic external memory”
“Using Fast Weights to Attend to the Recent Past”, Ba et al 2016
“Achieving Human Parity in Conversational Speech Recognition”, Xiong et al 2016
“Achieving Human Parity in Conversational Speech Recognition”
“Video Pixel Networks”, Kalchbrenner et al 2016
“HyperNetworks”, Ha et al 2016
“Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation”, Wu et al 2016
“Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation”
“Multiplicative LSTM for Sequence Modelling”, Krause et al 2016
“Pointer Sentinel Mixture Models”, Merity et al 2016
“Image-to-Markup Generation With Coarse-to-Fine Attention”, Deng et al 2016
“Hierarchical Multiscale Recurrent Neural Networks”, Chung et al 2016
“Deep Learning Human Mind for Automated Visual Classification”, Spampinato et al 2016
“Deep Learning Human Mind for Automated Visual Classification”
“Decoupled Neural Interfaces Using Synthetic Gradients”, Jaderberg et al 2016
“Full Resolution Image Compression With Recurrent Neural Networks”, Toderici et al 2016
“Full Resolution Image Compression with Recurrent Neural Networks”
“Clockwork Convnets for Video Semantic Segmentation”, Shelhamer et al 2016
“Layer Normalization”, Ba et al 2016
“LSTMVis: A Tool for Visual Analysis of Hidden State Dynamics in Recurrent Neural Networks”, Strobelt et al 2016
“LSTMVis: A Tool for Visual Analysis of Hidden State Dynamics in Recurrent Neural Networks”
“Learning to Learn by Gradient Descent by Gradient Descent”, Andrychowicz et al 2016
“Iterative Alternating Neural Attention for Machine Reading”, Sordoni et al 2016
“Iterative Alternating Neural Attention for Machine Reading”
“Deep Reinforcement Learning for Dialogue Generation”, Li et al 2016
“Programming With a Differentiable Forth Interpreter”, Bošnjak et al 2016
“Training Deep Nets With Sublinear Memory Cost”, Chen et al 2016
“Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex”, Liao & Poggio 2016
“Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex”
“Improving Sentence Compression by Learning to Predict Gaze”, Klerke et al 2016
“Improving sentence compression by learning to predict gaze”
“Adaptive Computation Time for Recurrent Neural Networks”, Graves 2016
“Dynamic Memory Networks for Visual and Textual Question Answering”, Xiong et al 2016
“Dynamic Memory Networks for Visual and Textual Question Answering”
“PlaNet—Photo Geolocation With Convolutional Neural Networks”, Weyand et al 2016
“PlaNet—Photo Geolocation with Convolutional Neural Networks”
“Learning Distributed Representations of Sentences from Unlabeled Data”, Hill et al 2016
“Learning Distributed Representations of Sentences from Unlabeled Data”
“Exploring the Limits of Language Modeling”, Jozefowicz et al 2016
“PixelRNN: Pixel Recurrent Neural Networks”, Oord et al 2016
“Persistent RNNs: Stashing Recurrent Weights On-Chip”, Diamos et al 2016
“Deep-Spying: Spying Using Smartwatch and Deep Learning”, Beltramelli & Risi 2015
“On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models”, Schmidhuber 2015
“Neural GPUs Learn Algorithms”, Kaiser & Sutskever 2015
“Sequence Level Training With Recurrent Neural Networks”, Ranzato et al 2015
“Generating Sentences from a Continuous Space”, Bowman et al 2015
“Neural Programmer-Interpreters”, Reed & Freitas 2015
“Generative Concatenative Nets Jointly Learn to Write and Classify Reviews”, Lipton et al 2015
“Generative Concatenative Nets Jointly Learn to Write and Classify Reviews”
“Generating Images from Captions With Attention”, Mansimov et al 2015
“Semi-supervised Sequence Learning”, Dai & Le 2015
“RNN Metadata for Mimicking Author Style”, Gwern 2015
“Training Recurrent Networks Online without Backtracking”, Ollivier et al 2015
“Deep Recurrent Q-Learning for Partially Observable MDPs”, Hausknecht & Stone 2015
“Teaching Machines to Read and Comprehend”, Hermann et al 2015
“Scheduled Sampling for Sequence Prediction With Recurrent Neural Networks”, Bengio et al 2015
“Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks”
“Visualizing and Understanding Recurrent Networks”, Karpathy et al 2015
“The Unreasonable Effectiveness of Recurrent Neural Networks”, Karpathy 2015
“The Unreasonable Effectiveness of Recurrent Neural Networks”
“Deep Neural Networks for Large Vocabulary Handwritten Text Recognition”, Bluche 2015
“Deep Neural Networks for Large Vocabulary Handwritten Text Recognition”
“Reinforcement Learning Neural Turing Machines—Revised”, Zaremba & Sutskever 2015
“End-To-End Memory Networks”, Sukhbaatar et al 2015
“Inferring Algorithmic Patterns With Stack-Augmented Recurrent Nets”, Arm et al 2015
“Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets”
“DRAW: A Recurrent Neural Network For Image Generation”, Gregor et al 2015
“Neural Turing Machines”, Graves et al 2014
“Learning to Execute”, Zaremba & Sutskever 2014
“Neural Machine Translation by Jointly Learning to Align and Translate”, Bahdanau et al 2014
“Neural Machine Translation by Jointly Learning to Align and Translate”
“Distributed Representations of Sentences and Documents”, Le & Mikolov 2014
“A Clockwork RNN”, Koutník et al 2014
“One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling”, Chelba et al 2013
“One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling”
“Generating Sequences With Recurrent Neural Networks”, Graves 2013
“On the Difficulty of Training Recurrent Neural Networks”, Pascanu et al 2012
“Recurrent Neural Network Based Language Model”, Mikolov et al 2010
“Large Language Models in Machine Translation”, Brants et al 2007
“Learning to Learn Using Gradient Descent”, Hochreiter et al 2001
“Long Short-Term Memory”, Hochreiter & Schmidhuber 1997
“Flat Minima”, Hochreiter & Schmidhuber 1997
“Gradient-Based Learning Algorithms for Recurrent Networks and Their Computational Complexity”, Williams & Zipser 1995
“Gradient-Based Learning Algorithms for Recurrent Networks and Their Computational Complexity”
“A Focused Backpropagation Algorithm for Temporal Pattern Recognition”, Mozer 1995
“A Focused Backpropagation Algorithm for Temporal Pattern Recognition”
“Learning Complex, Extended Sequences Using the Principle of History Compression”, Schmidhuber 1992
“Learning Complex, Extended Sequences Using the Principle of History Compression”
“Learning to Control Fast-Weight Memories: An Alternative to Dynamic Recurrent Networks”, Schmidhuber 1992
“Learning to Control Fast-Weight Memories: An Alternative to Dynamic Recurrent Networks”
“Untersuchungen Zu Dynamischen Neuronalen Netzen [Studies of Dynamic Neural Networks]”, Hochreiter 1991
“Untersuchungen zu dynamischen neuronalen Netzen [Studies of dynamic neural networks]”
“Finding Structure In Time”, Elman 1990
“Connectionist Music Composition Based on Melodic, Stylistic, and Psychophysical Constraints [Technical Report CU-CS–495–90]”, Mozer 1990
“A Learning Algorithm for Continually Running Fully Recurrent Neural Networks”, Williams & Zipser 1989b
“A Learning Algorithm for Continually Running Fully Recurrent Neural Networks”
“A Local Learning Algorithm for Dynamic Feedforward and Recurrent Networks”, Schmidhuber 1989
“A Local Learning Algorithm for Dynamic Feedforward and Recurrent Networks”
“Experimental Analysis of the Real-time Recurrent Learning Algorithm”, Williams & Zipser 1989
“Experimental Analysis of the Real-time Recurrent Learning Algorithm”
“A Sticky-Bit Approach for Learning to Represent State”, Bachrach 1988
“The Utility Driven Dynamic Error Propagation Network (RTRL)”, Robinson & Fallside 1987
“The Utility Driven Dynamic Error Propagation Network (RTRL)”
“Serial Order: A Parallel Distributed Processing Approach”, Jordan 1986
“Attention and Augmented Recurrent Neural Networks”
“Deep Learning for Assisting the Process of Music Composition (part 3)”
“Deep learning for assisting the process of music composition (part 3)”
Sort By Magic
Annotations sorted by machine learning into inferred 'tags'. This provides an alternative way to browse: instead of by date order, one can browse in topic order. The 'sorted' list has been automatically clustered into multiple sections & auto-labeled for easier browsing.
Beginning with the newest annotation, it uses the embedding of each annotation to attempt to create a list of nearest-neighbor annotations, creating a progression of topics. For more details, see the link.
deeplearning
transformer
rnn-usage
sequence-modeling
Wikipedia
Miscellaneous
-
/doc/ai/nn/rnn/2021-jaegle-figure2-perceiverioarchitecture.png -
/doc/ai/nn/rnn/2021-droppo-figure5-lstmvstransfomerscaling.png -
/doc/ai/nn/rnn/2020-deepmind-agent57-figure3-deepreinforcementlearningtimeline.svg -
/doc/ai/nn/rnn/2017-khalifa-example3-incoherentdeeptinglesamplepromptedwithmobydickcallmeishmael.png -
/doc/ai/nn/rnn/2016-06-09-rossgoodwin-adventuresinnarratedreality-2.html -
/doc/ai/nn/rnn/2016-03-19-rossgoodwin-adventuresinnarratedreality-1.html -
/doc/ai/nn/rnn/2015-06-03-karpathy-charrnn-visualization.tar.xz -
https://ahrm.github.io/jekyll/update/2022/04/14/using-languge-models-to-read-faster.html -
https://cprimozic.net/blog/growing-sparse-computational-graphs-with-rnns/ -
https://homepages.inf.ed.ac.uk/abmayne/publications/sennrich2016NAACL.pdf -
https://mi.eng.cam.ac.uk/projects/cued-rnnlm/papers/Interspeech15.pdf -
https://patentimages.storage.googleapis.com/57/53/22/91b8a6792dbb1e/US20180204116A1.pdf#deepmind -
https://soundcloud.com/seaandsailor/sets/char-rnn-composes-irish-folk-music -
https://twitter.com/arankomatsuzaki/status/1639000379978403853 -
https://www.danieldjohnson.com/2015/08/03/composing-music-with-recurrent-neural-networks/ -
https://www.lesswrong.com/posts/bD4B2MF7nsGAfH9fj/basic-mathematics-of-predictive-coding -
https://www.reddit.com/r/MachineLearning/comments/umq908/r_rwkvv2rnn_a_parallelizable_rnn_with/
Link Bibliography
-
https://www.nature.com/articles/s41467-023-42875-2#deepmind: “Learning Few-shot Imitation As Cultural Transmission”, -
aunn: “Absolute Unit NNs: Regression-Based MLPs for Everything”, Gwern -
https://arxiv.org/abs/2305.13048: “RWKV: Reinventing RNNs for the Transformer Era”, -
https://arxiv.org/abs/2303.06349#deepmind: “Resurrecting Recurrent Neural Networks for Long Sequences”, Antonio Orvieto, Samuel L. Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, Soham De -
https://arxiv.org/abs/2302.13939: “SpikeGPT: Generative Pre-trained Language Model With Spiking Neural Networks”, Rui-Jie Zhu, Qihang Zhao, Jason K. Eshraghian -
2023-bures.pdf: “Organic Reaction Mechanism Classification Using Machine Learning”, Jordi Burés, Igor Larrosa -
https://arxiv.org/abs/2212.14052: “Hungry Hungry Hippos: Towards Language Modeling With State Space Models”, Daniel Y. Fu, Tri Dao, Khaled K. Saab, Armin W. Thomas, Atri Rudra, Christopher Ré -
https://arxiv.org/abs/2211.07638: “Legged Locomotion in Challenging Terrains Using Egocentric Vision”, Ananye Agarwal, Ashish Kumar, Jitendra Malik, Deepak Pathak -
https://arxiv.org/abs/2209.11737: “Semantic Scene Descriptions As an Objective of Human Vision”, Adrien Doerig, Tim C. Kietzmann, Emily Allen, Yihan Wu, Thomas Naselaris, Kendrick Kay, Ian Charest -
https://arxiv.org/abs/2205.01972: “Sequencer: Deep LSTM for Image Classification”, Yuki Tatsunami, Masato Taki -
2022-grand.pdf: “Semantic Projection Recovers Rich Human Knowledge of Multiple Object Features from Word Embeddings”, Gabriel Grand, Idan Asher Blank, Francisco Pereira, Evelina Fedorenko -
https://arxiv.org/abs/2203.07852: “Block-Recurrent Transformers”, DeLesley Hutchins, Imanol Schlag, Yuhuai Wu, Ethan Dyer, Behnam Neyshabur -
https://arxiv.org/abs/2202.07765#deepmind: “General-purpose, Long-context Autoregressive Modeling With Perceiver AR”, -
2022-miki.pdf: “Learning Robust Perceptive Locomotion for Quadrupedal Robots in the Wild”, Takahiro Miki, Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, Marco Hutter -
https://arxiv.org/abs/2111.00396: “S4: Efficiently Modeling Long Sequences With Structured State Spaces”, Albert Gu, Karan Goel, Christopher Ré -
https://elifesciences.org/articles/66039: “A Connectome of the Drosophila Central Complex Reveals Network Motifs Suitable for Flexible Navigation and Context-dependent Action Selection”, -
https://arxiv.org/abs/2107.14795#deepmind: “Perceiver IO: A General Architecture for Structured Inputs & Outputs”, -
https://proceedings.mlr.press/v139/vicol21a.html: “PES: Unbiased Gradient Estimation in Unrolled Computation Graphs With Persistent Evolution Strategies”, Paul Vicol, Luke Metz, Jascha Sohl-Dickstein -
2021-delul.pdf: “Shelley: A Crowd-sourced Collaborative Horror Writer”, Pinar Yanardag Delul, Manuel Cebrian, Iyad Rahwan -
2021-jouppi.pdf: “Ten Lessons From Three Generations Shaped Google’s TPUv4i”, -
https://arxiv.org/abs/2106.06981: “RASP: Thinking Like Transformers”, Gail Weiss, Yoav Goldberg, Eran Yahav -
https://arxiv.org/abs/2106.09488#amazon: “Scaling Laws for Acoustic Models”, Jasha Droppo, Oguz Elibol -
https://arxiv.org/abs/2103.03206#deepmind: “Perceiver: General Perception With Iterative Attention”, Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, Joao Carreira -
https://arxiv.org/abs/2102.04159: “Deep Residual Learning in Spiking Neural Networks”, Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier, Yonghong Tian -
https://arxiv.org/abs/2011.12692#tencent: “Towards Playing Full MOBA Games With Deep Reinforcement Learning”, -
https://arxiv.org/abs/2008.07669: “HiPPO: Recurrent Memory With Optimal Polynomial Projections”, Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, Christopher Re -
https://www.lesswrong.com/posts/Wnqua6eQkewL3bqsF/matt-botvinick-on-the-spontaneous-emergence-of-learning: “Matt Botvinick on the Spontaneous Emergence of Learning Algorithms”, Adam Scholl -
https://www.deepmind.com/blog/agent57-outperforming-the-human-atari-benchmark: “Agent57: Outperforming the Human Atari Benchmark”, Adrià Puigdomènech, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Charles Blundell -
https://arxiv.org/abs/2002.03629: “Accelerating Feedforward Computation via Parallel Nonlinear Equation Solving”, Yang Song, Chenlin Meng, Renjie Liao, Stefano Ermon -
https://arxiv.org/abs/2001.08361#openai: “Scaling Laws for Neural Language Models”, -
https://arxiv.org/abs/1911.11423: “Single Headed Attention RNN: Stop Thinking With Your Head”, Stephen Merity -
https://openreview.net/forum?id=HyxlRHBlUB: “Legendre Memory Units: Continuous-Time Representation in Recurrent Neural Networks”, Aaron R. Voelker, Ivana Kajić, Chris Eliasmith -
https://arxiv.org/abs/1910.06591#deepmind: “SEED RL: Scalable and Efficient Deep-RL With Accelerated Central Inference”, Lasse Espeholt, Raphaël Marinier, Piotr Stanczyk, Ke Wang, Marcin Michalski -
https://arxiv.org/abs/1909.01792#deepmind: “Mogrifier LSTM”, Gábor Melis, Tomáš Kočiský, Phil Blunsom -
https://arxiv.org/abs/1905.01320#deepmind: “Meta-learners’ Learning Dynamics Are unlike Learners’”, Neil C. Rabinowitz -
https://openreview.net/forum?id=r1lyTjAqYX#deepmind: “R2D2: Recurrent Experience Replay in Distributed Reinforcement Learning”, Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, Will Dabney -
https://arxiv.org/abs/1801.06146: “ULMFiT: Universal Language Model Fine-tuning for Text Classification”, Jeremy Howard, Sebastian Ruder -
https://arxiv.org/abs/1709.02755: “SRU: Simple Recurrent Units for Highly Parallelizable Recurrence”, Tao Lei, Yu Zhang, Sida I. Wang, Hui Dai, Yoav Artzi -
https://arxiv.org/abs/1704.05526: “Learning to Reason: End-to-End Module Networks for Visual Question Answering”, Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, Kate Saenko -
https://arxiv.org/abs/1704.05179: “SearchQA: A New Q&A Dataset Augmented With Context from a Search Engine”, Matthew Dunn, Levent Sagun, Mike Higgins, V. Ugur Guney, Volkan Cirik, Kyunghyun Cho -
https://arxiv.org/abs/1608.03609: “Clockwork Convnets for Video Semantic Segmentation”, Evan Shelhamer, Kate Rakelly, Judy Hoffman, Trevor Darrell -
rnn-metadata: “RNN Metadata for Mimicking Author Style”, Gwern -
2010-mikolov.pdf: “Recurrent Neural Network Based Language Model”, Tomas Mikolov, Martin Karafiat, Lukas Burget, Jan Cernocky, Sanjeev Khudanpur -
1991-hochreiter.pdf: “Untersuchungen Zu Dynamischen Neuronalen Netzen [Studies of Dynamic Neural Networks]”, Sepp Hochreiter -
1989-williams.pdf: “Experimental Analysis of the Real-time Recurrent Learning Algorithm”, Ronald J. Williams, David Zipser