“‘RNN’ Tag”,2019-08-30 (MLP NN, self-attention)⁠:

Bibliography for tag ai/nn/rnn, most recent first: 5 related tags, 368 annotations, & 40 links (parent).

• See Also
• Gwern
  • “Absolute Unit NNs: Regression-Based MLPs for Everything”, Gwern2023
  • “RNN Metadata for Mimicking Author Style”, Gwern2015
• Links
  • “Hymba: A Hybrid-Head Architecture for Small Language Models”, Dong et al 2024
  • “State-Space Models Can Learn In-Context by Gradient Descent”, Sushma et al 2024
  • “Were RNNs All We Needed?”, Feng et al 2024
  • “The Mamba in the Llama: Distilling and Accelerating Hybrid Models”, Wang et al 2024
  • “handwriter.ttf: Handwriting Synthesis With Harfbuzz WASM”, Jingyi2024
  • “Learning to (Learn at Test Time): RNNs With Expressive Hidden States”, Sun et al 2024
  • “An Empirical Study of Mamba-Based Language Models”, Waleffe et al 2024
  • “State Soup: In-Context Skill Learning, Retrieval and Mixing”, Pióro et al 2024
  • “Grokfast: Accelerated Grokking by Amplifying Slow Gradients”, Lee et al 2024
  • “Attention As an RNN”, Feng et al 2024
  • “XLSTM: Extended Long Short-Term Memory”, Beck et al 2024
  • “Megalodon: Efficient LLM Pretraining and Inference With Unlimited Context Length”, Ma et al 2024
  • “The Illusion of State in State-Space Models”, Merrill et al 2024
  • “An Accurate and Rapidly Calibrating Speech Neuroprosthesis”, Card et al 2024
  • “Does Transformer Interpretability Transfer to RNNs?”, Paulo et al 2024
  • “Mechanistic Design and Scaling of Hybrid Architectures”, Poli et al 2024
  • “GLE: Backpropagation through Space, Time, and the Brain”, Ellenberger et al 2024
  • “ZigMa: Zigzag Mamba Diffusion Model”, Hu et al 2024
  • “RNNs Are Not Transformers (Yet): The Key Bottleneck on In-Context Retrieval”, Wen et al 2024
  • “MambaByte: Token-Free Selective State Space Model”, Wang et al 2024
  • “MoE-Mamba: Efficient Selective State Space Models With Mixture of Experts”, Pióro et al 2024
  • “Evolving Reservoirs for Meta Reinforcement Learning”, Léger et al 2023
  • “Zoology: Measuring and Improving Recall in Efficient Language Models”, Arora et al 2023
  • “Mamba: Linear-Time Sequence Modeling With Selective State Spaces”, Gu & Dao2023
  • “Diffusion Models Without Attention”, Yan et al 2023
  • “Learning Few-Shot Imitation As Cultural Transmission”, Bhoopchand et al 2023
  • “Compositional Capabilities of Autoregressive Transformers: A Study on Synthetic, Interpretable Tasks”, Ramesh et al 2023
  • “HGRN: Hierarchically Gated Recurrent Neural Network for Sequence Modeling”, Qin et al 2023
  • “On Prefrontal Working Memory and Hippocampal Episodic Memory: Unifying Memories Stored in Weights and Activation Slots”, Whittington et al 2023
  • “GateLoop: Fully Data-Controlled Linear Recurrence for Sequence Modeling”, Katsch2023
  • “ProSG: Using Prompt Synthetic Gradients to Alleviate Prompt Forgetting of RNN-Like Language Models”, Luo et al 2023
  • “Transformers Learn Higher-Order Optimization Methods for In-Context Learning: A Study With Linear Models”, Fu et al 2023
  • “Generalization in Sensorimotor Networks Configured With Natural Language Instructions”, Riveland & Pouget2023
  • “Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priors”, Amos et al 2023
  • “Parallelizing Non-Linear Sequential Models over the Sequence Length”, Lim et al 2023
  • “A High-Performance Neuroprosthesis for Speech Decoding and Avatar Control”, Metzger et al 2023
  • “Learning to Model the World With Language”, Lin et al 2023
  • “Retentive Network: A Successor to Transformer for Large Language Models”, Sun et al 2023
  • “Using Sequences of Life-Events to Predict Human Lives”, Savcisens et al 2023
  • “Thought Cloning: Learning to Think While Acting by Imitating Human Thinking”, Hu & Clune2023
  • “RWKV: Reinventing RNNs for the Transformer Era”, Peng et al 2023
  • “Emergence of Belief-Like Representations through Reinforcement Learning”, Hennig et al 2023
  • “Model Scale versus Domain Knowledge in Statistical Forecasting of Chaotic Systems”, Gilpin2023
  • “Resurrecting Recurrent Neural Networks for Long Sequences”, Orvieto et al 2023
  • “SpikeGPT: Generative Pre-Trained Language Model With Spiking Neural Networks”, Zhu et al 2023
  • “Organic Reaction Mechanism Classification Using Machine Learning”, Burés & Larrosa2023
  • “A High-Performance Speech Neuroprosthesis”, Willett et al 2023
  • “Hungry Hungry Hippos: Towards Language Modeling With State Space Models”, Fu et al 2022
  • “Pretraining Without Attention”, Wang et al 2022
  • “A 64-Core Mixed-Signal In-Memory Compute Chip Based on Phase-Change Memory for Deep Neural Network Inference”, Gallo et al 2022
  • “Melting Pot 2.0”, Agapiou et al 2022
  • “VeLO: Training Versatile Learned Optimizers by Scaling Up”, Metz et al 2022
  • “Legged Locomotion in Challenging Terrains Using Egocentric Vision”, Agarwal et al 2022
  • “Massively Multilingual ASR on 70 Languages: Tokenization, Architecture, and Generalization Capabilities”, Tjandra et al 2022
  • “Perfectly Secure Steganography Using Minimum Entropy Coupling”, Witt et al 2022
  • “Transformers Learn Shortcuts to Automata”, Liu et al 2022
  • “Omnigrok: Grokking Beyond Algorithmic Data”, Liu et al 2022
  • “Semantic Scene Descriptions As an Objective of Human Vision”, Doerig et al 2022
  • “Benchmarking Compositionality With Formal Languages”, Valvoda et al 2022
  • “Learning to Generalize With Object-Centric Agents in the Open World Survival Game Crafter”, Stanić et al 2022
  • “PI-ARS: Accelerating Evolution-Learned Visual-Locomotion With Predictive Information Representations”, Lee et al 2022
  • “Spatial Representation by Ramping Activity of Neurons in the Retrohippocampal Cortex”, Tennant et al 2022
  • “Neural Networks and the Chomsky Hierarchy”, Delétang et al 2022
  • “BYOL-Explore: Exploration by Bootstrapped Prediction”, Guo et al 2022
  • “AnimeSR: Learning Real-World Super-Resolution Models for Animation Videos”, Wu et al 2022
  • “Task-Agnostic Continual Reinforcement Learning: In Praise of a Simple Baseline (3RL)”, Caccia et al 2022
  • “Simple Recurrence Improves Masked Language Models”, Lei et al 2022
  • “Sequencer: Deep LSTM for Image Classification”, Tatsunami & Taki2022
  • “Data Distributional Properties Drive Emergent Few-Shot Learning in Transformers”, Chan et al 2022
  • “Semantic Projection Recovers Rich Human Knowledge of Multiple Object Features from Word Embeddings”, Grand et al 2022
  • “Block-Recurrent Transformers”, Hutchins et al 2022
  • “All You Need Is Supervised Learning: From Imitation Learning to Meta-RL With Upside Down RL”, Arulkumaran et al 2022
  • “Retrieval-Augmented Reinforcement Learning”, Goyal et al 2022
  • “Learning by Directional Gradient Descent”, Silver et al 2022
  • “General-Purpose, Long-Context Autoregressive Modeling With Perceiver AR”, Hawthorne et al 2022
  • “End-To-End Algorithm Synthesis With Recurrent Networks: Logical Extrapolation Without Overthinking”, Bansal et al 2022
  • “Data Scaling Laws in NMT: The Effect of Noise and Architecture”, Bansal et al 2022
  • “Active Predictive Coding Networks: A Neural Solution to the Problem of Learning Reference Frames and Part-Whole Hierarchies”, Gklezakos & Rao2022
  • “Learning Robust Perceptive Locomotion for Quadrupedal Robots in the Wild”, Miki et al 2022
  • “Inducing Causal Structure for Interpretable Neural Networks (IIT)”, Geiger et al 2021
  • “Evaluating Distributional Distortion in Neural Language Modeling”, Anonymous2021
  • “Gradients Are Not All You Need”, Metz et al 2021
  • “An Explanation of In-Context Learning As Implicit Bayesian Inference”, Xie et al 2021
  • “S4: Efficiently Modeling Long Sequences With Structured State Spaces”, Gu et al 2021
  • “Minimum Description Length Recurrent Neural Networks”, Lan et al 2021
  • “LSSL: Combining Recurrent, Convolutional, and Continuous-Time Models With Linear State-Space Layers”, Gu et al 2021
  • “A Connectome of the Drosophila Central Complex Reveals Network Motifs Suitable for Flexible Navigation and Context-Dependent Action Selection”, Hulse et al 2021
  • “Recurrent Model-Free RL Is a Strong Baseline for Many POMDPs”, Ni et al 2021
  • “Photos Are All You Need for Reciprocal Recommendation in Online Dating”, Neve & McConville2021
  • “Perceiver IO: A General Architecture for Structured Inputs & Outputs”, Jaegle et al 2021
  • “PES: Unbiased Gradient Estimation in Unrolled Computation Graphs With Persistent Evolution Strategies”, Vicol et al 2021
  • “Shelley: A Crowd-Sourced Collaborative Horror Writer”, Delul et al 2021
  • “Ten Lessons From Three Generations Shaped Google’s TPUv4i”, Jouppi et al 2021
  • “RASP: Thinking Like Transformers”, Weiss et al 2021
  • “Scaling Laws for Acoustic Models”, Droppo & Elibol2021
  • “Scaling End-To-End Models for Large-Scale Multilingual ASR”, Li et al 2021
  • “Sensitivity As a Complexity Measure for Sequence Classification Tasks”, Hahn et al 2021
  • “ALD: Efficient Transformers in Reinforcement Learning Using Actor-Learner Distillation”, Parisotto & Salakhutdinov2021
  • “Finetuning Pretrained Transformers into RNNs”, Kasai et al 2021
  • “Pretrained Transformers As Universal Computation Engines”, Lu et al 2021
  • “Perceiver: General Perception With Iterative Attention”, Jaegle et al 2021
  • “When Attention Meets Fast Recurrence: Training SRU++ Language Models With Reduced Compute”, Lei2021
  • “Generative Speech Coding With Predictive Variance Regularization”, Kleijn et al 2021
  • “Predictive Coding Is a Consequence of Energy Efficiency in Recurrent Neural Networks”, Ali et al 2021
  • “Deep Residual Learning in Spiking Neural Networks”, Fang et al 2021
  • “Distilling Large Language Models into Tiny and Effective Students Using PQRNN”, Kaliamoorthi et al 2021
  • “Meta Learning Backpropagation And Improving It”, Kirsch & Schmidhuber2020
  • “On the Binding Problem in Artificial Neural Networks”, Greff et al 2020
  • “A Recurrent Vision-And-Language BERT for Navigation”, Hong et al 2020
  • “Towards Playing Full MOBA Games With Deep Reinforcement Learning”, Ye et al 2020
  • “Multimodal Dynamics Modeling for Off-Road Autonomous Vehicles”, Tremblay et al 2020
  • “Adversarial Vulnerabilities of Human Decision-Making”, Dezfouli et al 2020
  • “Learning to Summarize Long Texts With Memory Compression and Transfer”, Park et al 2020
  • “Human-Centric Dialog Training via Offline Reinforcement Learning”, Jaques et al 2020
  • “AFT: An Attention Free Transformer”, Anonymous2020
  • “Deep Reinforcement Learning for Closed-Loop Blood Glucose Control”, Fox et al 2020
  • “HiPPO: Recurrent Memory With Optimal Polynomial Projections”, Gu et al 2020
  • “Adding Recurrence to Pretrained Transformers for Improved Efficiency and Context Size”, Yoshida et al 2020
  • “Matt Botvinick on the Spontaneous Emergence of Learning Algorithms”, Scholl2020
  • “Cultural Influences on Word Meanings Revealed through Large-Scale Semantic Alignment”, Thompson et al 2020
  • “DeepSinger: Singing Voice Synthesis With Data Mined From the Web”, Ren et al 2020
  • “High-Performance Brain-To-Text Communication via Imagined Handwriting”, Willett et al 2020
  • “Transformers Are RNNs: Fast Autoregressive Transformers With Linear Attention”, Katharopoulos et al 2020
  • “The Recurrent Neural Tangent Kernel”, Alemohammad et al 2020
  • “Untangling Tradeoffs between Recurrence and Self-Attention in Neural Networks”, Kerg et al 2020
  • “Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing”, Dai et al 2020
  • “Learning Music Helps You Read: Using Transfer to Study Linguistic Structure in Language Models”, Papadimitriou & Jurafsky2020
  • “Syntactic Structure from Deep Learning”, Linzen & Baroni2020
  • “Agent57: Outperforming the Human Atari Benchmark”, Puigdomènech et al 2020
  • “Machine Translation of Cortical Activity to Text With an Encoder-Decoder Framework”, Makin et al 2020
  • “Learning-Based Memory Allocation for C++ Server Workloads”, Maas et al 2020
  • “Accelerating Feedforward Computation via Parallel Nonlinear Equation Solving”, Song et al 2020
  • “Direct Fit to Nature: An Evolutionary Perspective on Biological and Artificial Neural Networks”, Hasson et al 2020
  • “Scaling Laws for Neural Language Models”, Kaplan et al 2020
  • “Estimating the Deep Replicability of Scientific Findings Using Human and Artificial Intelligence”, Yang et al 2020
  • “Placing Language in an Integrated Understanding System: Next Steps toward Human-Level Performance in Neural Language Models”, McClelland et al 2020
  • “Measuring Compositional Generalization: A Comprehensive Method on Realistic Data”, Keysers et al 2019
  • “SimpleBooks: Long-Term Dependency Book Dataset With Simplified English Vocabulary for Word-Level Language Modeling”, Nguyen2019
  • “Single Headed Attention RNN: Stop Thinking With Your Head”, Merity2019
  • “Excavate”, Lynch2019
  • “MuZero: Mastering Atari, Go, Chess and Shogi by Planning With a Learned Model”, Schrittwieser et al 2019
  • “CommonGen: A Constrained Text Generation Challenge for Generative Commonsense Reasoning”, Lin et al 2019
  • “High Fidelity Video Prediction With Large Stochastic Recurrent Neural Networks”, Villegas et al 2019
  • “Legendre Memory Units: Continuous-Time Representation in Recurrent Neural Networks”, Voelker et al 2019
  • “SEED RL: Scalable and Efficient Deep-RL With Accelerated Central Inference”, Espeholt et al 2019
  • “Mixed-Signal Neuromorphic Processors: Quo Vadis?”, Bavandpour et al 2019
  • “Restoring Ancient Text Using Deep Learning (Pythia): a Case Study on Greek Epigraphy”, Assael et al 2019
  • “Mogrifier LSTM”, Melis et al 2019
  • “R2D3: Making Efficient Use of Demonstrations to Solve Hard Exploration Problems”, Paine et al 2019
  • “Language Modeling State-Of-The-Art Leaderboards”, paperswithcode.com 2019
  • “Metalearned Neural Memory”, Munkhdalai et al 2019
  • “Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank”, Socher et al 2019
  • “Generating Text With Recurrent Neural Networks”, Sutskever et al 2019
  • “XLNet: Generalized Autoregressive Pretraining for Language Understanding”, Yang et al 2019
  • “Playing the Lottery With Rewards and Multiple Languages: Lottery Tickets in RL and NLP”, Yu et al 2019
  • “MoGlow: Probabilistic and Controllable Motion Synthesis Using Normalizing Flows”, Henter et al 2019
  • “Reinforcement Learning, Fast and Slow”, Botvinick et al 2019
  • “Meta-Learners’ Learning Dynamics Are unlike Learners’”, Rabinowitz2019
  • “Speech Synthesis from Neural Decoding of Spoken Sentences”, Anumanchipalli et al 2019
  • “Good News, Everyone! Context Driven Entity-Aware Captioning for News Images”, Biten et al 2019
  • “Surrogate Gradient Learning in Spiking Neural Networks”, Neftci et al 2019
  • “On the Turing Completeness of Modern Neural Network Architectures”, Pérez et al 2019
  • “Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context”, Dai et al 2019
  • “Natural Questions: A Benchmark for Question Answering Research”, Kwiatkowski et al 2019
  • “High Fidelity Video Prediction With Large Stochastic Recurrent Neural Networks: Videos”, Villegas et al 2019
  • “Bayesian Layers: A Module for Neural Network Uncertainty”, Tran et al 2018
  • “Meta-Learning: Learning to Learn Fast”, Weng2018
  • “Piano Genie”, Donahue et al 2018
  • “Learning Recurrent Binary/Ternary Weights”, Ardakani et al 2018
  • “R2D2: Recurrent Experience Replay in Distributed Reinforcement Learning”, Kapturowski et al 2018
  • “HotpotQA: A Dataset for Diverse, Explainable Multi-Hop Question Answering”, Yang et al 2018
  • “Adversarial Reprogramming of Text Classification Neural Networks”, Neekhara et al 2018
  • “Object Hallucination in Image Captioning”, Rohrbach et al 2018
  • “This Time With Feeling: Learning Expressive Musical Performance”, Oore et al 2018
  • “Character-Level Language Modeling With Deeper Self-Attention”, Al-Rfou et al 2018
  • “General Value Function Networks”, Schlegel et al 2018
  • “Deep-Speare: A Joint Neural Model of Poetic Language, Meter and Rhyme”, Lau et al 2018
  • “Universal Transformers”, Dehghani et al 2018
  • “Accurate Uncertainties for Deep Learning Using Calibrated Regression”, Kuleshov et al 2018
  • “The Natural Language Decathlon: Multitask Learning As Question Answering”, McCann et al 2018
  • “Neural Ordinary Differential Equations”, Chen et al 2018
  • “Know What You Don’t Know: Unanswerable Questions for SQuAD”, Rajpurkar et al 2018
  • “DVRL: Deep Variational Reinforcement Learning for POMDPs”, Igl et al 2018
  • “Greedy Attack and Gumbel Attack: Generating Adversarial Examples for Discrete Data”, Yang et al 2018
  • “Hierarchical Neural Story Generation”, Fan et al 2018
  • “Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context”, Khandelwal et al 2018
  • “Newsroom: A Dataset of 1.3 Million Summaries With Diverse Extractive Strategies”, Grusky et al 2018
  • “A Tree Search Algorithm for Sequence Labeling”, Lao et al 2018
  • “An Analysis of Neural Language Modeling at Multiple Scales”, Merity et al 2018
  • “Reviving and Improving Recurrent Back-Propagation”, Liao et al 2018
  • “Learning Memory Access Patterns”, Hashemi et al 2018
  • “Learning Longer-Term Dependencies in RNNs With Auxiliary Losses”, Trinh et al 2018
  • “One Big Net For Everything”, Schmidhuber2018
  • “Efficient Neural Audio Synthesis”, Kalchbrenner et al 2018
  • “Deep Contextualized Word Representations”, Peters et al 2018
  • “M-Walk: Learning to Walk over Graphs Using Monte Carlo Tree Search”, Shen et al 2018
  • “Overcoming the Vanishing Gradient Problem in Plain Recurrent Networks”, Hu et al 2018
  • “ULMFiT: Universal Language Model Fine-Tuning for Text Classification”, Howard & Ruder2018
  • “Large-Scale Comparison of Machine Learning Methods for Drug Target Prediction on ChEMBL”, Mayr et al 2018
  • “A Flexible Approach to Automated RNN Architecture Generation”, Schrimpf et al 2017
  • “The NarrativeQA Reading Comprehension Challenge”, Kočiský et al 2017
  • “Learning Compact Recurrent Neural Networks With Block-Term Tensor Decomposition”, Ye et al 2017
  • “Mastering the Dungeon: Grounded Language Learning by Mechanical Turker Descent”, Yang et al 2017
  • “Evaluating Prose Style Transfer With the Bible”, Carlson et al 2017
  • “Breaking the Softmax Bottleneck: A High-Rank RNN Language Model”, Yang et al 2017
  • “Neural Speed Reading via Skim-RNN”, Seo et al 2017
  • “Unsupervised Machine Translation Using Monolingual Corpora Only”, Lample et al 2017
  • “Generalization without Systematicity: On the Compositional Skills of Sequence-To-Sequence Recurrent Networks”, Lake & Baroni2017
  • “Mixed Precision Training”, Micikevicius et al 2017
  • “To Prune, or Not to Prune: Exploring the Efficacy of Pruning for Model Compression”, Zhu & Gupta2017
  • “Dynamic Evaluation of Neural Sequence Models”, Krause et al 2017
  • “Online Learning of a Memory for Learning Rates”, Meier et al 2017
  • “Why Pay More When You Can Pay Less: A Joint Learning Framework for Active Feature Acquisition and Classification”, Shim et al 2017
  • “N2N Learning: Network to Network Compression via Policy Gradient Reinforcement Learning”, Ashok et al 2017
  • “SRU: Simple Recurrent Units for Highly Parallelizable Recurrence”, Lei et al 2017
  • “Learning to Look Around: Intelligently Exploring Unseen Environments for Unknown Tasks”, Jayaraman & Grauman2017
  • “Twin Networks: Matching the Future for Sequence Generation”, Serdyuk et al 2017
  • “Skip RNN: Learning to Skip State Updates in Recurrent Neural Networks”, Campos et al 2017
  • “Revisiting Activation Regularization for Language RNNs”, Merity et al 2017
  • “Bayesian Sparsification of Recurrent Neural Networks”, Lobacheva et al 2017
  • “On the State-Of-The-Art of Evaluation in Neural Language Models”, Melis et al 2017
  • “Controlling Linguistic Style Aspects in Neural Language Generation”, Ficler & Goldberg2017
  • “Device Placement Optimization With Reinforcement Learning”, Mirhoseini et al 2017
  • “Six Challenges for Neural Machine Translation”, Koehn & Knowles2017
  • “Towards Synthesizing Complex Programs from Input-Output Examples”, Chen et al 2017
  • “Language Generation With Recurrent Generative Adversarial Networks without Pre-Training”, Press et al 2017
  • “Biased Importance Sampling for Deep Neural Network Training”, Katharopoulos & Fleuret2017
  • “Deriving Neural Architectures from Sequence and Graph Kernels”, Lei et al 2017
  • “A Deep Reinforced Model for Abstractive Summarization”, Paulus et al 2017
  • “TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension”, Joshi et al 2017
  • “DeepTingle”, Khalifa et al 2017
  • “A Neural Network System for Transformation of Regional Cuisine Style”, Kazama et al 2017
  • “Sharp Models on Dull Hardware: Fast and Accurate Neural Machine Translation Decoding on the CPU”, Devlin2017
  • “Adversarial Neural Machine Translation”, Wu et al 2017
  • “SearchQA: A New Q&A Dataset Augmented With Context from a Search Engine”, Dunn et al 2017
  • “Learning to Reason: End-To-End Module Networks for Visual Question Answering”, Hu et al 2017
  • “Exploring Sparsity in Recurrent Neural Networks”, Narang et al 2017
  • “Get To The Point: Summarization With Pointer-Generator Networks”, See et al 2017
  • “DeepAR: Probabilistic Forecasting With Autoregressive Recurrent Networks”, Salinas et al 2017
  • “Bayesian Recurrent Neural Networks”, Fortunato et al 2017
  • “Recurrent Environment Simulators”, Chiappa et al 2017
  • “Learning to Generate Reviews and Discovering Sentiment”, Radford et al 2017
  • “Learning Simpler Language Models With the Differential State Framework”, II et al 2017
  • “I2T2I: Learning Text to Image Synthesis With Textual Data Augmentation”, Dong et al 2017
  • “Improving Neural Machine Translation With Conditional Sequence Generative Adversarial Nets”, Yang et al 2017
  • “Learned Optimizers That Scale and Generalize”, Wichrowska et al 2017
  • “Parallel Multiscale Autoregressive Density Estimation”, Reed et al 2017
  • “Tracking the World State With Recurrent Entity Networks”, Henaff et al 2017
  • “Optimization As a Model for Few-Shot Learning”, Ravi & Larochelle2017
  • “Neural Combinatorial Optimization With Reinforcement Learning”, Bello et al 2017
  • “Frustratingly Short Attention Spans in Neural Language Modeling”, Daniluk et al 2017
  • “Tuning Recurrent Neural Networks With Reinforcement Learning”, Jaques et al 2017
  • “Outrageously Large Neural Networks: The Sparsely-Gated Mixture-Of-Experts Layer”, Shazeer et al 2017
  • “Neural Data Filter for Bootstrapping Stochastic Gradient Descent”, Fan et al 2017
  • “Learning the Enigma With Recurrent Neural Networks”, Greydanus2017
  • “Your TL;DR by an AI: A Deep Reinforced Model for Abstractive Summarization”, Paulus2017
  • “SampleRNN: An Unconditional End-To-End Neural Audio Generation Model”, Mehri et al 2016
  • “Improving Neural Language Models With a Continuous Cache”, Grave et al 2016
  • “NewsQA: A Machine Comprehension Dataset”, Trischler et al 2016
  • “Neural Combinatorial Optimization With Reinforcement Learning”, Bello et al 2016
  • “Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation”, Johnson et al 2016
  • “Learning to Learn without Gradient Descent by Gradient Descent”, Chen et al 2016
  • “RL²: Fast Reinforcement Learning via Slow Reinforcement Learning”, Duan et al 2016
  • “DeepCoder: Learning to Write Programs”, Balog et al 2016
  • “QRNNs: Quasi-Recurrent Neural Networks”, Bradbury et al 2016
  • “Neural Architecture Search With Reinforcement Learning”, Zoph & Le2016
  • “Bidirectional Attention Flow for Machine Comprehension”, Seo et al 2016
  • “Hybrid Computing Using a Neural Network With Dynamic External Memory”, Graves et al 2016
  • “Scaling Memory-Augmented Neural Networks With Sparse Reads and Writes”, Rae et al 2016
  • “Using Fast Weights to Attend to the Recent Past”, Ba et al 2016
  • “Achieving Human Parity in Conversational Speech Recognition”, Xiong et al 2016
  • “VPN: Video Pixel Networks”, Kalchbrenner et al 2016
  • “HyperNetworks”, Ha et al 2016
  • “Pointer Sentinel Mixture Models”, Merity et al 2016
  • “Multiplicative LSTM for Sequence Modeling”, Krause et al 2016
  • “Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation”, Wu et al 2016
  • “Image-To-Markup Generation With Coarse-To-Fine Attention”, Deng et al 2016
  • “Hierarchical Multiscale Recurrent Neural Networks”, Chung et al 2016
  • “Deep Learning Human Mind for Automated Visual Classification”, Spampinato et al 2016
  • “Using the Output Embedding to Improve Language Models”, Press & Wolf2016
  • “Full Resolution Image Compression With Recurrent Neural Networks”, Toderici et al 2016
  • “Decoupled Neural Interfaces Using Synthetic Gradients”, Jaderberg et al 2016
  • “Clockwork Convnets for Video Semantic Segmentation”, Shelhamer et al 2016
  • “Layer Normalization”, Ba et al 2016
  • “LSTMVis: A Tool for Visual Analysis of Hidden State Dynamics in Recurrent Neural Networks”, Strobelt et al 2016
  • “Learning to Learn by Gradient Descent by Gradient Descent”, Andrychowicz et al 2016
  • “Iterative Alternating Neural Attention for Machine Reading”, Sordoni et al 2016
  • “Deep Reinforcement Learning for Dialogue Generation”, Li et al 2016
  • “Programming With a Differentiable Forth Interpreter”, Bošnjak et al 2016
  • “Training Deep Nets With Sublinear Memory Cost”, Chen et al 2016
  • “Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex”, Liao & Poggio2016
  • “Improving Sentence Compression by Learning to Predict Gaze”, Klerke et al 2016
  • “Adaptive Computation Time for Recurrent Neural Networks”, Graves2016
  • “Dynamic Memory Networks for Visual and Textual Question Answering”, Xiong et al 2016
  • “PlaNet—Photo Geolocation With Convolutional Neural Networks”, Weyand et al 2016
  • “Learning Distributed Representations of Sentences from Unlabeled Data”, Hill et al 2016
  • “Exploring the Limits of Language Modeling”, Jozefowicz et al 2016
  • “PixelRNN: Pixel Recurrent Neural Networks”, Oord et al 2016
  • “Persistent RNNs: Stashing Recurrent Weights On-Chip”, Diamos et al 2016
  • “Exploring the Limits of Language Modeling § 5.9: Samples from the Model”
  • “Deep-Spying: Spying Using Smartwatch and Deep Learning”, Beltramelli & Risi2015
  • “On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models”, Schmidhuber2015
  • “Neural GPUs Learn Algorithms”, Kaiser & Sutskever2015
  • “Sequence Level Training With Recurrent Neural Networks”, Ranzato et al 2015
  • “Neural Programmer-Interpreters”, Reed & Freitas2015
  • “Generating Sentences from a Continuous Space”, Bowman et al 2015
  • “Generative Concatenative Nets Jointly Learn to Write and Classify Reviews”, Lipton et al 2015
  • “Generating Images from Captions With Attention”, Mansimov et al 2015
  • “Semi-Supervised Sequence Learning”, Dai & Le2015
  • “BPEs: Neural Machine Translation of Rare Words With Subword Units”, Sennrich et al 2015
  • “Training Recurrent Networks Online without Backtracking”, Ollivier et al 2015
  • “Deep Recurrent Q-Learning for Partially Observable MDPs”, Hausknecht & Stone2015
  • “Teaching Machines to Read and Comprehend”, Hermann et al 2015
  • “Scheduled Sampling for Sequence Prediction With Recurrent Neural Networks”, Bengio et al 2015
  • “Visualizing and Understanding Recurrent Networks”, Karpathy et al 2015
  • “The Unreasonable Effectiveness of Recurrent Neural Networks”, Karpathy2015
  • “Deep Neural Networks for Large Vocabulary Handwritten Text Recognition”, Bluche2015
  • “Reinforcement Learning Neural Turing Machines—Revised”, Zaremba & Sutskever2015
  • “End-To-End Memory Networks”, Sukhbaatar et al 2015
  • “LSTM: A Search Space Odyssey”, Greff et al 2015
  • “Inferring Algorithmic Patterns With Stack-Augmented Recurrent Nets”, Holdings et al 2015
  • “DRAW: A Recurrent Neural Network For Image Generation”, Gregor et al 2015
  • “Ensemble of Generative and Discriminative Techniques for Sentiment Analysis of Movie Reviews”, Mesnil et al 2014
  • “Neural Turing Machines”, Graves et al 2014
  • “Learning to Execute”, Zaremba & Sutskever2014
  • “Neural Machine Translation by Jointly Learning to Align and Translate”, Bahdanau et al 2014
  • “Identifying and Attacking the Saddle Point Problem in High-Dimensional Non-Convex Optimization”, Dauphin et al 2014
  • “GRU: Learning Phrase Representations Using RNN Encoder-Decoder for Statistical Machine Translation”, Cho et al 2014
  • “doc2vec: Distributed Representations of Sentences and Documents”, Le & Mikolov2014
  • “A Clockwork RNN”, Koutník et al 2014
  • “One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling”, Chelba et al 2013
  • “Generating Sequences With Recurrent Neural Networks”, Graves2013
  • “On the Difficulty of Training Recurrent Neural Networks”, Pascanu et al 2012
  • “Recurrent Neural Network Based Language Model”, Mikolov et al 2010
  • “Large Language Models in Machine Translation”, Brants et al 2007
  • “Learning to Learn Using Gradient Descent”, Hochreiter et al 2001
  • “Long Short-Term Memory”, Hochreiter & Schmidhuber1997
  • “Flat Minima”, Hochreiter & Schmidhuber1997
  • “Gradient-Based Learning Algorithms for Recurrent Networks and Their Computational Complexity”, Williams & Zipser1995
  • “A Focused Backpropagation Algorithm for Temporal Pattern Recognition”, Mozer1995
  • “Learning Complex, Extended Sequences Using the Principle of History Compression”, Schmidhuber1992
  • “Learning to Control Fast-Weight Memories: An Alternative to Dynamic Recurrent Networks”, Schmidhuber1992
  • “Untersuchungen Zu Dynamischen Neuronalen Netzen [Studies of Dynamic Neural Networks]”, Hochreiter1991
  • “Finding Structure In Time”, Elman1990
  • “Connectionist Music Composition Based on Melodic, Stylistic, and Psychophysical Constraints [Technical Report CU-CS–495–90]”, Mozer1990
  • “A Learning Algorithm for Continually Running Fully Recurrent Neural Networks”, Williams & Zipser1989b
  • “Recurrent Backpropagation and Hopfield Networks”, Almeida & Neto1989b
  • “Backpropagation in Perceptrons With Feedback”, Almeida1989
  • “Experimental Analysis of the Real-Time Recurrent Learning Algorithm”, Williams & Zipser1989
  • “A Local Learning Algorithm for Dynamic Feedforward and Recurrent Networks”, Schmidhuber1989
  • “A Sticky-Bit Approach for Learning to Represent State”, Bachrach1988
  • “Generalization of Backpropagation With Application to a Recurrent Gas Market Model”, Werbos1988
  • “Generalization of Back-Propagation to Recurrent Neural Networks”, Pineda1987
  • “The Utility Driven Dynamic Error Propagation Network (RTRL)”, Robinson & Fallside1987
  • “A Self-Optimizing, Non-Symmetrical Neural Net for Content Addressable Memory and Pattern Recognition”, Lapedes & Farber1986
  • “Programming a Massively Parallel, Computation Universal System: Static Behavior”, Lapedes & Farber1986b
  • “Serial Order: A Parallel Distributed Processing Approach”, Jordan1986
  • “Hypernetworks [Blog]”, Ha2024
  • “Safety-First AI for Autonomous Data Center Cooling and Industrial Control”
  • “Attention and Augmented Recurrent Neural Networks”
  • “BlinkDL/RWKV-LM: RWKV Is an RNN With Transformer-Level LLM Performance. It Can Be Directly Trained like a GPT (parallelizable). So It’s Combining the Best of RNN and Transformer—Great Performance, Fast Inference, Saves VRAM, Fast Training, “Infinite” Ctx_len, and Free Sentence Embedding.”
  • “Efficient, Reusable RNNs and LSTMs for Torch”
  • “Updated Training?”
  • “Minimaxir/textgenrnn: Easily Train Your Own Text-Generating Neural Network of Any Size and Complexity on Any Text Dataset With a Few Lines of Code.”
  • “Deep Learning for Assisting the Process of Music Composition (part 3)”
  • “Metalearning or Learning to Learn Since 1987”
  • “Stream Seaandsailor”
  • “Composing Music With Recurrent Neural Networks”
  • Wikipedia
• Miscellaneous
• Bibliography