Bibliography:

‘neural net’ tag
‘MLP NN’ tag
‘compressed Transformers’ tag
‘recurrent Transformers’ tag
‘Transformer’ tag
‘MuZero’ tag
Absolute Unit NNs: Regression-Based MLPs for Everything
RNN Metadata for Mimicking Author Style
FlashRNN: Optimizing Traditional RNNs on Modern Hardware
Hymba: A Hybrid-head Architecture for Small Language Models
State-space models can learn in-context by gradient descent
Were RNNs All We Needed?
The Mamba in the Llama: Distilling and Accelerating Hybrid Models
handwriter.ttf: Handwriting Synthesis With Harfbuzz WASM
Learning to (Learn at Test Time): RNNs with Expressive Hidden States
An Empirical Study of Mamba-based Language Models
State Soup: In-Context Skill Learning, Retrieval and Mixing
Grokfast: Accelerated Grokking by Amplifying Slow Gradients
Attention as an RNN
xLSTM: Extended Long Short-Term Memory
Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Length
The Illusion of State in State-Space Models
An accurate and rapidly calibrating speech neuroprosthesis
Does Transformer Interpretability Transfer to RNNs?
Mechanistic Design and Scaling of Hybrid Architectures
GLE: Backpropagation through space, time, and the brain
ZigMa: Zigzag Mamba Diffusion Model
RNNs are not Transformers (Yet): The Key Bottleneck on In-context Retrieval
MambaByte: Token-free Selective State Space Model
MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts
Evolving Reservoirs for Meta Reinforcement Learning
Zoology: Measuring and Improving Recall in Efficient Language Models
Mamba: Linear-Time Sequence Modeling with Selective State Spaces
Diffusion Models Without Attention
Learning few-shot imitation as cultural transmission
Compositional Capabilities of Autoregressive Transformers: A Study on Synthetic, Interpretable Tasks
HGRN: Hierarchically Gated Recurrent Neural Network for Sequence Modeling
On prefrontal working memory and hippocampal episodic memory: Unifying memories stored in weights and activation slots
GateLoop: Fully Data-Controlled Linear Recurrence for Sequence Modeling
ProSG: Using Prompt Synthetic Gradients to Alleviate Prompt Forgetting of RNN-like Language Models
Transformers Learn Higher-Order Optimization Methods for In-Context Learning: A Study with Linear Models
Generalization in Sensorimotor Networks Configured with Natural Language Instructions
Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priors
Parallelizing non-linear sequential models over the sequence length
A high-performance neuroprosthesis for speech decoding and avatar control
Learning to Model the World with Language
Retentive Network: A Successor to Transformer for Large Language Models
Using Sequences of Life-events to Predict Human Lives
Thought Cloning: Learning to Think while Acting by Imitating Human Thinking
RWKV: Reinventing RNNs for the Transformer Era
Emergence of belief-like representations through reinforcement learning
Model scale versus domain knowledge in statistical forecasting of chaotic systems
Resurrecting Recurrent Neural Networks for Long Sequences
SpikeGPT: Generative Pre-trained Language Model with Spiking Neural Networks
Organic reaction mechanism classification using machine learning
A high-performance speech neuroprosthesis
Hungry Hungry Hippos: Towards Language Modeling with State Space Models
Pretraining Without Attention
A 64-core mixed-signal in-memory compute chip based on phase-change memory for deep neural network inference
Melting Pot 2.0
VeLO: Training Versatile Learned Optimizers by Scaling Up
Legged Locomotion in Challenging Terrains using Egocentric Vision
Massively Multilingual ASR on 70 Languages: Tokenization, Architecture, and Generalization Capabilities
Perfectly Secure Steganography Using Minimum Entropy Coupling
Transformers Learn Shortcuts to Automata
Omnigrok: Grokking Beyond Algorithmic Data
Semantic scene descriptions as an objective of human vision
Benchmarking Compositionality with Formal Languages
Learning to Generalize with Object-centric Agents in the Open World Survival Game Crafter
PI-ARS: Accelerating Evolution-Learned Visual-Locomotion with Predictive Information Representations
Spatial representation by ramping activity of neurons in the retrohippocampal cortex
Neural Networks and the Chomsky Hierarchy
BYOL-Explore: Exploration by Bootstrapped Prediction
AnimeSR: Learning Real-World Super-Resolution Models for Animation Videos
Task-Agnostic Continual Reinforcement Learning: In Praise of a Simple Baseline (3RL)
Simple Recurrence Improves Masked Language Models
Sequencer: Deep LSTM for Image Classification
Data Distributional Properties Drive Emergent Few-Shot Learning in Transformers
Semantic projection recovers rich human knowledge of multiple object features from word embeddings
Block-Recurrent Transformers
All You Need Is Supervised Learning: From Imitation Learning to Meta-RL With Upside Down RL
Retrieval-Augmented Reinforcement Learning
Learning by Directional Gradient Descent
General-purpose, long-context autoregressive modeling with Perceiver AR
End-to-end Algorithm Synthesis with Recurrent Networks: Logical Extrapolation Without Overthinking
Data Scaling Laws in NMT: The Effect of Noise and Architecture
Active Predictive Coding Networks: A Neural Solution to the Problem of Learning Reference Frames and Part-Whole Hierarchies
Learning robust perceptive locomotion for quadrupedal robots in the wild
Inducing Causal Structure for Interpretable Neural Networks (IIT)
Evaluating Distributional Distortion in Neural Language Modeling
Gradients are Not All You Need
An Explanation of In-context Learning as Implicit Bayesian Inference
S4: Efficiently Modeling Long Sequences with Structured State Spaces
Minimum Description Length Recurrent Neural Networks
LSSL: Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers
A connectome of the Drosophila central complex reveals network motifs suitable for flexible navigation and context-dependent action selection
Recurrent Model-Free RL is a Strong Baseline for Many POMDPs
Photos Are All You Need for Reciprocal Recommendation in Online Dating
Perceiver IO: A General Architecture for Structured Inputs & Outputs
PES: Unbiased Gradient Estimation in Unrolled Computation Graphs with Persistent Evolution Strategies
Shelley: A Crowd-sourced Collaborative Horror Writer
Ten Lessons From Three Generations Shaped Google’s TPUv4i
RASP: Thinking Like Transformers
Scaling Laws for Acoustic Models
Scaling End-to-End Models for Large-Scale Multilingual ASR
Sensitivity as a Complexity Measure for Sequence Classification Tasks
ALD: Efficient Transformers in Reinforcement Learning using Actor-Learner Distillation
Finetuning Pretrained Transformers into RNNs
Pretrained Transformers as Universal Computation Engines
Perceiver: General Perception with Iterative Attention
When Attention Meets Fast Recurrence: Training SRU++ Language Models with Reduced Compute
Generative Speech Coding with Predictive Variance Regularization
Predictive coding is a consequence of energy efficiency in recurrent neural networks
Deep Residual Learning in Spiking Neural Networks
Distilling Large Language Models into Tiny and Effective Students using pQRNN
Meta Learning Backpropagation And Improving It
On the Binding Problem in Artificial Neural Networks
A Recurrent Vision-and-Language BERT for Navigation
Towards Playing Full MOBA Games with Deep Reinforcement Learning
Multimodal dynamics modeling for off-road autonomous vehicles
Adversarial vulnerabilities of human decision-making
Learning to Summarize Long Texts with Memory Compression and Transfer
Human-centric Dialog Training via Offline Reinforcement Learning
AFT: An Attention Free Transformer
Deep Reinforcement Learning for Closed-Loop Blood Glucose Control
HiPPO: Recurrent Memory with Optimal Polynomial Projections
Adding Recurrence to Pretrained Transformers for Improved Efficiency and Context Size
Matt Botvinick on the spontaneous emergence of learning algorithms
Cultural influences on word meanings revealed through large-scale semantic alignment
DeepSinger: Singing Voice Synthesis with Data Mined From the Web
High-performance brain-to-text communication via imagined handwriting
Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
The Recurrent Neural Tangent Kernel
Untangling tradeoffs between recurrence and self-attention in neural networks
Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing
Learning Music Helps You Read: Using Transfer to Study Linguistic Structure in Language Models
Syntactic Structure from Deep Learning
Agent57: Outperforming the human Atari benchmark
Machine Translation of Cortical Activity to Text With an Encoder-Decoder Framework
Learning-based Memory Allocation for C++ Server Workloads
Accelerating Feedforward Computation via Parallel Nonlinear Equation Solving
Direct Fit to Nature: An Evolutionary Perspective on Biological and Artificial Neural Networks
Scaling Laws for Neural Language Models
Estimating the deep replicability of scientific findings using human and artificial intelligence
Placing language in an integrated understanding system: Next steps toward human-level performance in neural language models
Measuring Compositional Generalization: A Comprehensive Method on Realistic Data
SimpleBooks: Long-term dependency book dataset with simplified English vocabulary for word-level language modeling
Single Headed Attention RNN: Stop Thinking With Your Head
Excavate
MuZero: Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model
CommonGen: A Constrained Text Generation Challenge for Generative Commonsense Reasoning
High Fidelity Video Prediction with Large Stochastic Recurrent Neural Networks
Legendre Memory Units: Continuous-Time Representation in Recurrent Neural Networks
SEED RL: Scalable and Efficient Deep-RL with Accelerated Central Inference
Mixed-Signal Neuromorphic Processors: Quo vadis?
Restoring ancient text using deep learning (Pythia): a case study on Greek epigraphy
Mogrifier LSTM
R2D3: Making Efficient Use of Demonstrations to Solve Hard Exploration Problems
Language Modeling State-of-the-art leaderboards
Metalearned Neural Memory
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Generating Text with Recurrent Neural Networks
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLP
MoGlow: Probabilistic and controllable motion synthesis using normalizing flows
Reinforcement Learning, Fast and Slow
Meta-learners’ learning dynamics are unlike learners’
Speech synthesis from neural decoding of spoken sentences
Good News, Everyone! Context driven entity-aware captioning for news images
Surrogate Gradient Learning in Spiking Neural Networks
On the Turing Completeness of Modern Neural Network Architectures
Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
Natural Questions: A Benchmark for Question Answering Research
High Fidelity Video Prediction with Large Stochastic Recurrent Neural Networks: Videos
Bayesian Layers: A Module for Neural Network Uncertainty
Meta-Learning: Learning to Learn Fast
Piano Genie
Learning Recurrent Binary/Ternary Weights
R2D2: Recurrent Experience Replay in Distributed Reinforcement Learning
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
Adversarial Reprogramming of Text Classification Neural Networks
Object Hallucination in Image Captioning
This Time with Feeling: Learning Expressive Musical Performance
Character-Level Language Modeling with Deeper Self-Attention
General Value Function Networks
Deep-speare: A Joint Neural Model of Poetic Language, Meter and Rhyme
Universal Transformers
Accurate Uncertainties for Deep Learning Using Calibrated Regression
The Natural Language Decathlon: Multitask Learning as Question Answering
Neural Ordinary Differential Equations
Know What You Don’t Know: Unanswerable Questions for SQuAD
DVRL: Deep Variational Reinforcement Learning for POMDPs
Greedy Attack and Gumbel Attack: Generating Adversarial Examples for Discrete Data
Hierarchical Neural Story Generation
Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context
Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategies
A Tree Search Algorithm for Sequence Labeling
An Analysis of Neural Language Modeling at Multiple Scales
Reviving and Improving Recurrent Back-Propagation
Learning Memory Access Patterns
Learning Longer-term Dependencies in RNNs with Auxiliary Losses
One Big Net For Everything
Efficient Neural Audio Synthesis
Deep contextualized word representations
M-Walk: Learning to Walk over Graphs using Monte Carlo Tree Search
Overcoming the vanishing gradient problem in plain recurrent networks
ULMFiT: Universal Language Model Fine-tuning for Text Classification
Large-scale comparison of machine learning methods for drug target prediction on ChEMBL
A Flexible Approach to Automated RNN Architecture Generation
The NarrativeQA Reading Comprehension Challenge
Learning Compact Recurrent Neural Networks with Block-Term Tensor Decomposition
Mastering the Dungeon: Grounded Language Learning by Mechanical Turker Descent
Evaluating prose style transfer with the Bible
Breaking the Softmax Bottleneck: A High-Rank RNN Language Model
Neural Speed Reading via Skim-RNN
Unsupervised Machine Translation Using Monolingual Corpora Only
Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks
Mixed Precision Training
To prune, or not to prune: exploring the efficacy of pruning for model compression
Dynamic Evaluation of Neural Sequence Models
Online Learning of a Memory for Learning Rates
Why Pay More When You Can Pay Less: A Joint Learning Framework for Active Feature Acquisition and Classification
N2N Learning: Network to Network Compression via Policy Gradient Reinforcement Learning
SRU: Simple Recurrent Units for Highly Parallelizable Recurrence
Learning to Look Around: Intelligently Exploring Unseen Environments for Unknown Tasks
Twin Networks: Matching the Future for Sequence Generation
Skip RNN: Learning to Skip State Updates in Recurrent Neural Networks
Revisiting Activation Regularization for Language RNNs
Bayesian Sparsification of Recurrent Neural Networks
On the State-of-the-Art of Evaluation in Neural Language Models
Controlling Linguistic Style Aspects in Neural Language Generation
Device Placement Optimization with Reinforcement Learning
Six Challenges for Neural Machine Translation
Towards Synthesizing Complex Programs from Input-Output Examples
Language Generation with Recurrent Generative Adversarial Networks without Pre-training
Biased Importance Sampling for Deep Neural Network Training
Deriving Neural Architectures from Sequence and Graph Kernels
A Deep Reinforced Model for Abstractive Summarization
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
DeepTingle
A neural network system for transformation of regional cuisine style
Sharp Models on Dull Hardware: Fast and Accurate Neural Machine Translation Decoding on the CPU
Adversarial Neural Machine Translation
SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine
Learning to Reason: End-to-End Module Networks for Visual Question Answering
Exploring Sparsity in Recurrent Neural Networks
Get To The Point: Summarization with Pointer-Generator Networks
DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks
Bayesian Recurrent Neural Networks
Recurrent Environment Simulators
Learning to Generate Reviews and Discovering Sentiment
Learning Simpler Language Models with the Differential State Framework
I2T2I: Learning Text to Image Synthesis with Textual Data Augmentation
Improving Neural Machine Translation with Conditional Sequence Generative Adversarial Nets
Learned Optimizers that Scale and Generalize
Parallel Multiscale Autoregressive Density Estimation
Tracking the World State with Recurrent Entity Networks
Optimization as a Model for Few-Shot Learning
Neural Combinatorial Optimization with Reinforcement Learning
Frustratingly Short Attention Spans in Neural Language Modeling
Tuning Recurrent Neural Networks with Reinforcement Learning
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
Neural Data Filter for Bootstrapping Stochastic Gradient Descent
Learning the Enigma With Recurrent Neural Networks
Your TL;DR by an AI: A Deep Reinforced Model for Abstractive Summarization
SampleRNN: An Unconditional End-to-End Neural Audio Generation Model
Improving Neural Language Models with a Continuous Cache
NewsQA: A Machine Comprehension Dataset
Neural Combinatorial Optimization with Reinforcement Learning
Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation
Learning to Learn without Gradient Descent by Gradient Descent
RL²: Fast Reinforcement Learning via Slow Reinforcement Learning
DeepCoder: Learning to Write Programs
QRNNs: Quasi-Recurrent Neural Networks
Neural Architecture Search with Reinforcement Learning
Bidirectional Attention Flow for Machine Comprehension
Hybrid computing using a neural network with dynamic external memory
Scaling Memory-Augmented Neural Networks with Sparse Reads and Writes
Using Fast Weights to Attend to the Recent Past
Achieving Human Parity in Conversational Speech Recognition
VPN: Video Pixel Networks
HyperNetworks
Pointer Sentinel Mixture Models
Multiplicative LSTM for sequence modeling
Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Image-to-Markup Generation with Coarse-to-Fine Attention
Hierarchical Multiscale Recurrent Neural Networks
Deep Learning Human Mind for Automated Visual Classification
Using the Output Embedding to Improve Language Models
Full Resolution Image Compression with Recurrent Neural Networks
Decoupled Neural Interfaces using Synthetic Gradients
Clockwork Convnets for Video Semantic Segmentation
Layer Normalization
LSTMVis: A Tool for Visual Analysis of Hidden State Dynamics in Recurrent Neural Networks
Learning to learn by gradient descent by gradient descent
Iterative Alternating Neural Attention for Machine Reading
Deep Reinforcement Learning for Dialogue Generation
Programming with a Differentiable Forth Interpreter
Training Deep Nets with Sublinear Memory Cost
Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex
Improving sentence compression by learning to predict gaze
Adaptive Computation Time for Recurrent Neural Networks
Dynamic Memory Networks for Visual and Textual Question Answering
PlaNet—Photo Geolocation with Convolutional Neural Networks
Learning Distributed Representations of Sentences from Unlabeled Data
Exploring the Limits of Language Modeling
PixelRNN: Pixel Recurrent Neural Networks
Persistent RNNs: Stashing Recurrent Weights On-Chip
Exploring the Limits of Language Modeling § 5.9: Samples from the Model
ec477c75170386e6fd6aff677d064cf95eb87a20.pdf#page=8&org=google
Deep-Spying: Spying using Smartwatch and Deep Learning
On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models
Neural GPUs Learn Algorithms
Sequence Level Training with Recurrent Neural Networks
Neural Programmer-Interpreters
Generating Sentences from a Continuous Space
Generative Concatenative Nets Jointly Learn to Write and Classify Reviews
Generating Images from Captions with Attention
Semi-supervised Sequence Learning
BPEs: Neural Machine Translation of Rare Words with Subword Units
Training recurrent networks online without backtracking
Deep Recurrent Q-Learning for Partially Observable MDPs
Teaching Machines to Read and Comprehend
Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
Visualizing and Understanding Recurrent Networks
The Unreasonable Effectiveness of Recurrent Neural Networks
Deep Neural Networks for Large Vocabulary Handwritten Text Recognition
Reinforcement Learning Neural Turing Machines—Revised
End-To-End Memory Networks
LSTM: A Search Space Odyssey
Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets
DRAW: A Recurrent Neural Network For Image Generation
Ensemble of Generative and Discriminative Techniques for Sentiment Analysis of Movie Reviews
Neural Turing Machines
Learning to Execute
Neural Machine Translation by Jointly Learning to Align and Translate
Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
GRU: Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
doc2vec: Distributed Representations of Sentences and Documents
A Clockwork RNN
One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling
Generating Sequences With Recurrent Neural Networks
On the difficulty of training Recurrent Neural Networks
Recurrent Neural Network Based Language Model
Large Language Models in Machine Translation
Learning to Learn Using Gradient Descent
Long Short-Term Memory
Flat Minima
Gradient-Based Learning Algorithms for Recurrent Networks and Their Computational Complexity
A Focused Backpropagation Algorithm for Temporal Pattern Recognition
Learning Complex, Extended Sequences Using the Principle of History Compression
Learning to Control Fast-Weight Memories: An Alternative to Dynamic Recurrent Networks
Untersuchungen zu dynamischen neuronalen Netzen [Studies of dynamic neural networks]
Finding Structure In Time
Connectionist Music Composition Based on Melodic, Stylistic, and Psychophysical Constraints [Technical report CU-CS–495–90]
A Learning Algorithm for Continually Running Fully Recurrent Neural Networks
Recurrent Backpropagation and Hopfield Networks
Backpropagation in Perceptrons with Feedback
Experimental Analysis of the Real-time Recurrent Learning Algorithm
A Local Learning Algorithm for Dynamic Feedforward and Recurrent Networks
A Sticky-Bit Approach for Learning to Represent State
Generalization of backpropagation with application to a recurrent gas market model
Generalization of back-propagation to recurrent neural networks
The Utility Driven Dynamic Error Propagation Network (RTRL)
A self-optimizing, non-symmetrical neural net for content addressable memory and pattern recognition
Programming a massively parallel, computation universal system: Static behavior
Serial Order: A Parallel Distributed Processing Approach
Hypernetworks [Blog]
Safety-First AI for Autonomous Data Center Cooling and Industrial Control
Attention and Augmented Recurrent Neural Networks
BlinkDL/RWKV-LM: RWKV Is an RNN With Transformer-Level LLM Performance. It Can Be Directly Trained like a GPT (parallelizable). So It’s Combining the Best of RNN and Transformer—Great Performance, Fast Inference, Saves VRAM, Fast Training, "Infinite" Ctx_len, and Free Sentence Embedding.
Efficient, Reusable RNNs and LSTMs for Torch
Updated Training?
Minimaxir/textgenrnn: Easily Train Your Own Text-Generating Neural Network of Any Size and Complexity on Any Text Dataset With a Few Lines of Code.
Deep Learning for Assisting the Process of Music Composition (part 3)
Metalearning or Learning to Learn Since 1987
76c24cf0db4abf4ce2b77d22182272d8e62d1a28.html
Stream Seaandsailor
Composing Music With Recurrent Neural Networks
0108f05c124cfb8b547e784dba32a6a5be813f44.html
2021-droppo-figure5-lstmvstransformerscaling.png
2021-jaegle-figure2-perceiverioarchitecture.png
2021-li-figure1-werasrscaling.png
2020-deepmind-agent57-figure3-deepreinforcementlearningtimeline.svg
2020-deepmind-agent57-performancetable.svg
2017-khalifa-example3-incoherentdeeptinglesamplepromptedwithmobydickcallmeishmael.png
2017-krause-figure2-dynamicevaluationrnnpredictionofwikipediaandspanishtextshowingtesttimeadaptation.png
2016-06-09-rossgoodwin-adventuresinnarratedreality-2.html
2016-03-19-rossgoodwin-adventuresinnarratedreality-1.html
2015-06-03-karpathy-charrnn-visualization.tar.xz
http://www.byronknoll.com/cmix.html
2fcab2e668d671915958f32dc7e5d4607a1bd18b.html
https://aclanthology.org/D13-1176.pdf
8ecd1e30c66400dbf395e3f5b7852c8bb70a02d2.pdf
https://ahrm.github.io/jekyll/update/2022/04/14/using-languge-models-to-read-faster.html
https://bellard.org/ts_server/ts_zip.html
https://blog.ought.com/dalca-4d47a90edd92
8a06a3f7afb28912aed273184beb600d643b5f09.html
https://blog.rwkv.com/p/eagle-7b-soaring-past-transformers
383941f46b66a161efb2b39936e04635da090a5f.html
https://cprimozic.net/blog/growing-sparse-computational-graphs-with-rnns/
c9d32a6c96999e0be1f657d4316c77ec606c7677.html
https://homepages.inf.ed.ac.uk/abmayne/publications/sennrich2016NAACL.pdf
https://jackcook.com/2024/02/23/mamba.html
77db36a2dade6a747ac6d14f6d42e0b2b2285a5a.html
https://magenta.tensorflow.org/blog/2017/06/01/waybackprop
https://manifestai.com/blogposts/faster-after-all/
eaed3ff82cffc0550ba85e7c9c89febbdd547d4e.html
https://mi.eng.cam.ac.uk/projects/cued-rnnlm/papers/Interspeech15.pdf
https://nlp.stanford.edu/~johnhew/rnns-hierarchy.html
9e6e8d0ed92bbee8142ad97f0de3567c20de436c.html
https://patentimages.storage.googleapis.com/57/53/22/91b8a6792dbb1e/US20180204116A1.pdf#deepmind
https://sander.ai/2023/07/20/perspectives.html
https://wandb.ai/wandb_fc/articles/reports/Image-to-LaTeX--Vmlldzo1NDQ0MTAx
https://www.ai21.com/blog/announcing-jamba
4eb64a22dba344b9bdff4f6c9767f8e64482e521.html
https://www.canva.dev/blog/engineering/ship-shape/
https://www.cs.utoronto.ca/~ilya/pubs/ilya_sutskever_phd_thesis.pdf#page=67
15e5dc3ad2c783e88d87d6299e7ade5b9d35ba16.pdf#page=67
https://www.lesswrong.com/posts/bD4B2MF7nsGAfH9fj/basic-mathematics-of-predictive-coding
https://www.lesswrong.com/posts/mxa7XZ8ajE2oarWcr/lawrencec-s-shortform#pEqfzPMpqsnhaGrNK
https://www.nature.com/articles/s41598-023-35597-4
https://www.reddit.com/r/MachineLearning/comments/11nre6t/p_rwkv_14b_is_a_strong_chatbot_despite_only/
https://www.reddit.com/r/MachineLearning/comments/umq908/r_rwkvv2rnn_a_parallelizable_rnn_with/
2a40a13e632302df9433aa93306ce591f3ac2412.html
https://www.reddit.com/r/MachineLearning/comments/yxt8sa/r_rwkv4_7b_release_an_attentionfree_rnn_language/
ad5ae41ac26ebe772323d4cfbfd6bf360e7efa3b.html
https://x.com/BlinkDL_AI/status/1677593798531223552
https://x.com/BlinkDL_AI/status/1784496793075744966
https://x.com/RichardSocher/status/1736161332259614989
https://x.com/arankomatsuzaki/status/1639000379978403853
https://x.com/mayfer/status/1732269798934106133
FlashRNN: Optimizing Traditional RNNs on Modern Hardware
https%253A%252F%252Farxiv.org%252Fabs%252F2412.07752.html
Were RNNs All We Needed?
https%253A%252F%252Farxiv.org%252Fabs%252F2410.01201.html
The Mamba in the Llama: Distilling and Accelerating Hybrid Models
Junxiong Wang
https://rush-nlp.com/
Tri Dao
https%253A%252F%252Farxiv.org%252Fabs%252F2408.15237.html
An Empirical Study of Mamba-based Language Models
Tri Dao
Albert Gu
https%253A%252F%252Farxiv.org%252Fabs%252F2406.07887.html
Grokfast: Accelerated Grokking by Amplifying Slow Gradients
https%253A%252F%252Farxiv.org%252Fabs%252F2405.20233.html
Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Length
Luke Zettlemoyer
Omer Levy
https%253A%252F%252Farxiv.org%252Fabs%252F2404.08801%2523facebook.html
Does Transformer Interpretability Transfer to RNNs?
https%253A%252F%252Farxiv.org%252Fabs%252F2404.05971%2523eleutherai.html
Mechanistic Design and Scaling of Hybrid Architectures
Stefano Ermon
https%253A%252F%252Farxiv.org%252Fabs%252F2403.17844.html
ZigMa: Zigzag Mamba Diffusion Model
https%253A%252F%252Farxiv.org%252Fabs%252F2403.13802.html
Zoology: Measuring and Improving Recall in Efficient Language Models
https%253A%252F%252Farxiv.org%252Fabs%252F2312.04927.html
Mamba: Linear-Time Sequence Modeling with Selective State Spaces
Albert Gu
Tri Dao
https%253A%252F%252Farxiv.org%252Fabs%252F2312.00752.html
Learning few-shot imitation as cultural transmission
https%253A%252F%252Fwww.nature.com%252Farticles%252Fs41467-023-42875-2%2523deepmind.html
Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priors
https%253A%252F%252Farxiv.org%252Fabs%252F2310.02980.html
Thought Cloning: Learning to Think while Acting by Imitating Human Thinking
Jeff Clune—Professor—Computer Science—University of British Columbia
https%253A%252F%252Farxiv.org%252Fabs%252F2306.00323.html
RWKV: Reinventing RNNs for the Transformer Era
https%253A%252F%252Farxiv.org%252Fabs%252F2305.13048.html
Resurrecting Recurrent Neural Networks for Long Sequences
Albert Gu
https://sites.google.com/view/razp/home
https%253A%252F%252Farxiv.org%252Fabs%252F2303.06349%2523deepmind.html
SpikeGPT: Generative Pre-trained Language Model with Spiking Neural Networks
https%253A%252F%252Farxiv.org%252Fabs%252F2302.13939.html
Organic reaction mechanism classification using machine learning
%252Fdoc%252Fscience%252F2023-bures.pdf.html
Hungry Hungry Hippos: Towards Language Modeling with State Space Models
Tri Dao
https%253A%252F%252Farxiv.org%252Fabs%252F2212.14052.html
Pretraining Without Attention
Junxiong Wang
Albert Gu
https://rush-nlp.com/
https%253A%252F%252Farxiv.org%252Fabs%252F2212.10544.html
Legged Locomotion in Challenging Terrains using Egocentric Vision
https%253A%252F%252Farxiv.org%252Fabs%252F2211.07638.html
Omnigrok: Grokking Beyond Algorithmic Data
https%253A%252F%252Farxiv.org%252Fabs%252F2210.01117.html
Semantic scene descriptions as an objective of human vision
https%253A%252F%252Farxiv.org%252Fabs%252F2209.11737.html
Sequencer: Deep LSTM for Image Classification
https%253A%252F%252Farxiv.org%252Fabs%252F2205.01972.html
Semantic projection recovers rich human knowledge of multiple object features from word embeddings
%252Fdoc%252Fai%252Fnn%252Frnn%252F2022-grand.pdf.html
Block-Recurrent Transformers
Yuhuai (Tony) Wu’s Home Page
Behnam Neyshabur
https%253A%252F%252Farxiv.org%252Fabs%252F2203.07852.html
General-purpose, long-context autoregressive modeling with Perceiver AR
https%253A%252F%252Farxiv.org%252Fabs%252F2202.07765%2523deepmind.html
Learning robust perceptive locomotion for quadrupedal robots in the wild
%252Fdoc%252Freinforcement-learning%252Fmeta-learning%252F2022-miki.pdf.html
S4: Efficiently Modeling Long Sequences with Structured State Spaces
Albert Gu
https%253A%252F%252Farxiv.org%252Fabs%252F2111.00396.html
A connectome of the Drosophila central complex reveals network motifs suitable for flexible navigation and context-dependent action selection
https%253A%252F%252Felifesciences.org%252Farticles%252F66039.html
Perceiver IO: A General Architecture for Structured Inputs & Outputs
https%253A%252F%252Farxiv.org%252Fabs%252F2107.14795%2523deepmind.html
PES: Unbiased Gradient Estimation in Unrolled Computation Graphs with Persistent Evolution Strategies
Luke Metz
Jascha Sohl-Dickstein
https%253A%252F%252Fproceedings.mlr.press%252Fv139%252Fvicol21a.html.html
Shelley: A Crowd-sourced Collaborative Horror Writer
%252Fdoc%252Fai%252Fnn%252Frnn%252F2021-delul.pdf.html
Ten Lessons From Three Generations Shaped Google’s TPUv4i
%252Fdoc%252Fai%252Fscaling%252Fhardware%252F2021-jouppi.pdf.html
RASP: Thinking Like Transformers
https%253A%252F%252Farxiv.org%252Fabs%252F2106.06981.html
Scaling Laws for Acoustic Models
https%253A%252F%252Farxiv.org%252Fabs%252F2106.09488%2523amazon.html
Perceiver: General Perception with Iterative Attention
https%253A%252F%252Farxiv.org%252Fabs%252F2103.03206%2523deepmind.html
Deep Residual Learning in Spiking Neural Networks
https%253A%252F%252Farxiv.org%252Fabs%252F2102.04159.html
Towards Playing Full MOBA Games with Deep Reinforcement Learning
https%253A%252F%252Farxiv.org%252Fabs%252F2011.12692%2523tencent.html
HiPPO: Recurrent Memory with Optimal Polynomial Projections
Albert Gu
Tri Dao
Stefano Ermon
https%253A%252F%252Farxiv.org%252Fabs%252F2008.07669.html
Matt Botvinick on the spontaneous emergence of learning algorithms
https%253A%252F%252Fwww.lesswrong.com%252Fposts%252FWnqua6eQkewL3bqsF%252Fmatt-botvinick-on-the-spontaneous-emergence-of-learning.html
Agent57: Outperforming the human Atari benchmark
https%253A%252F%252Fdeepmind.google%252Fdiscover%252Fblog%252Fagent57-outperforming-the-human-atari-benchmark%252F.html
Accelerating Feedforward Computation via Parallel Nonlinear Equation Solving
Stefano Ermon
https%253A%252F%252Farxiv.org%252Fabs%252F2002.03629.html
Scaling Laws for Neural Language Models
Jared Kaplan
Sam McCandlish
Alec Radford
https%253A%252F%252Farxiv.org%252Fabs%252F2001.08361%2523openai.html
Single Headed Attention RNN: Stop Thinking With Your Head
State of the Smerity
https%253A%252F%252Farxiv.org%252Fabs%252F1911.11423.html
Legendre Memory Units: Continuous-Time Representation in Recurrent Neural Networks
https%253A%252F%252Fopenreview.net%252Fforum%253Fid%253DHyxlRHBlUB.html
SEED RL: Scalable and Efficient Deep-RL with Accelerated Central Inference
https%253A%252F%252Farxiv.org%252Fabs%252F1910.06591%2523deepmind.html
Mogrifier LSTM
https%253A%252F%252Farxiv.org%252Fabs%252F1909.01792%2523deepmind.html
Language Modeling State-of-the-art leaderboards
https%253A%252F%252Fpaperswithcode.com%252Ftask%252Flanguage-modelling.html
Meta-learners’ learning dynamics are unlike learners’
https%253A%252F%252Farxiv.org%252Fabs%252F1905.01320%2523deepmind.html
Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
Zihang Dai
Zhilin Yang
https://www.cs.cmu.edu/~./yiming/
https%253A%252F%252Farxiv.org%252Fabs%252F1901.02860.html
R2D2: Recurrent Experience Replay in Distributed Reinforcement Learning
https%253A%252F%252Fopenreview.net%252Fforum%253Fid%253Dr1lyTjAqYX%2523deepmind.html
ULMFiT: Universal Language Model Fine-tuning for Text Classification
https%253A%252F%252Farxiv.org%252Fabs%252F1801.06146.html
Dynamic Evaluation of Neural Sequence Models
https%253A%252F%252Farxiv.org%252Fabs%252F1709.07432.html
SRU: Simple Recurrent Units for Highly Parallelizable Recurrence
https%253A%252F%252Farxiv.org%252Fabs%252F1709.02755.html
SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine
Kyunghyun Cho
https%253A%252F%252Farxiv.org%252Fabs%252F1704.05179.html
Learning to Reason: End-to-End Module Networks for Visual Question Answering
Jacob Andreas @ MIT
https%253A%252F%252Farxiv.org%252Fabs%252F1704.05526.html
Clockwork Convnets for Video Semantic Segmentation
https%253A%252F%252Farxiv.org%252Fabs%252F1608.03609.html
End-To-End Memory Networks
https%253A%252F%252Farxiv.org%252Fabs%252F1503.08895.html
Recurrent Neural Network Based Language Model
%252Fdoc%252Fai%252Fnn%252Frnn%252F2010-mikolov.pdf.html
Untersuchungen zu dynamischen neuronalen Netzen [Studies of dynamic neural networks]
%252Fdoc%252Fai%252Fnn%252Frnn%252F1991-hochreiter.pdf.html
Experimental Analysis of the Real-time Recurrent Learning Algorithm
%252Fdoc%252Fai%252Fnn%252Frnn%252F1989-williams.pdf.html
Generalization of backpropagation with application to a recurrent gas market model
%252Fdoc%252Fai%252Fnn%252Frnn%252F1988-werbos.pdf.html
Generalization of back-propagation to recurrent neural networks
%252Fdoc%252Fai%252Fnn%252Frnn%252F1987-pineda.pdf.html
Wikipedia Bibliography: