“‘MLP NN’ Tag”,2019-08-31 (RNN, Transformer)⁠:

Bibliography for tag ai/nn/fully-connected, most recent first: 2 related tags, 206 annotations, & 45 links (parent).

[page summary]

• See Also
• Gwern
  • “Research Ideas”, Gwern2017
  • “Absolute Unit NNs: Regression-Based MLPs for Everything”, Gwern2023
  • “Modular Brain AUNNs for Uploads”, Gwern2023
  • “Language-Conditioned Absolute Unit NNs”, Gwern2022
  • “Fully-Connected Neural Nets”, Gwern2021
  • “Efficient Attention: Breaking The Quadratic Transformer Bottleneck”, Gwern2020
• Links
  • “AUNN: Simple Implementation of Gwern’s AUNN Proposal”, Roland2024
  • “The Slingshot Helps With Learning”, Wu2024
  • “SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning”, Lee et al 2024
  • “NGPT: Normalized Transformer With Representation Learning on the Hypersphere”, Loshchilov et al 2024
  • “How Feature Learning Can Improve Neural Scaling Laws”, Bordelon et al 2024
  • “Magika: AI-Powered Content-Type Detection”, Fratantonio et al 2024
  • “On the Complexity of Neural Computation in Superposition”, Adler & Shavit2024
  • “Masked Mixers for Language Generation and Retrieval”, Badger2024
  • “GSoC 2024: Differentiable Logic for Interactive Systems and Generative Music”
  • “What Matters in Transformers? Not All Attention Is Needed”, He et al 2024
  • “When Parts Are Greater Than Sums: Individual LLM Components Can Outperform Full Models”, Chang et al 2024
  • “Probing the Decision Boundaries of In-Context Learning in Large Language Models”, Zhao et al 2024
  • “MAR: Autoregressive Image Generation without Vector Quantization”, Li et al 2024
  • “Grokking Modular Polynomials”, Doshi et al 2024
  • “Grokfast: Accelerated Grokking by Amplifying Slow Gradients”, Lee et al 2024
  • “Lateralization MLP: A Simple Brain-Inspired Architecture for Diffusion”, Hu & Rostami2024
  • “MLPs Learn In-Context”, Tong & Pehlevan2024
  • “Verified Neural Compressed Sensing”, Bunel et al 2024
  • “Neural Redshift: Random Networks Are Not Random Functions”, Teney et al 2024
  • “Neural Spline Fields for Burst Image Fusion and Layer Separation”, Chugunov et al 2023
  • “SwitchHead: Accelerating Transformers With Mixture-Of-Experts Attention”, Csordás et al 2023
  • “SMERF: Streamable Memory Efficient Radiance Fields for Real-Time Large-Scene Exploration”, Duckworth et al 2023
  • “Grokking Group Multiplication With Cosets”, Stander et al 2023
  • “Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks As an Alternative to Attention Layers in Transformers”, Bozic et al 2023
  • “HyperFields: Towards Zero-Shot Generation of NeRFs from Text”, Babu et al 2023
  • “Grokking Beyond Neural Networks: An Empirical Exploration With Model Complexity”, Miller et al 2023
  • “To Grok or Not to Grok: Disentangling Generalization and Memorization on Corrupted Algorithmic Datasets”, Doshi et al 2023
  • “Polynomial Time Cryptanalytic Extraction of Neural Network Models”, Shamir et al 2023
  • “One Wide Feedforward Is All You Need”, Pires et al 2023
  • “Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchilla”, Lieberum et al 2023
  • “Self Expanding Neural Networks”, Mitchell et al 2023
  • “The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networks”, Zhong et al 2023
  • “Scaling MLPs: A Tale of Inductive Bias”, Bachmann et al 2023
  • “Any Deep ReLU Network Is Shallow”, Villani & Schoots2023
  • “Does the First Letter of One’s Name Affect Life Decisions? A Natural Language Processing Examination of Nominative Determinism”, Chatterjee et al 2023
  • “Learning and Memorization”, Chatterjee2023
  • “How Does GPT-2 Compute Greater-Than?: Interpreting Mathematical Abilities in a Pre-Trained Language Model”, Hanna et al 2023
  • “Two-Step Training: Adjustable Sketch Colorization via Reference Image and Text Tag”, Yan et al 2023
  • “HyperDiffusion: Generating Implicit Neural Fields With Weight-Space Diffusion”, Erkoç et al 2023
  • “The Quantization Model of Neural Scaling”, Michaud et al 2023
  • “TSMixer: An All-MLP Architecture for Time Series Forecasting”, Chen et al 2023
  • “Loss Landscapes Are All You Need: Neural Network Generalization Can Be Explained Without the Implicit Bias of Gradient Descent”, Chiang et al 2023
  • “A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operations”, Chughtai et al 2023
  • “Looped Transformers As Programmable Computers”, Giannou et al 2023
  • “Organic Reaction Mechanism Classification Using Machine Learning”, Burés & Larrosa2023
  • “DataMUX: Data Multiplexing for Neural Networks”, Murahari et al 2023
  • “Merging Enzymatic and Synthetic Chemistry With Computational Synthesis Planning”, Levin et al 2022
  • “Magic3D: High-Resolution Text-To-3D Content Creation”, Lin et al 2022
  • “How Much Does Attention Actually Attend? Questioning the Importance of Attention in Pretrained Transformers”, Hassid et al 2022
  • “Deep Differentiable Logic Gate Networks”, Petersen et al 2022
  • “The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers”, Li et al 2022
  • “The Unreasonable Effectiveness of Fully-Connected Layers for Low-Data Regimes”, Kocsis et al 2022
  • “Scaling Forward Gradient With Local Losses”, Ren et al 2022
  • “Omnigrok: Grokking Beyond Algorithmic Data”, Liu et al 2022
  • “DreamFusion: Text-To-3D Using 2D Diffusion”, Poole et al 2022
  • “g.pt: Learning to Learn With Generative Models of Neural Network Checkpoints”, Peebles et al 2022
  • “Random Initializations Performing above Chance and How to Find Them”, Benzing et al 2022
  • “Scaling Laws vs Model Architectures: How Does Inductive Bias Influence Scaling?”, Tay et al 2022
  • “Why Do Tree-Based Models Still Outperform Deep Learning on Tabular Data?”, Grinsztajn et al 2022
  • “Revisiting Pretraining Objectives for Tabular Deep Learning”, Rubachev et al 2022
  • “RHO-LOSS: Prioritized Training on Points That Are Learnable, Worth Learning, and Not Yet Learnt”, Mindermann et al 2022
  • “MLP-3D: A MLP-Like 3D Architecture With Grouped Time Mixing”, Qiu et al 2022
  • “ChordMixer: A Scalable Neural Attention Model for Sequences With Different Lengths”, Khalitov et al 2022
  • “Sparse Mixers: Combining MoE and Mixing to Build a More Efficient BERT”, Lee-Thorp & Ainslie2022
  • “Towards Understanding Grokking: An Effective Theory of Representation Learning”, Liu et al 2022
  • “Paramixer: Parameterizing Mixing Links in Sparse Factors Works Better Than Dot-Product Self-Attention”, Yu et al 2022
  • “Deep Learning Meets Nonparametric Regression: Are Weight-Decayed DNNs Locally Adaptive?”, Zhang & Wang2022
  • “Efficient Language Modeling With Sparse All-MLP”, Yu et al 2022
  • “HyperMixer: An MLP-Based Low Cost Alternative to Transformers”, Mai et al 2022
  • “MLP-ASR: Sequence-Length Agnostic All-MLP Architectures for Speech Recognition”, Sakuma et al 2022
  • “Mixing and Shifting: Exploiting Global and Local Dependencies in Vision MLPs”, Zheng et al 2022
  • “PNLP-Mixer: an Efficient All-MLP Architecture for Language”, Fusco et al 2022
  • “Data-Driven Emergence of Convolutional Structure in Neural Networks”, Ingrosso & Goldt2022
  • “When Shift Operation Meets Vision Transformer: An Extremely Simple Alternative to Attention Mechanism (ShiftViT)”, Wang et al 2022
  • “ConvMixer: Patches Are All You Need?”, Trockman & Kolter2022
  • “MAXIM: Multi-Axis MLP for Image Processing”, Tu et al 2022
  • “Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets [Paper]”, Power et al 2022
  • “The GatedTabTransformer: An Enhanced Deep Learning Architecture for Tabular Modeling”, Cholakov & Kolev2022
  • “MLP Architectures for Vision-And-Language Modeling: An Empirical Study”, Nie et al 2021
  • “Noether Networks: Meta-Learning Useful Conserved Quantities”, Alet et al 2021
  • “Zero-Shot Text-Guided Object Generation With Dream Fields”, Jain et al 2021
  • “MorphMLP: A Self-Attention Free, MLP-Like Backbone for Image and Video”, Zhang et al 2021
  • “PointMixer: MLP-Mixer for Point Cloud Understanding”, Choe et al 2021
  • “MetaFormer Is Actually What You Need for Vision”, Yu et al 2021
  • “Deep Learning without Shortcuts: Shaping the Kernel With Tailored Rectifiers”, Zhang et al 2021
  • “ZerO Initialization: Initializing Residual Networks With Only Zeros and Ones”, Zhao et al 2021
  • “Wide Neural Networks Forget Less Catastrophically”, Mirzadeh et al 2021
  • “ADOP: Approximate Differentiable One-Pixel Point Rendering”, Rückert et al 2021
  • “Rapid Training of Deep Neural Networks without Skip Connections or Normalization Layers Using Deep Kernel Shaping”, Martens et al 2021
  • “Exploring the Limits of Large Scale Pre-Training”, Abnar et al 2021
  • “Sparse MLP for Image Recognition: Is Self-Attention Really Necessary?”, Tang et al 2021
  • “ConvMLP: Hierarchical Convolutional MLPs for Vision”, Li et al 2021
  • “Sparse-MLP: A Fully-MLP Architecture With Conditional Computation”, Lou et al 2021
  • “A Battle of Network Structures: An Empirical Study of CNN, Transformer, and MLP”, Zhao et al 2021
  • “Hire-MLP: Vision MLP via Hierarchical Rearrangement”, Guo et al 2021
  • “RaftMLP: How Much Can Be Done Without Attention and With Less Spatial Locality?”, Tatsunami & Taki2021
  • “S²-MLPv2: Improved Spatial-Shift MLP Architecture for Vision”, Yu et al 2021
  • “CycleMLP: A MLP-Like Architecture for Dense Prediction”, Chen et al 2021
  • “AS-MLP: An Axial Shifted MLP Architecture for Vision”, Lian et al 2021
  • “Vision Permutator: A Permutable MLP-Like Architecture for Visual Recognition”, Hou et al 2021
  • “Real-Time Neural Radiance Caching for Path Tracing”, Müller et al 2021
  • “Towards Biologically Plausible Convolutional Networks”, Pogodin et al 2021
  • “Well-Tuned Simple Nets Excel on Tabular Datasets”, Kadra et al 2021
  • “MLP Singer: Towards Rapid Parallel Korean Singing Voice Synthesis”, Tae et al 2021
  • “PairConnect: A Compute-Efficient MLP Alternative to Attention”, Xu et al 2021
  • “S²-MLP: Spatial-Shift MLP Architecture for Vision”, Yu et al 2021
  • “When Vision Transformers Outperform ResNets without Pre-Training or Strong Data Augmentations”, Chen et al 2021
  • “Container: Context Aggregation Network”, Gao et al 2021
  • “MixerGAN: An MLP-Based Architecture for Unpaired Image-To-Image Translation”, Cazenavette & Guevara2021
  • “One4all User Representation for Recommender Systems in E-Commerce”, Shin et al 2021
  • “Pay Attention to MLPs”, Liu et al 2021
  • “FNet: Mixing Tokens With Fourier Transforms”, Lee-Thorp et al 2021
  • “ResMLP: Feedforward Networks for Image Classification With Data-Efficient Training”, Touvron et al 2021
  • “Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNet”, Melas-Kyriazi2021
  • “Multi-Scale Inference of Genetic Trait Architecture Using Biologically Annotated Neural Networks”, Demetci et al 2021
  • “RepMLP: Re-Parameterizing Convolutions into Fully-Connected Layers for Image Recognition”, Ding et al 2021
  • “MLP-Mixer: An All-MLP Architecture for Vision”, Tolstikhin et al 2021
  • “Grokking: Generalization Beyond Overfitting On Small Algorithmic Datasets”, Power et al 2021
  • “Sifting out the Features by Pruning: Are Convolutional Networks the Winning Lottery Ticket of Fully Connected Ones?”, Pellegrini & Biroli2021
  • “Revisiting Simple Neural Probabilistic Language Models”, Sun & Iyyer2021
  • “KiloNeRF: Speeding up Neural Radiance Fields With Thousands of Tiny MLPs”, Reiser et al 2021
  • “Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows”, Liu et al 2021
  • “Attention Is Not All You Need: Pure Attention Loses Rank Doubly Exponentially With Depth”, Dong et al 2021
  • “Clusterability in Neural Networks”, Filan et al 2021
  • “Training Larger Networks for Deep Reinforcement Learning”, Ota et al 2021
  • “Explaining Neural Scaling Laws”, Bahri et al 2021
  • “Neural Geometric Level of Detail: Real-Time Rendering With Implicit 3D Shapes”, Takikawa et al 2021
  • “Is MLP-Mixer a CNN in Disguise? As Part of This Blog Post, We Look at the MLP Mixer Architecture in Detail and Also Understand Why It Is Not Considered Convolution Free.”
  • “Transformer Feed-Forward Layers Are Key-Value Memories”, Geva et al 2020
  • “AdnFM: An Attentive DenseNet Based Factorization Machine for CTR Prediction”, Wang et al 2020
  • “TabTransformer: Tabular Data Modeling Using Contextual Embeddings”, Huang et al 2020
  • “Scaling down Deep Learning”, Greydanus2020
  • “Image Generators With Conditionally-Independent Pixel Synthesis”, Anokhin et al 2020
  • “D2RL: Deep Dense Architectures in Reinforcement Learning”, Sinha et al 2020
  • “Fourier Neural Operator for Parametric Partial Differential Equations”, Li et al 2020
  • “AFT: An Attention Free Transformer”, Anonymous2020
  • “Towards Learning Convolutions from Scratch”, Neyshabur2020
  • “Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains”, Tancik et al 2020
  • “SIREN: Implicit Neural Representations With Periodic Activation Functions”, Sitzmann et al 2020
  • “Linformer: Self-Attention With Linear Complexity”, Wang et al 2020
  • “A Map of Object Space in Primate Inferotemporal Cortex”, Bao et al 2020
  • “Synthesizer: Rethinking Self-Attention in Transformer Models”, Tay et al 2020
  • “Deep Learning Training in Facebook Data Centers: Design of Scale-Up and Scale-Out Systems”, Naumov et al 2020
  • “NeRF: Representing Scenes As Neural Radiance Fields for View Synthesis”, Mildenhall et al 2020
  • “Cryptanalytic Extraction of Neural Network Models”, Carlini et al 2020
  • “ReZero Is All You Need: Fast Convergence at Large Depth”, Bachlechner et al 2020
  • “Train-By-Reconnect: Decoupling Locations of Weights from Their Values (LaPerm)”, Qiu & Suda2020
  • “Can Increasing Input Dimensionality Improve Deep Reinforcement Learning?”, Ota et al 2020
  • “Quasi-Equivalence of Width and Depth of Neural Networks”, Fan et al 2020
  • “Gesticulator: A Framework for Semantically-Aware Speech-Driven Gesture Generation”, Kucherenko et al 2020
  • “What’s Hidden in a Randomly Weighted Neural Network?”, Ramanujan et al 2019
  • “Understanding the Generalization of ‘Lottery Tickets’ in Neural Networks”, Morcos & Tian2019
  • “The Bouncer Problem: Challenges to Remote Explainability”, Merrer & Tredan2019
  • “3D Human Pose Estimation via Human Structure-Aware Fully Connected Network”, Zhang et al 2019d
  • “Finding the Needle in the Haystack With Convolutions: on the Benefits of Architectural Bias”, d’Ascoli et al 2019
  • “MoGlow: Probabilistic and Controllable Motion Synthesis Using Normalizing Flows”, Henter et al 2019
  • “Fixup Initialization: Residual Learning Without Normalization”, Zhang et al 2019
  • “SwitchNet: a Neural Network Model for Forward and Inverse Scattering Problems”, Khoo & Ying2018
  • “A Jamming Transition from Under-Parameterization to Over-Parameterization Affects Loss Landscape and Generalization”, Spigler et al 2018
  • “Neural Arithmetic Logic Units”, Trask et al 2018
  • “The Goldilocks Zone: Towards Better Understanding of Neural Network Loss Landscapes”, Fort & Scherlis2018
  • “Scalable Training of Artificial Neural Networks With Adaptive Sparse Connectivity Inspired by Network Science”, Mocanu et al 2018
  • “Deep Learning Generalizes Because the Parameter-Function Map Is Biased towards Simple Functions”, Valle-Pérez et al 2018
  • “Bidirectional Learning for Robust Neural Networks”, Pontes-Filho & Liwicki2018
  • “NAIS-Net: Stable Deep Networks from Non-Autonomous Differential Equations”, Ciccone et al 2018
  • “Meta-Learning Update Rules for Unsupervised Representation Learning”, Metz et al 2018
  • “Repurposing High-Throughput Image Assays Enables Biological Activity Prediction for Drug Discovery”, Simm et al 2018
  • “Improving Palliative Care With Deep Learning”, An et al 2018
  • “Learning to Play Chess With Minimal Lookahead and Deep Value Neural Networks”, Sabatelli2017 (page 3)
  • “Neural Collaborative Filtering”, He et al 2017
  • “Sharp Models on Dull Hardware: Fast and Accurate Neural Machine Translation Decoding on the CPU”, Devlin2017
  • “The Shattered Gradients Problem: If Resnets Are the Answer, Then What Is the Question?”, Balduzzi et al 2017
  • “Gender-From-Iris or Gender-From-Mascara?”, Kuehlkamp et al 2017
  • “Skip Connections Eliminate Singularities”, Orhan & Pitkow2017
  • “Deep Information Propagation”, Schoenholz et al 2016
  • “Topology and Geometry of Half-Rectified Network Optimization”, Freeman & Bruna2016
  • “On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima”, Keskar et al 2016
  • “Decoupled Neural Interfaces Using Synthetic Gradients”, Jaderberg et al 2016
  • “Learning to Optimize”, Li & Malik2016
  • “Do Deep Convolutional Nets Really Need to Be Deep and Convolutional?”, Urban et al 2016
  • “Network Morphism”, Wei et al 2016
  • “Adding Gradient Noise Improves Learning for Very Deep Networks”, Neelakantan et al 2015
  • “How Far Can We Go without Convolution: Improving Fully-Connected Networks”, Lin et al 2015
  • “BinaryConnect: Training Deep Neural Networks With Binary Weights during Propagations”, Courbariaux et al 2015
  • “Tensorizing Neural Networks”, Novikov et al 2015
  • “A Neural Attention Model for Abstractive Sentence Summarization”, Rush et al 2015
  • “Deep Neural Networks for Large Vocabulary Handwritten Text Recognition”, Bluche2015
  • “In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning”, Neyshabur et al 2014
  • “The Loss Surfaces of Multilayer Networks”, Choromanska et al 2014
  • “On the Number of Linear Regions of Deep Neural Networks”, Montúfar et al 2014
  • “Do Deep Nets Really Need to Be Deep?”, Ba & Caruana2013
  • “On the Number of Response Regions of Deep Feed Forward Networks With Piece-Wise Linear Activations”, Pascanu et al 2013
  • “Network In Network”, Lin et al 2013
  • “Deep Big Multilayer Perceptrons for Digit Recognition”, Cireşan et al 2012
  • “Deep Big Simple Neural Nets Excel on Handwritten Digit Recognition”, Ciresan et al 2010
  • “Compositional Pattern Producing Networks: A Novel Abstraction of Development”, Stanley2007
  • “Extraction De Séquences Numériques Dans Des Documents Manuscrits Quelconques”, Chatelain2006
  • “Best Practices for Convolutional Neural Networks Applied to Visual Document Analysis”, Simard et al 2003
  • “NEAT: Evolving Neural Networks through Augmenting Topologies”, Stanley & Miikkulainen2002
  • “DARPA and the Quest for Machine Intelligence, 1983^–₁₀1993_31ya”, Roland & Shiman2002
  • “Quantitative Analysis of Multivariate Data Using Artificial Neural Networks: A Tutorial Review and Applications to the Deconvolution of Pyrolysis Mass Spectra”, Goodacre et al 1996
  • “Statistical Mechanics of Generalization”, Opper & Kinzel1996
  • “On the Ability of the Optimal Perceptron to Generalize”, Opper et al 1990
  • “Learning To Tell Two Spirals Apart”, Lang & Witbrock1988
  • “Learning Internal Representations by Error Propagation”, Rumelhart et al 1986
  • “Neural Networks and Physical Systems With Emergent Collective Computational Abilities”, Hopfield1982
  • Wikipedia
• Miscellaneous
• Bibliography