“MLPs Learn In-Context”, William L. Tong, Cengiz Pehlevan2024-05-24 (MLP NN, meta-learning; backlinks)⁠:

In-context learning (ICL), the remarkable ability to solve a task from only input exemplars, has commonly been assumed to be a unique hallmark of Transformer models.

In this study, we demonstrate that multi-layer perceptrons (MLPs) can also learn in-context. Moreover, we find that MLPs, and the closely related MLP-Mixer models, learn in-context competitively with Transformers given the same compute budget.

We further show that MLPs outperform Transformers on a subset of ICL tasks designed to test relational reasoning. These results suggest that in-context learning is not exclusive to Transformers and highlight the potential of exploring this phenomenon beyond attention-based architectures.

In addition, MLPs’ surprising success on relational tasks challenges prior assumptions about simple connectionist models. Altogether, our results endorse the broad trend that “less inductive bias is better” and contribute to the growing interest in all-MLP alternatives to task-specific architectures.

Backlinks: