Transformers in Vision

Introduction

Initially introduced with the now-famous Attention is all you need¹, the Transformer has dominated the field of Natural Language Processing (NLP) for years. Particularly worth noting is the effort gone into scaling up Transformer-based models, such as BERT², MegatronLM³, T5⁴, and the various GPTs (GPT⁵, GPT-2⁶ and GPT-3⁷), due to their favourable scaling characteristics^8,9.

The success of transformers in NLP has not gone unnoticed in other fields, where they have been responsible for significant breakthroughs such as AlphaFold 2 in the field of protein folding.

Important works adapting transformers (and self-attention) to vision include Attention Augmented Convolutional Networks¹⁰, Stand-Alone Self-Attention models¹¹ (SASA models), DETR¹², Visual Transformers¹³ and LambdaNetworks¹⁴; as well as Image Transformers¹⁵ and Axial Transformers¹⁶ in the generative domain.

For an in-depth introduction to these works, we recommend two recent reviews: A Survey in Visual Transformers¹⁷ and Transformers in Vision: A Survey¹⁸.

Overview

This blog post aims to summarize recent research in applying transformers and self-attention to vision, with a focus on (but scope not limited to) image classification. While by no means exhaustive, it can hopefully represent a starting point for a more in-depth dive into the literature.

We first introduce the Vision Transformer, a simple yet powerful architecture that has had a significant influence on recent research due to its performance in large data regimes. We then continue with the many works studying how to achieve similar high performance using transformers (and self-attention) when data is not as plentiful. Finally, we discuss papers studying the robustness of these models to perturbations as well as their performance in self-supervised, medical and video tasks.

Figures are taken from their respective papers unless the source is explicitly provided in the caption.

Before we begin, it is worth noting that familiarity with transformers and self-attention is recommended; great resources include Jay Alammar’s Illustrated Transformer and Peter Bloem’s Transformers from scratch. For high-quality implementations of many models discussed in this post, check out Ross Wightman’s PyTorch Image Models as well as Phil Wang’s work.

Vision Transformers for Image Recognition

An image is worth 16x16 words: the Vision Transformer

First introduced in An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale¹⁹, Vision Transformers (ViTs) have taken computer vision by storm, leading to hundreds of citations in the span of a few months. The paper’s main goal was to show that a vanilla Transformer, once adapted to deal with data from the visual domain, could compete with some of the most performant convolutional neural networks (CNNs) developed up to that point.

The Vision Transformer architecture is conceptually simple: divide the image into patches, flatten and project them into a D$D$-dimensional embedding space obtaining the so-called patch embeddings, add positional embeddings (a set of learnable vectors allowing the model to retain positional information) and concatenate a (learnable) class token, then let the Transformer encoder do its magic. Finally, a classification head is applied to the class token to obtain the model’s logits.