“PIXEL: Language Modeling With Pixels”, Phillip Rust, Jonas F. Lotz, Emanuele Bugliarello, Elizabeth Salesky, Miryam de Lhoneux, Desmond Elliott2022-07-14 (, , ; backlinks)⁠:

[cf. Mansimov et al 2020, Hinami et al 2020, Schmidhuber2018. Extremely useful for large-scale pretraining on PDFs?] Language models are defined over a finite set of inputs, which creates a vocabulary bottleneck when we attempt to scale the number of supported languages. Tackling this bottleneck results in a trade-off between what can be represented in the embedding matrix and computational issues in the output layer.

This paper introduces PIXEL, the Pixel-based Encoder of Language, which suffers from neither of these issues. PIXEL is a pretrained language model that renders text as images, making it possible to transfer representations across languages based on orthographic similarity or the co-activation of pixels. PIXEL is trained to reconstruct the pixels of masked patches, instead of predicting a distribution over tokens.

We pretrain the 86M parameter PIXEL model on the same English data as BERT and evaluate on syntactic and semantic tasks in typologically diverse languages, including various non-Latin scripts.

We find that PIXEL substantially outperforms BERT on syntactic and semantic processing tasks on scripts that are not found in the pretraining data, but PIXEL is slightly weaker than BERT when working with Latin scripts. Furthermore, we find that PIXEL is more robust to noisy text inputs than BERT, further confirming the benefits of modeling language with pixels.

Figure 1: Overview of PIXEL’s architecture. Following He et al 2022, we use a masked autoencoder with a ViT architecture and a lightweight decoder for pretraining (left). At finetuning time (right), the decoder is replaced by a task-specific classification head that sits on top of the encoder.

…We propose to rethink language modeling as a visual recognition task, which removes the need for a finite vocabulary. Our proposal is inspired by Salesky et al 2021, who showed how to train a machine translation model with “visual text representations” in the encoder instead of subwords. Our Pixel-based Encoder of Language (PIXEL) is built on the Masked Autoencoding Visual Transformer (ViT-MAE; He et al 2022). ViT-MAE is a Transformer-based encoder-decoder trained to reconstruct the pixels in masked image patches. PIXEL does not have a vocabulary embedding layer; instead, it renders text as a sequence of fixed-sized patches and processes the patches using a vision transformer encoder (Dosovitskiy et al 2021). PIXEL also does not have a computationally expensive output layer when it reconstructs the pixels of the masked patches. In effect, PIXEL provides a solution to the vocabulary bottleneck without paying the cost of prohibitively long sequences.

Figure 3: PIXEL image reconstructions after 100k, 500k, and 1M steps of pretraining. We overlay the masked original image with the model’s predictions. Images are wrapped into squares and resized for visualization purposes only. The texts were not part of the training data. We see that the fully trained PIXEL (1M) predicts masked spans more clearly and accurately. For longer spans with a larger possible prediction space, multiple predictions may appear together creating blurred text.

See Also:

View PDF:

PIXEL: Language Modeling With Pixels