“Training a T5 Using Lab-Sized Resources”, Manuel R. Ciosici, Leon Derczynski2022-08-25 (T5 Transformer)⁠:

Training large neural language models on large datasets is resource/time-intensive. These requirements create a barrier to entry, where those with fewer resources cannot build competitive models.

This paper presents various techniques for making it possible to (1) train a large language model using resources that a modest research lab might have, and (2) train it in a reasonable amount of time.

We provide concrete recommendations for practitioners, which we illustrate with a case study: a T5 model for Danish, the first for this language.