Bibliography (14):

  1. https://github.com/socialfoundations/tttlm

  2. Large-Scale Retrieval for Reinforcement Learning

  3. In-Context Pretraining (ICP): Language Modeling Beyond Document Boundaries

  4. โ€‹ GPT-3 Creative Fiction ยง Prompts As Programming

  5. https://github.com/EleutherAI/The-Pile

  6. The Pile: An 800GB Dataset of Diverse Text for Language Modeling

  7. EleutherAI/gpt-Neo: An Implementation of Model Parallel GPT-2 and GPT-3-Style Models Using the Mesh-Tensorflow Library.

  8. Recurrent Neural Network Based Language Model ยง Dynamic Evaluation

  9. Language Models are Unsupervised Multitask Learners

  10. 2023-hardt-figure6-perplexitiesdecreasewhentrainingonincreasinglymoreneighborsusinggpt2onthepile.jpg

  11. Wikipedia Bibliography:

    1. https://en.wikipedia.org/wiki/Language_modelโ€‰โ€Š:

      https://en.wikipedia.org/wiki/Language_model

    2. Nearest neighbor search

    3. https://en.wikipedia.org/wiki/GPT-2โ€‰โ€Š:

      https://en.wikipedia.org/wiki/GPT-2

    4. Byte