“RNN Metadata for Mimicking Author Style”, Gwern2015-09-12 (RNN, NN sampling, poetry by AI, CLI, tutorial; backlinks)⁠:

Teaching a text-generating char-RNN to automatically imitate many different authors by labeling the input text by author; additional experiments include imitating Geocities and retraining GPT-2 on a large Project Gutenberg poetry corpus.

Char-RNNs are unsupervised generative models which learn to mimic text sequences. I suggest extending char-RNNs with inline metadata such as genre or author prefixed to each line of input, allowing for better & more efficient metadata, and more controllable sampling of generated output by feeding in desired metadata. A 2015 experiment using torch-rnn on a set of ~30 Project Gutenberg e-books (1 per author) to train a large char-RNN shows that a char-RNN can learn to remember metadata such as authors, learn associated prose styles, and often generate text visibly similar to that of a specified author.

I further try & fail to train a char-RNN on Geocities HTML for unclear reasons.

More successfully, I experiment in 2019 with a recently-developed alternative to char-RNNs, the Transformer NN architecture, by finetuning training OpenAI’s GPT-2-117M Transformer model on a much larger (117MB) Project Gutenberg poetry corpus using both unlabeled lines & lines with inline metadata (the source book). The generated poetry is much better. And GPT-3 is better still.

• Handling Multiple Corpuses
• Implementation
  • Data
  • Unlabeled
  • Training With Prefixes
    • Small RNN
    • Larger RNN
    • Larger Author Count
    • Class Imbalance Fix
      • Success
  • Training With Prefixes+suffixes
  • Classification
  • Transforms
• Conclusions
• External Links
• Appendix
  • Geocities Char-RNN
    • Data Extraction
    • Training
    • Data Cleaning
    • The Bounce Continues
  • Finetuning the GPT-2-117M Transformer for English Poetry Generation

Backlinks: