See Also

Miscellaneous

• /doc/ai/nn/transformer/t5/2022-wallace-figure2-berkeleycrosswordsolversolutionpipelineofqatoloopybeliefpropagationtobyt5trefinement.png

• /doc/ai/nn/transformer/t5/2022-scialom-table5-ablationofsheerparameterscalevsscaleduppretraininginenablingcontinuallearningwithoutforgetting.png

• /doc/ai/nn/transformer/t5/2022-scialom-figure2-t0languagemodelpreservesperformanceofpreviouslylearnedtasksasnewtasksareintroducedwithminimalrehearsalsolvingcontinuallearning.png

• /doc/ai/nn/transformer/t5/2022-patel-figure2-mt5fewshotpromptingbootstrapselfdistillationprocess.png

• /doc/ai/nn/transformer/t5/2021-tay-figure1-t5pretrainingvsfinetuningtransferscaling.png

• /doc/ai/nn/transformer/t5/2019-raffel-figure6-effectsofdatasetduplicationont5traininglosscurves.png

• https://colab.research.google.com/drive/1-ROO7L09EupLFLQM-TWgDHa5-FIOdLLh

• https://github.com/PiotrNawrot/nanoT5

• https://github.com/deep-floyd/IF

• https://github.com/google-research/byt5

• https://github.com/google-research/google-research/tree/master/ul2

• https://github.com/mbzuai-nlp/LaMini-LM

• https://huggingface.co/models?search=flan-t5

• https://threadreaderapp.com/thread/1187161460033458177.html

• https://twitter.com/Mascobot/status/1618246707267141632

• https://twitter.com/RamaswmySridhar/status/1621870497070981121

• https://twitter.com/_jasonwei/status/1621333297891790848

• https://twitter.com/abacaj/status/1618050431657324545

• https://www.forbes.com/sites/rashishrivastava/2023/04/11/writer-generative-ai/

• https://www.yitay.net/blog/flan-ul2-20b

Link Bibliography

1. https://arxiv.org/abs/2307.06440: “No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Models”, Jean Kaddour, Oscar Key, Piotr Nawrot, Pasquale Minervini, Matt J. Kusner

   link-bibliography

2. https://arxiv.org/abs/2305.09636#google: “SoundStorm: Efficient Parallel Audio Generation”, Zalán Borsos, Matt Sharifi, Damien Vincent, Eugene Kharitonov, Neil Zeghidour, Marco Tagliasacchi

   link-bibliography

3. https://arxiv.org/abs/2304.13731: “TANGO: Text-to-Audio Generation Using Instruction-Tuned LLM and Latent Diffusion Model”, Deepanway Ghosal, Navonil Majumder, Ambuj Mehrish, Soujanya Poria

   link-bibliography

4. https://arxiv.org/abs/2304.08467: “Learning to Compress Prompts With Gist Tokens”, Jesse Mu, Xiang Lisa Li, Noah Goodman

   link-bibliography

5. https://arxiv.org/abs/2301.12597#salesforce: “BLIP-2: Bootstrapping Language-Image Pre-training With Frozen Image Encoders and Large Language Models”, Junnan Li, Dongxu Li, Silvio Savarese, Steven Hoi

   link-bibliography

6. https://arxiv.org/abs/2301.00704#google: “Muse: Text-To-Image Generation via Masked Generative Transformers”, Huiwen Chang, Han Zhang, Jarred Barber, A. J. Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy

   link-bibliography

7. https://arxiv.org/abs/2212.09741: “One Embedder, Any Task: Instruction-Finetuned Text Embeddings (INSTRUCTOR)”, Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A. Smith

   link-bibliography

8. https://arxiv.org/abs/2212.05055#google: “Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints”, Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie

   link-bibliography

9. https://arxiv.org/abs/2211.01324#nvidia: “EDiff-I: Text-to-Image Diffusion Models With an Ensemble of Expert Denoisers”, Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila

   link-bibliography

10. https://arxiv.org/abs/2210.13669: “Help Me Write a Poem: Instruction Tuning As a Vehicle for Collaborative Poetry Writing (CoPoet)”, Tuhin Chakrabarty, Vishakh Padmakumar, He He

    link-bibliography

11. https://arxiv.org/abs/2210.11416#google: “FLAN: Scaling Instruction-Finetuned Language Models”, Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang

    link-bibliography

12. https://arxiv.org/abs/2210.02414#baai: “GLM-130B: An Open Bilingual Pre-trained Model”, Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng

    link-bibliography

13. https://arxiv.org/abs/2209.14500: “SAP: Bidirectional Language Models Are Also Few-shot Learners”, Ajay Patel, Bryan Li, Mohammad Sadegh Rasooli, Noah Constant, Colin Raffel, Chris Callison-Burch

    link-bibliography

14. https://arxiv.org/abs/2208.11663#facebook: “PEER: A Collaborative Language Model”, Timo Schick, Jane Dwivedi-Yu, Zhengbao Jiang, Fabio Petroni, Patrick Lewis, Gautier Izacard, Qingfei You

    link-bibliography

15. https://arxiv.org/abs/2208.09770#microsoft: “Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization”, Pengcheng He, Baolin Peng, Liyang Lu, Song Wang, Jie Mei, Yang Liu, Ruochen Xu, Hany Hassan Awadalla

    link-bibliography

16. https://arxiv.org/abs/2206.15474: “Forecasting Future World Events With Neural Networks”, Andy Zou, Tristan Xiao, Ryan Jia, Joe Kwon, Mantas Mazeika, Richard Li, Dawn Song, Jacob Steinhardt, Owain Evans

    link-bibliography

17. https://arxiv.org/abs/2206.07808#amazon: “Alexa Teacher Model: Pretraining and Distilling Multi-Billion-Parameter Encoders for Natural Language Understanding Systems”, Jack FitzGerald, Shankar Ananthakrishnan, Konstantine Arkoudas, Davide Bernardi, Abhishek Bhagia, Claudio Delli Bovi

    link-bibliography

18. https://openreview.net/forum?id=0ZbPmmB61g#google: “Boosting Search Engines With Interactive Agents”, Massimiliano Ciaramita, Leonard Adolphs, Michelle Chen Huebscher, Sascha Rothe, Christian Buck, Thomas Hofmann

    link-bibliography

19. https://arxiv.org/abs/2205.12393: “CT0: Fine-tuned Language Models Are Continual Learners”, Thomas Scialom, Tuhin Chakrabarty, Smaranda Muresan

    link-bibliography

20. https://arxiv.org/abs/2205.12209#google: “EdiT5: Semi-Autoregressive Text-Editing With T5 Warm-Start”, Jonathan Mallinson, Jakub Adamek, Eric Malmi, Aliaksei Severyn

    link-bibliography

21. https://arxiv.org/abs/2205.11487#google: “Imagen: Photorealistic Text-to-Image Diffusion Models With Deep Language Understanding”, Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour

    link-bibliography

22. https://arxiv.org/abs/2205.09665#bair: “Automated Crossword Solving”, Eric Wallace, Nicholas Tomlin, Albert Xu, Kevin Yang, Eshaan Pathak, Matthew Ginsberg, Dan Klein

    link-bibliography

23. https://arxiv.org/abs/2205.05131#google: “Unifying Language Learning Paradigms”, Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng

    link-bibliography

24. https://arxiv.org/abs/2204.07705: “Tk-Instruct: Benchmarking Generalization via In-Context Instructions on 1,600+ Language Tasks”, Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar

    link-bibliography

25. https://arxiv.org/abs/2204.03067: “ByT5 Model for Massively Multilingual Grapheme-to-phoneme Conversion”, Jian Zhu, Cong Zhang, David Jurgens

    link-bibliography

26. https://arxiv.org/abs/2203.00759: “HyperPrompt: Prompt-based Task-Conditioning of Transformers”, Yun He, Huaixiu Steven Zheng, Yi Tay, Jai Gupta, Yu Du, Vamsi Aribandi, Zhe Zhao, YaGuang Li, Zhao Chen

    link-bibliography

27. https://arxiv.org/abs/2202.11822#google: “Using Natural Language Prompts for Machine Translation”, Xavier Garcia, Orhan Firat

    link-bibliography

28. https://arxiv.org/abs/2202.09368#google: “Mixture-of-Experts With Expert Choice Routing”, Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew Dai, Zhifeng Chen, Quoc Le

    link-bibliography

29. https://arxiv.org/abs/2201.11473#microsoft: “Reasoning Like Program Executors”, Xinyu Pi, Qian Liu, Bei Chen, Morteza Ziyadi, Zeqi Lin, Yan Gao, Qiang Fu, Jian-Guang Lou, Weizhu Chen

    link-bibliography

30. https://arxiv.org/abs/2112.07916#google: “LongT5: Efficient Text-To-Text Transformer for Long Sequences”, Mandy Guo, Joshua Ainslie, David Uthus, Santiago Ontanon, Jianmo Ni, Yun-Hsuan Sung, Yinfei Yang

    link-bibliography

31. https://arxiv.org/abs/2112.07899#google: “Large Dual Encoders Are Generalizable Retrievers”, Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernández Ábrego, Ji Ma, Vincent Y. Zhao, Yi Luan, Keith B. Hall

    link-bibliography

32. https://arxiv.org/abs/2112.11446#deepmind: “Scaling Language Models: Methods, Analysis & Insights from Training Gopher”, Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides

    link-bibliography

33. https://arxiv.org/abs/2110.11309: “Fast Model Editing at Scale”, Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, Christopher D. Manning

    link-bibliography

34. https://arxiv.org/abs/2109.10686#google: “Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers”, Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama

    link-bibliography

35. https://arxiv.org/abs/2109.07958: “TruthfulQA: Measuring How Models Mimic Human Falsehoods”, Stephanie Lin, Jacob Hilton, Owain Evans

    link-bibliography

36. https://arxiv.org/abs/2109.02593#allen: “General-Purpose Question-Answering With Macaw”, Oyvind Tafjord, Peter Clark

    link-bibliography

37. https://arxiv.org/abs/2108.08877#google: “Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models”, Jianmo Ni, Gustavo Hernández Ábrego, Noah Constant, Ji Ma, Keith B. Hall, Daniel Cer, Yinfei Yang

    link-bibliography

38. https://arxiv.org/abs/2106.00737: “Implicit Representations of Meaning in Neural Language Models”, Belinda Z. Li, Maxwell Nye, Jacob Andreas

    link-bibliography

39. https://arxiv.org/abs/2105.13626#google: “ByT5: Towards a Token-free Future With Pre-trained Byte-to-byte Models”, Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, Colin Raffel

    link-bibliography

40. https://arxiv.org/abs/2104.10350#google: “Carbon Emissions and Large Neural Network Training”, David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So

    link-bibliography

41. https://arxiv.org/abs/2104.08691#google: “The Power of Scale for Parameter-Efficient Prompt Tuning”, Brian Lester, Rami Al-Rfou, Noah Constant

    link-bibliography

42. https://arxiv.org/abs/2103.13009#allen: “UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmark”, Nicholas Lourie, Ronan Le Bras, Chandra Bhagavatula, Yejin Choi

    link-bibliography

43. https://arxiv.org/abs/2102.13019: “Investigating the Limitations of the Transformers With Simple Arithmetic Tasks”, Rodrigo Nogueira, Zhiying Jiang, Jimmy Li

    link-bibliography

44. https://arxiv.org/abs/2101.03961#google: “Switch Transformers: Scaling to Trillion Parameter Models With Simple and Efficient Sparsity”, William Fedus, Barret Zoph, Noam Shazeer

    link-bibliography

45. https://arxiv.org/abs/2009.03300: “MMLU: Measuring Massive Multitask Language Understanding”, Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, Jacob Steinhardt

    link-bibliography

46. https://arxiv.org/abs/2007.06225: “ProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised Deep Learning and High Performance Computing”, Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rihawi, Yu Wang, Llion Jones, Tom Gibbs

    link-bibliography