See Also

Miscellaneous

Link Bibliography

1. https://arxiv.org/abs/2301.12597#salesforce: “BLIP-2: Bootstrapping Language-Image Pre-training With Frozen Image Encoders and Large Language Models”⁠, Junnan Li, Dongxu Li, Silvio Savarese, Steven Hoi:

   link-bibliography

2. https://arxiv.org/abs/2301.00704#google: “Muse: Text-To-Image Generation via Masked Generative Transformers”⁠, Huiwen Chang, Han Zhang, Jarred Barber, A. J. Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy:

   link-bibliography

3. https://arxiv.org/abs/2212.09741: “One Embedder, Any Task: Instruction-Finetuned Text Embeddings (INSTRUCTOR)”⁠, Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A. Smith:

   link-bibliography

4. https://arxiv.org/abs/2212.05055#google: “Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints”⁠, Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie:

   link-bibliography

5. https://arxiv.org/abs/2211.01324#nvidia: “EDiff-I: Text-to-Image Diffusion Models With an Ensemble of Expert Denoisers”⁠, Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila:

   link-bibliography

6. https://arxiv.org/abs/2210.13669: “Help Me Write a Poem: Instruction Tuning As a Vehicle for Collaborative Poetry Writing (CoPoet)”⁠, Tuhin Chakrabarty, Vishakh Padmakumar, He He:

   link-bibliography

7. https://arxiv.org/abs/2210.11416#google: “FLAN: Scaling Instruction-Finetuned Language Models”⁠, Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang:

   link-bibliography

8. https://arxiv.org/abs/2210.02414#baai: “GLM-130B: An Open Bilingual Pre-trained Model”⁠, Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng:

   link-bibliography

9. https://arxiv.org/abs/2209.14500: “SAP: Bidirectional Language Models Are Also Few-shot Learners”⁠, Ajay Patel, Bryan Li, Mohammad Sadegh Rasooli, Noah Constant, Colin Raffel, Chris Callison-Burch:

   link-bibliography

10. https://arxiv.org/abs/2208.11663#facebook: “PEER: A Collaborative Language Model”⁠, Timo Schick, Jane Dwivedi-Yu, Zhengbao Jiang, Fabio Petroni, Patrick Lewis, Gautier Izacard, Qingfei You:

    link-bibliography

11. https://arxiv.org/abs/2206.07808#amazon: “Alexa Teacher Model: Pretraining and Distilling Multi-Billion-Parameter Encoders for Natural Language Understanding Systems”⁠, Jack FitzGerald, Shankar Ananthakrishnan, Konstantine Arkoudas, Davide Bernardi, Abhishek Bhagia, Claudio Delli Bovi:

    link-bibliography

12. https://openreview.net/forum?id=0ZbPmmB61g#google: “Boosting Search Engines With Interactive Agents”⁠, Massimiliano Ciaramita, Leonard Adolphs, Michelle Chen Huebscher, Sascha Rothe, Christian Buck, Thomas Hofmann:

    link-bibliography

13. https://arxiv.org/abs/2205.12393#facebook: “CT0: Fine-tuned Language Models Are Continual Learners”⁠, Thomas Scialom, Tuhin Chakrabarty, Smaranda Muresan:

    link-bibliography

14. https://arxiv.org/abs/2205.12209#google: “EdiT5: Semi-Autoregressive Text-Editing With T5 Warm-Start”⁠, Jonathan Mallinson, Jakub Adamek, Eric Malmi, Aliaksei Severyn:

    link-bibliography

15. https://arxiv.org/abs/2205.11487#google: “Imagen: Photorealistic Text-to-Image Diffusion Models With Deep Language Understanding”⁠, Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour:

    link-bibliography

16. https://arxiv.org/abs/2205.09665#bair: “Automated Crossword Solving”⁠, Eric Wallace, Nicholas Tomlin, Albert Xu, Kevin Yang, Eshaan Pathak, Matthew Ginsberg, Dan Klein:

    link-bibliography

17. https://arxiv.org/abs/2205.05131#google: “Unifying Language Learning Paradigms”⁠, Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng:

    link-bibliography

18. https://arxiv.org/abs/2204.07705: “T<em>k< / ​em>-Instruct: Benchmarking Generalization via In-Context Instructions on 1,600+ Language Tasks”⁠, Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar:

    link-bibliography

19. https://arxiv.org/abs/2204.03067: “ByT5 Model for Massively Multilingual Grapheme-to-phoneme Conversion”⁠, Jian Zhu, Cong Zhang, David Jurgens:

    link-bibliography

20. https://arxiv.org/abs/2203.00759: “HyperPrompt: Prompt-based Task-Conditioning of Transformers”⁠, Yun He, Huaixiu Steven Zheng, Yi Tay, Jai Gupta, Yu Du, Vamsi Aribandi, Zhe Zhao, YaGuang Li, Zhao Chen:

    link-bibliography

21. https://arxiv.org/abs/2202.11822#google: “Using Natural Language Prompts for Machine Translation”⁠, Xavier Garcia, Orhan Firat:

    link-bibliography

22. https://arxiv.org/abs/2202.09368#google: “Mixture-of-Experts With Expert Choice Routing”⁠, Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew Dai, Zhifeng Chen, Quoc Le:

    link-bibliography

23. https://arxiv.org/abs/2201.11473#microsoft: “Reasoning Like Program Executors”⁠, Xinyu Pi, Qian Liu, Bei Chen, Morteza Ziyadi, Zeqi Lin, Yan Gao, Qiang Fu, Jian-Guang Lou, Weizhu Chen:

    link-bibliography

24. https://arxiv.org/abs/2112.07916#google: “LongT5: Efficient Text-To-Text Transformer for Long Sequences”⁠, Mandy Guo, Joshua Ainslie, David Uthus, Santiago Ontanon, Jianmo Ni, Yun-Hsuan Sung, Yinfei Yang:

    link-bibliography

25. https://arxiv.org/abs/2112.07899#google: “Large Dual Encoders Are Generalizable Retrievers”⁠, Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernández Ábrego, Ji Ma, Vincent Y. Zhao, Yi Luan, Keith B. Hall:

    link-bibliography

26. https://arxiv.org/abs/2112.11446#deepmind: “Scaling Language Models: Methods, Analysis & Insights from Training Gopher”⁠, Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides:

    link-bibliography

27. https://arxiv.org/abs/2110.11309: “Fast Model Editing at Scale”⁠, Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, Christopher D. Manning:

    link-bibliography

28. https://arxiv.org/abs/2109.10686#google: “Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers”⁠, Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama:

    link-bibliography

29. https://arxiv.org/abs/2109.07958: “TruthfulQA: Measuring How Models Mimic Human Falsehoods”⁠, Stephanie Lin, Jacob Hilton, Owain Evans:

    link-bibliography

30. https://arxiv.org/abs/2109.02593#allen: “General-Purpose Question-Answering With Macaw”⁠, Oyvind Tafjord, Peter Clark:

    link-bibliography

31. https://arxiv.org/abs/2108.08877#google: “Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models”⁠, Jianmo Ni, Gustavo Hernández Ábrego, Noah Constant, Ji Ma, Keith B. Hall, Daniel Cer, Yinfei Yang:

    link-bibliography

32. https://arxiv.org/abs/2106.00737: “Implicit Representations of Meaning in Neural Language Models”⁠, Belinda Z. Li, Maxwell Nye, Jacob Andreas:

    link-bibliography

33. https://arxiv.org/abs/2105.13626#google: “ByT5: Towards a Token-free Future With Pre-trained Byte-to-byte Models”⁠, Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, Colin Raffel:

    link-bibliography

34. https://arxiv.org/abs/2104.10350#google: “Carbon Emissions and Large Neural Network Training”⁠, David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So:

    link-bibliography

35. https://arxiv.org/abs/2104.08691#google: “The Power of Scale for Parameter-Efficient Prompt Tuning”⁠, Brian Lester, Rami Al-Rfou, Noah Constant:

    link-bibliography

36. https://arxiv.org/abs/2103.13009#allen: “UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmark”⁠, Nicholas Lourie, Ronan Le Bras, Chandra Bhagavatula, Yejin Choi:

    link-bibliography

37. https://arxiv.org/abs/2102.13019: “Investigating the Limitations of the Transformers With Simple Arithmetic Tasks”⁠, Rodrigo Nogueira, Zhiying Jiang, Jimmy Li:

    link-bibliography

38. https://arxiv.org/abs/2101.03961#google: “Switch Transformers: Scaling to Trillion Parameter Models With Simple and Efficient Sparsity”⁠, William Fedus, Barret Zoph, Noam Shazeer:

    link-bibliography

39. https://arxiv.org/abs/2009.03300: “MMLU: Measuring Massive Multitask Language Understanding”⁠, Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, Jacob Steinhardt:

    link-bibliography

40. https://arxiv.org/abs/2007.06225: “ProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised Deep Learning and High Performance Computing”⁠, Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rihawi, Yu Wang, Llion Jones, Tom Gibbs:

    link-bibliography