![](data:image/svg+xml,<svg xmlns="http://www.w3.org/2000/svg" width="73" height="73"><rect fill-opacity="0"/></svg>)
The 🌟Flan Collection🌟 (1st used in Flan-PaLM bit.ly/3Zu7bU2):
➕ Merges Flan 2021, P3, NIv2, CoT instruction-datasets into 1800+ dataset collection
➕ Data augmentations and mixing strategies
➕ 100s new templates
2/
This yields the best performing instruction tuning collection that has been compiled and released into one repo.
See our survey Figure of the prior works we built on to produce this compilation.
3/
Q: But why are the results strong?
Our breakdown of the Flan Collection shows *why* it works. The most important methods:
🌟Finding 1🌟 Fine-tuning on zero-shot and few-shot prompts together significantly improves both settings (not a trade-off)!
4/
🌟Finding 2🌟 Input inversion and data source balancing (as proposed and corroborated by MetaICL, T0, OPT-IML and others...) are incredibly important for successful instruction tuning.
See our ablations Table.
5/
🌟Finding 3 🌟 The Flan-T5 model converges higher and faster than T5 on single-task fine-tuning.
➡️ Recommendation: Use Flan-T5 as your base model for new tasks.
✅Better computational-efficiency and performance!
6/
➡️ It's promising these results don't use any RLHF data, or human "alignment", which is expensive to collect and less publicly available.
We hope this release supports the open source community, and improves instruction tuning methods and research!
arxiv.org/abs/2301.13688
7/
Please don’t hesitate to reach out with questions, thoughts, and critiques. We are always open to feedback! 😃
8/
Lastly, a heartfelt thank you to my awesome mentors at Google – @_jasonwei, @barret_zoph, @YiTayML, @denny_zhou, @quocleix, and @ada_rob.
9/
As well as my fantastic co-contributors and colleagues @Hou_Le, @hwchung27, @tuvuumass, and @albertwebson, who ran many experiments and led the open sourcing infrastructure.
10/10
