- See Also
-
Links
- “Google Is Asking Employees to Test Potential ChatGPT Competitors, including a Chatbot Called 'Apprentice Bard'”, Elias 2023
- “Creative Writing With Wordcraft, an AI-Powered Writing Assistant: Perspectives from Professional Writers”, Ippolito et al 2022
- “Language Model Cascades”, Dohan et al 2022
- “Exploring Length Generalization in Large Language Models”, Anil et al 2022
- “Least-to-Most Prompting Enables Complex Reasoning in Large Language Models”, Zhou et al 2022
- “Google Is Beta Testing Its AI Future: After Mistakes and Challenges, the Company Is Moving a Little Slower With AI Language Models”, Vincent 2022
- “PaLM: Scaling Language Modeling With Pathways”, Chowdhery et al 2022
- “Self-Consistency Improves Chain-of-Thought Reasoning in Language Models”, Wang et al 2022
- “PromptChainer: Chaining Large Language Model Prompts through Visual Programming”, Wu et al 2022
- “Using Natural Language Prompts for Machine Translation”, Garcia & Firat 2022
- “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models”, Wei et al 2022
- “LaMDA: Language Models for Dialog Applications”, Thoppilan et al 2022
- “SynthBio: A Case Study in Faster Curation of Text Datasets”, Yuan et al 2022
- “Discovering the Syntax and Strategies of Natural Language Programming With Generative Language Models”, Jiang et al 2022
- “GLaM: Efficient Scaling of Language Models With Mixture-of-Experts”, Du et al 2021
- “Show Your Work: Scratchpads for Intermediate Computation With Language Models”, Nye et al 2021
- “AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts”, Wu et al 2021
- “A Recipe For Arbitrary Text Style Transfer With Large Language Models”, Reif et al 2021
- “GenLine and GenForm: Two Tools for Interacting With Generative Language Models in a Code Editor”, Jiang et al 2021b
- “FLAN: Finetuned Language Models Are Zero-Shot Learners”, Wei et al 2021
- “Program Synthesis With Large Language Models”, Austin et al 2021
- “Towards a Human-like Open-Domain Chatbot”, Adiwardana et al 2020
- “LaMDA: Our Breakthrough Conversation Technology”
- Sort By Magic
- Miscellaneous
- Link Bibliography
Google LaMDA is a large 137b-parameter dense Transformer neural network model, announced by Google in May 2021, as a followup to Meena; it is most similar to OpenAI’s May 2020 GPT-3 (175b) in both design and capabilities. This parity may be due to higher-quality training data used for LaMDA, particularly the large dialogue training dataset inherited from Meena.
LaMDA is one of the standard testbeds for Google scaling research and examining the many surprising capabilities scaled-up models turn out to have, and many papers have been published about it. Mysteriously, Googlers were not allowed to name LaMDA in those papers, or even to confirm or deny whether it is LaMDA when asked; instead, the early papers vaguely alluded to a series of large Transformers (eg. “we used pre-trained dense decoder-only Transformer language models, ranging in size from 2 million to 137 billion parameters. These models were pre-trained on web documents and dialog data”), leading to confusion.
This index collates LaMDA papers: typically, if a Google paper uses a model size <20b, then it is probably a T5 bidirectional Transformer; >200b-parameters, it is actually a mixture-of-experts model (eg. Switch); if a >150b-parameter model is specified to be dense, then it may be a different model like DeepMind’s 280b-parameter Gopher.
See Also
Links
“Google Is Asking Employees to Test Potential ChatGPT Competitors, including a Chatbot Called 'Apprentice Bard'”, Elias 2023
“Creative Writing With Wordcraft, an AI-Powered Writing Assistant: Perspectives from Professional Writers”, Ippolito et al 2022
“Language Model Cascades”, Dohan et al 2022
“Exploring Length Generalization in Large Language Models”, Anil et al 2022
“Least-to-Most Prompting Enables Complex Reasoning in Large Language Models”, Zhou et al 2022
“Least-to-Most Prompting Enables Complex Reasoning in Large Language Models”
“Google Is Beta Testing Its AI Future: After Mistakes and Challenges, the Company Is Moving a Little Slower With AI Language Models”, Vincent 2022
“PaLM: Scaling Language Modeling With Pathways”, Chowdhery et al 2022
“Self-Consistency Improves Chain-of-Thought Reasoning in Language Models”, Wang et al 2022
“Self-Consistency Improves Chain-of-Thought Reasoning in Language Models”
“PromptChainer: Chaining Large Language Model Prompts through Visual Programming”, Wu et al 2022
“PromptChainer: Chaining Large Language Model Prompts through Visual Programming”
“Using Natural Language Prompts for Machine Translation”, Garcia & Firat 2022
“Chain-of-Thought Prompting Elicits Reasoning in Large Language Models”, Wei et al 2022
“Chain-of-Thought Prompting Elicits Reasoning in Large Language Models”
“LaMDA: Language Models for Dialog Applications”, Thoppilan et al 2022
“SynthBio: A Case Study in Faster Curation of Text Datasets”, Yuan et al 2022
“SynthBio: A Case Study in Faster Curation of Text Datasets”
“Discovering the Syntax and Strategies of Natural Language Programming With Generative Language Models”, Jiang et al 2022
“GLaM: Efficient Scaling of Language Models With Mixture-of-Experts”, Du et al 2021
“GLaM: Efficient Scaling of Language Models with Mixture-of-Experts”
“Show Your Work: Scratchpads for Intermediate Computation With Language Models”, Nye et al 2021
“Show Your Work: Scratchpads for Intermediate Computation with Language Models”
“AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts”, Wu et al 2021
“A Recipe For Arbitrary Text Style Transfer With Large Language Models”, Reif et al 2021
“A Recipe For Arbitrary Text Style Transfer with Large Language Models”
“GenLine and GenForm: Two Tools for Interacting With Generative Language Models in a Code Editor”, Jiang et al 2021b
“GenLine and GenForm: Two Tools for Interacting with Generative Language Models in a Code Editor”
“FLAN: Finetuned Language Models Are Zero-Shot Learners”, Wei et al 2021
“Program Synthesis With Large Language Models”, Austin et al 2021
“Towards a Human-like Open-Domain Chatbot”, Adiwardana et al 2020
“LaMDA: Our Breakthrough Conversation Technology”
Sort By Magic
Annotations sorted by machine learning into inferred 'tags'. This provides an alternative way to browse: instead of by date order, one can browse in topic order. The 'sorted' list has been automatically clustered into multiple sections & auto-labeled for easier browsing.
Beginning with the newest annotation, it uses the embedding of each annotation to attempt to create a list of nearest-neighbor annotations, creating a progression of topics. For more details, see the link.
chatbot
coding-lm
prompt-chain
language-models
Miscellaneous
-
https://blog.google/technology/ai/bard-google-ai-search-updates/ -
https://medium.com/@blaisea/do-large-language-models-understand-us-6f881d6d8e75 -
https://wordcraft-writers-workshop.appspot.com/stories/allison-parrish -
https://wordcraft-writers-workshop.appspot.com/stories/diana-hamilton -
https://wordcraft-writers-workshop.appspot.com/stories/eugenia-triantafyllou -
https://www.technologyreview.com/2021/05/20/1025135/ai-large-language-models-bigscience-project/
Link Bibliography
-
https://www.cnbc.com/2023/01/31/google-testing-chatgpt-like-chatbot-apprentice-bard-with-employees.html: “Google Is Asking Employees to Test Potential ChatGPT Competitors, including a Chatbot Called 'Apprentice Bard'”, Jennifer Elias -
https://arxiv.org/abs/2211.05030#google: “Creative Writing With Wordcraft, an AI-Powered Writing Assistant: Perspectives from Professional Writers”, Daphne Ippolito, Ann Yuan, Andy Coenen, Sehmon Burnam -
https://arxiv.org/abs/2205.10625#google: “Least-to-Most Prompting Enables Complex Reasoning in Large Language Models”, -
https://www.theverge.com/2022/5/11/23065072/google-ai-app-test-kitchen-future-io-2022: “Google Is Beta Testing Its AI Future: After Mistakes and Challenges, the Company Is Moving a Little Slower With AI Language Models”, James Vincent -
https://arxiv.org/abs/2204.02311#google: “PaLM: Scaling Language Modeling With Pathways”, -
https://arxiv.org/abs/2203.11171#google: “Self-Consistency Improves Chain-of-Thought Reasoning in Language Models”, Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Denny Zhou -
https://arxiv.org/abs/2202.11822#google: “Using Natural Language Prompts for Machine Translation”, Xavier Garcia, Orhan Firat -
https://arxiv.org/abs/2201.11903#google: “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models”, Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, Denny Zhou -
2021-jiang-2.pdf: “GenLine and GenForm: Two Tools for Interacting With Generative Language Models in a Code Editor”, Ellen Jiang, Edwin Toh, Alejandra Molina, Aaron Donsbach, Carrie Cai, Michael Terry