- See Also
-
Links
- “Google Is Asking Employees to Test Potential ChatGPT Competitors, including a Chatbot Called ‘Apprentice Bard’”, 2023
- “Creative Writing With Wordcraft, an AI-Powered Writing Assistant: Perspectives from Professional Writers”, Et Al 2022
- “Language Model Cascades”, Et Al 2022
- “Exploring Length Generalization in Large Language Models”, Et Al 2022
- “Least-to-Most Prompting Enables Complex Reasoning in Large Language Models”, Et Al 2022
- “Google Is Beta Testing Its AI Future: After Mistakes and Challenges, the Company Is Moving a Little Slower With AI Language Models”, 2022
- “PaLM: Scaling Language Modeling With Pathways”, Et Al 2022
- “Self-Consistency Improves Chain of Thought Reasoning in Language Models”, Et Al 2022
- “PromptChainer: Chaining Large Language Model Prompts through Visual Programming”, Et Al 2022
- “Using Natural Language Prompts for Machine Translation”, 2022
- “Chain of Thought Prompting Elicits Reasoning in Large Language Models”, Et Al 2022
- “LaMDA: Language Models for Dialog Applications”, Et Al 2022
- “SynthBio: A Case Study in Faster Curation of Text Datasets”, Et Al 2022
- “Discovering the Syntax and Strategies of Natural Language Programming With Generative Language Models”, Et Al 2022
- “GLaM: Efficient Scaling of Language Models With Mixture-of-Experts”, Et Al 2021
- “Show Your Work: Scratchpads for Intermediate Computation With Language Models”, Et Al 2021
- “AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts”, Et Al 2021
- “A Recipe For Arbitrary Text Style Transfer With Large Language Models”, Et Al 2021
- “GenLine and GenForm: Two Tools for Interacting With Generative Language Models in a Code Editor”, Et Al 2021
- “FLAN: Finetuned Language Models Are Zero-Shot Learners”, Et Al 2021
- “Program Synthesis With Large Language Models”, Et Al 2021
- “Towards a Human-like Open-Domain Chatbot”, Et Al 2020
- “LaMDA: Our Breakthrough Conversation Technology”
- Miscellaneous
- Link Bibliography
Google LaMDA is a large 137b-parameter dense Transformer neural network model, announced by Google in May 2021, as a followup to Meena; it is most similar to OpenAI’s May 2020 GPT-3 (175b) in both design and capabilities. This parity may be due to higher-quality training data used for LaMDA, particularly the large dialogue training dataset inherited from Meena.
LaMDA is one of the standard testbeds for Google scaling research and examining the many surprising capabilities scaled-up models turn out to have, and many papers have been published about it. Mysteriously, Googlers were not allowed to name LaMDA in those papers, or even to confirm or deny whether it is LaMDA when asked; instead, the early papers vaguely alluded to a series of large Transformers (eg. “we used pre-trained dense decoder-only Transformer language models, ranging in size from 2 million to 137 billion parameters. These models were pre-trained on web documents and dialog data”), leading to confusion.
This index collates LaMDA papers: typically, if a Google paper uses a model size <20b, then it is probably a T5 bidirectional Transformer; >200b-parameters, it is actually a mixture-of-experts model (eg. Switch); if a >150b-parameter model is specified to be dense, then it may be a different model like DeepMind’s 280b-parameter Gopher.
See Also
Links
“Google Is Asking Employees to Test Potential ChatGPT Competitors, including a Chatbot Called ‘Apprentice Bard’”, 2023
“Google is asking employees to test potential ChatGPT competitors, including a chatbot called 'Apprentice Bard'”, 2023-01-31 ( ; similar; bibliography)
“Creative Writing With Wordcraft, an AI-Powered Writing Assistant: Perspectives from Professional Writers”, Et Al 2022
“Creative Writing with Wordcraft, an AI-Powered Writing Assistant: Perspectives from Professional Writers”, 2022-11-09 ( ; similar; bibliography)
“Language Model Cascades”, Et Al 2022
“Language Model Cascades”, 2022-07-21 ( ; similar)
“Exploring Length Generalization in Large Language Models”, Et Al 2022
“Exploring Length Generalization in Large Language Models”, 2022-07-11 ( ; similar)
“Least-to-Most Prompting Enables Complex Reasoning in Large Language Models”, Et Al 2022
“Least-to-Most Prompting Enables Complex Reasoning in Large Language Models”, 2022-05-21 ( ; similar; bibliography)
“Google Is Beta Testing Its AI Future: After Mistakes and Challenges, the Company Is Moving a Little Slower With AI Language Models”, 2022
“Google is beta testing its AI future: After mistakes and challenges, the company is moving a little slower with AI language models”, 2022-05-11 (similar; bibliography)
“PaLM: Scaling Language Modeling With Pathways”, Et Al 2022
“PaLM: Scaling Language Modeling with Pathways”, 2022-04-05 ( ; similar; bibliography)
“Self-Consistency Improves Chain of Thought Reasoning in Language Models”, Et Al 2022
“Self-Consistency Improves Chain of Thought Reasoning in Language Models”, 2022-03-21 ( ; similar; bibliography)
“PromptChainer: Chaining Large Language Model Prompts through Visual Programming”, Et Al 2022
“PromptChainer: Chaining Large Language Model Prompts through Visual Programming”, 2022-03-13 ( ; similar)
“Using Natural Language Prompts for Machine Translation”, 2022
“Using natural language prompts for machine translation”, 2022-02-23 ( ; similar; bibliography)
“Chain of Thought Prompting Elicits Reasoning in Large Language Models”, Et Al 2022
“Chain of Thought Prompting Elicits Reasoning in Large Language Models”, 2022-01-28 ( ; similar; bibliography)
“LaMDA: Language Models for Dialog Applications”, Et Al 2022
“LaMDA: Language Models for Dialog Applications”, 2022-01-20 ( ; similar)
“SynthBio: A Case Study in Faster Curation of Text Datasets”, Et Al 2022
“SynthBio: A Case Study in Faster Curation of Text Datasets”, 2022-01-13 ( ; similar)
“Discovering the Syntax and Strategies of Natural Language Programming With Generative Language Models”, Et Al 2022
“Discovering the Syntax and Strategies of Natural Language Programming with Generative Language Models”, 2022-01-06 ( ; backlinks; similar)
“GLaM: Efficient Scaling of Language Models With Mixture-of-Experts”, Et Al 2021
“GLaM: Efficient Scaling of Language Models with Mixture-of-Experts”, 2021-12-13 ( ; similar)
“Show Your Work: Scratchpads for Intermediate Computation With Language Models”, Et Al 2021
“Show Your Work: Scratchpads for Intermediate Computation with Language Models”, 2021-10-05 ( ; similar)
“AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts”, Et Al 2021
“AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts”, 2021-10-04 ( ; similar)
“A Recipe For Arbitrary Text Style Transfer With Large Language Models”, Et Al 2021
“A Recipe For Arbitrary Text Style Transfer with Large Language Models”, 2021-09-08 ( ; similar)
“GenLine and GenForm: Two Tools for Interacting With Generative Language Models in a Code Editor”, Et Al 2021
“GenLine and GenForm: Two Tools for Interacting with Generative Language Models in a Code Editor”, 2021-09-07 ( ; similar; bibliography)
“FLAN: Finetuned Language Models Are Zero-Shot Learners”, Et Al 2021
“FLAN: Finetuned Language Models Are Zero-Shot Learners”, 2021-09-03 ( ; similar)
“Program Synthesis With Large Language Models”, Et Al 2021
“Program Synthesis with Large Language Models”, 2021-08-16 ( ; similar)
“Towards a Human-like Open-Domain Chatbot”, Et Al 2020
“Towards a Human-like Open-Domain Chatbot”, 2020-01-27 ( ; similar)
“LaMDA: Our Breakthrough Conversation Technology”
Miscellaneous
Link Bibliography
-
https://www.cnbc.com/2023/01/31/google-testing-chatgpt-like-chatbot-apprentice-bard-with-employees.html
: “Google Is Asking Employees to Test Potential ChatGPT Competitors, including a Chatbot Called 'Apprentice Bard'”, Jennifer Elias: -
https://arxiv.org/abs/2211.05030#google
: “Creative Writing With Wordcraft, an AI-Powered Writing Assistant: Perspectives from Professional Writers”, Daphne Ippolito, Ann Yuan, Andy Coenen, Sehmon Burnam: -
https://arxiv.org/abs/2205.10625#google
: “Least-to-Most Prompting Enables Complex Reasoning in Large Language Models”, : -
https://www.theverge.com/2022/5/11/23065072/google-ai-app-test-kitchen-future-io-2022
: “Google Is Beta Testing Its AI Future: After Mistakes and Challenges, the Company Is Moving a Little Slower With AI Language Models”, James Vincent: -
https://arxiv.org/abs/2204.02311#google
: “PaLM: Scaling Language Modeling With Pathways”, : -
https://arxiv.org/abs/2203.11171#google
: “Self-Consistency Improves Chain of Thought Reasoning in Language Models”, Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Denny Zhou: -
https://arxiv.org/abs/2202.11822#google
: “Using Natural Language Prompts for Machine Translation”, Xavier Garcia, Orhan Firat: -
https://arxiv.org/abs/2201.11903#google
: “Chain of Thought Prompting Elicits Reasoning in Large Language Models”, Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, Denny Zhou: -
2021-jiang.pdf
: “GenLine and GenForm: Two Tools for Interacting With Generative Language Models in a Code Editor”, Ellen Jiang, Edwin Toh, Alejandra Molina, Aaron Donsbach, Carrie Cai, Michael Terry: