--- title: '‘AI hardware’ directory' description: "Bibliography for directory ai/scaling/hardware, most recent first: 1 related tag, 295 annotations, & 99 links ()." thumbnail: /doc/ai/scaling/hardware/2021-jouppi-table1-keycharacteristicsoftpus.png thumbnail-text: '' thumbnail-css: "outline" created: 2019-08-31 modified: 2026-09-26 status: in progress previous: /doc/ai/scaling/emergence/grokking/index next: /doc/ai/scaling/mixture-of-experts/index confidence: log importance: 0 css-extension: dropcaps-not index: True backlink: False ... # See Also ::: {#see-alsos .directory-indexes .columns} - [Parent ('*`AI scaling`{=html}*' tag)](/doc/ai/scaling/index "Link to parent directory 'doc/ai/scaling/' (ascending)"){#_bEPR9TH8 .link-annotated .link-tag .directory-indexes-upwards rel="tag"} - [*`low-precision NN`{=html}*](/doc/ai/nn/sparsity/low-precision/index "‘low-precision NN’ directory"){#_Jey1dXgh .link-annotated .link-tag .directory-indexes-sideways rel="tag"} ::: # Gwern {.display-pop-not} ## `“GPU World”, Gwern et al 2026`{=html} {#gwern-et-al-2026-section} **[`GPU World`{=html}](https://www.gpuworld.org/ "'GPU World', Gwern et al 2026"){.link-modified-recently .link-annotated-partial .id-not .include-annotation}** ## `“LLMs Can Be Faster Than You Think”, Gwern 2025`{=html} {#gwern-2025-llms-can-be-faster-section} **[`LLMs Can Be Faster Than You Think`{=html}](https://www.reddit.com/r/mlscaling/comments/1ik3401/mistral_offers_deepseek_r1_llama70b_at_1500/mbjb9t4/ "'LLMs Can Be Faster Than You Think', Gwern 2025"){.link-annotated .id-not .include-annotation link-icon="deepseek" link-icon-type="svg" link-icon-color="#4d6bfe"}** ## `“Hardware Hedging Scaling Risks”, Gwern 2024`{=html} {#gwern-2024-hardware-hedging-section} **[`Hardware hedging scaling risks`{=html}](https://www.reddit.com/r/mlscaling/comments/1eyophn/hardware_hedging_against_scaling_regime_shifts/ "'Hardware hedging scaling risks', Gwern 2024"){.link-annotated .id-not .include-annotation link-icon="reddit" link-icon-type="svg" link-icon-color="#ff4500"}** ## `“Computer Optimization: Your Computer Is Faster Than You Think”, Gwern 2021`{=html} {#gwern-note-faster-section} **[`Computer Optimization: Your Computer Is Faster Than You Think`{=html}](/note/faster "'Computer Optimization: Your Computer Is Faster Than You Think', Gwern 2021"){.link-annotated-partial .id-not .include-annotation}** ## `“Slowing Moore’s Law: How It Could Happen”, Gwern 2012`{=html} {#gwern-slowing-moores-law-section} **[`Slowing Moore’s Law: How It Could Happen`{=html}](/slowing-moores-law "'Slowing Moore’s Law: How It Could Happen', Gwern 2012"){.link-annotated .id-not .include-annotation}** # Links {.display-pop-not} ## `“DeepSeek Elastic Compute (DSec): A Sandbox Infrastructure for Effective Agentic Training at Scale”, Huang et al 2026`{=html} {#huang-et-al-2026-section} **[`DeepSeek Elastic Compute (DSec): A Sandbox Infrastructure for Effective Agentic Training at Scale`{=html}](https://arxiv.org/abs/2609.22978#deepseek "'DeepSeek Elastic Compute (DSec): A Sandbox Infrastructure for Effective Agentic Training at Scale', Huang et al 2026"){.link-modified-recently .link-annotated .id-not .include-annotation link-icon="deepseek" link-icon-type="svg" link-icon-color="#4d6bfe"}** ## `“Another Potential Headache for US Data Centers—Trump Tariffs: The Possibility of New Duties Is Rattling the Tech Industry, Which Is Already Grappling With a Shortage of High-End Semiconductors”, Hawkins & Miller 2026`{=html} {#hawkins-miller-2026-section} **[`Another potential headache for US data centers—Trump tariffs: The possibility of new duties is rattling the tech industry, which is already grappling with a shortage of high-end semiconductors`{=html}](https://www.politico.com/news/2026/08/27/data-centers-chips-tariffs-threat-01050957 "'Another potential headache for US data centers—Trump tariffs: The possibility of new duties is rattling the tech industry, which is already grappling with a shortage of high-end semiconductors', Hawkins & Miller 2026"){.link-modified-recently .link-annotated-partial .id-not .include-annotation}** ## `“FlashAccel: Leveraging High-Bandwidth Flash (HBF) for High-Throughput LLM Inference”, Wang et al 2026`{=html} {#wang-et-al-2026-section} **[`FlashAccel: Leveraging High-Bandwidth Flash (HBF) for High-Throughput LLM Inference`{=html}](https://arxiv.org/abs/2607.10186 "'FlashAccel: Leveraging High-Bandwidth Flash (HBF) for High-Throughput LLM Inference', Wang et al 2026"){.link-modified-recently .link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“NZ Inc Is Going All in on AI Datacenters: The Government Wants $25–35 Billion in Datacenter Investment, and It Just Gathered Some of the Country’s Most Powerful Business Leaders to Make the Case”, MacManus 2026`{=html} {#macmanus-2026-section} **[`NZ Inc is going all in on AI datacenters: The government wants $25–35 billion in datacenter investment, and it just gathered some of the country’s most powerful business leaders to make the case`{=html}](https://thespinoff.co.nz/business/06-07-2026/nz-inc-is-going-all-in-on-ai-data-centres "'NZ Inc is going all in on AI datacenters: The government wants $25–35 billion in datacenter investment, and it just gathered some of the country’s most powerful business leaders to make the case', MacManus 2026"){.link-annotated-partial .id-not .include-annotation}** ## `“Cerebras: Faster Tokens Please; OpenAI and AWS Partnerships, Tokenomics Explainer, Architecture Deep Dive, Datacenter Ramp, Technical Roadmap”, Xie et al 2026`{=html} {#xie-et-al-2026-section} **[`Cerebras: Faster Tokens Please; OpenAI and AWS Partnerships, Tokenomics Explainer, Architecture Deep Dive, Datacenter Ramp, Technical Roadmap`{=html}](https://newsletter.semianalysis.com/p/cerebras-faster-tokens-please "'Cerebras: Faster Tokens Please; OpenAI and AWS Partnerships, Tokenomics Explainer, Architecture Deep Dive, Datacenter Ramp, Technical Roadmap', Xie et al 2026"){.link-annotated-partial .id-not .include-annotation link-icon="C" link-icon-type="text,sans" link-icon-color="#f05a29"}** ## `“Notes from inside China’s AI Labs: Lessons from My Trip to Talk to Most of the Leading AI Labs in China”, Lambert 2026`{=html} {#lambert-2026-section} **[`Notes from inside China’s AI labs: Lessons from my trip to talk to most of the leading AI labs in China`{=html}](https://www.interconnects.ai/p/notes-from-inside-chinas-ai-labs "'Notes from inside China’s AI labs: Lessons from my trip to talk to most of the leading AI labs in China', Lambert 2026"){.link-annotated-partial .id-not .include-annotation}** ## `“Bit-Flip Vulnerability of Shared KV-Cache Blocks in LLM Serving Systems”, Yamamoto & Matsuura 2026`{=html} {#yamamoto-matsuura-2026-section} **[`Bit-Flip Vulnerability of Shared KV-Cache Blocks in LLM Serving Systems`{=html}](https://arxiv.org/abs/2604.17249 "'Bit-Flip Vulnerability of Shared KV-Cache Blocks in LLM Serving Systems', Yamamoto & Matsuura 2026"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“MegaTrain: Full Precision Training of 100B+ Parameter Large Language Models on a Single GPU”, Yuan et al 2026`{=html} {#yuan-et-al-2026-section} **[`MegaTrain: Full Precision Training of 100B+ Parameter Large Language Models on a Single GPU`{=html}](https://arxiv.org/abs/2604.05091 "'MegaTrain: Full Precision Training of 100B+ Parameter Large Language Models on a Single GPU', Yuan et al 2026"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Soulplayer-C64: A Real 25k-Parameter Transformer Running on a Commodore 64!”, gizmo64k 2026`{=html} {#gizmo64k-2026-section} **[`soulplayer-c64: A real 25k-parameter transformer running on a Commodore 64!`{=html}](https://github.com/gizmo64k/soulplayer-c64 "'soulplayer-c64: A real 25k-parameter transformer running on a Commodore 64!', gizmo64k 2026"){.link-annotated .id-not .include-annotation link-icon="github" link-icon-type="svg"}** ## `“Is Almost Everyone Wrong about America’s AI Power Problem? Why Power Is Less of a Bottleneck Than You Think”, Ho et al 2025`{=html} {#ho-et-al-2025-section} **[`Is almost everyone wrong about America’s AI power problem? Why power is less of a bottleneck than you think`{=html}](https://epochai.substack.com/p/is-almost-everyone-wrong-about-americas "'Is almost everyone wrong about America’s AI power problem? Why power is less of a bottleneck than you think', Ho et al 2025"){.link-annotated-partial .id-not .include-annotation link-icon="substack" link-icon-type="svg" link-icon-color="#ff6719"}** ## `“AI in 2025: Gestalt”, technicalities 2025`{=html} {#technicalities-2025-section} **[`AI in 2025: gestalt`{=html}](https://www.lesswrong.com/posts/Q9ewXs8pQSAX5vL7H/ai-in-2025-gestalt "'AI in 2025: gestalt', technicalities 2025"){.link-annotated-partial .id-not .include-annotation link-icon="LW" link-icon-type="text" link-icon-color="#7faf83"}** ## `“Micron Announces Exit from Crucial™ Consumer RAM Business [In Favor of AI-Only RAM]”, Micron 2025`{=html} {#micron-2025-section} **[`Micron Announces Exit from Crucial™ Consumer RAM Business [in favor of AI-only RAM]`{=html}](https://investors.micron.com/news-releases/news-release-details/micron-announces-exit-crucial-consumer-business "'Micron Announces Exit from Crucial™ Consumer RAM Business [in favor of AI-only RAM]', Micron 2025"){.link-annotated .id-not .include-annotation}** ## `“Silicon Valley’s Man in the White House Is Benefiting Himself and His Friends: David Sacks, the Trump Administration’s AI & Crypto Czar, Has Helped Formulate Policies That Aid His Silicon Valley Friends & Many of His Own Tech Investments”, Kang et al 2025`{=html} {#kang-et-al-2025-section} **[`Silicon Valley’s Man in the White House Is Benefiting Himself and His Friends: David Sacks, the Trump administration’s AI & crypto czar, has helped formulate policies that aid his Silicon Valley friends & many of his own tech investments`{=html}](https://www.nytimes.com/2025/11/30/technology/david-sacks-white-house-profits.html "'Silicon Valley’s Man in the White House Is Benefiting Himself and His Friends: David Sacks, the Trump administration’s AI & crypto czar, has helped formulate policies that aid his Silicon Valley friends & many of his own tech investments', Kang et al 2025"){.link-annotated-partial .id-not .include-annotation link-icon="new-york-times" link-icon-type="svg"}** ## `“Google Must Double AI Serving Capacity Every 6 Months to Meet Demand”, Elias 2025`{=html} {#elias-2025-section} **[`Google must double AI serving capacity every 6 months to meet demand`{=html}](https://www.cnbc.com/2025/11/21/google-must-double-ai-serving-capacity-every-6-months-to-meet-demand.html "'Google must double AI serving capacity every 6 months to meet demand', Elias 2025"){.id-not .include-annotation link-icon="alphabet" link-icon-type="svg" link-icon-color="#4285f4"}** ## `“Expanding Our Use of Google Cloud TPUs and Services [>1m TPUs, >1GW]”, Anthropic 2025`{=html} {#anthropic-2025-section} **[`Expanding our use of Google Cloud TPUs and Services [>1m TPUs, >1GW]`{=html}](https://www.anthropic.com/news/expanding-our-use-of-google-cloud-tpus-and-services "'Expanding our use of Google Cloud TPUs and Services [>1m TPUs, >1GW]', Anthropic 2025"){.link-annotated-partial .id-not .include-annotation link-icon="alphabet" link-icon-type="svg" link-icon-color="#4285f4"}** ## `“I Talked to Sam Altman about the GPT-5 Launch Fiasco: Over Dinner, OpenAI CEO’s Addressed Criticism of GPT-5’s Rollout, the AI Bubble, Brain-Computer Interfaces, Buying Google Chrome, and More”, Heath 2025`{=html} {#heath-2025-section} **[`I talked to Sam Altman about the GPT-5 launch fiasco: Over dinner, OpenAI CEO’s addressed criticism of GPT-5’s rollout, the AI bubble, brain-computer interfaces, buying Google Chrome, and more`{=html}](https://www.theverge.com/command-line-newsletter/759897/sam-altman-chatgpt-openai-social-media-google-chrome-interview "'I talked to Sam Altman about the GPT-5 launch fiasco: Over dinner, OpenAI CEO’s addressed criticism of GPT-5’s rollout, the AI bubble, brain-computer interfaces, buying Google Chrome, and more', Heath 2025"){.link-annotated .id-not .include-annotation link-icon="alphabet" link-icon-type="svg" link-icon-color="#4285f4"}** ## `“Nvidia and AMD to Pay 15% of China Chip Sale Revenues to US Government: Chipmakers Agree to Unusual Arrangement to Secure Export Licences from Trump Administration”, Sevastopulo & Acton 2025`{=html} {#sevastopulo-acton-2025-section} **[`Nvidia and AMD to pay 15% of China chip sale revenues to US government: Chipmakers agree to unusual arrangement to secure export licences from Trump administration`{=html}](https://www.ft.com/content/cd1a0729-a8ab-41e1-a4d2-8907f4c01cac "'Nvidia and AMD to pay 15% of China chip sale revenues to US government: Chipmakers agree to unusual arrangement to secure export licences from Trump administration', Sevastopulo & Acton 2025"){.id-not .include-annotation link-icon="FT" link-icon-type="text" link-icon-color="#e3b68e"}** ::: aux-links-transclude-file ::: collapse **View External Link**: [`https://www.ft.com/content/cd1a0729-a8ab-41e1-a4d2-8907f4c01cac`](https://www.ft.com/content/cd1a0729-a8ab-41e1-a4d2-8907f4c01cac){.id-not .link-annotated-not .include-content .include-lazy link-icon="FT" link-icon-type="text" link-icon-color="#e3b68e"} ::: ::: ## `khoomeik @ "2025-08-07"`{=html} {#pandey-2025-section} **[`[GPT-5 was a <100× GPT-4 scaleup]`{=html}](https://x.com/khoomeik/status/1953560406381015259 "'[GPT-5 was a <100× GPT-4 scaleup]', Pandey 2025"){.link-annotated .id-not .include-annotation link-icon="twitter" link-icon-type="svg" link-icon-color="#1da1f2"}** ## `“Nvidia Chips Become the First GPUs to Fall to Rowhammer Bit-Flip Attacks: GPUhammer Is the First to Flip Bits in Onboard GPU Memory. It Likely Won’t Be the Last”, Goodin 2025`{=html} {#goodin-2025-section} **[`Nvidia chips become the first GPUs to fall to Rowhammer bit-flip attacks: GPUhammer is the first to flip bits in onboard GPU memory. It likely won’t be the last`{=html}](https://arstechnica.com/security/2025/07/nvidia-chips-become-the-first-gpus-to-fall-to-rowhammer-bit-flip-attacks/ "'Nvidia chips become the first GPUs to fall to Rowhammer bit-flip attacks: GPUhammer is the first to flip bits in onboard GPU memory. It likely won’t be the last', Goodin 2025"){.id-not .include-annotation link-icon="n" link-icon-type="text,sans,italic" link-icon-color="#77ba00"}** ## `“DeepSeek R2 Launch Stalled As CEO Balks at Progress, The Information Reports”, Sophia 2025`{=html} {#sophia-2025-section} **[`DeepSeek R2 launch stalled as CEO balks at progress, The Information reports`{=html}](https://www.reuters.com/world/china/deepseek-r2-launch-stalled-ceo-balks-progress-information-reports-2025-06-26/ "'DeepSeek R2 launch stalled as CEO balks at progress, The Information reports', Sophia 2025"){.link-annotated .id-not .include-annotation link-icon="deepseek" link-icon-type="svg" link-icon-color="#4d6bfe"}** ## `“Musings on AI Companies of 2025–2026 (June 2025)”, Vladimir_Nesov 2025`{=html} {#vladimir_nesov-2025-section} **[`Musings on AI Companies of 2025–2026 (June 2025)`{=html}](https://www.lesswrong.com/posts/9rKWm8BzTYAiCCFjx/musings-on-ai-companies-of-2025-2026-jun-2025 "'Musings on AI Companies of 2025–2026 (June 2025)', Vladimir_Nesov 2025"){.link-annotated-partial .id-not .include-annotation link-icon="LW" link-icon-type="text" link-icon-color="#7faf83"}** ## `“Inference Economics of Language Models”, Erdil 2025`{=html} {#erdil-2025-section} **[`Inference economics of language models`{=html}](https://arxiv.org/abs/2506.04645#epoch "'Inference economics of language models', Erdil 2025"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Estimating the Substitutability between Compute and Cognitive Labor in AI Research”`{=html} {#_Qmbk1Y0n-section} **[`Estimating the Substitutability between Compute and Cognitive Labor in AI Research`{=html}](https://forum.effectivealtruism.org/posts/xoX936hEvpxToeuLw/estimating-the-substitutability-between-compute-and "Estimating the Substitutability between Compute and Cognitive Labor in AI Research"){.link-annotated-partial .id-not .include-annotation link-icon="EA" link-icon-type="text" link-icon-color="#06819a"}** ## `“America Makes AI Chip Diffusion Deal With UAE and KSA”, Mowshowitz 2025`{=html} {#mowshowitz-2025-section} **[`America Makes AI Chip Diffusion Deal with UAE and KSA`{=html}](https://thezvi.wordpress.com/2025/05/19/america-makes-ai-chip-diffusion-deal-with-uae-and-ksa/ "'America Makes AI Chip Diffusion Deal with UAE and KSA', Mowshowitz 2025"){.link-annotated-partial .id-not .include-annotation}** ## `“AI Arrives In The Middle East: US Strikes A Deal With UAE and KSA; 5 GW Datacenter, HUMAIN, G42, Diversion and Misuse Risks, Security Requirements, American AI Wins”, Patel et al 2025`{=html} {#patel-et-al-2025-section} **[`AI Arrives In The Middle East: US Strikes A Deal with UAE and KSA; 5 GW Datacenter, HUMAIN, G42, Diversion and Misuse Risks, Security Requirements, American AI Wins`{=html}](https://semianalysis.com/2025/05/16/ai-arrives-in-the-middle-east-us-strikes-a-deal-with-uae-and-ksa/ "'AI Arrives In The Middle East: US Strikes A Deal with UAE and KSA; 5 GW Datacenter, HUMAIN, G42, Diversion and Misuse Risks, Security Requirements, American AI Wins', Patel et al 2025"){.link-annotated-partial .id-not .include-annotation}** ## `“UAE and US Presidents Attend the Unveiling of Phase 1 of New 5GW AI Campus in Abu Dhabi”, Commerce 2025`{=html} {#commerce-2025-section} **[`UAE and US Presidents attend the unveiling of Phase 1 of new 5GW AI campus in Abu Dhabi`{=html}](https://www.commerce.gov/news/press-releases/2025/05/uae-and-us-presidents-attend-unveiling-phase-1-new-5gw-ai-campus-abu "'UAE and US Presidents attend the unveiling of Phase 1 of new 5GW AI campus in Abu Dhabi', Commerce 2025"){.link-annotated-partial .id-not .include-annotation}** ## `“Outsourcer in Chief: Is Trump Trading Away America’s Tech Future? Big Deals to Sell Chips to the UAE and Saudi Arabia Have Divided the US Government over Whether They Could Be Remembered for Shipping Cutting-Edge AI Overseas”, Mickle & Swanson 2025`{=html} {#mickle-swanson-2025-section} **[`Outsourcer in Chief: Is Trump Trading Away America’s Tech Future? Big deals to sell chips to the UAE and Saudi Arabia have divided the US government over whether they could be remembered for shipping cutting-edge AI overseas`{=html}](https://www.nytimes.com/2025/05/15/business/economy/trump-chips-ai-uae.html "'Outsourcer in Chief: Is Trump Trading Away America’s Tech Future? Big deals to sell chips to the UAE and Saudi Arabia have divided the US government over whether they could be remembered for shipping cutting-edge AI overseas', Mickle & Swanson 2025"){.link-annotated-partial .id-not .include-annotation link-icon="new-york-times" link-icon-type="svg"}** ## `“Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures”, Zhao et al 2025`{=html} {#zhao-et-al-2025-section} **[`Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures`{=html}](https://arxiv.org/abs/2505.09343#deepseek "'Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures', Zhao et al 2025"){.link-annotated .id-not .include-annotation link-icon="deepseek" link-icon-type="svg" link-icon-color="#4d6bfe"}** ## `“Pangu Ultra MoE: How to Train Your Big MoE on Ascend NPUs”, Tang et al 2025`{=html} {#tang-et-al-2025-section} **[`Pangu Ultra MoE: How to Train Your Big MoE on Ascend NPUs`{=html}](https://arxiv.org/abs/2505.04519#huawei "'Pangu Ultra MoE: How to Train Your Big MoE on Ascend NPUs', Tang et al 2025"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Alphabet, Nvidia Invest in OpenAI Co-Founder Sutskever’s SSI, Source Says”, Cai & Hu 2025`{=html} {#cai-hu-2025-section} **[`Alphabet, Nvidia invest in OpenAI co-founder Sutskever’s SSI, source says`{=html}](https://www.reuters.com/technology/artificial-intelligence/alphabet-nvidia-invest-openai-co-founder-sutskevers-ssi-source-says-2025-04-12/ "'Alphabet, Nvidia invest in OpenAI co-founder Sutskever’s SSI, source says', Cai & Hu 2025"){.link-annotated .id-not .include-annotation link-icon="n" link-icon-type="text,sans,italic" link-icon-color="#77ba00"}** ## `“What’s Wrong With Apple? Even Before the Threat of President Trump’s Tariffs, There Were Questions about the Company’s Inability to Make Good on New Ideas”, Mickle 2025`{=html} {#mickle-2025-section} **[`What’s Wrong With Apple? Even before the threat of President Trump’s tariffs, there were questions about the company’s inability to make good on new ideas`{=html}](https://www.nytimes.com/2025/04/11/technology/apple-issues-trump-tariffs.html "'What’s Wrong With Apple? Even before the threat of President Trump’s tariffs, there were questions about the company’s inability to make good on new ideas', Mickle 2025"){.link-annotated .id-not .include-annotation link-icon="new-york-times" link-icon-type="svg"}** ## `“Pre-Training GPT-4.5”, Tootoonchian et al 2025`{=html} {#tootoonchian-et-al-2025-section} **[`Pre-Training GPT-4.5`{=html}](https://www.youtube.com/watch?v=6nJZopACRuQ "'Pre-Training GPT-4.5', Tootoonchian et al 2025"){.id-not .include-annotation link-icon="youtube" link-icon-type="svg" link-icon-color="#ff0033"}** ::: aux-links-transclude-file [`https://www.youtube.com/watch?v=6nJZopACRuQ`](https://www.youtube.com/watch?v=6nJZopACRuQ){.link-annotated-not .include-content .width-full} ::: ## `“OpenAI’s First Stargate Site to Hold Up to 400,000 Nvidia Chips”, Ford et al 2025`{=html} {#ford-et-al-2025-section} **[`OpenAI’s First Stargate Site to Hold Up to 400,000 Nvidia Chips`{=html}](https://www.bloomberg.com/news/articles/2025-03-18/openai-s-first-stargate-site-to-hold-up-to-400-000-nvidia-chips "'OpenAI’s First Stargate Site to Hold Up to 400,000 Nvidia Chips', Ford et al 2025"){.id-not .include-annotation link-icon="n" link-icon-type="text,sans,italic" link-icon-color="#77ba00"}** ## `“Measuring Automated Kernel Engineering”`{=html} {#_AZljyvfL-section} **[`Measuring Automated Kernel Engineering`{=html}](https://metr.org/blog/2025-02-14-measuring-automated-kernel-engineering/ "Measuring Automated Kernel Engineering"){.link-annotated-partial .id-not .include-annotation}** ## `“How To Scale Your Model on TPUs”, Austin et al 2025`{=html} {#austin-et-al-2025-section} **[`How To Scale Your Model on TPUs`{=html}](https://jax-ml.github.io/scaling-book/#google "'How To Scale Your Model on TPUs', Austin et al 2025"){.id-not .include-annotation link-icon="alphabet" link-icon-type="svg" link-icon-color="#4285f4"}** ::: aux-links-transclude-file ::: collapse **View HTML**: [`/doc/www/jax-ml.github.io/d2a695dfc9fc630e8d46aa398cfecad580c1da78.html#google`](/doc/www/jax-ml.github.io/d2a695dfc9fc630e8d46aa398cfecad580c1da78.html#google){.id-not .link-annotated-not .include-content .include-lazy link-icon="alphabet" link-icon-type="svg" link-icon-color="#4285f4"} ::: ::: ## `“Learning-By-Doing in the Semiconductor Industry”, Decker 2025`{=html} {#decker-2025-section} **[`Learning-by-doing in the Semiconductor Industry`{=html}](https://nicholasdecker.substack.com/p/learning-by-doing-in-the-semiconductor "'Learning-by-doing in the Semiconductor Industry', Decker 2025"){.id-not .include-annotation link-icon="substack" link-icon-type="svg" link-icon-color="#ff6719"}** ## `“Cerebras Launches World’s Fastest DeepSeek R1 Llama-70B Inference”, Wang 2025`{=html} {#wang-2025-r1-section} **[`Cerebras Launches World’s Fastest DeepSeek R1 Llama-70B Inference`{=html}](https://cerebras.ai/blog/cerebras-launches-worlds-fastest-deepseek-r1-llama-70b-inference "'Cerebras Launches World’s Fastest DeepSeek R1 Llama-70B Inference', Wang 2025"){.link-annotated .id-not .include-annotation link-icon="deepseek" link-icon-type="svg" link-icon-color="#4d6bfe"}** ## `“Neuromorphic Computing at Scale”, Kudithipudi et al 2025`{=html} {#kudithipudi-et-al-2025-section} **[`Neuromorphic computing at scale`{=html}](/doc/ai/scaling/hardware/2025-kudithipudi.pdf "'Neuromorphic computing at scale', Kudithipudi et al 2025"){.link-annotated .id-not .include-annotation link-icon="pdf" link-icon-type="svg" link-icon-color="#f40f02"}** ## `“Good Things Come in Small Packages: Should We Adopt Lite-GPUs in AI Infrastructure?”, Canakci et al 2025`{=html} {#canakci-et-al-2025-section} **[`Good things come in small packages: Should we adopt Lite-GPUs in AI infrastructure?`{=html}](https://arxiv.org/abs/2501.10187#microsoft "'Good things come in small packages: Should we adopt Lite-GPUs in AI infrastructure?', Canakci et al 2025"){.link-annotated .id-not .include-annotation link-icon="MS" link-icon-type="text,sans,italic"}** ## `“How Much Economic Growth from AI Should We Expect, How Soon?”, Wiseman & McClements 2025`{=html} {#wiseman-mcclements-2025-section} **[`How much economic growth from AI should we expect, how soon?`{=html}](https://inferencemagazine.substack.com/p/how-much-economic-growth-from-ai "'How much economic growth from AI should we expect, how soon?', Wiseman & McClements 2025"){.link-annotated-partial .id-not .include-annotation link-icon="substack" link-icon-type="svg" link-icon-color="#ff6719"}** ## `“Why New Zealand Is an Overlooked AI Hyperscaler Opportunity”, Alethios 2025`{=html} {#alethios-2025-section} **[`Why New Zealand is an Overlooked AI Hyperscaler Opportunity`{=html}](https://alethios.substack.com/p/why-new-zealand-is-an-overlooked "'Why New Zealand is an Overlooked AI Hyperscaler Opportunity', Alethios 2025"){.id-not .include-annotation link-icon="substack" link-icon-type="svg" link-icon-color="#ff6719"}** ## `“Christophe Fouquet, CEO ASML: ‘Je Moest Eens Weten Hoeveel Fuck-Ups Er Nodig Zijn Om De Meest Complexe Machine Ter Wereld Te Maken’—NRC”`{=html} {#_m3BenI1E-section} **[`Christophe Fouquet, CEO ASML: ‘Je moest eens weten hoeveel fuck-ups er nodig zijn om de meest complexe machine ter wereld te maken’—NRC`{=html}](https://www.nrc.nl/nieuws/2024/12/18/christophe-fouquet-ceo-asml-je-moest-eens-weten-hoeveel-fuck-ups-er-nodig-zijn-om-de-meest-complexe-machine-ter-wereld-te-maken-a4877089 "Christophe Fouquet, CEO ASML: ‘Je moest eens weten hoeveel fuck-ups er nodig zijn om de meest complexe machine ter wereld te maken’—NRC"){.link-annotated-partial .id-not .include-annotation}** ## `“Fast, Scalable, Clean, and Cheap Enough: How Off-Grid Solar Microgrids Can Power the AI Race”, Baranko et al 2024`{=html} {#baranko-et-al-2024-section} **[`Fast, scalable, clean, and cheap enough: How off-grid solar microgrids can power the AI race`{=html}](https://www.offgridai.us/ "'Fast, scalable, clean, and cheap enough: How off-grid solar microgrids can power the AI race', Baranko et al 2024"){.id-not .include-annotation}** ## `“Why a US AI "Manhattan Project" Could Backfire: Notes from Conversations in China”, Todd 2024`{=html} {#todd-2024-section} **[`Why a US AI "Manhattan Project" could backfire: notes from conversations in China`{=html}](https://benjamintodd.substack.com/p/why-a-us-ai-manhattan-project-could "'Why a US AI "Manhattan Project" could backfire: notes from conversations in China', Todd 2024"){.link-annotated .id-not .include-annotation link-icon="substack" link-icon-type="svg" link-icon-color="#ff6719"}** ## `“Getting AI Datacenters in the UK: Why the UK Needs to Create Special Compute Zones; and How to Do It”, Wiseman et al 2024`{=html} {#wiseman-et-al-2024-section} **[`Getting AI datacenters in the UK: Why the UK needs to create Special Compute Zones; and how to do it`{=html}](https://inferencemagazine.substack.com/p/getting-ai-datacentres-in-the-uk "'Getting AI datacenters in the UK: Why the UK needs to create Special Compute Zones; and how to do it', Wiseman et al 2024"){.link-annotated-partial .id-not .include-annotation link-icon="substack" link-icon-type="svg" link-icon-color="#ff6719"}** ## `“The Future of Compute: Nvidia’s Crown Is Slipping”, Dagarwal 2024`{=html} {#dagarwal-2024-section} **[`The Future of Compute: Nvidia’s Crown is Slipping`{=html}](https://mohitdagarwal.substack.com/p/from-dominance-to-dilemma-nvidia "'The Future of Compute: Nvidia’s Crown is Slipping', Dagarwal 2024"){.id-not .include-annotation link-icon="n" link-icon-type="text,sans,italic" link-icon-color="#77ba00"}** ## `“Jake Sullivan: The American Who Waged a Tech War on China”`{=html} {#_yq48Ye1l-section} **[`Jake Sullivan: The American Who Waged a Tech War on China`{=html}](https://www.wired.com/story/jake-sullivan-china-tech-profile/ "Jake Sullivan: The American Who Waged a Tech War on China"){.id-not .include-annotation link-icon="wired" link-icon-type="svg"}** ::: aux-links-transclude-file ::: collapse **View External Link**: [`https://www.wired.com/story/jake-sullivan-china-tech-profile/`](https://www.wired.com/story/jake-sullivan-china-tech-profile/){.id-not .link-annotated-not .include-content .include-lazy link-icon="wired" link-icon-type="svg"} ::: ::: ## `“Nvidia’s AI Chips Are Cheaper to Rent in China Than US: Supply of Processors Helps Chinese Start-Ups Advance Artificial Intelligence Technology despite Washington’s Restrictions”, McMorrow & Olcott 2024`{=html} {#mcmorrow-olcott-2024-section} **[`Nvidia’s AI chips are cheaper to rent in China than US: Supply of processors helps Chinese start-ups advance artificial intelligence technology despite Washington’s restrictions`{=html}](https://www.ft.com/content/10aacfa3-e966-4b50-bbee-66e13560deb4 "'Nvidia’s AI chips are cheaper to rent in China than US: Supply of processors helps Chinese start-ups advance artificial intelligence technology despite Washington’s restrictions', McMorrow & Olcott 2024"){.link-annotated .id-not .include-annotation link-icon="FT" link-icon-type="text" link-icon-color="#e3b68e"}** ## `“Benchmarking the Performance of Large Language Models on the Cerebras Wafer Scale Engine”, Zhang et al 2024`{=html} {#zhang-et-al-2024-section} **[`Benchmarking the Performance of Large Language Models on the Cerebras Wafer Scale Engine`{=html}](https://arxiv.org/abs/2409.00287#cerebras "'Benchmarking the Performance of Large Language Models on the Cerebras Wafer Scale Engine', Zhang et al 2024"){.link-annotated .id-not .include-annotation link-icon="C" link-icon-type="text,sans" link-icon-color="#f05a29"}** ## `“Chips or Not, Chinese AI Pushes Ahead: A Host of Chinese AI Startups Are Attempting to Write More Efficient Code for Large Language Models”, Kao & Huang 2024`{=html} {#kao-huang-2024-section} **[`Chips or Not, Chinese AI Pushes Ahead: A host of Chinese AI startups are attempting to write more efficient code for large language models`{=html}](https://www.wsj.com/tech/ai/chips-or-not-chinese-ai-pushes-ahead-31034e3d "'Chips or Not, Chinese AI Pushes Ahead: A host of Chinese AI startups are attempting to write more efficient code for large language models', Kao & Huang 2024"){.link-annotated .id-not .include-annotation link-icon="WSJ" link-icon-type="text,tri"}** ## `“Can AI Scaling Continue Through 2030?”, Sevilla et al 2024`{=html} {#sevilla-et-al-2024-section} **[`Can AI Scaling Continue Through 2030?`{=html}](https://epochai.org/blog/can-ai-scaling-continue-through-2030 "'Can AI Scaling Continue Through 2030?', Sevilla et al 2024"){.link-annotated-partial .id-not .include-annotation}** ## `“UK Government Shelves £1.3bn UK Tech and AI Plans”`{=html} {#_hV6l_Q4u-section} **[`UK Government shelves £1.3bn UK tech and AI plans`{=html}](https://www.bbc.com/news/articles/cyx5x44vnyeo "UK Government shelves £1.3bn UK tech and AI plans"){.id-not .include-annotation link-icon="BBC" link-icon-type="text,tri,sans"}** ## `“OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Training”, Jaghouar et al 2024`{=html} {#jaghouar-et-al-2024-section} **[`OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Training`{=html}](https://arxiv.org/abs/2407.07852 "'OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Training', Jaghouar et al 2024"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Huawei Faces Production Challenges With 20% Yield Rate for AI Chip”, Trendforce 2024`{=html} {#trendforce-2024-section} **[`Huawei Faces Production Challenges with 20% Yield Rate for AI Chip`{=html}](https://www.trendforce.com/news/2024/06/28/news-huawei-faces-production-challenges-with-20-yield-rate-for-ai-chip/ "'Huawei Faces Production Challenges with 20% Yield Rate for AI Chip', Trendforce 2024"){.link-annotated .id-not .include-annotation}** ## `“RAM Is Practically Endless Now”, fxtentacles 2024`{=html} {#fxtentacles-2024-section} **[`RAM is practically endless now`{=html}](https://news.ycombinator.com/item?id=40655567 "'RAM is practically endless now', fxtentacles 2024"){.link-annotated .id-not .include-annotation link-icon="hacker-news" link-icon-type="svg" link-icon-color="#f26522"}** ## `“Huawei ‘Unable to Secure 3.5 Nanometer Chips’”, Choi 2024`{=html} {#choi-2024-section} **[`Huawei ‘Unable to Secure 3.5 Nanometer Chips’`{=html}](https://www.businesskorea.co.kr/news/articleView.html?idxno=218699 "'Huawei ‘Unable to Secure 3.5 Nanometer Chips’', Choi 2024"){.link-annotated .id-not .include-annotation}** ## `“China Is Losing the Chip War: Xi Jinping Picked a Fight over Semiconductor Technology—One He Can’t Win”, Schuman 2024`{=html} {#schuman-2024-section} **[`China Is Losing the Chip War: Xi Jinping picked a fight over semiconductor technology—one he can’t win`{=html}](https://www.theatlantic.com/international/archive/2024/06/china-microchip-technology-competition/678612/ "'China Is Losing the Chip War: Xi Jinping picked a fight over semiconductor technology—one he can’t win', Schuman 2024"){.link-annotated .id-not .include-annotation link-icon="A" link-icon-type="text,italic" link-icon-color="#e7131a"}** ## `“Scalable Matmul-Free Language Modeling”, Zhu et al 2024`{=html} {#zhu-et-al-2024-2-section} **[`Scalable Matmul-free Language Modeling`{=html}](https://arxiv.org/abs/2406.02528 "'Scalable Matmul-free Language Modeling', Zhu et al 2024"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Elon Musk Ordered Nvidia to Ship Thousands of AI Chips Reserved for Tesla to Twitter/xAI”, Kolodny 2024`{=html} {#kolodny-2024-section} **[`Elon Musk ordered Nvidia to ship thousands of AI chips reserved for Tesla to Twitter/xAI`{=html}](https://www.cnbc.com/2024/06/04/elon-musk-told-nvidia-to-ship-ai-chips-reserved-for-tesla-to-x-xai.html "'Elon Musk ordered Nvidia to ship thousands of AI chips reserved for Tesla to Twitter/xAI', Kolodny 2024"){.link-annotated .id-not .include-annotation link-icon="n" link-icon-type="text,sans,italic" link-icon-color="#77ba00"}** ## `“Earnings Call: Tesla Discusses Q1 2024 Challenges and AI Expansion”, Abdulkadir 2024`{=html} {#abdulkadir-2024-section} **[`Earnings call: Tesla Discusses Q1 2024 Challenges and AI Expansion`{=html}](https://www.investing.com/news/stock-market-news/earnings-call-tesla-discusses-q1-challenges-and-ai-expansion-93CH-3393955 "'Earnings call: Tesla Discusses Q1 2024 Challenges and AI Expansion', Abdulkadir 2024"){.link-annotated .id-not .include-annotation}** ## `“Microsoft, OpenAI Plan $100 Billion Data-Center Project, Media Report Says”, Reuters 2024`{=html} {#reuters-2024-section} **[`Microsoft, OpenAI plan $100 billion data-center project, media report says`{=html}](https://www.reuters.com/technology/microsoft-openai-planning-100-billion-data-center-project-information-reports-2024-03-29/ "'Microsoft, OpenAI plan $100 billion data-center project, media report says', Reuters 2024"){.link-annotated .id-not .include-annotation link-icon="R" link-icon-type="text,sans" link-icon-color="#e56218"}** ## `“AI and Memory Wall”, Gholami et al 2024`{=html} {#gholami-et-al-2024-section} **[`AI and Memory Wall`{=html}](https://arxiv.org/abs/2403.14123 "'AI and Memory Wall', Gholami et al 2024"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Singapore’s Temasek in Discussions to Invest in OpenAI: State-Backed Group in Talks With ChatGPT Maker’s Chief Sam Altman Who Is Seeking Funding to Build Chips Business”, Murgia & Ruehl 2024`{=html} {#murgia-ruehl-2024-section} **[`Singapore’s Temasek in discussions to invest in OpenAI: State-backed group in talks with ChatGPT maker’s chief Sam Altman who is seeking funding to build chips business`{=html}](https://www.ft.com/content/8e8a65a0-a990-4c77-a6e8-ec4e5d247f80 "'Singapore’s Temasek in discussions to invest in OpenAI: State-backed group in talks with ChatGPT maker’s chief Sam Altman who is seeking funding to build chips business', Murgia & Ruehl 2024"){.link-annotated .id-not .include-annotation link-icon="FT" link-icon-type="text" link-icon-color="#e3b68e"}** ## `“China’s Military and Government Acquire Nvidia Chips despite US Ban”, Baptista 2024`{=html} {#baptista-2024-section} **[`China’s military and government acquire Nvidia chips despite US ban`{=html}](https://www.reuters.com/technology/chinas-military-government-acquire-nvidia-chips-despite-us-ban-2024-01-14/ "'China’s military and government acquire Nvidia chips despite US ban', Baptista 2024"){.link-annotated .id-not .include-annotation link-icon="n" link-icon-type="text,sans,italic" link-icon-color="#77ba00"}** ## `“Generative AI Beyond LLMs: System Implications of Multi-Modal Generation”, Golden et al 2023`{=html} {#golden-et-al-2023-section} **[`Generative AI Beyond LLMs: System Implications of Multi-Modal Generation`{=html}](https://arxiv.org/abs/2312.14385#facebook "'Generative AI Beyond LLMs: System Implications of Multi-Modal Generation', Golden et al 2023"){.link-annotated .id-not .include-annotation link-icon="facebook" link-icon-type="svg" link-icon-color="#1877f2"}** ## `“Real-Time AI & The Future of AI Hardware”, Uberti 2023`{=html} {#uberti-2023-section} **[`Real-Time AI & The Future of AI Hardware`{=html}](/doc/ai/scaling/hardware/2023-12-12-gavinuberti-podcastinterview-realtimeaiandthefutureofaihardwareecthedai.pdf "'Real-Time AI & The Future of AI Hardware', Uberti 2023"){.link-annotated-partial .id-not .include-annotation link-icon="pdf" link-icon-type="svg" link-icon-color="#f40f02"}** ## `“LLM in a Flash: Efficient Large Language Model Inference With Limited Memory”, Alizadeh et al 2023`{=html} {#alizadeh-et-al-2023-section} **[`LLM in a flash: Efficient Large Language Model Inference with Limited Memory`{=html}](https://arxiv.org/abs/2312.11514 "'LLM in a flash: Efficient Large Language Model Inference with Limited Memory', Alizadeh et al 2023"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“How Jensen Huang’s Nvidia Is Powering the AI Revolution: The Company’s CEO Bet It All on a New Kind of Chip. Now That Nvidia Is One of the Biggest Companies in the World, What Will He Do Next?”, Witt 2023`{=html} {#witt-2023-section} **[`How Jensen Huang’s Nvidia Is Powering the AI Revolution: The company’s CEO bet it all on a new kind of chip. Now that Nvidia is one of the biggest companies in the world, what will he do next?`{=html}](https://www.newyorker.com/magazine/2023/12/04/how-jensen-huangs-nvidia-is-powering-the-ai-revolution "'How Jensen Huang’s Nvidia Is Powering the AI Revolution: The company’s CEO bet it all on a new kind of chip. Now that Nvidia is one of the biggest companies in the world, what will he do next?', Witt 2023"){.link-annotated .id-not .include-annotation link-icon="n" link-icon-type="text,sans,italic" link-icon-color="#77ba00"}** ## `“OpenAI Agreed to Buy $51 Million of AI Chips From a Startup Backed by CEO Sam Altman”, Dave 2023`{=html} {#dave-2023-2-section} **[`OpenAI Agreed to Buy $51 Million of AI Chips From a Startup Backed by CEO Sam Altman`{=html}](https://www.wired.com/story/openai-buy-ai-chips-startup-sam-altman/ "'OpenAI Agreed to Buy $51 Million of AI Chips From a Startup Backed by CEO Sam Altman', Dave 2023"){.link-annotated .id-not .include-annotation link-icon="wired" link-icon-type="svg"}** ## `“Microsoft Swallows OpenAI’s Core Team § Compute Is King”, Patel & Nishball 2023`{=html} {#patel-nishball-2023-1-section} **[`Microsoft Swallows OpenAI’s Core Team § Compute Is King`{=html}](https://www.semianalysis.com/p/microsoft-swallows-openais-core-team#%C2%A7compute-is-king "'Microsoft Swallows OpenAI’s Core Team § Compute Is King', Patel & Nishball 2023"){.link-annotated .id-not .include-annotation}** ## `“Altman Sought Billions For Chip Venture Before OpenAI Ouster: Altman Was Fundraising in the Middle East for New Chip Venture; The Project, Code-Named Tigris, Is Intended to Rival Nvidia”, Ludlow & Vance 2023`{=html} {#ludlow-vance-2023-section} **[`Altman Sought Billions For Chip Venture Before OpenAI Ouster: Altman was fundraising in the Middle East for new chip venture; The project, code-named Tigris, is intended to rival Nvidia`{=html}](https://www.bloomberg.com/news/articles/2023-11-19/altman-sought-billions-for-ai-chip-venture-before-openai-ouster "'Altman Sought Billions For Chip Venture Before OpenAI Ouster: Altman was fundraising in the Middle East for new chip venture; The project, code-named Tigris, is intended to rival Nvidia', Ludlow & Vance 2023"){.link-annotated .id-not .include-annotation link-icon="𝐁" link-icon-type="text"}** ## `“DiLoCo: Distributed Low-Communication Training of Language Models”, Douillard et al 2023`{=html} {#douillard-et-al-2023-section} **[`DiLoCo: Distributed Low-Communication Training of Language Models`{=html}](https://arxiv.org/abs/2311.08105 "'DiLoCo: Distributed Low-Communication Training of Language Models', Douillard et al 2023"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“LSS Transformer: Ultra-Long Sequence Distributed Transformer”, Wang et al 2023`{=html} {#wang-et-al-2023-08-section} **[`LSS Transformer: Ultra-Long Sequence Distributed Transformer`{=html}](https://arxiv.org/abs/2311.02382 "'LSS Transformer: Ultra-Long Sequence Distributed Transformer', Wang et al 2023"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“ChipNeMo: Domain-Adapted LLMs for Chip Design”, Liu et al 2023`{=html} {#liu-et-al-2023-04-section} **[`ChipNeMo: Domain-Adapted LLMs for Chip Design`{=html}](https://arxiv.org/abs/2311.00176#nvidia "'ChipNeMo: Domain-Adapted LLMs for Chip Design', Liu et al 2023"){.link-annotated .id-not .include-annotation link-icon="n" link-icon-type="text,sans,italic" link-icon-color="#77ba00"}** ## `wagieeacc @ "2023-10-17"`{=html} {#shkreli-2023-section} **[`GPT-5 hardware rumor`{=html}](https://x.com/wagieeacc/status/1714458385624506679 "'GPT-5 hardware rumor', Shkreli 2023"){.link-annotated-partial .id-not .include-annotation link-icon="twitter" link-icon-type="svg" link-icon-color="#1da1f2"}** ## `“Saudi-China Collaboration Raises Concerns about Access to AI Chips: Fears Grow at Gulf Kingdom’s Top University That Ties to Chinese Researchers Risk Upsetting US Government”, Kerr et al 2023`{=html} {#kerr-et-al-2023-section} **[`Saudi-China collaboration raises concerns about access to AI chips: Fears grow at Gulf kingdom’s top university that ties to Chinese researchers risk upsetting US government`{=html}](https://www.ft.com/content/2a636cee-b0d2-45c2-a815-11ca32371763 "'Saudi-China collaboration raises concerns about access to AI chips: Fears grow at Gulf kingdom’s top university that ties to Chinese researchers risk upsetting US government', Kerr et al 2023"){.link-annotated .id-not .include-annotation link-icon="FT" link-icon-type="text" link-icon-color="#e3b68e"}** ## `“Efficient Video and Audio Processing With Loihi 2”, Shrestha et al 2023`{=html} {#shrestha-et-al-2023-section} **[`Efficient Video and Audio processing with Loihi 2`{=html}](https://arxiv.org/abs/2310.03251#intel "'Efficient Video and Audio processing with Loihi 2', Shrestha et al 2023"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Biden Is Beating China on Chips. It May Not Be Enough.”, Wang 2023`{=html} {#wang-2023-1-section} **[`Biden Is Beating China on Chips. It May Not Be Enough.`{=html}](https://www.nytimes.com/2023/07/16/opinion/biden-china-ai-chips-trade.html "'Biden Is Beating China on Chips. It May Not Be Enough.', Wang 2023"){.link-annotated .id-not .include-annotation link-icon="new-york-times" link-icon-type="svg"}** ## `“Deep Mind’s Chief on AI’s Dangers—And the UK’s £900 Million Supercomputer: Demis Hassabis Says We Shouldn’t Let AI Fall into the Wrong Hands and the Government’s Plan to Build a Supercomputer for AI Is Likely to Be out of Date Before It Has Even Started”, Sellman 2023`{=html} {#sellman-2023-section} **[`Deep Mind’s chief on AI’s dangers—and the UK’s £900 million supercomputer: Demis Hassabis says we shouldn’t let AI fall into the wrong hands and the government’s plan to build a supercomputer for AI is likely to be out of date before it has even started`{=html}](https://archive.is/c5jTk "'Deep Mind’s chief on AI’s dangers—and the UK’s £900 million supercomputer: Demis Hassabis says we shouldn’t let AI fall into the wrong hands and the government’s plan to build a supercomputer for AI is likely to be out of date before it has even started', Sellman 2023"){.link-annotated .id-not .include-annotation link-icon="internet-archive" link-icon-type="svg"}** ## `“Inflection AI Announces $1.3 Billion of Funding Led by Current Investors, Microsoft, and NVIDIA”, AI 2023`{=html} {#ai-2023-section} **[`Inflection AI announces $1.3 billion of funding led by current investors, Microsoft, and NVIDIA`{=html}](https://inflection.ai/inflection-ai-announces-1-3-billion-of-funding "'Inflection AI announces $1.3 billion of funding led by current investors, Microsoft, and NVIDIA', AI 2023"){.link-annotated .id-not .include-annotation}** ## `“U.S. Considers New Curbs on AI Chip Exports to China: Restrictions Come amid Concerns That China Could Use AI Chips from Nvidia and Others for Weapon Development and Hacking”, Fitch et al 2023`{=html} {#fitch-et-al-2023-section} **[`U.S. Considers New Curbs on AI Chip Exports to China: Restrictions come amid concerns that China could use AI chips from Nvidia and others for weapon development and hacking`{=html}](https://www.wsj.com/articles/u-s-considers-new-curbs-on-ai-chip-exports-to-china-56b17feb "'U.S. Considers New Curbs on AI Chip Exports to China: Restrictions come amid concerns that China could use AI chips from Nvidia and others for weapon development and hacking', Fitch et al 2023"){.link-annotated .id-not .include-annotation link-icon="WSJ" link-icon-type="text,tri"}** ## `“Unleashing True Utility Computing With Quicksand”, Ruan et al 2023`{=html} {#ruan-et-al-2023-section} **[`Unleashing True Utility Computing with Quicksand`{=html}](/doc/ai/scaling/hardware/2023-ruan.pdf "'Unleashing True Utility Computing with Quicksand', Ruan et al 2023"){.link-annotated .id-not .include-annotation link-icon="pdf" link-icon-type="svg" link-icon-color="#f40f02"}** ## `“The AI Boom Runs on Chips, but It Can’t Get Enough: ‘It’s like Toilet Paper during the Pandemic.’ Startups, Investors Scrounge for Computational Firepower”, Seetharaman & Dotan 2023`{=html} {#seetharaman-dotan-2023-section} **[`The AI Boom Runs on Chips, but It Can’t Get Enough: ‘It’s like toilet paper during the pandemic.’ Startups, investors scrounge for computational firepower`{=html}](https://www.wsj.com/articles/the-ai-boom-runs-on-chips-but-it-cant-get-enough-9f76f554 "'The AI Boom Runs on Chips, but It Can’t Get Enough: ‘It’s like toilet paper during the pandemic.’ Startups, investors scrounge for computational firepower', Seetharaman & Dotan 2023"){.link-annotated .id-not .include-annotation link-icon="WSJ" link-icon-type="text,tri"}** ## `“Cerebras Architecture Deep Dive: First Look Inside the HW/SW Co-Design for Deep Learning [Updated]”, Lie 2023`{=html} {#lie-2023-section} **[`Cerebras Architecture Deep Dive: First Look Inside the HW/SW Co-Design for Deep Learning [Updated]`{=html}](https://cerebras.ai/blog/cerebras-architecture-deep-dive-first-look-inside-the-hw/sw-co-design-for-deep-learning "'Cerebras Architecture Deep Dive: First Look Inside the HW/SW Co-Design for Deep Learning [Updated]', Lie 2023"){.id-not .include-annotation link-icon="C" link-icon-type="text,sans" link-icon-color="#f05a29"}** ## `“Big-PERCIVAL: Exploring the Native Use of 64-Bit Posit Arithmetic in Scientific Computing”, Mallasén et al 2023`{=html} {#mallasén-et-al-2023-section} **[`Big-PERCIVAL: Exploring the Native Use of 64-Bit Posit Arithmetic in Scientific Computing`{=html}](https://arxiv.org/abs/2305.06946 "'Big-PERCIVAL: Exploring the Native Use of 64-Bit Posit Arithmetic in Scientific Computing', Mallasén et al 2023"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“TPU V4: An Optically Reconfigurable Supercomputer for Machine Learning With Hardware Support for Embeddings”, Jouppi et al 2023`{=html} {#jouppi-et-al-2023-section} **[`TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings`{=html}](https://arxiv.org/abs/2304.01433#google "'TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings', Jouppi et al 2023"){.link-annotated .id-not .include-annotation link-icon="alphabet" link-icon-type="svg" link-icon-color="#4285f4"}** ## `“Accelerating Large GPT Training With Sparse Pre-Training and Dense Fine-Tuning”, Thangarasa 2023`{=html} {#thangarasa-2023-section} **[`Accelerating Large GPT Training with Sparse Pre-Training and Dense Fine-Tuning`{=html}](https://cerebras.ai/machine-learning/accelerating-large-gpt-training-with-sparse-pre-training-and-dense-fine-tuning/ "'Accelerating Large GPT Training with Sparse Pre-Training and Dense Fine-Tuning', Thangarasa 2023"){.id-not .include-annotation link-icon="C" link-icon-type="text,sans" link-icon-color="#f05a29"}** ## `davidtayar5 @ "2023-02-10"`{=html} {#stanley-2023-section} **[`Context on the NVIDIA ChatGPT opportunity—and ramifications of large language model enthusiasm`{=html}](https://x.com/davidtayar5/status/1627690520456691712 "'Context on the NVIDIA ChatGPT opportunity—and ramifications of large language model enthusiasm', Stanley 2023"){.link-annotated .id-not .include-annotation link-icon="twitter" link-icon-type="svg" link-icon-color="#1da1f2"}** ## `“SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient”, Ryabinin et al 2023`{=html} {#ryabinin-et-al-2023-section} **[`SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient`{=html}](https://arxiv.org/abs/2301.11913 "'SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient', Ryabinin et al 2023"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Microsoft and OpenAI Extend Partnership”, Microsoft 2023`{=html} {#microsoft-2023-section} **[`Microsoft and OpenAI extend partnership`{=html}](https://blogs.microsoft.com/blog/2023/01/23/microsoftandopenaiextendpartnership/ "'Microsoft and OpenAI extend partnership', Microsoft 2023"){.link-annotated .id-not .include-annotation link-icon="MS" link-icon-type="text,sans,italic"}** ## `“A 64-Core Mixed-Signal In-Memory Compute Chip Based on Phase-Change Memory for Deep Neural Network Inference”, Gallo et al 2022`{=html} {#gallo-et-al-2022-section} **[`A 64-core mixed-signal in-memory compute chip based on phase-change memory for deep neural network inference`{=html}](https://arxiv.org/abs/2212.02872#ibm "'A 64-core mixed-signal in-memory compute chip based on phase-change memory for deep neural network inference', Gallo et al 2022"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“How to Optimize a CUDA Matmul Kernel for CuBLAS-Like Performance: a Worklog”, Boehm 2022`{=html} {#boehm-2022-section} **[`How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance: a Worklog`{=html}](https://siboehm.com/articles/22/CUDA-MMM "'How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance: a Worklog', Boehm 2022"){.link-annotated-partial .id-not .include-annotation}** ## `“MultiRay: Optimizing Efficiency for Large-Scale AI Models”, Gupta et al 2022`{=html} {#gupta-et-al-2022-2-section} **[`MultiRay: Optimizing efficiency for large-scale AI models`{=html}](https://ai.meta.com/blog/multiray-large-scale-AI-models/ "'MultiRay: Optimizing efficiency for large-scale AI models', Gupta et al 2022"){.link-annotated .id-not .include-annotation link-icon="facebook" link-icon-type="svg" link-icon-color="#1877f2"}** ## `“Efficiently Scaling Transformer Inference”, Pope et al 2022`{=html} {#pope-et-al-2022-section} **[`Efficiently Scaling Transformer Inference`{=html}](https://arxiv.org/abs/2211.05102#google "'Efficiently Scaling Transformer Inference', Pope et al 2022"){.link-annotated .id-not .include-annotation link-icon="alphabet" link-icon-type="svg" link-icon-color="#4285f4"}** ## `“Reserve Capacity of NVIDIA HGX H100s on CoreWeave Now: Available at Scale in Q1 2023 Starting at $2.23/hr”, CoreWeave 2022`{=html} {#coreweave-2022-section} **[`Reserve capacity of NVIDIA HGX H100s on CoreWeave now: available at scale in Q1 2023 starting at $2.23/hr`{=html}](https://www.businesswire.com/news/home/20221107005057/en/CoreWeave-Among-First-Cloud-Providers-to-Offer-NVIDIA-HGX-H100-Supercomputers-Set-to-Transform-AI-Landscape "'Reserve capacity of NVIDIA HGX H100s on CoreWeave now: available at scale in Q1 2023 starting at $2.23/hr', CoreWeave 2022"){.link-annotated-partial .id-not .include-annotation}** ## `“Petals: Collaborative Inference and Fine-Tuning of Large Models”, Borzunov et al 2022`{=html} {#borzunov-et-al-2022-1-section} **[`Petals: Collaborative Inference and Fine-tuning of Large Models`{=html}](https://arxiv.org/abs/2209.01188#yandex "'Petals: Collaborative Inference and Fine-tuning of Large Models', Borzunov et al 2022"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Zeus: Understanding and Optimizing GPU Energy Consumption of DNN Training”, You et al 2022`{=html} {#you-et-al-2022-2-section} **[`Zeus: Understanding and Optimizing GPU Energy Consumption of DNN Training`{=html}](https://arxiv.org/abs/2208.06102 "'Zeus: Understanding and Optimizing GPU Energy Consumption of DNN Training', You et al 2022"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Is Integer Arithmetic Enough for Deep Learning Training?”, Ghaffari et al 2022`{=html} {#ghaffari-et-al-2022-section} **[`Is Integer Arithmetic Enough for Deep Learning Training?`{=html}](https://arxiv.org/abs/2207.08822 "'Is Integer Arithmetic Enough for Deep Learning Training?', Ghaffari et al 2022"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Efficient NLP Inference at the Edge via Elastic Pipelining”, Guo et al 2022`{=html} {#guo-et-al-2022-2-section} **[`Efficient NLP Inference at the Edge via Elastic Pipelining`{=html}](https://arxiv.org/abs/2207.05022 "'Efficient NLP Inference at the Edge via Elastic Pipelining', Guo et al 2022"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Training Transformers Together”, Borzunov et al 2022`{=html} {#borzunov-et-al-2022-2-section} **[`Training Transformers Together`{=html}](https://arxiv.org/abs/2207.03481 "'Training Transformers Together', Borzunov et al 2022"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Tutel: Adaptive Mixture-Of-Experts at Scale”, Hwang et al 2022`{=html} {#hwang-et-al-2022-2-section} **[`Tutel: Adaptive Mixture-of-Experts at Scale`{=html}](https://arxiv.org/abs/2206.03382#microsoft "'Tutel: Adaptive Mixture-of-Experts at Scale', Hwang et al 2022"){.link-annotated .id-not .include-annotation link-icon="MS" link-icon-type="text,sans,italic"}** ## `“8-Bit Numerical Formats for Deep Neural Networks”, Noune et al 2022`{=html} {#noune-et-al-2022-section} **[`8-bit Numerical Formats for Deep Neural Networks`{=html}](https://arxiv.org/abs/2206.02915#graphcore "'8-bit Numerical Formats for Deep Neural Networks', Noune et al 2022"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers”, Yao et al 2022`{=html} {#yao-et-al-2022-2-section} **[`ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers`{=html}](https://arxiv.org/abs/2206.01861#microsoft "'ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers', Yao et al 2022"){.link-annotated .id-not .include-annotation link-icon="MS" link-icon-type="text,sans,italic"}** ## `“FlashAttention: Fast and Memory-Efficient Exact Attention With IO-Awareness”, Dao et al 2022`{=html} {#dao-et-al-2022-1-section} **[`FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness`{=html}](https://arxiv.org/abs/2205.14135 "'FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness', Dao et al 2022"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“A Low-Latency Communication Design for Brain Simulations”, Du 2022`{=html} {#du-2022-section} **[`A Low-latency Communication Design for Brain Simulations`{=html}](https://arxiv.org/abs/2205.07125 "'A Low-latency Communication Design for Brain Simulations', Du 2022"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Reducing Activation Recomputation in Large Transformer Models”, Korthikanti et al 2022`{=html} {#korthikanti-et-al-2022-section} **[`Reducing Activation Recomputation in Large Transformer Models`{=html}](https://arxiv.org/abs/2205.05198#nvidia "'Reducing Activation Recomputation in Large Transformer Models', Korthikanti et al 2022"){.link-annotated .id-not .include-annotation link-icon="n" link-icon-type="text,sans,italic" link-icon-color="#77ba00"}** ## `“What Language Model to Train If You Have One Million GPU Hours?”, Scao et al 2022`{=html} {#scao-et-al-2022-section} **[`What Language Model to Train if You Have One Million GPU Hours?`{=html}](https://openreview.net/forum?id=rI7BL3fHIZq "'What Language Model to Train if You Have One Million GPU Hours?', Scao et al 2022"){.link-annotated .id-not .include-annotation link-icon="OR" link-icon-type="text,sans" link-icon-color="#8c1b13"}** ## `“Monarch: Expressive Structured Matrices for Efficient and Accurate Training”, Dao et al 2022`{=html} {#dao-et-al-2022-2-section} **[`Monarch: Expressive Structured Matrices for Efficient and Accurate Training`{=html}](https://arxiv.org/abs/2204.00595 "'Monarch: Expressive Structured Matrices for Efficient and Accurate Training', Dao et al 2022"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Pathways: Asynchronous Distributed Dataflow for ML”, Barham et al 2022`{=html} {#barham-et-al-2022-section} **[`Pathways: Asynchronous Distributed Dataflow for ML`{=html}](https://arxiv.org/abs/2203.12533#google "'Pathways: Asynchronous Distributed Dataflow for ML', Barham et al 2022"){.link-annotated .id-not .include-annotation link-icon="alphabet" link-icon-type="svg" link-icon-color="#4285f4"}** ## `“LiteTransformerSearch: Training-Free Neural Architecture Search for Efficient Language Models”, Javaheripi et al 2022`{=html} {#javaheripi-et-al-2022-section} **[`LiteTransformerSearch: Training-free Neural Architecture Search for Efficient Language Models`{=html}](https://arxiv.org/abs/2203.02094#microsoft "'LiteTransformerSearch: Training-free Neural Architecture Search for Efficient Language Models', Javaheripi et al 2022"){.link-annotated .id-not .include-annotation link-icon="MS" link-icon-type="text,sans,italic"}** ## `“Singularity: Planet-Scale, Preemptive and Elastic Scheduling of AI Workloads”, Shukla et al 2022`{=html} {#shukla-et-al-2022-2-section} **[`Singularity: Planet-Scale, Preemptive and Elastic Scheduling of AI Workloads`{=html}](https://arxiv.org/abs/2202.07848#microsoft "'Singularity: Planet-Scale, Preemptive and Elastic Scheduling of AI Workloads', Shukla et al 2022"){.link-annotated .id-not .include-annotation link-icon="MS" link-icon-type="text,sans,italic"}** ## `“Maximizing Communication Efficiency for Large-Scale Training via 0/1 Adam”, Lu et al 2022`{=html} {#lu-et-al-2022-7-section} **[`Maximizing Communication Efficiency for Large-scale Training via 0/1 Adam`{=html}](https://arxiv.org/abs/2202.06009#microsoft "'Maximizing Communication Efficiency for Large-scale Training via 0/1 Adam', Lu et al 2022"){.link-annotated .id-not .include-annotation link-icon="MS" link-icon-type="text,sans,italic"}** ## `“Introducing the AI Research SuperCluster—Facebook’s Cutting-Edge AI Supercomputer for AI Research”, Lee & Sengupta 2022`{=html} {#lee-sengupta-2022-section} **[`Introducing the AI Research SuperCluster—Facebook’s cutting-edge AI supercomputer for AI research`{=html}](https://ai.meta.com/blog/ai-rsc/ "'Introducing the AI Research SuperCluster—Facebook’s cutting-edge AI supercomputer for AI research', Lee & Sengupta 2022"){.link-annotated .id-not .include-annotation link-icon="facebook" link-icon-type="svg" link-icon-color="#1877f2"}** ## `“Is Programmable Overhead Worth The Cost? How Much Do We Pay for a System to Be Programmable? It Depends upon Who You Ask”, Bailey 2022`{=html} {#bailey-2022-section} **[`Is Programmable Overhead Worth The Cost? How much do we pay for a system to be programmable? It depends upon who you ask`{=html}](https://semiengineering.com/is-programmable-overhead-worth-the-cost/ "'Is Programmable Overhead Worth The Cost? How much do we pay for a system to be programmable? It depends upon who you ask', Bailey 2022"){.link-annotated .id-not .include-annotation}** ## `“Spiking Neural Networks and Their Applications: A Review”, Yamazaki et al 2022`{=html} {#yamazaki-et-al-2022-section} **[`Spiking Neural Networks and Their Applications: A Review`{=html}](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9313413/ "'Spiking Neural Networks and Their Applications: A Review', Yamazaki et al 2022"){.link-annotated .id-not .include-annotation link-icon="nlm-ncbi" link-icon-type="svg" link-icon-color="#20558a"}** ## `“On the Working Memory of Humans and Great Apes: Strikingly Similar or Remarkably Different?”, Read et al 2021`{=html} {#read-et-al-2021-section} **[`On the Working Memory of Humans and Great Apes: Strikingly Similar or Remarkably Different?`{=html}](https://www.sciencedirect.com/science/article/pii/S0149763421005674 "'On the Working Memory of Humans and Great Apes: Strikingly Similar or Remarkably Different?', Read et al 2021"){.link-annotated .id-not .include-annotation link-icon="E" link-icon-type="text" link-icon-color="#eb6500"}** ## `“Sustainable AI: Environmental Implications, Challenges and Opportunities”, Wu et al 2021`{=html} {#wu-et-al-2021-05-section} **[`Sustainable AI: Environmental Implications, Challenges and Opportunities`{=html}](https://arxiv.org/abs/2111.00364#facebook "'Sustainable AI: Environmental Implications, Challenges and Opportunities', Wu et al 2021"){.link-annotated .id-not .include-annotation link-icon="facebook" link-icon-type="svg" link-icon-color="#1877f2"}** ## `“China Has Already Reached Exascale—On Two Separate Systems”, Hemsoth 2021`{=html} {#hemsoth-2021-section} **[`China Has Already Reached Exascale—On Two Separate Systems`{=html}](https://www.nextplatform.com/2021/10/26/china-has-already-reached-exascale-on-two-separate-systems/ "'China Has Already Reached Exascale—On Two Separate Systems', Hemsoth 2021"){.link-annotated .id-not .include-annotation link-icon="NEXT" link-icon-type="text,quad,sans" link-icon-color="#ff7200"}** ## `“The Efficiency Misnomer”, Dehghani et al 2021`{=html} {#dehghani-et-al-2021-section} **[`The Efficiency Misnomer`{=html}](https://arxiv.org/abs/2110.12894#google "'The Efficiency Misnomer', Dehghani et al 2021"){.link-annotated .id-not .include-annotation link-icon="alphabet" link-icon-type="svg" link-icon-color="#4285f4"}** ## `“Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning”, Rudin et al 2021`{=html} {#rudin-et-al-2021-section} **[`Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning`{=html}](https://arxiv.org/abs/2109.11978#nvidia "'Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning', Rudin et al 2021"){.link-annotated .id-not .include-annotation link-icon="n" link-icon-type="text,sans,italic" link-icon-color="#77ba00"}** ## `“WarpDrive: Extremely Fast End-To-End Deep Multi-Agent Reinforcement Learning on a GPU”, Lan et al 2021`{=html} {#lan-et-al-2021-2-section} **[`WarpDrive: Extremely Fast End-to-End Deep Multi-Agent Reinforcement Learning on a GPU`{=html}](https://arxiv.org/abs/2108.13976#salesforce "'WarpDrive: Extremely Fast End-to-End Deep Multi-Agent Reinforcement Learning on a GPU', Lan et al 2021"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning”, Makoviychuk et al 2021`{=html} {#makoviychuk-et-al-2021-section} **[`Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning`{=html}](https://arxiv.org/abs/2108.10470#nvidia "'Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning', Makoviychuk et al 2021"){.link-annotated .id-not .include-annotation link-icon="n" link-icon-type="text,sans,italic" link-icon-color="#77ba00"}** ## `“PatrickStar: Parallel Training of Pre-Trained Models via Chunk-Based Memory Management”, Fang et al 2021`{=html} {#fang-et-al-2021-3-section} **[`PatrickStar: Parallel Training of Pre-trained Models via Chunk-based Memory Management`{=html}](https://arxiv.org/abs/2108.05818#tencent "'PatrickStar: Parallel Training of Pre-trained Models via Chunk-based Memory Management', Fang et al 2021"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Demonstration of Decentralized, Physics-Driven Learning”, Dillavou et al 2021`{=html} {#dillavou-et-al-2021-section} **[`Demonstration of Decentralized, Physics-Driven Learning`{=html}](https://arxiv.org/abs/2108.00275 "'Demonstration of Decentralized, Physics-Driven Learning', Dillavou et al 2021"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Chimera: Efficiently Training Large-Scale Neural Networks With Bidirectional Pipelines”, Li & Hoefler 2021`{=html} {#li-hoefler-2021-section} **[`Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelines`{=html}](https://arxiv.org/abs/2107.06925 "'Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelines', Li & Hoefler 2021"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“First-Generation Inference Accelerator Deployment at Facebook”, Anderson et al 2021`{=html} {#anderson-et-al-2021-1-section} **[`First-Generation Inference Accelerator Deployment at Facebook`{=html}](https://arxiv.org/abs/2107.04140#facebook "'First-Generation Inference Accelerator Deployment at Facebook', Anderson et al 2021"){.link-annotated .id-not .include-annotation link-icon="facebook" link-icon-type="svg" link-icon-color="#1877f2"}** ## `“Single-Chip Photonic Deep Neural Network for Instantaneous Image Classification”, Ashtiani et al 2021`{=html} {#ashtiani-et-al-2021-section} **[`Single-chip photonic deep neural network for instantaneous image classification`{=html}](https://arxiv.org/abs/2106.11747 "'Single-chip photonic deep neural network for instantaneous image classification', Ashtiani et al 2021"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Distributed Deep Learning in Open Collaborations”, Diskin et al 2021`{=html} {#diskin-et-al-2021-section} **[`Distributed Deep Learning in Open Collaborations`{=html}](https://arxiv.org/abs/2106.10207 "'Distributed Deep Learning in Open Collaborations', Diskin et al 2021"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Ten Lessons From Three Generations Shaped Google’s TPUv4i”, Jouppi et al 2021`{=html} {#jouppi-et-al-2021-section} **[`Ten Lessons From Three Generations Shaped Google’s TPUv4i`{=html}](/doc/ai/scaling/hardware/2021-jouppi.pdf "'Ten Lessons From Three Generations Shaped Google’s TPUv4i', Jouppi et al 2021"){.link-annotated .id-not .include-annotation link-icon="pdf" link-icon-type="svg" link-icon-color="#f40f02"}** ## `“Maximizing 3-D Parallelism in Distributed Training for Huge Neural Networks”, Bian et al 2021`{=html} {#bian-et-al-2021-section} **[`Maximizing 3-D Parallelism in Distributed Training for Huge Neural Networks`{=html}](https://arxiv.org/abs/2105.14450 "'Maximizing 3-D Parallelism in Distributed Training for Huge Neural Networks', Bian et al 2021"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“2.5-Dimensional Distributed Model Training”, Wang et al 2021`{=html} {#wang-et-al-2021-07-section} **[`2.5-dimensional distributed model training`{=html}](https://arxiv.org/abs/2105.14500 "'2.5-dimensional distributed model training', Wang et al 2021"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“A Full-Stack Accelerator Search Technique for Vision Applications”, Zhang et al 2021`{=html} {#zhang-et-al-2021-nas-section} **[`A Full-stack Accelerator Search Technique for Vision Applications`{=html}](https://arxiv.org/abs/2105.12842#google "'A Full-stack Accelerator Search Technique for Vision Applications', Zhang et al 2021"){.link-annotated .id-not .include-annotation link-icon="alphabet" link-icon-type="svg" link-icon-color="#4285f4"}** ## `“ChinAI #141: The PanGu Origin Story: Notes from an Informative Zhihu Thread on PanGu”, Ding 2021`{=html} {#ding-2021-1-section} **[`ChinAI #141: The PanGu Origin Story: Notes from an informative Zhihu Thread on PanGu`{=html}](https://chinai.substack.com/p/chinai-141-the-pangu-origin-story "'ChinAI #141: The PanGu Origin Story: Notes from an informative Zhihu Thread on PanGu', Ding 2021"){.link-annotated .id-not .include-annotation link-icon="substack" link-icon-type="svg" link-icon-color="#ff6719"}** ## `“GSPMD: General and Scalable Parallelization for ML Computation Graphs”, Xu et al 2021`{=html} {#xu-et-al-2021-7-section} **[`GSPMD: General and Scalable Parallelization for ML Computation Graphs`{=html}](https://arxiv.org/abs/2105.04663#google "'GSPMD: General and Scalable Parallelization for ML Computation Graphs', Xu et al 2021"){.link-annotated .id-not .include-annotation link-icon="alphabet" link-icon-type="svg" link-icon-color="#4285f4"}** ## `“PanGu-Α: Large-Scale Autoregressive Pretrained Chinese Language Models With Auto-Parallel Computation”, Zeng et al 2021`{=html} {#zeng-et-al-2021-3-section} **[`PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation`{=html}](https://arxiv.org/abs/2104.12369#huawei "'PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation', Zeng et al 2021"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning”, Rajbhandari et al 2021`{=html} {#rajbhandari-et-al-2021-section} **[`ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning`{=html}](https://arxiv.org/abs/2104.07857#microsoft "'ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning', Rajbhandari et al 2021"){.link-annotated .id-not .include-annotation link-icon="MS" link-icon-type="text,sans,italic"}** ## `“How to Train BERT With an Academic Budget”, Izsak et al 2021`{=html} {#izsak-et-al-2021-section} **[`How to Train BERT with an Academic Budget`{=html}](https://arxiv.org/abs/2104.07705 "'How to Train BERT with an Academic Budget', Izsak et al 2021"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Podracer Architectures for Scalable Reinforcement Learning”, Hessel et al 2021`{=html} {#hessel-et-al-2021-2-section} **[`Podracer architectures for scalable Reinforcement Learning`{=html}](https://arxiv.org/abs/2104.06272#deepmind "'Podracer architectures for scalable Reinforcement Learning', Hessel et al 2021"){.link-annotated .id-not .include-annotation link-icon="deepmind" link-icon-type="svg" link-icon-color="#4185f4"}** ## `“High-Performance, Distributed Training of Large-Scale Deep Learning Recommendation Models (DLRMs)”, Mudigere et al 2021`{=html} {#mudigere-et-al-2021-section} **[`High-performance, Distributed Training of Large-scale Deep Learning Recommendation Models (DLRMs)`{=html}](https://arxiv.org/abs/2104.05158#facebook "'High-performance, Distributed Training of Large-scale Deep Learning Recommendation Models (DLRMs)', Mudigere et al 2021"){.link-annotated .id-not .include-annotation link-icon="facebook" link-icon-type="svg" link-icon-color="#1877f2"}** ## `“An Efficient 2D Method for Training Super-Large Deep Learning Models”, Xu et al 2021`{=html} {#xu-et-al-2021-8-section} **[`An Efficient 2D Method for Training Super-Large Deep Learning Models`{=html}](https://arxiv.org/abs/2104.05343 "'An Efficient 2D Method for Training Super-Large Deep Learning Models', Xu et al 2021"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Efficient Large-Scale Language Model Training on GPU Clusters”, Narayanan et al 2021`{=html} {#narayanan-et-al-2021-section} **[`Efficient Large-Scale Language Model Training on GPU Clusters`{=html}](https://arxiv.org/abs/2104.04473#nvidia "'Efficient Large-Scale Language Model Training on GPU Clusters', Narayanan et al 2021"){.link-annotated .id-not .include-annotation link-icon="n" link-icon-type="text,sans,italic" link-icon-color="#77ba00"}** ## `“Large Batch Simulation for Deep Reinforcement Learning”, Shacklett et al 2021`{=html} {#shacklett-et-al-2021-section} **[`Large Batch Simulation for Deep Reinforcement Learning`{=html}](https://arxiv.org/abs/2103.07013 "'Large Batch Simulation for Deep Reinforcement Learning', Shacklett et al 2021"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Warehouse-Scale Video Acceleration (Argos): Co-Design and Deployment in the Wild”, Ranganathan et al 2021`{=html} {#ranganathan-et-al-2021-section} **[`Warehouse-Scale Video Acceleration (Argos): Co-design and Deployment in the Wild`{=html}](/doc/cs/hardware/2021-ranganathan.pdf#google "'Warehouse-Scale Video Acceleration (Argos): Co-design and Deployment in the Wild', Ranganathan et al 2021"){.link-annotated .id-not .include-annotation link-icon="alphabet" link-icon-type="svg" link-icon-color="#4285f4"}** ## `“TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models”, Li et al 2021`{=html} {#li-et-al-2021-8-section} **[`TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models`{=html}](https://arxiv.org/abs/2102.07988 "'TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models', Li et al 2021"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“PipeTransformer: Automated Elastic Pipelining for Distributed Training of Transformers”, He et al 2021`{=html} {#he-et-al-2021-4-section} **[`PipeTransformer: Automated Elastic Pipelining for Distributed Training of Transformers`{=html}](https://arxiv.org/abs/2102.03161 "'PipeTransformer: Automated Elastic Pipelining for Distributed Training of Transformers', He et al 2021"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“ZeRO-Offload: Democratizing Billion-Scale Model Training”, Ren et al 2021`{=html} {#ren-et-al-2021-2-section} **[`ZeRO-Offload: Democratizing Billion-Scale Model Training`{=html}](https://arxiv.org/abs/2101.06840#microsoft "'ZeRO-Offload: Democratizing Billion-Scale Model Training', Ren et al 2021"){.link-annotated .id-not .include-annotation link-icon="MS" link-icon-type="text,sans,italic"}** ## `“The Design Process for Google’s Training Chips: TPUv2 and TPUv3”, Norrie et al 2021`{=html} {#norrie-et-al-2021-section} **[`The Design Process for Google’s Training Chips: TPUv2 and TPUv3`{=html}](/doc/ai/scaling/hardware/2021-norrie.pdf#google "'The Design Process for Google’s Training Chips: TPUv2 and TPUv3', Norrie et al 2021"){.link-annotated .id-not .include-annotation link-icon="alphabet" link-icon-type="svg" link-icon-color="#4285f4"}** ## `“Hardware Beyond Backpropagation: a Photonic Co-Processor for Direct Feedback Alignment”, Launay et al 2020`{=html} {#launay-et-al-2020-section} **[`Hardware Beyond Backpropagation: a Photonic Co-Processor for Direct Feedback Alignment`{=html}](https://arxiv.org/abs/2012.06373#lighton "'Hardware Beyond Backpropagation: a Photonic Co-Processor for Direct Feedback Alignment', Launay et al 2020"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Parallel Training of Deep Networks With Local Updates”, Laskin et al 2020`{=html} {#laskin-et-al-2020-1-section} **[`Parallel Training of Deep Networks with Local Updates`{=html}](https://arxiv.org/abs/2012.03837#google-graphcore "'Parallel Training of Deep Networks with Local Updates', Laskin et al 2020"){.link-annotated .id-not .include-annotation link-icon="alphabet" link-icon-type="svg" link-icon-color="#4285f4"}** ## `“Exploring the Limits of Concurrency in ML Training on Google TPUs”, Kumar et al 2020`{=html} {#kumar-et-al-2020-1-section} **[`Exploring the limits of Concurrency in ML Training on Google TPUs`{=html}](https://arxiv.org/abs/2011.03641#google "'Exploring the limits of Concurrency in ML Training on Google TPUs', Kumar et al 2020"){.link-annotated .id-not .include-annotation link-icon="alphabet" link-icon-type="svg" link-icon-color="#4285f4"}** ## `“BytePS: A Unified Architecture for Accelerating Distributed DNN Training in Heterogeneous GPU/CPU Clusters”, Jiang et al 2020`{=html} {#jiang-et-al-2020-2-section} **[`BytePS: A Unified Architecture for Accelerating Distributed DNN Training in Heterogeneous GPU/CPU Clusters`{=html}](/doc/ai/scaling/hardware/2020-jiang.pdf "'BytePS: A Unified Architecture for Accelerating Distributed DNN Training in Heterogeneous GPU/CPU Clusters', Jiang et al 2020"){.link-annotated .id-not .include-annotation link-icon="pdf" link-icon-type="svg" link-icon-color="#f40f02"}** ## `“Training EfficientNets at Supercomputer Scale: 83% ImageNet Top-1 Accuracy in One Hour”, Wongpanich et al 2020`{=html} {#wongpanich-et-al-2020-section} **[`Training EfficientNets at Supercomputer Scale: 83% ImageNet Top-1 Accuracy in One Hour`{=html}](https://arxiv.org/abs/2011.00071#google "'Training EfficientNets at Supercomputer Scale: 83% ImageNet Top-1 Accuracy in One Hour', Wongpanich et al 2020"){.link-annotated .id-not .include-annotation link-icon="alphabet" link-icon-type="svg" link-icon-color="#4285f4"}** ## `“Matrix Engines for High Performance Computing:A Paragon of Performance or Grasping at Straws?”, Domke et al 2020`{=html} {#domke-et-al-2020-section} **[`Matrix Engines for High Performance Computing:A Paragon of Performance or Grasping at Straws?`{=html}](https://arxiv.org/abs/2010.14373 "'Matrix Engines for High Performance Computing:A Paragon of Performance or Grasping at Straws?', Domke et al 2020"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures”, Launay et al 2020b`{=html} {#launay-et-al-2020b-section} **[`Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures`{=html}](/doc/ai/scaling/hardware/2020-launay-2.pdf "'Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures', Launay et al 2020b"){.link-annotated .id-not .include-annotation link-icon="pdf" link-icon-type="svg" link-icon-color="#f40f02"}** ## `“Interlocking Backpropagation: Improving Depthwise Model-Parallelism”, Gomez et al 2020`{=html} {#gomez-et-al-2020-section} **[`Interlocking Backpropagation: Improving depthwise model-parallelism`{=html}](https://arxiv.org/abs/2010.04116 "'Interlocking Backpropagation: Improving depthwise model-parallelism', Gomez et al 2020"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“DeepSpeed: Extreme-Scale Model Training for Everyone”, Team et al 2020`{=html} {#team-et-al-2020-section} **[`DeepSpeed: Extreme-scale model training for everyone`{=html}](https://www.microsoft.com/en-us/research/blog/deepspeed-extreme-scale-model-training-for-everyone/ "'DeepSpeed: Extreme-scale model training for everyone', Team et al 2020"){.link-annotated .id-not .include-annotation link-icon="MS" link-icon-type="text,sans,italic"}** ## `“Measuring Hardware Overhang”, Hippke 2020`{=html} {#hippke-2020-section} **[`Measuring hardware overhang`{=html}](https://www.lesswrong.com/posts/75dnjiD8kv2khe9eQ/measuring-hardware-overhang "'Measuring hardware overhang', Hippke 2020"){.link-annotated .id-not .include-annotation link-icon="LW" link-icon-type="text" link-icon-color="#7faf83"}** ## `“The Node Is Nonsense: There Are Better Ways to Measure Progress Than the Old Moore’s Law Metric”, Moore 2020`{=html} {#moore-2020-section} **[`The Node Is Nonsense: There are better ways to measure progress than the old Moore’s law metric`{=html}](https://ieeexplore.ieee.org/abstract/document/9150552 "'The Node Is Nonsense: There are better ways to measure progress than the old Moore’s law metric', Moore 2020"){.link-annotated .id-not .include-annotation link-icon="IEEE" link-icon-type="text,mono,quad" link-icon-color="#006699"}** ## `“Are We in an AI Overhang?”, Jones 2020`{=html} {#jones-2020-section} **[`Are we in an AI overhang?`{=html}](https://www.lesswrong.com/posts/N6vZEnCn6A95Xn39p/are-we-in-an-ai-overhang "'Are we in an AI overhang?', Jones 2020"){.link-annotated .id-not .include-annotation link-icon="LW" link-icon-type="text" link-icon-color="#7faf83"}** ## `“HOBFLOPS CNNs: Hardware Optimized Bitslice-Parallel Floating-Point Operations for Convolutional Neural Networks”, Garland & Gregg 2020`{=html} {#garland-gregg-2020-section} **[`HOBFLOPS CNNs: Hardware Optimized Bitslice-Parallel Floating-Point Operations for Convolutional Neural Networks`{=html}](https://arxiv.org/abs/2007.06563#facebook "'HOBFLOPS CNNs: Hardware Optimized Bitslice-Parallel Floating-Point Operations for Convolutional Neural Networks', Garland & Gregg 2020"){.link-annotated .id-not .include-annotation link-icon="facebook" link-icon-type="svg" link-icon-color="#1877f2"}** ## `“The Computational Limits of Deep Learning”, Thompson et al 2020`{=html} {#thompson-et-al-2020-1-section} **[`The Computational Limits of Deep Learning`{=html}](https://arxiv.org/abs/2007.05558 "'The Computational Limits of Deep Learning', Thompson et al 2020"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Data Movement Is All You Need: A Case Study on Optimizing Transformers”, Ivanov et al 2020`{=html} {#ivanov-et-al-2020-section} **[`Data Movement Is All You Need: A Case Study on Optimizing Transformers`{=html}](https://arxiv.org/abs/2007.00072 "'Data Movement Is All You Need: A Case Study on Optimizing Transformers', Ivanov et al 2020"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“PyTorch Distributed: Experiences on Accelerating Data Parallel Training”, Li et al 2020`{=html} {#li-et-al-2020-3-section} **[`PyTorch Distributed: Experiences on Accelerating Data Parallel Training`{=html}](https://arxiv.org/abs/2006.15704#facebook "'PyTorch Distributed: Experiences on Accelerating Data Parallel Training', Li et al 2020"){.link-annotated .id-not .include-annotation link-icon="facebook" link-icon-type="svg" link-icon-color="#1877f2"}** ## `“Japanese Supercomputer Is Crowned World’s Speediest: In the Race for the Most Powerful Computers, Fugaku, a Japanese Supercomputer, Recently Beat American and Chinese Machines”, Clark 2020`{=html} {#clark-2020-1-section} **[`Japanese Supercomputer Is Crowned World’s Speediest: In the race for the most powerful computers, Fugaku, a Japanese supercomputer, recently beat American and Chinese machines`{=html}](https://www.nytimes.com/2020/06/22/technology/japanese-supercomputer-fugaku-tops-american-chinese-machines.html "'Japanese Supercomputer Is Crowned World’s Speediest: In the race for the most powerful computers, Fugaku, a Japanese supercomputer, recently beat American and Chinese machines', Clark 2020"){.link-annotated .id-not .include-annotation link-icon="new-york-times" link-icon-type="svg"}** ## `“Sample Factory: Egocentric 3D Control from Pixels at 100,000 FPS With Asynchronous Reinforcement Learning”, Petrenko et al 2020`{=html} {#petrenko-et-al-2020-section} **[`Sample Factory: Egocentric 3D Control from Pixels at 100,000 FPS with Asynchronous Reinforcement Learning`{=html}](https://arxiv.org/abs/2006.11751#intel "'Sample Factory: Egocentric 3D Control from Pixels at 100,000 FPS with Asynchronous Reinforcement Learning', Petrenko et al 2020"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“PipeDream-2BW: Memory-Efficient Pipeline-Parallel DNN Training”, Narayanan et al 2020`{=html} {#narayanan-et-al-2020-section} **[`PipeDream-2BW: Memory-Efficient Pipeline-Parallel DNN Training`{=html}](https://arxiv.org/abs/2006.09503#microsoft "'PipeDream-2BW: Memory-Efficient Pipeline-Parallel DNN Training', Narayanan et al 2020"){.link-annotated .id-not .include-annotation link-icon="MS" link-icon-type="text,sans,italic"}** ## `“There’s Plenty of Room at the Top: What Will Drive Computer Performance After Moore’s Law?”, Leiserson et al 2020`{=html} {#leiserson-et-al-2020-section} **[`There’s plenty of room at the Top: What will drive computer performance after Moore’s law?`{=html}](/doc/cs/hardware/2020-leiserson.pdf "'There’s plenty of room at the Top: What will drive computer performance after Moore’s law?', Leiserson et al 2020"){.link-annotated .id-not .include-annotation link-icon="pdf" link-icon-type="svg" link-icon-color="#f40f02"}** ## `“A Domain-Specific Supercomputer for Training Deep Neural Networks”, Jouppi et al 2020`{=html} {#jouppi-et-al-2020-section} **[`A domain-specific supercomputer for training deep neural networks`{=html}](/doc/ai/scaling/hardware/2020-jouppi.pdf#google "'A domain-specific supercomputer for training deep neural networks', Jouppi et al 2020"){.link-annotated .id-not .include-annotation link-icon="alphabet" link-icon-type="svg" link-icon-color="#4285f4"}** ## `“Microsoft Announces New Supercomputer, Lays out Vision for Future AI Work”, Langston 2020`{=html} {#langston-2020-section} **[`Microsoft announces new supercomputer, lays out vision for future AI work`{=html}](https://news.microsoft.com/source/features/ai/openai-azure-supercomputer/ "'Microsoft announces new supercomputer, lays out vision for future AI work', Langston 2020"){.link-annotated .id-not .include-annotation link-icon="MS" link-icon-type="text,sans,italic"}** ## `“AI and Efficiency: We’re Releasing an Analysis Showing That Since 2012 the Amount of Compute Needed to Train a Neural Net to the Same Performance on ImageNet Classification Has Been Decreasing by a Factor of 2 Every 16 Months”, Hernandez & Brown 2020`{=html} {#hernandez-brown-2020-blog-section} **[`AI and Efficiency: We’re releasing an analysis showing that since 2012 the amount of compute needed to train a neural net to the same performance on ImageNet classification has been decreasing by a factor of 2 every 16 months`{=html}](https://openai.com/research/ai-and-efficiency "'AI and Efficiency: We’re releasing an analysis showing that since 2012 the amount of compute needed to train a neural net to the same performance on ImageNet classification has been decreasing by a factor of 2 every 16 months', Hernandez & Brown 2020"){.link-annotated .id-not .include-annotation link-icon="openai" link-icon-type="svg"}** ## `“Computation in the Human Cerebral Cortex Uses Less Than 0.2 Watts yet This Great Expense Is Optimal When considering Communication Costs”, Levy & Calvert 2020`{=html} {#levy-calvert-2020-section} **[`Computation in the human cerebral cortex uses less than 0.2 watts yet this great expense is optimal when considering communication costs`{=html}](https://www.biorxiv.org/content/10.1101/2020.04.23.057927.full "'Computation in the human cerebral cortex uses less than 0.2 watts yet this great expense is optimal when considering communication costs', Levy & Calvert 2020"){.link-annotated .id-not .include-annotation link-icon="chi-dna" link-icon-type="svg" link-icon-color="#bd2736"}** ## `“Startup Tenstorrent Shows AI Is Changing Computing and vice Versa: Tenstorrent Is One of the Rush of AI Chip Makers Founded in 2016 and Finally Showing Product. The New Wave of Chips Represent a Substantial Departure from How Traditional Computer Chips Work, but Also Point to Ways That Neural Network Design May Change in the Years to Come”, Ray 2020`{=html} {#ray-2020-section} **[`Startup Tenstorrent shows AI is changing computing and vice versa: Tenstorrent is one of the rush of AI chip makers founded in 2016 and finally showing product. The new wave of chips represent a substantial departure from how traditional computer chips work, but also point to ways that neural network design may change in the years to come`{=html}](https://www.zdnet.com/article/startup-tenstorrent-and-competitors-show-how-computing-is-changing-ai-and-vice-versa/ "'Startup Tenstorrent shows AI is changing computing and vice versa: Tenstorrent is one of the rush of AI chip makers founded in 2016 and finally showing product. The new wave of chips represent a substantial departure from how traditional computer chips work, but also point to ways that neural network design may change in the years to come', Ray 2020"){.link-annotated .id-not .include-annotation}** ## `“AI Chips: What They Are and Why They Matter—An AI Chips Reference”, Khan & Mann 2020`{=html} {#khan-mann-2020-section} **[`AI Chips: What They Are and Why They Matter—An AI Chips Reference`{=html}](/doc/ai/scaling/hardware/2020-khan.pdf "'AI Chips: What They Are and Why They Matter—An AI Chips Reference', Khan & Mann 2020"){.link-annotated .id-not .include-annotation link-icon="pdf" link-icon-type="svg" link-icon-color="#f40f02"}** ## `“2019 Recent Trends in GPU Price per FLOPS”, Bergal 2020`{=html} {#bergal-2020-section} **[`2019 recent trends in GPU price per FLOPS`{=html}](https://aiimpacts.org/2019-recent-trends-in-gpu-price-per-flops/ "'2019 recent trends in GPU price per FLOPS', Bergal 2020"){.link-annotated .id-not .include-annotation link-icon="AII" link-icon-type="text,tri" link-icon-color="#2396ce"}** ## `“Pipelined Backpropagation at Scale: Training Large Models without Batches”, Kosson et al 2020`{=html} {#kosson-et-al-2020-section} **[`Pipelined Backpropagation at Scale: Training Large Models without Batches`{=html}](https://arxiv.org/abs/2003.11666#cerebras "'Pipelined Backpropagation at Scale: Training Large Models without Batches', Kosson et al 2020"){.link-annotated .id-not .include-annotation link-icon="C" link-icon-type="text,sans" link-icon-color="#f05a29"}** ## `“Ultrafast Machine Vision With 2D Material Neural Network Image Sensors”, Mennel et al 2020`{=html} {#mennel-et-al-2020-section} **[`Ultrafast machine vision with 2D material neural network image sensors`{=html}](/doc/ai/scaling/hardware/2020-mennel.pdf "'Ultrafast machine vision with 2D material neural network image sensors', Mennel et al 2020"){.link-annotated-partial .id-not .include-annotation link-icon="pdf" link-icon-type="svg" link-icon-color="#f40f02"}** ## `“L2L: Training Large Neural Networks With Constant Memory Using a New Execution Algorithm”, Pudipeddi et al 2020`{=html} {#pudipeddi-et-al-2020-section} **[`L2L: Training Large Neural Networks with Constant Memory using a New Execution Algorithm`{=html}](https://arxiv.org/abs/2002.05645#microsoft "'L2L: Training Large Neural Networks with Constant Memory using a New Execution Algorithm', Pudipeddi et al 2020"){.link-annotated .id-not .include-annotation link-icon="MS" link-icon-type="text,sans,italic"}** ## `“Towards Spike-Based Machine Intelligence With Neuromorphic Computing”, Roy et al 2019`{=html} {#roy-et-al-2019-section} **[`Towards spike-based machine intelligence with neuromorphic computing`{=html}](/doc/ai/scaling/hardware/2019-roy.pdf "'Towards spike-based machine intelligence with neuromorphic computing', Roy et al 2019"){.link-annotated .id-not .include-annotation link-icon="pdf" link-icon-type="svg" link-icon-color="#f40f02"}** ## `“Checkmate: Breaking the Memory Wall With Optimal Tensor Rematerialization”, Jain et al 2019`{=html} {#jain-et-al-2019-section} **[`Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerialization`{=html}](https://arxiv.org/abs/1910.02653 "'Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerialization', Jain et al 2019"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Training Kinetics in 15 Minutes: Large-Scale Distributed Training on Videos”, Lin et al 2019`{=html} {#lin-et-al-2019-3-section} **[`Training Kinetics in 15 Minutes: Large-scale Distributed Training on Videos`{=html}](https://arxiv.org/abs/1910.00932#google "'Training Kinetics in 15 Minutes: Large-scale Distributed Training on Videos', Lin et al 2019"){.link-annotated .id-not .include-annotation link-icon="alphabet" link-icon-type="svg" link-icon-color="#4285f4"}** ## `“Energy and Policy Considerations for Deep Learning in NLP”, Strubell et al 2019`{=html} {#strubell-et-al-2019-section} **[`Energy and Policy Considerations for Deep Learning in NLP`{=html}](https://arxiv.org/abs/1906.02243 "'Energy and Policy Considerations for Deep Learning in NLP', Strubell et al 2019"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Large Batch Optimization for Deep Learning: Training BERT in 76 Minutes”, You et al 2019`{=html} {#you-et-al-2019-section} **[`Large Batch Optimization for Deep Learning: Training BERT in 76 minutes`{=html}](https://arxiv.org/abs/1904.00962#google "'Large Batch Optimization for Deep Learning: Training BERT in 76 minutes', You et al 2019"){.link-annotated .id-not .include-annotation link-icon="alphabet" link-icon-type="svg" link-icon-color="#4285f4"}** ## `“GAP: Generalizable Approximate Graph Partitioning Framework”, Nazi et al 2019`{=html} {#nazi-et-al-2019-section} **[`GAP: Generalizable Approximate Graph Partitioning Framework`{=html}](https://arxiv.org/abs/1903.00614#google "'GAP: Generalizable Approximate Graph Partitioning Framework', Nazi et al 2019"){.link-annotated .id-not .include-annotation link-icon="alphabet" link-icon-type="svg" link-icon-color="#4285f4"}** ## `“An Empirical Model of Large-Batch Training”, McCandlish et al 2018`{=html} {#mccandlish-et-al-2018-largebatchtraining-section} **[`An Empirical Model of Large-Batch Training`{=html}](https://arxiv.org/abs/1812.06162#openai "'An Empirical Model of Large-Batch Training', McCandlish et al 2018"){.link-annotated .id-not .include-annotation link-icon="openai" link-icon-type="svg"}** ## `“Bayesian Layers: A Module for Neural Network Uncertainty”, Tran et al 2018`{=html} {#tran-et-al-2018-section} **[`Bayesian Layers: A Module for Neural Network Uncertainty`{=html}](https://arxiv.org/abs/1812.03973#google "'Bayesian Layers: A Module for Neural Network Uncertainty', Tran et al 2018"){.link-annotated .id-not .include-annotation link-icon="alphabet" link-icon-type="svg" link-icon-color="#4285f4"}** ## `“GPipe: Efficient Training of Giant Neural Networks Using Pipeline Parallelism”, Huang et al 2018`{=html} {#huang-et-al-2018-3-section} **[`GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism`{=html}](https://arxiv.org/abs/1811.06965#google "'GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism', Huang et al 2018"){.link-annotated .id-not .include-annotation link-icon="alphabet" link-icon-type="svg" link-icon-color="#4285f4"}** ## `“Measuring the Effects of Data Parallelism on Neural Network Training”, Shallue et al 2018`{=html} {#shallue-et-al-2018-section} **[`Measuring the Effects of Data Parallelism on Neural Network Training`{=html}](https://arxiv.org/abs/1811.03600#google "'Measuring the Effects of Data Parallelism on Neural Network Training', Shallue et al 2018"){.link-annotated .id-not .include-annotation link-icon="alphabet" link-icon-type="svg" link-icon-color="#4285f4"}** ## `“Mesh-TensorFlow: Deep Learning for Supercomputers”, Shazeer et al 2018`{=html} {#shazeer-et-al-2018-section} **[`Mesh-TensorFlow: Deep Learning for Supercomputers`{=html}](https://arxiv.org/abs/1811.02084#google "'Mesh-TensorFlow: Deep Learning for Supercomputers', Shazeer et al 2018"){.link-annotated .id-not .include-annotation link-icon="alphabet" link-icon-type="svg" link-icon-color="#4285f4"}** ## `“There Is Plenty of Time at the Bottom: the Economics, Risk and Ethics of Time Compression”, Sandberg 2018`{=html} {#sandberg-2018-2-section} **[`There is plenty of time at the bottom: the economics, risk and ethics of time compression`{=html}](/doc/ai/scaling/hardware/2018-sandberg.pdf "'There is plenty of time at the bottom: the economics, risk and ethics of time compression', Sandberg 2018"){.link-annotated .id-not .include-annotation link-icon="pdf" link-icon-type="svg" link-icon-color="#f40f02"}** ## `“Highly Scalable Deep Learning Training System With Mixed-Precision: Training ImageNet in 4 Minutes”, Jia et al 2018`{=html} {#jia-et-al-2018-section} **[`Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in 4 Minutes`{=html}](https://arxiv.org/abs/1807.11205#tencent "'Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in 4 Minutes', Jia et al 2018"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“AI and Compute”, Amodei et al 2018`{=html} {#amodei-et-al-2018-section} **[`AI and Compute`{=html}](https://openai.com/research/ai-and-compute "'AI and Compute', Amodei et al 2018"){.link-annotated .id-not .include-annotation link-icon="openai" link-icon-type="svg"}** ## `“Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions”, Vasilache et al 2018`{=html} {#vasilache-et-al-2018-section} **[`Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions`{=html}](https://arxiv.org/abs/1802.04730 "'Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions', Vasilache et al 2018"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Loihi: A Neuromorphic Manycore Processor With On-Chip Learning”, Davies et al 2018`{=html} {#davies-et-al-2018-2-section} **[`Loihi: A Neuromorphic Manycore Processor with On-Chip Learning`{=html}](/doc/ai/scaling/hardware/2018-davies.pdf "'Loihi: A Neuromorphic Manycore Processor with On-Chip Learning', Davies et al 2018"){.link-annotated .id-not .include-annotation link-icon="pdf" link-icon-type="svg" link-icon-color="#f40f02"}** ## `“Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training”, Lin et al 2017`{=html} {#lin-et-al-2017-1-section} **[`Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training`{=html}](https://arxiv.org/abs/1712.01887 "'Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training', Lin et al 2017"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Mixed Precision Training”, Micikevicius et al 2017`{=html} {#micikevicius-et-al-2017-section} **[`Mixed Precision Training`{=html}](https://arxiv.org/abs/1710.03740#nvidia "'Mixed Precision Training', Micikevicius et al 2017"){.link-annotated .id-not .include-annotation link-icon="n" link-icon-type="text,sans,italic" link-icon-color="#77ba00"}** ## `“Apache Arrow and the ‘10 Things I Hate About Pandas’”, McKinney 2017`{=html} {#mckinney-2017-section} **[`Apache Arrow and the ‘10 Things I Hate About pandas’`{=html}](https://wesmckinney.com/blog/apache-arrow-pandas-internals/ "'Apache Arrow and the ‘10 Things I Hate About pandas’', McKinney 2017"){.link-annotated-partial .id-not .include-annotation}** ## `“On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima”, Keskar et al 2016`{=html} {#keskar-et-al-2016-section} **[`On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima`{=html}](https://arxiv.org/abs/1609.04836#intel "'On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima', Keskar et al 2016"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Training Deep Nets With Sublinear Memory Cost”, Chen et al 2016`{=html} {#chen-et-al-2016-4-section} **[`Training Deep Nets with Sublinear Memory Cost`{=html}](https://arxiv.org/abs/1604.06174 "'Training Deep Nets with Sublinear Memory Cost', Chen et al 2016"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“GeePS: Scalable Deep Learning on Distributed GPUs With a GPU-Specialized Parameter Server”, Cui et al 2016`{=html} {#cui-et-al-2016-section} **[`GeePS: scalable deep learning on distributed GPUs with a GPU-specialized parameter server`{=html}](/doc/ai/scaling/hardware/2016-cui.pdf "'GeePS: scalable deep learning on distributed GPUs with a GPU-specialized parameter server', Cui et al 2016"){.link-annotated .id-not .include-annotation link-icon="pdf" link-icon-type="svg" link-icon-color="#f40f02"}** ## `“Convolutional Networks for Fast, Energy-Efficient Neuromorphic Computing”, Esser et al 2016`{=html} {#esser-et-al-2016-section} **[`Convolutional Networks for Fast, Energy-Efficient Neuromorphic Computing`{=html}](https://arxiv.org/abs/1603.08270 "'Convolutional Networks for Fast, Energy-Efficient Neuromorphic Computing', Esser et al 2016"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Communication-Efficient Learning of Deep Networks from Decentralized Data”, McMahan et al 2016`{=html} {#mcmahan-et-al-2016-section} **[`Communication-Efficient Learning of Deep Networks from Decentralized Data`{=html}](https://arxiv.org/abs/1602.05629#google "'Communication-Efficient Learning of Deep Networks from Decentralized Data', McMahan et al 2016"){.link-annotated .id-not .include-annotation link-icon="alphabet" link-icon-type="svg" link-icon-color="#4285f4"}** ## `“Persistent RNNs: Stashing Recurrent Weights On-Chip”, Diamos et al 2016`{=html} {#diamos-et-al-2016-section} **[`Persistent RNNs: Stashing Recurrent Weights On-Chip`{=html}](/doc/ai/nn/rnn/2016-diamos.pdf#baidu "'Persistent RNNs: Stashing Recurrent Weights On-Chip', Diamos et al 2016"){.link-annotated .id-not .include-annotation link-icon="pdf" link-icon-type="svg" link-icon-color="#f40f02"}** ## `“The Brain As a Universal Learning Machine”, Cannell 2015`{=html} {#cannell-2015-section} **[`The Brain as a Universal Learning Machine`{=html}](https://www.lesswrong.com/posts/9Yc7Pp7szcjPgPsjf/the-brain-as-a-universal-learning-machine "'The Brain as a Universal Learning Machine', Cannell 2015"){.link-annotated .id-not .include-annotation link-icon="LW" link-icon-type="text" link-icon-color="#7faf83"}** ## `“Scaling Distributed Machine Learning With the Parameter Server”, Li et al 2014`{=html} {#li-et-al-2014-2-section} **[`Scaling Distributed Machine Learning with the Parameter Server`{=html}](/doc/ai/scaling/hardware/2014-li-2.pdf#google "'Scaling Distributed Machine Learning with the Parameter Server', Li et al 2014"){.link-annotated .id-not .include-annotation link-icon="alphabet" link-icon-type="svg" link-icon-color="#4285f4"}** ## `“Multi-Column Deep Neural Network for Traffic Sign Classification”, Cireşan et al 2012b`{=html} {#cireşan-et-al-2012b-section} **[`Multi-column deep neural network for traffic sign classification`{=html}](/doc/ai/nn/cnn/2012-ciresan-2.pdf "'Multi-column deep neural network for traffic sign classification', Cireşan et al 2012b"){.link-annotated .id-not .include-annotation link-icon="pdf" link-icon-type="svg" link-icon-color="#f40f02"}** ## `“Multi-Column Deep Neural Networks for Image Classification”, Cireşan et al 2012`{=html} {#ciresan-et-al-2012-1-section} **[`Multi-column Deep Neural Networks for Image Classification`{=html}](https://arxiv.org/abs/1202.2745#schmidhuber "'Multi-column Deep Neural Networks for Image Classification', Cireşan et al 2012"){.link-annotated .id-not .include-annotation link-icon="SMDH" link-icon-type="text,quad,sans"}** ## `“Building High-Level Features Using Large Scale Unsupervised Learning”, Le et al 2011`{=html} {#le-et-al-2011-section} **[`Building high-level features using large scale unsupervised learning`{=html}](https://arxiv.org/abs/1112.6209#google "'Building high-level features using large scale unsupervised learning', Le et al 2011"){.link-annotated .id-not .include-annotation link-icon="alphabet" link-icon-type="svg" link-icon-color="#4285f4"}** ## `“Implications of Historical Trends in the Electrical Efficiency of Computing”, Koomey et al 2011`{=html} {#koomey-et-al-2011-section} **[`Implications of Historical Trends in the Electrical Efficiency of Computing`{=html}](/doc/cs/hardware/2011-koomey.pdf "'Implications of Historical Trends in the Electrical Efficiency of Computing', Koomey et al 2011"){.link-annotated .id-not .include-annotation link-icon="pdf" link-icon-type="svg" link-icon-color="#f40f02"}** ## `“HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent”, Niu et al 2011`{=html} {#niu-et-al-2011-section} **[`HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent`{=html}](https://arxiv.org/abs/1106.5730 "'HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent', Niu et al 2011"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“DanNet: Flexible, High Performance Convolutional Neural Networks for Image Classification”, Ciresan et al 2011`{=html} {#ciresan-et-al-2011-section} **[`DanNet: Flexible, High Performance Convolutional Neural Networks for Image Classification`{=html}](https://arxiv.org/abs/1102.0183#schmidhuber "'DanNet: Flexible, High Performance Convolutional Neural Networks for Image Classification', Ciresan et al 2011"){.link-annotated .id-not .include-annotation link-icon="SMDH" link-icon-type="text,quad,sans"}** ## `“Goodbye 2010”, Legg 2010`{=html} {#legg-2010-section} **[`Goodbye 2010`{=html}](https://www.vetta.org/2010/12/goodbye-2010/ "'Goodbye 2010', Legg 2010"){.link-annotated .id-not .include-annotation link-icon="Legg" link-icon-type="text,quad,sans" link-icon-color="#4185f4"}** ## `“Deep Big Simple Neural Nets Excel on Handwritten Digit Recognition”, Ciresan et al 2010`{=html} {#ciresan-et-al-2010-section} **[`Deep Big Simple Neural Nets Excel on Handwritten Digit Recognition`{=html}](https://arxiv.org/abs/1003.0358#schmidhuber "'Deep Big Simple Neural Nets Excel on Handwritten Digit Recognition', Ciresan et al 2010"){.link-annotated .id-not .include-annotation link-icon="SMDH" link-icon-type="text,quad,sans"}** ## `“The Cat Is out of the Bag: Cortical Simulations With 109 Neurons, 1013 Synapses”, Ananthanarayanan et al 2009`{=html} {#ananthanarayanan-et-al-2009-section} **[`The cat is out of the bag: cortical simulations with 109 neurons, 1013 synapses`{=html}](/doc/ai/scaling/hardware/2009-ananthanarayanan.pdf "'The cat is out of the bag: cortical simulations with 109 neurons, 1013 synapses', Ananthanarayanan et al 2009"){.link-annotated .id-not .include-annotation link-icon="pdf" link-icon-type="svg" link-icon-color="#f40f02"}** ## `“Large-Scale Deep Unsupervised Learning Using Graphics Processors”, Raina et al 2009`{=html} {#raina-et-al-2009-section} **[`Large-scale deep unsupervised learning using graphics processors`{=html}](/doc/ai/scaling/hardware/2009-raina.pdf "'Large-scale deep unsupervised learning using graphics processors', Raina et al 2009"){.link-annotated .id-not .include-annotation link-icon="pdf" link-icon-type="svg" link-icon-color="#f40f02"}** ## `“Bandwidth Optimal All-Reduce Algorithms for Clusters of Workstations”, Patarasuk & Yuan 2009`{=html} {#patarasuk-yuan-2009-section} **[`Bandwidth optimal all-reduce algorithms for clusters of workstations`{=html}](/doc/ai/scaling/hardware/2009-patarasuk.pdf "'Bandwidth optimal all-reduce algorithms for clusters of workstations', Patarasuk & Yuan 2009"){.link-annotated .id-not .include-annotation link-icon="pdf" link-icon-type="svg" link-icon-color="#f40f02"}** ## `“Whole Brain Emulation: A Roadmap”`{=html} {#_2RRianzb-section} **[`Whole Brain Emulation: A Roadmap`{=html}](/doc/ai/scaling/hardware/2008-sandberg-wholebrainemulationroadmap.pdf "Whole Brain Emulation: A Roadmap"){.link-annotated-partial .id-not .include-annotation link-icon="pdf" link-icon-type="svg" link-icon-color="#f40f02"}** ## `“Moore’s Law and the Technology S-Curve”, Bowden 2004`{=html} {#bowden-2004-section} **[`Moore’s law and the Technology S-Curve`{=html}](/doc/technology/2004-bowden.pdf "'Moore’s law and the Technology S-Curve', Bowden 2004"){.link-annotated-partial .id-not .include-annotation link-icon="pdf" link-icon-type="svg" link-icon-color="#f40f02"}** ## `“GPU Implementation of Neural Networks”, Oh & Jung 2004`{=html} {#oh-jung-2004-section} **[`GPU implementation of neural networks`{=html}](/doc/ai/scaling/hardware/2004-oh.pdf "'GPU implementation of neural networks', Oh & Jung 2004"){.link-annotated .id-not .include-annotation link-icon="pdf" link-icon-type="svg" link-icon-color="#f40f02"}** ## `“DARPA and the Quest for Machine Intelligence, 1983–1993”, Roland & Shiman 2002`{=html} {#roland-shiman-2002-section} **[`DARPA and the Quest for Machine Intelligence, 1983–1993`{=html}](/doc/cs/hardware/2002-roland-strategiccomputing-darpaandthequestformachineintelligence19831993.pdf "'DARPA and the Quest for Machine Intelligence, 1983–1993', Roland & Shiman 2002"){.link-annotated .id-not .include-annotation link-icon="pdf" link-icon-type="svg" link-icon-color="#f40f02"}** ## `“Ultimate Physical Limits to Computation”, Lloyd 1999`{=html} {#lloyd-1999-section} **[`Ultimate physical limits to computation`{=html}](https://arxiv.org/abs/quant-ph/9908043 "'Ultimate physical limits to computation', Lloyd 1999"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Matrioshka Brains”`{=html} {#_pDE3pgEI-section} **[`Matrioshka Brains`{=html}](/doc/ai/scaling/hardware/1999-bradbury-matrioshkabrains.pdf "Matrioshka Brains"){.link-annotated-partial .id-not .include-annotation link-icon="pdf" link-icon-type="svg" link-icon-color="#f40f02"}** ## `“When Will Computer Hardware Match the Human Brain?”, Moravec 1998`{=html} {#moravec-1998-section} **[`When will computer hardware match the human brain?`{=html}](https://jetpress.org/volume1/moravec.htm "'When will computer hardware match the human brain?', Moravec 1998"){.link-annotated .id-not .include-annotation}** ## `“Superhumanism: According to Hans Moravec § AI Scaling”, Platt 1995`{=html} {#platt-1995-1-section} **[`Superhumanism: According to Hans Moravec § AI Scaling`{=html}](https://www.wired.com/1995/10/moravec/#scaling "'Superhumanism: According to Hans Moravec § AI Scaling', Platt 1995"){.link-annotated .id-not .include-annotation link-icon="wired" link-icon-type="svg"}** ## `“A Sociological Study of the Official History of the Perceptrons Controversy [1993]”, Olazaran 1993`{=html} {#olazaran-1993-section} **[`A Sociological Study of the Official History of the Perceptrons Controversy [1993]`{=html}](/doc/ai/nn/1993-olazaran.pdf "'A Sociological Study of the Official History of the Perceptrons Controversy [1993]', Olazaran 1993"){.link-annotated .id-not .include-annotation link-icon="pdf" link-icon-type="svg" link-icon-color="#f40f02"}** ## `“What Is Massively Parallel Computing, and Why Is It Important?”, Hillis 1992`{=html} {#hillis-1992-section} **[`What Is Massively Parallel Computing, and Why Is It Important?`{=html}](/doc/ai/scaling/hardware/1992-hillis.pdf "'What Is Massively Parallel Computing, and Why Is It Important?', Hillis 1992"){.link-annotated-partial .id-not .include-annotation link-icon="pdf" link-icon-type="svg" link-icon-color="#f40f02"}** ## `“Intelligence As an Emergent Behavior; Or, The Songs of Eden”, Hillis 1988`{=html} {#hillis-1988-section} **[`Intelligence as an Emergent Behavior; or, The Songs of Eden`{=html}](/doc/ai/scaling/hardware/1988-hillis.pdf "'Intelligence as an Emergent Behavior; or, The Songs of Eden', Hillis 1988"){.link-annotated-partial .id-not .include-annotation link-icon="pdf" link-icon-type="svg" link-icon-color="#f40f02"}** ## `“Computing With Connections”, Sejnowski 1987`{=html} {#sejnowski-1987-section} **[`Computing with Connections`{=html}](/doc/ai/scaling/hardware/1987-sejnowski.pdf "'Computing with Connections', Sejnowski 1987"){.link-annotated .id-not .include-annotation link-icon="pdf" link-icon-type="svg" link-icon-color="#f40f02"}** ## `“The Role Of RAW POWER In INTELLIGENCE”, Moravec 1976`{=html} {#moravec-1976-section} **[`The Role Of RAW POWER In INTELLIGENCE`{=html}](https://web.archive.org/web/20230710000944/https://frc.ri.cmu.edu/~hpm/project.archive/general.articles/1975/Raw.Power.html "'The Role Of RAW POWER In INTELLIGENCE', Moravec 1976"){.link-annotated .id-not .include-annotation link-icon="internet-archive" link-icon-type="svg"}** ## `“Brain Performance in FLOPS”`{=html} {#_UIMUHPS4-section} **[`Brain performance in FLOPS`{=html}](https://aiimpacts.org/brain-performance-in-flops/ "Brain performance in FLOPS"){.link-annotated-partial .id-not .include-annotation link-icon="AII" link-icon-type="text,tri" link-icon-color="#2396ce"}** ## `“Google Demonstrates Leading Performance in Latest MLPerf Benchmarks”`{=html} {#_F1v3DtXz-section} **[`Google demonstrates leading performance in latest MLPerf Benchmarks`{=html}](https://blog.tensorflow.org/2021/06/google-demonstrates-leading-performance-in-latest-MLPerf-benchmarks.html "Google demonstrates leading performance in latest MLPerf Benchmarks"){.link-annotated-partial .id-not .include-annotation link-icon="alphabet" link-icon-type="svg" link-icon-color="#4285f4"}** ## `“H100 GPUs Set Standard for Gen AI in Debut MLPerf Benchmark”`{=html} {#_NsVD3pud-section} **[`H100 GPUs Set Standard for Gen AI in Debut MLPerf Benchmark`{=html}](https://blogs.nvidia.com/blog/generative-ai-debut-mlperf/ "H100 GPUs Set Standard for Gen AI in Debut MLPerf Benchmark"){.id-not .include-annotation link-icon="n" link-icon-type="text,sans,italic" link-icon-color="#77ba00"}** ## `“Introducing Cerebras Inference: AI at Instant Speed”, Cerebras 2026`{=html} {#_5l8Mwa9K-section} **[`Introducing Cerebras Inference: AI at Instant Speed`{=html}](https://cerebras.ai/blog/introducing-cerebras-inference-ai-at-instant-speed "'Introducing Cerebras Inference: AI at Instant Speed', Cerebras 2026"){.link-annotated-partial .id-not .include-annotation link-icon="C" link-icon-type="text,sans" link-icon-color="#f05a29"}** ## `“Llama-3.1-405B Now Runs at 969 Tokens/s on Cerebras Inference”`{=html} {#_C5LkfXii-section} **[`Llama-3.1-405B now runs at 969 tokens/s on Cerebras Inference`{=html}](https://cerebras.ai/blog/llama-405b-inference "Llama-3.1-405B now runs at 969 tokens/s on Cerebras Inference"){.link-annotated-partial .id-not .include-annotation link-icon="C" link-icon-type="text,sans" link-icon-color="#f05a29"}** ## `“Cerebras Systems Unveils World’s Fastest AI Chip With Whopping 4 Trillion Transistors”, Cerebras 2026`{=html} {#_Gx4txv-Y-section} **[`Cerebras Systems Unveils World’s Fastest AI Chip with Whopping 4 Trillion Transistors`{=html}](https://cerebras.ai/press-release/cerebras-announces-third-generation-wafer-scale-engine "'Cerebras Systems Unveils World’s Fastest AI Chip with Whopping 4 Trillion Transistors', Cerebras 2026"){.link-annotated-partial .id-not .include-annotation link-icon="C" link-icon-type="text,sans" link-icon-color="#f05a29"}** ## `“Crusoe Expands AI Data Center Campus in Abilene to 1.2 Gigawatts”`{=html} {#_dp0p43cv-section} **[`Crusoe Expands AI Data Center Campus in Abilene to 1.2 Gigawatts`{=html}](https://crusoe.ai/newsroom/crusoe-expands-ai-data-center-campus-in-abilene-to-1-2-gigawatts "Crusoe Expands AI Data Center Campus in Abilene to 1.2 Gigawatts"){.id-not .include-annotation}** ## `“NVIDIA Hopper Architecture In-Depth”`{=html} {#_qLwc3l4h-section} **[`NVIDIA Hopper Architecture In-Depth`{=html}](https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/ "NVIDIA Hopper Architecture In-Depth"){.id-not .include-annotation link-icon="n" link-icon-type="text,sans,italic" link-icon-color="#77ba00"}** ## `“AI Progress Is about to Speed Up”`{=html} {#_5a1B6TXy-section} **[`AI progress is about to speed up`{=html}](https://epoch.ai/gradient-updates/ai-progress-is-about-to-speed-up "AI progress is about to speed up"){.link-annotated-partial .id-not .include-annotation}** ## `“Trends in GPU Price-Performance”`{=html} {#_txo6bG9f-section} **[`Trends in GPU Price-Performance`{=html}](https://epochai.org/blog/trends-in-gpu-price-performance "Trends in GPU Price-Performance"){.link-annotated-partial .id-not .include-annotation}** ## `“NVIDIA/Megatron-LM: Ongoing Research Training Transformer Models at Scale”`{=html} {#_ADbPyrQV-section} **[`NVIDIA/Megatron-LM: Ongoing research training transformer models at scale`{=html}](https://github.com/nvidia/megatron-lm "NVIDIA/Megatron-LM: Ongoing research training transformer models at scale"){.link-annotated-partial .id-not .include-annotation link-icon="n" link-icon-type="text,sans,italic" link-icon-color="#77ba00"}** ## `“12 Hours Later, Groq Deploys Llama-3-Instruct (8 & 70B)”`{=html} {#_xEvUiyhv-section} **[`12 Hours Later, Groq Deploys Llama-3-instruct (8 & 70B)`{=html}](https://groq.com/12-hours-later-groq-is-running-llama-3-instruct-8-70b-by-meta-ai-on-its-lpu-inference-enginge/ "12 Hours Later, Groq Deploys Llama-3-instruct (8 & 70B)"){.link-annotated-partial .id-not .include-annotation}** ## `“The Technology Behind BLOOM Training”`{=html} {#_4Gg1tdoa-section} **[`The Technology Behind BLOOM Training`{=html}](https://huggingface.co/blog/bloom-megatron-deepspeed "The Technology Behind BLOOM Training"){.id-not .include-annotation link-icon="🤗" link-icon-type="text" link-icon-color="#ffcc4d"}** ## `“From Bare Metal to a 70B Model: Infrastructure Set-Up and Scripts”`{=html} {#_fn7M8Log-section} **[`From bare metal to a 70B model: infrastructure set-up and scripts`{=html}](https://imbue.com/research/70b-infrastructure/ "From bare metal to a 70B model: infrastructure set-up and scripts"){.id-not .include-annotation}** ## `“AI Accelerators, Part IV: The Very Rich Landscape”, Fuchs 2026`{=html} {#_SYS8xHne-section} **[`AI Accelerators, Part IV: The Very Rich Landscape`{=html}](https://medium.com/@adi.fu7/ai-accelerators-part-iv-the-very-rich-landscape-17481be80917 "'AI Accelerators, Part IV: The Very Rich Landscape', Fuchs 2026"){.id-not .include-annotation link-icon="𝐌" link-icon-type="text"}** ## `“Cerebras’s Next Generation CS-4: Fast Just Got Faster”`{=html} {#_vSPVvmtR-section} **[`Cerebras’s Next Generation CS-4: Fast Just Got Faster`{=html}](https://newsletter.semianalysis.com/p/cerebrass-next-generation-cs-4-fast "Cerebras’s Next Generation CS-4: Fast Just Got Faster"){.link-modified-recently .link-annotated-partial .id-not .include-annotation link-icon="C" link-icon-type="text,sans" link-icon-color="#f05a29"}** ## `“NVIDIA Announces DGX H100 Systems: World’s Most Advanced Enterprise AI Infrastructure”`{=html} {#_DQMWk-Is-section} **[`NVIDIA Announces DGX H100 Systems: World’s Most Advanced Enterprise AI Infrastructure`{=html}](https://nvidianews.nvidia.com/news/nvidia-announces-dgx-h100-systems-worlds-most-advanced-enterprise-ai-infrastructure "NVIDIA Announces DGX H100 Systems: World’s Most Advanced Enterprise AI Infrastructure"){.id-not .include-annotation link-icon="n" link-icon-type="text,sans,italic" link-icon-color="#77ba00"}** ## `“NVIDIA Launches UK’s Most Powerful Supercomputer, for Research in AI and Healthcare”`{=html} {#_hvvyrWeb-section} **[`NVIDIA Launches UK’s Most Powerful Supercomputer, for Research in AI and Healthcare`{=html}](https://nvidianews.nvidia.com/news/nvidia-launches-uks-most-powerful-supercomputer-for-research-in-ai-and-healthcare "NVIDIA Launches UK’s Most Powerful Supercomputer, for Research in AI and Healthcare"){.id-not .include-annotation link-icon="n" link-icon-type="text,sans,italic" link-icon-color="#77ba00"}** ## `“Perlmutter, Said to Be the World’s Fastest AI Supercomputer, Comes Online”`{=html} {#_9xrinzX_-section} **[`Perlmutter, said to be the world’s fastest AI supercomputer, comes online`{=html}](https://siliconangle.com/2021/05/27/perlmutter-said-worlds-fastest-ai-supercomputer-comes-online/ "Perlmutter, said to be the world’s fastest AI supercomputer, comes online"){.link-annotated-partial .id-not .include-annotation}** ## `“TensorFlow Research Cloud (TRC): Accelerate Your Cutting-Edge Machine Learning Research With Free Cloud TPUs”, TRC 2026`{=html} {#_dpuECoS8-section} **[`TensorFlow Research Cloud (TRC): Accelerate your cutting-edge machine learning research with free Cloud TPUs`{=html}](https://sites.research.google/trc/ "'TensorFlow Research Cloud (TRC): Accelerate your cutting-edge machine learning research with free Cloud TPUs', TRC 2026"){.link-annotated .id-not .include-annotation link-icon="alphabet" link-icon-type="svg" link-icon-color="#4285f4"}** ## `“Cerebras’ Tech Trains "Brain-Scale" AIs”`{=html} {#_sb6D26sP-section} **[`Cerebras’ Tech Trains "Brain-Scale" AIs`{=html}](https://spectrum.ieee.org/cerebras-ai-computers "Cerebras’ Tech Trains "Brain-Scale" AIs"){.link-annotated-partial .id-not .include-annotation link-icon="C" link-icon-type="text,sans" link-icon-color="#f05a29"}** ## `“Fugaku Holds Top Spot, Exascale Remains Elusive”`{=html} {#_tR7EyLsM-section} **[`Fugaku Holds Top Spot, Exascale Remains Elusive`{=html}](https://top500.org/news/fugaku-holds-top-spot-exascale-remains-elusive/ "Fugaku Holds Top Spot, Exascale Remains Elusive"){.link-annotated-partial .id-not .include-annotation}** ## `“342 Transistors for Every Person In the World: Cerebras 2nd Gen Wafer Scale Engine Teased”`{=html} {#_5kXuF4uw-section} **[`342 Transistors for Every Person In the World: Cerebras 2nd Gen Wafer Scale Engine Teased`{=html}](https://www.anandtech.com/show/16000/342-transistors-for-every-person-in-the-world-cerebras-2nd-gen-wafer-scale-engine-teased "342 Transistors for Every Person In the World: Cerebras 2nd Gen Wafer Scale Engine Teased"){.link-annotated-partial .id-not .include-annotation link-icon="C" link-icon-type="text,sans" link-icon-color="#f05a29"}** ## `“Jim Keller Becomes CTO at Tenstorrent: ‘The Most Promising Architecture Out There’”`{=html} {#_NNdG-CCr-section} **[`Jim Keller Becomes CTO at Tenstorrent: ‘The Most Promising Architecture Out There’`{=html}](https://www.anandtech.com/show/16354/jim-keller-becomes-cto-at-tenstorrent-the-most-promising-architecture-out-there "Jim Keller Becomes CTO at Tenstorrent: ‘The Most Promising Architecture Out There’"){.link-annotated-partial .id-not .include-annotation}** ## `“NVIDIA Unveils Grace: A High-Performance Arm Server CPU For Use In Big AI Systems”`{=html} {#_CAIYcUU9-section} **[`NVIDIA Unveils Grace: A High-Performance Arm Server CPU For Use In Big AI Systems`{=html}](https://www.anandtech.com/show/16610/nvidia-unveils-grace-a-highperformance-arm-server-cpu-for-use-in-ai-systems "NVIDIA Unveils Grace: A High-Performance Arm Server CPU For Use In Big AI Systems"){.link-annotated-partial .id-not .include-annotation link-icon="n" link-icon-type="text,sans,italic" link-icon-color="#77ba00"}** ## `“Cerebras Unveils Wafer Scale Engine Two (WSE2): 2.6 Trillion Transistors, 100% Yield”`{=html} {#_o3XHDPd2-section} **[`Cerebras Unveils Wafer Scale Engine Two (WSE2): 2.6 Trillion Transistors, 100% Yield`{=html}](https://www.anandtech.com/show/16626/cerebras-unveils-wafer-scale-engine-two-wse2-26-trillion-transistors-100-yield "Cerebras Unveils Wafer Scale Engine Two (WSE2): 2.6 Trillion Transistors, 100% Yield"){.link-annotated-partial .id-not .include-annotation link-icon="C" link-icon-type="text,sans" link-icon-color="#f05a29"}** ## `“AMD Announces Instinct MI200 Accelerator Family: Taking Servers to Exascale and Beyond”`{=html} {#_C-XoiAv0-section} **[`AMD Announces Instinct MI200 Accelerator Family: Taking Servers to Exascale and Beyond`{=html}](https://www.anandtech.com/show/17054/amd-announces-instinct-mi200-accelerator-family-cdna2-exacale-servers "AMD Announces Instinct MI200 Accelerator Family: Taking Servers to Exascale and Beyond"){.link-annotated-partial .id-not .include-annotation}** ## `“NVIDIA Hopper GPU Architecture and H100 Accelerator Announced: Working Smarter and Harder”`{=html} {#_96-j__XI-section} **[`NVIDIA Hopper GPU Architecture and H100 Accelerator Announced: Working Smarter and Harder`{=html}](https://www.anandtech.com/show/17327/nvidia-hopper-gpu-architecture-and-h100-accelerator-announced "NVIDIA Hopper GPU Architecture and H100 Accelerator Announced: Working Smarter and Harder"){.link-annotated-partial .id-not .include-annotation link-icon="n" link-icon-type="text,sans,italic" link-icon-color="#77ba00"}** ## `“Biological Anchors: A Trick That Might Or Might Not Work”`{=html} {#_J-lieiEU-section} **[`Biological Anchors: A Trick That Might Or Might Not Work`{=html}](https://www.astralcodexten.com/p/biological-anchors-a-trick-that-might "Biological Anchors: A Trick That Might Or Might Not Work"){.id-not .include-annotation link-icon="SSC" link-icon-type="text,tri" link-icon-color="#5175c2"}** ## `“Accelerating GPT-5.6 Sol Ultrafast”, Cerebras 2026`{=html} {#_UPn_1wZy-section} **[`Accelerating GPT-5.6 Sol Ultrafast`{=html}](https://www.cerebras.ai/blog/accelerating-gpt-5-6-sol-ultrafast-with-openai "'Accelerating GPT-5.6 Sol Ultrafast', Cerebras 2026"){.link-modified-recently .link-annotated-partial .id-not .include-annotation link-icon="C" link-icon-type="text,sans" link-icon-color="#f05a29"}** ## `“Scaling Up and Out: Training Massive Models on Cerebras Systems Using Weight Streaming”`{=html} {#_WWfNe-zN-section} **[`Scaling Up and Out: Training Massive Models on Cerebras Systems using Weight Streaming`{=html}](https://www.cerebras.net/blog/scaling-up-and-out-training-massive-models-on-cerebras-systems-using-weight-streaming/ "Scaling Up and Out: Training Massive Models on Cerebras Systems using Weight Streaming"){.id-not .include-annotation link-icon="C" link-icon-type="text,sans" link-icon-color="#f05a29"}** ::: aux-links-transclude-file ::: collapse **View External Link**: [`https://www.cerebras.net/blog/scaling-up-and-out-training-massive-models-on-cerebras-systems-using-weight-streaming/`](https://www.cerebras.net/blog/scaling-up-and-out-training-massive-models-on-cerebras-systems-using-weight-streaming/){.id-not .link-annotated-not .include-content .include-lazy link-icon="C" link-icon-type="text,sans" link-icon-color="#f05a29"} ::: ::: ## `“DeepSeek: The Quiet Giant Leading China’s AI Race”`{=html} {#_W0ma9R25-section} **[`DeepSeek: The Quiet Giant Leading China’s AI Race`{=html}](https://www.chinatalk.media/p/deepseek-ceo-interview-with-chinas "DeepSeek: The Quiet Giant Leading China’s AI Race"){.link-annotated-partial .id-not .include-annotation link-icon="deepseek" link-icon-type="svg" link-icon-color="#4d6bfe"}** ## `“DeepSeek: The View from China”`{=html} {#_gUsHoTwI-section} **[`DeepSeek: The View from China`{=html}](https://www.chinatalk.media/p/deepseek-the-view-from-china "DeepSeek: The View from China"){.link-annotated-partial .id-not .include-annotation link-icon="deepseek" link-icon-type="svg" link-icon-color="#4d6bfe"}** ## `“DeepSeek’s Edge”`{=html} {#_Z53t3Ypg-section} **[`DeepSeek’s Edge`{=html}](https://www.chinatalk.media/p/deepseeks-edge "DeepSeek’s Edge"){.link-annotated-partial .id-not .include-annotation link-icon="deepseek" link-icon-type="svg" link-icon-color="#4d6bfe"}** ## `“Fermi Estimate of Future Training Runs”`{=html} {#_t70R8K5C-section} **[`Fermi estimate of future training runs`{=html}](https://www.danieldewey.net/risk/estimates.html "Fermi estimate of future training runs"){.link-annotated-partial .id-not .include-annotation}** ## `“Carl Shulman #2: AI Takeover, Bio & Cyber Attacks, Detecting Deception, & Humanity's Far Future”`{=html} {#_4g8eBGfs-section} **[`Carl Shulman #2: AI Takeover, Bio & Cyber Attacks, Detecting Deception, & Humanity's Far Future`{=html}](https://www.dwarkesh.com/p/carl-shulman-2 "Carl Shulman #2: AI Takeover, Bio & Cyber Attacks, Detecting Deception, & Humanity's Far Future"){.id-not .include-annotation link-icon="Dwkh" link-icon-type="text,sans,quad" link-icon-color="#f3c016"}** ## `“Verifiable Compute Press Release”`{=html} {#_LxNznYFS-section} **[`Verifiable Compute Press Release`{=html}](https://www.eqtylab.io/blog/verifiable-compute-press-release "Verifiable Compute Press Release"){.link-annotated-partial .id-not .include-annotation}** ## `“Etched Is Making the Biggest Bet in AI”`{=html} {#_fkFvJXqO-section} **[`Etched is Making the Biggest Bet in AI`{=html}](https://www.etched.com/announcing-etched "Etched is Making the Biggest Bet in AI"){.link-annotated-partial .id-not .include-annotation}** ## `“The Emerging Age of AI Diplomacy: To Compete With China, the United States Must Walk a Tightrope in the Gulf”`{=html} {#_t2xXeCtn-section} **[`The Emerging Age of AI Diplomacy: To Compete With China, the United States Must Walk a Tightrope in the Gulf`{=html}](https://www.foreignaffairs.com/united-states/emerging-age-ai-diplomacy "The Emerging Age of AI Diplomacy: To Compete With China, the United States Must Walk a Tightrope in the Gulf"){.id-not .include-annotation}** ## `“How Far Can AI Progress Before Hitting Effective Physical Limits?”`{=html} {#_p0WghLg--section} **[`How Far Can AI Progress Before Hitting Effective Physical Limits?`{=html}](https://www.forethought.org/research/how-far-can-ai-progress-before-hitting-effective-physical-limits "How Far Can AI Progress Before Hitting Effective Physical Limits?"){.id-not .include-annotation}** ## `“The Resilience Myth: Fatal Flaws in the Push to Secure Chip Supply Chains”`{=html} {#_oqazTC1x-section} **[`The resilience myth: fatal flaws in the push to secure chip supply chains`{=html}](https://www.ft.com/content/f76534bf-b501-4cbf-9a46-80be9feb670c "The resilience myth: fatal flaws in the push to secure chip supply chains"){.link-annotated-partial .id-not .include-annotation link-icon="FT" link-icon-type="text" link-icon-color="#e3b68e"}** ## `“Prime Minister Sets out Blueprint to Turbocharge AI”`{=html} {#_ULPFV8Hu-section} **[`Prime Minister sets out blueprint to turbocharge AI`{=html}](https://www.gov.uk/government/news/prime-minister-sets-out-blueprint-to-turbocharge-ai "Prime Minister sets out blueprint to turbocharge AI"){.link-annotated-partial .id-not .include-annotation}** ## `“Compute Funds and Pre-Trained Models”`{=html} {#_Cm1pTGat-section} **[`Compute Funds and Pre-trained Models`{=html}](https://www.governance.ai/post/compute-funds-and-pre-trained-models "Compute Funds and Pre-trained Models"){.link-annotated-partial .id-not .include-annotation}** ## `“The Next Big Thing: Introducing IPU-POD128 and IPU-POD256”`{=html} {#_8kaNvQOj-section} **[`The Next Big Thing: Introducing IPU-POD128 and IPU-POD256`{=html}](https://www.graphcore.ai/posts/the-next-big-thing-introducing-ipu-pod128-and-ipu-pod256 "The Next Big Thing: Introducing IPU-POD128 and IPU-POD256"){.link-annotated-partial .id-not .include-annotation}** ## `“The WoW Factor: Graphcore Systems Get Huge Power and Efficiency Boost”`{=html} {#_ElmGXFtu-section} **[`The WoW Factor: Graphcore systems get huge power and efficiency boost`{=html}](https://www.graphcore.ai/posts/the-wow-factor-graphcore-systems-get-huge-power-and-efficiency-boost "The WoW Factor: Graphcore systems get huge power and efficiency boost"){.link-annotated-partial .id-not .include-annotation}** ## `“AWS Enables 4,000-GPU UltraClusters With New P4 A100 Instances”`{=html} {#_CLKtFsJV-section} **[`AWS Enables 4,000-GPU UltraClusters with New P4 A100 Instances`{=html}](https://www.hpcwire.com/2020/11/02/aws-ultraclusters-with-new-p4-a100-instances/ "AWS Enables 4,000-GPU UltraClusters with New P4 A100 Instances"){.link-annotated-partial .id-not .include-annotation}** ## `“Estimating Training Compute of Deep Learning Models”`{=html} {#_k64kG7GD-section} **[`Estimating training compute of Deep Learning models`{=html}](https://www.lesswrong.com/posts/HvqQm6o8KnwxbdmhZ/estimating-training-compute-of-deep-learning-models "Estimating training compute of Deep Learning models"){.id-not .include-annotation link-icon="LW" link-icon-type="text" link-icon-color="#7faf83"}** ::: aux-links-transclude-file ::: collapse **View External Link**: [`https://www.lesswrong.com/posts/HvqQm6o8KnwxbdmhZ/estimating-training-compute-of-deep-learning-models`](https://www.lesswrong.com/posts/HvqQm6o8KnwxbdmhZ/estimating-training-compute-of-deep-learning-models){.id-not .link-annotated-not .include-content .include-lazy link-icon="LW" link-icon-type="text" link-icon-color="#7faf83"} ::: ::: ## `“The Case for Fine-Grained Tracking of Compute for AI”`{=html} {#_-fMwKy5z-section} **[`The case for fine-grained tracking of compute for AI`{=html}](https://www.lesswrong.com/posts/LFkzAxGCzEz2r4CtE/the-case-for-fine-grained-tracking-of-compute-for-ai "The case for fine-grained tracking of compute for AI"){.link-annotated-partial .id-not .include-annotation link-icon="LW" link-icon-type="text" link-icon-color="#7faf83"}** ## `“What O3 Becomes by 2028”, Nesov 2026`{=html} {#_XD67Dc6j-section} **[`What o3 Becomes by 2028`{=html}](https://www.lesswrong.com/posts/NXTkEiaLA4JdS5vSZ/what-o3-becomes-by-2028 "'What o3 Becomes by 2028', Nesov 2026"){.link-annotated-partial .id-not .include-annotation link-icon="LW" link-icon-type="text" link-icon-color="#7faf83"}** ## `“The Colliding Exponentials of AI”`{=html} {#_zhDHJ3Yi-section} **[`The Colliding Exponentials of AI`{=html}](https://www.lesswrong.com/posts/QWuegBA9kGBv3xBFy/the-colliding-exponentials-of-ai "The Colliding Exponentials of AI"){.id-not .include-annotation link-icon="LW" link-icon-type="text" link-icon-color="#7faf83"}** ::: aux-links-transclude-file ::: collapse **View External Link**: [`https://www.lesswrong.com/posts/QWuegBA9kGBv3xBFy/the-colliding-exponentials-of-ai`](https://www.lesswrong.com/posts/QWuegBA9kGBv3xBFy/the-colliding-exponentials-of-ai){.id-not .link-annotated-not .include-content .include-lazy link-icon="LW" link-icon-type="text" link-icon-color="#7faf83"} ::: ::: ## `“Moore’s Law, AI, and the pace of Progress”`{=html} {#_0ZB2-MBI-section} **[`Moore’s Law, AI, and the pace of progress`{=html}](https://www.lesswrong.com/posts/aNAFrGbzXddQBMDqh/moore-s-law-ai-and-the-pace-of-progress "Moore’s Law, AI, and the pace of progress"){.id-not .include-annotation link-icon="LW" link-icon-type="text" link-icon-color="#7faf83"}** ::: aux-links-transclude-file ::: collapse **View External Link**: [`https://www.lesswrong.com/posts/aNAFrGbzXddQBMDqh/moore-s-law-ai-and-the-pace-of-progress`](https://www.lesswrong.com/posts/aNAFrGbzXddQBMDqh/moore-s-law-ai-and-the-pace-of-progress){.id-not .link-annotated-not .include-content .include-lazy link-icon="LW" link-icon-type="text" link-icon-color="#7faf83"} ::: ::: ## `“Introducing LIMBO: Managing the Simulation to Maintain Optimal P(DOOM)”`{=html} {#_qX47n6I7-section} **[`Introducing LIMBO: Managing the Simulation to Maintain Optimal P(DOOM)`{=html}](https://www.lesswrong.com/posts/f7YxHQHedsPdWywCG/introducing-limbo-maintaining-optimal-p-doom-and-a-call-for "Introducing LIMBO: Managing the Simulation to Maintain Optimal P(DOOM)"){.link-annotated-partial .id-not .include-annotation link-icon="LW" link-icon-type="text" link-icon-color="#7faf83"}** ## `“How Fast Can We Perform a Forward Pass?”`{=html} {#_rxjEVj3b-section} **[`How fast can we perform a forward pass?`{=html}](https://www.lesswrong.com/posts/gPmGTND8Kroxgpgsn/how-fast-can-we-perform-a-forward-pass "How fast can we perform a forward pass?"){.id-not .include-annotation link-icon="LW" link-icon-type="text" link-icon-color="#7faf83"}** ::: aux-links-transclude-file ::: collapse **View External Link**: [`https://www.lesswrong.com/posts/gPmGTND8Kroxgpgsn/how-fast-can-we-perform-a-forward-pass`](https://www.lesswrong.com/posts/gPmGTND8Kroxgpgsn/how-fast-can-we-perform-a-forward-pass){.id-not .link-annotated-not .include-content .include-lazy link-icon="LW" link-icon-type="text" link-icon-color="#7faf83"} ::: ::: ## `“"AI and Compute" Trend Isn’t Predictive of What Is Happening”`{=html} {#_g3wbstCX-section} **[`"AI and Compute" trend isn’t predictive of what is happening`{=html}](https://www.lesswrong.com/posts/wfpdejMWog4vEDLDg/ai-and-compute-trend-isn-t-predictive-of-what-is-happening ""AI and Compute" trend isn’t predictive of what is happening"){.id-not .include-annotation link-icon="LW" link-icon-type="text" link-icon-color="#7faf83"}** ::: aux-links-transclude-file ::: collapse **View External Link**: [`https://www.lesswrong.com/posts/wfpdejMWog4vEDLDg/ai-and-compute-trend-isn-t-predictive-of-what-is-happening`](https://www.lesswrong.com/posts/wfpdejMWog4vEDLDg/ai-and-compute-trend-isn-t-predictive-of-what-is-happening){.id-not .link-annotated-not .include-content .include-lazy link-icon="LW" link-icon-type="text" link-icon-color="#7faf83"} ::: ::: ## `“Brain Efficiency: Much More Than You Wanted to Know”`{=html} {#_NeP6pHLP-section} **[`Brain Efficiency: Much More than You Wanted to Know`{=html}](https://www.lesswrong.com/posts/xwBuoE9p8GE7RAuhd/brain-efficiency-much-more-than-you-wanted-to-know "Brain Efficiency: Much More than You Wanted to Know"){.id-not .include-annotation link-icon="LW" link-icon-type="text" link-icon-color="#7faf83"}** ::: aux-links-transclude-file ::: collapse **View External Link**: [`https://www.lesswrong.com/posts/xwBuoE9p8GE7RAuhd/brain-efficiency-much-more-than-you-wanted-to-know`](https://www.lesswrong.com/posts/xwBuoE9p8GE7RAuhd/brain-efficiency-much-more-than-you-wanted-to-know){.id-not .link-annotated-not .include-content .include-lazy link-icon="LW" link-icon-type="text" link-icon-color="#7faf83"} ::: ::: ## `“DeepSpeed: Accelerating Large-Scale Model Inference and Training via System Optimizations and Compression”`{=html} {#_IfgwBSeg-section} **[`DeepSpeed: Accelerating large-scale model inference and training via system optimizations and compression`{=html}](https://www.microsoft.com/en-us/research/blog/deepspeed-accelerating-large-scale-model-inference-and-training-via-system-optimizations-and-compression/ "DeepSpeed: Accelerating large-scale model inference and training via system optimizations and compression"){.link-annotated-partial .id-not .include-annotation link-icon="MS" link-icon-type="text,sans,italic"}** ## `“ZeRO-Infinity and DeepSpeed: Unlocking Unprecedented Model Scale for Deep Learning Training”`{=html} {#_OwsPppLM-section} **[`ZeRO-Infinity and DeepSpeed: Unlocking unprecedented model scale for deep learning training`{=html}](https://www.microsoft.com/en-us/research/blog/zero-infinity-and-deepspeed-unlocking-unprecedented-model-scale-for-deep-learning-training/ "ZeRO-Infinity and DeepSpeed: Unlocking unprecedented model scale for deep learning training"){.link-annotated-partial .id-not .include-annotation link-icon="MS" link-icon-type="text,sans,italic"}** ## `“The World’s Largest Computer Chip”`{=html} {#_qhHzblLK-section} **[`The World’s Largest Computer Chip`{=html}](https://www.newyorker.com/tech/annals-of-technology/the-worlds-largest-computer-chip "The World’s Largest Computer Chip"){.id-not .include-annotation link-icon="the-new-yorker" link-icon-type="svg"}** ## `“The Billion Dollar AI Problem That Just Keeps Scaling”`{=html} {#_TOptn_j7-section} **[`The Billion Dollar AI Problem That Just Keeps Scaling`{=html}](https://www.nextplatform.com/2021/02/11/the-billion-dollar-ai-problem-that-just-keeps-scaling/ "The Billion Dollar AI Problem That Just Keeps Scaling"){.link-annotated-partial .id-not .include-annotation link-icon="NEXT" link-icon-type="text,quad,sans" link-icon-color="#ff7200"}** ## `“TSMC Confirms 3nm Tech for 2022, Could Enable Epic 80 Billion Transistor GPUs”`{=html} {#_ODKksLpW-section} **[`TSMC confirms 3nm tech for 2022, could enable epic 80 billion transistor GPUs`{=html}](https://www.pcgamer.com/tsmc-confirms-3nm-tech-for-2022-could-enable-epic-80-billion-transistor-gpus/ "TSMC confirms 3nm tech for 2022, could enable epic 80 billion transistor GPUs"){.link-annotated-partial .id-not .include-annotation}** ## `“ORNL’s Frontier First to Break the Exaflop Ceiling”`{=html} {#_7h6GKpFP-section} **[`ORNL’s Frontier First to Break the Exaflop Ceiling`{=html}](https://www.top500.org/news/ornls-frontier-first-to-break-the-exaflop-ceiling/ "ORNL’s Frontier First to Break the Exaflop Ceiling"){.link-annotated-partial .id-not .include-annotation}** ## `“A Spymaster Sheikh Controls a $1.5 Trillion Fortune. Tahnoun Bin Zayed Al Nahyan Wants to Use It to Dominate AI”`{=html} {#_R4l-sdFN-section} **[`A Spymaster Sheikh Controls a $1.5 Trillion Fortune. Tahnoun bin Zayed al Nahyan Wants to Use It to Dominate AI`{=html}](https://www.wired.com/story/uae-intelligence-chief-ai-money/ "A Spymaster Sheikh Controls a $1.5 Trillion Fortune. Tahnoun bin Zayed al Nahyan Wants to Use It to Dominate AI"){.link-annotated-partial .id-not .include-annotation link-icon="wired" link-icon-type="svg"}** ## `“Returning to Google DeepMind”, Tay 2026`{=html} {#_ggyOqgNC-section} **[`Returning to Google DeepMind`{=html}](https://www.yitay.net/blog/returning-to-google-deepmind "'Returning to Google DeepMind', Tay 2026"){.link-annotated-partial .id-not .include-annotation link-icon="deepmind" link-icon-type="svg" link-icon-color="#4185f4"}** ## `“How to Accelerate Innovation With AI at Scale”`{=html} {#_QqufgWc5-section} **[`How to accelerate innovation with AI at Scale`{=html}](https://www.youtube.com/watch?v=DLw-wC4zntw?t=233 "How to accelerate innovation with AI at Scale"){.id-not .include-annotation link-icon="youtube" link-icon-type="svg" link-icon-color="#ff0033"}** ::: aux-links-transclude-file [`https://www.youtube.com/watch?v=DLw-wC4zntw?t=233`](https://www.youtube.com/watch?v=DLw-wC4zntw?t=233){.link-annotated-not .include-content .width-full} ::: ## `“48:44—Tesla Vision • 1:13:12—Planning and Control • 1:24:35—Manual Labeling • 1:28:11—Auto Labeling • 1:35:15—Simulation • 1:42:10—Hardware Integration • 1:45:40—Dojo”`{=html} {#_oIfs0L6H-section} **[`48:44—Tesla Vision • 1:13:12—Planning and Control • 1:24:35—Manual Labeling • 1:28:11—Auto Labeling • 1:35:15—Simulation • 1:42:10—Hardware Integration • 1:45:40—Dojo`{=html}](https://www.youtube.com/watch?v=j0z4FweCy4M&t=2928 "48:44—Tesla Vision • 1:13:12—Planning and Control • 1:24:35—Manual Labeling • 1:28:11—Auto Labeling • 1:35:15—Simulation • 1:42:10—Hardware Integration • 1:45:40—Dojo"){.id-not .include-annotation link-icon="youtube" link-icon-type="svg" link-icon-color="#ff0033"}** ::: aux-links-transclude-file [`https://www.youtube.com/watch?v=j0z4FweCy4M&t=2928`](https://www.youtube.com/watch?v=j0z4FweCy4M&t=2928){.link-annotated-not .include-content .width-full} ::: ## `ID_AA_Carmack`{=html} {#_yVtRmeiN-section} **[`Seymour Cray was famous for packing, powering, and cooling circuits incredibly densely. Classic Crays were made obsolete by microprocessors, but we may yet do similar things at a larger scale. Hyperscale data centers and even national supercomputers are loosely coupled things today, but if challenges demanded it, there is a world with a zetta scale, tightly integrated, low latency matrix dissipating a gigawatt in a swimming pool of circulating fluorinert.`{=html}](https://x.com/ID_AA_Carmack/status/1300280139071270914 "'Seymour Cray was famous for packing, powering, and cooling circuits incredibly densely. Classic Crays were made obsolete by microprocessors, but we may yet do similar things at a larger scale. Hyperscale data centers and even national supercomputers are loosely coupled things today, but if challenges demanded it, there is a world with a zetta scale, tightly integrated, low latency matrix dissipating a gigawatt in a swimming pool of circulating fluorinert.', Carmack 2026"){.link-annotated-partial .id-not .include-annotation link-icon="twitter" link-icon-type="svg" link-icon-color="#1da1f2"}** ## `lepikhin`{=html} {#_Kycu7V4Y-section} **[`We ran MoE (2048E,60L) with bfloat16 activations with total of 1 trillion model weights. Although trainable with manual diagnostics, with deep 1 trillion model we encountered several trainability issues with numerical stability. Will follow up.`{=html}](https://x.com/lepikhin/status/1278174444528132098 "'We ran MoE (2048E,60L) with bfloat16 activations with total of 1 trillion model weights. Although trainable with manual diagnostics, with deep 1 trillion model we encountered several trainability issues with numerical stability. Will follow up.', lepikhin 2026"){.link-annotated-partial .id-not .include-annotation link-icon="twitter" link-icon-type="svg" link-icon-color="#1da1f2"}** ## Sort By Magic ::: {demo-type="sort-by-magic-preface"} Annotations sorted by machine learning into [inferred \'tags\'](/design#future-tag-features){#gwern-design--future-tag-features .link-annotated-partial}. This provides an alternative way to browse: instead of by *date* order, one can browse in *topic* order. The \'sorted\' list has been automatically clustered into multiple sections & auto-labeled for easier browsing. Beginning with the newest annotation, it uses the embedding of each annotation to attempt to create a list of nearest-neighbor annotations, creating a progression of topics. For more details, see the link. ::: ### `retro-ai` {.collapse title="Machine-generated tag name for the following cluster of links."} [\[see previous entry\]](#gizmo64k-2026-section){.include .include-even-when-collapsed} ### `cache-security` {.collapse title="Machine-generated tag name for the following cluster of links."} [\[see previous entry\]](#yamamoto-matsuura-2026-section){.include .include-even-when-collapsed} [\[see previous entry\]](#read-et-al-2021-section){.include .include-even-when-collapsed} ### `ram-innovation ram-exit ai-ram future-memory ram-transition hardware-evolution` {.collapse title="Machine-generated tag name for the following cluster of links."} [\[see previous entry\]](#fxtentacles-2024-section){.include .include-even-when-collapsed} [\[see previous entry\]](#micron-2025-section){.include .include-even-when-collapsed} ### `sociology-machine` {.collapse title="Machine-generated tag name for the following cluster of links."} [\[see previous entry\]](#yamazaki-et-al-2022-section){.include .include-even-when-collapsed} [\[see previous entry\]](#roland-shiman-2002-section){.include .include-even-when-collapsed} [\[see previous entry\]](#olazaran-1993-section){.include .include-even-when-collapsed} ### `graph-partitioning` {.collapse title="Machine-generated tag name for the following cluster of links."} [\[see previous entry\]](#nazi-et-al-2019-section){.include .include-even-when-collapsed} [\[see previous entry\]](#tran-et-al-2018-section){.include .include-even-when-collapsed} ### `video-acceleration` {.collapse title="Machine-generated tag name for the following cluster of links."} [\[see previous entry\]](#ranganathan-et-al-2021-section){.include .include-even-when-collapsed} [\[see previous entry\]](#canakci-et-al-2025-section){.include .include-even-when-collapsed} ### `edge-ml` {.collapse title="Machine-generated tag name for the following cluster of links."} [\[see previous entry\]](#lloyd-1999-section){.include .include-even-when-collapsed} [\[see previous entry\]](#levy-calvert-2020-section){.include .include-even-when-collapsed} [\[see previous entry\]](#cannell-2015-section){.include .include-even-when-collapsed} [\[see previous entry\]](#dillavou-et-al-2021-section){.include .include-even-when-collapsed} [\[see previous entry\]](#launay-et-al-2020-section){.include .include-even-when-collapsed} [\[see previous entry\]](#launay-et-al-2020b-section){.include .include-even-when-collapsed} [\[see previous entry\]](#barham-et-al-2022-section){.include .include-even-when-collapsed} [\[see previous entry\]](#shukla-et-al-2022-2-section){.include .include-even-when-collapsed} [\[see previous entry\]](#ruan-et-al-2023-section){.include .include-even-when-collapsed} [\[see previous entry\]](#huang-et-al-2026-section){.include .include-even-when-collapsed} [\[see previous entry\]](#fang-et-al-2021-3-section){.include .include-even-when-collapsed} [\[see previous entry\]](#borzunov-et-al-2022-1-section){.include .include-even-when-collapsed} [\[see previous entry\]](#izsak-et-al-2021-section){.include .include-even-when-collapsed} [\[see previous entry\]](#erdil-2025-section){.include .include-even-when-collapsed} [\[see previous entry\]](#guo-et-al-2022-2-section){.include .include-even-when-collapsed} [\[see previous entry\]](#diamos-et-al-2016-section){.include .include-even-when-collapsed} [\[see previous entry\]](#raina-et-al-2009-section){.include .include-even-when-collapsed} [\[see previous entry\]](#le-et-al-2011-section){.include .include-even-when-collapsed} [\[see previous entry\]](#ciresan-et-al-2012-1-section){.include .include-even-when-collapsed} [\[see previous entry\]](#cireşan-et-al-2012b-section){.include .include-even-when-collapsed} [\[see previous entry\]](#ciresan-et-al-2011-section){.include .include-even-when-collapsed} [\[see previous entry\]](#ciresan-et-al-2010-section){.include .include-even-when-collapsed} [\[see previous entry\]](#ashtiani-et-al-2021-section){.include .include-even-when-collapsed} [\[see previous entry\]](#oh-jung-2004-section){.include .include-even-when-collapsed} [\[see previous entry\]](#domke-et-al-2020-section){.include .include-even-when-collapsed} [\[see previous entry\]](#vasilache-et-al-2018-section){.include .include-even-when-collapsed} [\[see previous entry\]](#hwang-et-al-2022-2-section){.include .include-even-when-collapsed} [\[see previous entry\]](#jain-et-al-2019-section){.include .include-even-when-collapsed} [\[see previous entry\]](#you-et-al-2022-2-section){.include .include-even-when-collapsed} [\[see previous entry\]](#dehghani-et-al-2021-section){.include .include-even-when-collapsed} [\[see previous entry\]](#sandberg-2018-2-section){.include .include-even-when-collapsed} [\[see previous entry\]](#legg-2010-section){.include .include-even-when-collapsed} [\[see previous entry\]](#bergal-2020-section){.include .include-even-when-collapsed} [\[see previous entry\]](#clark-2020-1-section){.include .include-even-when-collapsed} [\[see previous entry\]](#hemsoth-2021-section){.include .include-even-when-collapsed} ### `innovation-crisis` {.collapse title="Machine-generated tag name for the following cluster of links."} [\[see previous entry\]](#mickle-2025-section){.include .include-even-when-collapsed} [\[see previous entry\]](#moore-2020-section){.include .include-even-when-collapsed} ### `ai-chips` {.collapse title="Machine-generated tag name for the following cluster of links."} [\[see previous entry\]](#_dpuECoS8-section){.include .include-even-when-collapsed} [\[see previous entry\]](#jouppi-et-al-2020-section){.include .include-even-when-collapsed} [\[see previous entry\]](#norrie-et-al-2021-section){.include .include-even-when-collapsed} [\[see previous entry\]](#jouppi-et-al-2021-section){.include .include-even-when-collapsed} [\[see previous entry\]](#jouppi-et-al-2023-section){.include .include-even-when-collapsed} [\[see previous entry\]](#bailey-2022-section){.include .include-even-when-collapsed} [\[see previous entry\]](#khan-mann-2020-section){.include .include-even-when-collapsed} [\[see previous entry\]](#todd-2024-section){.include .include-even-when-collapsed} [\[see previous entry\]](#schuman-2024-section){.include .include-even-when-collapsed} [\[see previous entry\]](#wang-2023-1-section){.include .include-even-when-collapsed} [\[see previous entry\]](#fitch-et-al-2023-section){.include .include-even-when-collapsed} [\[see previous entry\]](#baptista-2024-section){.include .include-even-when-collapsed} [\[see previous entry\]](#mcmorrow-olcott-2024-section){.include .include-even-when-collapsed} [\[see previous entry\]](#kao-huang-2024-section){.include .include-even-when-collapsed} [\[see previous entry\]](#choi-2024-section){.include .include-even-when-collapsed} [\[see previous entry\]](#trendforce-2024-section){.include .include-even-when-collapsed} [\[see previous entry\]](#kerr-et-al-2023-section){.include .include-even-when-collapsed} [\[see previous entry\]](#dave-2023-2-section){.include .include-even-when-collapsed} [\[see previous entry\]](#ludlow-vance-2023-section){.include .include-even-when-collapsed} [\[see previous entry\]](#murgia-ruehl-2024-section){.include .include-even-when-collapsed} [\[see previous entry\]](#sellman-2023-section){.include .include-even-when-collapsed} [\[see previous entry\]](#heath-2025-section){.include .include-even-when-collapsed} [\[see previous entry\]](#stanley-2023-section){.include .include-even-when-collapsed} [\[see previous entry\]](#jones-2020-section){.include .include-even-when-collapsed} [\[see previous entry\]](#amodei-et-al-2018-section){.include .include-even-when-collapsed} [\[see previous entry\]](#hernandez-brown-2020-blog-section){.include .include-even-when-collapsed} [\[see previous entry\]](#team-et-al-2020-section){.include .include-even-when-collapsed} [\[see previous entry\]](#rajbhandari-et-al-2021-section){.include .include-even-when-collapsed} [\[see previous entry\]](#ren-et-al-2021-2-section){.include .include-even-when-collapsed} [\[see previous entry\]](#yuan-et-al-2026-section){.include .include-even-when-collapsed} [\[see previous entry\]](#narayanan-et-al-2021-section){.include .include-even-when-collapsed} [\[see previous entry\]](#bian-et-al-2021-section){.include .include-even-when-collapsed} [\[see previous entry\]](#wang-et-al-2021-07-section){.include .include-even-when-collapsed} [\[see previous entry\]](#xu-et-al-2021-8-section){.include .include-even-when-collapsed} [\[see previous entry\]](#shazeer-et-al-2018-section){.include .include-even-when-collapsed} [\[see previous entry\]](#kumar-et-al-2020-1-section){.include .include-even-when-collapsed} [\[see previous entry\]](#xu-et-al-2021-7-section){.include .include-even-when-collapsed} [\[see previous entry\]](#li-et-al-2021-8-section){.include .include-even-when-collapsed} [\[see previous entry\]](#he-et-al-2021-4-section){.include .include-even-when-collapsed} [\[see previous entry\]](#narayanan-et-al-2020-section){.include .include-even-when-collapsed} [\[see previous entry\]](#li-hoefler-2021-section){.include .include-even-when-collapsed} [\[see previous entry\]](#huang-et-al-2018-3-section){.include .include-even-when-collapsed} [\[see previous entry\]](#kosson-et-al-2020-section){.include .include-even-when-collapsed} [\[see previous entry\]](#shallue-et-al-2018-section){.include .include-even-when-collapsed} [\[see previous entry\]](#mccandlish-et-al-2018-largebatchtraining-section){.include .include-even-when-collapsed} [\[see previous entry\]](#keskar-et-al-2016-section){.include .include-even-when-collapsed} [\[see previous entry\]](#laskin-et-al-2020-1-section){.include .include-even-when-collapsed} [\[see previous entry\]](#gomez-et-al-2020-section){.include .include-even-when-collapsed} [\[see previous entry\]](#korthikanti-et-al-2022-section){.include .include-even-when-collapsed} [\[see previous entry\]](#pudipeddi-et-al-2020-section){.include .include-even-when-collapsed} [\[see previous entry\]](#alizadeh-et-al-2023-section){.include .include-even-when-collapsed} [\[see previous entry\]](#wang-et-al-2026-section){.include .include-even-when-collapsed} [\[see previous entry\]](#dao-et-al-2022-1-section){.include .include-even-when-collapsed} [\[see previous entry\]](#pope-et-al-2022-section){.include .include-even-when-collapsed} [\[see previous entry\]](#yao-et-al-2022-2-section){.include .include-even-when-collapsed} [\[see previous entry\]](#lu-et-al-2022-7-section){.include .include-even-when-collapsed} [\[see previous entry\]](#lin-et-al-2017-1-section){.include .include-even-when-collapsed} [\[see previous entry\]](#douillard-et-al-2023-section){.include .include-even-when-collapsed} [\[see previous entry\]](#jaghouar-et-al-2024-section){.include .include-even-when-collapsed} [\[see previous entry\]](#diskin-et-al-2021-section){.include .include-even-when-collapsed} [\[see previous entry\]](#borzunov-et-al-2022-2-section){.include .include-even-when-collapsed} [\[see previous entry\]](#ryabinin-et-al-2023-section){.include .include-even-when-collapsed} [\[see previous entry\]](#scao-et-al-2022-section){.include .include-even-when-collapsed} [\[see previous entry\]](#zhang-et-al-2024-section){.include .include-even-when-collapsed} [\[see previous entry\]](#gholami-et-al-2024-section){.include .include-even-when-collapsed} [\[see previous entry\]](#ivanov-et-al-2020-section){.include .include-even-when-collapsed} [\[see previous entry\]](#you-et-al-2019-section){.include .include-even-when-collapsed} [\[see previous entry\]](#wongpanich-et-al-2020-section){.include .include-even-when-collapsed} [\[see previous entry\]](#jia-et-al-2018-section){.include .include-even-when-collapsed} [\[see previous entry\]](#micikevicius-et-al-2017-section){.include .include-even-when-collapsed} [\[see previous entry\]](#noune-et-al-2022-section){.include .include-even-when-collapsed} [\[see previous entry\]](#ghaffari-et-al-2022-section){.include .include-even-when-collapsed} [\[see previous entry\]](#garland-gregg-2020-section){.include .include-even-when-collapsed} [\[see previous entry\]](#mallasén-et-al-2023-section){.include .include-even-when-collapsed} [\[see previous entry\]](#gallo-et-al-2022-section){.include .include-even-when-collapsed} [\[see previous entry\]](#esser-et-al-2016-section){.include .include-even-when-collapsed} [\[see previous entry\]](#roy-et-al-2019-section){.include .include-even-when-collapsed} [\[see previous entry\]](#kudithipudi-et-al-2025-section){.include .include-even-when-collapsed} [\[see previous entry\]](#davies-et-al-2018-2-section){.include .include-even-when-collapsed} [\[see previous entry\]](#shrestha-et-al-2023-section){.include .include-even-when-collapsed} [\[see previous entry\]](#zhao-et-al-2025-section){.include .include-even-when-collapsed} [\[see previous entry\]](#tang-et-al-2025-section){.include .include-even-when-collapsed} [\[see previous entry\]](#zeng-et-al-2021-3-section){.include .include-even-when-collapsed} [\[see previous entry\]](#ding-2021-1-section){.include .include-even-when-collapsed} [\[see previous entry\]](#langston-2020-section){.include .include-even-when-collapsed} [\[see previous entry\]](#microsoft-2023-section){.include .include-even-when-collapsed} [\[see previous entry\]](#reuters-2024-section){.include .include-even-when-collapsed} [\[see previous entry\]](#patel-nishball-2023-1-section){.include .include-even-when-collapsed} [\[see previous entry\]](#seetharaman-dotan-2023-section){.include .include-even-when-collapsed} [\[see previous entry\]](#witt-2023-section){.include .include-even-when-collapsed} [\[see previous entry\]](#kolodny-2024-section){.include .include-even-when-collapsed} [\[see previous entry\]](#abdulkadir-2024-section){.include .include-even-when-collapsed} [\[see previous entry\]](#lee-sengupta-2022-section){.include .include-even-when-collapsed} [\[see previous entry\]](#gupta-et-al-2022-2-section){.include .include-even-when-collapsed} [\[see previous entry\]](#mudigere-et-al-2021-section){.include .include-even-when-collapsed} [\[see previous entry\]](#li-et-al-2020-3-section){.include .include-even-when-collapsed} [\[see previous entry\]](#li-et-al-2014-2-section){.include .include-even-when-collapsed} [\[see previous entry\]](#cui-et-al-2016-section){.include .include-even-when-collapsed} [\[see previous entry\]](#jiang-et-al-2020-2-section){.include .include-even-when-collapsed} [\[see previous entry\]](#patarasuk-yuan-2009-section){.include .include-even-when-collapsed} [\[see previous entry\]](#mcmahan-et-al-2016-section){.include .include-even-when-collapsed} [\[see previous entry\]](#niu-et-al-2011-section){.include .include-even-when-collapsed} [\[see previous entry\]](#chen-et-al-2016-4-section){.include .include-even-when-collapsed} [\[see previous entry\]](#zhu-et-al-2024-2-section){.include .include-even-when-collapsed} [\[see previous entry\]](#dao-et-al-2022-2-section){.include .include-even-when-collapsed} [\[see previous entry\]](#javaheripi-et-al-2022-section){.include .include-even-when-collapsed} [\[see previous entry\]](#zhang-et-al-2021-nas-section){.include .include-even-when-collapsed} [\[see previous entry\]](#anderson-et-al-2021-1-section){.include .include-even-when-collapsed} [\[see previous entry\]](#wang-2025-r1-section){.include .include-even-when-collapsed} [\[see previous entry\]](#sophia-2025-section){.include .include-even-when-collapsed} [\[see previous entry\]](#ray-2020-section){.include .include-even-when-collapsed} [\[see previous entry\]](#liu-et-al-2023-04-section){.include .include-even-when-collapsed} [\[see previous entry\]](#golden-et-al-2023-section){.include .include-even-when-collapsed} [\[see previous entry\]](#wang-et-al-2023-08-section){.include .include-even-when-collapsed} [\[see previous entry\]](#lin-et-al-2019-3-section){.include .include-even-when-collapsed} [\[see previous entry\]](#rudin-et-al-2021-section){.include .include-even-when-collapsed} [\[see previous entry\]](#petrenko-et-al-2020-section){.include .include-even-when-collapsed} [\[see previous entry\]](#shacklett-et-al-2021-section){.include .include-even-when-collapsed} [\[see previous entry\]](#lan-et-al-2021-2-section){.include .include-even-when-collapsed} [\[see previous entry\]](#hessel-et-al-2021-2-section){.include .include-even-when-collapsed} [\[see previous entry\]](#makoviychuk-et-al-2021-section){.include .include-even-when-collapsed} [\[see previous entry\]](#du-2022-section){.include .include-even-when-collapsed} [\[see previous entry\]](#ananthanarayanan-et-al-2009-section){.include .include-even-when-collapsed} [\[see previous entry\]](#sejnowski-1987-section){.include .include-even-when-collapsed} [\[see previous entry\]](#moravec-1998-section){.include .include-even-when-collapsed} [\[see previous entry\]](#platt-1995-1-section){.include .include-even-when-collapsed} [\[see previous entry\]](#moravec-1976-section){.include .include-even-when-collapsed} [\[see previous entry\]](#leiserson-et-al-2020-section){.include .include-even-when-collapsed} [\[see previous entry\]](#hippke-2020-section){.include .include-even-when-collapsed} [\[see previous entry\]](#koomey-et-al-2011-section){.include .include-even-when-collapsed} [\[see previous entry\]](#thompson-et-al-2020-1-section){.include .include-even-when-collapsed} [\[see previous entry\]](#strubell-et-al-2019-section){.include .include-even-when-collapsed} [\[see previous entry\]](#wu-et-al-2021-05-section){.include .include-even-when-collapsed} [\[see previous entry\]](#ho-et-al-2025-section){.include .include-even-when-collapsed} [\[see previous entry\]](#pandey-2025-section){.include .include-even-when-collapsed} [\[see previous entry\]](#shkreli-2023-section){.include .include-even-when-collapsed} [\[see previous entry\]](#ai-2023-section){.include .include-even-when-collapsed} [\[see previous entry\]](#cai-hu-2025-section){.include .include-even-when-collapsed} ## Wikipedia (18) {#titled-links-wikipedia .collapse} 1. **[`ASML Holding`{=html}](https://en.wikipedia.org/wiki/ASML_Holding "ASML Holding"){.id-not .include-annotation link-icon="wikipedia" link-icon-type="svg"}** ::: aux-links-transclude-file [`https://en.wikipedia.org/wiki/ASML_Holding`](https://en.wikipedia.org/wiki/ASML_Holding "ASML Holding"){.id-not .include-content include-template="$annotationFileIncludeTemplate"} ::: 2. **[`Ampere (microarchitecture)`{=html}](https://en.wikipedia.org/wiki/Ampere_(microarchitecture) "Ampere (microarchitecture)"){.link-annotated-partial .id-not .include-annotation link-icon="wikipedia" link-icon-type="svg"}** 3. **[`Cerebras`{=html}](https://en.wikipedia.org/wiki/Cerebras "Cerebras"){.link-annotated-partial .id-not .include-annotation link-icon="wikipedia" link-icon-type="svg"}** 4. **[`Extreme ultraviolet lithography`{=html}](https://en.wikipedia.org/wiki/Extreme_ultraviolet_lithography "Extreme ultraviolet lithography"){.link-annotated-partial .id-not .include-annotation link-icon="wikipedia" link-icon-type="svg"}** 5. **[`GPGPU`{=html}](https://en.wikipedia.org/wiki/GPGPU "GPGPU"){.id-not .include-annotation link-icon="wikipedia" link-icon-type="svg"}** ::: aux-links-transclude-file [`https://en.wikipedia.org/wiki/GPGPU`](https://en.wikipedia.org/wiki/GPGPU "GPGPU"){.id-not .include-content include-template="$annotationFileIncludeTemplate"} ::: 6. **[`HBM2`{=html}](https://en.wikipedia.org/wiki/HBM2 "HBM2"){.id-not .include-annotation link-icon="wikipedia" link-icon-type="svg"}** ::: aux-links-transclude-file [`https://en.wikipedia.org/wiki/HBM2`](https://en.wikipedia.org/wiki/HBM2 "HBM2"){.id-not .include-content include-template="$annotationFileIncludeTemplate"} ::: 7. **[`HBM3`{=html}](https://en.wikipedia.org/wiki/HBM3 "HBM3"){.id-not .include-annotation link-icon="wikipedia" link-icon-type="svg"}** ::: aux-links-transclude-file [`https://en.wikipedia.org/wiki/HBM3`](https://en.wikipedia.org/wiki/HBM3 "HBM3"){.id-not .include-content include-template="$annotationFileIncludeTemplate"} ::: 8. **[`Hopper (microarchitecture)`{=html}](https://en.wikipedia.org/wiki/Hopper_(microarchitecture) "Hopper (microarchitecture)"){.link-annotated-partial .id-not .include-annotation link-icon="wikipedia" link-icon-type="svg"}** 9. **[`Minifloat`{=html}](https://en.wikipedia.org/wiki/Minifloat "Minifloat"){.id-not .include-annotation link-icon="wikipedia" link-icon-type="svg"}** ::: aux-links-transclude-file [`https://en.wikipedia.org/wiki/Minifloat`](https://en.wikipedia.org/wiki/Minifloat "Minifloat"){.id-not .include-content include-template="$annotationFileIncludeTemplate"} ::: 10. **[`Nvidia DGX`{=html}](https://en.wikipedia.org/wiki/Nvidia_DGX "Nvidia DGX"){.link-annotated-partial .id-not .include-annotation link-icon="wikipedia" link-icon-type="svg"}** 11. **[`Nvidia Tesla`{=html}](https://en.wikipedia.org/wiki/Nvidia_Tesla "Nvidia Tesla"){.link-annotated-partial .id-not .include-annotation link-icon="wikipedia" link-icon-type="svg"}** 12. **[`Roofline model`{=html}](https://en.wikipedia.org/wiki/Roofline_model "Roofline model"){.id-not .include-annotation link-icon="wikipedia" link-icon-type="svg"}** ::: aux-links-transclude-file [`https://en.wikipedia.org/wiki/Roofline_model`](https://en.wikipedia.org/wiki/Roofline_model "Roofline model"){.id-not .include-content include-template="$annotationFileIncludeTemplate"} ::: 13. **[`Spiking neural network`{=html}](https://en.wikipedia.org/wiki/Spiking_neural_network "Spiking neural network"){.link-annotated-partial .id-not .include-annotation link-icon="wikipedia" link-icon-type="svg"}** 14. **[`Stargate LLC`{=html}](https://en.wikipedia.org/wiki/Stargate_LLC "Stargate LLC"){.id-not .include-annotation link-icon="wikipedia" link-icon-type="svg"}** ::: aux-links-transclude-file [`https://en.wikipedia.org/wiki/Stargate_LLC`](https://en.wikipedia.org/wiki/Stargate_LLC "Stargate LLC"){.id-not .include-content include-template="$annotationFileIncludeTemplate"} ::: 15. **[`Summit (supercomputer)`{=html}](https://en.wikipedia.org/wiki/Summit_(supercomputer) "Summit (supercomputer)"){.link-annotated-partial .id-not .include-annotation link-icon="wikipedia" link-icon-type="svg"}** 16. **[`TSMC`{=html}](https://en.wikipedia.org/wiki/TSMC "TSMC"){.link-annotated-partial .id-not .include-annotation link-icon="wikipedia" link-icon-type="svg"}** 17. **[`Tensor Processing Unit § Third generation TPU`{=html}](https://en.wikipedia.org/wiki/Tensor_Processing_Unit#Third_generation_TPU "Tensor Processing Unit § Third generation TPU"){.id-not .include-annotation link-icon="wikipedia" link-icon-type="svg"}** ::: aux-links-transclude-file [`https://en.wikipedia.org/wiki/Tensor_Processing_Unit#Third_generation_TPU`](https://en.wikipedia.org/wiki/Tensor_Processing_Unit#Third_generation_TPU "Tensor Processing Unit § Third generation TPU"){.id-not .include-content include-template="$annotationFileIncludeTemplate"} ::: 18. **[`Tianhe-1A`{=html}](https://en.wikipedia.org/wiki/Tianhe-1A "Tianhe-1A"){.id-not .include-annotation link-icon="wikipedia" link-icon-type="svg"}** ::: aux-links-transclude-file [`https://en.wikipedia.org/wiki/Tianhe-1A`](https://en.wikipedia.org/wiki/Tianhe-1A "Tianhe-1A"){.id-not .include-content include-template="$annotationFileIncludeTemplate"} ::: # Miscellaneous - **[`https://www.nytimes.com/2026/04/17/technology/cerebras-public-offering-ai.html`{=html}](https://www.nytimes.com/2026/04/17/technology/cerebras-public-offering-ai.html){.id-not .include-annotation link-icon="C" link-icon-type="text,sans" link-icon-color="#f05a29"}** - **[`https://x.com/theAnujSaharan/status/1953513241885180162`{=html}](https://x.com/theAnujSaharan/status/1953513241885180162 "theAnujSaharan 2025"){.id-not .include-annotation link-icon="twitter" link-icon-type="svg" link-icon-color="#1da1f2"}** ::: aux-links-transclude-file [`https://x.com/theAnujSaharan/status/1953513241885180162`](https://x.com/theAnujSaharan/status/1953513241885180162 "theAnujSaharan 2025"){.id-not .include-content} ::: - **[`https://www.reuters.com/technology/artificial-intelligence/openai-builds-first-chip-with-broadcom-tsmc-scales-back-foundry-ambition-2024-10-29/`{=html}](https://www.reuters.com/technology/artificial-intelligence/openai-builds-first-chip-with-broadcom-tsmc-scales-back-foundry-ambition-2024-10-29/){.id-not .include-annotation link-icon="R" link-icon-type="text,sans" link-icon-color="#e56218"}** - **[`/doc/ai/scaling/mixture-of-experts/2021-04-12-jensenhuang-gtc2021keynote-ean_oizwuxa.en.vtt.txt`{=html}](/doc/ai/scaling/mixture-of-experts/2021-04-12-jensenhuang-gtc2021keynote-ean_oizwuxa.en.vtt.txt){.link-annotated-partial .id-not .include-annotation link-icon="txt" link-icon-type="svg"}** - **[`/doc/ai/scaling/hardware/2021-jouppi-table1-keycharacteristicsoftpus.png`{=html}](/doc/ai/scaling/hardware/2021-jouppi-table1-keycharacteristicsoftpus.png){.link-annotated-partial .id-not .include-annotation link-icon="image" link-icon-type="svg"}** - **[`/doc/ai/scaling/hardware/2021-ren-zerooffload-cpugpudataflow.png`{=html}](/doc/ai/scaling/hardware/2021-ren-zerooffload-cpugpudataflow.png){.link-annotated-partial .id-not .include-annotation link-icon="image" link-icon-type="svg"}** - **[`/doc/ai/scaling/hardware/2020-08-05-hippke-measuringhardwareoverhang-chessscaling19902020.png`{=html}](/doc/ai/scaling/hardware/2020-08-05-hippke-measuringhardwareoverhang-chessscaling19902020.png){.link-annotated-partial .id-not .include-annotation link-icon="image" link-icon-type="svg"}** - **[`/doc/ai/scaling/hardware/2020-kumar-figure11-tpumultipodspeedups.png`{=html}](/doc/ai/scaling/hardware/2020-kumar-figure11-tpumultipodspeedups.png){.link-annotated-partial .id-not .include-annotation link-icon="image" link-icon-type="svg"}** - **[`/doc/ai/scaling/hardware/2017-jouppi.pdf`{=html}](/doc/ai/scaling/hardware/2017-jouppi.pdf){.link-annotated-partial .id-not .include-annotation link-icon="pdf" link-icon-type="svg" link-icon-color="#f40f02"}** - **[`/doc/ai/scaling/hardware/1998-moravec-figure2-evolutionofcomputerpowercost19001998.csv`{=html}](/doc/ai/scaling/hardware/1998-moravec-figure2-evolutionofcomputerpowercost19001998.csv){.link-annotated-partial .id-not .include-annotation link-icon="csv" link-icon-type="svg"}** - **[`/doc/ai/scaling/hardware/1998-moravec-figure2-evolutionofcomputerpowercost19001998.jpg`{=html}](/doc/ai/scaling/hardware/1998-moravec-figure2-evolutionofcomputerpowercost19001998.jpg){.link-annotated-partial .id-not .include-annotation link-icon="image" link-icon-type="svg"}** - **[`/doc/ai/scaling/hardware/1998-moravec-figure3-peakcomputeuseinai19501998.jpg`{=html}](/doc/ai/scaling/hardware/1998-moravec-figure3-peakcomputeuseinai19501998.jpg){.link-annotated-partial .id-not .include-annotation link-icon="image" link-icon-type="svg"}** - **[`/doc/ai/scaling/hardware/1987-sejnowski-figure1-historyofsupercomputersextrapolationvshumanbraincomputepower.jpg`{=html}](/doc/ai/scaling/hardware/1987-sejnowski-figure1-historyofsupercomputersextrapolationvshumanbraincomputepower.jpg){.link-annotated-partial .id-not .include-annotation link-icon="image" link-icon-type="svg"}** - **[`https://ai.meta.com/blog/meta-training-inference-accelerator-AI-MTIA/`{=html}](https://ai.meta.com/blog/meta-training-inference-accelerator-AI-MTIA/){.id-not .include-annotation link-icon="facebook" link-icon-type="svg" link-icon-color="#1877f2"}** - **[`https://ai.meta.com/blog/next-generation-meta-training-inference-accelerator-AI-MTIA/`{=html}](https://ai.meta.com/blog/next-generation-meta-training-inference-accelerator-AI-MTIA/){.link-annotated-partial .id-not .include-annotation link-icon="facebook" link-icon-type="svg" link-icon-color="#1877f2"}** - **[`https://andromeda.ai/`{=html}](https://andromeda.ai/){.id-not .include-annotation}** - **[`https://apps.fz-juelich.de/jsc/hps/juwels/configuration.html#hardware-configuration-of-the-system-name-booster-module`{=html}](https://apps.fz-juelich.de/jsc/hps/juwels/configuration.html#hardware-configuration-of-the-system-name-booster-module){.id-not .include-annotation}** ::: aux-links-transclude-file ::: collapse **View HTML**: [`/doc/www/apps.fz-juelich.de/4234343ce438622214a38f09f3c93ebc81081679.html#hardware-configuration-of-the-system-name-booster-module`](/doc/www/apps.fz-juelich.de/4234343ce438622214a38f09f3c93ebc81081679.html#hardware-configuration-of-the-system-name-booster-module){.id-not .link-annotated-not .include-content .include-lazy link-icon="internet-archive" link-icon-type="svg"} ::: ::: - **[`https://austinvernon.site/blog/datacenterpv.html`{=html}](https://austinvernon.site/blog/datacenterpv.html){.link-annotated-partial .id-not .include-annotation}** - **[`https://blogs.nvidia.com/blog/2021/04/12/cpu-grace-cscs-alps/`{=html}](https://blogs.nvidia.com/blog/2021/04/12/cpu-grace-cscs-alps/){.id-not .include-annotation link-icon="n" link-icon-type="text,sans,italic" link-icon-color="#77ba00"}** - **[`https://blogs.nvidia.com/blog/2022/09/08/hopper-mlperf-inference/`{=html}](https://blogs.nvidia.com/blog/2022/09/08/hopper-mlperf-inference/){.id-not .include-annotation link-icon="n" link-icon-type="text,sans,italic" link-icon-color="#77ba00"}** - **[`https://carnegieendowment.org/2022/11/22/after-chips-act-limits-of-reshoring-and-next-steps-for-u.s.-semiconductor-policy-pub-88439`{=html}](https://carnegieendowment.org/2022/11/22/after-chips-act-limits-of-reshoring-and-next-steps-for-u.s.-semiconductor-policy-pub-88439){.link-annotated-partial .id-not .include-annotation}** - **[`https://caseyhandmer.wordpress.com/2024/03/12/how-to-feed-the-ais/`{=html}](https://caseyhandmer.wordpress.com/2024/03/12/how-to-feed-the-ais/){.link-annotated-partial .id-not .include-annotation}** - **[`https://chipsandcheese.com/p/nvidias-h100-funny-l2-and-tons-of-bandwidth`{=html}](https://chipsandcheese.com/p/nvidias-h100-funny-l2-and-tons-of-bandwidth){.id-not .include-annotation link-icon="n" link-icon-type="text,sans,italic" link-icon-color="#77ba00"}** - **[`https://cloud.google.com/blog/products/compute/the-worlds-largest-distributed-llm-training-job-on-tpu-v5e`{=html}](https://cloud.google.com/blog/products/compute/the-worlds-largest-distributed-llm-training-job-on-tpu-v5e){.id-not .include-annotation link-icon="alphabet" link-icon-type="svg" link-icon-color="#4285f4"}** - **[`https://cloud.google.com/blog/topics/systems/the-evolution-of-googles-jupiter-data-center-network`{=html}](https://cloud.google.com/blog/topics/systems/the-evolution-of-googles-jupiter-data-center-network){.link-annotated-partial .id-not .include-annotation link-icon="alphabet" link-icon-type="svg" link-icon-color="#4285f4"}** - **[`https://cset.georgetown.edu/wp-content/uploads/AI-and-Compute-How-Much-Longer-Can-Computing-Power-Drive-Artificial-Intelligence-Progress.pdf`{=html}](https://cset.georgetown.edu/wp-content/uploads/AI-and-Compute-How-Much-Longer-Can-Computing-Power-Drive-Artificial-Intelligence-Progress.pdf){.link-annotated-partial .id-not .include-annotation link-icon="pdf" link-icon-type="svg" link-icon-color="#f40f02"}** - **[`https://edoras.sdsu.edu/~vinge/misc/singularity.html`{=html}](https://edoras.sdsu.edu/~vinge/misc/singularity.html){.link-annotated-partial .id-not .include-annotation}** - **[`https://evabehrens.substack.com/p/the-agi-race-between-the-us-and-china`{=html}](https://evabehrens.substack.com/p/the-agi-race-between-the-us-and-china){.link-annotated-partial .id-not .include-annotation link-icon="substack" link-icon-type="svg" link-icon-color="#ff6719"}** - **[`https://github.com/deepseek-ai/open-infra-index/blob/main/202502OpenSourceWeek/day_6_one_more_thing_deepseekV3R1_inference_system_overview`{=html}](https://github.com/deepseek-ai/open-infra-index/blob/main/202502OpenSourceWeek/day_6_one_more_thing_deepseekV3R1_inference_system_overview){.link-annotated-partial .id-not .include-annotation link-icon="deepseek" link-icon-type="svg" link-icon-color="#4d6bfe"}** - **[`https://gpus.llm-utils.org/nvidia-h100-gpus-supply-and-demand/`{=html}](https://gpus.llm-utils.org/nvidia-h100-gpus-supply-and-demand/){.link-annotated-partial .id-not .include-annotation link-icon="n" link-icon-type="text,sans,italic" link-icon-color="#77ba00"}** - **[`https://gpus.llm-utils.org/nvidia-h100-gpus-supply-and-demand/#how-do-the-big-clouds-compare`{=html}](https://gpus.llm-utils.org/nvidia-h100-gpus-supply-and-demand/#how-do-the-big-clouds-compare){.id-not .include-annotation link-icon="n" link-icon-type="text,sans,italic" link-icon-color="#77ba00"}** ::: aux-links-transclude-file ::: collapse **View HTML**: [`/doc/www/gpus.llm-utils.org/d89d4f82112bca85cbf3807ffd2d98f634332f73.html#how-do-the-big-clouds-compare`](/doc/www/gpus.llm-utils.org/d89d4f82112bca85cbf3807ffd2d98f634332f73.html#how-do-the-big-clouds-compare){.id-not .link-annotated-not .include-content .include-lazy link-icon="internet-archive" link-icon-type="svg"} ::: ::: - **[`https://groq.com/wp-content/uploads/2020/06/ISCA-TSP.pdf`{=html}](https://groq.com/wp-content/uploads/2020/06/ISCA-TSP.pdf){.link-annotated-partial .id-not .include-annotation link-icon="pdf" link-icon-type="svg" link-icon-color="#f40f02"}** - **[`https://hpc.stability.ai/`{=html}](https://hpc.stability.ai/){.id-not .include-annotation}** - **[`https://newsletter.pragmaticengineer.com/p/scaling-chatgpt#%C2%A7five-scaling-challenges`{=html}](https://newsletter.pragmaticengineer.com/p/scaling-chatgpt#%C2%A7five-scaling-challenges){.id-not .include-annotation}** ::: aux-links-transclude-file ::: collapse **View HTML**: [`/doc/www/newsletter.pragmaticengineer.com/de8235663546200800f6174df20b20cb9e48f951.html#%C2%A7five-scaling-challenges`](/doc/www/newsletter.pragmaticengineer.com/de8235663546200800f6174df20b20cb9e48f951.html#%C2%A7five-scaling-challenges){.id-not .link-annotated-not .include-content .include-lazy link-icon="internet-archive" link-icon-type="svg"} ::: ::: - **[`https://openai.com/blog/techniques-for-training-large-neural-networks/`{=html}](https://openai.com/blog/techniques-for-training-large-neural-networks/){.id-not .include-annotation link-icon="openai" link-icon-type="svg"}** - **[`https://openai.com/index/a-business-that-scales-with-the-value-of-intelligence/`{=html}](https://openai.com/index/a-business-that-scales-with-the-value-of-intelligence/){.link-annotated-partial .id-not .include-annotation link-icon="openai" link-icon-type="svg"}** - **[`https://openai.com/research/scaling-kubernetes-to-7500-nodes`{=html}](https://openai.com/research/scaling-kubernetes-to-7500-nodes){.id-not .include-annotation link-icon="openai" link-icon-type="svg"}** - **[`https://openreview.net/forum?id=e40dYCosQd`{=html}](https://openreview.net/forum?id=e40dYCosQd){.link-annotated-partial .id-not .include-annotation link-icon="OR" link-icon-type="text,sans" link-icon-color="#8c1b13"}** - **[`https://research.google/blog/tensorstore-for-high-performance-scalable-array-storage/`{=html}](https://research.google/blog/tensorstore-for-high-performance-scalable-array-storage/){.id-not .include-annotation link-icon="alphabet" link-icon-type="svg" link-icon-color="#4285f4"}** - **[`https://restofworld.org/2024/tsmc-arizona-expansion/`{=html}](https://restofworld.org/2024/tsmc-arizona-expansion/){.link-annotated-partial .id-not .include-annotation}** - **[`https://rhg.com/research/running-on-ice/`{=html}](https://rhg.com/research/running-on-ice/){.id-not .include-annotation}** - **[`https://spectrum.ieee.org/computing/hardware/the-future-of-deep-learning-is-photonic`{=html}](https://spectrum.ieee.org/computing/hardware/the-future-of-deep-learning-is-photonic){.link-annotated-partial .id-not .include-annotation link-icon="IEEE" link-icon-type="text,mono,quad" link-icon-color="#006699"}** - **[`https://spectrum.ieee.org/generative-ai-training`{=html}](https://spectrum.ieee.org/generative-ai-training){.id-not .include-annotation link-icon="IEEE" link-icon-type="text,mono,quad" link-icon-color="#006699"}** - **[`https://thechipletter.substack.com/p/googles-first-tpu-architecture`{=html}](https://thechipletter.substack.com/p/googles-first-tpu-architecture){.id-not .include-annotation link-icon="alphabet" link-icon-type="svg" link-icon-color="#4285f4"}** - **[`https://thezvi.substack.com/p/on-the-executive-order`{=html}](https://thezvi.substack.com/p/on-the-executive-order){.id-not .include-annotation link-icon="substack" link-icon-type="svg" link-icon-color="#ff6719"}** - **[`https://venturebeat.com/2020/11/17/cerebras-wafer-size-chip-is-10000-times-faster-than-a-gpu/`{=html}](https://venturebeat.com/2020/11/17/cerebras-wafer-size-chip-is-10000-times-faster-than-a-gpu/){.link-annotated-partial .id-not .include-annotation link-icon="C" link-icon-type="text,sans" link-icon-color="#f05a29"}** - **[`https://warontherocks.com/2024/04/how-washington-can-save-its-semiconductor-controls-on-china/`{=html}](https://warontherocks.com/2024/04/how-washington-can-save-its-semiconductor-controls-on-china/){.link-annotated-partial .id-not .include-annotation link-icon="WOTR" link-icon-type="text,quad,sans"}** - **[`https://www.abortretry.fail/p/the-rise-and-fall-of-silicon-graphics`{=html}](https://www.abortretry.fail/p/the-rise-and-fall-of-silicon-graphics){.link-annotated-partial .id-not .include-annotation}** - **[`https://www.akamai.com/newsroom/press-release/akamai-announces-11-6-billion-multi-year-agreement-with-anthropic-to-support-growing-demand`{=html}](https://www.akamai.com/newsroom/press-release/akamai-announces-11-6-billion-multi-year-agreement-with-anthropic-to-support-growing-demand){.link-annotated-partial .id-not .include-annotation}** - **[`https://www.bloomberg.com/news/articles/2022-10-10/china-chip-stocks-drop-as-biden-tightens-rules-on-us-tech-access`{=html}](https://www.bloomberg.com/news/articles/2022-10-10/china-chip-stocks-drop-as-biden-tightens-rules-on-us-tech-access){.id-not .include-annotation link-icon="𝐁" link-icon-type="text"}** - **[`https://www.businesswire.com/news/home/20241015910376/en/Crusoe-Blue-Owl-Capital-and-Primary-Digital-Infrastructure-Enter-3.4-billion-Joint-Venture-for-AI-Data-Center-Development`{=html}](https://www.businesswire.com/news/home/20241015910376/en/Crusoe-Blue-Owl-Capital-and-Primary-Digital-Infrastructure-Enter-3.4-billion-Joint-Venture-for-AI-Data-Center-Development){.id-not .include-annotation}** - **[`https://www.cerebras.net/blog/introducing-gigagpt-gpt-3-sized-models-in-565-lines-of-code`{=html}](https://www.cerebras.net/blog/introducing-gigagpt-gpt-3-sized-models-in-565-lines-of-code){.link-annotated-partial .id-not .include-annotation link-icon="C" link-icon-type="text,sans" link-icon-color="#f05a29"}** - **[`https://www.cerebras.net/press-release/cerebras-announces-third-generation-wafer-scale-engine`{=html}](https://www.cerebras.net/press-release/cerebras-announces-third-generation-wafer-scale-engine){.link-annotated-partial .id-not .include-annotation link-icon="C" link-icon-type="text,sans" link-icon-color="#f05a29"}** - **[`https://www.chinatalk.media/p/new-chip-export-controls-explained`{=html}](https://www.chinatalk.media/p/new-chip-export-controls-explained){.link-annotated-partial .id-not .include-annotation}** - **[`https://www.chinatalk.media/p/new-sexport-controls-semianalysis`{=html}](https://www.chinatalk.media/p/new-sexport-controls-semianalysis){.id-not .include-annotation}** - **[`https://www.ft.com/content/25337df3-5b98-4dd1-b7a9-035dcc130d6a`{=html}](https://www.ft.com/content/25337df3-5b98-4dd1-b7a9-035dcc130d6a){.link-annotated-partial .id-not .include-annotation link-icon="FT" link-icon-type="text" link-icon-color="#e3b68e"}** - **[`https://www.ft.com/content/8799b33d-f07c-4a03-82f0-bf5d3d1d29e9`{=html}](https://www.ft.com/content/8799b33d-f07c-4a03-82f0-bf5d3d1d29e9){.link-modified-recently .id-not .include-annotation link-icon="FT" link-icon-type="text" link-icon-color="#e3b68e"}** ::: aux-links-transclude-file ::: collapse **View External Link**: [`https://www.ft.com/content/8799b33d-f07c-4a03-82f0-bf5d3d1d29e9`](https://www.ft.com/content/8799b33d-f07c-4a03-82f0-bf5d3d1d29e9){.id-not .link-annotated-not .include-content .include-lazy link-icon="FT" link-icon-type="text" link-icon-color="#e3b68e"} ::: ::: - **[`https://www.fujitsu.com/global/about/resources/news/press-releases/2024/0510-01.html`{=html}](https://www.fujitsu.com/global/about/resources/news/press-releases/2024/0510-01.html){.link-annotated-partial .id-not .include-annotation}** - **[`https://www.ibm.com/blogs/research/2020/12/ibm-ai-edge/`{=html}](https://www.ibm.com/blogs/research/2020/12/ibm-ai-edge/){.link-annotated-partial .id-not .include-annotation}** - **[`https://www.lesswrong.com/posts/KsKfvLx7nFBZnWtEu/no-human-brains-are-not-much-more-efficient-than-computers`{=html}](https://www.lesswrong.com/posts/KsKfvLx7nFBZnWtEu/no-human-brains-are-not-much-more-efficient-than-computers){.id-not .include-annotation link-icon="LW" link-icon-type="text" link-icon-color="#7faf83"}** ::: aux-links-transclude-file ::: collapse **View External Link**: [`https://www.lesswrong.com/posts/KsKfvLx7nFBZnWtEu/no-human-brains-are-not-much-more-efficient-than-computers`](https://www.lesswrong.com/posts/KsKfvLx7nFBZnWtEu/no-human-brains-are-not-much-more-efficient-than-computers){.id-not .link-annotated-not .include-content .include-lazy link-icon="LW" link-icon-type="text" link-icon-color="#7faf83"} ::: ::: - **[`https://www.lesswrong.com/posts/YKfNZAmiLdepDngwi/gpt-175bee`{=html}](https://www.lesswrong.com/posts/YKfNZAmiLdepDngwi/gpt-175bee){.link-annotated-partial .id-not .include-annotation link-icon="LW" link-icon-type="text" link-icon-color="#7faf83"}** - **[`https://www.lesswrong.com/posts/cB2Rtnp7DBTpDy3ii/memory-bandwidth-constraints-imply-economies-of-scale-in-ai`{=html}](https://www.lesswrong.com/posts/cB2Rtnp7DBTpDy3ii/memory-bandwidth-constraints-imply-economies-of-scale-in-ai){.id-not .include-annotation link-icon="LW" link-icon-type="text" link-icon-color="#7faf83"}** ::: aux-links-transclude-file ::: collapse **View External Link**: [`https://www.lesswrong.com/posts/cB2Rtnp7DBTpDy3ii/memory-bandwidth-constraints-imply-economies-of-scale-in-ai`](https://www.lesswrong.com/posts/cB2Rtnp7DBTpDy3ii/memory-bandwidth-constraints-imply-economies-of-scale-in-ai){.id-not .link-annotated-not .include-content .include-lazy link-icon="LW" link-icon-type="text" link-icon-color="#7faf83"} ::: ::: - **[`https://www.nytimes.com/2022/10/13/us/politics/biden-china-technology-semiconductors.html`{=html}](https://www.nytimes.com/2022/10/13/us/politics/biden-china-technology-semiconductors.html){.id-not .include-annotation link-icon="new-york-times" link-icon-type="svg"}** - **[`https://www.nytimes.com/2023/07/12/magazine/semiconductor-chips-us-china.html`{=html}](https://www.nytimes.com/2023/07/12/magazine/semiconductor-chips-us-china.html){.id-not .include-annotation link-icon="new-york-times" link-icon-type="svg"}** - **[`https://www.reddit.com/r/MachineLearning/comments/1dlsogx/d_academic_ml_labs_how_many_gpus/`{=html}](https://www.reddit.com/r/MachineLearning/comments/1dlsogx/d_academic_ml_labs_how_many_gpus/){.link-annotated-partial .id-not .include-annotation link-icon="reddit" link-icon-type="svg" link-icon-color="#ff4500"}** - **[`https://www.reddit.com/r/MachineLearning/comments/1s4hxgu/d_1m_tokenssecond_serving_qwen_35_27b_on_b200/`{=html}](https://www.reddit.com/r/MachineLearning/comments/1s4hxgu/d_1m_tokenssecond_serving_qwen_35_27b_on_b200/){.link-annotated-partial .id-not .include-annotation link-icon="reddit" link-icon-type="svg" link-icon-color="#ff4500"}** - **[`https://www.reuters.com/technology/coreweave-raises-23-billion-debt-collateralized-by-nvidia-chips-2023-08-03/`{=html}](https://www.reuters.com/technology/coreweave-raises-23-billion-debt-collateralized-by-nvidia-chips-2023-08-03/){.link-annotated-partial .id-not .include-annotation link-icon="n" link-icon-type="text,sans,italic" link-icon-color="#77ba00"}** - **[`https://www.reuters.com/technology/inside-metas-scramble-catch-up-ai-2023-04-25/`{=html}](https://www.reuters.com/technology/inside-metas-scramble-catch-up-ai-2023-04-25/){.id-not .include-annotation link-icon="R" link-icon-type="text,sans" link-icon-color="#e56218"}** - **[`https://www.theinformation.com/articles/bytedance-planned-to-spend-7-billion-on-nvidia-chips-next-year`{=html}](https://www.theinformation.com/articles/bytedance-planned-to-spend-7-billion-on-nvidia-chips-next-year){.id-not .include-annotation link-icon="the-information" link-icon-type="svg" link-icon-color="#f32a52"}** - **[`https://www.theinformation.com/articles/microsoft-and-openai-plot-100-billion-stargate-ai-supercomputer`{=html}](https://www.theinformation.com/articles/microsoft-and-openai-plot-100-billion-stargate-ai-supercomputer){.link-annotated-partial .id-not .include-annotation link-icon="the-information" link-icon-type="svg" link-icon-color="#f32a52"}** - **[`https://www.theregister.com/2023/11/07/bing_gpu_oracle/`{=html}](https://www.theregister.com/2023/11/07/bing_gpu_oracle/){.id-not .include-annotation link-icon="𓅐" link-icon-type="text" link-icon-color="#ff0000"}** ::: aux-links-transclude-file ::: collapse **View External Link**: [`https://www.theregister.com/2023/11/07/bing_gpu_oracle/`](https://www.theregister.com/2023/11/07/bing_gpu_oracle/){.id-not .link-annotated-not .include-content .include-lazy link-icon="𓅐" link-icon-type="text" link-icon-color="#ff0000"} ::: ::: - **[`https://www.yitay.net/blog/training-great-llms-entirely-from-ground-zero-in-the-wilderness`{=html}](https://www.yitay.net/blog/training-great-llms-entirely-from-ground-zero-in-the-wilderness){.id-not .include-annotation}** - **[`https://x.com/Altimor/status/1668902393386237953`{=html}](https://x.com/Altimor/status/1668902393386237953 "Crivello 2026"){.link-annotated-partial .id-not .include-annotation link-icon="twitter" link-icon-type="svg" link-icon-color="#1da1f2"}** - **[`https://x.com/BrettHarrison/status/2086824054535655660`{=html}](https://x.com/BrettHarrison/status/2086824054535655660 "BrettHarrison 2026"){.link-modified-recently .id-not .include-annotation link-icon="twitter" link-icon-type="svg" link-icon-color="#1da1f2"}** ::: aux-links-transclude-file [`https://x.com/BrettHarrison/status/2086824054535655660`](https://x.com/BrettHarrison/status/2086824054535655660 "BrettHarrison 2026"){.link-modified-recently .id-not .include-content} ::: - **[`https://x.com/DanHendrycks/status/1825926885370728881`{=html}](https://x.com/DanHendrycks/status/1825926885370728881 "Hendrycks 2026"){.id-not .include-annotation link-icon="twitter" link-icon-type="svg" link-icon-color="#1da1f2"}** ::: aux-links-transclude-file [`https://x.com/DanHendrycks/status/1825926885370728881`](https://x.com/DanHendrycks/status/1825926885370728881 "Hendrycks 2026"){.id-not .include-content} ::: - **[`https://x.com/EMostaque/status/1674479761429504017`{=html}](https://x.com/EMostaque/status/1674479761429504017 "Mostaque 2026"){.link-annotated-partial .id-not .include-annotation link-icon="twitter" link-icon-type="svg" link-icon-color="#1da1f2"}** - **[`https://x.com/EmilWallner/status/1591007449691336704`{=html}](https://x.com/EmilWallner/status/1591007449691336704 "EmilWallner 2026"){.link-annotated-partial .id-not .include-annotation link-icon="twitter" link-icon-type="svg" link-icon-color="#1da1f2"}** - **[`https://x.com/JensenHuang/status/2096700264569090384`{=html}](https://x.com/JensenHuang/status/2096700264569090384 "Huang 2026"){.link-modified-recently .id-not .include-annotation link-icon="twitter" link-icon-type="svg" link-icon-color="#1da1f2"}** ::: aux-links-transclude-file [`https://x.com/JensenHuang/status/2096700264569090384`](https://x.com/JensenHuang/status/2096700264569090384 "Huang 2026"){.link-modified-recently .id-not .include-content} ::: - **[`https://x.com/_JohnnyJones/status/2097357027202777199`{=html}](https://x.com/_JohnnyJones/status/2097357027202777199 "_JohnnyJones 2026"){.link-modified-recently .id-not .include-annotation link-icon="twitter" link-icon-type="svg" link-icon-color="#1da1f2"}** ::: aux-links-transclude-file [`https://x.com/_JohnnyJones/status/2097357027202777199`](https://x.com/_JohnnyJones/status/2097357027202777199 "_JohnnyJones 2026"){.link-modified-recently .id-not .include-content} ::: - **[`https://x.com/dylan522p/status/1859302712803807696`{=html}](https://x.com/dylan522p/status/1859302712803807696 "Patel 2026"){.id-not .include-annotation link-icon="twitter" link-icon-type="svg" link-icon-color="#1da1f2"}** ::: aux-links-transclude-file [`https://x.com/dylan522p/status/1859302712803807696`](https://x.com/dylan522p/status/1859302712803807696 "Patel 2026"){.id-not .include-content} ::: - **[`https://x.com/elonmusk/status/1797382701541990841`{=html}](https://x.com/elonmusk/status/1797382701541990841 "Musk 2026"){.id-not .include-annotation link-icon="twitter" link-icon-type="svg" link-icon-color="#1da1f2"}** ::: aux-links-transclude-file [`https://x.com/elonmusk/status/1797382701541990841`](https://x.com/elonmusk/status/1797382701541990841 "Musk 2026"){.id-not .include-content} ::: - **[`https://x.com/emollick/status/1759633391098732967`{=html}](https://x.com/emollick/status/1759633391098732967 "Mollick 2026"){.id-not .include-annotation link-icon="twitter" link-icon-type="svg" link-icon-color="#1da1f2"}** ::: aux-links-transclude-file [`https://x.com/emollick/status/1759633391098732967`](https://x.com/emollick/status/1759633391098732967 "Mollick 2026"){.id-not .include-content} ::: - **[`https://x.com/jordanschnyc/status/1580889342402129921`{=html}](https://x.com/jordanschnyc/status/1580889342402129921 "jordanschnyc 2026"){.link-annotated-partial .id-not .include-annotation link-icon="twitter" link-icon-type="svg" link-icon-color="#1da1f2"}** - **[`https://x.com/jukan05/status/2047516566149816627`{=html}](https://x.com/jukan05/status/2047516566149816627 "jukan05 2026"){.id-not .include-annotation link-icon="twitter" link-icon-type="svg" link-icon-color="#1da1f2"}** ::: aux-links-transclude-file [`https://x.com/jukan05/status/2047516566149816627`](https://x.com/jukan05/status/2047516566149816627 "jukan05 2026"){.id-not .include-content} ::: - **[`https://x.com/miolini/status/1634982361757790209`{=html}](https://x.com/miolini/status/1634982361757790209 "Andreenko 2026"){.link-annotated-partial .id-not .include-annotation link-icon="twitter" link-icon-type="svg" link-icon-color="#1da1f2"}** - **[`https://x.com/mkratsios47/status/2079933645888880708`{=html}](https://x.com/mkratsios47/status/2079933645888880708 "mkratsios47 2026"){.link-annotated-partial .id-not .include-annotation link-icon="twitter" link-icon-type="svg" link-icon-color="#1da1f2"}** - **[`https://x.com/ohlennart/status/1882967853177192876`{=html}](https://x.com/ohlennart/status/1882967853177192876 "ohlennart 2026"){.id-not .include-annotation link-icon="twitter" link-icon-type="svg" link-icon-color="#1da1f2"}** ::: aux-links-transclude-file [`https://x.com/ohlennart/status/1882967853177192876`](https://x.com/ohlennart/status/1882967853177192876 "ohlennart 2026"){.id-not .include-content} ::: - **[`https://x.com/pirroh/status/1694516986561307022`{=html}](https://x.com/pirroh/status/1694516986561307022 "pirroh 2026"){.id-not .include-annotation link-icon="twitter" link-icon-type="svg" link-icon-color="#1da1f2"}** ::: aux-links-transclude-file [`https://x.com/pirroh/status/1694516986561307022`](https://x.com/pirroh/status/1694516986561307022 "pirroh 2026"){.id-not .include-content} ::: - **[`https://x.com/ptrschmdtnlsn/status/1669590814329036803`{=html}](https://x.com/ptrschmdtnlsn/status/1669590814329036803 "ptrschmdtnlsn 2026"){.link-annotated-partial .id-not .include-annotation link-icon="twitter" link-icon-type="svg" link-icon-color="#1da1f2"}** - **[`https://x.com/sama/status/1739360234832052641`{=html}](https://x.com/sama/status/1739360234832052641 "Altman 2026"){.id-not .include-annotation link-icon="twitter" link-icon-type="svg" link-icon-color="#1da1f2"}** ::: aux-links-transclude-file [`https://x.com/sama/status/1739360234832052641`](https://x.com/sama/status/1739360234832052641 "Altman 2026"){.id-not .include-content} ::: - **[`https://x.com/scott_e_reed/status/1884672654658941257`{=html}](https://x.com/scott_e_reed/status/1884672654658941257 "scott_e_reed 2026"){.link-annotated-partial .id-not .include-annotation link-icon="twitter" link-icon-type="svg" link-icon-color="#1da1f2"}** - **[`https://x.com/teortaxesTex/status/2080142814550147376`{=html}](https://x.com/teortaxesTex/status/2080142814550147376 "teortaxesTex 2026"){.id-not .include-annotation link-icon="twitter" link-icon-type="svg" link-icon-color="#1da1f2"}** ::: aux-links-transclude-file [`https://x.com/teortaxesTex/status/2080142814550147376`](https://x.com/teortaxesTex/status/2080142814550147376 "teortaxesTex 2026"){.id-not .include-content} ::: - **[`https://x.com/teortaxesTex/status/2080143652207100339`{=html}](https://x.com/teortaxesTex/status/2080143652207100339 "teortaxesTex 2026"){.id-not .include-annotation link-icon="twitter" link-icon-type="svg" link-icon-color="#1da1f2"}** ::: aux-links-transclude-file [`https://x.com/teortaxesTex/status/2080143652207100339`](https://x.com/teortaxesTex/status/2080143652207100339 "teortaxesTex 2026"){.id-not .include-content} ::: - **[`https://x.com/teortaxesTex/status/2080146784853045312`{=html}](https://x.com/teortaxesTex/status/2080146784853045312 "teortaxesTex 2026"){.id-not .include-annotation link-icon="twitter" link-icon-type="svg" link-icon-color="#1da1f2"}** ::: aux-links-transclude-file [`https://x.com/teortaxesTex/status/2080146784853045312`](https://x.com/teortaxesTex/status/2080146784853045312 "teortaxesTex 2026"){.id-not .include-content} ::: - **[`https://x.com/teortaxesTex/status/2080152343383810316`{=html}](https://x.com/teortaxesTex/status/2080152343383810316 "teortaxesTex 2026"){.id-not .include-annotation link-icon="twitter" link-icon-type="svg" link-icon-color="#1da1f2"}** ::: aux-links-transclude-file [`https://x.com/teortaxesTex/status/2080152343383810316`](https://x.com/teortaxesTex/status/2080152343383810316 "teortaxesTex 2026"){.id-not .include-content} ::: - **[`https://x.com/thiteanish/status/1635188333705043969`{=html}](https://x.com/thiteanish/status/1635188333705043969 "thiteanish 2026"){.link-annotated-partial .id-not .include-annotation link-icon="twitter" link-icon-type="svg" link-icon-color="#1da1f2"}** - **[`https://x.com/transitive_bs/status/1628118163874516992`{=html}](https://x.com/transitive_bs/status/1628118163874516992 "transitive_bs 2026"){.link-annotated-partial .id-not .include-annotation link-icon="twitter" link-icon-type="svg" link-icon-color="#1da1f2"}** - **[`https://x.com/ylecun/status/1612182019861094402`{=html}](https://x.com/ylecun/status/1612182019861094402 "LeCun 2026"){.link-annotated-partial .id-not .include-annotation link-icon="twitter" link-icon-type="svg" link-icon-color="#1da1f2"}** - **[`https://x.com/zeeshanp_/status/2050737682440175903`{=html}](https://x.com/zeeshanp_/status/2050737682440175903 "Patel 2026"){.id-not .include-annotation link-icon="twitter" link-icon-type="svg" link-icon-color="#1da1f2"}** ::: aux-links-transclude-file [`https://x.com/zeeshanp_/status/2050737682440175903`](https://x.com/zeeshanp_/status/2050737682440175903 "Patel 2026"){.id-not .include-content} ::: # Bibliography {#link-bibliography-section} 1. `https://www.theverge.com/command-line-newsletter/759897/sam-altman-chatgpt-openai-social-media-google-chrome-interview`: [`“I Talked to Sam Altman about the GPT-5 Launch Fiasco: Over Dinner, OpenAI CEO’s Addressed Criticism of GPT-5’s Rollout, the AI Bubble, Brain-Computer Interfaces, Buying Google Chrome, and More”`{=html}](https://www.theverge.com/command-line-newsletter/759897/sam-altman-chatgpt-openai-social-media-google-chrome-interview "'I talked to Sam Altman about the GPT-5 launch fiasco: Over dinner, OpenAI CEO’s addressed criticism of GPT-5’s rollout, the AI bubble, brain-computer interfaces, buying Google Chrome, and more', Heath 2025"){#heath-2025 .link-annotated}, [[Alex Heath](https://en.wikipedia.org/wiki/Alex_Heath "Alex Heath"){.id-not}]{.author .cite-author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Fwww.theverge.com%252Fcommand-line-newsletter%252F759897%252Fsam-altman-chatgpt-openai-social-media-google-chrome-interview.html "Directory-tag link-bibliography for link https://www.theverge.com/command-line-newsletter/759897/sam-altman-chatgpt-openai-social-media-google-chrome-interview"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 2. `https://x.com/khoomeik/status/1953560406381015259`: [`“[GPT-5 Was a <100× GPT-4 Scaleup]”`{=html}](https://x.com/khoomeik/status/1953560406381015259 "'[GPT-5 was a <100× GPT-4 scaleup]', Pandey 2025"){#pandey-2025 .link-annotated}, [[Rohan Pandey](https://rpandey.tech/ "'Rohan Pandey homepage', Pandey 2026"){.link-annotated .id-not}]{.author .cite-author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Fx.com%252Fkhoomeik%252Fstatus%252F1953560406381015259.html "Directory-tag link-bibliography for link https://x.com/khoomeik/status/1953560406381015259"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 3. `https://www.reuters.com/world/china/deepseek-r2-launch-stalled-ceo-balks-progress-information-reports-2025-06-26/`: [`“DeepSeek R2 Launch Stalled As CEO Balks at Progress, The Information Reports”`{=html}](https://www.reuters.com/world/china/deepseek-r2-launch-stalled-ceo-balks-progress-information-reports-2025-06-26/ "'DeepSeek R2 launch stalled as CEO balks at progress, The Information reports', Sophia 2025"){#sophia-2025 .link-annotated}, [Deborah Sophia]{.author .cite-author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Fwww.reuters.com%252Fworld%252Fchina%252Fdeepseek-r2-launch-stalled-ceo-balks-progress-information-reports-2025-06-26%252F.html "Directory-tag link-bibliography for link https://www.reuters.com/world/china/deepseek-r2-launch-stalled-ceo-balks-progress-information-reports-2025-06-26/"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 4. `https://arxiv.org/abs/2505.09343#deepseek`: [`“Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures”`{=html}](https://arxiv.org/abs/2505.09343#deepseek "'Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures', Zhao et al 2025"){#zhao-et-al-2025 .link-annotated}, [Chenggang Zhao, Chengqi Deng, Chong Ruan[, Damai Dai, Huazuo Gao, Jiashi Li, Liyue Zhang, Panpan Huang, Shangyan Zhou, Shirong Ma, [Liang Wenfeng](https://en.wikipedia.org/wiki/Liang_Wenfeng "Liang Wenfeng"){.id-not}, Ying He, Yuqing Wang, Yuxuan Liu, Y. X. Wei]{.collapse}]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2505.09343%2523deepseek.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2505.09343#deepseek"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 5. `https://www.reuters.com/technology/artificial-intelligence/alphabet-nvidia-invest-openai-co-founder-sutskevers-ssi-source-says-2025-04-12/`: [`“Alphabet, Nvidia Invest in OpenAI Co-Founder Sutskever’s SSI, Source Says”`{=html}](https://www.reuters.com/technology/artificial-intelligence/alphabet-nvidia-invest-openai-co-founder-sutskevers-ssi-source-says-2025-04-12/ "'Alphabet, Nvidia invest in OpenAI co-founder Sutskever’s SSI, source says', Cai & Hu 2025"){#cai-hu-2025 .link-annotated}, [Kenrick Cai, [Krystal Hu](https://www.reuters.com/authors/krystal-hu/){.id-not}]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Fwww.reuters.com%252Ftechnology%252Fartificial-intelligence%252Falphabet-nvidia-invest-openai-co-founder-sutskevers-ssi-source-says-2025-04-12%252F.html "Directory-tag link-bibliography for link https://www.reuters.com/technology/artificial-intelligence/alphabet-nvidia-invest-openai-co-founder-sutskevers-ssi-source-says-2025-04-12/"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 6. `https://benjamintodd.substack.com/p/why-a-us-ai-manhattan-project-could`: [`“Why a US AI "Manhattan Project" Could Backfire: Notes from Conversations in China”`{=html}](https://benjamintodd.substack.com/p/why-a-us-ai-manhattan-project-could "'Why a US AI "Manhattan Project" could backfire: notes from conversations in China', Todd 2024"){#todd-2024 .link-annotated}, [[Benjamin Todd](https://benjamintodd.org/#about "Benjamin Todd"){.id-not}]{.author .cite-author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Fbenjamintodd.substack.com%252Fp%252Fwhy-a-us-ai-manhattan-project-could.html "Directory-tag link-bibliography for link https://benjamintodd.substack.com/p/why-a-us-ai-manhattan-project-could"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 7. `https://www.theatlantic.com/international/archive/2024/06/china-microchip-technology-competition/678612/`: [`“China Is Losing the Chip War: Xi Jinping Picked a Fight over Semiconductor Technology—One He Can’t Win”`{=html}](https://www.theatlantic.com/international/archive/2024/06/china-microchip-technology-competition/678612/ "'China Is Losing the Chip War: Xi Jinping picked a fight over semiconductor technology—one he can’t win', Schuman 2024"){#schuman-2024 .link-annotated}, [Michael Schuman]{.author .cite-author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Fwww.theatlantic.com%252Finternational%252Farchive%252F2024%252F06%252Fchina-microchip-technology-competition%252F678612%252F.html "Directory-tag link-bibliography for link https://www.theatlantic.com/international/archive/2024/06/china-microchip-technology-competition/678612/"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 8. `https://www.ft.com/content/8e8a65a0-a990-4c77-a6e8-ec4e5d247f80`: [`“Singapore’s Temasek in Discussions to Invest in OpenAI: State-Backed Group in Talks With ChatGPT Maker’s Chief Sam Altman Who Is Seeking Funding to Build Chips Business”`{=html}](https://www.ft.com/content/8e8a65a0-a990-4c77-a6e8-ec4e5d247f80 "'Singapore’s Temasek in discussions to invest in OpenAI: State-backed group in talks with ChatGPT maker’s chief Sam Altman who is seeking funding to build chips business', Murgia & Ruehl 2024"){#murgia-ruehl-2024 .link-annotated}, [Madhumita Murgia, Mercedes Ruehl]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Fwww.ft.com%252Fcontent%252F8e8a65a0-a990-4c77-a6e8-ec4e5d247f80.html "Directory-tag link-bibliography for link https://www.ft.com/content/8e8a65a0-a990-4c77-a6e8-ec4e5d247f80"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 9. `https://www.newyorker.com/magazine/2023/12/04/how-jensen-huangs-nvidia-is-powering-the-ai-revolution`: [`“How Jensen Huang’s Nvidia Is Powering the AI Revolution: The Company’s CEO Bet It All on a New Kind of Chip. Now That Nvidia Is One of the Biggest Companies in the World, What Will He Do Next?”`{=html}](https://www.newyorker.com/magazine/2023/12/04/how-jensen-huangs-nvidia-is-powering-the-ai-revolution "'How Jensen Huang’s Nvidia Is Powering the AI Revolution: The company’s CEO bet it all on a new kind of chip. Now that Nvidia is one of the biggest companies in the world, what will he do next?', Witt 2023"){#witt-2023 .link-annotated}, [Stephen Witt]{.author .cite-author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Fwww.newyorker.com%252Fmagazine%252F2023%252F12%252F04%252Fhow-jensen-huangs-nvidia-is-powering-the-ai-revolution.html "Directory-tag link-bibliography for link https://www.newyorker.com/magazine/2023/12/04/how-jensen-huangs-nvidia-is-powering-the-ai-revolution"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 10. `https://www.wired.com/story/openai-buy-ai-chips-startup-sam-altman/`: [`“OpenAI Agreed to Buy $51 Million of AI Chips From a Startup Backed by CEO Sam Altman”`{=html}](https://www.wired.com/story/openai-buy-ai-chips-startup-sam-altman/ "'OpenAI Agreed to Buy $51 Million of AI Chips From a Startup Backed by CEO Sam Altman', Dave 2023"){#dave-2023-2 .link-annotated}, [Paresh Dave]{.author .cite-author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Fwww.wired.com%252Fstory%252Fopenai-buy-ai-chips-startup-sam-altman%252F.html "Directory-tag link-bibliography for link https://www.wired.com/story/openai-buy-ai-chips-startup-sam-altman/"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 11. `https://www.semianalysis.com/p/microsoft-swallows-openais-core-team#%C2%A7compute-is-king`: [`“Microsoft Swallows OpenAI’s Core Team § Compute Is King”`{=html}](https://www.semianalysis.com/p/microsoft-swallows-openais-core-team#%C2%A7compute-is-king "'Microsoft Swallows OpenAI’s Core Team § Compute Is King', Patel & Nishball 2023"){#patel-nishball-2023-1 .link-annotated}, [Dylan Patel, Daniel Nishball]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Fwww.semianalysis.com%252Fp%252Fmicrosoft-swallows-openais-core-team%2523%2525C2%2525A7compute-is-king.html "Directory-tag link-bibliography for link https://www.semianalysis.com/p/microsoft-swallows-openais-core-team#%C2%A7compute-is-king"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 12. `https://www.ft.com/content/2a636cee-b0d2-45c2-a815-11ca32371763`: [`“Saudi-China Collaboration Raises Concerns about Access to AI Chips: Fears Grow at Gulf Kingdom’s Top University That Ties to Chinese Researchers Risk Upsetting US Government”`{=html}](https://www.ft.com/content/2a636cee-b0d2-45c2-a815-11ca32371763 "'Saudi-China collaboration raises concerns about access to AI chips: Fears grow at Gulf kingdom’s top university that ties to Chinese researchers risk upsetting US government', Kerr et al 2023"){#kerr-et-al-2023 .link-annotated}, [Simeon Kerr, Samer Al-Atrush, Qianer Liu, Madhumita Murgia]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Fwww.ft.com%252Fcontent%252F2a636cee-b0d2-45c2-a815-11ca32371763.html "Directory-tag link-bibliography for link https://www.ft.com/content/2a636cee-b0d2-45c2-a815-11ca32371763"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 13. `https://www.nytimes.com/2023/07/16/opinion/biden-china-ai-chips-trade.html`: [`“Biden Is Beating China on Chips. It May Not Be Enough.”`{=html}](https://www.nytimes.com/2023/07/16/opinion/biden-china-ai-chips-trade.html "'Biden Is Beating China on Chips. It May Not Be Enough.', Wang 2023"){#wang-2023-1 .link-annotated}, [Dan Wang]{.author .cite-author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Fwww.nytimes.com%252F2023%252F07%252F16%252Fopinion%252Fbiden-china-ai-chips-trade.html.html "Directory-tag link-bibliography for link https://www.nytimes.com/2023/07/16/opinion/biden-china-ai-chips-trade.html"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 14. `https://archive.is/c5jTk`: [`“Deep Mind’s Chief on AI’s Dangers—And the UK’s £900 Million Supercomputer: Demis Hassabis Says We Shouldn’t Let AI Fall into the Wrong Hands and the Government’s Plan to Build a Supercomputer for AI Is Likely to Be out of Date Before It Has Even Started”`{=html}](https://archive.is/c5jTk "'Deep Mind’s chief on AI’s dangers—and the UK’s £900 million supercomputer: Demis Hassabis says we shouldn’t let AI fall into the wrong hands and the government’s plan to build a supercomputer for AI is likely to be out of date before it has even started', Sellman 2023"){#sellman-2023 .link-annotated}, [Mark Sellman]{.author .cite-author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farchive.is%252Fc5jTk.html "Directory-tag link-bibliography for link https://archive.is/c5jTk"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 15. `https://www.wsj.com/articles/the-ai-boom-runs-on-chips-but-it-cant-get-enough-9f76f554`: [`“The AI Boom Runs on Chips, but It Can’t Get Enough: ‘It’s like Toilet Paper during the Pandemic.’ Startups, Investors Scrounge for Computational Firepower”`{=html}](https://www.wsj.com/articles/the-ai-boom-runs-on-chips-but-it-cant-get-enough-9f76f554 "'The AI Boom Runs on Chips, but It Can’t Get Enough: ‘It’s like toilet paper during the pandemic.’ Startups, investors scrounge for computational firepower', Seetharaman & Dotan 2023"){#seetharaman-dotan-2023 .link-annotated}, [[Deepa Seetharaman](https://x.com/dseetharaman "dseetharaman 2026"){.id-not}, Tom Dotan]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Fwww.wsj.com%252Farticles%252Fthe-ai-boom-runs-on-chips-but-it-cant-get-enough-9f76f554.html "Directory-tag link-bibliography for link https://www.wsj.com/articles/the-ai-boom-runs-on-chips-but-it-cant-get-enough-9f76f554"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 16. `https://arxiv.org/abs/2305.06946`: [`“Big-PERCIVAL: Exploring the Native Use of 64-Bit Posit Arithmetic in Scientific Computing”`{=html}](https://arxiv.org/abs/2305.06946 "'Big-PERCIVAL: Exploring the Native Use of 64-Bit Posit Arithmetic in Scientific Computing', Mallasén et al 2023"){#mallasén-et-al-2023 .link-annotated}, [David Mallasén, Alberto A. Del Barrio, Manuel Prieto-Matias]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2305.06946.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2305.06946"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 17. `https://arxiv.org/abs/2304.01433#google`: [`“TPU V4: An Optically Reconfigurable Supercomputer for Machine Learning With Hardware Support for Embeddings”`{=html}](https://arxiv.org/abs/2304.01433#google "'TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings', Jouppi et al 2023"){#jouppi-et-al-2023 .link-annotated}, [[Norman P. Jouppi](https://en.wikipedia.org/wiki/Norman_P._Jouppi "Norman P. Jouppi"){.id-not}, [George Kurian](https://en.wikipedia.org/wiki/George_Kurian "George Kurian"){.id-not}, Sheng Li[, Peter Ma, Rahul Nagarajan, Lifeng Nai, Nishant Patil, Suvinay Subramanian, Andy Swing, Brian Towles, Cliff Young, Xiang Zhou, Zongwei Zhou, David Patterson]{.collapse}]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2304.01433%2523google.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2304.01433#google"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 18. `https://x.com/davidtayar5/status/1627690520456691712`: [`“Context on the NVIDIA ChatGPT Opportunity—And Ramifications of Large Language Model Enthusiasm”`{=html}](https://x.com/davidtayar5/status/1627690520456691712 "'Context on the NVIDIA ChatGPT opportunity—and ramifications of large language model enthusiasm', Stanley 2023"){#stanley-2023 .link-annotated}, [[Morgan Stanley](https://en.wikipedia.org/wiki/Morgan_Stanley "Morgan Stanley"){.id-not}]{.author .cite-author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Fx.com%252Fdavidtayar5%252Fstatus%252F1627690520456691712.html "Directory-tag link-bibliography for link https://x.com/davidtayar5/status/1627690520456691712"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 19. `https://blogs.microsoft.com/blog/2023/01/23/microsoftandopenaiextendpartnership/`: [`“Microsoft and OpenAI Extend Partnership”`{=html}](https://blogs.microsoft.com/blog/2023/01/23/microsoftandopenaiextendpartnership/ "'Microsoft and OpenAI extend partnership', Microsoft 2023"){#microsoft-2023 .link-annotated}, [Microsoft]{.author .cite-author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Fblogs.microsoft.com%252Fblog%252F2023%252F01%252F23%252Fmicrosoftandopenaiextendpartnership%252F.html "Directory-tag link-bibliography for link https://blogs.microsoft.com/blog/2023/01/23/microsoftandopenaiextendpartnership/"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 20. `https://ai.meta.com/blog/multiray-large-scale-AI-models/`: [`“MultiRay: Optimizing Efficiency for Large-Scale AI Models”`{=html}](https://ai.meta.com/blog/multiray-large-scale-AI-models/ "'MultiRay: Optimizing efficiency for large-scale AI models', Gupta et al 2022"){#gupta-et-al-2022-2 .link-annotated}, [[Nikhil Gupta](https://www.linkedin.com/in/gnikhil){.id-not}, [Michael Gschwind](https://en.wikipedia.org/wiki/Michael_Gschwind "Michael Gschwind"){.id-not}, Don Husa[, Christopher Dewan, Madian Khabsa]{.collapse}]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Fai.meta.com%252Fblog%252Fmultiray-large-scale-AI-models%252F.html "Directory-tag link-bibliography for link https://ai.meta.com/blog/multiray-large-scale-AI-models/"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 21. `https://arxiv.org/abs/2211.05102#google`: [`“Efficiently Scaling Transformer Inference”`{=html}](https://arxiv.org/abs/2211.05102#google "'Efficiently Scaling Transformer Inference', Pope et al 2022"){#pope-et-al-2022 .link-annotated}, [Reiner Pope, [Sholto Douglas](https://en.wikipedia.org/wiki/Sholto_Douglas "Sholto Douglas"){.id-not}, Aakanksha Chowdhery[, Jacob Devlin, [James Bradbury](https://x.com/jekbradbury "Bradbury 2026"){.id-not}, Anselm Levskaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal, [Jeff Dean](https://en.wikipedia.org/wiki/Jeff_Dean "Jeff Dean"){.link-annotated-partial .id-not}]{.collapse}]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2211.05102%2523google.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2211.05102#google"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 22. `https://arxiv.org/abs/2206.03382#microsoft`: [`“Tutel: Adaptive Mixture-Of-Experts at Scale”`{=html}](https://arxiv.org/abs/2206.03382#microsoft "'Tutel: Adaptive Mixture-of-Experts at Scale', Hwang et al 2022"){#hwang-et-al-2022-2 .link-annotated}, [Changho Hwang, Wei Cui, Yifan Xiong[, Ziyue Yang, Ze Liu, Han Hu, Zilong Wang, [Rafael Salas](https://en.wikipedia.org/wiki/Rafael_Salas "Rafael Salas"){.id-not}, Jithin Jose, Prabhat Ram, Joe Chau, [Peng Cheng](https://en.wikipedia.org/wiki/Peng_Cheng "Peng Cheng"){.id-not}, Fan Yang, Mao Yang, Yongqiang Xiong]{.collapse}]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2206.03382%2523microsoft.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2206.03382#microsoft"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 23. `https://arxiv.org/abs/2206.01861#microsoft`: [`“ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers”`{=html}](https://arxiv.org/abs/2206.01861#microsoft "'ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers', Yao et al 2022"){#yao-et-al-2022-2 .link-annotated}, [Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang[, Xiaoxia Wu, Conglong Li, Yuxiong He]{.collapse}]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2206.01861%2523microsoft.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2206.01861#microsoft"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 24. `https://arxiv.org/abs/2205.14135`: [`“FlashAttention: Fast and Memory-Efficient Exact Attention With IO-Awareness”`{=html}](https://arxiv.org/abs/2205.14135 "'FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness', Dao et al 2022"){#dao-et-al-2022-1 .link-annotated}, [[Tri Dao](https://tridao.me/ "Tri Dao"){.link-annotated-partial .id-not}, Daniel Y. Fu, [Stefano Ermon](https://scholar.google.com/citations?user=ogXTOZ4AAAAJ "Stefano Ermon"){.id-not}[, [Atri Rudra](https://cse.buffalo.edu/faculty/atri/ "Atri Rudra"){.id-not}, [Christopher Ré](https://en.wikipedia.org/wiki/Christopher_R%C3%A9 "Christopher Ré"){.id-not}]{.collapse}]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2205.14135.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2205.14135"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 25. `https://arxiv.org/abs/2204.00595`: [`“Monarch: Expressive Structured Matrices for Efficient and Accurate Training”`{=html}](https://arxiv.org/abs/2204.00595 "'Monarch: Expressive Structured Matrices for Efficient and Accurate Training', Dao et al 2022"){#dao-et-al-2022-2 .link-annotated}, [[Tri Dao](https://tridao.me/ "Tri Dao"){.link-annotated-partial .id-not}, Beidi Chen, Nimit Sohoni[, Arjun Desai, Michael Poli, Jessica Grogan, Alexander Liu, Aniruddh Rao, [Atri Rudra](https://cse.buffalo.edu/faculty/atri/ "Atri Rudra"){.id-not}, [Christopher Ré](https://en.wikipedia.org/wiki/Christopher_R%C3%A9 "Christopher Ré"){.id-not}]{.collapse}]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2204.00595.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2204.00595"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 26. `https://arxiv.org/abs/2203.02094#microsoft`: [`“LiteTransformerSearch: Training-Free Neural Architecture Search for Efficient Language Models”`{=html}](https://arxiv.org/abs/2203.02094#microsoft "'LiteTransformerSearch: Training-free Neural Architecture Search for Efficient Language Models', Javaheripi et al 2022"){#javaheripi-et-al-2022 .link-annotated}, [Mojan Javaheripi, Gustavo H. de Rosa, Subhabrata Mukherjee[, Shital Shah, Tomasz L. Religa, Caio C. T. Mendes, [Sebastien Bubeck](https://en.wikipedia.org/wiki/Sebastien_Bubeck "Sebastien Bubeck"){.id-not}, [Farinaz Koushanfar](https://en.wikipedia.org/wiki/Farinaz_Koushanfar "Farinaz Koushanfar"){.id-not}, Debadeepta Dey]{.collapse}]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2203.02094%2523microsoft.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2203.02094#microsoft"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 27. `https://arxiv.org/abs/2202.06009#microsoft`: [`“Maximizing Communication Efficiency for Large-Scale Training via 0/1 Adam”`{=html}](https://arxiv.org/abs/2202.06009#microsoft "'Maximizing Communication Efficiency for Large-scale Training via 0/1 Adam', Lu et al 2022"){#lu-et-al-2022-7 .link-annotated}, [Yucheng Lu, Conglong Li, Minjia Zhang[, Christopher De Sa, Yuxiong He]{.collapse}]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2202.06009%2523microsoft.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2202.06009#microsoft"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 28. `https://ai.meta.com/blog/ai-rsc/`: [`“Introducing the AI Research SuperCluster—Facebook’s Cutting-Edge AI Supercomputer for AI Research”`{=html}](https://ai.meta.com/blog/ai-rsc/ "'Introducing the AI Research SuperCluster—Facebook’s cutting-edge AI supercomputer for AI research', Lee & Sengupta 2022"){#lee-sengupta-2022 .link-annotated}, [Kevin Lee, Shubho Sengupta]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Fai.meta.com%252Fblog%252Fai-rsc%252F.html "Directory-tag link-bibliography for link https://ai.meta.com/blog/ai-rsc/"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 29. `https://semiengineering.com/is-programmable-overhead-worth-the-cost/`: [`“Is Programmable Overhead Worth The Cost? How Much Do We Pay for a System to Be Programmable? It Depends upon Who You Ask”`{=html}](https://semiengineering.com/is-programmable-overhead-worth-the-cost/ "'Is Programmable Overhead Worth The Cost? How much do we pay for a system to be programmable? It depends upon who you ask', Bailey 2022"){#bailey-2022 .link-annotated}, [Brian Bailey]{.author .cite-author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Fsemiengineering.com%252Fis-programmable-overhead-worth-the-cost%252F.html "Directory-tag link-bibliography for link https://semiengineering.com/is-programmable-overhead-worth-the-cost/"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 30. `https://arxiv.org/abs/2106.10207`: [`“Distributed Deep Learning in Open Collaborations”`{=html}](https://arxiv.org/abs/2106.10207 "'Distributed Deep Learning in Open Collaborations', Diskin et al 2021"){#diskin-et-al-2021 .link-annotated}, [Michael Diskin, Alexey Bukhtiyarov, Max Ryabinin[, Lucile Saulnier, Quentin Lhoest, Anton Sinitsin, Dmitry Popov, Dmitry Pyrkin, Maxim Kashirin, Alexander Borzunov, Albert Villanova del Moral, Denis Mazur, Ilia Kobelev, Yacine Jernite, [Thomas Wolf](https://thomwolf.io/ "Thomas Wolf"){.link-annotated-partial .id-not}, Gennady Pekhimenko]{.collapse}]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2106.10207.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2106.10207"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 31. `2021-jouppi.pdf`: [`“Ten Lessons From Three Generations Shaped Google’s TPUv4i”`{=html}](/doc/ai/scaling/hardware/2021-jouppi.pdf "'Ten Lessons From Three Generations Shaped Google’s TPUv4i', Jouppi et al 2021"){#jouppi-et-al-2021 .link-annotated}, [[Norman P. Jouppi](https://en.wikipedia.org/wiki/Norman_P._Jouppi "Norman P. Jouppi"){.id-not}, Doe Hyun Yoon, Matthew Ashcraft[, Mark Gottscho, Thomas B. Jablin, [George Kurian](https://en.wikipedia.org/wiki/George_Kurian "George Kurian"){.id-not}, James Laudon, Sheng Li, Peter Ma, Xiaoyu Ma, Thomas Norrie, Nishant Patil, Sushma Prasad, Cliff Young, Zongwei Zhou, David Patterson]{.collapse}]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/%252Fdoc%252Fai%252Fscaling%252Fhardware%252F2021-jouppi.pdf.html "Directory-tag link-bibliography for link /doc/ai/scaling/hardware/2021-jouppi.pdf"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 32. `https://chinai.substack.com/p/chinai-141-the-pangu-origin-story`: [`“ChinAI #141: The PanGu Origin Story: Notes from an Informative Zhihu Thread on PanGu”`{=html}](https://chinai.substack.com/p/chinai-141-the-pangu-origin-story "'ChinAI #141: The PanGu Origin Story: Notes from an informative Zhihu Thread on PanGu', Ding 2021"){#ding-2021-1 .link-annotated}, [[Jeffrey Ding](https://jeffreyjding.github.io/ "Jeffrey Ding"){.id-not}]{.author .cite-author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Fchinai.substack.com%252Fp%252Fchinai-141-the-pangu-origin-story.html "Directory-tag link-bibliography for link https://chinai.substack.com/p/chinai-141-the-pangu-origin-story"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 33. `https://arxiv.org/abs/2104.06272#deepmind`: [`“Podracer Architectures for Scalable Reinforcement Learning”`{=html}](https://arxiv.org/abs/2104.06272#deepmind "'Podracer architectures for scalable Reinforcement Learning', Hessel et al 2021"){#hessel-et-al-2021-2 .link-annotated}, [Matteo Hessel, Manuel Kroiss, Aidan Clark[, Iurii Kemaev, [John Quan](https://scholar.google.com/citations?user=rcojd6kAAAAJ "John Quan"){.id-not}, Thomas Keck, Fabio Viola, [Hado van Hasselt](https://scholar.google.com/citations?user=W80oBMkAAAAJ "Hado van Hasselt"){.id-not}]{.collapse}]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2104.06272%2523deepmind.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2104.06272#deepmind"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 34. `https://arxiv.org/abs/2104.05343`: [`“An Efficient 2D Method for Training Super-Large Deep Learning Models”`{=html}](https://arxiv.org/abs/2104.05343 "'An Efficient 2D Method for Training Super-Large Deep Learning Models', Xu et al 2021"){#xu-et-al-2021-8 .link-annotated}, [Qifan Xu, Shenggui Li, Chaoyu Gong, [Yang You](https://en.wikipedia.org/wiki/Yang_You "Yang You"){.id-not}]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2104.05343.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2104.05343"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 35. `https://arxiv.org/abs/2102.07988`: [`“TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models”`{=html}](https://arxiv.org/abs/2102.07988 "'TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models', Li et al 2021"){#li-et-al-2021-8 .link-annotated}, [Zhuohan Li, Siyuan Zhuang, Shiyuan Guo[, Danyang Zhuo, Hao Zhang, [Dawn Song](https://en.wikipedia.org/wiki/Dawn_Song "Dawn Song"){.id-not}, [Ion Stoica](https://en.wikipedia.org/wiki/Ion_Stoica "Ion Stoica"){.id-not}]{.collapse}]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2102.07988.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2102.07988"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 36. `https://arxiv.org/abs/2102.03161`: [`“PipeTransformer: Automated Elastic Pipelining for Distributed Training of Transformers”`{=html}](https://arxiv.org/abs/2102.03161 "'PipeTransformer: Automated Elastic Pipelining for Distributed Training of Transformers', He et al 2021"){#he-et-al-2021-4 .link-annotated}, [Chaoyang He, [Shen Li](https://en.wikipedia.org/wiki/Shen_Li "Shen Li"){.id-not}, Mahdi Soltanolkotabi, [Salman Avestimehr](https://en.wikipedia.org/wiki/Salman_Avestimehr "Salman Avestimehr"){.id-not}]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2102.03161.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2102.03161"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 37. `2020-jiang.pdf`: [`“BytePS: A Unified Architecture for Accelerating Distributed DNN Training in Heterogeneous GPU/CPU Clusters”`{=html}](/doc/ai/scaling/hardware/2020-jiang.pdf "'BytePS: A Unified Architecture for Accelerating Distributed DNN Training in Heterogeneous GPU/CPU Clusters', Jiang et al 2020"){#jiang-et-al-2020-2 .link-annotated}, [Yimin Jiang, Yibo Zhu, Chang Lan[, Bairen Yi, Yong Cui, Chuanxiong Guo]{.collapse}]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/%252Fdoc%252Fai%252Fscaling%252Fhardware%252F2020-jiang.pdf.html "Directory-tag link-bibliography for link /doc/ai/scaling/hardware/2020-jiang.pdf"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 38. `https://arxiv.org/abs/2011.00071#google`: [`“Training EfficientNets at Supercomputer Scale: 83% ImageNet Top-1 Accuracy in One Hour”`{=html}](https://arxiv.org/abs/2011.00071#google "'Training EfficientNets at Supercomputer Scale: 83% ImageNet Top-1 Accuracy in One Hour', Wongpanich et al 2020"){#wongpanich-et-al-2020 .link-annotated}, [Arissa Wongpanich, Hieu Pham, [James Demmel](https://en.wikipedia.org/wiki/James_Demmel "James Demmel"){.id-not}[, Mingxing Tan, [Quoc V. Le](https://en.wikipedia.org/wiki/Quoc_V._Le "Quoc V. Le"){.id-not}, [Yang You](https://en.wikipedia.org/wiki/Yang_You "Yang You"){.id-not}, Sameer Kumar]{.collapse}]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2011.00071%2523google.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2011.00071#google"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 39. `2020-launay-2.pdf`: [`“Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures”`{=html}](/doc/ai/scaling/hardware/2020-launay-2.pdf "'Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures', Launay et al 2020b"){#launay-et-al-2020b .link-annotated}, [Julien Launay, Iacopo Poli, François Boniface, [Florent Krzakala](https://en.wikipedia.org/wiki/Florent_Krzakala "Florent Krzakala"){.id-not}]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/%252Fdoc%252Fai%252Fscaling%252Fhardware%252F2020-launay-2.pdf.html "Directory-tag link-bibliography for link /doc/ai/scaling/hardware/2020-launay-2.pdf"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 40. `https://www.microsoft.com/en-us/research/blog/deepspeed-extreme-scale-model-training-for-everyone/`: [`“DeepSpeed: Extreme-Scale Model Training for Everyone”`{=html}](https://www.microsoft.com/en-us/research/blog/deepspeed-extreme-scale-model-training-for-everyone/ "'DeepSpeed: Extreme-scale model training for everyone', Team et al 2020"){#team-et-al-2020 .link-annotated}, [DeepSpeed Team, Rangan Majumder, Junhua Wang]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Fwww.microsoft.com%252Fen-us%252Fresearch%252Fblog%252Fdeepspeed-extreme-scale-model-training-for-everyone%252F.html "Directory-tag link-bibliography for link https://www.microsoft.com/en-us/research/blog/deepspeed-extreme-scale-model-training-for-everyone/"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 41. `https://news.microsoft.com/source/features/ai/openai-azure-supercomputer/`: [`“Microsoft Announces New Supercomputer, Lays out Vision for Future AI Work”`{=html}](https://news.microsoft.com/source/features/ai/openai-azure-supercomputer/ "'Microsoft announces new supercomputer, lays out vision for future AI work', Langston 2020"){#langston-2020 .link-annotated}, [Jennifer Langston]{.author .cite-author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Fnews.microsoft.com%252Fsource%252Ffeatures%252Fai%252Fopenai-azure-supercomputer%252F.html "Directory-tag link-bibliography for link https://news.microsoft.com/source/features/ai/openai-azure-supercomputer/"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 42. `https://www.zdnet.com/article/startup-tenstorrent-and-competitors-show-how-computing-is-changing-ai-and-vice-versa/`: [`“Startup Tenstorrent Shows AI Is Changing Computing and vice Versa: Tenstorrent Is One of the Rush of AI Chip Makers Founded in 2016 and Finally Showing Product. The New Wave of Chips Represent a Substantial Departure from How Traditional Computer Chips Work, but Also Point to Ways That Neural Network Design May Change in the Years to Come”`{=html}](https://www.zdnet.com/article/startup-tenstorrent-and-competitors-show-how-computing-is-changing-ai-and-vice-versa/ "'Startup Tenstorrent shows AI is changing computing and vice versa: Tenstorrent is one of the rush of AI chip makers founded in 2016 and finally showing product. The new wave of chips represent a substantial departure from how traditional computer chips work, but also point to ways that neural network design may change in the years to come', Ray 2020"){#ray-2020 .link-annotated}, [Tiernan Ray]{.author .cite-author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Fwww.zdnet.com%252Farticle%252Fstartup-tenstorrent-and-competitors-show-how-computing-is-changing-ai-and-vice-versa%252F.html "Directory-tag link-bibliography for link https://www.zdnet.com/article/startup-tenstorrent-and-competitors-show-how-computing-is-changing-ai-and-vice-versa/"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 43. `2020-khan.pdf`: [`“AI Chips: What They Are and Why They Matter—An AI Chips Reference”`{=html}](/doc/ai/scaling/hardware/2020-khan.pdf "'AI Chips: What They Are and Why They Matter—An AI Chips Reference', Khan & Mann 2020"){#khan-mann-2020 .link-annotated}, [Saif M. Khan, [Alexander Mann](https://en.wikipedia.org/wiki/Alexander_Mann "Alexander Mann"){.id-not}]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/%252Fdoc%252Fai%252Fscaling%252Fhardware%252F2020-khan.pdf.html "Directory-tag link-bibliography for link /doc/ai/scaling/hardware/2020-khan.pdf"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 44. `https://arxiv.org/abs/1904.00962#google`: [`“Large Batch Optimization for Deep Learning: Training BERT in 76 Minutes”`{=html}](https://arxiv.org/abs/1904.00962#google "'Large Batch Optimization for Deep Learning: Training BERT in 76 minutes', You et al 2019"){#you-et-al-2019 .link-annotated}, [[Yang You](https://en.wikipedia.org/wiki/Yang_You "Yang You"){.id-not}, Jing Li, Sashank Reddi[, Jonathan Hseu, [Sanjiv Kumar](https://scholar.google.com/citations?user=08CNqrYAAAAJ&view_op=list_works&sortby=pubdate "Sanjiv Kumar"){.id-not}, Srinadh Bhojanapalli, Xiaodan Song, [James Demmel](https://en.wikipedia.org/wiki/James_Demmel "James Demmel"){.id-not}, [Kurt Keutzer](https://en.wikipedia.org/wiki/Kurt_Keutzer "Kurt Keutzer"){.id-not}, Cho-Jui Hsieh]{.collapse}]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F1904.00962%2523google.html "Directory-tag link-bibliography for link https://arxiv.org/abs/1904.00962#google"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 45. `https://arxiv.org/abs/1811.06965#google`: [`“GPipe: Efficient Training of Giant Neural Networks Using Pipeline Parallelism”`{=html}](https://arxiv.org/abs/1811.06965#google "'GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism', Huang et al 2018"){#huang-et-al-2018-3 .link-annotated}, [[Yanping Huang](https://scholar.google.com/citations?user=uEtBQScAAAAJ "Yanping Huang"){.id-not}, Youlong Cheng, Ankur Bapna[, [Orhan Firat](https://orhanfirat.com/ "Orhan Fırat"){.link-annotated-partial .id-not}, Mia Xu Chen, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, [Quoc V. Le](https://en.wikipedia.org/wiki/Quoc_V._Le "Quoc V. Le"){.id-not}, [Yonghui Wu](https://scholar.google.com/citations?user=bMWMlk4AAAAJ "Yonghui Wu"){.id-not}, [Zhifeng Chen](https://scholar.google.com/citations?user=EemUE4gAAAAJ "Zhifeng Chen"){.id-not}]{.collapse}]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F1811.06965%2523google.html "Directory-tag link-bibliography for link https://arxiv.org/abs/1811.06965#google"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 46. `https://arxiv.org/abs/1811.02084#google`: [`“Mesh-TensorFlow: Deep Learning for Supercomputers”`{=html}](https://arxiv.org/abs/1811.02084#google "'Mesh-TensorFlow: Deep Learning for Supercomputers', Shazeer et al 2018"){#shazeer-et-al-2018 .link-annotated}, [[Noam Shazeer](https://en.wikipedia.org/wiki/Noam_Shazeer "Noam Shazeer"){.id-not}, Youlong Cheng, [Niki Parmar](https://scholar.google.com/citations?user=q2YXPSgAAAAJ "Niki Parmar"){.id-not}[, Dustin Tran, [Ashish Vaswani](https://en.wikipedia.org/wiki/Ashish_Vaswani "Ashish Vaswani"){.id-not}, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, Blake Hechtman]{.collapse}]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F1811.02084%2523google.html "Directory-tag link-bibliography for link https://arxiv.org/abs/1811.02084#google"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 47. `https://openai.com/research/ai-and-compute`: [`“AI and Compute”`{=html}](https://openai.com/research/ai-and-compute "'AI and Compute', Amodei et al 2018"){#amodei-et-al-2018 .link-annotated}, [[Dario Amodei](https://en.wikipedia.org/wiki/Dario_Amodei "Dario Amodei"){.link-annotated-partial .id-not}, [Danny Hernandez](https://en.wikipedia.org/wiki/Danny_Hernandez "Danny Hernandez"){.id-not}, [Girish Sastry](https://girishsastry.com/ "Girish Sastry"){.link-annotated-partial .id-not}[, [Jack Clark](https://jack-clark.net/about/){.link-annotated-partial .id-not}, [Greg Brockman](https://en.wikipedia.org/wiki/Greg_Brockman "Greg Brockman"){.link-annotated-partial .id-not}, [Ilya Sutskever](https://en.wikipedia.org/wiki/Ilya_Sutskever "Ilya Sutskever"){.link-annotated-partial .id-not}]{.collapse}]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Fopenai.com%252Fresearch%252Fai-and-compute.html "Directory-tag link-bibliography for link https://openai.com/research/ai-and-compute"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 48. `https://arxiv.org/abs/1712.01887`: [`“Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training”`{=html}](https://arxiv.org/abs/1712.01887 "'Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training', Lin et al 2017"){#lin-et-al-2017-1 .link-annotated}, [Yujun Lin, [Song Han](https://hanlab.mit.edu/songhan "Song Han—Associate Professor, MIT EECS"){.link-annotated-partial .id-not}, Huizi Mao[, Yu Wang, [William J. Dally](https://en.wikipedia.org/wiki/William_J._Dally "William J. Dally"){.id-not}]{.collapse}]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F1712.01887.html "Directory-tag link-bibliography for link https://arxiv.org/abs/1712.01887"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 49. `https://arxiv.org/abs/1102.0183#schmidhuber`: [`“DanNet: Flexible, High Performance Convolutional Neural Networks for Image Classification”`{=html}](https://arxiv.org/abs/1102.0183#schmidhuber "'DanNet: Flexible, High Performance Convolutional Neural Networks for Image Classification', Ciresan et al 2011"){#ciresan-et-al-2011 .link-annotated}, [Dan Claudiu Ciresan, Ueli Meier, Jonathan Masci[, [Luca Maria Gambardella](https://en.wikipedia.org/wiki/Luca_Maria_Gambardella "Luca Maria Gambardella"){.id-not}, [Jürgen Schmidhuber](https://en.wikipedia.org/wiki/J%C3%BCrgen_Schmidhuber "Jürgen Schmidhuber"){.link-annotated-partial .id-not}]{.collapse}]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F1102.0183%2523schmidhuber.html "Directory-tag link-bibliography for link https://arxiv.org/abs/1102.0183#schmidhuber"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 50. `https://www.vetta.org/2010/12/goodbye-2010/`: [`“Goodbye 2010”`{=html}](https://www.vetta.org/2010/12/goodbye-2010/ "'Goodbye 2010', Legg 2010"){#legg-2010 .link-annotated}, [[Shane Legg](https://en.wikipedia.org/wiki/Shane_Legg "Shane Legg"){.link-annotated-partial .id-not}]{.author .cite-author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Fwww.vetta.org%252F2010%252F12%252Fgoodbye-2010%252F.html "Directory-tag link-bibliography for link https://www.vetta.org/2010/12/goodbye-2010/"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 51. `2009-raina.pdf`: [`“Large-Scale Deep Unsupervised Learning Using Graphics Processors”`{=html}](/doc/ai/scaling/hardware/2009-raina.pdf "'Large-scale deep unsupervised learning using graphics processors', Raina et al 2009"){#raina-et-al-2009 .link-annotated}, [Rajat Raina, Anand Madhavan, Andrew Y. Ng]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/%252Fdoc%252Fai%252Fscaling%252Fhardware%252F2009-raina.pdf.html "Directory-tag link-bibliography for link /doc/ai/scaling/hardware/2009-raina.pdf"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 52. `2009-patarasuk.pdf`: [`“Bandwidth Optimal All-Reduce Algorithms for Clusters of Workstations”`{=html}](/doc/ai/scaling/hardware/2009-patarasuk.pdf "'Bandwidth optimal all-reduce algorithms for clusters of workstations', Patarasuk & Yuan 2009"){#patarasuk-yuan-2009 .link-annotated}, [Pitch Patarasuk, Xin Yuan]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/%252Fdoc%252Fai%252Fscaling%252Fhardware%252F2009-patarasuk.pdf.html "Directory-tag link-bibliography for link /doc/ai/scaling/hardware/2009-patarasuk.pdf"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 53. `https://www.wired.com/1995/10/moravec/#scaling`: [`“Superhumanism: According to Hans Moravec § AI Scaling”`{=html}](https://www.wired.com/1995/10/moravec/#scaling "'Superhumanism: According to Hans Moravec § AI Scaling', Platt 1995"){#platt-1995-1 .link-annotated}, [Charles Platt]{.author .cite-author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Fwww.wired.com%252F1995%252F10%252Fmoravec%252F%2523scaling.html "Directory-tag link-bibliography for link https://www.wired.com/1995/10/moravec/#scaling"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 54. `1993-olazaran.pdf`: [`“A Sociological Study of the Official History of the Perceptrons Controversy [1993]”`{=html}](/doc/ai/nn/1993-olazaran.pdf "'A Sociological Study of the Official History of the Perceptrons Controversy [1993]', Olazaran 1993"){#olazaran-1993 .link-annotated}, [[Mikel Olazaran](https://scholar.google.com/citations?user=XoUNbRoAAAAJ "Mikel Olazaran"){.id-not}]{.author .cite-author} [[link-bibliography](/metadata/annotation/link-bibliography/%252Fdoc%252Fai%252Fnn%252F1993-olazaran.pdf.html "Directory-tag link-bibliography for link /doc/ai/nn/1993-olazaran.pdf"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 55. `1987-sejnowski.pdf`: [`“Computing With Connections”`{=html}](/doc/ai/scaling/hardware/1987-sejnowski.pdf "'Computing with Connections', Sejnowski 1987"){#sejnowski-1987 .link-annotated}, [[Terrence J. Sejnowski](https://en.wikipedia.org/wiki/Terrence_J._Sejnowski "Terrence J. Sejnowski"){.id-not}]{.author .cite-author} [[link-bibliography](/metadata/annotation/link-bibliography/%252Fdoc%252Fai%252Fscaling%252Fhardware%252F1987-sejnowski.pdf.html "Directory-tag link-bibliography for link /doc/ai/scaling/hardware/1987-sejnowski.pdf"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 56. `https://web.archive.org/web/20230710000944/https://frc.ri.cmu.edu/~hpm/project.archive/general.articles/1975/Raw.Power.html`: [`“The Role Of RAW POWER In INTELLIGENCE”`{=html}](https://web.archive.org/web/20230710000944/https://frc.ri.cmu.edu/~hpm/project.archive/general.articles/1975/Raw.Power.html "'The Role Of RAW POWER In INTELLIGENCE', Moravec 1976"){#moravec-1976 .link-annotated}, [[Hans Moravec](https://en.wikipedia.org/wiki/Hans_Moravec "Hans Moravec"){.link-annotated-partial .id-not}]{.author .cite-author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Fweb.archive.org%252Fweb%252F20230710000944%252Fhttps%253A%252F%252Ffrc.ri.cmu.edu%252F~hpm%252Fproject.archive%252Fgeneral.articles%252F1975%252FRaw.Power.html.html "Directory-tag link-bibliography for link https://web.archive.org/web/20230710000944/https://frc.ri.cmu.edu/~hpm/project.archive/general.articles/1975/Raw.Power.html"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 57. `https://sites.research.google/trc/`: [`“TensorFlow Research Cloud (TRC): Accelerate Your Cutting-Edge Machine Learning Research With Free Cloud TPUs”`{=html}](https://sites.research.google/trc/ "'TensorFlow Research Cloud (TRC): Accelerate your cutting-edge machine learning research with free Cloud TPUs', TRC 2026"){#_dpuECoS8 .link-annotated}, [TRC]{.author .cite-author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Fsites.research.google%252Ftrc%252F.html "Directory-tag link-bibliography for link https://sites.research.google/trc/"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block}