Bibliography (16):

1. Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, the World’s Largest and Most Powerful Generative Language Model

2. Microsoft announces new supercomputer, lays out vision for future AI work

3. ZeRO-2 & DeepSpeed: Shattering barriers of deep learning speed & scale

4. Naver unveils first ‘hyperscale’ AI platform

5. Chinese AI lab challenges Google, OpenAI with a model of 1.75 trillion parameters

6. Wikipedia Bibliography:

   1. Volta (microarchitecture) § Products  :

      https://en.wikipedia.org/wiki/Volta_(microarchitecture)#Products

   2. https://en.wikipedia.org/wiki/Ampere_(microarchitecture)#A100_accelerator_and_DGX_A100  :

      https://en.wikipedia.org/wiki/Ampere_(microarchitecture)#A100_accelerator_and_DGX_A100

   3. Graphics processing unit

   4. InfiniBand

   5. https://en.wikipedia.org/wiki/Clos_network  :

      https://en.wikipedia.org/wiki/Clos_network

   6. https://en.wikipedia.org/wiki/Penguin_Computing  :

      https://en.wikipedia.org/wiki/Penguin_Computing

   7. AMD

   8. https://en.wikipedia.org/wiki/Epyc  :

      https://en.wikipedia.org/wiki/Epyc

   9. https://en.wikipedia.org/wiki/Perlmutter_(supercomputer)  :

      https://en.wikipedia.org/wiki/Perlmutter_(supercomputer)

   10. https://en.wikipedia.org/wiki/Exascale_computing  :

       https://en.wikipedia.org/wiki/Exascale_computing

   11. Summit (supercomputer)