“Chinese AI Lab Challenges Google, OpenAI With a Model of 1.75 Trillion Parameters”, 2021-06-01 (; similar):
[WP] The Beijing Academy of Artificial Intelligence, styled as BAAI and known in Chinese as 北京智源人工智能研究院, launched the latest version of Wu Dao 悟道, a pre-trained deep learning model that the lab dubbed as “China’s first”, and “the world’s largest ever”, with a whopping 1.75 trillion parameters.
…Unlike conventional deep learning models that are usually task-specific, Wu Dao is a multi-modal model trained to tackle both text and image, 2 dramatically different sets of problems. At BAAI’s annual academic conference on Tuesday, the institution demonstrated Wu Dao performing tasks such as natural language processing, text generation, image recognition, image generation, etc.
The model is capable of writing poems and couplets in the traditional Chinese styles, answer questions, write essays, generate alt text for images, and generate corresponding images from natural language description with a decent level of photorealism. It is even able to power “virtual idols”, with the help of Xiaoice, a Chinese company spun off of Microsoft—so there can be voice support too, in addition to text and image.
…Very interestingly, this model with 1.75 trillion parameters is already the 2.0 version of Wu Dao, whose first version was just launched less than 3 months ago. One of the main reasons the Chinese researchers made progress quickly was that they were able to tap into China’s supercomputing clusters, with the help of a few of its core members who also worked on the national supercomputing projects.
A little more technical explanation: BAAI researchers developed and open-sourced a deep learning system called FastMoE, which allowed Wu Dao to be trained on both supercomputers and regular GPUs with substantially more parameters, giving the model, in theory, more flexibility than Google’s take on the MoE, or Mixture-of-Experts. This is because Google’s system requires the company’s dedicated TPU hardware and distributed training framework, while BAAI’s FastMoE works with at least one industry-standard open-source framework, namely PyTorch, and can be operated on off-the-shelf hardware.
The Chinese lab claims that Wu Dao’s sub-models achieved better performance than previous models, beating OpenAI’s CLIP and Google’s ALIGN on English image and text indexing in the Microsoft COCO dataset. For image generation from text, a novel task, BAAI claims that Wu Dao’s sub-model CogView beat OpenAI’s DALL·E 1, a state-of-the-art neural network launched in January this year with 12 billion parameters.
“The way to artificial general intelligence is big models and big computer”, said Dr. Zhang Hongjiang, chairman of BAAI, “What we are building is a power plant for the future of AI, with mega data, mega computing power, and mega models, we can transform data to fuel the AI applications of the future.”
…However, while OpenAI and DeepMind are privately funded, a key distinction for BAAI is that it’s formed and funded with substantial help from China’s Ministry of Science and Technology, as well as Beijing’s municipal government.