Meta: Creating the Llama 3 model requires the use of 24,500 NVIDIA H100 GPUs

Meta: Creating the Llama 3 model requires the use of 24,500 NVIDIA H100 GPUs

We knew that Meta engineers had been working on themodel training generative Llama 3. Designed to overcome the limitations of current artificial intelligence solutions, the new Llama 3 (which should be publicly released in July 2024) should allow the development of complex “cognitive abilities”, without forgetting that we are still dealing with stochastic models.

Differently from closed and proprietary models such as OpenAI GPT-4 and Google Gemini, Meta Llama 3 will be open source and will offer numerous customization possibilities to the benefit of developers.

To train Llama 3, Meta uses clusters of more than 24,500 NVIDIA H100 GPUs

In a recently published in-depth study, Meta engineers announced a major investment in the future of artificial intelligencewith the presentation of two clusters composed of 24,576 GPU NVIDIA H100. In the post, the company founded by Mark Zuckerberg reveals some details about the hardware, network, storage, design, performance and software used to achieve high reliability and throughput important for the most demanding workloads, such as the Llama training 3.

While an important milestone, the launch of Llama 3 is not an end point. Meta says that by the end of 2024 the company aims to continue expanding its infrastructure, which will include 350.000 GPU NVIDIA H100 as part of a portfolio that will offer computing power equivalent to nearly 600,000 H100s.

Creating Llama required the use of over 24,500 NVIDIA H100 GPUs

Towards the development of an open and responsible AGI

The Meta managers don’t beat around the bush: all this is enormous computing power aims to develop, in the medium to long term, AGI, general artificial intelligence.

And’AGIo general artificial intelligence, aims to replicate human general intelligence, capable of understanding, learning and solving a wide range of different tasks in a similar way to a human. Unlike specialized AIs, which are designed to perform specific tasks or solve narrow problems, an AGI should be able to adapt and learn new tasks without significant reconfiguration. The goal is to ensure that the machine can tackle a wide range of tasks and problems with it flexibility e creativity of a human being.

And if Meta is preparing to work on the development of an AGI, the company is trying to scale its clusters to reach the ambitious goal. The journey towards AGI leads to the creation of new products, new features and innovative computing devices centered around AI.

For Meta, its AGI must necessarily be open and built in a way responsible, so as to be widely available and for the benefit of all. An aspect that Elon Musk, for example, has repeatedly criticized OpenAIa reality which according to his point of view would have reneged on the initial agreements.

AI Research Supercluster (RSC)

Meta has a long history of developinginfrastructure for artificial intelligence applications. In 2022, the SuperCluster AI (RSC) made up of 16,000 NVIDIA A100 GPUs has helped accelerate open and responsible AI research, helping Meta develop the first generation of advanced AI models.

That “seed” sown at the time played an important role in the development of Llama and Llama 2, as well as advanced AI models for computer visionNLP (natural language processing) and speech recognition, image generation and data coding.

Meta’s new AI clusters build on the successes and lessons learned from the RSC SuperCluster. The objective is in fact to push on the construction of sistemi IA end-to-end, with a particular emphasis on the experience and productivity of researchers and developers. The efficiency of high-performance networks within these clusters, some key storage choices, the use of more than 24,500 NVIDIA Tensor Core H100 GPUs in each cluster, enable support for larger and more complex models and open the doors to the development of innovative solutions GenAI.

Opening image credit: – BlackJack3D

Leave a Reply

Your email address will not be published. Required fields are marked *