
Large language models and the applications they power enable unprecedented opportunities for organizations to get deeper insights from their data reservoirs and to build entirely new classes of applications.
But with opportunities often come challenges.
Both on premises and in the cloud, applications that are expected to run in real time place significant demands on data center infrastructure to simultaneously deliver high throughput and low latency with one platform investment.
To drive continuous performance improvements and improve the return on infrastructure investments, NVIDIA regularly optimizes the state-of-the-art community models, including Meta's Llama, Google's Gemma, Microsoft's Phi and our own NVLM-D-72B, released just a few weeks ago.
Relentless Improvements Performance improvements let our customers and partners serve more complex models and reduce the needed infrastructure to host them. NVIDIA optimizes performance at every layer of the technology stack, including TensorRT-LLM, a purpose-built library to deliver state-of-the-art performance on the latest LLMs. With improvements to the open-source Llama 70B model, which delivers very high accuracy, we've already improved minimum latency performance by 3.5x in less than a year.
We're constantly improving our platform performance and regularly publish performance updates. Each week, improvements to NVIDIA software libraries are published, allowing customers to get more from the very same GPUs. For example, in just a few months' time, we've improved our low-latency Llama 70B performance by 3.5x.
NVIDIA has increased performance on the Llama 70B model by 3.5x. In the most recent round of MLPerf Inference 4.1, we made our first-ever submission with the Blackwell platform. It delivered 4x more performance than the previous generation.
This submission was also the first-ever MLPerf submission to use FP4 precision. Narrower precision formats, like FP4, reduces memory footprint and memory traffic, and also boost computational throughput. The process takes advantage of Blackwell's second-generation Transformer Engine, and with advanced quantization techniques that are part of TensorRT Model Optimizer, the Blackwell submission met the strict accuracy targets of the MLPerf benchmark.
Blackwell B200 delivers up to 4x more performance versus previous generation on MLPerf Inference v4.1's Llama 2 70B workload. Improvements in Blackwell haven't stopped the continued acceleration of Hopper. In the last year, Hopper performance has increased 3.4x in MLPerf on H100 thanks to regular software advancements. This means that NVIDIA's peak performance today, on Blackwell, is 10x faster than it was just one year ago on Hopper.
These results track progress on the MLPerf Inference Llama 2 70B Offline scenario over the past year. Our ongoing work is incorporated into TensorRT-LLM, a purpose-built library to accelerate LLMs that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM is built on top of the TensorRT Deep Learning Inference library and leverages much of TensorRT's deep learning optimizations with additional LLM-specific improvements.
Improving Llama in Leaps and Bounds More recently, we've continued optimizing variants of Meta's Llama models, including versions 3.1 and 3.2 as well as model sizes 70B and the biggest model, 405B. These optimizations include custom quantization recipes, as well as efficient use of parallelization techniques to more efficiently split the model across multiple GPUs, leveraging NVIDIA NVLink and NVSwitch interconnect technologies. Cutting-edge LLMs like Llama 3.1 405B are very demanding and require the combined performance of multiple state-of-the-art GPUs for fast responses.
Parallelism techniques require a hardware platform with a robust GPU-to-GPU interconnect fabric to get maximum performance and avoid communication bottlenecks. Each NVIDIA H200 Tensor Core GPU features fourth-generation NVLink, which provides a whopping 900GB/s of GPU-to-GPU bandwidth. Every eight-GPU HGX H200 platform also ships with four NVLink Switches, enabling every H200 GPU to communicate with any other H200 GPU at 900GB/s, simultaneously.
Many LLM deployments use parallelism over choosing to keep the workload on a single GPU, which can have compute bottlenecks. LLMs seek to balance low latency and high throughput, with the optimal parallelization technique depending on application requirements.
For instance, if lowest latency is the priority, tensor parallelism is critical, as the combined compute performance of multiple GPUs can be used to serve tokens to users more quickly. However, for use cases where peak throughput across all users is prioritized, pipeline parallelism can efficiently boost overall server throughput.
The table below shows that tensor parallelism can deliver over 5x more throughput in minimum latency scenarios, whereas pipeline parallelism brings 50% more performance for maximum throughput use cases.
For production deployments that seek to maximize throughput within a given latency budget, a platform needs to provide the ability to effectively combine both techniques like in TensorRT-LLM.
Read the technical blog on boosting Llama 3.1 405B throughput to learn more about these techniques.
Different scenarios have different requirements, and parallelism techniques bring optimal performance for each of these scenarios. The Virtuous Cycle Over the lifecycle of our architectures, we deliver significant performance gains from ongoing software tuning and optimization. These improvements translate into additional value for customers who train and deploy on our platforms. They're able to create more capable models and applications and deploy their existing models using less infrastructure, enhancing th
Most recent headlines
04/09/2025
Monumental Sports & Entertainment (MSE), in collaboration with Dalet, has been a...
19/04/2025
SDVI, the leading platform provider for cloud-native media supply chains, today announced that the company earned multiple awards at the 2025 NAB Show, with two...
19/04/2025
Ateliere Creative Technologies, a leading GenAI media software solutions company, today announced that Dan Goman has stepped down as CEO and David Bortis, Ateli...
19/04/2025
As Director of Media and Aerial Production at Terrible Herbst Motorsports, Bryan Moore is setting new standards in off-road racing media coverage thanks to his ...
19/04/2025
A next-generation collaboration device that redefines connectivity for meeting environments
Lightware, an industry-leading manufacturer of signal management so...
19/04/2025
Calrec is today announcing that its True Control 2.0 is a Remote Production winner in the 2025 NAB Show Product of the Year Awards. This official awards program...
19/04/2025
Appear, a global leader in live production technology, proudly announces it has been recognised alongside NBCUniversal with the prestigious NAB Show Delivery Pr...
19/04/2025
Deity Microphones, a leader in innovative audio equipment, is proud to announce the expected release of our Ultra-Wide Band IFB to the market. The THEOS DIFB wi...
19/04/2025
A world renowned broadcaster and long-standing LiveU customer has successfully completed a series of live connectivity tests using LiveU's revolutionary, aw...
19/04/2025
BitFire (bitfire.tv), a longtime leader in live video transport, today announced dual NAB Show award wins at the 2025 NAB Show in Las Vegas. The company's M...
19/04/2025
BitFire (bitfire.tv), a longtime leader in live video transport, today announced three major award wins at the 2025 NAB Show, April 5-9, in Las Vegas. The compa...
19/04/2025
AI video discovery company Moments Lab and Satisfaction Group, a leading independent unscripted television production company, are proud to announce a unique st...
19/04/2025
As the media industry navigates the triple challenge of AI-driven production, distributed teams, and skyrocketing content demand, DigitalGlue s creative.space h...
19/04/2025
Network technology startup Miri Technologies Inc. capped off its tremendously successful NAB Show debut by winning two prestigious industry awards for its cutti...
19/04/2025
CINCINNATI Scripp's Nuvyyo USA has concluded a deal with Warner Bros. Discovery to bring 45 FAST channels to Nuvyyo's Tablo TV device....
19/04/2025
In a ruling that could have broader implications on the legality of regulatory agencies levying fines through administrative proceedings, the 5th U.S. Circuit C...
19/04/2025
WASHINGTON Federal Communications Commission chair Brendan Carr has blasted Comcast over MSNBC's coverage of the deportation of Kilmar Abrego Garcia in a so...
19/04/2025
Berklee NYC and NYC Media Launch Season 3 of Inside Power Station @BerkleeNYC This season features faculty member Arun Pandian as the new host and interviews ...
18/04/2025
Director Andrew Ahn, alongside actors Youn Yuh-jung and Joan Chen, takes a photo of the audience after the premiere of his film The Wedding Banquet at Eccles ...
18/04/2025
In a ruling that could have a major impact on the digital advertising market, a federal judge has ruled that Google has monopolized some types of advertising te...
18/04/2025
Broadcast and cable TV news outlets saw strong social media growth in March, according to new data from the social video analytics company Tubular Labs ....
18/04/2025
Berklee Student Yukai Yang Named 2025 Yamaha Young Performing Artist The drummer secured a spot among the elite winners in this years competition.
By
Maddie...
18/04/2025
Boston Conservatory Alums Bring Real Women Have Curves to Broadway The Latin American immigrant community takes center stage in a new musical featuring Tatian...
18/04/2025
WASHINGTON The FCC's call for public comments and suggestions on outdated regulations that it should be eliminated, has prompted a slew of fillings from bro...
18/04/2025
In a ruling that could have a major impact on the digital advertising market, a federal judge has ruled that Google has monopolized some types of advertising te...
18/04/2025
PEARL RIVER, N.Y. Global media solutions company Active Media Services (AMS) has formed a new relationship with VideoAmp, a measurement company for linear TV, c...
18/04/2025
Netflix reported generally positive results for first-quarter 2025, with revenue up 13% year-over-year to $10.543 billion and operating income growing by 27% to...
18/04/2025
NHL Playoffs 2025: TNT Sports Hits the Road for Onsite Productions With Mobile U...
18/04/2025
EVSs S bastien Verlaine on U.S. Expansion, Next-Generation Products Beyond replay, offerings also target asset management and media infrastructure By Ken Kersc...
18/04/2025
ESPN Unleashes 4DREPLAY as NCAA Women's Gymnastics Championships Hit ABC Men's championships to follow Saturday night on ESPN2 By Brandon Costa, Direct...
18/04/2025
Visualizing Victory: The Latest in AR, XR, and Virtual Production in Live Sports This panel discussion featured leaders from ESPN, CBS Sports, Warner Bros. Disc...
18/04/2025
NHL Playoffs 2025: With 16 Games in First Six Days, ESPN Deploys Variety of Remo...
17/04/2025
Emilie Blichfeldt attends the 2025 Sundance Film Festival premiere of The Ugly ...
17/04/2025
R-GPS gives warfighters a decisive battlefield advantage by punching through adv...
17/04/2025
This year's NAB Show in Las Vegas marked a noticeable shift in the priorities of media and broadcast organisations. Gone are the days of chasing flashy, or ...
17/04/2025
class=attachment-thumbnail size-thumbnail f-align-center alt= decoding=async data-lazy-srcset=https://www.antonbauer.com/wp-content/uploads/2024/12/Amy-Daniel-1...
17/04/2025
SAN JOSE, Calif. Roku and Adobe have announced that they are collaborating on a real time data platform made possible by a a new integration of the Roku Data C...
17/04/2025
NEW YORK Internet advertising revenues demonstrated strong growth in 2024, increasing 14.9% year-over-year to $258.6 billion, according to the IAB Internet Adv...
17/04/2025
SDVI Earns Both Product and Project of the Year Awards at 2025 NAB Show
Brie Clayton April 17, 2025
0 Comments
Left to right, Geoff Stedman, CMO, SDVI...
17/04/2025
Singapore Polytechnic Readies Aspiring AV Professionals for Live IP Productions ...
17/04/2025
Calrec Wins 2025 NAB Show Product of the Year Award for True Control 2.0
Brie Clayton April 17, 2025
0 Comments
Image: The Calrec True Control 2.o on ...
17/04/2025
In Return to Berklee, Lucius Looks Back and Moves Forward From mood boards to live demos, the alumni band gave students an exclusive look at the process behin...
17/04/2025
DirecTV's free streaming service MyFree DirecTV has just added another eight channels from NBCUniversal....
17/04/2025
LOS ANGELES The virtual production company GameChanger has announced that it is expanding its global footprint by bringing its virtual production technology to ...
17/04/2025
DENVER The International Broadcaster Coalition Against Piracy (IBCAP) has announced that it has developed a proprietary, automated software-based system to iden...
17/04/2025
Pixalate's new CTV Device Market Share report for Q1 2025 shows that Roku has the highest open programmatic CTV device market share in the United States, wi...
17/04/2025
Edward J. Lewis III Named Senior Vice President of Institutional Advancement Lewis has more than 20 years of industry experience, leading fundraising initiati...
17/04/2025
The Curling Group Puts On Inaugural Curling All-Star Game in Nashville The location in Music City is intended to broaden the sport's appeal By Dan Daley, ...
17/04/2025
April 17th, 2025 Press Materials Available Here
Tribeca Festival 2025 Announces TV & NOW Lineup
World Premieres and Exclusive Cast Panels with Apple TV '...
17/04/2025
SVG Sit-Down: Cisco's Bryan Bedford on Providing End-to-End Support for Clie...