Sony Pixel Power calrec Sony

Striking Performance: Large Language Models up to 4x Faster on RTX With TensorRT-LLM for Windows

17/10/2023

Generative AI is one of the most important trends in the history of personal computing, bringing advancements to gaming, creativity, video, productivity, development and more.

And GeForce RTX and NVIDIA RTX GPUs, which are packed with dedicated AI processors called Tensor Cores, are bringing the power of generative AI natively to more than 100 million Windows PCs and workstations.

Today, generative AI on PC is getting up to 4x faster via TensorRT-LLM for Windows, an open-source library that accelerates inference performance for the latest AI large language models, like Llama 2 and Code Llama. This follows the announcement of TensorRT-LLM for data centers last month.

NVIDIA has also released tools to help developers accelerate their LLMs, including scripts that optimize custom models with TensorRT-LLM, TensorRT-optimized open-source models and a developer reference project that showcases both the speed and quality of LLM responses.

TensorRT acceleration is now available for Stable Diffusion in the popular Web UI by Automatic1111 distribution. It speeds up the generative AI diffusion model by up to 2x over the previous fastest implementation.

Plus, RTX Video Super Resolution (VSR) version 1.5 is available as part of today's Game Ready Driver release - and will be available in the next NVIDIA Studio Driver, releasing early next month.

Supercharging LLMs With TensorRT LLMs are fueling productivity - engaging in chat, summarizing documents and web content, drafting emails and blogs - and are at the core of new pipelines of AI and other software that can automatically analyze data and generate a vast array of content.

TensorRT-LLM, a library for accelerating LLM inference, gives developers and end users the benefit of LLMs that can now operate up to 4x faster on RTX-powered Windows PCs.

At higher batch sizes, this acceleration significantly improves the experience for more sophisticated LLM use - like writing and coding assistants that output multiple, unique auto-complete results at once. The result is accelerated performance and improved quality that lets users select the best of the bunch.

TensorRT-LLM acceleration is also beneficial when integrating LLM capabilities with other technology, such as in retrieval-augmented generation (RAG), where an LLM is paired with a vector library or vector database. RAG enables the LLM to deliver responses based on a specific dataset, like user emails or articles on a website, to provide more targeted answers.

To show this in practical terms, when the question How does NVIDIA ACE generate emotional responses? was asked of the LLaMa 2 base model, it returned an unhelpful response.

Better responses, faster. Conversely, using RAG with recent GeForce news articles loaded into a vector library and connected to the same Llama 2 model not only returned the correct answer - using NeMo SteerLM - but did so much quicker with TensorRT-LLM acceleration. This combination of speed and proficiency gives users smarter solutions.

TensorRT-LLM will soon be available to download from the NVIDIA Developer website. TensorRT-optimized open source models and the RAG demo with GeForce news as a sample project are available at ngc.nvidia.com and GitHub.com/NVIDIA.

Automatic Acceleration Diffusion models, like Stable Diffusion, are used to imagine and create stunning, novel works of art. Image generation is an iterative process that can take hundreds of cycles to achieve the perfect output. When done on an underpowered computer, this iteration can add up to hours of wait time.

TensorRT is designed to accelerate AI models through layer fusion, precision calibration, kernel auto-tuning and other capabilities that significantly boost inference efficiency and speed. This makes it indispensable for real-time applications and resource-intensive tasks.

And now, TensorRT doubles the speed of Stable Diffusion.

Compatible with the most popular distribution, WebUI from Automatic1111, Stable Diffusion with TensorRT acceleration helps users iterate faster and spend less time waiting on the computer, delivering a final image sooner. On a GeForce RTX 4090, it runs 7x faster than the top implementation on Macs with an Apple M2 Ultra. The extension is available for download today.

The TensorRT demo of a Stable Diffusion pipeline provides developers with a reference implementation on how to prepare diffusion models and accelerate them using TensorRT. This is the starting point for developers interested in turbocharging a diffusion pipeline and bringing lightning-fast inferencing to applications.

Video That's Super AI is improving everyday PC experiences for all users. Streaming video - from nearly any source, like YouTube, Twitch, Prime Video, Disney+ and countless others - is among the most popular activities on a PC. Thanks to AI and RTX, it's getting another update in image quality.

RTX VSR is a breakthrough in AI pixel processing that improves the quality of streamed video content by reducing or eliminating artifacts caused by video compression. It also sharpens edges and details.

Available now, RTX VSR version 1.5 further improves visual quality with updated models, de-artifacts content played in its native resolution and adds support for RTX GPUs based on the NVIDIA Turing architecture - both professional RTX and GeForce RTX 20 Series GPUs.

Retraining the VSR AI model helped it learn to accurately identify the difference between subtle details and compression artifacts. As a result, AI-enhanced images more accurately preserve details during the upscaling process. Finer details are more visible, and the overall image looks sharper and crisper.

RTX Video Super Resolution v1.5 improves detail and sharpness. New with version 1.5 is the ability to de-artifact video played at the display's native resolution. The original release only enhanced video when it was
LINK: https://blogs.nvidia.com/blog/2023/10/17/tensorrt-llm-windows-stable-d...
See more stories from nvidia

Most recent headlines

17/02/2025

How BBC Wales returned live to 1985

TVBEurope talks to the team who brought a piece of broadcasting history back to life on BBC Wales last night By Jenny Priestley Published: February 14, 2025 ...

17/02/2025

Film studio tax relief programme gets underway in England

Eligible studios will receive the relief directly from their local authorities, with no need to submit a claim By Matthew Corrigan Published: February 17, 20...

17/02/2025

Empowering progress from within

As DEI initiatives come under pressure in the USA, supporters of Rise, which promotes diversity in the media and entertainment sector, discuss the continuing im...

17/02/2025

EIB Global Lends 125 Million To Finance SES's Medium Earth Orbit Satellites

EIB Global loan will partially fund SES's continued O3b mPOWER medium earth orbit (MEO) fleet expansion enabling additional capacity worldwide. O3b mPOWER s...

15/02/2025

Nielsen's Business Resource Groups celebrate inclusion with BRinG It! Campaign

At Nielsen, Business Resource Groups (BRGs) play a critical role in nurturing a ...

15/02/2025

Innovation to empower creativity: Explore the latest in Lightroom

Innovation to empower creativity: Explore the latest in Lightroom Rob Christensen February 15, 2025 0 Comments Whether you want to remove distracting ...

15/02/2025

Wisycom Expands Into Argentina and Uruguay With EQUAPHON

Wisycom, a worldwide leader in RF technology and provider of wireless solutions, announces a new distribution agreement with EQUAPHON. An Argentinian profession...

15/02/2025

DPA Microphones Adds to Sales and Marketing Teams

DPA Microphones, the leading manufacturer of high-quality microphone solutions, is pleased to welcome Nicholas Nick Mariano and Vincent Vince Divine to the ...

15/02/2025

Tuxera brings media-optimized SMB technology to NAB Show...

Tuxera, a leading provider of quality-assured file systems and networking solutions, will showcase how Tuxera Fusion SMB enables unprecedented performance for m...

15/02/2025

Bitmovin Report Ad Insertion Tops Streaming Challenges a...

Bitmovin's latest Video Developer Report indicates an increasing role of AI, the slow adoption of next-generation codecs, and continuing challenges when it ...

15/02/2025

Oberc Lights Michelin-Star Hotel Guide Promo with DoPchoi...

When Michelin Enterprises set out to promote its prestigious Hotel Guide in the new Sleepless in the Kitchen , campaign, veteran German/Slovenian gaffer Robert...

15/02/2025

Formula E Supercharges Fan Experience with AI-Driven High...

Formula E, the world's first all-electric racing series, revolutionizing motorsport with cutting-edge technology and a commitment to sustainability has sele...

15/02/2025

Agile Content is Now AgileTV

Agile Content, a global leader in TV and video technology solutions, announces its rebranding to AgileTV. The transition to AgileTV stems from a commitment to ...

15/02/2025

Hiltron to Promote Latest-Generation Satellite Communicat...

Hiltron Communications will promote the latest additions to its range of satellite communication products, systems and supporting services at GovSatCom 2025. Th...

15/02/2025

DoPchoice Debuts Airglow Float - Inflatable Round Topligh...

Following the success of the original Airglow Booklight, DoPchoice debuts a new addition to their inflatable line the Float. This lightweight, inflatable, round...

15/02/2025

Marshall to Debut RCP Plus Controller at 2025 NAB Show

LAS VEGAS Marshall Electronics has announced that it will debut its newest camera controller, the RCP Plus, at the 2025 NAB Show at Booth N2649 between April 5 ...

15/02/2025

Gray Media Names Blake Sebo GM of Stations in Anchorage, Alaska

ATLANTA Gray Media has named Blake Sebo as the next general manager of KTUU and KYES, Gray's NBC and CBS affiliates in Anchorage, Alaska....

15/02/2025

DPA Microphones To Introduce CORE+ Technology at NAB Show

KOKKEDAL, Denmark DPA Microphones will feature its new CORE+ technology for distortion-free microphone sound at the 2025 NAB Show, April 5-9, in Las Vegas....

15/02/2025

QuickLink To Unveil StudioPro-NDI, StudioPro-3 at 2025 NAB Show

SWANSEA, U.K. QuickLink will launch StudioPro-NDI and StudioPro-3, two new models of its QuickLine StudioPro video production platform, at the 2025 NAB Show, Ap...

15/02/2025

Live From 2025 NBA All-Star Game Weekend: After 25 Years, Turner Sports All-In For Final Production

Live From 2025 NBA All-Star Game Weekend: After 25 Years, Turner Sports All-In F...

15/02/2025

NASCAR Driver Cam on Max Embraces Cloud-Based Workflow for 2025 Season

NASCAR Driver Cam on Max Embraces Cloud-Based Workflow for 2025 Season Chris Brown of WBD Sports says Mobii will play key part in delivering signals from the tr...

14/02/2025

Spotify's Partner Program Helps Creators Increase Revenue and Consumption of Video Podcasts

Last month, we officially launched the Spotify Partner Program, a powerful new m...

14/02/2025

Four Ways L3Harris Targeting and Sensor Systems are Defining the Future of Land Missions

The future of electro-optical/infrared (EO/IR) land systems is evolving so that ...

14/02/2025

ISE 2025: 5 Reasons to Join Calrec at ISE

Here are 5 reasons why you should attend ISE 2025 Discover more about Calrec: Learn more about Calrec's legacy of innovation in broadcast audio solutions ...

14/02/2025

World Radio Day: On the same wavelength

World Radio Day: On the same wavelength By Graham Murray, International Sales Manager at Calrec To mark World Radio Day, Calrec's International Sales Ma...

14/02/2025

Tegna Inks Deal for Dallas Wings Broadcasts

DALLAS and TYSONS, Va. Tegna has concluded a broadcast rights agreement that makes Tegna's KFAA in Dallas the exclusive local television partner of the Dall...

14/02/2025

Prime Video Makes Significant Upgrades to Its Apple TV App

Amazon's Prime Video has released a new app for Apple TV that the streamer said delivers an improved streaming experiences with faster scrolling, improved s...

14/02/2025

Telos Alliance To Showcase Quasar V3.2 AoIP Software at NAB Show

CLEVELAND Telos Alliance announced a software update for its Axia Quasar XR and SR line of audio-over-IP (AoIP) mixing consoles and will show the offering at th...

14/02/2025

Puget Systems Returns to HPA Tech Retreat 2025 with New Analysis Tools for Puget Bench for Creators

Puget Systems Returns to HPA Tech Retreat 2025 with New Analysis Tools for Puge...

14/02/2025

Glass Animals: Tour of Earth Uses Blackmagic Live Production Workflow

Glass Animals: Tour of Earth Uses Blackmagic Live Production Workflow Brie Clayton February 14, 2025 0 Comments 80six deploys Blackmagic URSA Broadcas...

14/02/2025

Paramount, Youtube TV Avoid Channel Blackout, Will Continue Negotiations

A dispute between Paramount and Youtube TV that would have pulled CBS stations and Paramount's cable channels from the streaming service has been temporaril...

14/02/2025

Stop-motion triumph

Colourist Deidre (Dee) McClelland details how she helped shape the distinct look and feel of whimsical Oscar-nominated stop-motion film, Memoir of a Snail By C...

14/02/2025

BitFire appoints Jim Akimchuk as CEO

BitFire said it intends to introduce forward-looking services that simplify the launch of live production workflows, and enhance real-time collaboration By Jen...

14/02/2025

Arena TV liquidators dismiss banks' defence as unsustainable'

The claim by the liquidators is the latest twist in the ongoing saga of the companys collapse at the end of 2021 By Matthew Corrigan Published: February 14, ...

14/02/2025

YouTube TV Likely To Drop Paramount Channels Today

A carriage dispute between YouTube TV and Paramount could come to a head today (Feb. 13), with the virtual pay TV provider threatening to pull all of the progra...

14/02/2025

LG Opens Its First Virtual Production Studio

ENGLEWOOD CLIFFS, N.J. LG Electronics USA has opened a new in-house, state-of-the-art virtual production studio at its 360,000-square-foot North American headqu...

14/02/2025

CIMM, TVB Release Local TV and Video Innovation Showcase

NEW YORK In a bid to help local broadcasters find new ways of monetizing their content, the Coalition for Innovative Media Measurement (CIMM) and TVB have relea...

14/02/2025

U.S. Senators Blast Weaponization of FCC' Against Broadcasters

WASHINGTON Three Democratic U.S. Senators have sent a stern letter blasting FCC Chair Brendan Carr for the weaponization of the agency for attacking broadcast...

14/02/2025

The Apple TV App Comes to Android

Years after launching the streaming service Apple TV+ in November of 2019, Apple has finally released a version of its steaming app for Android devices....

14/02/2025

Red Adds Nikon Z Mount To V-RAPTOR [X], KOMODO-X Camera Systems

FOOTHILL RANCH, Calif. Red Digital Cinema will make its V-RAPTOR [X] and KOMODO-X camera systems available with Nikon's premier Z Mount, providing filmmaker...

14/02/2025

Fox Sports Revs Up for Daytona 500 With New NASCAR Deal, Some New Tech

DAYTONA, Fla. The Fox Sports production of this year's Daytona 500 will see the addition of a few new tech elements and the continued unfolding of the broad...

14/02/2025

Watch Bryson Battles Blind Audition on The Voice Get a Four-Chair Turn

Watch Bryson Battles Blind Audition on The Voice Get a Four-Chair Turn Michael Bubl said the Boston Conservatory at Berklee musical theater senior might have...

14/02/2025

2025 Daytona 500: FOX Sports Looks To Ride Super Bowl LIX Momentum Into NASCAR Season

2025 Daytona 500: FOX Sports Looks To Ride Super Bowl LIX Momentum Into NASCAR S...

14/02/2025

VEON's Kyivstar and the Come Back Alive Foundation Raise USD 2.6 Million for Demining in Ukraine

14 Feb 2025 VEON's Kyivstar and the Come Back Alive Foundation Raise USD 2....

14/02/2025

Max Renews THE PITT, Starring Noah Wyle, For A Second Season

Max Renews THE PITT, Starring Noah Wyle, For A Second Season Max Warner Bros. Television Group The Pitt New Episodes Of The Medical Drama From John...

14/02/2025

110 Years of Love in Every Frame: A Technicolor Celebration of Romance

This year, Technicolor proudly celebrates 110 years of shaping the magic of cinema pioneering color, visual effects, and innovation that bring unforgettable l...

14/02/2025

NBA All-Star: More Games, More Teams, More Music, More Sound From the Court

NBA All-Star: More Games, More Teams, More Music, More Sound From the Court Turner Sports' team will deploy familiar tech to capture all the audio By Dan D...

14/02/2025

LIV Golf Continues To Innovate With Drones, Data-Driven Graphics, and All-Access Features in Year 4

LIV Golf Continues To Innovate With Drones, Data-Driven Graphics, and All-Access...