Editor's note: This post is part of the AI Decoded series, which demystifies AI by making the technology more accessible, and showcases new hardware, software, tools and accelerations for RTX PC users.
Large language models are driving some of the most exciting developments in AI with their ability to quickly understand, summarize and generate text-based content.
These capabilities power a variety of use cases, including productivity tools, digital assistants, non-playable characters in video games and more. But they're not a one-size-fits-all solution, and developers often must fine-tune LLMs to fit the needs of their applications.
The NVIDIA RTX AI Toolkit makes it easy to fine-tune and deploy AI models on RTX AI PCs and workstations through a technique called low-rank adaptation, or LoRA. A new update, available today, enables support for using multiple LoRA adapters simultaneously within the NVIDIA TensorRT-LLM AI acceleration library, improving the performance of fine-tuned models by up to 6x.
Fine-Tuned for Performance LLMs must be carefully customized to achieve higher performance and meet growing user demands.
These foundational models are trained on huge amounts of data but often lack the context needed for a developer's specific use case. For example, a generic LLM can generate video game dialogue, but it will likely miss the nuance and subtlety needed to write in the style of a woodland elf with a dark past and a barely concealed disdain for authority.
To achieve more tailored outputs, developers can fine-tune the model with information related to the app's use case.
Take the example of developing an app to generate in-game dialogue using an LLM. The process of fine-tuning starts with using the weights of a pretrained model, such as information on what a character may say in the game. To get the dialogue in the right style, a developer can tune the model on a smaller dataset of examples, such as dialogue written in a more spooky or villainous tone.
In some cases, developers may want to run all of these different fine-tuning processes simultaneously. For example, they may want to generate marketing copy written in different voices for various content channels. At the same time, they may want to summarize a document and make stylistic suggestions - as well as draft a video game scene description and imagery prompt for a text-to-image generator.
It's not practical to run multiple models simultaneously, as they won't all fit in GPU memory at the same time. Even if they did, their inference time would be impacted by memory bandwidth - how fast data can be read from memory into GPUs.
Lo(RA) and Behold A popular way to address these issues is to use fine-tuning techniques such as low-rank adaptation. A simple way of thinking of it is as a patch file containing the customizations from the fine-tuning process.
Once trained, customized LoRA adapters can integrate seamlessly with the foundation model during inference, adding minimal overhead. Developers can attach the adapters to a single model to serve multiple use cases. This keeps the memory footprint low while still providing the additional details needed for each specific use case.
Architecture overview of supporting multiple clients and use-cases with a single foundation model using multi-LoRA capabilities In practice, this means that an app can keep just one copy of the base model in memory, alongside many customizations using multiple LoRA adapters.
This process is called multi-LoRA serving. When multiple calls are made to the model, the GPU can process all of the calls in parallel, maximizing the use of its Tensor Cores and minimizing the demands of memory and bandwidth so developers can efficiently use AI models in their workflows. Fine-tuned models using multi-LoRA adapters perform up to 6x faster.
LLM inference performance on GeForce RTX 4090 Desktop GPU for Llama 3B int4 with LoRA adapters applied at runtime. Input sequence length is 43 tokens and output sequence length is 100 tokens. LoRA adapter max rank is 64. In the example of the in-game dialogue application described earlier, the app's scope could be expanded, using multi-LoRA serving, to generate both story elements and illustrations - driven by a single prompt.
The user could input a basic story idea, and the LLM would flesh out the concept, expanding on the idea to provide a detailed foundation. The application could then use the same model, enhanced with two distinct LoRA adapters, to refine the story and generate corresponding imagery. One LoRA adapter generates a Stable Diffusion prompt to create visuals using a locally deployed Stable Diffusion XL model. Meanwhile, the other LoRA adapter, fine-tuned for story writing, could craft a well-structured and engaging narrative.
In this case, the same model is used for both inference passes, ensuring that the space required for the process doesn't significantly increase. The second pass, which involves both text and image generation, is performed using batched inference, making the process exceptionally fast and efficient on NVIDIA GPUs. This allows users to rapidly iterate through different versions of their stories, refining the narrative and the illustrations with ease.
This process is outlined in more detail in a recent technical blog.
LLMs are becoming one of the most important components of modern AI. As adoption and integration grows, demand for powerful, fast LLMs with application-specific customizations will only increase. The multi-LoRA support added today to the RTX AI Toolkit gives developers a powerful new way to accelerate these capabilities.
Most recent headlines
09/12/2024
Dalet, a leading technology and service provider for media-rich organizations, today announced that it has been named an IDC Innovator in the IDC Innovators: ...
09/11/2024
Dalet, a leading technology and service provider for media-rich organizations, today announced three new members of its executive team. Tara Bryant joins as Chi...
13/09/2024
By Bailey Pennick
One of the most exciting things about the Sundance Film Festival is having a front-row seat for the bright future of independent filmmaking. ...
13/09/2024
SNS is excited to preview the latest features coming to the award-winning EVO Suite. Be among the first to test drive these workflow innovations at IBC2024, and...
13/09/2024
Spotify is committed to bringing the best listening experience to all our users, and that includes parents and families. That's why we've introduced a p...
13/09/2024
As fans patiently await their favorite artists' new album releases, the anti...
13/09/2024
What I've always loved about his films is that incredible warmth and that incredible imagination, cinematographer Haris Zambarloukos, BSC, GSC says of dire...
13/09/2024
SALT LAKE CITY, Sept. 12, 2024 - L3Harris Technologies (NYSE:LHX) has received a...
13/09/2024
Rendering of L3Harris Tranche 2 Tracking (T2TRK) imagery for the Space Developme...
13/09/2024
Last month, L3Harris delivered the third U.S. Air Force EA-37B Compass Call Cross Deck aircraft integrated with sophisticated electromagnetic attack (EA) miss...
13/09/2024
Targeted at low-to-mid tier productions, the new products set new standards for ...
13/09/2024
The next generation products will enhance the Densit Modular Platform with unma...
13/09/2024
Revolution in Storage: EditShare Reveals Game-Changing New Line Up at IBC 2024
More power, more speed, more flexibility, more capacity, more security, more co...
13/09/2024
AMSTERDAM Calrec Audio has unveiled its compact Argo M audio console, ImPulseV cloud audio mixing solution and True Control 2.0 at IBC 2024, Sept. 13-16....
13/09/2024
A study of how the media and broadcast industry feels about artificial intelligence (AI) has revealed most media and broadcast companies view its adoption as a ...
13/09/2024
AMSTERDAM AI-driven digital asset management system (DAM) Starchive announced at IBC 2024 a collaboration with Seagate that fully integrates its toolset with Se...
13/09/2024
AMSTERDAM DPA Microphones has introduced its 2061 Miniature Omnidirectional Microphone for applications ranging from broadcast studios and ENG to theaters, even...
13/09/2024
Rodney Alejandro Named Dean of Professional Writing and Music Technology Alejandro, Berklees current Songwriting Department chair and an alumnus of the colleg...
13/09/2024
The presidential debate between Vice President Kamala Harris and former President Donald Trump was watched by 67.1 million viewers, according to Nielsen....
13/09/2024
Brightcove introduced the Brightcove AI Suite, which uses AI to provide content creation, audience growth and engagement and increased revenue generation....
13/09/2024
Warner Bros. Discovery and Charter Communications said the reached a new multi-y...
13/09/2024
The History Channel's History This Week podcast returns for a fifth season on September 16....
13/09/2024
NEW YORK The ABC News Presidential Debate between Vice President Kamala Harris and former President Donald Trump drew an estimated audience of 67.1 million view...
13/09/2024
WASHINGTON, D.C. In response to Donald Trump's call for ABC stations to lose their broadcast licenses, Federal Communications Commission chairwoman Jessica ...
13/09/2024
IRVINE, Calif. Teradek will launch its new Prism 877 4K HEVC encoder/decoder cards at the IBC 2024, Sept. 13-16, at the RAI Amsterdam Convention Center....
13/09/2024
ATLANTA The Georgia Association of Broadcasters honored nine station employees at the early stage of their broadcast career as members of its first-ever class o...
13/09/2024
MONTREAL Grass Valley and broadcast equipment supply and service company ES Broadcast have inked a strategic partnership agreement that continues and expands a ...
13/09/2024
NEWARK, N.J. Panasonic Connect has announced the new AK-UCX100 4K studio camera and two new 4K multi-purpose cameras. In addition, the company said it will exte...
13/09/2024
OTTAWA, Canada Ross Video has launched Ultricore Tally, a full-featured, versatile tally control and management solution and will showcase it at IBC 2024, Sept....
13/09/2024
Both appointees join the company following extensive careers in the industry
By Matthew Corrigan
Published: September 13, 2024
Both appointees join the co...
13/09/2024
Hollyland Elevates Wireless Communications for Large-Scale Film and TV Productio...
13/09/2024
Blackmagic Design Announces Blackmagic PYXIS Monitor!
Brie Clayton September 12, 2024
0 Comments
New cinematic monitoring solution for Blackmagic PYXI...
13/09/2024
Sculpting Reimagined: Maxon Releases ZBrush for iPad
Brie Clayton September 13, 2024
0 Comments
Take the power of ZBrush on the go with reimagined UI,...
13/09/2024
WCVB Spotlights Berklee Institute for Accessible Arts Education The institutes ABLE summer camp program was featured in the 5 for Good segment.
By
Daniel Pe...
13/09/2024
13 Sep 2024
VEON General Counsel Omiyinka Doris Named Among Top 15 Legal Leader...
13/09/2024
Sky Studios Elstree opens Sky Up Academy to inspire 10,000 students a year to pu...
13/09/2024
Countering the autonomous aerial threat In an interview, Martin Woywod, Product Manager Counter-UAS (CUAS) Systems, Rohde & Schwarz , talks about the need for...
13/09/2024
Amsterdam September 13, 2024
WDR Relies on Riedel Backbone for Remote Production of UEFA Euro 2024The German regional public broadcaster Westdeutscher Rundfu...
13/09/2024
Amsterdam September 13, 2024
Riedel Expands Its Range of NSA Network Stream AdaptersRiedel Communications today announced the launch of two new additions to i...
13/09/2024
Amsterdam September 13, 2024
Riedel Communications Launches Virtual SmartPanel at IBC2024At IBC2024, Riedel Communications announced the launch of its ground...
13/09/2024
Amsterdam September 13, 2024
Riedel Unveils Revolutionary SAME Smart Audio & Mixing Engine at IBC2024Riedel SAME Delivers Ultra-Low-Latency Audio Processing ...
13/09/2024
Back to All News
Netflix Greenlights Love Untangled': A Nostalgic Korean Y...
13/09/2024
SAN JOSE, Calif. - Sept. 13, 2024 - Harmonic (NASDAQ: HLIT) is pleased to announ...
13/09/2024
Photo credit: GETTY IMAGES
Press release - 13 September 2024 09:50
Eutelsat ...
13/09/2024
Photo credit: AdobeStock_305149408- nuchao
Press release - 13 September 2024...
13/09/2024
Today, Apple Maps on the web is available in public beta, allowing users around the world to access Maps directly from their browser.1...
13/09/2024
Apple and Major League Baseball (MLB) today announced the September game schedule for Friday Night Baseball, a weekly doubleheader available to Apple TV+ subs...
13/09/2024
Starting with iOS 18.1, developers will be able to offer NFC contactless transactions using the Secure Element from within their own apps on iPhone, separate fr...
13/09/2024
Starting tomorrow, September 13, at 5 a.m. PDT, customers can pre-order Apple's powerful new iPhone 16 and iPhone 16 Pro models on apple.com/uk and in the r...
Editor's note: This post is part of the AI Decoded series, which demystifies AI by making the technology more accessible, and showcases new hardware, software, tools and accelerations for RTX PC users.
