Editor's note: This post is part of the AI Decoded series, which demystifies AI by making the technology more accessible, and showcases new hardware, software, tools and accelerations for GeForce RTX PC and NVIDIA RTX workstation users.
Large language models (LLMs) are reshaping productivity. They're capable of drafting documents, summarizing web pages and, having been trained on vast quantities of data, accurately answering questions about nearly any topic.
LLMs are at the core of many emerging use cases in generative AI, including digital assistants, conversational avatars and customer service agents.
Many of the latest LLMs can run locally on PCs or workstations. This is useful for a variety of reasons: users can keep conversations and content private on-device, use AI without the internet, or simply take advantage of the powerful NVIDIA GeForce RTX GPUs in their system. Other models, because of their size and complexity, do no't fit into the local GPU's video memory (VRAM) and require hardware in large data centers.
However, Iit i's possible to accelerate part of a prompt on a data-center-class model locally on RTX-powered PCs using a technique called GPU offloading. This allows users to benefit from GPU acceleration without being as limited by GPU memory constraints.
Size and Quality vs. Performance There's a tradeoff between the model size and the quality of responses and the performance. In general, larger models deliver higher-quality responses, but run more slowly. With smaller models, performance goes up while quality goes down.
This tradeoff isn't always straightforward. There are cases where performance might be more important than quality. Some users may prioritize accuracy for use cases like content generation, since it can run in the background. A conversational assistant, meanwhile, needs to be fast while also providing accurate responses.
The most accurate LLMs, designed to run in the data center, are tens of gigabytes in size, and may not fit in a GPU's memory. This would traditionally prevent the application from taking advantage of GPU acceleration.
However, GPU offloading uses part of the LLM on the GPU and part on the CPU. This allows users to take maximum advantage of GPU acceleration regardless of model size.
Optimize AI Acceleration With GPU Offloading and LM Studio LM Studio is an application that lets users download and host LLMs on their desktop or laptop computer, with an easy-to-use interface that allows for extensive customization in how those models operate. LM Studio is built on top of llama.cpp, so it's fully optimized for use with GeForce RTX and NVIDIA RTX GPUs.
LM Studio and GPU offloading takes advantage of GPU acceleration to boost the performance of a locally hosted LLM, even if the model can't be fully loaded into VRAM.
With GPU offloading, LM Studio divides the model into smaller chunks, or subgraphs, which represent layers of the model architecture. Subgraphs aren't permanently fixed on the GPU, but loaded and unloaded as needed. With LM Studio's GPU offloading slider, users can decide how many of these layers are processed by the GPU.
LM Studio's interface makes it easy to decide how much of an LLM should be loaded to the GPU. For example, imagine using this GPU offloading technique with a large model like Gemma-2-27B. 27B refers to the number of parameters in the model, informing an estimate as to how much memory is required to run the model.
According to 4-bit quantization, a technique for reducing the size of an LLM without significantly reducing accuracy, each parameter takes up a half byte of memory. This means that the model should require about 13.5 billion bytes, or 13.5GB - plus some overhead, which generally ranges from 1-5GB.
Accelerating this model entirely on the GPU requires 19GB of VRAM, available on the GeForce RTX 4090 desktop GPU. With GPU offloading, the model can run on a system with a lower-end GPU and still benefit from acceleration.
In LM Studio, it's possible to assess the performance impact of different levels of GPU offloading, compared with CPU only. The below table shows the results of running the same query across different offloading levels on a GeForce RTX 4090 desktop GPU.
Depending on the percent of the model offloaded to GPU, users see increasing throughput performance compared with running on CPUs alone. For the Gemma-2-27B, performance goes from an anemic 2.1 tokens per second to increasingly usable speeds the more the GPU is used. This enables users to benefit from the performance of larger models that they otherwise would've been unable to run. On this particular model, even users with an 8GB GPU can enjoy a meaningful speedup versus running only on CPUs. Of course, an 8GB GPU can always run a smaller model that fits entirely in GPU memory and get full GPU acceleration.
Achieving Optimal Balance LM Studio's GPU offloading feature is a powerful tool for unlocking the full potential of LLMs designed for the data center, like Gemma-2-27B, locally on RTX AI PCs. It makes larger, more complex models accessible across the entire lineup of PCs powered by GeForce RTX and NVIDIA RTX GPUs.
Download LM Studio to try GPU offloading on larger models, or experiment with a variety of RTX-accelerated LLMs running locally on RTX AI PCs and workstations.
Generative AI is transforming gaming, videoconferencing and interactive experiences of all kinds. Make sense of what's new and what's next by subscribing to the AI Decoded newsletter.
Most recent headlines
09/12/2024
Dalet, a leading technology and service provider for media-rich organizations, today announced that it has been named an IDC Innovator in the IDC Innovators: ...
09/11/2024
Dalet, a leading technology and service provider for media-rich organizations, today announced three new members of its executive team. Tara Bryant joins as Chi...
24/10/2024
Independent Midwest Broadcaster selects Rohde & Schwarz TE1 liquid-cooled transm...
23/10/2024
By Lucy Spicer
One of the most exciting things about the Sundance Film Festival...
23/10/2024
Advanced, future-proof technology and rapid deployment builds on Phase One's...
23/10/2024
New Delhi, India - October 21, 2024 - Nielsen, a global leader in audience measurement, data and analytics, has today announced the expansion of YouTube Connect...
23/10/2024
Bangkok, Thailand - October 23, 2024 - Nielsen, a global leader in audience measurement, data and analytics, has today announced the expansion of YouTube Connec...
23/10/2024
IBC has opened submissions for the 2025 Accelerator Media Innovation Program, which will showcase proofs of concept at IBC2025....
23/10/2024
ENGLEWOOD CLIFFS, N.J. LG Electronics USA has launched a dedicated section for news called News Hub on LG Channels that provides weather reports, and local an...
23/10/2024
Cost-efficiency and ROI are front of mind for media business and technology leaders across Europe - LTN shares insight and case studies on cost-effective growth...
23/10/2024
Celebrating five years of Fresco with powerful new updates that unlock digital d...
23/10/2024
The company is employing the newest version of BATON Captions from Interra to de...
23/10/2024
Mark Smith, head of the Accelerator programme, tells TVBEurope it is about understanding what challenges technology buyers want to solve, and what is coming nex...
23/10/2024
As streaming services struggle to boost profits by raising prices, a new Parks Associates study has found more streaming consumers are opting for basic tiers wi...
23/10/2024
CommScope has introduced the new SYSTIMAX GigaSHIELD X10D end-to-end Cat 6A shielded platform, designed to help cable operators support the next generation of h...
23/10/2024
CUPERTINO, Calif. Interra Systems has announced that its long-time customer Starz has upgraded its caption generation and quality control workflows to the lates...
23/10/2024
Berklee Celebrates Annual BMI Day with Acclaimed Composer Mychael Danna Berklee's Sean McMahon and BMI's Tracy McKnight presented the BMI Film Scoring...
23/10/2024
NBA Tip-Off 2024: ESPN Gets Going With New Big Three in Philadelphia, Regular-Se...
23/10/2024
Shure Enters WMAS Realm With Axient Digital PSM New wireless in-ear monitoring system is also beta-testing as an IFB for broadcast sports By Dan Daley, Audio E...
23/10/2024
Sophomore-Year Streaming: Utah Jazz Begin Second Regular Season on Jazz+ Tonight...
23/10/2024
SVG New Sponsor Spotlight: Mo-Sys Founder Michael Geissler on the Rise of Virtua...
23/10/2024
Back to All News
The Shadow Strays Becomes a Global Hit, Trending in 85 Countri...
23/10/2024
Fox Corporation's President of Advertising Sales Jeff Collins Announces Stra...
23/10/2024
Press Release: Cartesian Supports Redge Technologies in Aligning Their Redge Med...
23/10/2024
Lionel Messi makes his historic MLS Cup Playoffs debut this Friday, October 25, as the playoffs return to MLS Season Pass on Apple TV for a free primetime match...
23/10/2024
NVIDIA founder and CEO Jensen Huang joined the king of Denmark to launch the cou...
23/10/2024
Editor's note: This post is part of the AI Decoded series, which demystifies AI by making the technology more accessible, and showcases new hardware, softwa...
23/10/2024
23 October 2024, 05:30 (PDT) Kenneth Branagh Theatre Company's Production o...
23/10/2024
Facebook
Twitter
LinkedIn
Order intake: 15.6 billion, up 23% on an organic basis1 ( 26% total change)
Sales: 14.1 billion, up 6.2% on an organic basis ...
22/10/2024
From L-R: Robert Steen, Trude Steen, Mia Steen and Benjamin Ree (Marc Sagliocco/...
22/10/2024
At Spotify, we know our users like to flex their creativity. We've seen this come to life across the more than 8 billion playlists listeners have created to...
22/10/2024
In 2022, Spotify's partnership with Spelman College took flight as part of our NextGen audio program that aims to infuse, activate, and grow podcast culture...
22/10/2024
October 22, 2024
Comercio TV Chooses Broadpeak to Optimize OTT Video Delivery and Monetization
CESSON-SEVIGNE, France - Oct. 22, 2024 - Broadpeak , a leadi...
22/10/2024
In September, audiences in Mexico decreased their streaming usage by 1 point compared to the previous month, accounting for 21.5% of TV viewing.
Disclaimer: YU...
22/10/2024
FOX exhibits largest monthly gain as multiple NFL and NCAA football distributors...
22/10/2024
NEW YORK Live sports TV streaming platform Fubo has launched standalone premium subscription services, the company said today....
22/10/2024
NEW YORK & ST. GALLEN, Switzerland In the run-up to the start of the NBA season, Sportradar has introduced a new suite of innovative fan-engagement solutions to...
22/10/2024
Digital Domain Relaunches Advertising Experiences Group With New Hires
Brie Clayton October 22, 2024
0 Comments
Digital Domain, a pioneer in visual ef...
22/10/2024
Hiroshi Okuyama Film My Sunshine Shot and Graded Using Blackmagic Design Product...
22/10/2024
InnovateUK and EU Approve £1.2 Million in Funding for Immersive Technology from ...
22/10/2024
Lauv Inspires Berklee Students with Authenticity and Chart-Topping Success Pop sensation Lauv shares insights on creativity, vulnerability, and overcoming the...
22/10/2024
Applications Open for Next Jazz Legacy 2025 Cohort Backed by a $1.25 million Mellon Foundation grant, the initiative from New Music USA and the Berklee Instit...
22/10/2024
This Rihanna Song Has Secret' '80s Easter Eggs The SOS songwriter revealed a nostalgic twist to the lyrics, and it's going viral on TikTok.
By...
22/10/2024
The CTO Office has been tasked with the gradual introduction and integration of new technologies to improve the efficiency, reach and quality of media productio...
22/10/2024
SOUTHPORT, Conn. and NEW YORK Diamond Sports Group and the online gaming company FanDuel have announced a wide-ranging long-term commercial partnership that wil...
22/10/2024
New research indicates that Roku remains the top connected TV (CTV) streaming device in North America, with a 37% share of the open programmatic ads sold on spe...
22/10/2024
The newly released S&P Global Market Intelligence Radio & TV Annual Outlook from Kagan finds some good news in this years ad-revenue outlook and some not-so-goo...
22/10/2024
BRISTOL, Conn. and PALM BEACH GARDENS, Fla. TGL, the new tech-infused golf league founded by Tiger Woods and Rory McIlroy's TMRW Sports in partnership with...
22/10/2024
WASHINGTON, D.C. The National Association of Broadcasters has filed a letter with the FCC expressing concerns about the impact the proposed merger between SES a...
Editor's note: This post is part of the AI Decoded series, which demystifies AI by making the technology more accessible, and showcases new hardware, software, tools and accelerations for GeForce RTX PC and NVIDIA RTX workstation users.
