How to Accelerate Larger LLMs Locally on RTX With LM Studio
23/10/2024
Large language models (LLMs) are reshaping productivity. They're capable of drafting documents, summarizing web pages and, having been trained on vast quantities of data, accurately answering questions about nearly any topic.
LLMs are at the core of many emerging use cases in generative AI, including digital assistants, conversational avatars and customer service agents.
Many of the latest LLMs can run locally on PCs or workstations. This is useful for a variety of reasons: users can keep conversations and content private on-device, use AI without the internet, or simply take advantage of the powerful NVIDIA GeForce RTX GPUs in their system. Other models, because of their size and complexity, do no't fit into the local GPU's video memory (VRAM) and require hardware in large data centers.
However, Iit i's possible to accelerate part of a prompt on a data-center-class model locally on RTX-powered PCs using a technique called GPU offloading. This allows users to benefit from GPU acceleration without being as limited by GPU memory constraints.
Size and Quality vs. Performance There's a tradeoff between the model size and the quality of responses and the performance. In general, larger models deliver higher-quality responses, but run more slowly. With smaller models, performance goes up while quality goes down.
This tradeoff isn't always straightforward. There are cases where performance might be more important than quality. Some users may prioritize accuracy for use cases like content generation, since it can run in the background. A conversational assistant, meanwhile, needs to be fast while also providing accurate responses.
The most accurate LLMs, designed to run in the data center, are tens of gigabytes in size, and may not fit in a GPU's memory. This would traditionally prevent the application from taking advantage of GPU acceleration.
However, GPU offloading uses part of the LLM on the GPU and part on the CPU. This allows users to take maximum advantage of GPU acceleration regardless of model size.
Optimize AI Acceleration With GPU Offloading and LM Studio LM Studio is an application that lets users download and host LLMs on their desktop or laptop computer, with an easy-to-use interface that allows for extensive customization in how those models operate. LM Studio is built on top of llama.cpp, so it's fully optimized for use with GeForce RTX and NVIDIA RTX GPUs.
LM Studio and GPU offloading takes advantage of GPU acceleration to boost the performance of a locally hosted LLM, even if the model can't be fully loaded into VRAM.
With GPU offloading, LM Studio divides the model into smaller chunks, or subgraphs, which represent layers of the model architecture. Subgraphs aren't permanently fixed on the GPU, but loaded and unloaded as needed. With LM Studio's GPU offloading slider, users can decide how many of these layers are processed by the GPU.
LM Studio's interface makes it easy to decide how much of an LLM should be loaded to the GPU. For example, imagine using this GPU offloading technique with a large model like Gemma-2-27B. 27B refers to the number of parameters in the model, informing an estimate as to how much memory is required to run the model.
According to 4-bit quantization, a technique for reducing the size of an LLM without significantly reducing accuracy, each parameter takes up a half byte of memory. This means that the model should require about 13.5 billion bytes, or 13.5GB - plus some overhead, which generally ranges from 1-5GB.
Accelerating this model entirely on the GPU requires 19GB of VRAM, available on the GeForce RTX 4090 desktop GPU. With GPU offloading, the model can run on a system with a lower-end GPU and still benefit from acceleration.
In LM Studio, it's possible to assess the performance impact of different levels of GPU offloading, compared with CPU only. The below table shows the results of running the same query across different offloading levels on a GeForce RTX 4090 desktop GPU.
Depending on the percent of the model offloaded to GPU, users see increasing throughput performance compared with running on CPUs alone. For the Gemma-2-27B, performance goes from an anemic 2.1 tokens per second to increasingly usable speeds the more the GPU is used. This enables users to benefit from the performance of larger models that they otherwise would've been unable to run. On this particular model, even users with an 8GB GPU can enjoy a meaningful speedup versus running only on CPUs. Of course, an 8GB GPU can always run a smaller model that fits entirely in GPU memory and get full GPU acceleration.
Achieving Optimal Balance LM Studio's GPU offloading feature is a powerful tool for unlocking the full potential of LLMs designed for the data center, like Gemma-2-27B, locally on RTX AI PCs. It makes larger, more complex models accessible across the entire lineup of PCs powered by GeForce RTX and NVIDIA RTX GPUs.
Download LM Studio to try GPU offloading on larger models, or experiment with a variety of RTX-accelerated LLMs running locally on RTX AI PCs and workstations.
Generative AI is transforming gaming, videoconferencing and interactive experiences of all kinds. Make sense of what's new and what's next by subscribing to the AI Decoded newsletter.
North America Stories
23/10/2024
Give Me the Backstory: Get to Know Greg Jardin, the Writer-Director of It's What's Inside
By Lucy Spicer One of the most exciting things about the Sundance Film Festival...
23/10/2024
Nielsen launches deduplicated YouTube CTV campaign measurement in India
New Delhi, India - October 21, 2024 - Nielsen, a global leader in audience measurement, data and analytics, has today announced the expansion of YouTube Connect...
23/10/2024
Nielsen launches deduplicated YouTube CTV campaign measurement in Thailand
Bangkok, Thailand - October 23, 2024 - Nielsen, a global leader in audience measurement, data and analytics, has today announced the expansion of YouTube Connec...
23/10/2024
Innovation Through Experimentation: IBC Launches 2025 Accelerator Program
IBC has opened submissions for the 2025 Accelerator Media Innovation Program, which will showcase proofs of concept at IBC2025....
23/10/2024
LG Channels Adds Local Stations to Free Streaming Lineup
ENGLEWOOD CLIFFS, N.J. LG Electronics USA has launched a dedicated section for news called News Hub on LG Channels that provides weather reports, and local an...
23/10/2024
Celebrating five years of Fresco with powerful new updates that unlock digital drawing and painting for everyone
Celebrating five years of Fresco with powerful new updates that unlock digital d...
23/10/2024
Cost-Conscious Consumers Embrace Ad-Supported Streaming, Study Finds
As streaming services struggle to boost profits by raising prices, a new Parks Associates study has found more streaming consumers are opting for basic tiers wi...
23/10/2024
CommScope Bows SYSTIMAX GigaSHIELD X10D Cat 6A Shielded Platform
CommScope has introduced the new SYSTIMAX GigaSHIELD X10D end-to-end Cat 6A shielded platform, designed to help cable operators support the next generation of h...
23/10/2024
Starz Upgrades Interra Systems' AI/ML-Based Captions Solution
CUPERTINO, Calif. Interra Systems has announced that its long-time customer Starz has upgraded its caption generation and quality control workflows to the lates...
23/10/2024
Berklee Celebrates Annual BMI Day with Acclaimed Composer Mychael Danna
Berklee Celebrates Annual BMI Day with Acclaimed Composer Mychael Danna Berklee's Sean McMahon and BMI's Tracy McKnight presented the BMI Film Scoring...
23/10/2024
NBA Tip-Off 2024: ESPN Gets Going With New Big Three in Philadelphia, Regular-Season Debut of Intuit Dome in L.A.
NBA Tip-Off 2024: ESPN Gets Going With New Big Three in Philadelphia, Regular-Se...
23/10/2024
Shure Enters WMAS Realm With Axient Digital PSM
Shure Enters WMAS Realm With Axient Digital PSM New wireless in-ear monitoring system is also beta-testing as an IFB for broadcast sports By Dan Daley, Audio E...
23/10/2024
Sophomore-Year Streaming: Utah Jazz Begin Second Regular Season on Jazz+ Tonight vs. the Memphis Grizzlies
Sophomore-Year Streaming: Utah Jazz Begin Second Regular Season on Jazz+ Tonight...
23/10/2024
SVG New Sponsor Spotlight: Mo-Sys Founder Michael Geissler on the Rise of Virtual Production Solutions
SVG New Sponsor Spotlight: Mo-Sys Founder Michael Geissler on the Rise of Virtua...
23/10/2024
'The Shadow Strays' Becomes a Global Hit, Trending in 85 Countries on Netflix
Back to All News The Shadow Strays Becomes a Global Hit, Trending in 85 Countri...
23/10/2024
Denmark Launches Leading Sovereign AI Supercomputer to Solve Scientific Challenges With Social Impact
NVIDIA founder and CEO Jensen Huang joined the king of Denmark to launch the cou...
23/10/2024
How to Accelerate Larger LLMs Locally on RTX With LM Studio
Editor's note: This post is part of the AI Decoded series, which demystifies AI by making the technology more accessible, and showcases new hardware, softwa...
22/10/2024
Benjamin Ree's Ibelin Takes Viewers to a Different World and Moves Them to Tears
From L-R: Robert Steen, Trude Steen, Mia Steen and Benjamin Ree (Marc Sagliocco/...
22/10/2024
The Gauge: Mexico September 2024
In September, audiences in Mexico decreased their streaming usage by 1 point compared to the previous month, accounting for 21.5% of TV viewing. Disclaimer: YU...
22/10/2024
Disney Retakes Top Spot in Nielsen's Media Distributor Gauge as Football Hikes TV Viewing in September
FOX exhibits largest monthly gain as multiple NFL and NCAA football distributors...
22/10/2024
Fubo Unveils Standalone Premium Subscription Networks
NEW YORK Live sports TV streaming platform Fubo has launched standalone premium subscription services, the company said today....
22/10/2024
Sportradar Launches New Suite of NBA Fan-Engagement Solutions
NEW YORK & ST. GALLEN, Switzerland In the run-up to the start of the NBA season, Sportradar has introduced a new suite of innovative fan-engagement solutions to...
22/10/2024
Digital Domain Relaunches Advertising Experiences Group With New Hires
Digital Domain Relaunches Advertising Experiences Group With New Hires Brie Clayton October 22, 2024 0 Comments Digital Domain, a pioneer in visual ef...
22/10/2024
Hiroshi Okuyama Film My Sunshine Shot and Graded Using Blackmagic Design Products
Hiroshi Okuyama Film My Sunshine Shot and Graded Using Blackmagic Design Product...
22/10/2024
InnovateUK and EU Approve 1.2 Million in Funding for Immersive Technology from Disguise
InnovateUK and EU Approve £1.2 Million in Funding for Immersive Technology from ...
22/10/2024
Lauv Inspires Berklee Students with Authenticity and Chart-Topping Success
Lauv Inspires Berklee Students with Authenticity and Chart-Topping Success Pop sensation Lauv shares insights on creativity, vulnerability, and overcoming the...
22/10/2024
Applications Open for Next Jazz Legacy 2025 Cohort
Applications Open for Next Jazz Legacy 2025 Cohort Backed by a $1.25 million Mellon Foundation grant, the initiative from New Music USA and the Berklee Instit...
22/10/2024
This Rihanna Song Has Secret' '80s Easter Eggs
This Rihanna Song Has Secret' '80s Easter Eggs The SOS songwriter revealed a nostalgic twist to the lyrics, and it's going viral on TikTok. By...
22/10/2024
Bally Sports Rebrands as FanDuel Sports Network
SOUTHPORT, Conn. and NEW YORK Diamond Sports Group and the online gaming company FanDuel have announced a wide-ranging long-term commercial partnership that wil...
22/10/2024
Roku Keeps 37% Share of North American Streaming Device Market
New research indicates that Roku remains the top connected TV (CTV) streaming device in North America, with a 37% share of the open programmatic ads sold on spe...
22/10/2024
S&P: TV-Station Advertising To Grow 14% to $24.95 Billion in 2024
The newly released S&P Global Market Intelligence Radio & TV Annual Outlook from Kagan finds some good news in this years ad-revenue outlook and some not-so-goo...
22/10/2024
ESPN To Bring Tech-Heavy TGL Golf to Primetime in 2025
BRISTOL, Conn. and PALM BEACH GARDENS, Fla. TGL, the new tech-infused golf league founded by Tiger Woods and Rory McIlroy's TMRW Sports in partnership with...
22/10/2024
NAB Expresses Concentration Concerns Over SES-Intelsat Merger
WASHINGTON, D.C. The National Association of Broadcasters has filed a letter with the FCC expressing concerns about the impact the proposed merger between SES a...
22/10/2024
Michael Malone, Long-Time B+C' Editor and Reporter, Dies at 55
Michael Malone, a long-time reporter, editor and content director at Broadcasting+Cable, Multichannel News and NextTV, died Saturday (Oct. 19) at Memorial Sloan...
22/10/2024
Educational Cosmetic Procedure Live Streams Powered by Blackmagic Design
Educational Cosmetic Procedure Live Streams Powered by Blackmagic Design Brie Clayton October 21, 2024 0 Comments Elijah Zane Echeveste, an independen...
22/10/2024
FilmLight Colour Awards reveals 2024 nominees
FilmLight Colour Awards reveals 2024 nominees Brie Clayton October 21, 2024 0 Comments Jury selects 25 nominees across six categories - winners to be ...
22/10/2024
Title Rebuild with the Boris FX Suite: Kraven The Hunter
Title Rebuild with the Boris FX Suite: Kraven The Hunter Brie Clayton October 21, 2024 0 Comments Epic 3D Title Rebuild: Kraven The Hunter Title Anima...
22/10/2024
Netflix Celebrates a Decade of Aussie Storytelling with New Sydney Office
Back to All News Netflix Celebrates a Decade of Aussie Storytelling with New Sydney Office Business 22 October 2024 GlobalAustraliaNew Zealand Link copied ...
22/10/2024
Ross Video to showcase multiformat IP integration at SATIS 2024 in France
Ottawa, Canada - October 22, 2024 - Ross Video, a global leader in video production technology, is participating in SATIS 2024, which will take place November 6...
22/10/2024
Ross Video prsentera l'intgration IP multiformat au salon SATIS en France
Ottawa, Canada - 22 Octobre 2024 - Ross Video, leader mondial des technologies de production vid o sera pr sente au SATIS 2024, qui se tiendra les 6 et 7 novem...
22/10/2024
Tribeca Films Announces New Output Deals with Kanopy and Kinema to Distribute the Best of Independent Cinema to Wider Audiences
October 22nd, 2024 TRIBECA FILMS ANNOUNCES NEW OUTPUT DEALS WITH KANOPY AND KIN...
22/10/2024
SVG Summit 2024: NBC Sports' Reflection on Paris Olympics Highlights a Loaded Main Stage Agenda
SVG Summit 2024: NBC Sports' Reflection on Paris Olympics Highlights a Loade...
22/10/2024
TNT Sports' REMI Remodel: A Conversation With WBD VP, Technology and Operations, Chris Brown
TNT Sports' REMI Remodel: A Conversation With WBD VP, Technology and Operati...
22/10/2024
NBA Tip-Off 2024: NBA on TNT Plans a Bash in Beantown for Opening Night
NBA Tip-Off 2024: NBA on TNT Plans a Bash in Beantown for Opening Night Doubleheader offers Knicks-Celtics at 7:30, Timberwolves-Lakers at 10 By Kristian Hern...
22/10/2024
SVG All-Stars: Barney Carleton, Senior Director, Broadcast Planning and Strategy, NBA
SVG All-Stars: Barney Carleton, Senior Director, Broadcast Planning and Strategy...
22/10/2024
Netflix Releases Trailer for 'The Helicopter Heist'
Back to All News Netflix Releases Trailer for The Helicopter HeistPlay Video Play Video Entertainment 22 October 2024 GlobalSweden Link copied to clipboar...
22/10/2024
Uncover the Secret Pulse of AI-Enhanced Tokyo: Tokyo Override' Premieres November 21
Back to All News Uncover the Secret Pulse of AI-Enhanced Tokyo: Tokyo Override...
22/10/2024
Top 10 Week of Oct. 14: Outer Banks' Strikes Gold for Second Week, Lonely Planet' Rises to #1
Back to All News Top 10 Week of Oct. 14: Outer Banks' Strikes Gold for Sec...
22/10/2024
What Is Agentic AI?
AI chatbots use generative AI to provide responses based on a single interaction. A person makes a query and the chatbot uses natural language processing to rep...
22/10/2024
NVIDIA Brings Generative AI Tools, Simulation and Perception Workflows to ROS Developer Ecosystem
At ROSCon in Odense, one of Denmark's oldest cities and a hub of automation,...