Sony Pixel Power calrec Sony

Fast, Low-Cost Inference Offers Key to Profitable AI

23/01/2025

Businesses across every industry are rolling out AI services this year. For Microsoft, Oracle, Perplexity, Snap and hundreds of other leading companies, using the NVIDIA AI inference platform - a full stack comprising world-class silicon, systems and software - is the key to delivering high-throughput and low-latency inference and enabling great user experiences while lowering cost.

NVIDIA's advancements in inference software optimization and the NVIDIA Hopper platform are helping industries serve the latest generative AI models, delivering excellent user experiences while optimizing total cost of ownership. The Hopper platform also helps deliver up to 15x more energy efficiency for inference workloads compared to previous generations.

AI inference is notoriously difficult, as it requires many steps to strike the right balance between throughput and user experience.

But the underlying goal is simple: generate more tokens at a lower cost. Tokens represent words in a large language model (LLM) system - and with AI inference services typically charging for every million tokens generated, this goal offers the most visible return on AI investments and energy used per task.

Full-stack software optimization offers the key to improving AI inference performance and achieving this goal.

Cost-Effective User Throughput Businesses are often challenged with balancing the performance and costs of inference workloads. While some customers or use cases may work with an out-of-the-box or hosted model, others may require customization. NVIDIA technologies simplify model deployment while optimizing cost and performance for AI inference workloads. In addition, customers can experience flexibility and customizability with the models they choose to deploy.

NVIDIA NIM microservices, NVIDIA Triton Inference Server and the NVIDIA TensorRT library are among the inference solutions NVIDIA offers to suit users' needs:

NVIDIA NIM inference microservices are prepackaged and performance-optimized for rapidly deploying AI foundation models on any infrastructure - cloud, data centers, edge or workstations.

NVIDIA Triton Inference Server, one of the company's most popular open-source projects, allows users to package and serve any model regardless of the AI framework it was trained on.

NVIDIA TensorRT is a high-performance deep learning inference library that includes runtime and model optimizations to deliver low-latency and high-throughput inference for production applications.

Available in all major cloud marketplaces, the NVIDIA AI Enterprise software platform includes all these solutions and provides enterprise-grade support, stability, manageability and security.

With the framework-agnostic NVIDIA AI inference platform, companies save on productivity, development, and infrastructure and setup costs. Using NVIDIA technologies can also boost business revenue by helping companies avoid downtime and fraudulent transactions, increase e-commerce shopping conversion rates and generate new, AI-powered revenue streams.

Cloud-Based LLM Inference To ease LLM deployment, NVIDIA has collaborated closely with every major cloud service provider to ensure that the NVIDIA inference platform can be seamlessly deployed in the cloud with minimal or no code required. NVIDIA NIM is integrated with cloud-native services such as:

Amazon SageMaker AI, Amazon Bedrock Marketplace, Amazon Elastic Kubernetes Service

Google Cloud's Vertex AI, Google Kubernetes Engine

Microsoft Azure AI Foundry coming soon, Azure Kubernetes Service

Oracle Cloud Infrastructure's data science tools, Oracle Cloud Infrastructure Kubernetes Engine

Plus, for customized inference deployments, NVIDIA Triton Inference Server is deeply integrated into all major cloud service providers.

For example, using the OCI Data Science platform, deploying NVIDIA Triton is as simple as turning on a switch in the command line arguments during model deployment, which instantly launches an NVIDIA Triton inference endpoint.

Similarly, with Azure Machine Learning, users can deploy NVIDIA Triton either with no-code deployment through the Azure Machine Learning Studio or full-code deployment with Azure Machine Learning CLI. AWS provides one-click deployment for NVIDIA NIM from SageMaker Marketplace and Google Cloud provides a one-click deployment option on Google Kubernetes Engine (GKE). Google Cloud provides a one-click deployment option on Google Kubernetes Engine, while AWS offers NVIDIA Triton on its AWS Deep Learning containers.

The NVIDIA AI inference platform also uses popular communication methods for delivering AI predictions, automatically adjusting to accommodate the growing and changing needs of users within a cloud-based infrastructure.

From accelerating LLMs to enhancing creative workflows and transforming agreement management, NVIDIA's AI inference platform is driving real-world impact across industries. Learn how collaboration and innovation are enabling the organizations below to achieve new levels of efficiency and scalability.

Serving 400 Million Search Queries Monthly With Perplexity AI Perplexity AI, an AI-powered search engine, handles over 435 million monthly queries. Each query represents multiple AI inference requests. To meet this demand, the Perplexity AI team turned to NVIDIA H100 GPUs, Triton Inference Server and TensorRT-LLM.

Supporting over 20 AI models, including Llama 3 variations like 8B and 70B, Perplexity processes diverse tasks such as search, summarization and question-answering. By using smaller classifier models to route tasks to GPU pods, managed by NVIDIA Triton, the company delivers cost-efficient, responsive service under strict service level agreements.

Through model parallelism, which splits LLMs across GPUs, Perplexity achieved a threefold cost reduction while maintaining low latency and high accurac
LINK: https://blogs.nvidia.com/blog/ai-inference-platform/...
See more stories from nvidia

Most recent headlines

12/03/2025

Jeff Lilly Named WGN-TV Director Of Technology

CHICAGO Jeff Lilly has been named WGN-TV director of technology effective March 17, 2025, according to Ric Harris, WGN-TV vice president and general manager....

12/03/2025

Survey: Consumers Want More Shoppable TV Experiences

MOUNTAIN VIEW, Calif. A new study from LG Ad Solutions indicates that consumers want more features that would allow them to shop for products on the connected T...

12/03/2025

IP Showcase To Focus On IP, Broadcast Tech Convergence at 2025 NAB Show

BOTHELL, Wash. The Alliance for IP Media Solutions (AIMS), Advanced Media Workflow Association (AMWA) and the Video Services Forum (VSF) will once again present...

12/03/2025

Comcast Boosts Internet Speeds for More Than 20 Million Customers

PHILADELPHIA Comcast announced that it has upgraded Xfinity Internet speeds for more than 20 million customers for no additional cost....

12/03/2025

Create with Maxon: Cinema 4D Fundamentals Workshop - March 12-14

Create with Maxon: Cinema 4D Fundamentals Workshop - March 12-14 Brie Clayton March 11, 2025 0 Comments Makin' Waffles with Elly Wade During Marc...

11/03/2025

Give Me the Backstory: Get to Know Amel Guellaty, the Writer-Director of Where the Wind Comes From

By Lucy Spicer One of the most exciting things about the Sundance Film Festival...

11/03/2025

Salsa's Rhythmic Revival: A New Generation Discovers the Genre's Timeless Appeal

Salsa is making a comeback, captivating new listeners with its infectious energy...

11/03/2025

Canada's Own Rupi Kaur Curates the Latest Edition of Local Spot, Revealing the Songs and Artists That Shaped Her

For many people, music can serve as a reflection of their roots and upbringing. ...

11/03/2025

ST Engineering iDirect's Public Safety Solution Wins MSUA Satellite Mobile Innovation Award 2025

The solution delivers reliable, scalable and secure connectivity for critical pu...

11/03/2025

Weigel Broadcasting Taps Harmonic for Cloud-Based Playout-to-Delivery

SAN JOSE, Calif. Harmonic has announced that Weigel Broadcasting has deployed Harmonics VOS Media Software, which offers playout-to-delivery capabilities, inclu...

11/03/2025

Wheatstone To Unveil Linux Audio Driver At 2025 NAB Show

NEW BERN, N.C. Wheatstone will introduce a Linux audio driver for its WheatNet IP audio network during the 2025 NAB Show, April 5-9, at the Las Vegas Convention...

11/03/2025

Study: Women's Sports Are a Big Win for Advertisers

NEW YORK A new study finds that as TV viewership for women's sports surged by 131% in 2024, the programming also saw a 56% year-over-year increase in ad im...

11/03/2025

Al Sharqiya prepares for Ultra HD with FOR-A

Powerful switcher for news studio FOR-A, a cutting-edge video broadcast technology company backed by more than 50 years experience, has installed its HVS-1200 ...

11/03/2025

Intinor Unveils Advancements in Video Quality and Workflo...

Intinor, Sweden's leading developer of high-quality video over the internet, is unveiling significant advancements to its Direkt series at NAB 2025. With a ...

11/03/2025

Glensound extends yet further its market-leading communic...

Glensound, a leader in high-quality audio systems, is bringing yet more innovation to NAB Show 2025 (Booth N2270, Las Vegas Convention Center, 6-9 April). Well-...

11/03/2025

Anaconda Street Productions Uses Laser Projectors for Cos...

Whittier, Calif.-based Anaconda Street Productions (ASP) is a leading film and television production company known for creating captivating content that resonat...

11/03/2025

DNAV and Sierra Automated Systems Introduce Intelligent R...

DNAV, a full-service systems integrator, consultant, and manufacturer's representative of leading broadcast, AV, lighting, and display equipment, announces ...

11/03/2025

MNC Software appoints Darren Frearson as CEO

MNC Software Inc., a global leader in network solutions, is pleased to announce the appointment of Darren Frearson as its new Chief Executive Officer, effective...

11/03/2025

Polar Graphics announces a new product line powered by Xe...

Polar Graphics, a UK leading distributor for the broadcast, post and pro-AV industries, has signed a licensing agreement with XenData to include its XenData Arc...

11/03/2025

Chyron PAINT 9.9 Delivers Sharper Visuals and Smoother Workflows for Sports Telestration

Chyron PAINT 9.9 Delivers Sharper Visuals and Smoother Workflows for Sports Tele...

11/03/2025

Alice in Wonderlight Filmed with URSA Mini Pro 12K OLPF

Alice in Wonderlight Filmed with URSA Mini Pro 12K OLPF Brie Clayton March 11, 2025 0 Comments Play captured in 8K60p as part of national initiative t...

11/03/2025

Twisting in the Wind - An Apple Motion Tutorial

Twisting in the Wind - An Apple Motion Tutorial Simon Ubsdell March 11, 2025 0 Comments Another very simple text-based project that uses an unusual co...

11/03/2025

The secret to looking inside a Project in Premiere Pro

The secret to looking inside a Project in Premiere Pro Colin Smith March 11, 2025 0 Comments This tutorial demonstrates the incredible capabilities of...

11/03/2025

New Book Explores How the Quiet Storm Shaped Modern R&B

New Book Explores How the Quiet Storm Shaped Modern R&B In The Quiet Storm, Berklee Online alumnus Amani Roberts explores how the radio format defined America...

11/03/2025

Slapshot app makes AI-powered VFX accessible to all

Slapshot aims to make VFX available for a range of users including independent editors, colourists, content creators and established visual effects companies B...

11/03/2025

Estonian campus to be first modern film complex in the Baltics'

Aiming to attract international film production, the 16 million complex is scheduled to open next year By Matthew Corrigan Published: March 11, 2025 Aimi...

11/03/2025

Meet the support and projects manager

Kamil Pietrzyk, support and projects manager at CueScript, tells TVBEurope how an interest in electronics and IT prompted a career in the broadcast industry By...

11/03/2025

Underwater DoP Ian Seabrook Lights Last Breath with Aster...

Underwater DoP Ian Seabrook on Last Breath Credit: Jon Borg / 2024 FOCUS FEATURES LLC Canadian/British Director of Photography, Ian Seabrook is one of the ...

11/03/2025

Los Angeles iconic Sound Factory installs custom mains wi...

Originally opened in the early 1960s as Moonglow Records, The Sound Factory name was coined by Producer David Hassinger, who purchased the studio in 1969. A lit...

11/03/2025

BCNEXXT Celebrates 10 Years of Innovation as Customers Co...

BCNEXXT, a trailblazer in virtualized, cloud-native systems for Linear, VoD, and OTT publishing, proudly marks a decade of redefining broadcast playout. This mi...

11/03/2025

Experience Commerce of Cheil India Wins Social and Media...

Experience Commerce, an integrated marketing agency within the Cheil Network, has been appointed as the official digital partner for Parle Candy Culture, reinfo...

11/03/2025

Survey: More Viewers Turn to Streaming for Sports

PORTSMOUTH, N.H. A new survey highlights how far major streaming platforms have come in terms of offering sports, with findings that show the number of people ...

11/03/2025

Tegna Names John Trevio President and GM at WKYC in Cleveland

TYSONS, Va. Tegna Inc. has announced today that John Trevi o has been named president and general manager at WKYC, the NBC affiliate serving Cleveland, Ohio, ef...

11/03/2025

Canela Media Selects Brightcove to Power Streaming Service

BOSTON Brightcove has announced that Canela Media is using Brightcove's technologies to power its streaming operations....

11/03/2025

Chunghwa Telecom Taps Broadpeak for Dynamic Ad Insertion

CESSON-SEVIGNE, France Chunghwa Telecom, the leading telecommunications operator in Taiwan, has selected Broadpeak to provide solutions for its streaming servic...

11/03/2025

Leader sets US debut of LPX500 Waveform Monitor for NAB 2025

Leader sets US debut of LPX500 Waveform Monitor for NAB 2025 Brie Clayton March 10, 2025 0 Comments Test & measurement innovator, Leader Instruments C...

11/03/2025

The Berklee Institute of Jazz and Gender Justice Presents the Grand Gathering

The Berklee Institute of Jazz and Gender Justice Presents the Grand Gathering The Signature Series concert will feature performances by all nine of the instit...

11/03/2025

Space42 and Viasat Announce Partnership to Explore Shared Global 5G Non-Terrestrial Network Initiative

Abu Dhabi, UAE and Carlsbad, California 12 March 2025 Space42, (ADX: SPACE42...

11/03/2025

Tribeca Enterprises, SIC, and the Lisbon City Council Announce Second Annual Tribeca Festival Lisboa October 30-November 1

March 11th, 2025 Tribeca Enterprises, SIC, and the Lisbon City Council Announce...

11/03/2025

Drones, Flycam To Highlight 60+-Camera Coverage of THE PLAYERS This Weekend

Drones, Flycam To Highlight 60+-Camera Coverage of THE PLAYERS This Weekend 60+ cameras, NEP's PGA TOUR fleet, 25 talent will be deployed at TPC Sawgrass B...

11/03/2025

The Art of 9:16: How Corporate Content Producers Have Embraced Vertical Video

The Art of 9:16: How Corporate Content Producers Have Embraced Vertical Video Leaders from the corporate space share their advice and success stories By SVG St...

11/03/2025

SVG Rewind: NFL's Tim Tubito on Elevating Gameday Activations of Super Bowl LIX With New Multi-Use Stage

SVG Rewind: NFL's Tim Tubito on Elevating Gameday Activations of Super Bowl ...

11/03/2025

New Sponsor Spotlight: Ventuz's David Paniego on the Increasing Synergy Between Broadcast Graphics and In-Venue Displays

New Sponsor Spotlight: Ventuz's David Paniego on the Increasing Synergy Betw...

11/03/2025

Best Snow Day Ever: NESN, Bruins Roll Out NHL's Latest Animated Data-Visualization Altcast With a STEM Twist

Best Snow Day Ever: NESN, Bruins Roll Out NHL's Latest Animated Data-Visuali...

11/03/2025

Trailer revealed for Sky Exclusive comedy series Hacks season 4, coming to screens 11 April

To view this content, please enable our use of cookies. To do so, click Privacy ...

11/03/2025

Netflix Reveals Official Trailer for 'The Lady's Companion'

Back to All News Netflix Reveals Official Trailer for The Ladys CompanionPlay Video Play Video Entertainment 11 March 2025 GlobalSpain Link copied to clip...

11/03/2025

Geeta Gandbhir's 'The Perfect Neighbor' to Release on Netflix in 2025

Back to All News Geeta Gandbhir's The Perfect Neighbor to Release on Netfli...

11/03/2025

The ROI of AI: New research on how AI is transforming B2B...

The ROI of AI: New research on how AI is transforming B2B sales Published on Mar 11, 2025 Categories: Research, Data and insights LinkedIn Corporate Commun...

11/03/2025

Weigel Broadcasting Embraces Cloud-Based Playout-to-Delivery Solution from Harmonic

SAN JOSE, Calif. - March 11, 2025 - Harmonic (NASDAQ: HLIT) today announced that...

11/03/2025

Al Sharqiya prepares for Ultra HD with FOR-A

Powerful new switcher for news studio...