Sony Pixel Power calrec Sony

Explaining Tokens - the Language and Currency of AI

17/03/2025

Under the hood of every AI application are algorithms that churn through data in their own language, one based on a vocabulary of tokens.

Tokens are tiny units of data that come from breaking down bigger chunks of information. AI models process tokens to learn the relationships between them and unlock capabilities including prediction, generation and reasoning. The faster tokens can be processed, the faster models can learn and respond.

AI factories - a new class of data centers designed to accelerate AI workloads - efficiently crunch through tokens, converting them from the language of AI to the currency of AI, which is intelligence.

With AI factories, enterprises can take advantage of the latest full-stack computing solutions to process more tokens at lower computational cost, creating additional value for customers. In one case, integrating software optimizations and adopting the latest generation NVIDIA GPUs reduced cost per token by 20x compared to unoptimized processes on previous-generation GPUs - delivering 25x more revenue in just four weeks.

By efficiently processing tokens, AI factories are manufacturing intelligence - the most valuable asset in the new industrial revolution powered by AI.

What Is Tokenization? Whether a transformer AI model is processing text, images, audio clips, videos or another modality, it will translate the data into tokens. This process is known as tokenization.

Efficient tokenization helps reduce the amount of computing power required for training and inference. There are numerous tokenization methods - and tokenizers tailored for specific data types and use cases can require a smaller vocabulary, meaning there are fewer tokens to process.

For large language models (LLMs), short words may be represented with a single token, while longer words may be split into two or more tokens.

The word darkness, for example, would be split into two tokens, dark and ness, with each token bearing a numerical representation, such as 217 and 655. The opposite word, brightness, would similarly be split into bright and ness, with corresponding numerical representations of 491 and 655.

In this example, the shared numerical value associated with ness can help the AI model understand that the words may have something in common. In other situations, a tokenizer may assign different numerical representations for the same word depending on its meaning in context.

For example, the word lie could refer to a resting position or to saying something untruthful. During training, the model would learn the distinction between these two meanings and assign them different token numbers.

For visual AI models that process images, video or sensor data, a tokenizer can help map visual inputs like pixels or voxels into a series of discrete tokens.

Models that process audio may turn short clips into spectrograms - visual depictions of sound waves over time that can then be processed as images. Other audio applications may instead focus on capturing the meaning of a sound clip containing speech, and use another kind of tokenizer that captures semantic tokens, which represent language or context data instead of simply acoustic information.

How Are Tokens Used During AI Training? Training an AI model starts with the tokenization of the training dataset.

Based on the size of the training data, the number of tokens can number in the billions or trillions - and, per the pretraining scaling law, the more tokens used for training, the better the quality of the AI model.

As an AI model is pretrained, it's tested by being shown a sample set of tokens and asked to predict the next token. Based on whether or not its prediction is correct, the model updates itself to improve its next guess. This process is repeated until the model learns from its mistakes and reaches a target level of accuracy, known as model convergence.

After pretraining, models are further improved by post-training, where they continue to learn on a subset of tokens relevant to the use case where they'll be deployed. These could be tokens with domain-specific information for an application in law, medicine or business - or tokens that help tailor the model to a specific task, like reasoning, chat or translation. The goal is a model that generates the right tokens to deliver a correct response based on a user's query - a skill better known as inference.

How Are Tokens Used During AI Inference and Reasoning? During inference, an AI receives a prompt - which, depending on the model, may be text, image, audio clip, video, sensor data or even gene sequence - that it translates into a series of tokens. The model processes these input tokens, generates its response as tokens and then translates it to the user's expected format.

Input and output languages can be different, such as in a model that translates English to Japanese, or one that converts text prompts into images.

To understand a complete prompt, AI models must be able to process multiple tokens at once. Many models have a specified limit, referred to as a context window - and different use cases require different context window sizes.

A model that can process a few thousand tokens at once might be able to process a single high-resolution image or a few pages of text. With a context length of tens of thousands of tokens, another model might be able to summarize a whole novel or an hourlong podcast episode. Some models even provide context lengths of a million or more tokens, allowing users to input massive data sources for the AI to analyze.

Reasoning AI models, the latest advancement in LLMs, can tackle more complex queries by treating tokens differently than before. Here, in addition to input and output tokens, the model generates a host of reasoning tokens over minutes or hours as it thinks about how to solve a given problem.

These
LINK: https://blogs.nvidia.com/blog/ai-tokens-explained/...
See more stories from nvidia

North America Stories

17/03/2025

Summit: Scripps Sports Carves Out its Unique Niche in Sports TV

In the final keynote of the 2025 TV Tech Tech Leadership Summit, March 13, TV Tech Content Director Tom Butts spoke with Brian Lawlor, president of Scripps Spor...

17/03/2025

Comcast Expands Multiview to Millions of Customers for March Madness

PHILADELPHIA In the run-up to the start of the men's and women's NCAA basketball tournament, Comcast has expanded Xfinity Multiview to more X1 devices, ...

17/03/2025

New Data on Political Advertising Shows Growing Importance of CTV Ads

WASHINGTON As political strategists prepare for the 2026 and 2028 U.S. elections, a new study highlights some of the trends, particularly the growth of CTV poli...

17/03/2025

Digital Alert Systems 2025 NAB Show Preview

At the 2025 NAB Show, Digital Alert Systems will offer a presentation on virtualized emergency alerting at the Broadcast Engineering and IT Conference. On the s...

17/03/2025

Triveni Digital Introduces Next Evolution of StreamScope...

Triveni Digital, a trusted leader in ATSC 1.0 and 3.0 service delivery, data broadcasting, and quality assurance solutions today announced the StreamScope MT-7...

17/03/2025

SDVI to Showcase Platform Enhancements and New Technology...

At the 2025 NAB Show, SDVI will demonstrate how its Rally media supply chain management platform is giving media operations teams the ability to meet the often ...

17/03/2025

Riedel Communications at the 2025 NAB Show

At the 2025 NAB Show, Riedel Communications will showcase its latest innovations in audio, networking, and control including its next-generation StageLink Ed...

17/03/2025

Projective Demonstrates Hybrid Post-Production Innovation...

Projective, a software provider that transforms post-production workflows for broadcasters, creative agencies, post-production facilities and content creators o...

17/03/2025

arkona Announces Smaller Smarter Simpler Routing and Proc...

With a mission of providing smaller, smarter and simpler IP core infrastructure solutions for broadcast productions of all sizes, arkona technologies GmbH is br...

17/03/2025

Shotoku Broadcast Systems to Bring Best of All Worlds Sol...

Shotoku USA, Shotoku Broadcast Systems' North American operation, is bringing its award-wining SoftRail software defined rail system to NAB 2025 signifying ...

17/03/2025

FOR-A Theme Connecting the Present Building the Future Co...

Company Leads the Way with New Software-Based Production Platform, 12G Switcher, Software-based Switcher, Enhanced LED Displays, Cloud & AI Technology FOR-A wi...

17/03/2025

Ikegami to Demonstrate Latest Advances in Broadcast Telev...

Ikegami USA (www.ikegami.com) will exhibit the latest additions to its range of broadcast quality production equipment at the April 6th-9th 2025 NAB Show in Las...

17/03/2025

Miri Technologies to Make NAB Show Debut with X510 Dual-C...

Miri Technologies Inc. is ramping up for its NAB Show debut in Las Vegas, where the network technology startup will have its own booth for the first time. Exhib...

17/03/2025

Mediaproxy Unveils Groundbreaking Solutions at NAB Show 2...

Mediaproxy, the global standard for software-based IP compliance monitoring and multiviewing solutions, is excited to announce the launch of several groundbreak...

17/03/2025

BPMI Presents 2025 Festival Artists at 23rd Annual Showcase

BPMI Presents 2025 Festival Artists at 23rd Annual Showcase The Berklee Popular Music Institute will showcase eight artists performing at this summer's ho...

17/03/2025

NBC Sports Group Technical Summit Brings Decision-Makers Together To Preview Latest Production Tech

NBC Sports Group Technical Summit Brings Decision-Makers Together To Preview Lat...

17/03/2025

Major Music Publisher Sues USC Over Copyright Infringement

Major Music Publisher Sues USC Over Copyright Infringement Sports and growth of NIL drive use of music on college athletes' social media By Dan Daley, Audi...

17/03/2025

LVP Scores With Top-Flight Football Esports Production for LaLiga and EA Sports

LVP scores with top-flight football esports production for LaLiga and EA Sports By Michael Burns Friday, March 14, 2025 - 10:06 Print This Story All photo...

17/03/2025

An Inside Look as TGL Production Efforts Shift Into Playoff Mode

An Inside Look as TGL Production Efforts Shift Into Playoff Mode Mixing the virtual and physical, the young golf league has evolved in its first season By Ken ...

17/03/2025

Cloud Questions: What is the Status of Media Creation in the Cloud in 2025?

Cloud Questions: What is the Status of Media Creation in the Cloud in 2025? Leaders from Microsoft Production Studio, CiscoTV, MovieLabs, LucidLink, and Chess.c...

17/03/2025

Explaining Tokens - the Language and Currency of AI

Under the hood of every AI application are algorithms that churn through data in their own language, one based on a vocabulary of tokens. Tokens are tiny units...

15/03/2025

'The Last Of Us' Season 2 Trailer Hits Record 158 Million Views

Warner Bros. Discovery is reporting that the trailer for The Last of Us season two amassed 158 million global views across platforms in its first three days, ...

15/03/2025

Scripps Sports to Air Las Vegas Aces Games on Vegas 34

LAS VEGAS & CINCINNATI The Las Vegas Aces and Scripps Sports have signed a multiyear agreement to make Scripps Vegas 34 station the official broadcast home of W...

15/03/2025

Cobalt Digital To Feature New IPMX Support, IP Multiview At 2025 NAB Show

CHAMPAIGNE, Ill. Cobalt Digital will feature IPMX-compliant products, including its SAPPHIRE mini converters and PACIFIC encoders enhanced with IPMX support, du...

15/03/2025

Sharp Smart TVs Powered by TiVo Launch in the U.S.

SAN JOSE, Calif. TiVo has announced that the latest release of Sharp Smart TVs Powered by TiVo are now available for purchase in the U.S....

15/03/2025

EditShare to Unveil Next-Generation Media Solutions at 2025 NAB Show

BOSTON Collaborative media workflow solution provider EditShare has announced that it will be showing a wide range of new innovations at the 2025 NAB Show that ...

15/03/2025

Editing and Color Grading Promo Codes for Audiio and MotionVFX!

Editing and Color Grading Promo Codes for Audiio and MotionVFX! Kasia Jarco March 14, 2025 0 Comments I wanted to share some promo codes for great too...

15/03/2025

Avid Makes Key Additions to Executive Leadership Team to Meet the Needs of the Evolving Media Landscape

Avid Makes Key Additions to Executive Leadership Team to Meet the Needs of the E...

15/03/2025

NAB 2025: Cobalt Announces Plans to Bring Plug and Play ST 2110 Experience to NAB with Expanded Line of IPMX-Compliant Products

NAB 2025: Cobalt Announces Plans to Bring Plug and Play ST 2110 Experience to NA...

15/03/2025

Student Spotlight: Rachel McFarlane

Student Spotlight: Rachel McFarlane The Canadian composer talks about being drawn to the storytelling power of scoring for film and video games, working with ...

14/03/2025

Give Me the Backstory: Get to Know Emilie Blichfeldt, the Writer-Director of The Ugly Stepsister

By Lucy Spicer One of the most exciting things about the Sundance Film Festival...

14/03/2025

NAB Show 2025 Exhibitor Insight: Bitcentral

TV Technology: What do you anticipate will be the most significant technology trends at the 2025 NAB Show?...

14/03/2025

Dalet welcomes you to NAB Show 2025

The Dalet NAB 2025 showcase will feature its market-leading enterprise News, Media Management and Supply Chain solutions designed for faster deployment and acce...

14/03/2025

IMAX to Unveil Breakthrough Quality Innovation for Premiu...

IMAX Corporation (NYSE: IMAX) will debut its latest advancements for premium live streaming at NAB 2025 (booth W1401), introducing new AI-driven technologies d...

14/03/2025

TAG Video Systems and Harmonic Partner to Deliver Enhance...

TAG Video Systems, the leader in software-based IP media probing, monitoring, visualization, and analytics, and Harmonic (NASDAQ: HLIT), the worldwide leader in...

14/03/2025

Unicanal and Trece TV revolutionize Digital TV in Paragua...

Unicanal and Trece TV, two of Paraguay s leading broadcasters, have successfully launched digital TV services in three main cities: Encarnaci n, Ciudad del Este...

14/03/2025

IP Showcase Returns to 2025 NAB Show With a Focus on IPMX...

The Alliance for IP Media Solutions (AIMS), Advanced Media Workflow Association (AMWA), and the Video Services Forum (VSF) today announced that the IP Showcase ...

14/03/2025

Nebraska Public Media Standardizes on Digital Alert Syste...

Digital Alert Systems, the global leader in emergency communications solutions for media providers, today announced that Nebraska Public Media is deploying the ...

14/03/2025

Comcast Technology Solutions Achieves Enhanced QC Efficie...

Interra Systems, a leading provider of end-to-end quality assurance solutions for the digital media industry, today announced that Comcast Technology Solutions ...

14/03/2025

Rightsline Unveils New AI Features and Media Workflow Eff...

At the 2025 National Association of Broadcasters (NAB) Show, Rightsline will showcase innovations designed to make rights and royalties management smarter, fast...

14/03/2025

PK Sound Launches Tx Series Intelligent Point Source Modu...

PK Sound, the robotic line array company, has formally released the Tx Series of intelligent multi-purpose point source modules, beginning with the ultra-versat...

14/03/2025

Avid Makes Key Additions to Executive Leadership Team to...

Avid today announced the expansion of its executive leadership team (ELT) with the appointment of three experienced executives: Tom Sharma as Chief Technology ...

14/03/2025

DigitalGlue Invites NAB Visitors to Experience New Featur...

DigitalGlue, a trusted systems integrator and provider of end-to-end media solutions, is redefining storage workflows with its innovative platform, creative.spa...

14/03/2025

CVP Supports Innovative Large Format Short Films Competit...

CVP, one of Europe s leading resellers and providers of professional video and broadcast solutions, is partnering with renowned director and cinematographer Joe...

14/03/2025

Cobalt Announces Plans to Bring Plug and Play ST 2110 Exp...

Cobalt Digital, the leading designer and manufacturer of award-winning signal processing products, and a founding partner in the openGear initiative has announ...

14/03/2025

Keepit named Best Cybersecurity Backup Service by Busines...

Keepit, a global provider of a comprehensive cloud backup and recovery platform, announced today its success at the Business Awards UK, 2025 Cybersecurity and R...

14/03/2025

Lightware simplifies complex AV integrations through stra...

Lightware, an industry-leading manufacturer of signal management solutions, has announced its commitment to fostering strong relationships with top technology c...

14/03/2025

Editing and Color Grading Promo Codes

Editing and Color Grading Promo Codes Kasia Jarco March 14, 2025 0 Comments I wanted to share some promo codes for great tools for your video editing ...

14/03/2025

Singaporean singer-songwriter JJ Lin on Craft, Courage, and Creativity

Singaporean Singer-Songwriter JJ Lin on Craft, Courage, and Creativity The artist shared candid insights into his musical evolution, personal challenges, and ...

14/03/2025

Avid Appoints New CTO, Adds to Its Executive Leadership Team

BURLINGTON, Mass. Avid has expanded its executive leadership team (ELT) with the appointment of three experienced executives: Tom Sharma as chief technology off...