When you type a question into an AI chatbot and receive an answer a few seconds later, a large amount of computation happens behind the scenes.

You type:

Explain how a CPU works.

The AI doesn’t simply look up a stored answer and send it back.

A trained model processes your input, converts it into tokens, runs those tokens through its neural-network layers, calculates probabilities for possible next tokens, selects a token, and then repeats the process until the response is complete.

This process is called AI inference.

Inference is the stage where a trained AI model is used to produce an output from new input. NVIDIA describes it as the application of a trained model to real-world data, while for generative AI it includes the process of generating tokens for the response. (nvidia.com)

Training and inference are therefore two very different phases of an AI system.

Training
   ↓
Learn model parameters
   ↓
Trained Model
   ↓
Inference
   ↓
Generate predictions / responses

For generative AI, inference is the part you experience every time you send a prompt.

---

What Is AI Inference?

At its simplest:

Inference is using an already-trained model to make a prediction or generate an output.

For a traditional machine-learning model:

Image
  ↓
Trained Model
  ↓
Cat

For a large language model:

Prompt
  ↓
Trained Model
  ↓
Generated Tokens
  ↓
Response

The important distinction is that the model is normally not learning new parameters during an ordinary inference request.

The weights that were learned during training are being used to perform computation.

---

Training vs Inference

It is useful to separate the two.

Training

During training, a model processes enormous amounts of data and adjusts its parameters to improve its predictions.

A simplified representation is:

Training Data
     ↓
Model
     ↓
Prediction
     ↓
Compare With Target
     ↓
Calculate Error
     ↓
Update Weights
     ↓
Repeat

This process can involve enormous amounts of computation.

Inference

During inference, the trained model is used to generate an output.

User Input
    ↓
Trained Model
    ↓
Prediction
    ↓
Output

The model’s parameters aren’t normally being updated because you asked a question.

The inference process is instead using the learned parameters to calculate an output.

---

How Does an AI Model Generate a Sentence?

This is where things get interesting.

Suppose you ask:

The capital of France is

The model doesn’t necessarily retrieve the complete sentence from a database.

It calculates probabilities for possible next tokens.

Conceptually:

The capital of France is

Paris      → high probability
London     → low probability
Berlin     → low probability
Tokyo      → very low probability

The system then chooses a token according to the generation strategy.

If it chooses:

Paris

the sequence becomes:

The capital of France is Paris

The model then predicts what should come next.

This process is called autoregressive generation.

Hugging Face’s documentation describes LLM generation as repeatedly predicting the next token using the prompt plus tokens that the model has already generated. (huggingface.co)

---

The Response Is Generated Token by Token

This is one of the most important concepts in understanding inference.

Suppose the model wants to generate:

Paris is the capital of France.

Conceptually, the process can look like:

Input
 ↓
"Paris"
 ↓
"is"
 ↓
"the"
 ↓
"capital"
 ↓
"of"
 ↓
"France"
 ↓
"."

Each generated token becomes part of the context for the next prediction.

So the model repeatedly performs:

Context
   ↓
Model
   ↓
Next-token probabilities
   ↓
Select token
   ↓
Add token to context
   ↓
Repeat

This is why generating a long response requires many generation steps.

---

What Exactly Is a Token?

A token is a unit of text processed by the model.

A token might represent:

• a complete word
• part of a word
• punctuation
• whitespace patterns
• symbols
• code fragments

For example, the sentence:

Artificial intelligence is changing software.

might be represented internally as a sequence of tokens rather than exactly five word units.

The exact tokenization depends on the model and tokenizer.

NVIDIA describes tokens as units of data processed by AI models during training and inference, while Hugging Face’s generation documentation explains that language models operate on token sequences rather than ordinary words. (nvidia.com)

---

Tokenization Happens Before the Model Starts Generating

When you send:

How does a GPU work?

the application first processes the input into tokens.

Conceptually:

Text
 ↓
Tokenizer
 ↓
Token IDs
 ↓
Model

The model doesn’t receive the original sentence in the same human-readable form you see.

It receives numerical representations associated with those tokens.

---

From Token IDs to Embeddings

The token IDs are then mapped into numerical vectors.

For example:

Token ID
   ↓
Embedding
   ↓
Vector

A simplified representation might look like:

[0.21, -0.14, 0.73, 0.08, ...]

Real model representations contain many dimensions.

These numerical representations are then processed by the model’s neural-network layers.

---

The Transformer Processes the Context

Modern language models are largely built around transformer architectures or architectures derived from them.

The transformer processes the input through multiple layers.

A simplified representation is:

Input Tokens
     ↓
Embeddings
     ↓
Transformer Layer
     ↓
Transformer Layer
     ↓
Transformer Layer
     ↓
...
     ↓
Output Representation

Each layer performs mathematical operations that transform the representations.

One important mechanism is attention.

---

What Does Attention Do?

Attention allows the model to calculate relationships between different tokens in the context.

Consider:

The developer deployed the application
because it was ready.

The model needs to process relationships between words and their surrounding context.

In programming code, the relationships can become even more important.

For example:

const user = getUser();

console.log(user.email);

The model can use the surrounding context to associate:

user

with:

user.email

Attention is one of the mechanisms that allows transformer models to process these relationships.

---

What Comes Out of the Model?

After processing the input through its layers, the model produces numerical outputs that can be converted into scores for possible next tokens.

Suppose the vocabulary contains thousands or millions of possible tokens.

The model can produce something conceptually like:

Token       Score
-------------------
"the"       2.81
"Paris"     6.92
"London"    1.73
"capital"   3.41
"."         0.92

These values are often referred to as logits before being converted into probabilities.

The model doesn’t simply say:

“The answer is Paris.”

It produces a distribution over possible next tokens.

---

From Logits to Probabilities

A mathematical operation such as softmax can convert logits into a probability distribution.

Conceptually:

Logits
  ↓
Softmax
  ↓
Probabilities
  ↓
Token Selection

For example:

Paris       0.82
London      0.05
Berlin      0.03
Madrid      0.02
Other       0.08

These numbers are illustrative rather than actual model probabilities.

The model then uses a decoding strategy to determine which token should be selected.

---

Does the Model Always Pick the Highest-Probability Token?

No.

This is an important part of text generation.

The system can use different decoding strategies.

One simple approach is greedy decoding:

Choose the highest-probability token.

But generative AI systems often use sampling strategies.

These can include:

• Temperature
• Top-k
• Top-p
• Repetition penalties
• Other decoding controls

Hugging Face’s inference documentation exposes parameters such as temperature, top-p/top-k-style controls, repetition penalties, and sampling options for text generation. (huggingface.co)

---

What Does Temperature Do?

Temperature affects how the probability distribution is used during sampling.

A simplified intuition is:

Lower temperature
      ↓
More predictable output

and:

Higher temperature
      ↓
More variation

For example, if the model has:

A → 0.70
B → 0.20
C → 0.10

a lower-temperature configuration can make the dominant choice more strongly favored.

A higher temperature can make less-probable choices more likely.

Temperature does not magically make the model “more intelligent” or “less intelligent.”

It changes the behavior of token selection.

---

What Is Top-p Sampling?

Top-p sampling limits the candidate tokens considered for selection to a probability mass chosen by the configured value.

Conceptually:

All possible tokens
        ↓
Sort by probability
        ↓
Keep enough tokens to reach p
        ↓
Sample from those candidates

This prevents extremely unlikely tokens from being considered while still allowing some variation.

Different models and applications use different decoding strategies.

---

Why Does the Same Prompt Sometimes Produce Different Answers?

Because generation does not necessarily have to be deterministic.

Suppose the model sees:

Explain JavaScript closures.

There can be multiple reasonable ways to answer.

If the generation configuration allows sampling, the model may choose different token paths.

That can produce:

Response A

on one request and:

Response B

on another.

Both can be valid.

Deterministic settings can reduce variation, but exact behavior depends on the model and serving system.

---

The Two Major Stages of LLM Inference

For modern LLM serving, inference is commonly discussed in two major phases:

Prompt
  ↓
PREFILL
  ↓
DECODE
  ↓
Response

These two stages behave differently.

---

Stage 1: Prefill

The prefill stage processes the input prompt.

Suppose your request contains:

You are a software engineer...

[large system instructions]

[conversation history] Explain how databases use indexes.

The model needs to process all of that input before it can begin generating the answer.

That processing is called prefill.

Conceptually:

Large Prompt
     ↓
Tokenizer
     ↓
Model
     ↓
Prompt Representations

NVIDIA’s description of LLM inference separates prompt processing into the prefill phase and response generation into the decode phase. (nvidia.com)

---

Stage 2: Decode

After the prompt has been processed, the model begins generating output.

Prompt
  ↓
Prefill
  ↓
Token 1
  ↓
Token 2
  ↓
Token 3
  ↓
Token 4
  ↓
...

Each generated token depends on the previous context.

This repeated generation is the decode stage.

---

Why Does the First Token Take Longer?

This explains an important user experience detail.

You send a request and see:

Thinking...

Then suddenly:

The

appears.

The time before the first token is called Time to First Token, or TTFT.

It includes work such as request processing and prompt prefill.

After that, the system generates additional tokens.

NVIDIA identifies TTFT and time per output token as important metrics for understanding LLM inference performance. (nvidia.com)

---

What Is Time Per Output Token?

After the first token appears, the model needs to generate the remaining tokens.

Suppose the response is:

500 tokens

The system needs to repeatedly generate those tokens.

A common performance metric is time per output token (TPOT).

Conceptually:

First token
   ↓
Token 2
   ↓
Token 3
   ↓
Token 4
   ↓
...

Lower time per token generally means the response can be produced faster.

So two different performance problems can exist:

Slow first token

or:

Fast first token
+
Slow generation afterward

Optimizing one doesn’t automatically solve the other.

---

What Is KV Cache?

Generating a long response repeatedly processing the entire previous context from scratch would be extremely inefficient.

Transformers therefore use a mechanism commonly called the KV cache during autoregressive generation.

KV stands for:

Key
Value

During attention calculations, information from previously processed tokens can be cached and reused during subsequent generation steps.

Conceptually:

Previous Context
      ↓
Key / Value Cache
      ↓
New Token
      ↓
Attention
      ↓
Next Token

This avoids recomputing certain information for every newly generated token.

KV caching is one of the major reasons efficient LLM inference systems can generate long sequences without completely repeating the same computation every time.

---

Why Does AI Inference Need GPUs?

Large AI models involve enormous numbers of mathematical operations.

GPUs are well suited to many of these operations because they can perform large numbers of calculations in parallel.

A simplified architecture looks like:

User Request
      ↓
Inference Server
      ↓
GPU
      ↓
Model Weights
      ↓
Token Generation
      ↓
Response

This doesn’t mean every AI inference workload must use a GPU.

Depending on the model and workload, inference can run on CPUs, GPUs, specialized accelerators, or other hardware.

But large generative AI workloads commonly rely heavily on accelerated computing.

NVIDIA’s inference documentation describes GPUs and specialized inference infrastructure as important for serving large generative models efficiently. (nvidia.com)

---

Why AI Inference Uses So Much Memory

A model’s parameters have to be stored somewhere while it runs.

Consider a simplified example.

If a model has:

70 billion parameters

and each parameter uses 2 bytes, the raw parameter storage alone would be approximately:

70 billion × 2 bytes
= 140 billion bytes
≈ 140 GB

That is only a simplified illustration.

Real inference memory requirements can also include:

• model weights
• KV cache
• activations
• temporary buffers
• runtime overhead
• batching-related memory

This is why large models often require multiple GPUs or specialized hardware configurations.

---

What Is Quantization?

One way to reduce inference memory requirements is quantization.

Instead of representing model weights using a higher-precision numerical format, they can sometimes be represented using lower-precision formats.

Conceptually:

Higher precision
      ↓
More memory
      ↓
Quantization
      ↓
Lower precision
      ↓
Less memory

For example, a model may be quantized to formats such as:

FP16
INT8
INT4

depending on the model and serving stack.

The tradeoff is that lower precision can affect accuracy or output quality, depending on the model and quantization method.

Inference frameworks such as Hugging Face’s tooling support various quantization approaches intended to reduce memory requirements and improve serving efficiency. (huggingface.co)

---

What Is Batching?

Imagine one user asks:

What is Python?

and another asks:

Explain neural networks.

If the server processes every request completely independently, hardware may not be used efficiently.

Instead, inference servers can combine requests into batches.

Request A ──┐
Request B ──┤
Request C ──┼──> GPU
Request D ──┘

Modern inference systems can use continuous batching, allowing new requests to join ongoing workloads rather than waiting for a traditional fixed batch to finish.

Hugging Face’s inference tooling lists continuous batching as one of the techniques used to improve total throughput. (huggingface.co)

---

What Is Streaming?

When you use an AI chatbot, you may see the answer appear gradually:

AI
is
generating
the
response...

The system doesn’t necessarily wait for the complete response before sending it to the client.

Instead, generated tokens can be streamed.

Model
 ↓
Token
 ↓
Client

Model
 ↓
Next Token
 ↓
Client

This makes the application feel much faster even if the total generation time hasn’t changed.

Streaming therefore improves perceived responsiveness.

Inference APIs can expose generated text token by token rather than returning the entire response at the end. Hugging Face’s inference client, for example, supports streaming generated text. (huggingface.co)

---

Why Longer Prompts Cost More

A prompt isn’t free from a computational perspective.

Suppose one request contains:

100 tokens

and another contains:

100,000 tokens

The second request requires dramatically more input processing.

The model must process that context during inference.

This is why AI applications often try to avoid sending unnecessary information.

A system that blindly sends:

Entire conversation
+
Entire repository
+
Entire documentation

for every request can become expensive and slow.

This is also why techniques such as retrieval, context selection, summarization, and caching are important.

---

Why AI Inference Gets Expensive at Scale

One user asking a question is manageable.

Now imagine:

1 user
↓
10 users
↓
1,000 users
↓
100,000 users
↓
Millions of requests

Every request consumes compute.

And generative models can produce many tokens per request.

The infrastructure therefore needs to optimize:

Latency
+
Throughput
+
Memory
+
Power
+
GPU utilization
+
Cost per token

This is why AI inference has become a major infrastructure problem.

---

Cost Per Token

AI APIs commonly measure usage in tokens.

There are usually two broad categories:

Input Tokens
+
Output Tokens

The exact pricing model varies between providers.

From an infrastructure perspective, however, token generation also represents real computational work.

A longer response generally requires more decode steps.

NVIDIA describes cost per token and token-processing performance as central considerations for AI inference infrastructure. (nvidia.com)

---

What Is Speculative Decoding?

One interesting inference optimization is speculative decoding.

The basic idea is to use a smaller or faster model to propose several tokens and then have the larger model verify those predictions.

Conceptually:

Small / Draft Model
       ↓
Token 1
Token 2
Token 3
Token 4
       ↓
Large Model verifies
       ↓
Accept / Reject

If many proposed tokens are correct, the larger model can make progress more efficiently.

Hugging Face describes speculative decoding as generating token candidates before the larger model verifies them, potentially reducing generation time when the predictions are accurate enough. (huggingface.co)

This is one example of how inference optimization can happen without fundamentally changing what the user sees.

---

Inference for Reasoning Models Can Be Different

Some modern AI models use additional computation during inference to solve more complex tasks.

Instead of:

Prompt
 ↓
Answer

the system may use additional internal generation or multiple inference steps.

Conceptually:

Prompt
 ↓
Reasoning / intermediate computation
 ↓
Additional inference
 ↓
Answer

NVIDIA describes this as test-time scaling, where reasoning models can use additional inference computation and generate more tokens to work through complex problems. (nvidia.com)

The important point is that inference isn’t necessarily a single simple forward pass followed by one output.

For some AI systems, more computation at inference time is itself part of the strategy for improving results.

---

What Happens When AI Calls a Tool?

Modern AI applications can also combine inference with external tools.

Suppose you ask:

What is the weather today?

A tool-using AI system might do:

User Question
     ↓
Model Inference
     ↓
Decide: Need weather tool
     ↓
Tool Call
     ↓
Weather Result
     ↓
Model Inference Again
     ↓
Final Answer

Now one user request can involve multiple inference passes.

This becomes especially important with AI agents.

An agent may:

Think / plan
 ↓
Search
 ↓
Read result
 ↓
Reason
 ↓
Call another tool
 ↓
Inspect output
 ↓
Generate final answer

So the computational cost of an agent can be much higher than a simple chatbot response.

---

Why Inference Optimization Matters

Imagine an AI application that generates a response in:

10 seconds

For one request, that might be acceptable.

But at large scale, the system must handle many requests simultaneously.

Engineers therefore optimize both hardware and software.

Common techniques include:

• Quantization
• KV caching
• Continuous batching
• Tensor parallelism
• Model parallelism
• Speculative decoding
• Efficient attention implementations
• Caching
• Request scheduling
• Hardware acceleration

Production inference frameworks are specifically designed around these challenges. Hugging Face’s inference tooling, for example, includes techniques such as tensor parallelism, continuous batching, quantization, and optimized attention implementations. (huggingface.co)

---

AI Inference Is More Than “Running the Model”

It is tempting to think of inference as:

Prompt
 ↓
Model
 ↓
Answer

But a production AI system looks more like:

User Request
      ↓
API Gateway
      ↓
Authentication
      ↓
Request Processing
      ↓
Tokenization
      ↓
Queue / Scheduler
      ↓
Prefill
      ↓
Decode
      ↓
Sampling
      ↓
Detokenization
      ↓
Streaming
      ↓
User

And behind the model server there can be:

GPU Cluster
+
Model Weights
+
KV Cache
+
Batching
+
Memory Management
+
Monitoring

This is why serving a large AI model in production is an engineering problem of its own.

---

Inference vs API Call

There is also an important distinction between an API request and inference.

When your application calls an AI API:

const response = await ai.generate({
    prompt: "Explain databases"
});

your application is making an API request.

Behind that API request, the provider performs inference.

Your Application
      ↓
API Request
      ↓
Provider Infrastructure
      ↓
Inference
      ↓
Generated Tokens
      ↓
API Response
      ↓
Your Application

So the API is the interface.

Inference is the computation happening behind it.

---

Why AI Responses Feel Instant Even Though They Require Huge Computation

Modern AI infrastructure is highly optimized.

When you see:

Hello! Here's how it works...

appearing one piece at a time, the system may already be performing thousands or millions of mathematical operations behind the scenes.

The experience feels simple because the infrastructure hides the complexity.

At the other end are data centers containing:

AI Accelerators
      ↓
High-speed Networking
      ↓
Inference Servers
      ↓
Model Weights
      ↓
Schedulers
      ↓
Millions of Requests

The AI response shown in a browser is therefore the final visible result of a very large computational pipeline.

---

The Complete AI Inference Pipeline

We can now put the entire process together.

Suppose you ask:

Explain how a GPU works.

A simplified inference pipeline looks like:

User Prompt
     ↓
Tokenization
     ↓
Token IDs
     ↓
Embedding
     ↓
Transformer Layers
     ↓
Logits
     ↓
Sampling / Decoding
     ↓
First Token
     ↓
KV Cache
     ↓
Next Token
     ↓
Repeat
     ↓
End-of-Sequence
     ↓
Detokenization
     ↓
Final Response

In a production environment, there are additional layers:

User
 ↓
API
 ↓
Queue / Scheduler
 ↓
Batching
 ↓
GPU / Accelerator
 ↓
Prefill
 ↓
Decode
 ↓
Streaming
 ↓
User

That is the basic machinery behind generative AI responses.

---

Final Thoughts

AI inference is the part of artificial intelligence that turns a trained model into a working application.

Training creates the model.

Inference uses that model.

For a large language model, inference involves much more than simply asking a question and receiving an answer.

The system needs to:

Process the prompt
       ↓
Convert text into tokens
       ↓
Run the model
       ↓
Calculate next-token probabilities
       ↓
Select a token
       ↓
Cache previous information
       ↓
Generate the next token
       ↓
Repeat
       ↓
Return the response

At production scale, another set of problems appears:

How fast can we generate?
How many users can we serve?
How much GPU memory is required?
How much does each token cost?
How can we reduce latency?
How can we increase throughput?

These questions are becoming just as important as model quality itself.

The next time an AI chatbot starts writing an answer token by token, remember what is happening underneath:

The model isn’t retrieving a finished paragraph from somewhere. It is repeatedly performing inference, calculating what should come next, selecting a token, and using that new token as part of the context for the next step.

That repeated process—combined with increasingly sophisticated hardware and inference software—is what turns a trained AI model into an interactive system.