You type:
Explain why the sky is blue.
A moment later, the model starts answering one token at a time.
Behind that simple response, the GPU repeatedly loads model weights, performs billions of mathematical operations, checks previous tokens through attention, produces probabilities for the next token, and repeats the process.
Understanding this flow makes concepts such as VRAM, memory bandwidth, KV cache, context length, and tokens per second much easier to reason about.
One Token Goes Through This Loop
At a high level:
Your prompt ↓ Tokenization ↓ Prompt processed by the model ↓ KV cache created ↓ GPU calculates next-token probabilities ↓ One token is selected ↓ Token added to the sequence ↓ Repeat until generation stops
An autoregressive LLM predicts one new token from all the tokens it has already seen, then repeats the process.
Let's follow that process.
1. Your Prompt Becomes Tokens
The model does not directly process:
Why is the sky blue?
A tokenizer first converts the text into token IDs.
Conceptually:
"Why is the sky blue?" ↓ [10445, 374, 279, 13180, 6437, 30]
The exact tokens depend on the model's tokenizer.
These IDs are then mapped to numerical vectors called embeddings that the neural network can process.
This part is important because GPU requirements are affected by tokens, not simply by words.
A 10,000-word document and another 10,000-word document may produce different token counts.
If you are still choosing the model itself, comparing open-source LLMs and their GPU requirements before deployment can save you from sizing the infrastructure around the wrong model.
2. The GPU Processes the Entire Prompt First
The first pass through your prompt is called prefill.
Suppose your prompt contains 2,000 tokens.
The model processes those tokens through its transformer layers, performing operations such as:
- Matrix multiplications
- Attention calculations
- Normalization
- Feed-forward network operations
Modern NVIDIA GPUs use specialized Tensor Cores to accelerate many matrix operations performed with lower-precision data types.
This is one reason GPUs are so effective for transformer workloads.
Prefill is also why a very long prompt can delay the moment when the first generated token appears.
3. Attention Checks What Came Before
Inside each transformer layer, attention helps determine which previous tokens are relevant to the current computation.
Consider:
The developer restarted the server because it had crashed.
To interpret “it,” the model needs information from earlier tokens.
Attention calculates relationships between tokens using query, key, and value vectors.
Doing all of this from scratch for every generated token would waste enormous amounts of computation.
That is where the KV cache helps.
4. The GPU Stores a KV Cache
During the initial prompt processing, the model calculates key and value vectors for the tokens it has already seen.
Instead of recalculating them every time another token is generated, it stores them in the KV cache.
Hugging Face describes the cache as a way to reuse previously calculated key and value information during autoregressive generation.
Think of it as the model keeping useful working notes:
Prompt tokens ↓ Keys + Values calculated ↓ Stored in GPU memory ↓ Reused for future tokens
This makes generation much faster.
But there is a cost.
The KV cache consumes GPU memory and grows as the sequence gets longer.
That is why increasing:
- Context length
- Number of concurrent users
- Batch size
can dramatically increase VRAM consumption.
Systems such as PagedAttention divide KV-cache memory into blocks to use GPU memory more efficiently.
5. The Model Produces Scores for Possible Next Tokens
After information moves through all transformer layers, the model produces a set of numbers called logits.
Conceptually, it might look like:
"The" 0.08 "Sky" 0.02 "Because" 0.54 "Light" 0.18 "Blue" 0.07 ...
The real vocabulary may contain tens of thousands or more possible tokens.
The logits are transformed into probabilities, and the decoding strategy decides what comes next.
Different strategies include:
- Greedy decoding
- Sampling
- Top-k
- Top-p
- Temperature-based sampling
Changing the decoding strategy can change the generated text even though the underlying model and GPU remain the same.
Eventually, one token is selected.
Perhaps:
Because
6. Then the GPU Does It Again
Now the sequence becomes:
Why is the sky blue? Because
The new token passes through the transformer.
Its new key and value information is added to the KV cache.
The model calculates another probability distribution.
Another token is selected.
Because → sunlight
Then:
Because sunlight → contains
And again:
Because sunlight contains → many
This continues until the model:
- Generates an end-of-sequence token
- Reaches the output limit
- Encounters another configured stopping condition
LLM text generation is therefore not one giant calculation.
It is a repeated predict one token → append it → predict another token loop.
Why GPU Memory Bandwidth Matters So Much
Generating one token requires the GPU to repeatedly access model weights and data stored in memory.
For many LLM inference workloads, particularly token-by-token decoding at relatively small batch sizes, moving this data quickly can become as important as raw computation.
This is why two GPUs with enough VRAM to hold the same model can still generate tokens at very different speeds.
For example, an L4 vs L40S comparison is not simply a 24GB-versus-48GB memory decision. Memory bandwidth and compute capability also affect how quickly inference runs.
Similarly, larger data-center GPUs such as the NVIDIA H100 cloud GPU become more relevant when higher throughput or demanding workloads justify their additional compute and memory capabilities.
Why More VRAM Does Not Automatically Mean Faster Tokens
VRAM answers:
Can my workload fit?
It does not fully answer:
How fast will it run?
Inference speed can also depend on:
- Memory bandwidth
- Model architecture
- Precision
- Quantization
- Batch size
- Serving engine
- Context length
- GPU utilization
- Concurrency
That is why selecting a GPU only by memory capacity can be misleading.
If several GPU generations are being considered, this broader L4 vs L40S vs A100 vs H100 vs H200 comparison is useful for seeing how memory capacity and architecture change across GPU classes.
Why the First Token Often Takes Longer
You may notice that an LLM pauses briefly and then starts generating text quickly.
Those are two different phases.
Prefill
The complete input prompt must first be processed.
Longer input generally means more work before generation begins.
This contributes to time to first token, or TTFT.
Decode
After prefill, the model generates one new token at a time while reusing the KV cache.
This determines much of the visible tokens-per-second generation speed.
So if an application has a problem, identify which one:
Slow before text starts? Investigate prefill and TTFT.
Text starts quickly but crawls afterward? Investigate decode performance.
That distinction is much more useful than simply saying “the GPU is slow.”
The Whole Process in One Picture
PROMPT "Why is the sky blue?" │ ▼ Tokenizer │ ▼ Token embeddings │ ▼ ┌─────────────────────────┐ │ Transformer GPU │ │ │ │ Attention │ │ ↓ │ │ KV Cache ← previous │ │ ↓ tokens │ │ Feed-forward layers │ │ ↓ │ │ Next-token logits │ └─────────────────────────┘ │ ▼ Sampling │ ▼ "Because" │ └─────────────┐ │ Feed back in │ ▼ Next token
That loop may run dozens or hundreds of times for a single answer.
What This Means When Choosing a GPU
Instead of asking only:
How many parameters does my model have?
Ask:
- Will the model weights fit?
- How much space will the KV cache need?
- What context length will users actually use?
- How many requests will run concurrently?
- What time to first token is acceptable?
- What generation speed do users need?
That is the difference between choosing a GPU that merely loads the model and choosing one that can actually serve the application well.
