I am sure when you are training your AI model you have seen the “out-of-memory” error at least once. But here is one thing that is kept on hanging while this error. It tells you exactly what happened: ‘Your GPU hit its memory limit’, but what's left hanging is “WHY?”, “WHERE?”, “WHAT?”.
It fires on the allocation that broke the camel's back, not on the one that caused the problem. So the line number in the traceback is rarely the real culprit. From a business point of view as well, knowing all the answers is really important for you.
Keep reading and you will know it. I have covered everything here in a quick to the point manner.
What does an out-of-memory error actually mean during training?
It means the training job asked for more memory than the device had free at that moment. Most of the time that device is the GPU, and the memory is VRAM. But not always, and that difference matters more than people expect, which I will come back to below.
The error appears at the point where memory peaks. A job can run fine for hours and then crash on one unusually long batch.
Why does training need so much more memory than the model's size?
Because the model's weights are only one of four things being stored. During training the GPU also holds gradients, optimizer states and activations.
Here is the usual accounting for mixed-precision training with the Adam optimizer:
| What is stored | Rough size per parameter | What it scales with |
| Weights (fp16 or bf16) | 2 bytes | Model size |
| Gradients | 2 bytes | Number of trainable parameters |
| fp32 master copy of weights | 4 bytes | Model size |
| Adam momentum and variance | 8 bytes | Number of trainable parameters |
| Activations | Varies a lot | Batch size, sequence length, model depth |
That adds up to roughly 16 bytes per parameter before a single activation is counted. Some frameworks count a little differently and land closer to 18, so treat 16 as a floor.
Can a 7B model be fully fine-tuned on one 80GB GPU?
Not with plain mixed-precision Adam. Seven billion parameters at 16 bytes each is about 112GB, and that is before activations. A smaller 1B model already needs around 16GB just to exist in the training loop.
That is why full fine-tuning at this size needs sharding across GPUs, an 8-bit optimizer, or a method like LoRA that trains far fewer parameters.
Now that we cleared up some foundation, let's get to the main agenda here. The causes for the memory error so you can avoid it happening in your next AI model training.
Batch size or sequence length too big
This is the most common cause, and it hits activations. Activations are the intermediate values the model saves during the forward pass so it can compute gradients later.
- Double your batch size and you roughly double the activations
- Longer sequences make attention memory grow much faster, because standard attention scales with the square of the sequence length
- Deeper and wider models store more of them at every layer
The fixes are a smaller micro-batch with gradient accumulation to keep your effective batch size, gradient checkpointing, and an efficient attention implementation such as FlashAttention. Gradient checkpointing recomputes activations instead of storing them, and it typically makes training about 20 to 30 percent slower.
Optimizer eating your memory
Adam keeps two extra values for every trainable parameter, and they sit in higher precision than the model itself. For many training runs, optimizer states are the largest single block of memory after the weights.
An 8-bit optimizer cuts this sharply for a small precision cost. LoRA helps too, since fewer trainable parameters means fewer gradients and fewer optimizer states.
There is one catch worth knowing. LoRA and quantization shrink the parts that scale with parameters. They do nothing for activations. If you are still out of memory after switching to LoRA, the batch size or sequence length is probably your real problem.
Accidentally holding on to old tensors
A leak looks like memory that climbs a little every step until the job dies hours in. The classic cause is keeping tensors that still carry their computation graph.
- Adding loss to a running total instead of loss.item() keeps every step's graph alive
- Storing predictions for later without detaching them has the same effect
- Running evaluation without turning off gradient tracking uses far more memory than it needs to
If your memory use grows steadily over time instead of jumping, look for this first.
Fragmentation, or another process
Sometimes the GPU has free memory, just not in one piece large enough for the allocation. A good sign is when reserved memory is far above allocated memory. In PyTorch, setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True often helps.
Also run nvidia-smi. A forgotten notebook kernel or an old process can quietly hold gigabytes that your new job needs.
RAM or VRAM that ran out
This is the ram vs vram mix-up, and it trips up many people. RAM is the computer's main memory, and VRAM is the GPU's own memory. They run out in different ways and need different fixes.
| What you see | Which memory | Usual cause |
| CUDA out of memory. Tried to allocate... | VRAM | Batch size, activations, model states |
| Process just says Killed, with no traceback | System RAM | The operating system's out-of-memory killer stepped in |
| MemoryError in Python | System RAM | A dataset loaded fully into memory |
On the system RAM side, the usual suspects are loading an entire dataset into memory at once and running many data loader workers that each hold their own copy of the data. Lower the worker count, stream the data from disk, and check memory with free -h or your system monitor.
How do you fix an out-of-memory error, in the right order?
Start with the cheap changes and move down the list only if you need to.
| Fix | What it shrinks | Trade-off |
| Smaller micro-batch plus gradient accumulation | Activations | More steps, slightly slower |
| Mixed precision (bf16) | Weights, gradients and activations | Needs a little numerical care |
| Gradient checkpointing | Activations | Roughly 20 to 30 percent slower |
| Shorter sequences or efficient attention | Activations | May change what the model sees |
| 8-bit optimizer | Optimizer states | Tiny precision loss |
| LoRA or QLoRA | Gradients, optimizer states and weights | Does not reduce activations |
| Sharding across GPUs, or a bigger card | Everything | Setup effort, or more cost |
Before changing anything, confirm which memory ran out. Fixing VRAM when RAM was the problem wastes an afternoon.
Conclusion
Out-of-memory errors feel random, but they have a short list of causes: model states, activations, leaks, fragmentation, and the wrong memory entirely. Work out which one you are facing before you change settings. Most jobs that crash can be rescued with a smaller micro-batch, gradient accumulation and checkpointing, without buying anything bigger.
Frequently asked questions
Does torch.cuda.empty_cache() fix out-of-memory errors?
Rarely. It releases cached blocks PyTorch is holding but not using, which can help with fragmentation. It cannot help when your model, gradients and activations genuinely do not fit.
Will adding more GPUs fix an out-of-memory error?
Only if you shard the model. Plain data parallel training keeps a full copy of the model on every GPU, so each card needs the same memory as before. Techniques that split optimizer states and weights across GPUs, such as ZeRO and FSDP, are what reduce the per-card load.
How can I estimate memory before I start?
Take about 16 bytes per trainable parameter for weights, gradients and optimizer states, then add activations. After that, run a few steps with a small batch and read the peak memory figure, rather than trusting any formula completely.