How to Fix CUDA Out of Memory Errors in Local AI Models

Running Llama, Stable Diffusion, vLLM, or other local AI workloads can quickly trigger torch.cuda.OutOfMemoryError. The message appears when model weights, activations, context tokens, and batch overhead demand more VRAM than the GPU can provide.

This guide shows how to reduce memory pressure without immediately buying a new graphics card.

---

What Triggers CUDA Memory Errors?

• Unquantized weights: A 7-billion-parameter model in FP16 can require roughly 14 GB just for its weights.
• Long context windows: The KV cache expands as a conversation grows.
• Large batches: Parallel prompts and images increase memory use.
• Fragmentation: Free VRAM may be split into blocks too small for a new allocation.

---

Step 1: Use 4-bit or 8-bit Quantization

Quantization stores model weights with fewer bits, dramatically reducing VRAM use with a relatively small quality trade-off.

Ollama and GGUF

ollama run llama3:8b-instruct-q4_K_M

PyTorch and Hugging Face

import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_use_double_quant=True
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Meta-Llama-3-8B-Instruct",
    quantization_config=config,
    device_map="auto"
)

---

Step 2: Tune the PyTorch Allocator

Use the allocator setting appropriate for your shell, then restart the workload.

Windows Command Prompt

set PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:512

PowerShell

$env:PYTORCH_CUDA_ALLOC_CONF="max_split_size_mb:512"

Linux

export PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:512

---

Step 3: Offload Layers and Optimize Attention

1. Use FlashAttention-2 on supported NVIDIA hardware to lower attention-memory overhead.
2. Add device_map="auto" so supported libraries can spill excess weights into system RAM.
3. Reduce context length, batch size, or image resolution before lowering model quality further.

---

Step 4: Clear Cached VRAM Between Runs

import gc
import torch

def clear_vram():
    gc.collect()
    torch.cuda.empty_cache()
    torch.cuda.ipc_collect()

clear_vram()

VRAM Optimization Matrix

Method	Typical saving	Trade-off	Best for
4-bit quantization	65–70%	Small quality/speed change	8–12 GB GPUs
FlashAttention-2	30–40%	Requires compatible hardware	Modern NVIDIA GPUs
CPU offloading	Up to 80%	Slower generation	4–6 GB GPUs
Allocator tuning	Prevents spikes	None in most workloads	Fragmentation crashes