How much VRAM do you need to run AI models locally?
The memory an LLM needs is driven by its parameter count and precision. As a rule of thumb, a model needs about 2 GB per billion parameters at FP16, 1 GB/B at 8-bit, and ~0.6 GB/B at 4-bit — plus ~20% runtime overhead. Here are the numbers for every common size, and what each GPU can run.
VRAM required by model size
| Model size | FP16 | 8-bit (Q8) | 4-bit (Q4) | 3-bit (Q3) |
|---|---|---|---|---|
| 1B | 3.4 GB | 2.2 GB | 1.7 GB | 1.5 GB |
| 3B | 8.2 GB | 4.6 GB | 3.0 GB | 2.5 GB |
| 7B | 18 GB | 9.4 GB | 5.6 GB | 4.5 GB |
| 8B | 20 GB | 11 GB | 6.3 GB | 5.0 GB |
| 13B | 32 GB | 17 GB | 9.6 GB | 7.6 GB |
| 14B | 35 GB | 18 GB | 10 GB | 8.1 GB |
| 34B | 83 GB | 42 GB | 23 GB | 18 GB |
| 8×7B (MoE, 47B) | 114 GB | 57 GB | 32 GB | 25 GB |
| 70B | 169 GB | 85 GB | 47 GB | 36 GB |
| 405B | 973 GB | 487 GB | 268 GB | 205 GB |
Includes ~20% runtime overhead. MoE models (e.g. 8×7B) load all experts, so size by total parameters, not active ones.
What each GPU can run (at 4-bit)
| GPU | VRAM | 7B | 13B | 34B | 70B | Max (~4-bit) |
|---|---|---|---|---|---|---|
| RTX 4060 | 8 GB | ~10B | ||||
| RTX 3060 (12GB) | 12 GB | ~16B | ||||
| RTX 4070 Super | 12 GB | ~16B | ||||
| RTX 4080 Super | 16 GB | ~21B | ||||
| RTX 4090 | 24 GB | ~32B | ||||
| RTX 5090 | 32 GB | ~43B | ||||
| A100 (80GB) | 80 GB | ~109B |
fits in VRAM runs with GPU+CPU offload too big
VRAM requirements — frequently asked questions
How much VRAM do I need to run a 7B model?
About 5 GB of VRAM at 4-bit quantization, or roughly 18 GB at full FP16 precision. A 7–8B model at 4-bit fits comfortably on any 8 GB+ GPU.
How much VRAM does a 13B or 70B model need?
At 4-bit: a 13B model needs ~9 GB and a 70B model ~40 GB. At FP16 they need roughly 4× that. A 70B model therefore needs a 48 GB+ GPU for 4-bit, or GPU+CPU offload on smaller cards.
Can I run a 70B model on a 24 GB GPU like an RTX 4090?
Not entirely in VRAM — 70B needs ~40 GB even at 4-bit. But you can split it across GPU + system RAM (offload) and it will run, just slower than a model that fits fully on the GPU.
How much VRAM does quantization save?
A lot. FP16 uses ~2 GB per billion parameters; 8-bit halves that to ~1 GB/B; 4-bit (Q4_K_M) drops to ~0.55 GB/B — about a quarter of FP16 — for only a small quality loss. It's what makes large models runnable on consumer GPUs.
Want the exact numbers for a specific model? Browse models or paste any Hugging Face model to check it against your own hardware. Curious what a specific card handles? Browse GPUs.