Can I run Qwen2.5-32B-Instruct on a NVIDIA RTX 5070?
Yes — with workarounds.Workarounds
Runs with GPU + CPU offload (~48% on GPU). On a NVIDIA RTX 5070 (12 GB VRAM), Qwen2.5-32B-Instruct needs about 23 GB (Q4) and should generate at roughly ~4.1 tok/s. Recommended: Split across GPU + RAM with Q4 (offload).
Qwen/Qwen2.5-32B-InstructWhy
- Too big for your VRAM alone, but at Q4 (~23 GB) it fits across VRAM + RAM. About 48% stays on the GPU and the rest runs from RAM — slower than full-GPU, but much faster than CPU-only.
How to run Qwen2.5-32B-Instruct on a RTX 5070
Run with Ollama (easiest)
ollama run qwen2.5:32bllama.cpp with GPU offload (~48% on GPU)
# -ngl = number of layers on the GPU. Raise it until VRAM is full,
# lower it if you hit out-of-memory. -1 tries to offload everything.
llama-cli -hf Qwen/Qwen2.5-32B-Instruct -ngl 99 -p "Hello"Serve with vLLM + CPU offload
# --cpu-offload-gb moves the overflow weights to system RAM
vllm serve Qwen/Qwen2.5-32B-Instruct --cpu-offload-gb 12Python (Transformers)
from transformers import pipeline, BitsAndBytesConfig
# pip install transformers accelerate bitsandbytes
quant = BitsAndBytesConfig(load_in_4bit=True)
pipe = pipeline("text-generation", model="Qwen/Qwen2.5-32B-Instruct", device_map="auto",
model_kwargs={"quantization_config": quant})
print(pipe("Hello", max_new_tokens=50))Want it to run comfortably at higher precision? The smallest GPU that runs Qwen2.5-32B-Instruct well is the AMD RX 7900 XT (20 GB).