Can I run Qwen2.5-32B-Instruct on a NVIDIA RTX 4090?

Yes.Runs

You can run this with Q3 quantization. On a NVIDIA RTX 4090 (24 GB VRAM), Qwen2.5-32B-Instruct needs about 18 GB (Q3) and should generate at roughly ~47 tok/s. Recommended: Run on your GPU with Q3.

Qwen/Qwen2.5-32B-Instruct

Why

Full precision needs ~80 GB which exceeds your accelerator, but 3-bit (Q3_K) needs only ~18 GB and fits.

How to run Qwen2.5-32B-Instruct on a RTX 4090

Run with Ollama (easiest)

ollama run qwen2.5:32b

Or with llama.cpp

llama-cli -hf Qwen/Qwen2.5-32B-Instruct -p "Hello"

Serve with vLLM (OpenAI-compatible API)

# Fast production serving on http://localhost:8000/v1
vllm serve Qwen/Qwen2.5-32B-Instruct

Python (Transformers)

from transformers import pipeline, BitsAndBytesConfig
# pip install transformers accelerate bitsandbytes
quant = BitsAndBytesConfig(load_in_4bit=True)
pipe = pipeline("text-generation", model="Qwen/Qwen2.5-32B-Instruct", device_map="auto",
                model_kwargs={"quantization_config": quant})
print(pipe("Hello", max_new_tokens=50))

Want it to run comfortably at higher precision? The smallest GPU that runs Qwen2.5-32B-Instruct well is the NVIDIA RTX 5090 (32 GB).

Try the full analyzer

Qwen2.5-32B-Instruct on other GPUs

RTX 509032 GB VRAM RTX 508016 GB VRAM RTX 5070 Ti16 GB VRAM RTX 507012 GB VRAM RTX 4080 Super16 GB VRAM RTX 4070 Ti Super16 GB VRAM

Other models on a RTX 4090

Qwen2.5-0.5Bon RTX 4090 GPT-2on RTX 4090 Phi-3-minion RTX 4090 Qwen2.5-Coderon RTX 4090 Mistral-7Bon RTX 4090 Llama-3.1-8Bon RTX 4090