Can I run Qwen2.5-0.5B-Instruct on a NVIDIA RTX 3090?
Yes.Runs
You can run this comfortably. On a NVIDIA RTX 3090 (24 GB VRAM), Qwen2.5-0.5B-Instruct needs about 2.2 GB (FP16) and should generate at roughly ~290 tok/s. Recommended: Run on your GPU with FP16.
Qwen/Qwen2.5-0.5B-InstructWhy
- At full precision this model needs ~2.2 GB, which fits your GPU VRAM.
How to run Qwen2.5-0.5B-Instruct on a RTX 3090
Serve with vLLM (OpenAI-compatible API)
# Fast production serving on http://localhost:8000/v1
vllm serve Qwen/Qwen2.5-0.5B-InstructPython (Transformers)
from transformers import pipeline
pipe = pipeline("text-generation", model="Qwen/Qwen2.5-0.5B-Instruct", device_map="auto")
print(pipe("Hello", max_new_tokens=50))Want it to run comfortably at higher precision? The smallest GPU that runs Qwen2.5-0.5B-Instruct well is the NVIDIA RTX 5090 (32 GB).