Can I run Qwen2.5-Coder-7B-Instruct on a NVIDIA RTX 6000 Ada?

Yes.Runs

You can run this comfortably. On a NVIDIA RTX 6000 Ada (48 GB VRAM), Qwen2.5-Coder-7B-Instruct needs about 19 GB (FP16) and should generate at roughly ~41 tok/s. Recommended: Run on your GPU with FP16.

Qwen/Qwen2.5-Coder-7B-Instruct

Why

At full precision this model needs ~19 GB, which fits your GPU VRAM.

How to run Qwen2.5-Coder-7B-Instruct on a RTX 6000 Ada

Serve with vLLM (OpenAI-compatible API)

# Fast production serving on http://localhost:8000/v1
vllm serve Qwen/Qwen2.5-Coder-7B-Instruct

Python (Transformers)

from transformers import pipeline
pipe = pipeline("text-generation", model="Qwen/Qwen2.5-Coder-7B-Instruct", device_map="auto")
print(pipe("Hello", max_new_tokens=50))

Want it to run comfortably at higher precision? The smallest GPU that runs Qwen2.5-Coder-7B-Instruct well is the NVIDIA H100 (80GB) (80 GB).

Try the full analyzer

Qwen2.5-Coder-7B-Instruct on other GPUs

RTX 509032 GB VRAM RTX 508016 GB VRAM RTX 5070 Ti16 GB VRAM RTX 507012 GB VRAM RTX 409024 GB VRAM RTX 4080 Super16 GB VRAM

Other models on a RTX 6000 Ada

Qwen2.5-0.5Bon RTX 6000 Ada GPT-2on RTX 6000 Ada Phi-3-minion RTX 6000 Ada Mistral-7Bon RTX 6000 Ada Llama-3.1-8Bon RTX 6000 Ada Gemma-2-9Bon RTX 6000 Ada