Can I run Qwen2.5-Coder-7B-Instruct on a NVIDIA RTX 3080 (10GB)?
Yes.Runs
You can run this with Q4 quantization. On a NVIDIA RTX 3080 (10GB) (10 GB VRAM), Qwen2.5-Coder-7B-Instruct needs about 6.0 GB (Q4) and should generate at roughly ~100 tok/s. Recommended: Run on your GPU with Q4.
Qwen/Qwen2.5-Coder-7B-InstructWhy
- Full precision needs ~19 GB which exceeds your accelerator, but 4-bit (Q4_K_M / INT4) needs only ~6.0 GB and fits.
How to run Qwen2.5-Coder-7B-Instruct on a RTX 3080 (10GB)
Run a GGUF build with Ollama
ollama run hf.co/Qwen/Qwen2.5-Coder-7B-InstructOr with llama.cpp
llama-cli -hf Qwen/Qwen2.5-Coder-7B-Instruct -p "Hello"Serve with vLLM (OpenAI-compatible API)
# Fast production serving on http://localhost:8000/v1
vllm serve Qwen/Qwen2.5-Coder-7B-InstructPython (Transformers)
from transformers import pipeline, BitsAndBytesConfig
# pip install transformers accelerate bitsandbytes
quant = BitsAndBytesConfig(load_in_4bit=True)
pipe = pipeline("text-generation", model="Qwen/Qwen2.5-Coder-7B-Instruct", device_map="auto",
model_kwargs={"quantization_config": quant})
print(pipe("Hello", max_new_tokens=50))Want it to run comfortably at higher precision? The smallest GPU that runs Qwen2.5-Coder-7B-Instruct well is the NVIDIA RTX 2080 Ti (11 GB).