Can I run DeepSeek-R1-Distill-Qwen-7B on a NVIDIA RTX 5070 Ti?
Yes.Runs
You can run this with Q8 quantization. On a NVIDIA RTX 5070 Ti (16 GB VRAM), DeepSeek-R1-Distill-Qwen-7B needs about 10 GB (Q8) and should generate at roughly ~71 tok/s. Recommended: Run on your GPU with Q8.
deepseek-ai/DeepSeek-R1-Distill-Qwen-7BWhy
- Full precision needs ~19 GB which exceeds your accelerator, but 8-bit (INT8 / Q8) needs only ~10 GB and fits.
How to run DeepSeek-R1-Distill-Qwen-7B on a RTX 5070 Ti
Run with Ollama (easiest)
ollama run deepseek-r1:7bOr with llama.cpp
llama-cli -hf deepseek-ai/DeepSeek-R1-Distill-Qwen-7B -p "Hello"Serve with vLLM (OpenAI-compatible API)
# Fast production serving on http://localhost:8000/v1
vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-7BPython (Transformers)
from transformers import pipeline, BitsAndBytesConfig
# pip install transformers accelerate bitsandbytes
quant = BitsAndBytesConfig(load_in_8bit=True)
pipe = pipeline("text-generation", model="deepseek-ai/DeepSeek-R1-Distill-Qwen-7B", device_map="auto",
model_kwargs={"quantization_config": quant})
print(pipe("Hello", max_new_tokens=50))Want it to run comfortably at higher precision? The smallest GPU that runs DeepSeek-R1-Distill-Qwen-7B well is the AMD RX 7900 XT (20 GB).