Can I run Phi-3-mini-4k-instruct on a NVIDIA RTX 5070 Ti?
Yes.Runs
You can run this comfortably. On a NVIDIA RTX 5070 Ti (16 GB VRAM), Phi-3-mini-4k-instruct needs about 10 GB (FP16) and should generate at roughly ~70 tok/s. Recommended: Run on your GPU with FP16.
microsoft/Phi-3-mini-4k-instructWhy
- At full precision this model needs ~10 GB, which fits your GPU VRAM.
How to run Phi-3-mini-4k-instruct on a RTX 5070 Ti
Serve with vLLM (OpenAI-compatible API)
# Fast production serving on http://localhost:8000/v1
vllm serve microsoft/Phi-3-mini-4k-instructPython (Transformers)
from transformers import pipeline
pipe = pipeline("text-generation", model="microsoft/Phi-3-mini-4k-instruct", device_map="auto")
print(pipe("Hello", max_new_tokens=50))Want it to run comfortably at higher precision? The smallest GPU that runs Phi-3-mini-4k-instruct well is the AMD RX 7900 XT (20 GB).