Can I run Mistral-7B-Instruct-v0.3 on a NVIDIA RTX 5090?
Yes.Runs
You can run this comfortably. On a NVIDIA RTX 5090 (32 GB VRAM), Mistral-7B-Instruct-v0.3 needs about 18 GB (FP16) and should generate at roughly ~74 tok/s. Recommended: Run on your GPU with FP16.
mistralai/Mistral-7B-Instruct-v0.3Why
- At full precision this model needs ~18 GB, which fits your GPU VRAM.
How to run Mistral-7B-Instruct-v0.3 on a RTX 5090
Serve with vLLM (OpenAI-compatible API)
# Fast production serving on http://localhost:8000/v1
vllm serve mistralai/Mistral-7B-Instruct-v0.3Python (Transformers)
from transformers import pipeline
pipe = pipeline("text-generation", model="mistralai/Mistral-7B-Instruct-v0.3", device_map="auto")
print(pipe("Hello", max_new_tokens=50))Want it to run comfortably at higher precision? The smallest GPU that runs Mistral-7B-Instruct-v0.3 well is the NVIDIA A100 (40GB) (40 GB).