Can I run Mixtral-8x7B-Instruct-v0.1 on a NVIDIA RTX 5090?
Yes.Runs
You can run this with Q3 quantization. On a NVIDIA RTX 5090 (32 GB VRAM), Mixtral-8x7B-Instruct-v0.1 needs about 25 GB (Q3) and should generate at roughly ~57 tok/s. Recommended: Run on your GPU with Q3.
mistralai/Mixtral-8x7B-Instruct-v0.1Why
- Full precision needs ~113 GB which exceeds your accelerator, but 3-bit (Q3_K) needs only ~25 GB and fits.
How to run Mixtral-8x7B-Instruct-v0.1 on a RTX 5090
Run with Ollama (easiest)
ollama run mixtral:8x7bOr with llama.cpp
llama-cli -hf mistralai/Mixtral-8x7B-Instruct-v0.1 -p "Hello"Serve with vLLM (OpenAI-compatible API)
# Fast production serving on http://localhost:8000/v1
vllm serve mistralai/Mixtral-8x7B-Instruct-v0.1Python (Transformers)
from transformers import pipeline, BitsAndBytesConfig
# pip install transformers accelerate bitsandbytes
quant = BitsAndBytesConfig(load_in_4bit=True)
pipe = pipeline("text-generation", model="mistralai/Mixtral-8x7B-Instruct-v0.1", device_map="auto",
model_kwargs={"quantization_config": quant})
print(pipe("Hello", max_new_tokens=50))Want it to run comfortably at higher precision? The smallest GPU that runs Mixtral-8x7B-Instruct-v0.1 well is the NVIDIA A100 (40GB) (40 GB).