Can I run Mixtral-8x7B-Instruct-v0.1 on a NVIDIA RTX 4090?
Yes — with workarounds.Workarounds
Runs with GPU + CPU offload (~68% on GPU). On a NVIDIA RTX 4090 (24 GB VRAM), Mixtral-8x7B-Instruct-v0.1 needs about 32 GB (Q4) and should generate at roughly ~4.5 tok/s. Recommended: Split across GPU + RAM with Q4 (offload).
mistralai/Mixtral-8x7B-Instruct-v0.1Why
- Too big for your VRAM alone, but at Q4 (~32 GB) it fits across VRAM + RAM. About 68% stays on the GPU and the rest runs from RAM — slower than full-GPU, but much faster than CPU-only.
How to run Mixtral-8x7B-Instruct-v0.1 on a RTX 4090
Run with Ollama (easiest)
ollama run mixtral:8x7bllama.cpp with GPU offload (~68% on GPU)
# -ngl = number of layers on the GPU. Raise it until VRAM is full,
# lower it if you hit out-of-memory. -1 tries to offload everything.
llama-cli -hf mistralai/Mixtral-8x7B-Instruct-v0.1 -ngl 99 -p "Hello"Serve with vLLM + CPU offload
# --cpu-offload-gb moves the overflow weights to system RAM
vllm serve mistralai/Mixtral-8x7B-Instruct-v0.1 --cpu-offload-gb 11Python (Transformers)
from transformers import pipeline, BitsAndBytesConfig
# pip install transformers accelerate bitsandbytes
quant = BitsAndBytesConfig(load_in_4bit=True)
pipe = pipeline("text-generation", model="mistralai/Mixtral-8x7B-Instruct-v0.1", device_map="auto",
model_kwargs={"quantization_config": quant})
print(pipe("Hello", max_new_tokens=50))Want it to run comfortably at higher precision? The smallest GPU that runs Mixtral-8x7B-Instruct-v0.1 well is the NVIDIA RTX 5090 (32 GB).