Can I run Mixtral-8x7B-Instruct-v0.1 on a NVIDIA H200?
Yes.Runs
You can run this comfortably. On a NVIDIA H200 (141 GB VRAM), Mixtral-8x7B-Instruct-v0.1 needs about 113 GB (FP16) and should generate at roughly ~34 tok/s. Recommended: Run on your GPU with FP16.
mistralai/Mixtral-8x7B-Instruct-v0.1Why
- At full precision this model needs ~113 GB, which fits your GPU VRAM.
How to run Mixtral-8x7B-Instruct-v0.1 on a H200
Serve with vLLM (OpenAI-compatible API)
# Fast production serving on http://localhost:8000/v1
vllm serve mistralai/Mixtral-8x7B-Instruct-v0.1Python (Transformers)
from transformers import pipeline
pipe = pipeline("text-generation", model="mistralai/Mixtral-8x7B-Instruct-v0.1", device_map="auto")
print(pipe("Hello", max_new_tokens=50))