Can I run Mistral-7B-Instruct-v0.3 on a AMD RX 7900 XT?

Yes.Runs

You can run this with Q8 quantization. On a AMD RX 7900 XT (20 GB VRAM), Mistral-7B-Instruct-v0.3 needs about 9.7 GB (Q8) and should generate at roughly ~67 tok/s. Recommended: Run on your GPU with Q8.

mistralai/Mistral-7B-Instruct-v0.3

Why

Full precision needs ~18 GB which exceeds your accelerator, but 8-bit (INT8 / Q8) needs only ~9.7 GB and fits.

How to run Mistral-7B-Instruct-v0.3 on a RX 7900 XT

Run with Ollama (easiest)

ollama run mistral:7b

Or with llama.cpp

llama-cli -hf mistralai/Mistral-7B-Instruct-v0.3 -p "Hello"

Serve with vLLM (OpenAI-compatible API)

# Fast production serving on http://localhost:8000/v1
vllm serve mistralai/Mistral-7B-Instruct-v0.3

Python (Transformers)

from transformers import pipeline, BitsAndBytesConfig
# pip install transformers accelerate bitsandbytes
quant = BitsAndBytesConfig(load_in_8bit=True)
pipe = pipeline("text-generation", model="mistralai/Mistral-7B-Instruct-v0.3", device_map="auto",
                model_kwargs={"quantization_config": quant})
print(pipe("Hello", max_new_tokens=50))

Want it to run comfortably at higher precision? The smallest GPU that runs Mistral-7B-Instruct-v0.3 well is the NVIDIA RTX 4090 (24 GB).

Try the full analyzer

Mistral-7B-Instruct-v0.3 on other GPUs

RTX 509032 GB VRAM RTX 508016 GB VRAM RTX 5070 Ti16 GB VRAM RTX 507012 GB VRAM RTX 409024 GB VRAM RTX 4080 Super16 GB VRAM

Other models on a RX 7900 XT

Qwen2.5-0.5Bon AMD RX 7900 XT GPT-2on AMD RX 7900 XT Phi-3-minion AMD RX 7900 XT Qwen2.5-Coderon AMD RX 7900 XT Llama-3.1-8Bon AMD RX 7900 XT Gemma-2-9Bon AMD RX 7900 XT