Can I run gemma-2-9b-it on a NVIDIA RTX 5070?
Yes.Runs
You can run this with Q4 quantization. On a NVIDIA RTX 5070 (12 GB VRAM), gemma-2-9b-it needs about 7.1 GB (Q4) and should generate at roughly ~78 tok/s. Recommended: Run on your GPU with Q4.
google/gemma-2-9b-itWhy
- Full precision needs ~23 GB which exceeds your accelerator, but 4-bit (Q4_K_M / INT4) needs only ~7.1 GB and fits.
How to run gemma-2-9b-it on a RTX 5070
Run with Ollama (easiest)
ollama run gemma2:9bOr with llama.cpp
llama-cli -hf google/gemma-2-9b-it -p "Hello"Serve with vLLM (OpenAI-compatible API)
# Fast production serving on http://localhost:8000/v1
vllm serve google/gemma-2-9b-itPython (Transformers)
from transformers import pipeline, BitsAndBytesConfig
# pip install transformers accelerate bitsandbytes
quant = BitsAndBytesConfig(load_in_4bit=True)
pipe = pipeline("text-generation", model="google/gemma-2-9b-it", device_map="auto",
model_kwargs={"quantization_config": quant})
print(pipe("Hello", max_new_tokens=50))Want it to run comfortably at higher precision? The smallest GPU that runs gemma-2-9b-it well is the NVIDIA RTX 5080 (16 GB).