Can I run Llama-3.1-70B-Instruct on a NVIDIA RTX 5070 Ti?
Yes — with workarounds.Workarounds
Runs with GPU + CPU offload (~39% on GPU). On a NVIDIA RTX 5070 Ti (16 GB VRAM), Llama-3.1-70B-Instruct needs about 37 GB (Q3) and should generate at roughly ~2.2 tok/s. Recommended: Split across GPU + RAM with Q3 (offload).
meta-llama/Llama-3.1-70B-InstructWhy
- Too big for your VRAM alone, but at Q3 (~37 GB) it fits across VRAM + RAM. About 39% stays on the GPU and the rest runs from RAM — slower than full-GPU, but much faster than CPU-only.
How to run Llama-3.1-70B-Instruct on a RTX 5070 Ti
Run with Ollama (easiest)
ollama run llama3.1:70bllama.cpp with GPU offload (~39% on GPU)
# -ngl = number of layers on the GPU. Raise it until VRAM is full,
# lower it if you hit out-of-memory. -1 tries to offload everything.
llama-cli -hf meta-llama/Llama-3.1-70B-Instruct -ngl 99 -p "Hello"Serve with vLLM + CPU offload
# --cpu-offload-gb moves the overflow weights to system RAM
vllm serve meta-llama/Llama-3.1-70B-Instruct --cpu-offload-gb 23Python (Transformers)
from transformers import pipeline, BitsAndBytesConfig
# pip install transformers accelerate bitsandbytes
quant = BitsAndBytesConfig(load_in_4bit=True)
pipe = pipeline("text-generation", model="meta-llama/Llama-3.1-70B-Instruct", device_map="auto",
model_kwargs={"quantization_config": quant})
print(pipe("Hello", max_new_tokens=50))Want it to run comfortably at higher precision? The smallest GPU that runs Llama-3.1-70B-Instruct well is the NVIDIA RTX 6000 Ada (48 GB).