Can I run Llama-3.1-70B-Instruct on a NVIDIA RTX 3060 (12GB)?
No.Too big
Not recommended on this machine. On a NVIDIA RTX 3060 (12GB) (12 GB VRAM), Llama-3.1-70B-Instruct memory needs are uncertain. Recommended: Use a cloud GPU or a smaller model.
meta-llama/Llama-3.1-70B-InstructWhy
- Even aggressively quantized (Q3) this needs ~37 GB — about 0.2 GB more than your GPU and RAM combined can offer.
How to run Llama-3.1-70B-Instruct on a RTX 3060 (12GB)
Python (Transformers)
from transformers import pipeline
pipe = pipeline("text-generation", model="meta-llama/Llama-3.1-70B-Instruct", device_map="auto")
print(pipe("Hello", max_new_tokens=50))Want it to run comfortably at higher precision? The smallest GPU that runs Llama-3.1-70B-Instruct well is the NVIDIA RTX 6000 Ada (48 GB).