Can I run gpt2 on a NVIDIA RTX 6000 Ada?
Yes.Runs
You can run this comfortably. On a NVIDIA RTX 6000 Ada (48 GB VRAM), gpt2 needs about 1.3 GB (FP16) and should generate at roughly ~420 tok/s. Recommended: Run on your GPU with FP16.
openai-community/gpt2Why
- At full precision this model needs ~1.3 GB, which fits your GPU VRAM.
How to run gpt2 on a RTX 6000 Ada
Serve with vLLM (OpenAI-compatible API)
# Fast production serving on http://localhost:8000/v1
vllm serve openai-community/gpt2Python (Transformers)
from transformers import pipeline
pipe = pipeline("text-generation", model="openai-community/gpt2", device_map="auto")
print(pipe("Hello", max_new_tokens=50))Want it to run comfortably at higher precision? The smallest GPU that runs gpt2 well is the NVIDIA H100 (80GB) (80 GB).