Run AI Models 35 Times Bigger Than Your Graphics Card: Set Up the Free AirLLM Library on 4GB
Big open AI models need an expensive graphics card, so most people never try one. AirLLM is a free, open-source Python library that runs a 70 billion parameter model on a single 4GB GPU by keeping only one layer of it in memory at a time. The install command, the code to paste in, ready-made prompts for Claude Code, and the model sizes from its own README
A 70 billion parameter model is about 140GB at 16-bit precision (70 billion numbers, two bytes each). That is why running one has meant buying a very expensive graphics card, and why most people never tried.
AirLLM is a free, open-source library (Apache-2.0 licence, over 35,000 stars on GitHub) that gets around the problem. Its README says it runs 70B models on a single 4GB GPU, without quantization, distillation, or pruning.
How it works
- A model is a stack of layers. A normal setup loads all of them into the graphics card at once, which is where the 140GB goes.
- AirLLM splits the model into its layers and saves them one by one on your disk the first time you run it.
- When you ask a question it loads one layer onto the GPU, runs it, frees it, and loads the next. Per its README it only ever keeps one layer on the GPU at a time.
- So the GPU memory you need depends on the size of one layer, not the whole model. Think of a giant book that you read one page at a time instead of cramming the whole book into your graphics card.
The trade-off is the disk. The README says the bottleneck is mainly disk loading, and that the original model is decomposed and saved layer-wise first, so you need free disk space for it.
What it can run, per its README
| Model | Size | GPU memory |
|---|---|---|
| Qwen3 / Mistral / Phi (about 8B) | 8B | about 1–2 GB |
| Qwen3.8-27B (dense, with vision) | 27B | 3.33 GB |
| Llama 3.x 70B (full precision) | 70B | about 4 GB |
| Qwen3-235B (MoE) | 235B | about 3 GB |
| Llama 3.1 405B | 405B | about 8 GB |
| DeepSeek-V3 | 671B | about 12 GB |
| Kimi K3, the largest open model released to date | 2.8T | 3.72 GB (measured on one RTX 6000 Ada) |
Same one line of code for all of them: you only change the Hugging Face model ID.
What you need first
- A computer with an NVIDIA graphics card (the README's example calls .cuda()), or an Apple silicon Mac (see the Mac note below).
- Python and pip.
- Free disk space: the model is downloaded and then split into layers, so plan for a lot. If disk is tight, the delete_original option removes the original download and keeps only the split model, which saves about half.
- A Hugging Face account and a token if the model is gated (for example the Llama models).
Free, for your email
Get the full setup steps + the copy-paste setup prompt
The rest of this guide (4 more sections, 4 copy-paste prompts) opens right here. The setup prompt, one paste into Claude that sets it all up, comes by email only. Use your real address.
50,000+ follow where these guides come from. No spam, unsubscribe in one click.