Most of us end up speccing a box for local models because an API bill or a privacy requirement made the decision for us, not because we wanted a hardware project. This is the sizing math I keep coming back to, aimed at developers who want a working setup rather than a benchmark chase.
Start With VRAM, Everything Else Follows
The baseline is roughly 2 GB of VRAM per billion parameters at FP16. A 7B model wants about 14 GB at full precision, about 7 GB at Q8, and roughly 3.5 GB at Q4 with some quality trade-off. That single ratio decides which models you can run at all.
On consumer cards, a used RTX 3090 at 24 GB is still the value pick, the 4090 matches it at 24 GB with more speed, and the 5090 moves the ceiling to 32 GB. Two 3090s give you 48 GB combined, which is enough for a 70B model at Q4, but multi-GPU adds power, cooling, and PCIe lane problems. For a single developer running agents, one high-VRAM card is simpler and more reliable than splitting across two.
CPU And RAM Are The Quiet Bottlenecks
For inference on one model, a modern 8 core CPU is plenty. The moment you host several agents at once or do any training, step up to 12 or more cores so tokenization and request routing do not stall behind each other.
System RAM should be about double your total VRAM. A 24 GB card wants 64 GB as a practical minimum, and a 70B model with CPU offloading can need 96 to 128 GB because the offloaded layers live in system memory. DDR5 over DDR4 for the bandwidth, and ECC if the box runs continuously.
Storage Decides How Fast You Iterate
A quantized 7B model is 4 to 8 GB on disk and a 70B is 35 to 70 GB, so a model collection reaches terabyte scale quickly. Sequential read speed is what you feel day to day: a PCIe 4.0 NVMe at 5,000 to 7,000 MB/s loads a 30 GB model in about 5 seconds, while a SATA SSD at 550 MB/s takes nearly a minute for the same file.
A 1 TB NVMe for the OS and active models plus a 2 TB secondary drive for archives and datasets covers most single-machine setups. The full AI server requirements guide breaks the same decisions down component by component if you want the longer version.
Pick A Tier, Not A Dream Build
Under $500 buys a used office PC with 32 GB of DDR4 and an 8 GB card, which runs 7B models at Q4 around 5 to 15 tokens per second. That is genuinely usable for a personal assistant or a single agent.
Between $500 and $2,000, a 3060 12 GB or a used 3090 with 64 GB of DDR5 and a 1 TB NVMe gives 20 to 40 tokens per second on 13B models and room for several concurrent agents. Above $2,000, a 4090 or 5090 with 128 GB of RAM handles most open weight models at useful quantization levels.
The Software Side Is Mostly Settled
Ubuntu Server LTS is the path of least resistance because that is what the frameworks are tested against. CUDA 12.x covers every NVIDIA card from the 30-series onward, and for serving you pick by workload: vLLM for batched throughput, llama.cpp for the widest hardware compatibility, Ollama when you want the simplest setup. Docker with the NVIDIA Container Toolkit keeps the CUDA versions from fighting each other.
Takeaway
Get the VRAM number right first, give the machine twice that in system RAM, put the models on NVMe, and the rest of the build stops being interesting in a good way. Almost every disappointing local setup I have seen failed on one of those three numbers rather than on the model choice.