Most local LLM threads end the same way. Someone posts a benchmark, someone else says their hardware cannot run it, and nobody agrees on which model to actually start with. Here is the shorter version of what holds up once you sit down and try it.
Start With the Job, Not the Leaderboard
The open source field in 2026 sorts pretty cleanly by task. Qwen 3 72B is the safe general purpose default and ships under Apache 2.0. DeepSeek R1 is what you reach for when the work is reasoning heavy, it hits 79.8% on AIME with verifiable chain of thought steps. GLM-5.2 leads on coding under an MIT license. Llama 4 Scout carries a 256K context window, which makes it the long document and RAG choice. Phi-4 and Gemma 3 only make sense when the hardware forces your hand.
The Quantization Rule That Saves the Most Money
A larger model at lower precision almost always beats a smaller model at higher precision. A 70B at Q4 outruns a 7B at FP16 while occupying a similar amount of memory. So the method is simple: pick the largest model your VRAM or unified memory can hold, then apply the highest quality quantization that still fits. Q4_K_M is the usual sweet spot, roughly 95% of full precision quality at about 25% of the memory footprint. Below Q3 the quality degrades faster than the savings are worth. The full breakdown of formats, GGUF versus GPTQ versus AWQ and where FP8 fits on Hopper and newer, is in our guide to open source LLMs.
Ollama Is Not a Serving Layer
Ollama is the right tool for getting a model running on your laptop in one command, and it exposes an OpenAI compatible API on localhost:11434 so existing client code only changes a URL. It is not what you put in front of concurrent users. vLLM's PagedAttention manages KV cache the way an OS manages virtual memory, and in 2026 benchmarks it delivers around 16x the throughput of Ollama under concurrent load. SGLang is the pick when your output has to conform to a schema, which covers most agent work.
Where Self Hosting Actually Wins
Under 100,000 tokens a day an API is cheaper, you would be paying for infrastructure you barely touch. Between 1 and 10 million a day, a single rented GPU breaks even with API pricing inside two to four weeks. Above 10 million a day, self hosting runs 4x to 10x less. Concretely: an A100 80GB rents for roughly $0.80 to $1.50 an hour and serves about 8,600 500-token responses a day, so $20 to $36 of compute against roughly $130 a day through a proprietary API at the same volume.
The costs people forget are the human ones, monitoring, updates, failover, and the team time spent building serving infrastructure instead of features.
What are you running locally, and what made you land on that model rather than the one next to it on the leaderboard?
