Estimating from the GGUF header is a good start, but the context cache is where these estimates usually drift, because it depends on the number of KV heads, the head dimension and whether the KV cache is quantized, not just layer count. On macOS I would also treat compressed and purgeable pages as reclaimable instead of reading only free memory, otherwise the tool may tell people to close apps that were not the real problem.
I kept rebooting my machine to run local LLMs, so I built canfit
2 Comments
@[inevolin] Agreed on both, and canfit already goes most of the way:
KV cache. The estimate isn’t layer count × a constant. It reads head_count_kv and the key/value lengths from the header (per layer when they differ, so hybrid models with zero-KV layers come out right). It also handles GQA and MLA latent caches and sizes by cache type: --cache-type f16|q8_0|q4_0.
In my tests it matches llama.cpp’s allocation exactly. The known drift is sliding-window models (Gemma 2/3), where canfit sizes a full cache for every layer and runs high. Every number is labelled measured / read from the file / estimated. When you want the real figure, canfit watch -- llama-server ... measures the peak of the whole process tree, shows it next to the estimate and saves it locally. canfit history --apps lists it later, and the next check answers from that measurement.
If you like the project, a ⭐ on GitHub (github.com/federicolaterra/canfit) means a lot and really helps. More features are coming in the next releases.
Please log in to add a comment.
Please log in to comment on this post.
More Posts
- © 2026 Coder Legion
- Feedback / Bug
- Privacy
- About Us
- Contacts
- You Tube
- Tiktok
- Premium Subscription
- Terms of Service
- Early Builders
I work across visual design, software architecture, and development. Interested in open-source software, local-first tools, knowledge systems, and cybersecurity.
Currently building Knemora, an open-source, local-first AI (SI) second brain that turns documents and datasets into linked knowledge. Show less
More From federicolaterra
Related Jobs
- Senior Staff SRE, Architecture Engineering: Hosted RunnersGitLab · Full time · United Kingdom
- Machine Learning Engineer 2 - FreewheelComcast · Full time · Sunnyvale, CA
- AIML - Machine Learning Engineer, MINDApple · Full time · Seattle, WA
Commenters (This Week)
Contribute meaningful comments to climb the leaderboard and earn badges!