Plenty of the self-hosting threads here stop at the point where a model finally answers on localhost. This is a map of everything that sits around that, because inference turns out to be the easy layer.
The Five Layers
Inference loads model weights and generates tokens. Ollama when you want it to just work, vLLM for production throughput, llama.cpp when you are squeezing an underpowered box.
Embeddings turn text into vectors and search them. Qdrant, ChromaDB or pgvector, paired with an embedding model like nomic-embed-text or BGE. This is the layer that lets a model answer questions about documents it never saw in training.
Memory gives agents knowledge that survives the session. Conversation history, session state, long term facts, knowledge graphs. Redis and Postgres do most of the real work here.
Tools connect the model to everything outside itself. An LLM only emits text, so reading files, calling APIs, querying databases and running code all live at this layer. MCP servers and function calling frameworks sit here.
Orchestration decides which model runs, when to retrieve context, which tool to call and how multiple agents hand off. LangGraph, n8n, Dify, or your own code. Without it you have five services. With it you have an agent.
Where Setups Actually Break
Most stalled self-hosting projects have layers one and two and nothing else, and the symptoms are consistent. The assistant is impressive on a single question and useless across a week, because there is no memory layer. Every component works alone and nothing composes, because there is no orchestration. Neither failure looks like a model problem, so people go shopping for a bigger model instead of the missing layer.
The Cost Math Has A Floor
Self-hosting is not automatically cheaper. A consumer GPU like a 12 GB RTX 3060 runs about 300 to 400 dollars and handles a 7B model comfortably, with maybe 10 to 20 dollars a month in electricity. Against cloud pricing of roughly 0.15 to 0.60 dollars per million input tokens, 100,000 tokens a day costs 15 to 60 dollars a month, so the card pays for itself somewhere between six months and two years.
Below that volume an API is genuinely cheaper and you should use one. Above it, and especially where the data cannot leave your hardware for compliance reasons, the fixed cost structure wins and keeps winning as volume grows. Rented GPU instances sit in between at 0.50 to 2.00 dollars an hour, which is fine for bursts and expensive for always-on.
Starting Order
Start with inference so you have something that responds, add embeddings when you need it to know your own documents, then memory, then tools, then orchestration. Adding them in that order means each layer has something to plug into, and you can stop at whatever depth your use case actually needs.
If you want the long version, the full five layer breakdown covers each layer's tooling options, a couple of ready-made stack combinations, and the cost comparison in more detail.