I didn't set out to write a tutorial. This is closer to a log of what happened when I tried to wire a locally-hosted Qwen model into Claude's third-party inference gateway, and the things that quietly broke before it worked.
Why bother with a local backend at all
Most of my work involves research and document generation that doesn't need to hit a hosted API every single time. I wanted something I could run on my own machine, mostly to experiment freely without worrying about racking up API calls for every small test. The idea wasn't to replace Claude, but to have a local option sitting behind the same interface, one I could point at when the task didn't call for a hosted model.
Qwen3.6 looked like a reasonable fit for this. Not because it's the newest or the most talked about, but because it's the kind of model you can actually run on a single machine without needing a rack of GPUs.
The basic shape of the setup
The architecture ended up being fairly simple on paper: llama-server runs the model locally, and LiteLLM sits in front of it as a proxy that speaks the API shape Claude's third-party inference gateway expects. LiteLLM runs inside a Docker container (via Colima on macOS), and llama-server runs directly on the host.
On paper, that's three components talking to each other. In practice, it took a bit of trial and error before they actually did.
The Docker networking trap
The first real snag came from something that seems obvious in hindsight: LiteLLM, running inside a Docker container, tried to reach llama-server via 127.0.0.1. That address, from inside a container, doesn't point to the host machine. It points to the container's own loopback interface, which has nothing running on it.
The fix was switching to host.docker.internal instead. It's a small detail, but it's the kind of thing that costs an hour of confused debugging if you haven't hit it before. If you're bridging a host-level service to something running inside Docker, this is worth checking first, before assuming the service itself is broken.
Hitting a context window wall
Once the plumbing was working, a different problem showed up during a longer conversation: a ContextWindowExceededError, with the context window set to 32768 tokens on llama-server. It's a reminder that a locally-served model doesn't quietly resize itself the way a hosted API sometimes appears to. The ceiling is whatever you set at launch, and it will tell you when you've hit it.
The trade-off nobody tells you about upfront
I started with Qwen3.6-27B, then moved to Qwen3.6-35B-A3B. The larger model scores well (Qwen3.6-27B benchmarks around 77.2% on SWE-bench), but it's also slower to respond. That trade-off between quality and speed is easy to state in the abstract, but it feels very different once you're the one waiting on time-to-first-token during an actual working session.
There's no universally correct answer here. It depends on whether the task in front of you needs the better score more than it needs the faster reply.
Where things stand now
LiteLLM is successfully connected to Claude's third-party inference gateway, with the local model mapped in the gateway configuration. It works. It is also, honestly, still a work in progress rather than a finished setup I'd call done. Configuration around slot persistence, context sizing, and model choice is something I expect to keep adjusting as I use it more.
If you've built something similar, or hit a different wall along the way, I'd be glad to compare notes. This kind of setup seems to have more undocumented edges than the clean tutorials suggest.
