Running an LLM locally is easy to demonstrate.
Start Ollama or vLLM, load a model, send a request, and watch it generate tokens.
The interesting problems start when you send more than one kind of request.
Imagine a local inference server handling two workloads:
Interactive requests where a user is waiting for an answer.
Background jobs generating longer responses that can tolerate delay.
With plenty of spare capacity, both workloads coexist peacefully.
Under contention, they don't.
A few large background requests can occupy enough inference capacity that interactive requests wait behind work that was never latency-sensitive in the first place.
At that point, adding another HTTP worker or increasing a concurrency limit doesn't necessarily solve the problem.
You have a scheduling problem.
Concurrency Is Not Capacity Management
A common first approach is to put a semaphore around inference:
maximum concurrent requests = 4
That's useful. It prevents unlimited work from entering the system.
But it doesn't answer a more important question:
Which four requests should be allowed to run?
Suppose all four slots are occupied by long-running batch generations.
An interactive request arrives.
The concurrency limit is working exactly as designed, yet the user still waits.
The system protected itself from overload, but it didn't protect the workload that mattered most.
This is where admission control becomes different from simple concurrency limiting.
Concurrency asks:
How much work can execute simultaneously?
Admission control asks:
Which work should consume scarce capacity right now?
Tokens Make the Problem More Interesting
LLM workloads also aren't uniform.
Consider two requests:
Request A
input: 500 tokens
maximum output: 200 tokens
Request B
input: 2,000 tokens
maximum output: 4,000 tokens
Counting both as "one request" hides a large difference in potential resource consumption.
A better admission decision can account for both request concurrency and estimated token footprint.
Conceptually:
estimated footprint = input tokens + maximum output tokens
Before admitting a request, the controller asks whether enough capacity remains for that footprint.
The estimate doesn't need to predict the exact number of generated tokens perfectly. It needs to provide a useful bound for deciding whether admitting more work is safe.
When the request finishes, unused capacity can be returned.
This gives the system another dimension of protection beyond request count alone.
Static Limits Still Leave a Policy Problem
You can go further and reserve capacity.
For example:
Interactive: 3 slots
Batch: 1 slot
Now interactive traffic is protected.
But you've created another problem.
What happens when there are no interactive requests?
If three reserved slots sit unused while batch work waits, utilization suffers.
The obvious improvement is borrowing.
Let batch jobs temporarily consume unused interactive capacity:
Interactive demand low
↓
Batch borrows spare capacity
↓
Interactive demand returns
↓
Borrowed capacity must be restored
That last step is where things become interesting.
Borrowing idle capacity is easy.
Safely reclaiming it is the hard part.
Borrowing Without Restoration Isn't Really a Reservation
Suppose an interactive class is guaranteed three units of capacity.
During an idle period, batch workloads borrow two of them.
Then interactive traffic suddenly returns.
If those batch requests can't be interrupted, the interactive reservation doesn't immediately exist anymore. It exists only after the borrowers finish.
That means the actual guarantee depends on restoration behavior.
A useful reservation policy therefore needs to define more than:
interactive floor = 3
It also needs to answer:
How quickly can that floor become usable again?
This distinction matters especially with inference because some resources can be reclaimed immediately while others cannot.
A controller may stop granting new batch work immediately.
But an upstream inference request already running on a GPU may continue until completion.
So "capacity restored" can mean at least two different things:
- The scheduler has restored the entitlement.
- The physical resource is actually available again.
Those are not necessarily the same moment.
Testing This Against a Real Local Model
I've been experimenting with this problem while building MoFlux, an admission-control system for shared workloads.
One of the useful milestones was running the controller against a real local Ollama instance instead of only synthetic workloads.
The first result wasn't interesting because MoFlux somehow made the model faster.
It didn't.
The interesting result was that the admission policy could operate in front of real local inference and observe the same contention problem that appears in simulations.
That changes what can be tested.
Instead of asking only:
Does the controller make the correct decision?
we can ask:
What happens to real inference when interactive and batch workloads
compete for the same model?
Now measurements such as these become meaningful:
- Time to first token
- End-to-end latency
- Interactive success rate
- Batch throughput
- Admission rejection rate
- Capacity restoration time
- Protected-capacity violations
This is where benchmarks start becoming more useful than raw throughput numbers.
Throughput Is Not Always the Goal
Imagine two configurations.
Configuration A completes 110 requests per minute.
Configuration B completes 100.
At first glance, A wins.
But suppose under A:
- interactive latency becomes unpredictable,
- user-facing requests frequently miss their SLO,
- background work dominates capacity during bursts.
Configuration B might complete fewer total requests while completing substantially more useful requests within their objectives.
That's the difference between throughput and goodput.
For shared inference infrastructure, the question isn't simply:
How many generations can this GPU complete?
A more useful question is:
How much useful work can this GPU complete while preserving the
guarantees of the workloads sharing it?
That is a capacity-allocation problem.
Why This Matters for Local Inference
Local inference is becoming increasingly practical.
Developers are running models on workstations, edge machines, private servers, and internal GPU clusters.
The first optimization instinct is often hardware-oriented:
Need more throughput?
Buy a faster GPU.
Sometimes that's the right answer.
But if the actual problem is uncontrolled contention between workloads, more hardware can simply postpone the same failure mode.
Before scaling hardware, it's worth understanding:
Who is consuming capacity?
How much are they allowed to consume?
Which workloads need protection?
Can idle capacity be borrowed?
How quickly can it be reclaimed?
What happens when demand suddenly changes?
Those questions exist whether your inference server is a laptop running Ollama or a production cluster serving thousands of requests.
The scale changes.
The scheduling problem doesn't.
The Bigger Lesson
A single local LLM request is mostly a model-serving problem.
A shared inference server is a distributed-systems problem.
Once multiple workloads compete for a finite model, GPU, memory budget, or concurrency pool, you need policy.
Rate limiting controls how often work arrives.
Concurrency limiting controls how much work runs simultaneously.
Admission control decides whether the system should accept the work at all, given its current obligations.
And workload isolation decides whose work gets protected when capacity becomes scarce.
You may eventually need another GPU.
But before buying one, send your local model two very different workloads at the same time.
You might discover that the first thing you need isn't more capacity.
It's a better way to decide who gets to use the capacity you already have.
