Why a great multi-provider gateway still leaves enterprise-grade gaps in prompt injection defense, PII handling, and compliance logging.
If you've put LiteLLM in front of your LLM traffic, you already know why it's popular. One proxy, over 100 providers, automatic failover, per-team spend tracking, rate limiting. It does the job well enough to earn tens of thousands of GitHub stars.
But routing traffic and securing traffic are not the same job, and it's easy to assume one buys you the other.
What's already built in
LiteLLM proxy has a guardrails block in config.yaml, and it plugs into more than 40 third-party guardrail providers. Between them you get prompt injection scanning, PII and PHI masking, secret detection, and basic content policy checks. Each guardrail runs in one of three modes:
pre_call. Runs before the request hits the model.during_call. Runs in parallel with the model call.post_call. Runs after the response comes back, checking input and output.
You wire it up per API key or per team, which is genuinely useful if your internal tools and customer-facing endpoints need different rules. For prototypes and most internal projects, this is enough.
Where it stops being enough
The gaps show up the moment someone asks you to prove your AI system is actually safe, not just configured.
Pattern matching isn't intent detection. Guardrail providers catch known injection signatures and known PII formats. They don't understand what a multi-turn conversation is actually trying to extract. Someone probing for your system prompt across ten harmless-looking messages slips right past pattern matching.
Routing decisions ignore jurisdiction. LiteLLM picks a provider based on cost, latency, and load. It has no concept of "this is EU personal data and shouldn't leave the EU," which means a misconfigured fallback can quietly create a compliance problem.
Callback logs aren't audit trails. Sending events to Langfuse or S3 is application logging. A regulator wants immutable, inference-level records: exact input, exact output, model version, timestamp, verifiable after the fact. Those are different artifacts built for different purposes.
Agent chains are invisible. Native guardrails see the outer API call. Once you're running orchestrator-to-subagent chains, tool calls, and memory access, everything happening inside that chain is a blind spot.
You can't guardrail what you haven't tested for. Configured guardrails only catch categories you already anticipated. OWASP's LLM Top 10 lays out ten attack classes that require deliberate adversarial testing to catch, not passive filtering.
Closing the gap without ripping out LiteLLM
You don't need to replace your gateway to fix this. NeuralTrust's TrustGuard plugs in as a custom guardrail class, so every app already pointed at your proxy gets covered without touching client code.
The integration is a Python file plus a few lines of config. It calls TrustGuard's evaluate endpoint before the model call and again on the response, then enforces whatever verdict comes back: allow, block, or transform.
guardrails:
- guardrail_name: "trustguard"
litellm_params:
guardrail: trustguard_guardrail.TrustGuardGuardrail
mode: ["pre_call", "post_call"]
api_base: os.environ/TRUSTGUARD_API_BASE
api_key: os.environ/TRUSTGUARD_API_KEY
fail_open: false
default_on: true
A couple of things worth deciding upfront:
- Fail-open vs fail-closed.
fail_open: false means if TrustGuard can't be reached, the proxy returns a 503 and nothing gets to the model. That's the safer default. fail_open: true keeps traffic flowing uninspected during an outage, only use it if uptime matters more than guaranteed inspection, and alert on it.
- Inspection scope. Checking only the current turn keeps payload size constant but misses attacks spread across a conversation. Checking the full transcript catches more but grows with every turn, and one flagged message blocks everything after it.
- Streaming changes your enforcement point. With
stream: true, output is already being sent to the user by the time the post-call check runs on the assembled response, so output-side blocking becomes detection after the fact. Input-side checks aren't affected.
Full setup steps are in NeuralTrust's TrustGuard LiteLLM integration guide, which also covers the exact guardrail class code.
One more option worth knowing
If you're not committed to LiteLLM and you're evaluating gateways from scratch, TrustGate is NeuralTrust's own gateway with this kind of security built in natively, rather than layered on. It's a replacement for LiteLLM, not something you'd run alongside it, stacking two gateways just adds latency for no gain.
For a broader look at how AI gateways get hardened in production in general, agentsecurity.com is a solid reference point outside the vendor-specific weeds.
The takeaway
LiteLLM's job is routing, and it does that job well. Security is a separate layer with separate requirements: semantic intent detection, jurisdiction-aware controls, tamper-evident audit records, visibility into agent chains, and adversarial testing before launch. None of that is a knock on LiteLLM, it just means "we have guardrails configured" and "we can prove this system is secure" are two different sentences, and only one of them satisfies a compliance review.
