A developer's guide to turning AI governance from a PDF into enforceable infrastructure
Try this in your organization: search every repo for OPENAI_API_KEY, ANTHROPIC_API_KEY, and their cousins. Count the hits. Then ask who can tell you, right now, which of those keys sent customer data to an external model last week.
In most companies, nobody can. There is usually an AI acceptable use policy somewhere, signed off by legal and security. But a policy is a document. It does not inspect a prompt, block a request, or write a log line. Every team that integrates an LLM makes its own calls about redaction, logging, and access. The result is dozens of slightly different interpretations of the same policy, or none at all.
That gap between what the policy says and what the code does is the real governance problem. NeuralTrust makes the case that an AI gateway is the layer that closes it, and the argument holds up well from an engineering point of view. Here is the condensed version, focused on what you actually build.
Why governance belongs in the network path
You can try to enforce AI rules inside each application. It works for one service. It falls apart at ten, because every app reimplements PII detection, every team logs a different shape of data, and nobody updates all of them when the policy changes.
An AI gateway moves those controls into a single proxy that sits between your applications and every model provider. Because all traffic passes through it, it is the one component that sees every prompt and every completion, regardless of which team, SDK, or model produced it. Change a rule once and it applies everywhere.
For most codebases the migration is small. OpenAI-compatible SDKs let you override the base URL, so pointing a service at a gateway is often just a config change:
import os
from openai import OpenAI
# Before: the app called the provider directly with a raw provider key.
# After: the app calls the gateway, which holds the provider credentials
# and applies policy before forwarding the request.
client = OpenAI(
base_url=os.environ["AI_GATEWAY_URL"], # e.g. https://ai-gateway.internal/v1
api_key=os.environ["AI_GATEWAY_KEY"], # scoped key issued by the gateway
default_headers={"X-End-User": "user-4821"}, # header name depends on your gateway
)
resp = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Summarize this support ticket..."}],
)
A useful side effect: provider keys disappear from application config entirely. Apps hold gateway-issued keys that can be scoped, rotated, and revoked from one place.
What to enforce on the request path
Once traffic flows through a single point, you can inspect it before it reaches a model. These are the controls that pay off first.
- Sensitive data redaction. Detect personal data, card numbers, and credentials in outgoing prompts and mask them before they leave your perimeter. This is the practical way to respect GDPR's data minimisation principle in Article 5(1)(c), which says personal data should be limited to what is necessary for the purpose.
- Prompt injection filtering. The 2025 edition of the OWASP Top 10 for LLM Applications lists prompt injection as LLM01, the top risk. Filtering at the gateway gives every app the same baseline defense, including internal tools that would never get a dedicated security review.
- Scope enforcement. A support bot should answer support questions. Gateway rules can reject requests that fall outside an application's declared purpose, which blocks users from turning it into a free general-purpose assistant.
- Data-aware routing. Requests carrying regulated data can be forced to a self-hosted or region-restricted model. Developers no longer have to write that routing logic into every service.
Identity is more than a shared API key
A single key shared by a whole team makes every request look identical. You cannot tell an intern's experiment from a production batch job, and you cannot apply different rules to each.
Gateway-level access control makes model access identity-aware. You can set token quotas per user, restrict which models each application may call, and cap how many tool calls an autonomous agent can make in a session. Token budgets also work as a scope signal. An app that suddenly burns through its budget is often handling prompts it was never meant to see, or an agent is stuck in a loop.
Agents raise the stakes because they act on behalf of users and chain calls across tools and other agents. If the gateway cannot carry the original user's identity through each hop, your logs will show that "the agent" did something without showing who authorized it. For a deeper look at threat models and controls specific to autonomous systems, AgentSecurity maintains frameworks and a glossary worth bookmarking.
Logs are the compliance artifact
Auditors do not accept "we have a policy." They want evidence that controls actually ran. A gateway produces that evidence as a byproduct, because every request already carries the metadata you need. A useful record looks something like this (illustrative schema, not a specific product's format):
{
"timestamp": "2026-09-14T10:32:07Z",
"app": "support-assistant",
"end_user": "user-4821",
"model_requested": "gpt-4o",
"model_served": "gpt-4o",
"provider_region": "eu",
"tokens": {"input": 812, "output": 164},
"policies": [
{"rule": "pii_redaction", "action": "masked", "entities": ["EMAIL"]},
{"rule": "prompt_injection", "action": "pass"}
],
"latency_ms": 940
}
That record maps onto real regulatory obligations. Article 12 of the EU AI Act requires high-risk AI systems to support automatic logging of events over their entire lifetime. A gateway will not make a system compliant by itself, but it gives you consistent, centrally collected logs instead of a patchwork of app-level ones. Store them somewhere tamper-evident and ship them to your SIEM so AI traffic shows up next to the rest of your security telemetry.
The same data drives day-to-day operations too: p95 and p99 latency per model, cost per team, fallback rates, and anomaly alerts. NeuralTrust's guide to LLM observability with an AI gateway covers which of those metrics are worth tracking and why app-level logging misses them.
A rollout path that doesn't stall
You do not need every control on day one. A sequence that works in practice:
- Route all model traffic through the gateway, even with no rules enabled. Visibility comes first.
- Inventory each application. Note which models it calls, what data it handles, and who uses it.
- Turn on PII redaction and prompt injection filtering for apps that touch customer or regulated data.
- Replace shared provider keys with scoped gateway keys and set per-user and per-app budgets.
- Connect gateway logs to your SIEM and review the first month of traffic for surprises.
Step 1 alone usually surfaces at least one integration nobody knew existed.
Where TrustGate fits
TrustGate is NeuralTrust's gateway for this layer. It routes calls to OpenAI, Anthropic, Azure OpenAI, and self-hosted models, and it also covers MCP tool calls and agent-to-agent handoffs. It forwards end-user identity through each hop and supports per-agent and per-tool RBAC. You can run it as SaaS, in a hybrid setup where the data plane stays inside your environment, or fully on-premises on Kubernetes.
Whatever tool you pick, the principle stays the same. Governance is only real when it runs on every request. Put it in the one place every request already passes through.
