Why request-level tracing has to happen at the gateway, not inside each app
You wrapped your OpenAI client, logged every request and response, and shipped it. Cool. Now answer three questions without opening a second dashboard: which team burned through your token budget last week, which model is quietly failing at p99 while p50 looks fine, and whether last night's traffic spike was a viral feature or a prompt injection sweep.
If you can't answer those from your current logs, it's not because you logged wrong. It's because you logged from the wrong place.
The per-app blind spot
SDK-level logging tells you what one application did. It says nothing about how that application's usage compares to the other five services also calling models in your org. Three problems show up as soon as you have more than one app talking to more than one model:
No cross-app view. Each app's logs live in isolation. There's no single place to ask "what did we spend on LLM calls across the whole company this month."
No visibility past your own client. Your app logs what it sent and what came back. It doesn't know if a router sent that request to a fallback model, or what got filtered out of the response before you saw it.
Instrumentation debt compounds. Every new service needs its own wrapper. Every model swap means updating instrumentation in N places instead of one.
None of this is a tooling gap. It's a structural one: you're trying to observe a multi-model, multi-team system from inside individual apps that can only see their own slice of it.
Move the observation point upstream
The fix is boring and effective: put an AI gateway between your applications and your model providers, and let it do the logging. Every request from every app to every model has to pass through one chokepoint, so that's where you get complete coverage without touching application code. NeuralTrust wrote up the full mechanics of this in their observability guide, which is worth a read if you want the deeper dive.
What to actually track
Skip the vanity metrics. These are the ones that catch real problems:
- Latency at p50/p95/p99, not average. A request that's fast 95% of the time and takes 8 seconds the other 5% will still show a fine average, and your enterprise customers will still be furious.
- Token usage per request. High variance across similar calls usually means unoptimized prompts or a runaway agent looping.
- Cost per call, attributed by team or app. Without this you can't allocate spend or catch the one service quietly eating your budget.
- Error rate, aggregated across providers. A spike here usually means either a provider incident or a prompt hitting a policy filter.
- Fallback rate. If a big chunk of your traffic is landing on your secondary model, your primary is unreliable or rate-limited, and you should know that before finance does.
- Anomaly detection on volume and cost. This is what actually catches prompt injection campaigns and misconfigured agents before they become a five-figure surprise.
Each of these is only useful in aggregate. Your app's individual error rate tells you almost nothing. The error rate across every app and every model tells you whether you have a provider problem, a prompt problem, or a config problem.
Use OpenTelemetry, don't reinvent tracing
If you're going to trace LLM calls, do it in a format your existing observability stack already understands. OpenTelemetry has published semantic conventions specifically for generative AI, under the gen_ai.* namespace, covering things like the model called, token counts, and operation type. A gateway that emits these spans natively means your LLM traces show up next to your regular service traces in Grafana, Datadog, or whatever you already run. No parallel toolchain for AI.
A minimal span for a chat call looks roughly like this:
from opentelemetry import trace
tracer = trace.get_tracer("llm-service")
with tracer.start_as_current_span("chat gpt-4") as span:
span.set_attribute("gen_ai.provider.name", "openai")
span.set_attribute("gen_ai.operation.name", "chat")
span.set_attribute("gen_ai.request.model", "gpt-4")
response = client.chat.completions.create(model="gpt-4", messages=messages)
span.set_attribute("gen_ai.usage.input_tokens", response.usage.prompt_tokens)
span.set_attribute("gen_ai.usage.output_tokens", response.usage.completion_tokens)
A gateway that does this automatically for every request means you don't hand-roll this block in every service you own.
Where this connects to agent security
Observability and security overlap more than teams expect. An anomaly in token volume or a sudden jump in fallback rate is often the first visible sign of a compromised agent or a prompt injection attempt, not just a cost problem. If you're building or evaluating agents that call tools and APIs on their own, it's worth looking at how the broader agent security space is approaching this, including resources at agentsecurity.com. NeuralTrust's own gateway product page and their open source TrustGate repo are also useful references if you want to see how a gateway-first approach captures this data in practice, including cost-side wins covered in their LLM cost optimization piece.
The takeaway
Stop trying to bolt observability onto each app individually. It doesn't scale past app number two, and it structurally can't see cross-model or cross-team patterns. Put the gateway in the path, emit OpenTelemetry-compatible traces, and track the six metrics above in aggregate. That's the whole trick.
