This article is adapted from Chapter 4 of "AI Governance: The Foundation for Organized
AI in Production" (Zenodo DOI:
10.5281/zenodo.19655870). Original work ©
Fabio Bastos / ThinkNEO.
Most teams deploying AI in production hit the same wall about 90 days in.
The MVP works. The demo impressed stakeholders. But now you have three different LLM
providers, five internal tools calling those providers in slightly different ways, no
unified logging, no cost visibility, and a growing list of "why did it do that?" tickets
from users.
You don't have an AI problem. You have an infrastructure problem.
What you're missing is an AI Control Plane.
## What Is an AI Control Plane?
In traditional distributed systems, a control plane manages how traffic flows — routing
decisions, health checks, policy enforcement — separate from the data plane, which
handles the actual traffic.
The same separation applies to AI systems in production.
Your data plane is the actual inference: prompts going out, completions coming back,
embeddings being generated. Your AI Control Plane sits above that and governs:
- Which model handles which request (routing)
- What policies apply to each call (guardrails, rate limits, cost caps)
- What gets logged and how (observability)
- What happens when a provider fails (fallback, retry, circuit breaking)
- Who is allowed to call what (access control)
Without this layer, your AI infrastructure is a collection of direct API calls scattered
across your codebase — ungovernable at scale.
## The Five Core Functions
### 1. Unified Routing
The control plane abstracts provider-specific APIs behind a single interface. Your
application code calls control_plane.complete(request) — not
openai.chat.completions.create() or anthropic.messages.create() directly.
`python
# Without control plane — provider-coupled, brittle
response = openai.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": prompt}]
)
# With control plane — provider-agnostic, governable
response = control_plane.complete(
request=AIRequest(
prompt=prompt,
policy="customer-facing",
budget_tier="standard"
)
)
- Policy Enforcement at Runtime
Policies define the rules that govern AI calls. They are evaluated at runtime, not
hardcoded in application logic. Policies live in configuration — your governance team can
update them without a deployment cycle.
# policy: customer-facing
guardrails:
input:
- pii_detection: block
- prompt_injection: block
output:
- toxicity_filter: warn
cost:
max_tokens: 2000
preferred_provider: openai
fallback_provider: anthropic
logging:
level: full
retention_days: 90
- Observability as a First-Class Concern
Every AI call routed through the control plane emits a structured trace event. At scale,
this feeds dashboards, cost allocation, anomaly detection, and compliance audit logs.
{
"trace_id": "ai-7f3a9c",
"timestamp": "2026-05-17T14:32:01Z",
"policy": "customer-facing",
"provider": "openai",
"model": "gpt-4o",
"input_tokens": 312,
"output_tokens": 187,
"latency_ms": 843,
"cost_usd": 0.0074,
"guardrails_triggered": [],
"session_id": "user-99142"
}
- Fallback and Circuit Breaking
Providers go down. Rate limits get hit. A model gets deprecated with two weeks' notice.
The control plane handles failure gracefully — application code never implements
retry/fallback logic itself.
Request → Primary Provider (OpenAI gpt-4o)
↓ [timeout / error / rate limit]
Fallback Provider (Anthropic claude-sonnet)
↓ [also unavailable]
Degraded Mode (cached response / queue for retry)
↓ [circuit open > threshold]
Reject with 503 + alert on-call
- Access Control and Quota Management
The control plane enforces:
- Service-level quotas — service X gets 10,000 tokens/minute
- Model access tiers — only approved services can call GPT-4o
- Cost attribution — spend tracked per service, per team, per feature flag
Reference Architecture
┌─────────────────────────────────────────────────┐
│ Application Layer │
│ (APIs, agents, chatbots, internal tools) │
└────────────────────┬────────────────────────────┘
│ AIRequest
┌────────────────────▼────────────────────────────┐
│ AI Control Plane │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────────┐ │
│ │ Router │ │ Policies │ │ Observability│ │
│ └──────────┘ └──────────┘ └──────────────┘ │
│ ┌──────────┐ ┌──────────┐ ┌──────────────┐ │
│ │Guardrails│ │ Fallback │ │Access Control│ │
│ └──────────┘ └──────────┘ └──────────────┘ │
└───────┬──────────────┬───────────────┬──────────┘
│ │ │
┌───────▼──┐ ┌──────▼───┐ ┌──────▼───┐
│ OpenAI │ │Anthropic │ │ Gemini │
└──────────┘ └──────────┘ └──────────┘
When Do You Actually Need This?
You don't need a full control plane on day one. But you need to start building toward it
the moment any of these are true:
- You have more than one LLM provider in use (or plan to)
- You have more than one team making AI calls
- You need to demonstrate cost accountability to leadership
- You're operating in a regulated industry where AI outputs must be auditable
- You've had your first "why did the AI say that?" incident in production
Practical Starting Point
Build incrementally:
- Week 1: Wrap all AI calls in a single internal module — even if it just passes through.
This is your future control plane boundary.
- Week 2: Add structured logging to every call. Token counts, latency, cost, provider.
- Week 3: Externalize your model selection into a config file. Stop hardcoding model
names in application logic.
- Week 4: Add a fallback rule for your most critical path. One rule. Test it.
By the end of the month, you have the skeleton of a control plane — and a codebase that's
actually governable.
Conclusion
The AI Control Plane is not a product you buy. It's an architectural layer you build —
deliberately, as a first-class concern — when you decide to run AI seriously in
production.
Every production AI system eventually needs routing, policy enforcement, observability,
fallback handling, and access control. The question is whether you build that layer on
purpose, or piece it together reactively after your first major incident.
Build it on purpose.
This article is adapted from "AI Governance: The Foundation for Organized AI in
Production" by Fabio Bastos. Available on Amazon KDP | Peer-citable via Zenodo:
10.5281/zenodo.19655870
About the author: Fabio Bastos is Founder & CEO of ThinkNEO, an enterprise AI platform
focused on deploying, orchestrating, and governing AI systems in production. Based in
Bangkok.