What finance, healthcare, and government actually require from your inference stack, and why the EU timeline just moved
Somewhere in your stack there is a line of code that picks a model endpoint. Maybe it reads a config value. Maybe it is hardcoded to an OpenAI or Bedrock URL. That line is where data sovereignty is either enforced or lost, and no amount of policy documentation upstream changes it.
This is the part that gets missed when engineers hear "AI compliance." The regulators in finance, healthcare, and government are not asking for a slide deck. They are asking whether you can prove where a specific inference ran, what went into it, what came out, and which model version produced it. Those are runtime properties. You either capture them or you do not.
The three sectors converge on the same technical requirements and diverge completely on the paperwork. Worth separating the two.
The shared engineering requirement
Strip away the acronyms and every regulated deployment needs four things:
- Jurisdiction-aware routing. A request carrying regulated data must resolve to an endpoint in an authorized legal territory. Not a preference. A hard constraint that fails closed.
- Content classification before dispatch. You cannot route on data you have not inspected. Detection has to happen before the request leaves your perimeter, otherwise you are logging a breach rather than preventing one.
- Inference-level audit records. Application logs are not enough. You need input, output, model version, and timestamp bound together per call.
- Pre-deployment adversarial testing with documented results, because "we tested it" is not a control until it produces an artifact.
Here is roughly what the routing layer looks like in practice:
routes:
- name: eu-regulated
match:
classification: [pii, phi, financial_record]
subject_jurisdiction: EU
endpoint: https://llm.internal.eu-west-1/v1
on_no_match: deny # fail closed, never fall through to public
audit:
retain_days: 2555 # 7 years
fields: [input_hash, output, model_version, ts, policy_id]
- name: general
match:
classification: [none]
endpoint: https://api.public-provider.example/v1
The on_no_match: deny line does more compliance work than most governance programs. If no compliant endpoint exists for a given classification, the request fails. It does not quietly degrade to the cheap public API.
For a deeper treatment of why this belongs at the gateway rather than scattered across every integration, NeuralTrust's write-up on how AI gateways maintain data sovereignty covers the enforcement architecture.
Where the three sectors diverge
Finance. DORA (Regulation (EU) 2022/2554) has applied since January 2025 and treats AI classified as a critical or important ICT function as regulated infrastructure. Article 28 is the one that bites engineering teams, because it extends oversight to your third-party cloud provider, not just your own policies. On retention, the commonly cited five-to-seven year window comes from MiFID II Article 16(7), which requires records of relevant communications for five years, extendable to seven at a competent authority's request. MiFIR Article 25 separately requires five years of order and transaction data. If a model influenced a trade, reconstructing that decision means storing the inputs and the model version, not just the outcome.
Healthcare. HIPAA has no AI-specific carve-out and does not need one. If your system touches protected health information, the Security Rule technical safeguards at 45 CFR 164.312 apply as written. The practical trap is that sending a clinical note to an external LLM endpoint is a data transmission under that rule. You need a Business Associate Agreement with the provider, and a BAA does not resolve the separate question of whether that provider's infrastructure sits under foreign legal jurisdiction. For clinical decision support, the FDA's AI-enabled medical device software guidance adds a Predetermined Change Control Plan requirement for models that adapt after deployment, which describes most production ML.
Government. FedRAMP is the entry ticket for cloud-hosted AI serving US federal agencies, assessed against NIST SP 800-53 Rev 5 and its 20 control families. The two most AI-relevant privacy controls are SI-19 (De-identification) and PT-3 (PII Processing Purposes). Note that the widely repeated "12 to 18 months to authorize" figure is now dated. FedRAMP 20x, which replaces static documentation with machine-readable Key Security Indicators, has been compressing that materially, and GSA finalized its Consolidated Rules in mid-2026. If you are budgeting a federal go-to-market, check the current program status rather than the older number. Classified workloads remain a separate world of air-gapped infrastructure and cleared personnel.
The EU timeline changed in July 2026
This one matters and a lot of published guidance is now stale. The EU AI Act (Regulation (EU) 2024/1689) classifies credit scoring, medical device software, and law enforcement AI as high-risk under Annex III. Those obligations were due 2 August 2026.
They are not. The Digital Omnibus on AI, Regulation (EU) 2026/1744, entered into force on 27 July 2026 and deferred standalone Annex III high-risk obligations to 2 December 2027, with embedded Annex I systems moving to 2 August 2028. Article 50 transparency duties were not deferred and applied from August 2026.
The deferral is not a reprieve on the engineering work. DORA and HIPAA did not move, and the Article 12 logging design you need for the AI Act is the same design those regimes already demand. Build it once.
Agents make this harder
Everything above assumes a request-response model. Agentic systems break that assumption. An agent chains tool calls, retrieves external content, and takes actions, which means a single user request can fan out into a dozen inferences hitting different endpoints with different data classifications. Your audit trail has to survive that fan-out, and your routing policy has to apply per tool call rather than per session. AgentSecurity maintains a useful threat model and benchmark library for exactly this class of problem, including indirect prompt injection through retrieved content, which is the most likely way regulated data escapes an otherwise compliant deployment.
Practical takeaway
Do not build three compliance programs. Build one enforcement layer with per-sector policy configuration on top. The routing rules, classification thresholds, and retention windows differ. The architecture does not.
If you want the sector-by-sector regulatory detail in full, the original NeuralTrust piece on sovereign AI for regulated industries goes deeper on each framework, and the Agent Gateway product page shows what the enforcement point looks like implemented.
Start with the endpoint selection code. That is where the audit will land.
