What an AI gateway is for, and what it is not

An organisation that has written an AI acceptable-use standard and has no gateway has added exposure without reducing any: a document now exists saying what people must not do, and nothing at all observes whether they do it. A gateway is the answer to the question every standard eventually runs into, which is “enforced by what?”

The term needs pinning down, because the market sells at least three different things under it.

The useful definition

An AI gateway is a mandatory path between your applications and any model, wherever the model runs. Everything the definition buys you follows from the word mandatory.

What it should own:

• Routing and allowlisting. Which models, which providers, which regions, which versions. Retiring a model becomes a configuration change rather than a hunt through repositories.
• Identity. Calls arrive with an application or workload identity, not a shared key pasted into an environment file. Provider credentials live at the gateway and never in a client. Revocation works.
• A record. Prompt and response metadata, the model used, the tools invoked, the caller, the cost. Retained under a policy someone has read, with the content itself redacted or omitted where the data classification requires it.
• Data controls on the way out. Classification-aware inspection of what leaves, the boundary at which a request must stay on a self-hosted or regionally-constrained model, and blocking for the categories that must never reach a third party.
• Cost and rate limits, per team, per application, per key. Unglamorous, and the control most likely to be the reason finance supports the project.

What it should not own: anything that requires understanding the application’s business context. A gateway cannot tell whether this user was entitled to this document. That is the application’s authorisation model, and no amount of inspection at the perimeter substitutes for it.

Why it is not a firewall for prompts

Most of the marketing energy in this space goes into prompt-injection detection. It is worth having and it is not a boundary. Detection of adversarial text is a classification problem on unbounded natural-language input, which means a false negative rate that is not zero and cannot be driven to zero by tuning. Any control design that only holds when detection succeeds is a control design that fails on the first phrasing nobody tested.

Treat classifiers as noise reduction and telemetry. Put the boundary where it can be enforced deterministically: in what the model is permitted to invoke, under whose identity, against which data, with which irreversible actions gated behind a human. A gateway is extremely good at the first four. Position it as a filter that makes unsafe model behaviour safe, and it will be believed, which is the actual risk.

The routing decision is a data-classification decision

The most valuable thing a gateway does in a regulated organisation is make one rule mechanical: material of a given classification may only be processed by models meeting a given standard. Public information can go to a frontier model on a commercial API. Client-confidential material may require a contractual arrangement with no training rights and a defined region. Some categories do not leave your tenancy at all.

Written in a standard, that rule is an aspiration that each delivery team interprets differently under deadline. Implemented as routing policy, it becomes a property of the platform, and it produces evidence continuously rather than at the point someone asks for it. That last part is what matters when a regulator, a client’s due diligence team or an internal auditor comes asking.

Build or buy

Both work. A thin service in front of the provider APIs, owned by the platform team, is often the right first step: it establishes the mandatory path, which is the hard part organisationally, and it can be extended. Commercial gateways buy you inspection, policy tooling and multi-provider support sooner.

Neither of those decides the outcome. What decides it is whether the gateway is mandatory: whether egress controls, key management and platform standards make the direct route to a provider API unavailable rather than merely discouraged. A gateway that teams can bypass is a reporting tool, and the traffic you most want to see is exactly the traffic that will not appear in it.

What to do first

1. Establish the mandatory path and centralise provider credentials. Nothing else works until this is true.
2. Get identity onto every call, so the log answers “who” and not only “what”.
3. Add routing by data classification, tied to your existing classification scheme rather than a new one invented for AI.
4. Then add inspection, evaluation and the rest.

Governance follows the control point, not the other way round. An organisation with a mandatory gateway and a two-page standard is in better shape than one with a forty-page standard and a spreadsheet of self-attestations.