Microsoft has released a dedicated AI Gateway tier of Azure API Management in public preview, a gateway resource whose control plane is organized around models, MCP servers and tools rather than APIs. It was built as a separate experience rather than another layer of policy on the existing gateway, though the classic and v2 tiers keep the AI gateway capabilities they already had.

The central idea is that teams now front many providers rather than one, and the tooling should reflect that. The preview publishes Foundry-hosted models including OpenAI, Anthropic and Mistral, alongside models in AWS Bedrock, Google Vertex AI and OpenAI directly. All OpenAI-compatible providers share one endpoint path, with the gateway routing on an exact match of the model field, so each published model needs a unique name. Anthropic is handled through a custom provider with Messages API passthrough.

Policies are configured as cards in the portal rather than XML and expressions, covering token and request limits, quotas, Content Safety and model fallback. A gateway provisions in about a minute with no scale units to plan. Telemetry is exported as OpenTelemetry token metrics to Application Insights, Datadog, Grafana, and other destinations the customer controls, and the resource runs in the customer's own subscription and Entra tenant.

For tools, the gateway federates backends from three sources: a remote MCP server by URL, an OpenAPI specification, or a built-in connector covering more than a thousand SaaS applications with no server to host. Each backend's operations become tools, and teams choose per backend whether the gateway authenticates with none, an API key, OAuth 2.0, or a managed identity.

(Source: Techcommunity blog post)

The intended operating model splits central control from team self-service. A platform group connects approved models and tools and publishes them; application teams exercise those assets in a test console and build against them without routing every change through the centre, while the platform group keeps the guardrails and the usage picture.

Reactions from architects and platform engineers have been broadly positive on the consolidation, while raising questions about where the governance boundary sits.

Paolo Perrone, who writes the AI Engineer newsletter on production AI systems, argued in a comment on the launch that the underrated part is putting cost governance at the gateway:

Most teams bolt on rate limiting and spend tracking only after an incident, and centralising it gives "one control plane instead of per-app patches".

Adolph White Jr., an enterprise AI systems architect, raised the sharpest open question, about what happens when an agent run does not finish cleanly:

If an agent produces useful work but the run ends without a clean completion, is that output preserved for auditable review, or does the gateway fail over and retry? He framed the distinction as one between "governing AI traffic and governing the full lifecycle".

Whether authority over what an agent's output may change belongs at the gateway or at the orchestration layer above it is not addressed in the announcement.

Not all of the response was favourable. Sreenivasulu Kandakuru, who works in data and AI at Aer Lingus, called the tier badly needed but judged Azure to be behind AWS and Databricks. Unity AI Gateway, announced at Data + AI Summit in June, extends Unity Catalog to govern models, agents, MCP services and skills at runtime, with hard spend caps, smart routing and content guardrails, and governs external coding agents such as Claude Code and Codex by routing them through a model service. Much of it is also still in beta, and Microsoft has shipped AI gateway capabilities for roughly two years, so the gap is narrower than the comparison implies.

One design decision deserves scrutiny before teams plan an onboarding model around it. A runtime access key is gateway-scoped, reaching every model and every tool published on that gateway. Microsoft's guidance is one key per application, but the blast radius of a leaked key is the whole gateway rather than a single product, and teams using APIM subscriptions today to scope consumers to a set of APIs will not find that boundary here.

The preview posture is worth reading carefully too. Availability is best effort with no SLA, and APIs, telemetry, limits, regions, and pricing can change before general availability. Preview quotas cap models, tools, runtime keys and throughput, with specific limits unpublished. Pricing is to be announced later in the preview, which makes the cost-governance argument the least settled part of the release.

The open question the announcement does not answer is coexistence, and the confusion is already visible. Rajib Mahapatro, an AI engineer and architect, responded to the launch by reporting four months of production use, citing caching, content safety, logging and token limits. Those are the existing capabilities rather than a tier that entered preview days earlier, which is precisely the ambiguity at issue. Microsoft's documentation describes the AI gateway as an extension of the existing API Management gateway rather than a separate offering. Organizations that already built one on Premium or Standard v2 have no published guidance on whether those investments carry across, run alongside, or migrate into the new tier.

The AI Gateway tier is available now in East US 2 and Sweden Central, free during preview, with labs covering how to publish a governed Foundry model and watch token rate limits in action.