Azure OpenAI Model Routing Cost: Model Router vs AI Gateway
Azure OpenAI model routing cost, checked against Microsoft Learn: what model router and API Management charge for, where each stops, what to verify.
Neither model router nor any Microsoft gateway in front of it will stop your bill at a fixed number. Azure OpenAI in Microsoft Foundry Models has no native hard dollar limit: Plan and manage costs for Microsoft Foundry says “While OpenAI has an option for hard limits that prevent you from going over your budget, Azure OpenAI doesn’t currently provide this functionality.” A separate post covers which AI spend controls stop spending and which only alert. In an Azure OpenAI model routing cost plan, the limits this article covers either count tokens or estimate spend. In the title, AI Gateway covers two of the four things that carry the name: the AI gateway capabilities in API Management and AI Gateway (preview).
Our order: model router first, the AI gateway capabilities in API Management once several apps share a deployment, and AI Gateway (preview) only as a pilot.
Four things share one name
After describing the AI features built into the API Management service, Microsoft’s AI Gateway and Azure API Management service page warns: “The phrase AI gateway in API Management documentation can refer to those capabilities rather than the AI Gateway product.”
| Name | What it is | Status |
|---|---|---|
| AI gateway capabilities in API Management | Capabilities on existing API Management tiers | Available |
| AI Gateway (preview) | A dedicated product with its own resource type, Microsoft.ApiManagement/aigateways | Public preview |
| AIGateway SKU | The earlier AI Gateway tier on Microsoft.ApiManagement/service | Deprecated |
| AI Gateway in a Foundry resource | Configured inside a Foundry resource and backed by an API Management v2 instance | Preview |
Sources:
- AI gateway in Azure API Management calls the first “a set of capabilities that help you manage your AI backends effectively” and says “it’s not a separate offering.”
- What is AI Gateway? states: “AI Gateway is currently in public preview.”
- Migrate from the AI Gateway tier in API Management (preview) says the AIGateway SKU is deprecated, new resources of that SKU can’t be created and existing ones “will continue to work until January 15, 2027.”
- Configure AI Gateway in your Foundry resources covers the fourth, and the Foundry portal GA overview, dated 2026-08-14, lists it as “Preview”.
Takeaway: before you read any pricing or vendor page, write down which of the four it means.
What model router decides, and what it leaves to you
Model router is a trained routing model. How model router works calls it “a purpose-built ML model optimized for fast inference” and adds “It is not an LLM itself.” It also says “The selected model is always disclosed in the API response via the model field.” Per-request routing metadata and session affinity are preview features.
- Status. What’s new in model router lists “Model router GA version” under November 2025.
- Reach. Version 2025-11-18 “now supports 35 models” and model router “is now available in 32 Azure regions.”
- Deployment types. Model router for Microsoft Foundry names Global Standard and Data Zone Standard deployments and no others.
It has three modes. Balanced, the default, looks within a small quality band of the best model and “picks the most cost-effective model.” Cost widens that band, and Quality “picks the highest quality rated model for the prompt, ignoring the cost.” Microsoft’s bands are “for example” figures, not guarantees: 1% to 2% for Balanced and 5% to 6% for Cost.
Four routing decisions stay with you:
- The model subset. It is also your fallback set, so “select model subsets with at least two models to benefit from the fallback capability.”
- The Azure Policy allowlist. Govern model router deployments with Azure Policy says that through the REST API, Azure CLI or ARM templates, “A model router deployment that includes an unapproved model is rejected with a policy violation.”
- Auto-update. When it is on and a new version arrives, “the set of underlying models also changes, which could affect the overall performance of the model and costs.”
- Claude models. You don’t deploy the supported models separately, “with the exception of the Claude models.”
Takeaway: you choose the subset and the version policy. The router only chooses within the boundary you set.
Our assembly: a gateway in front, model router behind
This design is our assembly of documented Microsoft parts, not a Microsoft reference architecture, and we have not run it.
The closest Microsoft text, Use a gateway in front of multiple model deployments or instances, is about a gateway routing across multiple back ends, not about model router: “Azure doesn’t provide a complete turnkey solution or reference architecture for building a gateway that’s focused on routing traffic across multiple back ends. However, API Management is preferred because the service provides a PaaS-based solution that uses built-in features such as back-end pools, circuit-breaking policies, and custom policies if needed.”
Microsoft’s Cloud Adoption Framework governance guidance recommends, but does not require, gateway controls as a pattern: “A generative AI gateway provides centralized cost controls for AI endpoints.”
None of the Microsoft pages we read shows model router working with the llm-token-limit policy, AI Gateway (preview) or Foundry’s AI Gateway (preview). Treat the combination as untested until you test it yourself.
As documented in AI Gateway concepts and the October 16 release note, model aliases in AI Gateway (preview) route by configured priority and weight, not by prompt.
Our design for the request path:
- 1
Caller
An app or agent sends a chat request with its own key or identity.
- 2
AI gateway capabilities in API Management
Checks who is calling, applies the per-caller token limit and emits token metrics. This is the only AI gateway in the path.
- 3
Model router deployment
Scores the prompt text and picks a model from the subset you approved.
- 4
Underlying model
The selected model answers, and its name comes back in the model field.
Takeaway: the API Management AI gateway capabilities decide who may call and how many tokens each caller gets. Model router decides which model answers.
Model router vs AI gateway: which one routes what
In the Azure OpenAI model router vs AI gateway question, AI gateway means two of the four things that share the name: the AI gateway capabilities in API Management and AI Gateway (preview).
| Model router | AI gateway capabilities in API Management | AI Gateway (preview) | |
|---|---|---|---|
| Routes on | Prompt text, per request | Backend pools: round-robin, weighted, priority-based or session-aware | Model alias with configured priority and weight, as documented |
| Status | GA per the Model router GA version heading, November 2025 | Available; core policies carry no preview label and remain supported | Public preview, East US 2 and Sweden Central |
| Limits it enforces | The deployment's TPM rate limit | Token rate and token quota per key | Token limits and cost limits on estimated spend |
| Failover | Built-in automatic failover inside the subset; a single-model subset disables it | Backend pools with priority groups and circuit breakers | Not documented in the pages we read. Microsoft's page on moving AI workloads to AI Gateway names failover among the dependencies that can be a reason to keep a workload in API Management or design an alternative. |
| Where it runs | A deployment in your Foundry resource | Policies on an API Management instance | Its own resource type, Microsoft.ApiManagement/aigateways |
Sources:
- Model router. Use model router says the “rate limit setting applies to all activity to and from the model router.” Model router for Microsoft Foundry and How model router works cover failover.
- API Management capabilities. llm-token-limit covers token rate and quota. Backends in API Management covers priority groups and circuit breakers.
- AI Gateway (preview). Move AI workloads to AI Gateway says “AI gateway capabilities in API Management remain supported.” Cost limits are in the August 28 release note. The in-product release notes are preview and still use the “AI Gateway tier” name. What is AI Gateway? names the two preview regions.
The AI Gateway and Azure API Management service page says “AI Gateway is the primary path for Microsoft’s latest AI gateway capabilities.” It also says “API Management service continues to support comprehensive API management and foundational AI capabilities” and “The products are complementary.” Move AI workloads to AI Gateway adds: “Full AI feature parity between the products isn’t planned.”
Foundry’s AI Gateway (preview), which sits on an API Management v2 instance, has a narrower scope:
- Per the generative AI gateway guide, it is “designed to support ingress to a single Foundry resource and doesn’t span multiple resources.”
- Once you enable it on the resource, new projects get it by default, but its configuration page says “Existing projects must be enabled manually.”
- The Foundry portal GA overview says “Standalone Azure OpenAI resources: Not supported in the new Foundry portal.”
If you still run a standalone resource, you can upgrade from Azure OpenAI to Microsoft Foundry, which What is Microsoft Foundry? says preserves “your endpoint, API keys, and existing state.”
Takeaway: match the tool to the decision: prompt difficulty to model router, each caller’s token allowance to the gateway.
Azure OpenAI model routing cost: what you actually pay for
For rates, see the Model Router pricing page and the Azure OpenAI pricing page, and check current rates.
| Cost line | What drives it |
|---|---|
| Router charge | Model router input charge, on its own meters |
| Underlying models | The models the router selects for each request |
| Gateway runtime | An API Management v2 instance |
| Foundry's AI Gateway (preview), Basic v2 instance | A listed allowance of 100,000 requests; no period or overage cost stated |
| AI Gateway (preview) | No published price |
| Semantic caching | An embeddings deployment plus Azure Managed Redis |
| Telemetry | Azure Monitor Logs, alerting and Application Insights |
| Spillover (provisioned deployments, not model router) | Spilled requests bill at standard token rates |
Sources:
- Router charge. The Azure Retail Prices API lists meters named “Model Routers GL 1M Tokens” and “Model Routers DZ 1M Tokens.”
- Gateway runtime. The generative AI gateway guide says: “All API gateways incur runtime costs that grow as you add reliability, security, and performance features.” The configuration page for Foundry’s AI Gateway (preview) adds: “Disabling a project alone doesn’t delete the gateway or stop API Management charges.”
- Foundry’s AI Gateway (preview), Basic v2 instance. For the instance Foundry creates, Microsoft’s API Management pricing page listed on 2026-11-07 “Free for up to 100,000 requests when created as an AI Gateway in Azure AI Foundry” (the old product name is Microsoft’s). The same configuration page adds: “Basic v2 is designed for development and testing with SLA support.”
- AI Gateway (preview). The same pricing page, under the old tier name, showed “Pricing details are coming soon!”
- Semantic caching. Enable semantic caching lists the embeddings deployment and the Redis instance as prerequisites.
- Telemetry. Plan and manage costs says Azure Monitor Logs and alerting “incur extra costs for those services.” llm-emit-token-metric sends token metrics to Application Insights. Our reasoning: that is one more destination to price.
- Spillover. Spillover for provisioned deployments bills spilled requests “at the associated input token, cached token, and output token rates.”
Our reasoning: the router sends traffic to models with different prices, so the same token limit on a router deployment can mean different spend.
On prompt caching, Model router for Microsoft Foundry says “caching benefits apply only when the same model handles consecutive requests with overlapping prompt prefixes.” Prompt caching adds that on GPT-5.6 and later, “cache writes can incur charges in addition to discounted cache reads.”
Takeaway: price the router charge, the underlying models, the gateway runtime and the telemetry as separate lines, then check each against your own bill.
Where it stops, and what to verify before you trust it
Latency. Model routing latency on Azure OpenAI has no millisecond figure in the pages we read, so measure it. The generative AI gateway guide says a gateway “introduces added latency for each API invocation,” and the Use model router page lists “Router overhead plus underlying model processing” as a cause. For a method, borrow a troubleshooting row for Foundry’s AI Gateway (preview) in Enforce token limits for models: “Call the model directly and compare the result with the call proxied through AI Gateway.”
Model router limits, from the router pages above:
- “The effective context window is limited by the smallest underlying model.”
- Routing uses text input only. Image inputs are accepted, and “Model router doesn’t process audio input.”
- Separate turns of a conversation can land on different models.
- No page we read documents model router with provisioned, batch, Flex or spillover.
Microsoft’s Well-Architected application design guidance, dated 2025-10-08, warns about routers as a pattern, not the Foundry product: “Model routers add flexibility but introduce extra variability.”
Gateway limits:
- llm-token-limit “doesn’t aggregate token counts across the entire instance.”
- Under it, “concurrent or near-concurrent requests can temporarily exceed the configured token limit.”
- “Completion tokens are also estimated when responses are streamed.”
- A rate limit returns “429 Too Many Requests” and a quota returns “403 Forbidden.”
- AI Gateway (preview) cost limits block new requests on cumulative estimated spend, and “a small temporary overshoot can occur.” They are documented under the old “tier” name, so confirm their behavior on the current resource type.
The direct path. The configuration page for Foundry’s AI Gateway (preview) says “All requests flow through the APIM instance once associated.” No page we read says the direct path is blocked, so test it.
For the benchmark, the Use model router page says: “Before you send production traffic to model router, benchmark it against your current model for response quality, estimated cost, and latency.” For telemetry, the generative AI gateway guide says of Foundry default metrics: “there’s latency with the availability of the data.”
Takeaway: don’t route production traffic through a stage until every checklist item for that stage is ticked on your own workload. The direct path and telemetry items apply once a gateway is in the path.
What to switch on first
This order is our design, not Microsoft’s: a cost order for one workload, not a position on whether your organization should require a gateway as a control plane.
- Model router in Balanced mode. Use a subset of two or more models and turn Auto-update off (our recommendation; the docs present Auto-update as a choice). Benchmark first, then follow Microsoft’s advice to “observe the routing distribution in Azure Monitor for a few weeks.”
- The API Management AI gateway capabilities on a v2 tier. Add llm-token-limit and llm-emit-token-metric once more than one app shares the deployment, and pilot first, because this step runs the combination no page we read documents. The v2 tier is our pick; the policy reference lists llm-token-limit on Classic and V2, not on Consumption. Microsoft says a gateway gives you “an opportunity to track usage per client,” but attributing that usage to teams stays your job, which is where token attribution and CI/CD cost gates come in.
- A pilot of AI Gateway (preview). Run it in a supported region if budgets on estimated spend matter to you, and test the combination with model router yourself.
Before you start step 2, write into your design document what each layer may decide: the router chooses models inside the boundary you set, and the gateway rations tokens per caller. Neither caps your bill, so reconcile your Azure OpenAI model routing cost against the invoice before you scale.
Takeaway: turn on one thing per stage, and verify it before the next.
A second pair of eyes before production
A routing setup is easiest to fix before production traffic reaches it. Citizen Development Academy runs a short review: it looks at what your team has built or plans to build with AI, names the top three risks, and recommends one next step.
Book a free 30-minute AI app review with Citizen Development Academy
Stay in the loop
Get new posts delivered to your inbox. No spam, unsubscribe anytime.
Related articles
Foundry Hosted Agents Finally Get Network Egress Controls: Put Them on Default-Deny Today
Azure AI Foundry's hosted agent egress preview confirms agents are untrusted workloads. Here is the default-deny posture to ship, and the rule pattern that works.
The APIM AI Gateway Tier Is Microsoft Admitting LLM Traffic Broke Your Gateway
Azure APIM's AI Gateway tier concedes that request-based governance fails for LLM traffic. What's genuinely new, what's repackaged, and when to adopt.
Why Uniform Agent Governance Fails: A Six-Pattern Estate Audit
Uniform agent governance breaks at scale, and Microsoft's six-pattern guidance reads like a concession. A four-step audit that tests your own agent estate for it.