- An LLM gateway is one control point in front of every model provider, so routing, keys, cost tracking, caching and logging live in one place instead of scattered through your services.
- It centralises five things: routing and failover across providers, key management with per team quotas, cost attribution, caching and guardrail or logging hooks.
- Every gateway adds a network hop. The honest trade is a little latency and a new dependency in exchange for control you would otherwise hand code into every service.
- A hosted gateway can become a single point of failure in front of the very providers you adopted for redundancy, which is the failure mode teams miss when they buy one.
- The clearest payoff is a routing policy: a cheap model for classification, a mid model for drafting, a premium model for the final answer, with a fallback chain, which can cut the cost of a feature by most of its bill.
You are on your third model provider. One handles your cheap classification calls, one drafts copy, one does the reasoning nobody else gets right, and the API keys for all three are copied into four different services. Nobody can say which feature is burning the budget, because the spend is split across three dashboards that do not agree. Then last month one provider rate limited you at the wrong moment, a service that only knew how to call that one provider fell over, and the product went down for an hour. Every one of those problems has the same shape. Each service talks to providers directly, so there is no single place to route, count, cache, or fail over. An LLM gateway is that single place.
What does an LLM gateway actually do?
It sits between your application and the model providers and becomes the one point every call passes through. That position is the whole value, because a control point is where you put the things that should be consistent across every call and are painful to reimplement in each service. There are five of them, and a good gateway owns all five.
Underneath all five sits one enabling trick, a unified API. Because the gateway speaks a single request format, usually the widely supported OpenAI shape, your services call the gateway the same way no matter which provider ultimately answers. That is what lets routing and failover work at all, since a call written once can be sent to any provider without a rewrite, and it is what LiteLLM means by drop in OpenAI compatibility across more than a hundred models. Without that single shape, every provider swap is a code change, and the other four responsibilities have nothing consistent to attach to.
These are not abstract. The open source LiteLLM proxy lists exactly this set in its own words: a unified API across more than a hundred models with drop in OpenAI compatibility, virtual keys for access control, cost tracking per project and user, load balancing across deployments, guardrails for policy, and logging through observability callbacks. A hosted option like Cloudflare AI Gateway describes the same job from the managed side: analytics on requests, tokens and cost, logging, caching that serves a request straight from cache instead of the provider, rate limiting, and request retry with model fallback, with those core features available on all plans. Two products, one shape. The table below turns the five responsibilities into the concrete question that matters, which is what each one replaces in your own code.
| Capability | What it replaces in your code | What it costs you |
|---|---|---|
| Routing and failover | Per service retry loops and hardcoded provider choice | One network hop, plus a routing config to maintain |
| Key management and quotas | Keys pasted into every service, no per team limits | A secret store the gateway must guard well |
| Cost attribution | Three provider dashboards that never reconcile | Nothing, this is close to free upside |
| Caching | An ad hoc cache bolted onto one service | Cache invalidation, and stale answers if misused |
| Guardrail and logging hooks | Copy pasted logging in each call site | Latency if a guard runs a second model inline |
Read the last column honestly, because it is the part vendors skip. Every gateway adds a hop between your service and the provider, so it adds latency, and it becomes a dependency your calls now route through. The trade is usually worth it, since the control you get back is real, but it is a trade and not a free win. Cost attribution is the one row that is pure upside. Once every call carries a team or project tag through one place, the question of which feature is burning the budget stops being a mystery, and if you have never priced what those calls add up to, our breakdown of what a real feature costs in LLM API pricing is where the numbers live.
Two of the five deserve a closer look, because they are where a gateway earns or loses your trust. Key management is the unglamorous one that pays off first. When every service holds its own copy of every provider key, rotating a leaked key means hunting it down across four codebases, and there is no way to say that this team may spend only this much. A gateway replaces the scattered keys with virtual keys it issues and controls, so you rotate in one place and set a per team or per project quota that the gateway enforces before a runaway job empties the budget. LiteLLM builds this in as virtual keys with cost tracking per project and user, which is the difference between a spend limit you hope holds and one the system actually enforces.
Caching is the one that looks like free money and can quietly hand a customer the wrong answer. A gateway can serve a repeated request straight from cache instead of paying the provider again, which Cloudflare's gateway does at the edge, and for identical cacheable requests that is a real saving on both cost and latency. The trap is that a language model request is rarely identical twice, and a cache keyed too loosely will return a stale or mismatched answer to a request that only looked the same. Cache exact cacheable calls, set a sane time to live, and never cache anything personalised, because the failure here is not a slow response, it is a confident wrong one served instantly.
Guardrail and logging hooks are the fifth, and their cost is latency you can see. A guard that inspects a prompt or a response by calling a second model adds that model's round trip to every request it runs on. That can be the right call for a high risk feature and the wrong one for a cheap high volume path, so a good gateway lets you scope which calls run the guard rather than taxing all of them. Logging is cheaper, but it is also where the gateway and your observability stack begin to overlap, which is the question we come back to at the end.
How much latency does a gateway add?
One hop, and how much depends on where the gateway runs relative to your service and the provider. A gateway you self host next to your application adds little more than a local network call, since it is close to you and still has to reach the provider either way. A hosted gateway at someone else's edge can add or remove time depending on geography, and caching can make a cached call faster than the direct one it replaces. The number that matters is not the average, it is the tail, because a gateway that is quick on the median and occasionally stalls becomes the slowest part of a request at the worst moment. Measure the gateway's own added latency as a distinct span, separate from the model's time, so that when a call is slow you can tell whether the gateway or the provider caused it. That separation is exactly the kind of signal an observability trace should carry, which is why the two tools end up side by side.
Should you build, buy, or self host?
Three options, and each fails in its own way, so the right question is which failure you can live with. Building your own thin router inside your app keeps you in control and adds no new service, but it quietly grows into a second product nobody owns, and the day you need cross provider failover you are writing a gateway anyway without meaning to. Self hosting an open source gateway gives you the full feature set and keeps every request inside your own infrastructure, at the cost of running and updating one more stateful service that now sits on your critical path. Buying a hosted gateway is the fastest to stand up and the least to operate, and it carries the failure mode teams most often miss.
| Approach | Three ways it fails | Best when |
|---|---|---|
| Build it yourself | Scope creep, no owner, reinvents failover badly | One service, one provider, simple needs |
| Self host open source | Another service to run, patch lag, a stateful hop you own | You want control and data stays in house |
| Buy hosted | Single point of failure, vendor lock, data leaves your walls | Small team, speed matters more than control |
The catch in the last row deserves a sentence of its own. You adopted several providers precisely so that one outage would not take you down. If every call then routes through a single hosted gateway, you have put a single point of failure directly in front of your redundancy, and when that gateway has a bad day your carefully diversified providers are all unreachable at once. That does not make hosted gateways wrong, plenty of teams accept it, but you should accept it on purpose, with a plan for the gateway itself failing, rather than discover it during the outage.
There are ways to soften that trade without giving up the gateway. You can keep a direct path to at least one provider as a break glass route your application switches to if the gateway is unreachable, so a gateway outage degrades you to one provider rather than to zero. You can self host the gateway inside your own infrastructure, so its fate is tied to systems you already run rather than a third party's edge. Or you can run the gateway in more than one location behind your own load balancer, which rebuilds the redundancy the single hop removed. Each adds work, and the right amount depends on how much an hour of downtime costs you, a number worth knowing before you choose.
What does a real routing policy look like?
This is where a gateway pays for its hop. Instead of sending every request to your most capable and most expensive model, you route each call to the cheapest model that can do that specific job, and you write the fallback chain once, in the gateway, rather than in every service. A worked policy for a content feature looks like three tiers. Classification, deciding what kind of request this is, goes to a small fast model, because the job is easy and the volume is high. Drafting goes to a mid tier model, which is good enough for a first pass. Only the final, customer facing answer goes to the premium model, because that is the one place the quality difference is visible. Each tier names a fallback, so if the premium provider rate limits you the call drops to the next best model instead of failing.
The cost delta is the point. If classification and drafting are the bulk of your call volume, and you move them off the premium model onto models that cost a fraction as much, the total bill for the feature falls by most of its value, while the one call that actually needs the premium model still gets it. Put rough numbers on it to see the shape, treating these as an illustration rather than a quoted benchmark. Say a feature makes a thousand calls a day, and eight in ten are classification or drafting while two in ten are the final answer. If you sent all thousand to the premium model you would pay the premium rate a thousand times over. Route the eight hundred cheap calls to models that cost a small fraction of the premium, and you keep the premium rate on only two hundred calls, so the daily bill drops toward a fifth of where it started, while the answers customers actually read stay the same because they still run on the premium model. The exact saving depends on your traffic mix and the price gap between tiers, which is why the policy is worth writing down and measuring rather than guessing, but the direction holds: most of an LLM feature's cost hides in the calls that never needed the expensive model.
The gateway is what makes this a config rather than a rewrite, and the fallback behavior is a real feature, not a promise. LiteLLM's routing documentation spells out the mechanics: strategies like simple shuffle, latency based, usage based and cost based selection, retries with backoff on rate limit errors, and automatic cooldowns that pull a failing deployment out of rotation after a threshold before re picking a healthy peer. That cooldown is exactly what would have kept the hour long outage from happening. Choosing which model sits in each tier is its own decision, and our grid for choosing a model rather than reading a leaderboard is the tool for that part.
Isn't this the same as observability?
They overlap, and the overlap confuses people, but the line is clean once you see it. A gateway sits in the request path and gets cost, caching and traffic across providers almost for free, because it is already in the path and can count every call as it passes. It sees the outside of the call: which model, how many tokens, how much it cost. What it does not see is inside your application, the retrieval step that fed the prompt or the tool result that poisoned the answer, because those never pass through it. That is the job of an observability tool, which watches the full trace inside your code. The two are complementary, and the reason teams end up running both is that each sees what the other cannot. We drew the same line from the other side in the piece on what to record for LLM observability, and the short version is that the gateway owns the request path while observability owns the trace.
So the decision order is simple. If your pain is scattered keys, no cost visibility, and a provider outage taking you down, you have a gateway problem, and a gateway is the smallest change that fixes all three at once. If your pain is that answers are sometimes wrong and you cannot tell why, that is not a gateway problem and a gateway will not fix it. Most teams that run several providers eventually want both, but they are different tools solving different problems, and buying the wrong one first is a common and expensive way to learn the difference. If you are picking the AI stack for a commerce build and want pricing you can predict rather than a floating meter across three providers, our pricing for the MaShop builder shows the flat credit approach that sidesteps this whole coordination problem for a small team.