Build or buy an LLM gateway for coding agents: the five questions that decide it

If your organisation puts Claude Code or Codex on an API key, a gateway stops being optional at about the point the first invoice arrives without names on it. You need per-developer attribution, a spend cap that actually stops spend, and the ability to change a default without asking four hundred engineers to edit a config file. The question is not whether to route the traffic through something — it is what that something should be.

This is the decision framework we used, written as questions rather than a recommendation, because the right answer genuinely differs by organisation. We ended up building; a team with different constraints should read the same five questions and buy.

What does a gateway for coding agents actually have to do?

Four jobs, and they are smaller than the feature lists suggest. Authenticate a per-developer or per-team credential so the provider key never leaves your infrastructure. Attribute every request to a person, a team and a session. Apply policy — a cheaper model for subagent traffic, a cache TTL, a token ceiling, a budget. Meter what each request cost, in a form your finance team can reconcile.

Anthropic’s own documentation lists three ways to attribute Claude Code spend per user on a cloud provider: OpenTelemetry export from each developer’s machine, a self-hosted Claude apps gateway, or an LLM gateway that tracks spend per key (Manage costs effectively). Only the last two also give you a place to enforce policy, which is why the gateway pattern wins for anyone who wants more than a report.

Notice what is not on the list: multi-provider routing, semantic caching, guardrails, prompt management, evaluation harnesses. Every general-purpose gateway ships them. If your problem is coding agents, you will use none of them, and you will still carry their failure modes.

Question 1: does it change your request bytes?

This is the question that costs the most money and gets asked the least. Anthropic’s prompt caching is prefix-based: a cache entry is matched by the exact serialised content of the prefix, up to a cache_control breakpoint. Two requests whose JSON differs only in key order are, to the cache, two different prefixes.

A gateway that parses a request into a normalised internal representation and re-serialises it for the provider — the standard design for anything that routes many providers behind one API — cannot guarantee byte-for-byte stability across versions, and a change in serialisation order is invisible in every test that checks semantics rather than bytes. The symptom is not an error. It is a cache hit rate that quietly drops and an invoice that quietly rises, because a cache read is billed at a fraction of the input rate while a rebuilt prefix is billed in full.

The evidence that this is a real constraint rather than a theoretical one is that gateways which do translate end up special-casing coding agents back into passthrough. When a project builds a full translation layer and then adds a mode that forwards the client’s bytes untouched for Claude Code, that mode is the design admitting what the traffic needs.

How to check a candidate: send the same request twice through the gateway and directly, and compare the provider’s reported cache_read_input_tokens on the second call. If the gateway’s number is lower, it is rewriting your prefix.

Question 2: what happens when it fails?

A gateway for coding agents sits on the critical path of your engineers’ working day, which puts it in a different reliability class from a gateway serving a product feature. The question to ask of any candidate is what its failure modes degrade toward.

There are three answers, in descending order of how much you will regret them. Degrade to passthrough: an internal error forwards the original request and you lose measurement for that call. Degrade to error: the request fails and the developer’s session breaks. Degrade to retry: the gateway re-sends, and if the failure was in the response path you pay twice for one answer.

Most general-purpose gateways are built for the second and third, because for a product feature a failed request is better than a wrong one. For coding agents the calculus inverts: nobody is served by an outage of the tool, and losing one row of a spend log costs you a rounding error in a monthly report.

How to check: kill the gateway’s policy store or its database while a session is running. If the session dies with it, the gateway’s dependencies are your dependencies.

Question 3: how big is the blast radius of its dependencies?

A gateway holds live provider credentials inside your network, which makes its supply chain part of your security posture. In March 2026 the litellm PyPI package shipped a compromised release that exfiltrated credentials for about forty minutes before it was pulled. That is not an argument against the project — a popular package is a target precisely because it is useful — but it is the argument for counting what you are installing.

Anthropic’s documentation puts the same point carefully when it names LiteLLM as the gateway several large enterprises reported using: the project “is unaffiliated with Anthropic and has not been audited for security.” Whatever you choose, someone in your organisation should be able to answer what runs inside it.

How to check: count the transitive dependency graph of each candidate and ask who would have to review it after an incident. The number is usually between several hundred and several thousand packages. Compare that against the number of people you have who could read them.

Question 4: is the policy surface the one you need?

Policy for coding agents is a short list, and it is not the list most gateways optimise for. The levers that move an agentic bill are documented in Uber’s public account of running AI tooling at scale and match what the harness vendors recommend: a cheaper model for subagent traffic, a cache TTL matched to how developers actually work, reasoning effort matched to task difficulty, compaction before the context becomes the cost, and tool schemas kept out of context until they are needed. We wrote up which of those levers are configuration rather than engineering.

Against that list, the feature depth of a general-purpose gateway is mostly irrelevant: routing rules across a dozen providers, semantic caching, prompt templates, evaluation hooks. You will configure none of it, and you will still read its docs to find the three settings you need.

The opposite failure exists too. A gateway that only routes and meters, with no way to rewrite a request, cannot apply any of these levers — it can only report that you are spending money.

How to check: write down the five policies you intend to run in your first quarter. Then find each one in the candidate’s documentation. If more than one requires a plugin you would have to write, you are building anyway, only inside someone else’s abstraction.

Question 5: can you reproduce its numbers?

If you plan to make decisions with a gateway’s cost figures — or, worse, to pay someone a share of savings computed from them — the numbers have to be reconstructible from something you control.

Three properties make that possible. The record is per request rather than per day, so an anomaly can be traced to a session rather than a date. It is in your infrastructure, in a format you can read without the vendor. And it is priced from the provider’s own reported usage, so the arithmetic can be checked against an invoice rather than against a dashboard.

We wrote separately about why a vendor’s per-developer benchmark is not a baseline and what a savings claim has to survive — the same discipline applies to a gateway’s own reporting.

How to check: ask a candidate to show you one request’s cost, end to end, and reconcile it with the provider’s billing for that request. If the answer involves a rollup, the number cannot be audited.

So when should you buy?

Buy when your problem is genuinely multi-provider. If a dozen teams call five providers from twenty services, the translation layer you would otherwise write is the product, and LiteLLM in particular is good at exactly that job — it exists because the problem is real.

Buy when nobody on the team wants to own a proxy on the critical path. That ownership is not free: someone carries the pager for a component that, when it fails, stops engineers from working.

Buy when the levers you need are already in the product and you can name them. The cost of adopting a gateway is dominated by learning it, not by running it, and a feature you can configure today beats one you could build next quarter.

Build when the scope is narrow and the constraints are sharp: two harnesses, two providers, byte fidelity that decides your cache bill, a policy list you can write on one page, and a reliability requirement that says the failure mode must be “forward the bytes.” That combination is small enough that a from-scratch implementation is not the ambitious choice — it is the conservative one, because the alternative is carrying a general-purpose system’s failure modes to solve a specific problem.

That is the position we ended up in, and it is why decost runs its own passthrough gateway inside the customer’s network rather than shipping a fork of someone else’s. The measurement it produces is the part we actually sell — savings are per-request counterfactuals that under-claim rather than inflate — and that only works if the record underneath it is one we can defend line by line.

Frequently asked

Does a gateway add latency to a coding session? A passthrough gateway adds one network hop plus the cost of an in-memory credential check; the dominant term is the provider’s own time to first token. A translating gateway additionally parses and re-serialises the request body, which for the large prefixes typical of coding agents is the part worth measuring before you adopt one.

Will a gateway break Claude Code or Codex when the vendor ships a new feature? It depends entirely on the answer to question 1. A gateway that forwards bytes carries new fields, headers and content blocks it has never heard of. A gateway that parses into a fixed schema drops or rejects them until it ships support, which is why “the provider shipped and the proxy 400s” is a recurring pattern in proxy issue trackers.

Can we just use OpenTelemetry instead of a gateway? For attribution, yes — Anthropic documents OTel export as the option that works on every provider. What it cannot do is enforce: telemetry describes what happened, and a budget that stops spend has to sit in the request path.

Is a self-hosted gateway a compliance win or a compliance problem? Usually a win, because the provider credential stays in your network and the request log never leaves it, but it moves the audit burden onto you. The question a security reviewer will ask first is what the gateway’s dependencies are, which is question 3 above.