LLM Routing Is Three Decisions, and Only One Saves $ Money

Dean Jain

Senior Staff Software Engineer · Enterprise AI, Data & Cloud Architect

September 6, 2026 · 18 min read

LLM RoutingLLM Cost OptimizationAI ArchitectureLLMsAgentic AIFinOps

The router makes all three decisions, but only the middle one splits the traffic. Each stream's width is its share, and that split is the whole saving.

I have been researching and experimenting with LLM cost and token optimization, and routing is where most of the advertised savings are supposed to live. So I went at it properly: the papers behind the headline numbers, the open-source routers, and what each claim is actually measured against. What follows is what I took away, including the parts that disappointed me.

The confusion is expensive. Teams buy a router expecting savings and get reliability instead. “LLM routing” is three decisions stacked on each other, and only one of them has real money in it. Which decision is in play is what decides what to buy, and when to buy nothing.

TL;DR

Three decisions wear the same name. Which endpoint gets called, which model answers at all, and which seller of that model serves it. Different problems, different value.
The gateway layer is plumbing. One endpoint, one key, budgets, logging. It buys portability, not savings.
Provider routing buys reliability. One open-weight model (its files are published, so anyone can run it) has many hosts at different speeds and prices. Failing over between them is the part I would never build myself.
Model routing is where the money is. Berkeley and Anyscale’s RouteLLM cut cost 85% on a standard public test set while holding 95% of GPT-4 quality, sending 14–26% of queries to the expensive frontier model, the biggest and priciest tier a vendor sells.
The router’s own latency is noise. Simple rules add under 1ms, embedding lookups about 5ms, a trained classifier model 50–100ms. The LLM call itself takes 500–2,000ms.
A router earns its place at the second something. The second model, the second team, or an uptime target above one vendor’s SLA. Most teams cross one of those lines within a year. Before that line, a router is a hop and a bill on a problem nobody has yet.

1. Three decisions hiding under one word

Every request that leaves an application faces three questions. Most products answer some of them and market all of them as “routing.”

---
config:
  theme: dark
  fontSize: 16
  themeVariables:
    fontFamily: "Comic Sans MS, Comic Neue, Chalkboard SE, cursive"
    fontSize: "16px"
  flowchart:
    wrappingWidth: 220
    nodeSpacing: 40
    rankSpacing: 45
---
flowchart TD
    Q["💬 A request arrives"]:::neutral
    Q --> L1["🔌 1 · Gateway<br/>which endpoint, whose key,<br/>whose budget?<br/><b>buys portability</b>"]:::obs
    L1 --> L3["🧠 2 · Model routing<br/>which model should<br/>answer this at all?<br/><b>buys cost savings</b>"]:::agent
    L3 --> L2["🏭 3 · Provider routing<br/>which seller of that<br/>model serves it?<br/><b>buys reliability</b>"]:::good
    L2 --> OUT["✅ Answer + metered spend"]:::gate
    classDef neutral fill:#ECECEC,stroke:#8A8A8A,stroke-width:2px,color:#0F172A
    classDef obs fill:#AED6F1,stroke:#2E86C1,stroke-width:2px,color:#0F172A
    classDef agent fill:#FFE08A,stroke:#E8A33D,stroke-width:2px,color:#0F172A
    classDef good fill:#BFEFC8,stroke:#3FA34D,stroke-width:2px,color:#0F172A
    classDef gate fill:#D7C3F2,stroke:#8E5BD0,stroke-width:2px,color:#0F172A

Figure 1: The three decisions, in the order they happen. Most tools sold as "routers" do one or two of these well and mention the third in the marketing.

Layer	The question	What it buys	Who does it
Gateway	Which endpoint, whose key, whose budget?	Portability and control. Not savings	LiteLLM, Portkey, Kong, Cloudflare
Model routing	Which model should answer this?	Cost, up to 85% against always-frontier	RouteLLM, `openrouter/auto`, Azure Model Router
Provider routing	Which seller of that model?	Uptime and price arbitrage	OpenRouter, LiteLLM

Provider routing is the layer most people have never thought about. It exists because an open-weight model is sold by many companies at once. The model’s files are public, so any host can serve it. Llama runs on a dozen hosts at different speeds and prices. “This host of Llama failed, try another host of the same Llama” is a completely different move from “Claude failed, try GPT.” It is also invisible unless someone says it is happening.

2. What actually happens to a request

Behind the abstraction, a well-built router runs the same seven steps every time.

---
config:
  theme: dark
  fontSize: 16
  themeVariables:
    fontFamily: "Comic Sans MS, Comic Neue, Chalkboard SE, cursive"
    fontSize: "16px"
  flowchart:
    wrappingWidth: 220
    nodeSpacing: 40
    rankSpacing: 45
---
flowchart TD
    A["1 · Normalise<br/>one wire format"]:::neutral --> B["2 · Authorise<br/>key · team · budget"]:::gate
    B --> C["3 · Cache check<br/>exact or semantic"]:::good
    C --> D["4 · Pick model<br/>rules · classifier · cascade"]:::agent
    D --> E["5 · Pick provider<br/>drop unhealthy, weight by price"]:::obs
    E --> F["6 · Call + fall back<br/>on 429, 5xx, timeout"]:::warn
    F --> G["7 · Meter + log<br/>tokens · cost · trace"]:::gate
    classDef neutral fill:#ECECEC,stroke:#8A8A8A,stroke-width:2px,color:#0F172A
    classDef gate fill:#D7C3F2,stroke:#8E5BD0,stroke-width:2px,color:#0F172A
    classDef good fill:#BFEFC8,stroke:#3FA34D,stroke-width:2px,color:#0F172A
    classDef agent fill:#FFE08A,stroke:#E8A33D,stroke-width:2px,color:#0F172A
    classDef obs fill:#AED6F1,stroke:#2E86C1,stroke-width:2px,color:#0F172A
    classDef warn fill:#FFE6A8,stroke:#E0A106,stroke-width:2px,color:#0F172A

Figure 2: The request lifecycle. Steps 4 and 5 are the two routing decisions. Everything else is the plumbing that makes them safe to operate.

Step 5 is worth making concrete, because good implementations are more careful than people assume. OpenRouter’s provider selection works like this:

Deprioritise any provider that had a significant outage in the last 30 seconds. Reliability filter first.
Among the remaining cheap providers, pick one weighted by the inverse square of price. A seller at $1 per million tokens is about 9× more likely to be picked than one at $3.
Keep the rest as an ordered fallback chain.

That ordering matters. It is a reliability mechanism that happens to optimise cost, not a cost mechanism that hopes for reliability. Anyone building this in-house gets the order backwards on the first attempt.

3. Model routing is where the money is

The other two layers cost a little and save a little. Model routing is the one with real numbers behind it.

Berkeley and Anyscale’s RouteLLM trained routers on human preference data from Chatbot Arena (now LMArena), the public site where people vote on which of two anonymous model answers is better. The headline result: 85% cost reduction on MT Bench while holding 95% of GPT-4’s quality, with 26% of queries reaching the expensive model, or 14% once the training data is augmented.

Read the conditions, because they carry the argument. MT Bench, MMLU and GSM8K are standard public test sets: open-ended chat, exam questions, and grade-school maths. The 85% is MT Bench alone. The 45% on MMLU came from a different router in the same family, trained on labelled MMLU data, and it still needed 54% of calls. And every one of these figures holds 95% of GPT-4’s quality, not all of it. A router that saves money by accepting a 5% quality loss is a different product from one that matches quality, and that gap is the conversation I end up having with every product owner.

That works because most requests are not hard. Today’s model families run a 5–25× price gap between their cheap and frontier tiers. A large share of production traffic never needed the frontier model at all. Classification, extraction, formatting, short answers. Routing is arbitrage on that gap. Buy the cheap tier, sell the same answer.

Four ways to make the call, with honest overheads:

Strategy	How it decides	Added latency	Best for
Rules	Prompt length, task tag, user tier	under 1ms	Known, well-segmented traffic
Embeddings	Similarity to labelled example queries	~5ms	Mixed traffic, fast to tune
Classifier	A small model reads the prompt and picks the answering model	50–100ms	Traffic that will not segment by rule
Cascade	The cheap model answers first; a bad answer is escalated	50–100ms plus a wasted cheap call	Async and batch work

Set that against a typical LLM call of 500–2,000ms. The router’s own latency is noise. The cost of routing is almost never latency. It is being wrong.

Rules: the signal is already in the application

The signal is usually already in the application, not in the prompt. The code knows the user hit /summarise and not /analyse. It knows the account tier, and whether the response has to satisfy a JSON schema. A classifier that infers the task from prompt text is guessing at something the application was handed for free.

Build it as middleware in front of the call, keyed on token count, the task tag, user tier, whether tools or structured output are required, and the conversation turn. Count tokens with the provider’s own counter, tiktoken for OpenAI or count_tokens for Anthropic. A count from the wrong tokenizer is a wrong route.

The stack is config, not code. LiteLLM’s tag-based routing tags each deployment and matches on tags carried by the request:

model_list:
  - model_name: chat
    litellm_params:
      model: anthropic/claude-haiku-4-5
      tags: ["cheap", "default"]     # untagged requests land here
  - model_name: chat
    litellm_params:
      model: anthropic/claude-opus-5
      tags: ["hard"]

router_settings:
  enable_tag_filtering: True

The caller then sends metadata: {"tags": ["hard"]}, or the header x-litellm-tags: hard, and the proxy picks the deployment. The application keeps the routing logic it already had; the proxy keeps the model names.

Gotchas. Prompt length is a proxy for difficulty, and a poor one. Length predicts cost well and hardness badly. “Prove this is NP-hard” is twenty tokens. Route on capability before difficulty. Strict schemas, long tool chains and 200K contexts are hard constraints, not quality preferences, and a cheap model that cannot emit the schema is a retry rather than a saving. Rules rot in silence. The traffic mix moves, the rule keeps matching, and nothing pages anyone, because every call returned 200.

Embeddings: match the prompt to labelled examples

An embedding turns a piece of text into a list of numbers, positioned so that text with similar meaning lands close by. Compare two of those lists for a similarity score, with no model call involved.

Write 10–30 real example prompts per route, embed them once, then embed each incoming prompt and take the closest match. No model call and no training run. It is the fastest of the three to stand up and the easiest to explain to whoever has to approve it.

semantic-router (Aurelio Labs, MIT) is the library. Keep the encoder (the model that produces the embeddings) running locally. HuggingFaceEncoder runs all-MiniLM-L6-v2 on CPU in single-digit milliseconds, while a hosted embedding API turns a 5ms router into a 150ms network hop and the whole latency argument collapses.

from semantic_router import Route
from semantic_router.encoders import HuggingFaceEncoder
from semantic_router.routers import SemanticRouter

cheap = Route(name="cheap", utterances=[
    "summarise this thread",
    "pull the invoice number out of this email",
    "translate the product description to German",
])
hard = Route(name="hard", utterances=[
    "why did this migration deadlock and how do I avoid it",
    "design a tenancy model for our billing schema",
])

router = SemanticRouter(encoder=HuggingFaceEncoder(),
                        routes=[cheap, hard], auto_sync="local")

choice = router("pull the total off this receipt").name   # -> "cheap"
model = CHEAP if choice == "cheap" else FRONTIER          # None -> FRONTIER

Building it from scratch is about forty lines: sentence-transformers to turn text into vectors, FAISS to find the closest one. Take the library for the threshold handling and route syncing, not for the search.

Gotchas. Handle the no-match case first. The router returns .name = None when no route clears the similarity threshold, and that branch is the whole design. A prompt that resembles nothing in the labelled set is a prompt with no evidence behind it, so it must go to the frontier model. Default it to cheap and the regression surfaces in a support ticket. This classifies topic, not difficulty. Two prompts about billing can differ by an order of magnitude in hardness. Seed the example prompts from production logs, never from imagination. Invented prompts are cleaner, better spelled and better punctuated than the ones that actually arrive. Pin the encoder. Swapping the embedding model silently re-decides every route, so it is a versioned dependency of the policy, not an implementation detail.

Classifiers and cascades: let a model make the call

Take the two words first, because both get used loosely.

A classifier is a small model trained to sort things into buckets. Here the buckets are “cheap model” and “frontier model”, and it reads the incoming prompt to pick one. It decides before anything has been generated.

A cascade runs the other way round. The cheap model answers first. Something scores that answer. A low score sends the request up to the frontier model, and the cheap answer is thrown away. It decides after.

So: a classifier guesses up front, a cascade checks afterwards. Everything else follows from that.

How a trained classifier works. The obvious move is to train a “how hard is this prompt” scorer, and it is a dead end, because nobody has labelled prompts by hardness.

RouteLLM gets around that by asking a different question. Not “is this prompt hard?” but “how likely is the expensive model to beat the cheap one on it?” That question already has millions of labels, and they come from Chatbot Arena, a public site where anyone types a prompt, gets two answers back from two unnamed models, and votes for the better one. Each of those votes is called a battle. Millions of them are published as an open dataset.

Four steps:

Label. Each battle is a prompt and a verdict: the strong model won, or it did not.
Train. Fit a model that turns prompt text into a win rate: one number from 0 to 1, the predicted chance the strong model wins.
Decide. At request time, compare that number to a cutoff. Above it, frontier model. Below it, cheap model. A low win rate means the two would have tied, and a tie means pay less.
Calibrate. Set the cutoff from the budget, not by feel.

RouteLLM ships four ways to do step 2:

Router	How it predicts the win rate
`sw_ranking`	Embeds the prompt, finds the most similar training prompts, then runs a similarity-weighted Elo (the chess rating system) over who won those. No training run; the work happens per request
`mf`	Matrix factorization, the recommender-system trick explained below. The one they recommend
`bert`	BERT is a small, older language model that reads text but does not generate any. Fine-tune it on prompt text and it predicts the verdict directly. Cheap to serve, sees only the words
`causal_llm`	An LLM fine-tuned to emit the verdict as its next token. Strongest signal, heaviest to serve

Why mf is the one they recommend. It is the same math as a film recommender. Netflix learns a hidden vector (a short list of numbers standing in for taste) for every user and every film, and the dot product of the two predicts a rating for a pair nobody has rated yet. RouteLLM learns a hidden vector for every model and every prompt, and the dot product predicts how well that model handles that prompt. The authors recommend it for most uses as “very strong and lightweight”, and it is what their default threshold is tuned against. On raw scores it is not always the best of the four, but it is the cheapest to serve that stays close.

All four routers transfer to model pairs they never saw in training. RouteLLM trained its routers to choose between GPT-4 and Mixtral 8x7B. It then pointed the same routers, with nothing retrained, at two pairs absent from the training data: Claude 3 Opus over Claude 3 Sonnet, and Llama 3.1 70B over Llama 3.1 8B. All four beat random routing on both, by margins of 31% to 57%. That is the property worth having. What these routers learned is which prompts need the stronger model, not what GPT-4 and Mixtral in particular happen to be good at. It is not special to matrix factorization. Every router in the family showed it.

In code it is a drop-in client:

from routellm.controller import Controller

client = Controller(routers=["mf"], strong_model=..., weak_model=...)
client.chat.completions.create(model="router-mf-0.11593", messages=[...])

0.11593 is the cutoff from step 3, not a version number. Every prompt scoring above it goes to the strong model.

How to build the cascade. The scorer is the way. FrugalGPT is a reference: a small DistilBERT scorer (a trimmed-down BERT, fast enough to run on every request) that takes (query, answer) and returns a number from 0 to 1, plus an escalation threshold. Training our own is realistic, because we already hold the data: every logged request, the answer it got, and whether anyone complained.

Gotchas

Routing fights prompt caching. Providers charge less for the opening chunk of a prompt when it repeats. That is a prefix cache, and each model keeps its own. Send turn one to the cheap model and turn two to the frontier model, and the cache starts cold again. The lost discount is often bigger than the routing saving. Route once per conversation, not per message.
Pin the model for a session. Swap models mid-conversation and the voice changes, the formatting changes, and what the model refuses changes. Users read that as the product breaking.
The prompt is tuned for one model. Examples, format instructions, the phrases that make a model reason step by step: all of it is model-specific. A prompt built against the frontier model does worse on the cheap one. The router gets blamed for what is really a prompt problem.
Measure the router, not the models. The question is not whether the cheap model is any good. It is how often the router sent a hard request to it. Answering that needs a set of real prompts, each labelled with the route it should have taken. Hold that set back, and re-run it on every policy change.
A wrong route is an invisible regression. The call succeeded. Latency was fine. The dashboard is green. The answer is just worse. Track escalation rate and route distribution as real metrics, not debug logs. I build the paved-road AI architecture for a bank, and one rule decides whether a router is allowed in the request path at all: log which model answered, and why, every time. “The router decided” is not an answer I want to give an auditor.

4. What an effective router must have

Routing logic is the interesting part. These are the unglamorous parts that decide whether it survives production.

Capability	Why it matters	What breaks without it
Health-aware failover	Providers fail constantly and briefly	Uptime equals the worst vendor’s uptime
Ordered fallback chains	Explicit is better than clever	A silent downgrade to a weaker model mid-incident
Per-team virtual keys and budgets	One runaway loop drains the shared account	A surprise five-figure invoice
Token-level metering	Cost lives in tokens, not requests	Spend cannot be attributed to a feature
Exact and semantic caching	A hit returns in under 5ms, against 500–2,000ms	The same answer gets billed twice
Per-request data policy	Some prompts must not reach a provider that stores them	A compliance finding, discovered in an audit
Full-fidelity tracing	The same prompt can go to a different model each time	No way to explain why a request went where it went
OpenAI-compatible wire format	One request shape everywhere	Migrating out becomes a rewrite

Two of those need a word. A virtual key is a capped stand-in the router issues for the real provider key, so one team cannot spend another team’s budget. Semantic caching reuses the answer to a similar question, not only an identical one.

That last row is the one that decides how expensive leaving is. A wire format is the shape of the request and the response: the field names, the nesting, the streaming events. Nearly every provider and proxy now copies OpenAI’s, so code written against it keeps working when the thing behind it changes. A router with its own format is one nobody can remove without rewriting every call site.

Exits matter more than they used to: Helicone went to maintenance mode after its March 2026 acquisition, and in August Stripe agreed to buy OpenRouter. Stripe did not disclose terms, and press reports put the price in the billions. Whoever runs the router sits in the path of every request, and ownership changes faster than migration plans.

Key takeaways

Name the decision before buying. Gateway, model routing, provider routing. Buying the wrong layer is how teams end up disappointed by a tool that worked exactly as designed.

Model routing is the only layer with large savings in it. The saving comes from the price gap between the cheap and frontier tiers, not from the router. And check what any vendor number is measured against before repeating it: RouteLLM’s 85% is against always-frontier, their 40% is against other routers.

One model, one team, one vendor needs no router. Buy one at the second model, the second team, or the first uptime target one vendor’s SLA cannot meet.

Keep the exit cheap and write the migration down. The wire format is the portability guarantee. Ownership of the layer in front of every request belongs in the risk assessment, not in a footnote.

Buy the plumbing. Own the policy. Failover, budgets, metering, tracing and caching are solved problems, and a vendor has already debugged them at a scale most teams will not reach. That is the buy. The routing policy is not for sale: it is a function of my traffic, my prompts and my quality bar. Even behind a bought router, the threshold, the labelled test set and the escalation rate stay mine to own. The one piece worth building yourself is the piece nobody can sell you.