Routing & failover

When a request names a model alias, the gateway chooses which deployment — and therefore which vendor — serves it. Routing is both budget-aware and health-aware, and it is entirely invisible to your applications: they only ever see the alias.

The routing sequence

1. Resolve. The alias resolves to the org's list of candidate deployments registered under it in the catalog.
2. Budget-aware filtering (cost-aware failover). Any candidate whose own per-deployment budget is exhausted is dropped, in one batched read. If you cap the expensive vendor and leave the cheap one open, traffic spills to the cheap one instead of being denied. A 429 happens only when every candidate is over budget. (Aggregate, all-deployment budgets are checked earlier, at the pre-call gate — see aggregate vs per-deployment.)
3. Health partition (circuit breaker). A per-deployment circuit breaker tracks failures. After N consecutive failures a deployment enters a cooldown window; a success clears it. Candidates split into healthy and cooling-down.
4. Load balancing. Each group is shuffled by a stateless random load balancer, spreading traffic evenly across equivalent backends.
5. Order and fall back. Healthy candidates are tried first; cooling-down ones stay at the end of the list as half-open probes — still reachable, so a recovered backend rejoins automatically. The pipeline tries each candidate in order until one succeeds, reporting every attempt's outcome back to the breaker.
6. Transient retries. Within a single attempt, transient failures (502, 503, 504, 529, dropped connections) are retried with exponential backoff and jitter — streaming-safe — before the attempt counts as a failure.

request "chat" ─▶ resolve alias ─▶ drop over-budget candidates
                                    │
                                    ▼
                    healthy ──shuffle──┐
                    cooling-down ──────┴─▶ try in order ─▶ success ─▶ respond + record
                                              │ hard failure
                                              ▼
                                        next candidate (fallback)

What this gives you in practice

Register two cheap backends and one premium one under an alias, price them, and put a per-deployment budget on the premium one. The gateway will:

• load-balance the cheap pair,
• fail over between them on errors,
• cool down a flaky backend and probe it back in when it recovers, and
• spill off the premium backend when its budget runs out —

all without your application knowing any of it happened. Every attempt (including failed ones) is recorded in the audit trail with the provider that served it.

Rules of thumb

• Group only interchangeable deployments under one alias. Fallback across heterogeneous models has sharp edges — a request with an image cannot fall back to a text-only model.
• Watch the served provider in Audit & usage (Govern ▸ Audit & usage) to confirm load-balancing and failover behave as intended before relying on them.
• The upstream echoes its own model name; the gateway may rewrite model in the response back to the requested alias so clients see a stable identifier.

Tuning

The circuit breaker is configured with application properties:

Breaker state is in-memory per gateway instance.

Related pages

• Cost-aware failover — a worked end-to-end example.
• Product architecture — where routing sits in the request pipeline.