Governance · Cost management · Deep dive

The model proxy: one door to every model

Every agent in a Brightwire environment reaches every language model through one proxy. Provider keys live there and nowhere else; each caller carries its own key with its own budget; every call is metered, allow-listed and logged without its content. This is the piece that turns "we use several AI vendors" into something a company can control.

Why one door

The usual way to adopt several model providers is to hand out API keys. Within a quarter there are keys in laptops, in CI variables, in three departments' scripts, and nobody can answer the two questions that matter: who is spending what, and can we turn it off.

We put a proxy in the path instead, and made it the only path. In an Axle environment a spoke's network is closed by default: its agents may reach the model proxy, the tool gateway, the control API, their own databases and an outbound proxy with an allowlist, and nothing else. No provider key exists in any tenant's namespace. If an agent wants a model, it asks the proxy, with its own key.

The proxy is LiteLLM, an open-source gateway that speaks one API to many providers. It runs as one pinned, digest-locked deployment per environment with its own database. As with the boards our agents work on, the tool matters less than the conventions around it.

Who calls, with which key

A virtual key is an identity the proxy issues. It never reaches a provider. It carries a budget, an allowlist tier and tags, and it is what the spend log attributes to.

What the proxy decides on every call

1. Authenticate the key. Unknown key, no call.
2. Allowlist. Each key is bound to a tier of models it may use. The tier is a file in the fleet repository; a control loop reconciles it onto every managed key every five minutes and writes each change to the audit log. Models marked not-for-tenants are simply absent from the tier.
3. Budget. Each key has a hard cap per 30 days. Over it, the proxy answers with a budget error and the call never reaches a provider.
4. Resolve the alias, with fallbacks. Agents ask for a name like default, simple, medium or complex, not a vendor's model id. The proxy maps the alias to a concrete model, and if that model is down or over its own limits it falls through a fixed chain to the next one, or to medium. The alias is what lets us change vendors without touching an agent.
5. Log the spend. Model, tokens, cost, latency and tags. Not the prompt. Not the answer.

What a budget stop looks like

Budgets are enforced where the money is spent, not reported after the fact. When a key crosses its cap the proxy refuses the call, and a webhook tells the control API, which writes a budget.exceeded event to that tenant's audit stream and posts a line in the hub's chat. Budgets reset on the first of the month.

proxy alerting:      webhook → control-api /api/hooks/litellm
alert types:         budget_alerts
mapping:             budget_crossed          → budget.exceeded
                     threshold_crossed,
                     soft_budget_crossed,
                     projected_limit_exceeded → budget.alert
each event:          tenant audit stream (append-only) + one line in hub chat
caps today:          hub agents 2,000 USD / 30 days · each spoke key 500 USD / 30 days

Changing a model or a vendor

Because agents only know aliases, a vendor change is a configuration change, and configuration changes go through the same door as code:

1. A pull request in the fleet repository adds the catalog entry and, if tenants may use it, the allowlist entry. A person merges it.
2. The environment syncs itself to the merged state; the control loop puts the new tier on every managed key; a one-token probe confirms the model answers.
3. A new provider additionally needs its API key placed in the secret store. That is a human-only step, and it is the only place the key ever exists.

Removing a model is the same pull request in reverse. Today the catalog holds about sixty entries across five providers: Google Vertex AI (which needs no stored key at all), Anthropic, OpenAI, xAI and Fireworks. Adding a sixth is an afternoon; the agents never notice.

What is logged, and what is not

• Logged, per call: the key (by alias), model requested and model served, tokens in and out, cost, latency, the tenant tag, and the router's notes on what it did and what the alternative would have cost.
• Never logged: the prompt or the response. Prompt storage is switched off in the proxy and the runbook forbids turning it on. Evidence about what an agent did lives on the board and in the tool gateway's audit, not in the model log.
• Read-only operator views: health, the model catalog, the allowlist and the list of keys, shown by alias and never by value.

What we do not claim

• Routing by task complexity, today. The router exists (a classifier picks simple, medium or complex, with a rule that never sends a tool-using turn to a model that cannot handle tools, and a clamp to medium once a tenant has spent 80 % of its budget). It is switched off in every environment while its classifier is measured against a hold-back set; today the auto alias resolves to a fixed model. Aliases and fallbacks are what route calls now.
• Spend by person or by persona. Attribution is per key: per tenant, per chat surface, per run. Per-persona keys and end-user attribution are the next phase of the budget work, not the current one.
• A soft landing before the cap. The design has warning thresholds and a gradual downgrade to cheaper models as a budget runs out. Today the stop is hard, and the warning threshold is not yet armed on the live keys.
• Per-tenant allowlists. There is one fleet-wide tier today; the mechanism supports more.
• A retention period for the spend log. We have not set one yet, so we do not quote one.

Soft next step: if you want to see what your own model spend looks like behind one door, start a conversation.