Model proxy and budgets
SpanDock can sit between your AI tools and the model providers. With the model proxy on, the server decides which model answers each call and records what it cost. You can swap models with rules, let SpanDock Mind pick a cheaper model when it is good enough, and keep each person within a budget. Everything is configured on the server under Model routing.
What the proxy does
Under Model routing, the server can:
- Swap models with rules. For example, send Claude Code's Opus requests to Sonnet.
- Let SpanDock Mind pick a model for each new conversation. SpanDock Mind is SpanDock's model advisor: it looks at the start of the conversation, the person's budget and the candidate models, and chooses a cheaper model when it is good enough.
- Keep each person within a budget.
For each call, the server decides in this order:
- The budget. Over the limit, the call is handled the way you chose (see below).
- The conversation's earlier choice. A conversation keeps its model for 6 hours, so the model doesn't change mid-task.
- The first matching rule.
- SpanDock Mind, if it is on.
- The requested model, unchanged.
Budgets
Budgets are per person. You assign clients to people; a client that isn't assigned counts as its own person. Each budget has a period (a day, a week or a month) and a limit.
- Near the limit (80% of it), or when spending is ahead of pace, only models cheaper than the requested one are used.
- Over the limit, the administrator chooses what happens: calls are blocked, sent to the cheapest model, or just flagged.
Spend is the total cost of the calls that went through the proxy, including SpanDock Mind's own cost.
Service keys and teams
From version 0.61, a service key and a team can have limits of their own:
- A key's models. Under Settings → Public endpoint → Limits, a key can be limited to some models: names, aliases such as
prod-chat, or patterns such asgpt-5-*. It is refused any other model, and its model lists show only these. - A key's budget. Its own day, week or month budget, beside the budget of the person it spends as.
- A team's budget. An access group (Access → Groups → Team budget) counts the spend of every computer and service in it.
Every budget that applies to a call is checked, and the strictest one decides. If you set a blocking budget below what was already spent this period, SpanDock asks before calls start being refused. Model routing → Budgets lists the keys' and teams' budgets.
On the command line: spandock service create CI -models gpt-5-* -budget 50/month.
Retries, backups and timeouts
From version 0.59, a call that fails is retried, and a rule can name backups that take it when its model fails. Set them under Model routing → Rules.
-
Retries. A call is retried up to 2 times, with a growing, randomised pause, on:
- connection errors and timeouts before the answer starts;
- 408, 429, 500, 502, 503, 504 and 529.
A provider's
Retry-Afteris honoured up to 30 seconds. A 429 asking for longer goes straight to the next backup. -
Backups. Each rule can have up to 4, tried in order once its model's retries are used up. A backup must speak the same API as the request; near or over the budget, only backups that cost no more than the requested model are used.
-
Prompts that are too long. A request too long for the model goes only to a backup marked Takes longer prompts. Without one, the tool gets the provider's error; Claude Code then compacts the conversation.
-
Content-policy refusals go to the next backup only if you turn that on.
-
Never retried: other 400 errors, 401, 403 and 404 go straight back to the tool, and nothing is retried once an answer has started.
-
Timeouts: 10 seconds to connect, 120 seconds for a streamed answer to start, 60 seconds of silence within a stream, and 600 seconds for an answer that isn't streamed.
-
Cooldown. A model that fails 3 times within a minute is skipped for a minute, as long as a backup can answer. If every model for a call is cooling down, the tool gets a short "try again" error. Each server keeps its own count.
Every value can be changed for the whole server and for each rule. Only the attempt that answered is charged, and Activity shows every attempt.
Upgrading: a server installed before version 0.59 keeps trying each call once until an administrator turns these settings on under Model routing → Rules → Retries and timeouts; SpanDock shows what changes before it applies anything. A rule with backups uses them either way.
Coming from LiteLLM? See Move from LiteLLM.
Rate limits
From version 0.62, calls can be limited in requests a minute, tokens a minute and calls at once, for a person, a team (access group), a service key and a model (a name, an alias or a pattern such as gpt-5-*). Set them under Model routing → Rules → Model limits, on a person, on an access group, or in a key's Limits.
Every limit that applies is checked. The first one reached refuses the call with HTTP 429, Retry-After, and the rate-limit headers the tool's API uses, so SDKs back off by themselves. Tokens are estimated from the request and corrected once the answer is in. Each server counts its own calls. Rate limits are in the Premium plan and up.
Aliases
A rule for a plain name, such as prod-chat, is an alias: apps ask for prod-chat, and the rule sends the call to a real model. The models lists (/v1/models) show the aliases a caller may use first. Model routing → Rules → Aliases lists each alias with its last use, and Test sends one short call through it.
Translation between API formats
A rule can allow translation, so Claude Code (Anthropic Messages) can run on GPT, Gemini, Llama or any OpenAI-compatible model, and Chat Completions apps can run on Claude. Streaming, tool calls and usage cross both ways. Turning it on lists what doesn't cross: thinking blocks, prompt-caching hints and Anthropic's server tools. It is off by default, per rule.
Embeddings, images and audio
From version 0.62, /embeddings, /images/…, /audio/… and /moderations under /llm/openai/v1 go to the provider with a key's model allow-list and budgets. They are metered by tokens, by image or by the minute. Price overrides can set a price per image and per minute. Batches, files, realtime and rerank aren't proxied, and the proxy answers so.
Load balancing
From version 0.63, a rule can share its requests with up to 7 more models: by weight, to the one with the fewest calls in flight, the fastest (time to the first byte), or the cheapest. A conversation stays on the model it started on, so prompt caching keeps working. If that model fails, the others take the request before the rule's backups. Set it under Model routing → Rules → Share its requests with.
Response caching
A rule can answer exact repeats of a request from a copy, without calling the provider: instant, and free. It is off until you turn it on for a rule and acknowledge that answers are kept on the server. Copies are encrypted, kept from 5 minutes to 24 hours, and given back only to the person or service key that made the request. They are never backed up, and they are deleted when records are deleted or the data is reset.
Guardrail vendors
A rule can have each request's latest prompt checked first by Lakera Guard, OpenAI's moderation, a self-hosted Presidio, or your own webhook. A flagged prompt is blocked or only recorded, and when the vendor is unreachable the rule fails open or closed, as you choose. Adding a vendor asks you to acknowledge that prompt text goes to it. SpanDock keeps only whether the prompt passed, and the Privacy Center lists the vendor.
Tags and end users
Label calls with X-SpanDock-Tags: team:data,job:nightly (or metadata.tags in the body) and the end user with X-SpanDock-End-User (or user, or Anthropic's metadata.user_id). Each call records them, so spend breaks down by tag and by end user.
Keys from the environment
Under Settings → AI models, a provider key can be env:OPENAI_API_KEY or file:/run/secrets/openai. SpanDock reads it each time it's used and never stores it, which fits Kubernetes secrets, Vault Agent and Doppler.
Monitoring and automation
- Prometheus:
GET /metricsexposes the proxy's calls, tokens, cost, savings, latency, calls in flight and cache size, by provider and model, never people. On the server's own machine it needs no key; from anywhere else, use a service key with the Prometheus metrics path. - Alerts: under Settings → Notifications → Alert channels, the server sends its notifications straight to Slack, Microsoft Teams or a webhook.
- Management API:
/manage/v1creates, changes, rotates and revokes service keys, manages people, and sets teams' budgets and limits. It needs a service key with the Management API path, which only the license owner can grant. Every change is in the audit log as that key.
Turn it on
The proxy runs on the server, and each computer opts in:
- On one computer, turn on Settings → Model proxy.
- For clients, turn it on from the server under Clients → Client defaults, or select clients and choose Edit settings.
Claude Code is then pointed at SpanDock automatically. Codex can be pointed at it with the snippet SpanDock shows.
Calls use the server's provider keys, set under Settings → AI models. The keys stay on the server, in the OS credential store; tools on the clients authenticate only with a local proxy key that SpanDock sets up for them.
Only API-key traffic is routed. Claude Pro and Max sign-ins aren't used while the proxy is on.
What is recorded
Every call through the proxy records the requested model and the model that answered, how the choice was made (rule, SpanDock Mind, budget, or unchanged), tokens, cost, savings, and SpanDock Mind's cost. These calls appear in Activity, Sessions, Models and the Dashboard, and the Model routing page summarises the month.
Cost estimation
Many tools report tokens but not cost. In that case, SpanDock estimates the cost from a price catalogue and marks it as estimated. A cost the tool reports itself always wins.
Prices are taken from, highest priority first:
- your overrides, under Settings → Model prices;
- built-in Claude prices;
- OpenRouter's public model list, refreshed daily.
Model names are matched exactly after removing provider prefixes and date suffixes, so a smaller model is never priced as its larger sibling. When the catalogue changes, older records that have tokens but no cost are priced too.
The model cost leaderboard
The Models page is a leaderboard of the models your tools used. It ranks them by spend, tokens, calls, speed, reliability, or value (cost per million tokens), and shows how each model moved since the previous period. Only calls that used tokens or cost are counted, and each model's list price is shown for comparison. On a server, the leaderboard covers every client or one of them.