Route work by risk

A cheap model that had been reliable for titles was assigned a multi-file application edit. It called the right tools, but repeatedly reread files, missed a stale revision, and exhausted the turn limit. Replacing it with the most expensive model for every request fixed the edit and tripled the cost of ordinary chat.

The useful question was not "which model is best?" It was "what does this turn require, and what damage can a poor decision cause?"

Route in layers

incoming work
    |
    v
execution mode: interactive or isolated
    |
    v
risk class: read, reversible write, consequential
    |
    v
work role: chat, build, cheap maintenance
    |
    v
compatible model chain
    |
    v
selected tools and budgets

Routing includes more than model choice. A scheduled turn should lose browser-only tools. A private turn should lose memory-write tools. A consequential action should keep the approval gate no matter which model runs.

Terms

• Role: A workload class with its own model chain, budget, and prompt guidance.
• Router: A small classifier that selects a role and optional instructions.
• Fallback chain: Ordered compatible models used when quota or transient failure blocks the preferred model.
• Capability gate: Code that includes or excludes a tool for the current context.
• Deterministic work: Work completed by code without a model decision.
• Quota failure: A provider response that says the selected model cannot currently serve more work.
• Replay compatibility: Whether another model can continue from the same message and tool transcript.

Use a small role set

Too many routes are hard to evaluate. Start with roles that differ materially:

• chat for conversation, research, memory, and follow-ups;
• build for multi-file creation and editing;
• cheap for titles, capture, summaries, consolidation, and low-risk triage;
• route for the small classification call itself.

Verification and design-rule checks should stay deterministic where possible. Do not pay a model to parse JavaScript, compare revisions, check a URL status, or run a browser flow when code can decide.

A role configuration can be plain:

roles:
  route:
    models: [fast-small]
    max_output_tokens: 300
    tools: []
  chat:
    models: [balanced-a, balanced-b]
    max_cost_usd: 0.35
  build:
    models: [code-a, code-b, balanced-b]
    max_cost_usd: 1.00
  cheap:
    models: [fast-small]
    max_cost_usd: 0.05

Model IDs belong in configuration or a versioned catalogue with price and endpoint metadata. Do not scatter them through feature code.

Keep the router weak

The route call should see only what it needs: pending user text, whether the conversation owns editable artifacts, and perhaps skills already loaded. It needs no tools and little output.

type Route = {
  role: "chat" | "build";
  skills: string[];
};

function parseRoute(value: unknown): Route {
  const parsed = RouteSchema.safeParse(value);
  return parsed.success ? parsed.data : { role: "chat", skills: [] };
}

If the route call fails, times out, or returns malformed JSON, use a safe default. Do not fail the user turn because its optimization failed.

Routing should not decide policy. A router may select build; it may not grant deployment bypasses or connected-account tools.

Route by risk before capability

Workload difficulty matters, but effect risk matters more.

A small model can safely propose a title because the output is easy to overwrite. The same model should not autonomously send an email even if the email is short. A strong model also should not send it without approval.

Useful risk questions are:

• Can the action be reversed?
• Does it communicate outside the system?
• Can it spend money or consume scarce quota?
• Does it expose private data?
• Will anyone be present to review the result?
• Can deterministic checks catch a bad result before the effect?

These answers set tool gates, approval, budgets, and verification. Model quality is one control among several.

Fallback without restarting the turn

Quota limits often apply per model. When the preferred model returns a clear quota response, move to the next model in the role chain and preserve the transcript. Record the fallback.

Do not replay completed tools. Continue from the existing messages and tool results. Restarting from the user request can duplicate effects and waste context.

Fallback requires protocol compatibility. Models using the same chat-completions tool format can often share a transcript. Moving between incompatible endpoint families may require lossy translation. Exclude incompatible models from one chain unless that translation has tests.

Differentiate quota from transient failure:

• quota or model entitlement failure: switch models immediately;
• timeout or 5xx: retry briefly on the same model, then fall back;
• malformed protocol response: record the model-specific failure and decide whether one replay is safe;
• authentication failure: stop, because another model under the same credentials will usually fail too.

Skills and prompt routing

A build model often needs specialized instructions, but loading every skill into every turn wastes context.

The route call can prefetch a small set of named skills. Record which skills a conversation has already read, and include bodies only when needed. Keep skills as versioned files with bounded length.

This mechanism should improve instructions, not hide capabilities. Tools still advertise their own schema and guidance. A missing skill should make the turn less efficient, not unsafe.

Decisions and rejected options

Do not use one model for everything. It is simple, but cheap background work steals quota and expensive models handle tasks code could complete.

Do not route solely on user wording. "Make this better" may be a build request only when an artifact exists. Include minimal structured state.

Do not let routing become a second agent. A long classifier with tools and memory adds latency before every turn and can fail in more ways than it saves.

Fallback only on defined conditions. Switching after every tool error hides application bugs and produces hard-to-explain model behavior.

Price usage locally when needed. Some gateways return tokens without cost. A versioned catalogue allows consistent budgets, but prices can change, so monitor drift.

Keep isolated turns conservative. Scheduled and event-driven work should default to chat-grade reasoning with a reduced tool set. They skip route calls when the likely role is known.

Failure modes

The router may classify an ordinary request as build and spend more. Track role decisions and correct rates using outcome data, not intuition.

A fallback model may not support a tool schema feature used earlier in the turn. Catalogue capabilities and filter chains before deployment.

Quota errors may arrive as several HTTP statuses or provider-specific bodies. Parse a narrow allowlist. Treat unknown authorization errors as terminal rather than cycling every model.

A cheap summarizer can corrupt durable memory. Use revision checks, bounded append formats, and human-editable documents. Low cost does not remove the need for safe writes.

Model labels in metrics can explode cardinality when providers return arbitrary names. Normalize against the catalogue and use unknown for the rest.

Routing checklist

• Roles correspond to materially different work.
• Deterministic jobs use code rather than a model.
• The router has no tools and a small output budget.
• Router failure falls back to a safe default.
• Policy and approval do not depend on router output.
• Tool selection considers private and isolated modes.
• Each role has a compatible ordered model chain.
• Quota, transient, protocol, and authentication failures differ.
• Fallback preserves the existing transcript and completed tool results.
• Price and capability metadata are versioned.
• Role, model, fallback, cost, and outcome are recorded.
• Production data reviews whether routing saves calls and latency.

The budgets in Context is a budget make routing measurable. Routing remains a hypothesis about cost, quality, and risk. The final chapter tests it with privacy-bounded evidence from the machine itself. Continue with Measure the machine.