LLM Router: Keep an Audit Trail When Models Change

An LLM router selects which language model handles a request. Once routing is enabled, the model name your agent sends can differ from the model that returns the answer. Keep both in the task record, preserve each request attempt you can observe, and leave unavailable model or cost data explicitly unknown. That gives a reviewer a usable audit trail when a coding task changes models or retries after a failure.

We built a local HTTP fixture to check that recording behavior. Ten scripted scenarios passed, including a successful fallback after HTTP 503, missing model information, absent usage, and a terminal authentication failure. The fixture called no real model provider. It tests an example recorder, rather than Copilot Fusion, Azure routing performance, or answer quality.

What does an LLM router do?

An LLM router directs a request to a model based on a routing policy. Depending on the system, that policy can consider the task, enabled models, cost, or expected response quality. An application can call one stable endpoint while different requests reach different underlying models. The routing decision becomes part of the execution history.

For a coding team, imagine an agent handling a task that includes summarizing a small diff and reviewing a complicated migration. A routing system may choose different paths for those requests. The task ID remains useful, but a record containing only that ID and the endpoint alias leaves out which model actually answered each step.

Start by naming the job you expect from the router. A budget policy, a model allowlist, and a fallback policy each need observable results. A routing rule that sounds sensible in a design document still needs evaluation on representative tasks before you trust it with important work.

Separate routing, gateway operations, and observability

The components overlap in products, so check behavior rather than relying on a category label. For the audit design in this article, treat model selection, request handling, and evidence collection as separate responsibilities.

Responsibility	Practical question	Evidence to preserve
Routing	Which model should handle this request?	Requested alias, policy reference, reported response model
Request handling	Did the call succeed, retry, or fail?	Visible attempt number and HTTP outcome
Observability	Can someone explain the task afterward?	Task linkage, available usage, missing-data markers
Human review	Is this change acceptable?	Review decision tied to the actual artifact and run

An agent can authenticate successfully and still lack a useful execution record. Our agent identity and project access guide covers attribution and authorization. The router record adds another dimension: which reported model produced the response associated with that actor's task.

Keep the task and output together. A reviewer approving one diff should be able to find the corresponding execution record, rather than guess from whichever model happens to be selected in a developer's interface today.

Why the request alias is insufficient

The request says what your application asked for. The response can report what served it. Store those as separate fields even when their values happen to match. Replacing the requested value with the response value loses configuration context; recording only the request hides the observed result.

For a documented example, Microsoft's model router guide describes inspecting the underlying model in responses. Read the documentation for your own provider and API version before mapping its fields into a shared record. A response field is provider-reported evidence, not independent verification of model weights or execution infrastructure.

GitHub's Copilot Fusion announcement describes per-task model routing and several execution patterns. That announcement is a reason to ask what evidence your client exposes. It does not establish that a generic chat-completion recorder can see every internal step of Fusion.

If the client exposes only a routing label, record that label and mark the underlying model unknown. A missing field should stay visible in the evidence. Filling it with the model you expected would make the record look more complete while making it less trustworthy.

The ten local scenarios we executed

The experiment ran on October 31, 2026. A Python HTTP server bound to loopback returned scripted JSON responses. The client sent requests using the alias team-router. Successful fixture responses identified either fixture-small or fixture-large, while other cases omitted the model or returned errors.

These names are synthetic. No provider credentials were used, and no prompt was sent to a hosted model. The fixture is useful because the audit recorder should behave predictably even when a request returns incomplete evidence.

Scenario	Required result
Requested alias differs from returned model	Preserve both values
Another task reports a different model	Record that task's response value
Response omits the model	Preserve null and mark attribution unknown
Response model is an empty string	Preserve unknown rather than an empty identity
HTTP 503 followed by success	Keep both visible attempt records
Final response includes usage after a retry	Keep total cost unknown without billing evidence
HTTP 401 ends the request	Keep the failure without inventing a response model
Response contains private message content	Omit that content from the audit record
Successful response omits usage	Keep usage null
Two consecutive HTTP 503 responses	Preserve both failures

All ten scenario assertions passed. The client also checked that the mock server received exactly as many requests as appeared in the visible attempt list. That ties the recorder's retry count to observed local HTTP traffic.

The experiment does not show that any model is cheaper, faster, or better at coding. It also does not establish what a real gateway bills for failed attempts. Those questions require a separate workload evaluation and provider billing evidence.

A small audit record with explicit unknowns

One fixture case returned HTTP 503 and then succeeded with fixture-large. The resulting record preserved the requested alias and both observable attempts. Its final usage described the successful response; the recorder made no claim about unreported internal work.

{
  "task_id": "synthetic-task",
  "requested_model_alias": "team-router",
  "response_model": "fixture-large",
  "visible_http_attempts": [
    {
      "attempt": 1,
      "http_status": 503,
      "reported_response_model": null,
      "usage": null
    },
    {
      "attempt": 2,
      "http_status": 200,
      "reported_response_model": "fixture-large",
      "usage": {"prompt_tokens": 20, "completion_tokens": 8}
    }
  ],
  "provider_internal_attempts": null,
  "cost_usd": null,
  "model_attribution": "known",
  "output_retained": false
}

This is an example application schema, not a provider API contract or a SyncStaff event format. Adapt it to the evidence your integration exposes. In a production record, also retain a timestamp, a non-secret request identifier when available, and the policy or configuration revision that governed the call.

The known label here means the response reported a nonempty model value. It does not mean the recorder verified the model independently. Define that distinction in your own schema so downstream reports do not give the field more authority than it has.

Handle retries without losing the first failure

A successful final response can conceal an application-visible failure if you overwrite the attempt record each time. Append an attempt for each request, then record the terminal outcome. This makes the sequence reviewable even when the task ultimately succeeds.

In our fixture, the retry rule allowed one more request after HTTP 503. A 401 stopped immediately. That rule belongs to the experiment. It is not a general recommendation for your production retry policy, which must account for the operation, provider guidance, and whether repeating it is safe.

The same distinction applies inside a managed router. Your application may observe one HTTP request while the service performs several internal steps. Keep application-visible attempts separate from provider-reported internal diagnostics. When the latter are unavailable, leave them unknown. Counting one outer request as one model invocation would claim knowledge the client does not have.

Cost attribution needs its own evidence

Token usage is useful, but the final response's token count does not prove the full cost of a routed task. The experiment deliberately kept cost_usd null after a failed attempt followed by a successful response with usage. Its recorder had no billing data, so it could not justify a dollar amount.

For real runs, preserve the usage you receive and identify its source. If you calculate an estimate, record the pricing revision and label the result as an estimate. Reconcile it with the billing source available to your organization before presenting it as an invoiced cost.

Compare equivalent outcomes as well as totals. A low-cost request that produces unusable code can generate more work through retries and review. Our Claude Code token-cost guide discusses the wider accounting problem for coding teams. Routing makes task-level attribution more useful because the selected model can change within the workflow.

Keep sensitive content out of routine audit records

An operational record can establish the task, attempt, status, and reported model without copying a prompt or completion. One fixture response contained a synthetic private response marker. The recorder intentionally excluded the response's message content, and the resulting record contained neither the prompt marker nor the response marker.

That assertion covers the record built by this example. It does not prove that an SDK, proxy, provider, or separate logging system retains nothing. Trace each collection point in your deployment and check its configuration. Decide who can access diagnostic content and how long it should remain available.

Start with the metadata needed to answer the investigation question. If a failed request requires a protected diagnostic capture, make that a deliberate workflow with access and retention rules. A shared audit dashboard should not quietly become a second copy of every repository snippet an agent has seen.

Put model changes in front of the reviewer

A coding review should preserve the link between a task, its generated artifact, and the execution evidence. If a run falls back to a different model or loses model attribution, the reviewer should be able to see that fact alongside the work being approved.

The appropriate response depends on the task. A missing model field on a low-risk summary might warrant a telemetry issue. The same gap on a sensitive migration may require another review step. Define that policy with the people accountable for the repository rather than leaving each agent to improvise it.

Our human-in-the-loop guide covers keeping meaningful decisions with people. For a routed task, make the reported serving model, visible failures, and missing evidence available before that decision. A successful request alone says little about whether the resulting change is acceptable.

Before enabling a new router policy, run a small attribution check: one successful response, one missing-model response, and one visible retry. Preserve the requested alias throughout. Then select a real task and verify that its reviewer can find the output and its execution record together. That is a concrete audit requirement a team can test before scaling its use of routing.