A client called us in a panic last spring. Their support agent had been running clean for four months, then over one weekend its resolution rate dropped from 91% to 68%. Nothing in their code had changed. What changed was the model behind an unpinned API alias, which the provider had quietly rolled forward to a newer version. The new model was objectively smarter. It also ignored a formatting instruction their downstream system depended on.

That failure is the single most common way a working AI system breaks. Here is the process we use so it does not happen to the agents we ship.

Why upgrades break things that were working

A production agent is not just a model. It is a model plus prompts, tool definitions, output schemas, retry logic, and a hundred small assumptions that got tuned against one specific version. Swap the model and every one of those assumptions is back in play.

The failure modes are predictable once you have seen enough of them:

• Format drift. The new model returns valid but differently shaped output. Downstream parsers choke.
• Verbosity creep. Newer models often explain more, which inflates token cost and breaks length-sensitive UI.
• Refusal shifts. Safety tuning changes mean edge-case inputs that used to process now get declined.
• Tool-calling changes. The model picks different tools, or calls them in a different order, for the same input.
• Latency swings. Reasoning-heavy models can double response time on tasks that never needed the extra thinking.

None of these show up as errors. That is what makes them dangerous. The system keeps returning 200s while quality quietly falls off a cliff.

Rule one: pin everything

Every model call in every agent we ship references a specific dated version, never a floating alias like latest. This is a one-line decision at build time that removes an entire category of production incident.

Pinning is not about avoiding new models. It is about controlling when you adopt them. An upgrade should be a change you deploy on a Tuesday morning with a rollback ready, not something that arrives while your team is asleep.

We keep a live inventory of every pinned model across every client system, with its deprecation date. When a provider announces a retirement, we already know exactly which agents are affected and how many days we have.

The inventory sounds like overhead until the first deprecation notice lands. A team without one spends the first week just finding out where the old model is still being called, and in a system with a dozen steps across three services, that search is rarely as quick as anyone expects.

Rule two: you cannot upgrade what you cannot measure

The eval set is the whole game. For every agent we build, we maintain a held-out test set of 100 to 300 real cases with known-good outputs, weighted toward the hard and weird inputs rather than the easy middle.

That set is built during the original project, not scrambled together during a migration. It covers:

• The 10 to 15 highest-frequency task types the agent handles
• Every historical failure case we have ever fixed, kept as a regression guard
• Adversarial and malformed inputs that should be handled gracefully
• Cost and latency baselines per task type

When a new model appears, we run the entire set through both versions and diff the results. A migration decision takes hours instead of weeks because the measurement infrastructure already exists. Teams without evals end up either upgrading blind or refusing to upgrade at all, and both options get expensive.

Rule three: shadow before you switch

Passing evals is necessary but not sufficient. Real traffic contains inputs no test set anticipated.

So we shadow. The new model runs in parallel against live production traffic, its outputs logged and scored but never returned to users. The old model still serves every request. We typically run 5 to 7 days or 1,000 requests, whichever comes first.

Shadow mode is where we catch the surprises: a model that aced the eval set but produces 30% longer outputs on real tickets, or one that handles a niche request type that only shows up twice a week and never made it into testing.

What we look at during shadow

We score four things, in this order:

1. Task success rate against human review on a sampled subset
2. Output validity, meaning schema conformance and parse success
3. Cost per completed task, including retries, not cost per token
4. P95 latency, because averages hide the failures users actually notice

If the new model does not clearly win on task success and at least tie on the other three, we stay put. Newer is not a reason to migrate.

Rule four: fix the prompts, not just the model

Roughly one in four upgrades regresses something on the first pass. That is normal and it is usually fixable.

Prompts written for an older model often carry scaffolding the new one no longer needs. Chain-of-thought instructions that boosted a 2024-era model can actively hurt a reasoning model that already thinks internally. Few-shot examples tuned for one style can anchor a newer model into worse behavior than it would produce cold.

Our migration pass includes stripping the prompt down and retesting, not just swapping the model string. On several projects, the upgrade only paid off after we deleted about 40% of the prompt. The instructions were compensating for weaknesses that no longer existed.

Rule five: roll out gradually with a rollback ready

Once shadow testing clears, we move traffic in stages: 10%, then 50%, then 100%, with at least 48 hours at each step and automatic alerting on the same four metrics we scored during shadow.

Rollback is a config change, not a deploy. If quality drops, we are back on the pinned old version in under a minute while we diagnose. We keep the previous version warm and available for the full deprecation window, never removing it the moment the new one goes live.

The staged rollout also gives the client's own team time to notice things our metrics cannot capture. Tone, phrasing, and judgment calls are hard to score automatically, and the people who read the agent's output every day will flag a shift in voice long before any dashboard does. We ask for that feedback explicitly at each stage rather than waiting for someone to complain.

One upgrade at a time

When a migration does regress, the diagnosis only stays cheap if you changed one thing. We do not bundle a model upgrade with a prompt rewrite, a new tool, or a schema change in the same release. Each one ships separately with its own eval run, even when that means three deploys instead of one. Bundled changes turn a 20-minute root cause into a two-day one.

What this costs and what it saves

Full migration on an instrumented agent runs 2 to 5 days of work. Without an eval set, the same migration takes 2 to 3 weeks, because you have to build measurement before you can trust any comparison.

The alternative, which we see constantly, is teams that freeze on an aging model because upgrading feels too risky. Eighteen months later they are paying more for worse output than a current small model would deliver, and then a deprecation notice forces the migration anyway, under deadline, with no test set.

Model upgrades are not a maintenance chore. They are the mechanism by which an AI system gets better after launch. Every provider release is a free performance improvement for teams equipped to evaluate it, and a live risk for teams that are not.

If you are running agents on unpinned models with no eval set, that is the highest-leverage thing to fix this quarter. If you want help building the measurement layer, let's talk about it.