Prompt regressions: when a text change breaks what already worked

Why can a prompt regress?

A prompt is not code with independent lines. Every word affects every possible response. When you add a rule to fix one case, you can change the weight of other instructions, and the model starts applying the rule where it does not belong.

There are two frequent sources of regression:

1. The text change. Adding “always confirm before answering” to fix a booking case, and suddenly the assistant asks for confirmation on menu questions.
2. The model change. A new version behaves differently even though nobody touched the prompt.

The second one surprises teams most, because nothing was edited. An independent Claude migration guide opens by saying that migrating a production workload is “rarely” just changing the model ID and shipping. Anthropic’s official documentation points the same way: its Claude Sonnet 5.5 guide asks you to run a fresh effort sweep against your own evals instead of keeping the setting you used on Claude Sonnet 5.

What can change when you migrate a model?

The official migration guide to Claude Sonnet 5.5 is a good concrete example. These are some of the changes it documents, depending on the starting model:

An independent guide adds that newer models tend to follow instructions more literally and that how often they use tools can shift. That is not in the official documentation consulted, so treat it as a hypothesis your test set should confirm or rule out.

The lesson does not depend on the specific model: every row is something you measure, not something you assume.

How do you detect a regression?

The method is the one used for any regression test, adapted to language. Hamel Husain puts it this way: like unit tests, evals can catch problems that return after a fix. He also warns that they often cost considerably more than unit tests to maintain and run.

1. Fix the case set. A versioned golden dataset with representative cases, cases that already failed and edge cases.
2. Run the current and the candidate version on the same set, side by side.
3. Grade on properties, not exact text. If the result must be valid JSON, check that it is. If it must call a tool, check that decision. Comparing sentences word by word creates false alarms. For subjective criteria, use an LLM judge validated against human labels.
4. Review the changes in sign. Cases that passed before and fail now are the most important signal. Cases that improve deserve a look too, because sometimes they improve for a reason you did not expect.
5. Keep the raw outputs. When something fails, you need to see what the model answered in order to diagnose it.

Husain recommends that analysis start from real traces rather than generic metrics. Langfuse turns that into a practical rule: every bug that escapes to production becomes a new case in the set.

How do you automate it?

Langfuse suggests moving the comparison into continuous integration, so that every change that touches a prompt runs the golden dataset automatically. To keep each review fast, it suggests a gate with a subset of tens to low hundreds of cases, and keeping the full suite for large changes or for before release.

On a small team this can start as a script that runs the set and lists the cases that went from passing to failing. What matters is that the prompt lives in a versioned file, so every text change leaves a trace and can be reverted.

How do you roll out without surprises?

A sequence that reduces the risk:

1. Inventory: find every place a model identifier appears, including configuration and test fixtures.
2. Pin the version during the migration so comparisons are reproducible, and keep the previous one available as a fallback.
3. Compare offline without real traffic, and review error rates, latency and cost.
4. Roll out in stages: a small share first, then more. The model identifier should live in configuration so you can roll back without redeploying.
5. Set rollback thresholds beforehand: limits on quality, errors and cost. If one is crossed, you revert.
6. Adjust the prompt separately, after migrating and measuring, so each effect has an identifiable cause.

When is this NOT worth the effort?

• When the prompt is short, the use case is simple and the test set would cost more than the errors it prevents. Even then, a 20-case set usually pays off.
• When versions cannot be compared. Without a fixed set, every change is an opinion.
• When the migration and the prompt rewrite happen in the same week. You lose the ability to attribute effects.

Example

Illustrative example: a medical appointment assistant has 120 cases in its golden dataset. The team adds a rule in capitals so the assistant always confirms the time before canceling. The quick gate passes, but when the full suite runs a regression appears: 9 price questions now ask for time confirmation, which makes no sense. They remove the emphasis, rewrite the rule as a condition (“if the user asks to cancel”) and rerun. That quarter they migrate the model: the set shows that the rate of unnecessary tool calls rose from 2% to 7%. They revert the model in configuration, adjust the prompt in a separate change and migrate again once the numbers hold.

To understand how to build the case set, read golden datasets. For evals as a continuous check, what are evals. And for measuring results against criteria, LLM-as-judge.

Common mistakes

• Changing the model identifier without running any case.
• Fixing one failure with strong emphasis that affects other behaviors.
• Comparing exact text and concluding there is a regression because one word differs.
• Migrating model and prompt in the same change, so effects cannot be attributed.
• Keeping the previous model’s effort level without measuring again.
• Not rolling back when a threshold is crossed, because “the new model is better overall”.

Sources

• Claude Sonnet 5.5 migration guide (Anthropic documentation): parameters that return errors, prefill, tokens, omitted thinking, refusals and effort.
• Prompting Claude Sonnet 5.5 (Anthropic documentation): rerunning the effort sweep against your own evals.
• Anthropic Claude Model Migration Guide (Hidekazu Konishi, independent blog): “rarely” enough to change the model ID; more literal instructions and shifting tool use.
• LLM Evals: Everything You Need to Know (Hamel Husain): evals as regression tests, error analysis on real traces.
• Golden dataset evaluation: build and maintain LLM test sets (Langfuse): CI gate, fast subset, new cases from failures.