Switch to a better, cheaper model without your users noticing

Switch without your users noticing

A newer model is often better and cheaper per token, but your users only see your prompt applied to their requests. A model that ranks higher on a leaderboard can still break your output format, ignore a rule you rely on or change its voice. The only way to find out is to run the prompt you ship on requests you've actually received, and read the two results side by side.

What to look for

• Format. Does the new model keep your structure: the JSON keys of an invoice extraction, the label set of a classifier, a hand-over marker your code looks for? Code that reads the result breaks on small changes.
• Rules. Does it still refuse to promise refunds, or stick to the facts it was given? Test the requests where your prompt forbids something.
• Who it fails for. A model that's fine in English can slip in Spanish, or on long or unusual inputs. Include the kinds of request your users really send.
• Tone and length. Users notice a change of voice before a change of quality, and longer isn't better if your interface expects short.
• Cost and latency. Cost per call depends on how much a model writes as well as its price per token, so a model that's cheaper per token but twice as wordy can cost more.

Mark what worked, then compare the real cost

Mark each result as worked or didn't work, the way a user would see it. The totals then show the share that worked first time and the real cost per result: what you pay for each result that worked, counting the failed tries. Example: model A costs $0.004 a call and 9 in 10 results work, so each one that worked costs about $0.0044. Model B is half the price per call, but if only 4 in 10 work, each one that worked costs $0.005. The cheaper model costs more per result that worked, before you count the customers who had to ask again.

How many inputs you need

Ten to twenty real inputs are enough to spot problems: a broken format, a refusal, a shift in tone. Include a few requests that went badly before. They aren't enough to show that one model works first time more often, because rates are noisy: going from 60% to 70% takes about 360 results per model to show. The A/B test sample size calculator works out the number for your own rates. Use this test to decide whether a model is worth trying with real users, then measure what works for them.

GPT-4o and GPT-4.1 leave OpenAI's API on January 14, 2027

OpenAI's deprecations page (opens in a new tab) lists January 14, 2027 as the API shutdown date for GPT-4o, GPT-4o mini, GPT-4.1 and GPT-4.1 mini, and names GPT-6 Luna as their replacement. That's why model A starts as GPT-4o here. Try the named replacement and one or two models from other providers, since the cheapest model that works for your prompt may not come from the same company. The model shutdown calendar lists the other dates.

Test prompt changes the same way

1. Set model A and model B to the same model and run your current prompt. The differences you see are the run-to-run variation that any real change has to beat.
2. Download the test set, then run the same inputs again with the new prompt and download that too. The system_prompt_hash field tells the two files apart.
3. Save the inputs. Run them again at every model or prompt change, so each change is checked against the same cases.