How many results before you know version B works first time more often?

Why compare how often results work first time

Few users click thumbs up or down, and the ones who do aren't typical. Whether a result worked first time is known for every result: the support customer didn't ask again, nobody fixed the extracted invoice fields, sales didn't overrule the lead score, the user didn't hit “try again”. It reflects each user's own judgement of their own request, and it's tied to money, because every failed try is a call you paid for. That makes it the most direct answer to “is version B better for my users?”

Don't peek and stop early

If you check every day and stop the moment B looks ahead, you'll declare far more false wins than your confidence level promises, because random swings cross the line along the way. Work out the sample size before you start, run until both versions reach it, then compare. While it runs, watch for breakage such as errors and cost spikes, not for a winner.

Test bigger changes when traffic is low

The results you need grow with the square of how small a difference you want to detect: halve the lift and you need about four times as many. At 100 results a day, telling 60% working first time from 70% takes about eight days, while 60% against 62% takes about six months. With little traffic, test changes big enough to matter, such as a new model or a rewritten prompt, rather than one reworded sentence.

Use offline replay as a pre-filter

Before you spend live traffic, replay a set of real past requests through each version and have an AI judge score the results. Check that judge against people first: one app's judge, Storybirdie's, had 0.28 precision against human labels, and a judge like that can't pick a winner. The judge agreement checker tells you whether yours can. Use replay to drop versions that are clearly worse, then confirm the survivor with a live test.

Split traffic by user, not by request

Assign each user to a version once, for example by hashing their user id, and keep them there. If you split by request, the same person sees both prompts: they try again on a weak result from A, get one from B that works, and B gets credit for A's failure. Splitting by user also keeps the app consistent for each person.

This calculator counts results. If a few heavy users make most of your requests, their habits dominate both groups, so aim for more results than the numbers above, or count users instead.