Xjyuk

Doing some code testing and I have a pre-fix environment where I ran 24 test scenarios and 20 out of 24 worked as expected—only 4 (16.67%) failed.

In the code where the fix exists, if I do another 24 tests, what are the chances that I get a 100% pass rate even if the code didn't fix the problem?

I feel like I probably need to do more tests but I can't remember my stats course and was hoping someone here could help

One straightforward way to analyze this situation is to assume for testing purposes that the fix made no difference. Under this assumption, you may view the assignment of the (potential) 48 observations into pre-fix and post-fix groups as being random. If (hypothetically) all the post-fix outcomes work as expected, it means you have observed 44 expected outcomes and 4 "failures" out of 48 (presumably) independent observations.

This permutation test frames your question as "what is the chance that a random partition of 48 test scenarios, of which four are failures, places all four failures in the first group?" Because all such random partitions are equally probable, this reduces the answer to a simple counting calculation:

• There are $24cdot 23 cdot 22 cdot 21$ equally-likely ways to place the four failures within the sequence of $24$ pre-fix scenarios.




• There are $48cdot 47cdot 46 cdot 45$ equally-likely ways to place the four failures within the sequence of all $48$ scenarios.

Ergo, under these hypotheses the chances that all four failures occurred before the fix are

$$frac24cdot 23 cdot 22 cdot 21 48cdot 47cdot 46 cdot 45 = frac771410 approx 5.5%.$$

The interpretation could be written in the following way:

Suppose four of 24 initial scenarios resulted in failure and none of 24 post-fix scenarios resulted in failure. This had a $5.5%$ chance of occurring when the fix changed nothing. Because $5.5%$ is small, it provides some evidence that the fix worked; but because it's not really that small--events with those chances happen all the time--it's probably not enough evidence to convince anyone who is sceptical that the fix worked. Unless collecting more data is very expensive, it might not be enough evidence to support consequential decisions related to the fix.

Even if the fix has already occurred, you have the flexibility to collect more post-fix data. You can experiment with analogous computations to see how many more post-fix scenarios you need to run (all without failure) in order to establish a convincing level of evidence that something changed for the better after the fix. For instance, if you need this chance to be less than $1%,$ you would want to run $49$ post-fix scenarios rather than $24.$

Of course, as soon as one post-fix scenario produces a failure you would likely stop your testing and try to improve the fix. Sophisticated versions of this approach (where you stop once you have enough information, without trying to guess how much you need beforehand) are called sequential testing procedures.