Xjyuk

Why do we care about the power of a hypothesis test if we no longer live in an age where computers are slow and it's too costly to bootstrap/do a permutation test on anything which is also non-parametric?

Is power-analysis irrelevant if I can bootstrap/permutation hypothesis test?

We can make the "sample size" infinity with bootstrapping so power goes up as a result of bootstrapping?

The amount of information relating to the hypotheses that you have is simply the information in the original data.

Resampling that information, whether bootstrapping, permutation testing or any other resampling, cannot add information that wasn't already there.

The point of bootstrapping is to estimate the sampling distribution of some quantity, in essence by using the sample cdf as an approximation of the population cdf from which it was drawn.

As normally understood, each bootstrap sample is the same size as the original sample (since taking a larger sample wouldn't tell you about the sampling variability at the sample size you have). What varies is the number of such bootstrap resamples.

Increasing the number of bootstrap samples gives a more "accurate" sense of that approximation, but it doesn't add any information that wasn't already there.

With a bootstrap test you can reduce the simulation error in a p-value calculation, but you can't shift the underlying p-value that you're approximating (which is just a function of the sample); your estimate of it is just less noisy.

For example, let's say I do a bootstrapped one-sample t-test (with a one-sided alternative) and look at what happens when we increase the number of bootstrap samples:

The blue line very close to 2 shows the t-statistic for our sample, which we see is unusually high (the estimated p-value is similar in both cases, but the estimated standard error of that p-value is about 30% as large for the second one)

A qualitatively similar picture - noisier vs less noisy versions of identical underlying distribution shapes - would result from sampling the permutation distribution of some statistic as well.

We see that the information hasn't changed; the basic shape of the bootstrap distribution of the statistic is the same, it's just that we get a slightly less noisy idea of it (and hence a slightly less noisy estimate of the p-value).

--

To do a power analysis with a bootstrap or permutation test is a little tricky since you have to specify things that you didn't need to assume in the test, such as the specific distribution shape of the population. You can evaluate power under some specific distributional assumption. Presumably you don't have a particularly good idea what distribution that is, or you'd have been able to use that information to help construct the test (e.g. by starting with something that would have good power for a distribution reflecting what you understand about it, then perhaps robustifying it somewhat). You can of course investigate a variety of possible candidate distributions and a variety of sequences of alternatives, depending on the circumstances.