Before proceeding to the essence of permutation tests, let’s review a situation that comes up frequently. You want to analyze an experiment, but you realize that you don’t have enough data to apply it. You have started the analysis but now realize that with the results so far, there isn’t enough statistical power to determine statistical significance. This is a typical scenario that often stems from the limited sample size used to achieve the statistical power.
However, what if we want to describe the metric with multiple predictors, or to extract statistics from some criterion? In this case, the bootstrap would not be very useful. Now you are ready for the explanation of the permutation tests and when they are needed.
Let's imagine that you decided to calculate your a/b test using a Student’s t-test. You assumed that your data is normally distributed and with equal variance (for example, you confirmed it using Bartlett’s test). Next, you would calculate the t-statistics and compare them to the theoretical distribution, and so on, in order to reject the H0.
This approach may remind you of Bayes. In R, the implementation of this approach is implemented in the Coin package, but you can also implement it using standard functions or the boot package for bootstrap.
Permutation tests are one way to handle a situation where the sample size is not enough to obtain sufficient statistical power to determine the significance of the results. However, you need to remember that no “little trick” will replace the sample size to achieve the optimum power of the experiment. You should only apply this method when you understand both its applicability and the nature of your data.