Permutation Tests vs Traditional Parametric Tests
Feature | Permutation Test | Parametric Tests (t-test and ANOVA) |
---|---|---|
Purpose | Non-parametric method for comparing groups when parametric assumptions are violated or data distribution is unknown | Parametric methods for comparing means of groups assuming normality and equal variances. |
Assumption | No assumptions about the underlying distribution of data, robust to violations of assumptions such as normality and homogeneity of variances | Assumes normal distribution of data and equal variances between groups |
Test statistic | Test statistic derived from permutations of the data | t-value for t-test, F-value for ANOVA |
Example | Comparing median income between two different cities | Comparing mean exam scores between different teaching methods |
Permutation tests in Machine Learning
Permutation tests become quite useful in these situations, offering a reliable and adaptable substitute for hypothesis testing. The basic idea behind permutation testing is to generate a null distribution by randomly permuting the observed data. This allows for inference to be made without making strict assumptions about the distribution of the data. This article aims to demystify permutation tests in machine learning.
Table of Content
- What are Permutation Tests?
- Permutation Tests vs Traditional Parametric Tests
- Estimating the p-value in Permutation Tests
- P-values and its Interpretation in Permutation Tests
- Permutation Test in Python
- Benefits and Limitations of Permutation Test
- Applications of Permutation Tests in Machine Learning
Contact Us