Aim and hypotheses

The Friedman test is used to compare \(k \geq 3\) related conditions (or points in time) in terms of a quantitative or ordinal variable measured on the same subjects. In our example, it helps us to answer the question: “Is the pain score different before, during and after the treatment?”.

Its principle is simple: the \(k\) measurements of each subject are ranked from 1 (the smallest) to \(k\) (the largest), with tied values receiving the average of their ranks, and the ranks are then summed for each condition. If the conditions do not differ, these sums, denoted \(R_1, \dots, R_k\), should all be close to \(n(k+1)/2\), where \(n\) is the number of subjects. The test statistic measures how far they are from this value:

\[Q = \frac{12}{n k (k+1)} \sum_{j=1}^k R_j^2 - 3n(k+1)\]

Under the null hypothesis, and if \(n\) is not too small, \(Q\) follows approximately a chi-square distribution with \(k - 1\) degrees of freedom (with a correction in case of ties).¹ Since the ranking is done within each subject, the general level of each patient plays no role, just as the repeated measures ANOVA removes the variability between subjects from its error term.

The null and alternative hypotheses of the Friedman test are:

• \(H_0\): the \(k\) related conditions have the same distribution, so no condition tends to receive higher or lower ranks than the others
• \(H_1\): at least one condition is different from the others

In our example, \(H_0\) means that the pain score has the same distribution before, during and after the treatment, and \(H_1\) that at least one of the three moments differs from the other two.

Be careful that, as for the ANOVA and the Kruskal-Wallis test, the alternative hypothesis is not that all conditions are different from each other. If the null hypothesis is rejected, we only know that at least one moment differs from the others; post-hoc tests, covered later, tell us which ones.

Note also that the Friedman test is often presented as a comparison of medians. This shortcut is only valid under additional assumptions (in particular, distributions with the same shape that only differ by a shift), which are not needed as long as the test is used to compare conditions in general, as we do here.

Assumptions

First, the Friedman test requires one dependent variable, at least ordinal, measured on the same subjects across \(k \geq 3\) related conditions or points in time: a within-subjects design in which each subject is measured once in each condition (subjects with a missing measurement are removed by friedman.test()). Here, the pain score is measured at 3 moments on the same 16 patients, so this assumption is met.²

Second, the subjects must be independent from each other. Independence is required between subjects, but not within them: the \(k\) measurements of a given patient are of course dependent (this is the whole point of the design, and exactly what the ranking within subjects accounts for), but the scores of one patient must not influence those of another. This is verified based on the design of the study: here, patients have been selected at random and treated individually.

Third, as a nonparametric test, the Friedman test does not require normality, and since it compares neither means nor variances of differences, it does not require sphericity either. This is precisely why it is preferred over the repeated measures ANOVA when the normality or sphericity assumptions of the latter are not satisfied.

In our example, normality can be assessed on the residuals of a model including the moment and the patient, as in the repeated measures ANOVA article:

# residuals of the model, taking the patient effect into account
res_lm <- lm(pain ~ time + patient, data = dat)

# install.packages("car")
library(car)

qqPlot(residuals(res_lm),
  id = FALSE # remove point identification
)

shapiro.test(residuals(res_lm))
## 
## 	Shapiro-Wilk normality test
## 
## data:  residuals(res_lm)
## W = 0.89685, p-value = 0.0004984

Several points of the QQ-plot lie far from the straight line and outside the confidence bands, and the Shapiro-Wilk test rejects the normality of the residuals (p-value < 0.05). With only 16 patients, the central limit theorem cannot be relied upon to make this deviation harmless, so the Friedman test is the appropriate choice.

This flexibility comes at a price, though: since only the order of the measurements within each subject is used, the Friedman test is generally less powerful than the repeated measures ANOVA when the assumptions of the latter are met. It is an alternative, not a default choice.

In R

friedman.test(pain ~ time | patient,
  data = dat
)
## 
## 	Friedman rank sum test
## 
## data:  pain and time and patient
## Friedman chi-squared = 17.375, df = 2, p-value = 0.0001687

# install.packages("tidyr")
library(tidyr)

# from the long format to the wide format
dat_wide <- dat %>%
  pivot_wider(names_from = time, values_from = pain)

head(dat_wide)
## # A tibble: 6 × 4
##   patient before during after
##   <fct>    <dbl>  <dbl> <dbl>
## 1 1         75.7   66.8  55.2
## 2 2         58.8   49    52.5
## 3 3         91.4   50.4  55  
## 4 4         67.8   53.9  51.5
## 5 5         92.4   52.8  51.7
## 6 6         60.7   58.8  46.1
# Friedman test on the 3 columns of scores (without the patient column)
friedman.test(as.matrix(dat_wide[, -1]))
## 
## 	Friedman rank sum test
## 
## data:  as.matrix(dat_wide[, -1])
## Friedman chi-squared = 17.375, df = 2, p-value = 0.0001687

# rank the 3 scores within each patient
ranks <- t(apply(dat_wide[, -1], 1, rank))

# sum of the ranks for each moment
R <- colSums(ranks)
R
## before during  after 
##     45     29     22
# test statistic
k <- 3
12 / (n * k * (k + 1)) * sum(R^2) - 3 * n * (k + 1)
## [1] 17.375

# install.packages("rstatix")
library(rstatix)

dat %>%
  friedman_test(pain ~ time | patient)
## # A tibble: 1 × 6
##   .y.       n statistic    df        p method       
## * <chr> <int>     <dbl> <dbl>    <dbl> <chr>        
## 1 pain     16      17.4     2 0.000169 Friedman test

dat %>%
  friedman_effsize(pain ~ time | patient)
## # A tibble: 1 × 5
##   .y.       n effsize method    magnitude
## * <chr> <int>   <dbl> <chr>     <ord>    
## 1 pain     16   0.543 Kendall W large

Interpretations

The p-value is smaller than the significance level \(\alpha = 0.05\), so we reject the null hypothesis and we conclude that the pain score is not the same at the three moments (\(\chi^2(2) = 17.38\), p-value < 0.001, Kendall’s \(W = 0.54\)).

(For the sake of illustration, with a p-value larger than 0.05, we could not have rejected the null hypothesis, and thus could not have concluded that the pain score changed over time.)

As any omnibus test, however, the Friedman test does not tell us which moments differ.

Pairwise Wilcoxon signed-rank tests

The Wilcoxon signed-rank test compares two paired samples. It belongs to the same family of rank-based tests as the Friedman test, and it is actually a one-sample Wilcoxon test applied to the differences within subjects. Pairwise tests with the Holm adjustment are obtained with pairwise_wilcox_test() and the argument paired = TRUE:³

dat %>%
  pairwise_wilcox_test(pain ~ time,
    paired = TRUE,
    p.adjust.method = "holm"
  )
## # A tibble: 3 × 9
##   .y.   group1 group2    n1    n2 statistic       p  p.adj p.adj.signif
## * <chr> <chr>  <chr>  <int> <int>     <dbl>   <dbl>  <dbl> <chr>       
## 1 pain  before during    16    16       117 0.00919 0.0184 *           
## 2 pain  before after     16    16       122 0.00336 0.0101 *           
## 3 pain  during after     16    16        95 0.175   0.175  ns

(With paired = TRUE, observations are paired according to their order in the dataset, so the patients must appear in the same order at each moment, which is the case here.)

The base R function pairwise.wilcox.test() gives the same p-values:

pairwise.wilcox.test(dat$pain, dat$time,
  paired = TRUE,
  p.adjust.method = "holm"
)
## 
## 	Pairwise comparisons using Wilcoxon signed rank exact test 
## 
## data:  dat$pain and dat$time 
## 
##        before during
## during 0.018  -     
## after  0.010  0.175 
## 
## P value adjustment method: holm

Comparing the adjusted p-values (the p.adj column of the first output) to the 5% significance level, we conclude that:

• the pain score differs significantly between before and during the treatment (adjusted p-value = 0.018),
• it differs significantly between before the treatment and one month after it (adjusted p-value = 0.010), and
• it does not differ significantly between during the treatment and one month after it (adjusted p-value = 0.175).

In other words, the treatment is associated with a significant decrease of the pain score (median from 68.5 to 57.45), which persists one month after the treatment (median of 52.1), while the further decrease after the treatment is not significant.

Nemenyi test

The Nemenyi test is available in the {PMCMRplus} package:

# install.packages("PMCMRplus")
library(PMCMRplus)

frdAllPairsNemenyiTest(pain ~ time | patient,
  data = dat
)
##        before  during 
## during 0.01299 -      
## after  0.00014 0.43104

These p-values (based on the studentized range distribution, which already accounts for multiple comparisons) lead to the same conclusions: the moment before the treatment differs significantly from both other moments, while during and after do not differ significantly.

Conover test

Like the Nemenyi test, the Conover test compares the rank sums of the Friedman test, but it relies on a Student’s t distribution with an error term estimated from the ranks, which generally makes it more powerful than the Nemenyi test. It is available in the same package, with the frdAllPairsConoverTest() function. By default, this function uses a single-step adjustment based on the studentized range distribution (as for the Nemenyi test), but other adjustments can be chosen with the p.adjust.method argument, in which case the p-values are computed from the t distribution and then adjusted. For consistency with the pairwise Wilcoxon tests, we use the Holm method. Note that, unlike frdAllPairsNemenyiTest(), this function does not accept a formula, so the response, the conditions and the subjects are passed as separate vectors:

frdAllPairsConoverTest(
  y = dat$pain,
  groups = dat$time,
  blocks = dat$patient,
  p.adjust.method = "holm"
)
##        before  during 
## during 0.00066 -      
## after  6.9e-06 0.08650

These adjusted p-values lead to the same conclusions: the pain score before the treatment differs significantly from the pain score during and one month after the treatment, while the pain scores during and after the treatment do not differ significantly.

Note that you may also come across this test under the name Durbin-Conover test, for instance in the {ggstatsplot} package used in the next section. The Durbin test is a generalization of the Friedman test to designs in which each subject is measured under only some of the conditions (the so-called balanced incomplete block designs), and the Durbin-Conover test is its post-hoc test. When all subjects are measured under all conditions, as in our example, the Durbin test reduces to the Friedman test and the Durbin-Conover test reduces to the Conover test: despite the different names, the computations are exactly the same (the durbinAllPairsTest() function of the {PMCMRplus} package would give the same results as above).