Introduction

In previous articles, we showed how to test whether two qualitative variables are related thanks to the Chi-square test of independence in R (and how to compute it by hand). Both articles insist on one important limitation: this test requires independent observations. If you have dependent observations (paired samples), that is, if the measurements have been collected on the same subjects, the McNemar’s or Cochran’s Q tests should be used instead, the Cochran’s Q test being an extension of the McNemar’s test when we have more than two related measures.

The case of exactly two related measurements has already been covered in the article about the McNemar’s test in R. The present article is dedicated to its generalization: the Cochran’s Q test, used to compare three or more related proportions, that is, the same binary outcome measured on the same subjects under \(k \geq 3\) conditions or time points.

It can therefore be seen as the dependent-samples counterpart of the Chi-square test of independence for a binary outcome: instead of comparing several independent groups, it compares several repeated measurements collected on the same individuals. It also follows the same logic as the other tests comparing three groups or more presented on this blog, such as the Kruskal-Wallis test (used for a quantitative variable and independent samples): a first test tells us whether at least one group differs from the others, and post-hoc tests, together with an adjustment of the \(p\)-values for multiple comparisons, then tell us which groups actually differ. If you are unsure about which test is appropriate for your own data, see this overview of the most common statistical tests.

In the remaining of the article, we present the data, the aim, the hypotheses and the assumptions of the test, and finally how to perform it in R, how to complement it with post-hoc tests and how to interpret the results.

Data

As already mentioned in the article about the McNemar’s test, a dataset with a repeated binary structure is not so easy to find among the datasets shipped with R, so we simulate our own data.

Suppose that we ask 200 randomly selected citizens whether they are in favor of a new policy in their city (answer “Yes” or “No”), and that we ask them exactly the same question at three different points in time:

1. before a public debate on this policy,
2. right after this debate, and
3. one month later (follow_up), in order to see whether the effect of the debate persists over time.

# number of respondents
n <- 200

# opinion before the debate
before <- sample(c("Yes", "No"),
  size = n,
  replace = TRUE,
  prob = c(0.4, 0.6)
)

# opinion right after the debate (respondents who were in favor
# tend to keep their opinion, while those who were against
# are more likely to change their mind)
after <- ifelse(before == "Yes",
  sample(c("Yes", "No"), size = n, replace = TRUE, prob = c(0.9, 0.1)),
  sample(c("Yes", "No"), size = n, replace = TRUE, prob = c(0.4, 0.6))
)

# opinion one month later (some of the respondents
# convinced by the debate go back to their initial opinion)
follow_up <- ifelse(after == "Yes",
  sample(c("Yes", "No"), size = n, replace = TRUE, prob = c(0.85, 0.15)),
  sample(c("Yes", "No"), size = n, replace = TRUE, prob = c(0.1, 0.9))
)

# dataset
dat <- data.frame(
  respondent = factor(1:n),
  before = factor(before, levels = c("No", "Yes")),
  after = factor(after, levels = c("No", "Yes")),
  follow_up = factor(follow_up, levels = c("No", "Yes"))
)

head(dat)
##   respondent before after follow_up
## 1          1    Yes   Yes       Yes
## 2          2    Yes   Yes       Yes
## 3          3     No   Yes       Yes
## 4          4    Yes   Yes       Yes
## 5          5    Yes   Yes       Yes
## 6          6     No    No        No

(Note that a seed has been set in the background with set.seed(42), so the simulated data and all the results presented below are reproducible.)

The data are stored in the wide format: one row per respondent, and one column per measurement. This is exactly the structure the Cochran’s Q test is designed for, with each respondent playing the role of a block inside which the three answers are related.

As always, it is a good practice to start with some descriptive statistics, here the proportion of respondents in favor of the policy at each of the three points in time:

# install.packages("dplyr")
library(dplyr)

dat %>%
  summarise(across(before:follow_up, ~ mean(.x == "Yes")))
##   before after follow_up
## 1   0.46  0.61      0.54

These proportions are easier to compare on a plot:

# install.packages("ggplot2")
library(ggplot2)

# install.packages("tidyr")
library(tidyr)

# from the wide format to the long format
dat_long <- dat %>%
  pivot_longer(
    cols = c(before, after, follow_up),
    names_to = "time",
    values_to = "opinion"
  ) %>%
  mutate(time = factor(time, levels = c("before", "after", "follow_up")))

dat_long %>%
  group_by(time) %>%
  summarise(prop_yes = mean(opinion == "Yes")) %>%
  ggplot() +
  aes(x = time, y = prop_yes) +
  geom_col(fill = "steelblue") +
  labs(
    x = "Moment of the survey",
    y = "Proportion in favor of the policy"
  )

In our sample, the proportion of citizens in favor of the policy increased from 46% before the debate to 61% right after it, and then decreased to 54% one month later. The question is whether these differences are large enough to be generalized to the population, or whether they could be explained by sampling fluctuations alone.

Note that the three proportions are computed on the same people, so comparing them as if they came from three independent groups would ignore the fact that the measurements are repeated. Taking this dependency into account is precisely the purpose of the Cochran’s Q test. Note also that, in the code above, we created a long format version of the dataset (dat_long), with one row per respondent and per measurement, since this is the format expected by the functions used in the rest of the article:

head(dat_long)
## # A tibble: 6 × 3
##   respondent time      opinion
##   <fct>      <fct>     <fct>  
## 1 1          before    Yes    
## 2 1          after     Yes    
## 3 1          follow_up Yes    
## 4 2          before    Yes    
## 5 2          after     Yes    
## 6 2          follow_up Yes

Cochran’s Q test

# install.packages("rstatix")
library(rstatix)

dat_long %>%
  cochran_qtest(opinion ~ time | respondent)
## # A tibble: 1 × 6
##   .y.         n statistic    df        p method          
## * <chr>   <int>     <dbl> <dbl>    <dbl> <chr>           
## 1 opinion   200      18.3     2 0.000108 Cochran's Q test

# Cochran's Q test on the first 2 measurements only
dat_long %>%
  filter(time %in% c("before", "after")) %>%
  mutate(time = droplevels(time)) %>%
  cochran_qtest(opinion ~ time | respondent)
## # A tibble: 1 × 6
##   .y.         n statistic    df         p method          
## * <chr>   <int>     <dbl> <dbl>     <dbl> <chr>           
## 1 opinion   200      17.3     1 0.0000318 Cochran's Q test
# McNemar's test on the same 2 measurements
mcnemar.test(table(dat$before, dat$after),
  correct = FALSE
)
## 
## 	McNemar's Chi-squared test
## 
## data:  table(dat$before, dat$after)
## McNemar's chi-squared = 17.308, df = 1, p-value = 3.179e-05

Post-hoc tests

We have just showed that the proportion of citizens in favor of the policy is not stable over time. Nonetheless, here comes the limitation of the test: it does not say which measurement(s) differ(s) from the others.

To know this, we need post-hoc tests (in Latin, “after this”, so after obtaining significant results for the Cochran’s Q test), also referred as multiple pairwise-comparison tests. The logic is the same as the one presented for the Kruskal-Wallis test: we compare the groups two by two, and we adjust the \(p\)-values because performing several tests on the same data increases the risk of finding a significant difference by chance alone.

Here, the natural post-hoc test is simply the McNemar’s test applied to each pair of measurements. The post-hoc step is thus literally a repeated application of the test presented in the article dedicated to the McNemar’s test, which makes sense given that the Cochran’s Q test is nothing more than its extension to more than two related measurements.

With 3 measurements, there are 3 pairs to compare. This is done with the pairwise_mcnemar_test() function of the {rstatix} package, with the Holm method to adjust the \(p\)-values:²

dat_long %>%
  pairwise_mcnemar_test(opinion ~ time | respondent,
    p.adjust.method = "holm"
  )
## # A tibble: 3 × 8
##   group1 group2    statistic    df         p    p.adj p.adj.signif method      
## * <chr>  <chr>         <dbl> <dbl>     <dbl>    <dbl> <chr>        <chr>       
## 1 before after         16.2      1 0.0000578 0.000173 ***          McNemar test
## 2 before follow_up      3.31     1 0.0689    0.0689   ns           McNemar test
## 3 after  follow_up      6.04     1 0.0140    0.0280   *            McNemar test

It is the p.adj column (the \(p\)-values adjusted for multiple comparisons) which is of interest, and not the p column (the unadjusted \(p\)-values). These adjusted \(p\)-values must be compared to the desired significance level (usually 5%).

Based on the output, we conclude that:

• the proportion of citizens in favor of the policy differs significantly between before and right after the debate (\(p\)-value < 0.001),
• it differs significantly between right after the debate and one month later (\(p\)-value = 0.028), and
• it does not differ significantly between before the debate and one month later (\(p\)-value = 0.069).

Combined with the proportions computed earlier, these post-hoc tests give a much more precise picture than the Cochran’s Q test alone: the debate significantly increased the support for the policy in the short run (from 46% to 61%), but this increase did not last. One month later, the support had significantly decreased compared to the level observed right after the debate, and it was back to a level no longer significantly different from the initial one.

Summary

In this article, we reviewed the aim, the hypotheses and the assumptions of the Cochran’s Q test, used to compare three or more related proportions. We then showed how to perform it in R with the cochran_qtest() function of the {rstatix} package, how to interpret its results by comparing the \(p\)-value with the significance level \(\alpha\), and, since a significant result only indicates that at least one measurement differs from the others, how to identify which ones thanks to pairwise McNemar’s tests with adjusted \(p\)-values. Remember, last but not least, that the McNemar’s test is the special case of the Cochran’s Q test for exactly two related measurements, and that with independent samples the Chi-square test of independence should be preferred.

Thanks for reading.

I hope this article helped you to understand the Cochran’s Q test and how to perform it in R.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.