Why Risk Ratio Is Preferred, What is Modified Poisson, and Why Do We Even Care About Standard Error?

[This article was first published on r on Everyday Is A School Day, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

Learnt about modified Poisson for the first time. Simulated RR vs OR, log-binomial and robust SE, then sprinkled in repeated measures. Turns out a 95% CI can be too narrow (confidently wrong) or too wide (overly cautious)! Still learning, but pretty convinced now 🙌

Motivations

I was working on a project recently and learnt a few things as we were going along and want to look into it a bit more with simulation to see for myself. These might sound like some random topics, but they’re all kind of tied together, at least in my mind they are. I’ve not heard of modified poisson before and what a great way to get from incompetent incompetence to at least competent incompetence. 🤣 Knowing what we don’t know is definitely better than not knowing what we don’t know (MNAR problem). Well let’s explore together and see for ourselves, will this be our next go-to model along with robust sandwich standard error? Oh, let’s also sprinkle a little bit of repeatd measure (some clustering) to beef up that SE to see an even better effect with this robust sandwich standard error. Let’s go! 🙌

Thought Process

But, how are we going to do this? Let’s go through the bare basics of the interpretation, let’s say we have binary outcome. A heuristic is to use logistic regression model with logit as link function. It’s a great model, but exponentiating on the coefficient of exposure turns it into odds ratio instead of risk ratio (RR). So, let's talk a bit about why risk ratio may be a bit easier to understand than odds ratio. After this we’ll look and see how can we simulate using baseline rate and RR. And use this simulated data to check on our regular poisson regression, log binomial regression, logistic regression with logit link, then poisson regression w robust SE (aka modified poisson). We’ll look at their wideness of 95%CI. Then, to sprinkle a little bit of sweetness on top, we’ll add on repeated measure, look at SE of all these models. Top it up with, g-computation using logistic regression & bootstrap based on subject and see how that matches up with modified poisson.

Objectives:

Why Risk Ratio?

Risk Ratio is easier to understand because it’s basically risk of event in exposed group divided by risk of event in unexposed group (p1/p0). So, if we have an RR of 0.6, we’re saying the exposed group has 40% less risk of event than the unexposed group. If we have an RR of 1.5, we’re saying the exposed group has 50% more risk of event than the unexposed group. That’s much more intuitive than the odds ratio, which only approximates RR when the outcome is rare (roughly <10%). When the outcome is common, OR exaggerates the effect, pulling further from 1 than the RR.

To get an RR, we can’t just exponentiate the exposure coefficient from a logistic regression, because that gives us an OR. Instead, we can use:

  • Log-binomial regression (binomial family, log link): this models RR directly, but it often fails to converge, especially with continuous covariates or high baseline risk.
  • Modified Poisson regression (Poisson family, log link, robust SEs): exponentiating the exposure coefficient gives us RR, how convenient! But there’s a catch. With a binary outcome, plain Poisson overestimates the variance, so the CIs are too wide. Use robust (sandwich) standard errors to fix this, e.g. sandwich::vcovHC() in R. Predicted “risks” can also exceed 1, so don’t use the model for predictions.

Another reason to use RR to communicate effect size is that OR is non-collapsible. What do that mean? 🤷‍♂️ I’ve tried to understand this several times in the past, I thought I got it but it was an illusion of understanding. 🤣 We’ll have to learn and go deeper on what that means next time with simulations and examples. Let’s focus on our current task.

Simulate Baseline Rate and RR

library(tidyverse)
library(broom)
library(sandwich)
library(lmtest)

set.seed(1)
n <- 1300
baseline <- log(0.06) # 6% base rate
rr <- log(0.6) # RR of 0.6, meaning treatment reduces risk for 40% compared to non-treatment (noticed how we switched the term exposure with treatment here)

x <- rbinom(n,1,0.5)
p <- exp(baseline + rr*x)
y <- rbinom(n, 1, p)

We’ve not simulated with the above setting before. So, let’s try to explain so that our future self can return and recall what’s going on here.

So in Poisson regression, we have log link function. So, we can model the log of the expected count as a linear combination of the predictors. In this case, we have a binary predictor x (0 or 1) and we want to simulate a binary outcome y (0 or 1) based on a baseline risk and a risk ratio. Poisson formula is log(λ) = β0 + β1*x, where λ is the expected count (or rate), β0 is the intercept (baseline log risk), and β1 is the coefficient for x (log risk ratio). To get probability of the Poisson distribution, we can exponentiate the linear predictor to get λ = exp(β0 + β1*x). Then we can use this λ as the probability of success in a Bernoulli trial to simulate our binary outcome y. Later on, we can add confounder to it on the equation as well.

table(y)

## y
##    0    1 
## 1228   72

Pretty small event isn’t it? Let’s start with this

Comparison

Poisson Regression

model_poisson <- glm(y ~ x, family = poisson)
(model_poisson_df <- tidy(model_poisson, exponentiate = T, conf.int = T))

## # A tibble: 2 Ă— 7
##   term        estimate std.error statistic  p.value conf.low conf.high
##   <chr>          <dbl>     <dbl>     <dbl>    <dbl>    <dbl>     <dbl>
## 1 (Intercept)   0.0675     0.147    -18.3  1.24e-74   0.0499    0.0890
## 2 x             0.622      0.245     -1.94 5.28e- 2   0.379     0.997

Remember, Poisson assumes that the mean is equal to the variance. But with binary outcome, the variance is p*(1-p), which is less than the mean (p). So, Poisson overestimates the variance, leading to wider confidence intervals. Also notice that the exponentiated coef of x is very close to our truth 0.6? 🙌 Wow! I would never have thought we can use Poisson model for binary data! Let’s keep an eye on the 95%CI width, it’s 0.6180093

Log Binomial Regression

model_bin <- tryCatch(
  glm(y ~ x, family = binomial("log")), #notice the link function here is log, not logit?
  error = function(e) NULL
)

(model_bin_df <- tidy(model_bin, exponentiate = T, conf.int = T))

## # A tibble: 2 Ă— 7
##   term        estimate std.error statistic  p.value conf.low conf.high
##   <chr>          <dbl>     <dbl>     <dbl>    <dbl>    <dbl>     <dbl>
## 1 (Intercept)   0.0675     0.142    -18.9  6.63e-80   0.0503    0.0880
## 2 x             0.622      0.239     -1.99 4.68e- 2   0.384     0.985

wow, same estimate! But smalled 95% CI, which is because of smaller SE! The width of 95%CI is 0.6007632. Now let’s take a look at logistic regression.

Now one might wonder, if log binomial regression is the OG, why don’t we use this? We can, but it has a convergence issue, especially when we have continuous covariates or high baseline risk. That is also why we sometimes have to use tryCatch for the error.

Logistic Regression with Logit Link

model_logit <- glm(y ~ x, family = binomial("logit"))
(model_logit_df <- tidy(model_logit, exponentiate = T, conf.int = T))

## # A tibble: 2 Ă— 7
##   term        estimate std.error statistic  p.value conf.low conf.high
##   <chr>          <dbl>     <dbl>     <dbl>    <dbl>    <dbl>     <dbl>
## 1 (Intercept)   0.0724     0.153    -17.2  3.06e-66   0.0530    0.0965
## 2 x             0.605      0.252     -1.99 4.62e- 2   0.365     0.984

OK, remember we mentioned early? the exponentiated coefficient does not give an RR, it’s OR, though very close to our truth, because the event is rare. But the width of 95%CI is 0.6189336. So, we can see that the logistic regression has a wider CI than the log-binomial regression, which is expected because the log-binomial model is more efficient for estimating RR when the outcome is binary.

Poisson Regression with Robust Standard Error (Modified Poisson)

(model_modpoisson <- coeftest(model_poisson, vcov = vcovHC(model_poisson, type = "HC0")) |>
  tidy(conf.int = TRUE) |> #apparently exp not supported in coeftest from broom
  mutate(across(c(estimate, conf.low, conf.high), exp)))

## # A tibble: 2 Ă— 7
##   term        estimate std.error statistic  p.value conf.low conf.high
##   <chr>          <dbl>     <dbl>     <dbl>    <dbl>    <dbl>     <dbl>
## 1 (Intercept)   0.0675     0.142    -18.9  6.67e-80   0.0511    0.0893
## 2 x             0.622      0.239     -1.99 4.68e- 2   0.389     0.993

Alright, accurate estimate (well, we didn’t change it from Poisson result), and smaller CI width than Poisson regression, but still wider than log-binomial regression. The width of 95%CI is 0.6040859 by very small amount! And doesn’t seem like there is convergence issue with modified poisson, which is good. 👍 We can read more about this here Zou G - A Modified Poisson Regression Approach to Prospective Studies with Binary Data

Let’s compare all of them with dataviz

High Event Rates

Let’s set baseline rate to 80%

## y
##   0   1 
## 441 859

## # A tibble: 4 Ă— 5
##   model            estimate conf.low conf.high width
##   <fct>               <dbl>    <dbl>     <dbl> <dbl>
## 1 Poisson             0.631    0.548     0.724 0.176
## 2 Log Binomial        0.631    0.577     0.686 0.109
## 3 Logistic            0.253    0.197     0.323 0.125
## 4 Modified Poisson    0.631    0.578     0.688 0.109

Wow, how about that! OR no longer is interchangeable as RR, 95% CI is wider for Poisson but for log binomial and modified Poisson, the 95% CIs are narrower and close to each other. Let’s increass the RR and see if log binomial breaks

High RR

Let’s set RR as 2 and base rate as 30%

## y
##   0   1 
## 749 551

## # A tibble: 4 Ă— 5
##   model            estimate conf.low conf.high width
##   <fct>               <dbl>    <dbl>     <dbl> <dbl>
## 1 Poisson              2.02     1.70      2.42 0.713
## 2 Log Binomial         2.02     1.77      2.33 0.558
## 3 Logistic             3.42     2.72      4.31 1.59 
## 4 Modified Poisson     2.02     1.77      2.32 0.556

Wow! log binomial and modified poisson still win! Look at logistic regression, the exponentiated coefficient is no longer close to RR. So far this still doesn’t break log bin, let’s push further.

Very High RR

Let’s set RR to 15? base rate of 6%

## y
##   0   1 
## 697 603

## # A tibble: 4 Ă— 5
##   model            estimate conf.low conf.high  width
##   <fct>               <dbl>    <dbl>     <dbl>  <dbl>
## 1 Poisson              13.3     9.98      18.2   8.26
## 2 Log Binomial         NA      NA         NA    NA   
## 3 Logistic            124.     84.2      187.  103.  
## 4 Modified Poisson     13.3    10.1       17.6   7.57

## Warning: Removed 1 row containing missing values or values outside the scale range
## (`geom_pointrange()`).

umm, ok that’s a bit hard to see. We notice that log binomial did not converge and logistic regression is estimating odds ratio, not RR lol. Let’s remove logisic regression and compare poisson to modified poisson.

Alright! We’ve seen in the past that log binomial and modified poisson estimates and SE are pretty close, and since our log binomial model could not converge, modified Poisson is our winner! 🥇

Repeat Measures

Alright, now that we’re convinced (though we really need to do lots of replicates of the above to “convince” ourselves that it’s not just one dataset that did this - let’s spawn an agent to push these to computer cluster and replicate for us while we go on… if you’re curious click here), let’s insert some interesting thing here. What if, we have repeated measure? Let’s simulate 65 subjects with 20 visits each, treatment fixed per subject and a subject-level random effect, then checked how often each method’s 95% CI actually contained the true RR (~0.63).

library(geepack)

set.seed(1)
n_id <- 65     # subjects
n_visit <- 20  # visits per subject (65 x 20 = 1300 rows)
baseline <- log(0.15) # 15% base rate
rr <- log(0.6) # RR of 0.6

id <- rep(1:n_id, each = n_visit)
x <- rep(rbinom(n_id, 1, 0.5), each = n_visit) # treatment fixed per subject
u <- rep(rnorm(n_id, 0, 1), each = n_visit)    # subject random effect, strong clustering
p <- pmin(exp(baseline + rr*x + u), 1)
y <- rbinom(n_id * n_visit, 1, p)

df <- tibble(id, x, y)

table(df$y)

## 
##    0    1 
## 1078  222

# true marginal RR (pmin caps a few risks at 1, so it's slightly off 0.6)
u_big <- rnorm(1e6, 0, 1)
true_rr <- mean(pmin(exp(baseline + rr + u_big), 1)) / mean(pmin(exp(baseline + u_big), 1))
true_rr

## [1] 0.6271079

# models
model_poisson_rm <- glm(y ~ x, family = poisson, data = df)
model_gee_rm <- geeglm(y ~ x, family = poisson("log"), data = df, id = id, corstr = "independence")

exp_ct <- function(ct) {
  ct |>
    tidy(conf.int = TRUE) |>
    mutate(across(c(estimate, conf.low, conf.high), exp))
}

results_rm <- bind_rows(
  tidy(model_poisson_rm, exponentiate = T, conf.int = T) |> mutate(model = "Poisson"),
  coeftest(model_poisson_rm, vcov = vcovHC(model_poisson_rm, type = "HC0")) |> exp_ct() |> mutate(model = "Modified Poisson"),
  coeftest(model_poisson_rm, vcov = vcovCL(model_poisson_rm, cluster = ~id, type = "HC0", cadjust = FALSE)) |> exp_ct() |> mutate(model = "Modified Poisson (cluster)"),
  tidy(model_gee_rm, exponentiate = T, conf.int = T) |> mutate(model = "GEE")
) |>
  filter(term == "x") |>
  select(model, estimate, std.error, conf.low, conf.high) |>
  mutate(width = conf.high - conf.low,
         model = factor(model, levels = c("Poisson", "Modified Poisson", "Modified Poisson (cluster)", "GEE")))

results_rm

## # A tibble: 4 Ă— 6
##   model                      estimate std.error conf.low conf.high width
##   <fct>                         <dbl>     <dbl>    <dbl>     <dbl> <dbl>
## 1 Poisson                        1.13     0.134    0.867      1.47 0.603
## 2 Modified Poisson               1.13     0.122    0.888      1.43 0.547
## 3 Modified Poisson (cluster)     1.13     0.243    0.701      1.82 1.12 
## 4 GEE                            1.13     0.243    0.701      1.82 1.12

results_rm |>
  ggplot(aes(x = model, y = estimate, ymin = conf.low, ymax = conf.high)) +
  geom_pointrange() +
  geom_hline(yintercept = true_rr, linetype = "dashed", color = "red") +
  ylim(0, 2.5)

Wow! Remember we said poisson SE is bigger than modified poisson, but now both poisson and modified poisson have narrower 95%CI than the clustering methods. Which is right? I mean, we did simulate repeated measures with strong clustering, so the clustering methods should be more appropriate, right? But the 95%CI didn’t even cover the truth… let’s call the agent again.

Holy Mac! Yea poisson and modified poisson without clusterings have narrow CI but look at their coverage! Poor! And the clustering methods have wider CI but their coverage is much better, close to 95%. So, we can see that the clustering methods are more appropriate for repeated measures data, even though they have wider confidence intervals. Alright, right now I’m pretty convinced about modified poisson and use clustering if appropriate. Next, let’s see if we can estimate RR with g-computation and bootstrap by subject to see if we can get a similar result as modified poisson with robust SE + clustering.

RR Without Modified Poisson with LR G-computation

# g-computation: fit logistic, predict everyone as treated vs untreated, take ratio of mean risks
gcomp_rr <- function(data) {
  m <- glm(y ~ x, family = binomial("logit"), data = data)
  p1 <- mean(predict(m, newdata = mutate(data, x = 1), type = "response"))
  p0 <- mean(predict(m, newdata = mutate(data, x = 0), type = "response"))
  p1 / p0
}

rr_gcomp <- gcomp_rr(df)

# bootstrap by subject: resample subjects (not rows), keep all their visits together
df_split <- split(df, df$id)

set.seed(1)
boot_rr <- map_dbl(1:1000, ~ {
  samp <- sample(names(df_split), replace = TRUE)
  boot_df <- bind_rows(df_split[samp], .id = "boot_id") # new id so duplicated subjects count separately
  gcomp_rr(boot_df)
})

results_gcomp <- tibble(
  model = "G-comp (cluster bootstrap)",
  estimate = rr_gcomp,
  conf.low = quantile(boot_rr, 0.025),
  conf.high = quantile(boot_rr, 0.975)
) |>
  mutate(width = conf.high - conf.low)

results_gcomp

## # A tibble: 1 Ă— 5
##   model                      estimate conf.low conf.high width
##   <chr>                         <dbl>    <dbl>     <dbl> <dbl>
## 1 G-comp (cluster bootstrap)     1.13    0.690      1.80  1.11

bind_rows(results_rm, results_gcomp) |>
  mutate(model = factor(model, levels = c("Poisson", "Modified Poisson", "Modified Poisson (cluster)", "GEE", "G-comp (cluster bootstrap)"))) |>
  ggplot(aes(x = model, y = estimate, ymin = conf.low, ymax = conf.high)) +
  geom_pointrange() +
  geom_hline(yintercept = true_rr, linetype = "dashed", color = "red") +
  ylim(0, 2.5)

Look at that! G-computation with logistic regression gives us the exact same RR as modified Poisson. That makes sense, because with only x in the model, both are just computing the risk in the treated divided by the risk in the untreated. And bootstrapping by subject (not by row!) gives a 95% CI almost identical to the cluster robust SE and GEE. Should we call the agent again?

Et viola! Same same! woo hoo! 🙌

Replicates of Our 4 Scenarios

Now we’ve seen it! All 4 scenarios with replicates showed pretty convincing evidence that modified poisson is a good choice for estimating risk ratios, especially when log-binomial regression fails to converge. Not only we care about convergence and narrow CI, we care about appropriate CI with appropriate coverage as we saw from the repeated measures simulation.

Opportunities For Improvement

  • No confounders yet! We only had one binary treatment. I heard this is where log-binomial really struggles to converge.
  • Explore non-collapsibility of OR
  • Other GEE correlations. We only used independence, to match modified Poisson. What about exchangeable?
  • Rescuing log-binomial. We let it fail. Starting values might fix it. Is it the model, or just the algorithm?
  • Risk difference. We only looked at RR. Absolute risk difference often matters more to patients and clinicians.
  • Need to try to solving robust/sandwich SE from scratch to know how it actually works

Lessons learnt

  • learnt what modified poisson with robust SE is
  • learnt about clustering with vcovCl or geepack package
  • learnt that modified poisson with robuse SE w/wo clustering (depending on data) really matters because SE could be wrong either too narrow or too wide!
  • thank goodness to Claude code where I can paste my rmarkdown of my simulation and models so that it can send replicates to cluster and create the plot for me! Saved me a lot of time trying to code from the ground up! Will definitely be using this more!

If you like this article:

To leave a comment for the author, please follow the link and comment on their blog: r on Everyday Is A School Day.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

Never miss an update!
Subscribe to R-bloggers to receive
e-mails with the latest R posts.
(You will not see this message again.)

Click here to close (This popup will not appear again)