# I don’t know Fisher’s exact test, but I know Stan

**Higher Order Functions**, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

A few days ago, I watched a terrific lecture by Bob Carpenter on

Bayesian models. He started with a Bayesian approach to Fisher’s exact test. I

had never heard of this classical procedure, so I was curious to play with the

example. In this post, I use the same data that he used in the lecture and

in an earlier, pre-Stan blog post. I show how I would go about

fitting the model in Stan and inspecting the results in R.

## Problem statement

We observed the following data.

sex | n left handed | n right handed |
---|---|---|

male | 9 | 43 |

female | 4 | 44 |

Question: Is the rate of left-handedness different between the male and female

groups? Specifically, is left-handedness more likely in the male group?

### Classical approach

In frequentist statistics, we might run Fisher’s exact test. At least, that’s what

the flow charts tell us. To do that, we first put the data in a

matrix.

```
<span class="c1"># Create a matrix representation of the data for the fisher test
</span><span class="n">m</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">matrix</span><span class="p">(</span><span class="nf">c</span><span class="p">(</span><span class="m">9</span><span class="p">,</span><span class="w"> </span><span class="m">43</span><span class="p">,</span><span class="w"> </span><span class="m">4</span><span class="p">,</span><span class="w"> </span><span class="m">44</span><span class="p">),</span><span class="w"> </span><span class="n">nrow</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">2</span><span class="p">,</span><span class="w"> </span><span class="n">byrow</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kc">TRUE</span><span class="p">,</span><span class="w">
</span><span class="n">dimnames</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nf">list</span><span class="p">(</span><span class="n">sex</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nf">c</span><span class="p">(</span><span class="s2">"male"</span><span class="p">,</span><span class="w"> </span><span class="s2">"female"</span><span class="p">),</span><span class="w">
</span><span class="n">handedness</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nf">c</span><span class="p">(</span><span class="s2">"left"</span><span class="p">,</span><span class="w"> </span><span class="s2">"right"</span><span class="p">)))</span><span class="w">
</span><span class="n">m</span><span class="w">
</span><span class="c1">#> handedness
#> sex left right
#> male 9 43
#> female 4 44
</span>
```

We can run the two-tailed test: Are the two groups different?

```
<span class="n">fisher.test</span><span class="p">(</span><span class="n">m</span><span class="p">)</span><span class="w">
</span><span class="c1">#>
#> Fisher's Exact Test for Count Data
#>
#> data: m
#> p-value = 0.2392
#> alternative hypothesis: true odds ratio is not equal to 1
#> 95 percent confidence interval:
#> 0.582996 10.927993
#> sample estimates:
#> odds ratio
#> 2.283832
</span>
```

The output is a little verbose, but I do like how it spells out a sentence

describing the alternative hypothesis.

We can also consider one-sided test: Is left-handedness greater in the male

group?

```
<span class="n">fisher.test</span><span class="p">(</span><span class="n">m</span><span class="p">,</span><span class="w"> </span><span class="n">alternative</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"greater"</span><span class="p">)</span><span class="w">
</span><span class="c1">#>
#> Fisher's Exact Test for Count Data
#>
#> data: m
#> p-value = 0.1502
#> alternative hypothesis: true odds ratio is greater than 1
#> 95 percent confidence interval:
#> 0.7006563 Inf
#> sample estimates:
#> odds ratio
#> 2.283832
</span>
```

In both tests, we cannot reject the null hypothesis because the *p*-value is

greater than .05 :x:, so we would conclude that the two groups are not different.

But I don’t really know this test that well. We never covered it in any of my

stats classes, and indeed, this post is the first time I ever used the function

`fisher.test()`

. If I had never heard of the test, I am not quite sure what I

would have done. Maybe a logistic regression (*p* = .191 :x:).

## Creating a Stan model

I’m very not fluent in the classical bag of tricks, but that’s okay. I know some

Stan, I have an idea about how the data could have been generated, and that’s

good enough :relaxed:. I can just write down my data-generating story in a model

and let Stan compute a posterior distribution for the difference in handedness

rates between the two groups.

For my model, I’m going to suppose that in each group, there is a probability of

being left-handed called *θ* and that the counts we see result from a

binomial process. The 9 left-handed males we observe are the number of successes

from 52 observations of a process that “succeeds” with probability

*θ*_{male}.

To fit the model, we need a prior distribution. The prior’s job is to generate

possible *θ*’s, and we will use our data to update the prior. The Beta

distribution generates values between 0 and 1, so it’s an obvious choice.

In math, the model for a group would be:

We can use a flat, uninformative prior by using *a* = 1, *b* = 1. This

prior considers all probabilities from 0 to 1 as equally plausible.

```
<span class="n">library</span><span class="p">(</span><span class="n">ggplot2</span><span class="p">)</span><span class="w">
</span><span class="n">steps</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">seq</span><span class="p">(</span><span class="n">from</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span><span class="p">,</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">1</span><span class="p">,</span><span class="w"> </span><span class="n">by</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">.01</span><span class="p">)</span><span class="w">
</span><span class="n">ggplot</span><span class="p">(</span><span class="n">data.frame</span><span class="p">(</span><span class="n">x</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">steps</span><span class="p">,</span><span class="w"> </span><span class="n">y</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">dbeta</span><span class="p">(</span><span class="n">steps</span><span class="p">,</span><span class="w"> </span><span class="n">shape1</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">1</span><span class="p">,</span><span class="w"> </span><span class="n">shape2</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">1</span><span class="p">)))</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">geom_line</span><span class="p">(</span><span class="n">aes</span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="w"> </span><span class="n">y</span><span class="p">))</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">scale_x_continuous</span><span class="p">(</span><span class="n">breaks</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="m">0</span><span class="o">:</span><span class="m">10</span><span class="p">)</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="m">10</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">ylim</span><span class="p">(</span><span class="m">0</span><span class="p">,</span><span class="w"> </span><span class="m">2</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">labs</span><span class="p">(</span><span class="n">x</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"p(left-handed)"</span><span class="p">,</span><span class="w"> </span><span class="n">y</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"density"</span><span class="p">,</span><span class="w"> </span><span class="n">title</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"beta(1,1)"</span><span class="p">)</span><span class="w">
</span>
```

But I also think that 10-ish% of people are left handed. (I don’t know where I

first heard this number, but it’ll serve as my prior information.) I toyed

around with `shape1`

and `shape2`

parameters in `dbeta()`

until I got the prior

Beta(5, 40), which is peaked around .1-ish but wide enough to keep .5 and .15 as

plausible values too.

```
<span class="n">ggplot</span><span class="p">(</span><span class="n">data.frame</span><span class="p">(</span><span class="n">x</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">steps</span><span class="p">,</span><span class="w"> </span><span class="n">y</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">dbeta</span><span class="p">(</span><span class="n">steps</span><span class="p">,</span><span class="w"> </span><span class="n">shape1</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">5</span><span class="p">,</span><span class="w"> </span><span class="n">shape2</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">40</span><span class="p">)))</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">geom_line</span><span class="p">(</span><span class="n">aes</span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="w"> </span><span class="n">y</span><span class="p">))</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">scale_x_continuous</span><span class="p">(</span><span class="n">breaks</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="m">0</span><span class="o">:</span><span class="m">10</span><span class="p">)</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="m">10</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">labs</span><span class="p">(</span><span class="n">x</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"p(left-handed)"</span><span class="p">,</span><span class="w"> </span><span class="n">y</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"density"</span><span class="p">,</span><span class="w"> </span><span class="n">title</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"beta(5,40)"</span><span class="p">)</span><span class="w">
</span>
```

Let’s write out a really simple model in Stan. Okay, it *used to* be really

simple. Then I made the parameters for the Beta prior data values, and then I

created an option to just sample the prior distribution. But the core of it is

simple. The most important lines are the one with `~`

symbols. These

correspond to the sampling statements in the mathematical description of the

model.

```
<span class="n">model_code</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="s2">"
data {
int<lower=0> beta_a;
int<lower=0> beta_b;
int<lower=0> n_total_1;
int<lower=0> n_total_2;
int<lower=0> n_hits_1;
int<lower=0> n_hits_2;
int<lower=0, upper=1> sample_prior_only;
}
parameters {
real<lower=0, upper=1> theta_1;
real<lower=0, upper=1> theta_2;
}
model {
theta_1 ~ beta(beta_a, beta_b);
theta_2 ~ beta(beta_a, beta_b);
if (sample_prior_only != 1) {
n_hits_1 ~ binomial(n_total_1, theta_1);
n_hits_2 ~ binomial(n_total_2, theta_2);
}
}
generated quantities {
real diff;
diff = theta_1 - theta_2;
}
"</span><span class="w">
</span>
```

The `generated quantities`

block runs on every sample of the posterior

distribution. Here, we compute the

*θ*_{male} − *θ*_{female} on every

draw. Computing the difference inside the model code means that Stan

will treat the `diff`

values like any other parameter of the model. It will show

up in summary functions and in plots of model parameters. That saves us some

work later on.

```
<span class="n">library</span><span class="p">(</span><span class="n">rstan</span><span class="p">)</span><span class="w">
</span><span class="c1">#> Loading required package: StanHeaders
#> rstan (Version 2.15.1, packaged: 2017-04-19 05:03:57 UTC, GitRev: 2e1f913d3ca3)
#> For execution on a local, multicore CPU with excess RAM we recommend calling
#> rstan_options(auto_write = TRUE)
#> options(mc.cores = parallel::detectCores())
</span>
```

I begin by compiling the model. This step will create an executable

program that can sample from the model. I do the compilation in its own step so

that I can re-use the program for different versions of the model.

```
<span class="n">model_program</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">stan_model</span><span class="p">(</span><span class="n">model_code</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">model_code</span><span class="p">)</span><span class="w">
</span>
```

For convenience, I wrote a function that fits different versions of this model.

This step is not necessary, but I don’t like repeating myself. (Normally, you

would use `sampling(model_program, stan_data)`

to get samples from a

model-program.)

```
<span class="n">run_model</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="k">function</span><span class="p">(</span><span class="n">beta_a</span><span class="p">,</span><span class="w"> </span><span class="n">beta_b</span><span class="p">,</span><span class="w"> </span><span class="n">sample_prior_only</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="n">stan_data</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="nf">list</span><span class="p">(</span><span class="w">
</span><span class="n">beta_a</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">beta_a</span><span class="p">,</span><span class="w">
</span><span class="n">beta_b</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">beta_b</span><span class="p">,</span><span class="w">
</span><span class="n">n_total_1</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">9</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="m">43</span><span class="p">,</span><span class="w">
</span><span class="n">n_total_2</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">4</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="m">44</span><span class="p">,</span><span class="w">
</span><span class="n">n_hits_1</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">9</span><span class="p">,</span><span class="w">
</span><span class="n">n_hits_2</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">4</span><span class="p">,</span><span class="w">
</span><span class="n">sample_prior_only</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">sample_prior_only</span><span class="p">)</span><span class="w">
</span><span class="c1"># Use quietly() to hide the sampler's output text
</span><span class="w"> </span><span class="n">model</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">purrr</span><span class="o">::</span><span class="n">quietly</span><span class="p">(</span><span class="n">sampling</span><span class="p">)(</span><span class="n">model_program</span><span class="p">,</span><span class="w"> </span><span class="n">stan_data</span><span class="p">)</span><span class="w">
</span><span class="c1"># But print any warnings that would have appeared
</span><span class="w"> </span><span class="nf">invisible</span><span class="p">(</span><span class="n">lapply</span><span class="p">(</span><span class="n">model</span><span class="o">$</span><span class="n">warnings</span><span class="p">,</span><span class="w"> </span><span class="n">warning</span><span class="p">,</span><span class="w"> </span><span class="n">call.</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kc">FALSE</span><span class="p">))</span><span class="w">
</span><span class="n">model</span><span class="o">$</span><span class="n">result</span><span class="w">
</span><span class="p">}</span><span class="w">
</span>
```

The actual left-handed versus right-handed numbers are hard-coded, but I can

adjust the parameters for the Beta prior and toggle between sampling from the

prior and the posterior. Stan normally prints out verbose progress information

but I suppress that by using `purrr::quietly()`

. I still want warnings, so I

print them if they arise.

### Checking our prior information

Now, let’s draw samples from the priors of each model. This step lets us check

that our program works as expected. We know how the values should be

distributed—we made up the numbers!

```
<span class="n">m_informative_pd</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">run_model</span><span class="p">(</span><span class="n">beta_a</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">5</span><span class="p">,</span><span class="w"> </span><span class="n">beta_b</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">40</span><span class="p">,</span><span class="w"> </span><span class="n">sample_prior_only</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">1</span><span class="p">)</span><span class="w">
</span><span class="n">m_informative_pd</span><span class="w">
</span><span class="c1">#> Inference for Stan model: 7e241131e72a0ec0b2cfd4f9a73290f1.
#> 4 chains, each with iter=2000; warmup=1000; thin=1;
#> post-warmup draws per chain=1000, total post-warmup draws=4000.
#>
#> mean se_mean sd 2.5% 25% 50% 75% 97.5% n_eff Rhat
#> theta_1 0.11 0.00 0.05 0.04 0.08 0.11 0.14 0.22 3047 1
#> theta_2 0.11 0.00 0.05 0.04 0.08 0.11 0.14 0.21 2953 1
#> diff 0.00 0.00 0.06 -0.13 -0.04 0.00 0.04 0.13 3072 1
#> lp__ -32.40 0.02 0.99 -35.01 -32.81 -32.11 -31.68 -31.42 1776 1
#>
#> Samples were drawn using NUTS(diag_e) at Tue May 16 14:05:44 2017.
#> For each parameter, n_eff is a crude measure of effective sample size,
#> and Rhat is the potential scale reduction factor on split chains (at
#> convergence, Rhat=1).
</span><span class="w">
</span><span class="n">m_flat_pd</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">run_model</span><span class="p">(</span><span class="n">beta_a</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">1</span><span class="p">,</span><span class="w"> </span><span class="n">beta_b</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">1</span><span class="p">,</span><span class="w"> </span><span class="n">sample_prior_only</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">1</span><span class="p">)</span><span class="w">
</span><span class="n">m_flat_pd</span><span class="w">
</span><span class="c1">#> Inference for Stan model: 7e241131e72a0ec0b2cfd4f9a73290f1.
#> 4 chains, each with iter=2000; warmup=1000; thin=1;
#> post-warmup draws per chain=1000, total post-warmup draws=4000.
#>
#> mean se_mean sd 2.5% 25% 50% 75% 97.5% n_eff Rhat
#> theta_1 0.49 0.01 0.28 0.03 0.25 0.49 0.74 0.97 3134 1
#> theta_2 0.50 0.00 0.28 0.03 0.26 0.51 0.74 0.97 3247 1
#> diff -0.01 0.01 0.40 -0.77 -0.29 -0.01 0.28 0.75 3001 1
#> lp__ -3.90 0.03 1.09 -6.86 -4.34 -3.56 -3.12 -2.81 1362 1
#>
#> Samples were drawn using NUTS(diag_e) at Tue May 16 14:05:45 2017.
#> For each parameter, n_eff is a crude measure of effective sample size,
#> and Rhat is the potential scale reduction factor on split chains (at
#> convergence, Rhat=1).
</span>
```

We can confirm (by inspecting the `diff`

row) that the difference in

left-handedness in both groups is 0 according to our priors. The

informative prior says that the values between 0.04 and

0.22 are plausible rates of left-handedness in each group.

It’s worth a moment to reflect on how obviously wrong the uninformative prior is.

The central *θ* value in each group is .5. Therefore, half of the prior

samples assert that there are more left-handed individuals than right-handed

ones! If there is anything we know about handedness, it’s that left-handedness is

less common than right-handedness. *Uninformative* sometimes connotes “unbiased”

or “letting the data speak for itself”, but in this case, I would say

“gullible”.

## Sampling the posterior

Now, the fun part. We update our prior information with our data.

```
<span class="n">m_informative</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">run_model</span><span class="p">(</span><span class="n">beta_a</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">5</span><span class="p">,</span><span class="w"> </span><span class="n">beta_b</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">40</span><span class="p">,</span><span class="w"> </span><span class="n">sample_prior_only</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span><span class="p">)</span><span class="w">
</span><span class="n">m_informative</span><span class="w">
</span><span class="c1">#> Inference for Stan model: 7e241131e72a0ec0b2cfd4f9a73290f1.
#> 4 chains, each with iter=2000; warmup=1000; thin=1;
#> post-warmup draws per chain=1000, total post-warmup draws=4000.
#>
#> mean se_mean sd 2.5% 25% 50% 75% 97.5% n_eff Rhat
#> theta_1 0.14 0.00 0.04 0.08 0.12 0.14 0.17 0.22 3288 1
#> theta_2 0.10 0.00 0.03 0.05 0.07 0.09 0.11 0.17 2837 1
#> diff 0.05 0.00 0.05 -0.05 0.02 0.05 0.08 0.14 3020 1
#> lp__ -70.65 0.02 1.05 -73.56 -71.09 -70.33 -69.89 -69.63 1834 1
#>
#> Samples were drawn using NUTS(diag_e) at Tue May 16 14:05:45 2017.
#> For each parameter, n_eff is a crude measure of effective sample size,
#> and Rhat is the potential scale reduction factor on split chains (at
#> convergence, Rhat=1).
</span><span class="w">
</span><span class="n">m_flat</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">run_model</span><span class="p">(</span><span class="n">beta_a</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">1</span><span class="p">,</span><span class="w"> </span><span class="n">beta_b</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">1</span><span class="p">,</span><span class="w"> </span><span class="n">sample_prior_only</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span><span class="p">)</span><span class="w">
</span><span class="n">m_flat</span><span class="w">
</span><span class="c1">#> Inference for Stan model: 7e241131e72a0ec0b2cfd4f9a73290f1.
#> 4 chains, each with iter=2000; warmup=1000; thin=1;
#> post-warmup draws per chain=1000, total post-warmup draws=4000.
#>
#> mean se_mean sd 2.5% 25% 50% 75% 97.5% n_eff Rhat
#> theta_1 0.19 0.00 0.05 0.09 0.15 0.18 0.22 0.30 2987 1
#> theta_2 0.10 0.00 0.04 0.04 0.07 0.10 0.13 0.20 2511 1
#> diff 0.09 0.00 0.07 -0.04 0.04 0.09 0.13 0.21 3042 1
#> lp__ -43.14 0.02 1.00 -45.82 -43.54 -42.84 -42.43 -42.15 1768 1
#>
#> Samples were drawn using NUTS(diag_e) at Tue May 16 14:05:46 2017.
#> For each parameter, n_eff is a crude measure of effective sample size,
#> and Rhat is the potential scale reduction factor on split chains (at
#> convergence, Rhat=1).
</span>
```

The flat model puts the difference at 0.09 and 90% of the plausible

values fall in the interval [-0.02, 0.19]. The informative

model is more skeptical of higher left-handedness rates, so it puts the difference

at 0.05 with 90% of the values between [-0.03, 0.13].

Both of these intervals contain 0 and negative values :x:, so there is not much

evidence for higher left-handedness in the male group.

To compute a “Bayesian *p*-value”, we could ask what proportion of differences

are 0 or negative. There are more proper ways to make this inference in a

Bayesian framework, but this approach is the easiest and it works for a model

this simple. If 10% of the plausible values for the group differences are

negative, then we assign a 10% probability to a negative group difference.

```
<span class="n">df_flat</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">as.data.frame</span><span class="p">(</span><span class="n">m_flat</span><span class="p">)</span><span class="w">
</span><span class="n">mean</span><span class="p">(</span><span class="n">df_flat</span><span class="o">$</span><span class="n">diff</span><span class="w"> </span><span class="o"><=</span><span class="w"> </span><span class="m">0</span><span class="p">)</span><span class="w">
</span><span class="c1">#> [1] 0.09575
</span><span class="w">
</span><span class="n">df_informative</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">as.data.frame</span><span class="p">(</span><span class="n">m_informative</span><span class="p">)</span><span class="w">
</span><span class="n">mean</span><span class="p">(</span><span class="n">df_informative</span><span class="o">$</span><span class="n">diff</span><span class="w"> </span><span class="o"><=</span><span class="w"> </span><span class="m">0</span><span class="p">)</span><span class="w">
</span><span class="c1">#> [1] 0.15525
</span>
```

It’s also worth comparing the two models. I’ve recently become a fan of the

ggmcmc package for quick visualization

of Stan models. The package uses a function `ggs()`

to create a long dataframe

of MCMC samples. Then you plug those dataframes into various plotting functions

that start with `ggs_`

. I especially like how the package returns a plain

ggplot2 plot that I can easily adjust with a few extra lines of code.

For example, not much effort is required—after some practice and

trial-and-error, of course—to visualize the posterior samples in each model.

```
<span class="n">library</span><span class="p">(</span><span class="n">ggmcmc</span><span class="p">)</span><span class="w">
</span><span class="c1">#> Loading required package: tidyr
#>
#> Attaching package: 'tidyr'
#> The following object is masked from 'package:rstan':
#>
#> extract
</span><span class="w">
</span><span class="c1"># A helper dataframe for relabeling parameters. I'm writing them in a way that
# works with ?plotmath conventions.
</span><span class="n">labels</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">data.frame</span><span class="p">(</span><span class="w">
</span><span class="n">Parameter</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nf">c</span><span class="p">(</span><span class="s2">"theta_1"</span><span class="p">,</span><span class="w"> </span><span class="s2">"theta_2"</span><span class="p">,</span><span class="w"> </span><span class="s2">"diff"</span><span class="p">),</span><span class="w">
</span><span class="n">Label</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nf">c</span><span class="p">(</span><span class="s2">"theta[male]"</span><span class="p">,</span><span class="w"> </span><span class="s2">"theta[female]"</span><span class="p">,</span><span class="w"> </span><span class="s2">"theta[male] - theta[female]"</span><span class="p">))</span><span class="w">
</span><span class="c1"># Get ggmcmc's tidy dataframe of each model.
# ggs() doesn't like that labels I made have brackets so I am suppressing its
# warnings.
</span><span class="n">ggs_flat</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">suppressWarnings</span><span class="p">(</span><span class="w">
</span><span class="n">ggs</span><span class="p">(</span><span class="n">m_flat</span><span class="p">,</span><span class="w"> </span><span class="n">description</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"flat"</span><span class="p">,</span><span class="w"> </span><span class="n">par_labels</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">labels</span><span class="p">)</span><span class="w">
</span><span class="p">)</span><span class="w">
</span><span class="n">ggs_informative</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">suppressWarnings</span><span class="p">(</span><span class="w">
</span><span class="n">ggs</span><span class="p">(</span><span class="n">m_informative</span><span class="p">,</span><span class="w"> </span><span class="n">description</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"informative"</span><span class="p">,</span><span class="w"> </span><span class="n">par_labels</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">labels</span><span class="p">)</span><span class="w">
</span><span class="p">)</span><span class="w">
</span><span class="n">ggs_density</span><span class="p">(</span><span class="n">ggs_flat</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">facet_grid</span><span class="p">(</span><span class="n">Parameter</span><span class="w"> </span><span class="o">~</span><span class="w"> </span><span class="n">.</span><span class="p">,</span><span class="w"> </span><span class="n">labeller</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">label_parsed</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">ggtitle</span><span class="p">(</span><span class="s2">"flat prior: beta(1, 1)"</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">theme_grey</span><span class="p">(</span><span class="n">base_size</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">14</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">theme</span><span class="p">(</span><span class="n">legend.position</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"bottom"</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="c1"># so the two models can be compared
</span><span class="w"> </span><span class="n">xlim</span><span class="p">(</span><span class="m">-.2</span><span class="p">,</span><span class="w"> </span><span class="m">.5</span><span class="p">)</span><span class="w">
</span><span class="n">ggs_density</span><span class="p">(</span><span class="n">ggs_informative</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">facet_grid</span><span class="p">(</span><span class="n">Parameter</span><span class="w"> </span><span class="o">~</span><span class="w"> </span><span class="n">.</span><span class="p">,</span><span class="w"> </span><span class="n">labeller</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">label_parsed</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">ggtitle</span><span class="p">(</span><span class="s2">"informative prior: beta(5, 40)"</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">theme_grey</span><span class="p">(</span><span class="n">base_size</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">14</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">theme</span><span class="p">(</span><span class="n">legend.position</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"bottom"</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="c1"># so the two models can be compared
</span><span class="w"> </span><span class="n">xlim</span><span class="p">(</span><span class="m">-.2</span><span class="p">,</span><span class="w"> </span><span class="m">.5</span><span class="p">)</span><span class="w">
</span>
```

We can also compare the models together in a single plot by passing a list of

model dataframes into `ggs_caterpillar()`

.

```
<span class="n">ggs_caterpillar</span><span class="p">(</span><span class="n">D</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nf">list</span><span class="p">(</span><span class="n">ggs_flat</span><span class="p">,</span><span class="w"> </span><span class="n">ggs_informative</span><span class="p">),</span><span class="w"> </span><span class="n">line</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span><span class="p">,</span><span class="w">
</span><span class="n">thick_ci</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nf">c</span><span class="p">(</span><span class="m">0.05</span><span class="p">,</span><span class="w"> </span><span class="m">0.95</span><span class="p">),</span><span class="w"> </span><span class="n">thin_ci</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nf">c</span><span class="p">(</span><span class="m">0.025</span><span class="p">,</span><span class="w"> </span><span class="m">0.975</span><span class="p">))</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="c1"># Parse the labels as formatted math
</span><span class="w"> </span><span class="n">scale_y_discrete</span><span class="p">(</span><span class="w">
</span><span class="n">breaks</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nf">as.character</span><span class="p">(</span><span class="n">unique</span><span class="p">(</span><span class="n">ggs_flat</span><span class="o">$</span><span class="n">Parameter</span><span class="p">)),</span><span class="w">
</span><span class="n">labels</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">parse</span><span class="p">(</span><span class="n">text</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nf">as.character</span><span class="p">(</span><span class="n">unique</span><span class="p">(</span><span class="n">ggs_flat</span><span class="o">$</span><span class="n">Parameter</span><span class="p">))))</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">labs</span><span class="p">(</span><span class="n">caption</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"Intervals: thick 90%, thin 95%. Point: median."</span><span class="p">,</span><span class="w">
</span><span class="n">y</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kc">NULL</span><span class="p">,</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kc">NULL</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">theme_grey</span><span class="p">(</span><span class="n">base_size</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">14</span><span class="p">)</span><span class="w">
</span>
```

There’s a lot of useful information here. First, the intervals in the flat prior

model are wider than the ones for the informative model. The two models largely

agree on the values in the female group, although the flat prior model is wider.

These wider intervals indicate greater uncertainty about the parameter values.

The two models disagree on the male group, because the informative model assigns

little prior probability to values greater than .25 but the flat model doesn’t

discount those possibilities.

The models also demonstrate the **regularizing effect of prior information**.

Regularization broadly refers to techniques to avoid overfitting a dataset.

Priors can regularize a model by making it skeptical of certain parameter

values—in this case, high values of left-handedness. The male probabilities in

the flat model are basically pulled towards the values in the informative prior

(.1-ish). We can see this effect in how the midpoint is shifted in the flat

model versus the informative model.

I would be remiss if I didn’t end with the following disclaimer/trivia. In a

way, the question behind this post is ill-posed because handedness is not quite

a binary measure. Some years ago, I had a class on stuttering and fluency

disorders. (I used to be a speech pathologist.) There once was a lot of research

on the association between handedness and stuttering, and at some point,

researchers figured out that they could measure handedness as a continuous

measure. They gave people a survey asking which hands they use for certain tasks

and then computed a so-called dextrality quotient from the responses. I

had the concept driven home when I once saw my wife casually switch

between her left and right hands while brushing her teeth. I would probably poke

a hole through my cheek if I used my left hand! So: Handedness may be a matter

of degree. (Or maybe not. I mostly wanted to mention the dextrality quotient.

It’s fun to think about.)

**leave a comment**for the author, please follow the link and comment on their blog:

**Higher Order Functions**.

R-bloggers.com offers

**daily e-mail updates**about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.