pilotr: Pilot the study before running it

A Study That Does Not Exist Yet

The imagined experiment is a lexical-decision task. Every participant sees every target word after both a related and an unrelated prime. Reaction times follow a shifted lognormal distribution, and the model includes item frequency, participant vocabulary, and an interaction between vocabulary and priming.

The main assumptions are set out below.

The specification determines the analysis formula as well as the data generator. That prevents a quiet mismatch between the design used for power and the model eventually fitted.

model_formula(spec)
#> .y ~ prime + z_freq + z_vocab + prime_z_vocab + (1 + prime |
#> subject) + (1 + prime | item)

Looking at the Imagined Data

A single simulated data set is useful before any power calculation (DeBruine & Barr, 2021). It shows whether the assumptions produce recognisable observations. In Figure 1 the simulated reaction times are right-skewed, as intended, and unrelated primes shift the distribution slightly to the right.

Figure 1: Simulated Reaction Times After Related and Unrelated Primes.

The average priming effect is only part of the picture. The specification gives participants and items their own priming slopes, and trial-level noise adds to that variation, so some show a much larger difference than others. Figure 2 gives each unit its own row, and in about a third of each set the difference runs the other way.

Figure 2: Simulated Priming Effects for Each Item and Each Participant.

Detection Is Not Estimation

The first design analysis simulates the study repeatedly, fits the mixed model to each simulated data set, and records conventional power and the number of singular fits. It also reports Type S error, the chance that a significant estimate has the wrong sign, and Type M error, the factor by which significant estimates exaggerate the true effect (Gelman & Carlin, 2014).

power_at_plan <- power_mixed(spec, n_sims = n_sims, workers = workers)
power_at_plan
#> Simulation-based power over 12 replicates (alpha = 0.05)
#> fits: 12 attempted, 12 returned, 6 converged cleanly, 6 singular, 6 with warnings
#> note: many fits were boundary-singular, so this model is richer than the design supports
#> effect true power mcse ci95 n_sig type_s type_m
#> prime 0.05 0.667 0.136 [0.391, 0.862] 8 0 1.039
#> z_freq -0.03 0.250 0.125 [0.089, 0.532] 3 0 1.648
#> z_vocab -0.02 0.000 0.000 [0.000, 0.242] 0 NA NA
#> prime_z_vocab 0.02 0.167 0.108 [0.047, 0.448] 2 0 2.016

With twelve replicates, one simulated success moves a power figure by more than eight percentage points. A planning run would use the same code with several hundred replicates, enough to bring the Monte Carlo standard error in the mcse column down to an acceptable level. The power curve in Figure 3 comes from repeating the analysis at six sample sizes, from 16 to 96 participants.

Figure 3: Detection Probability for the Priming Effect Across Participant Counts. 12 replicates per design. All simulated designs contain 24 items.

Power asks whether an effect would clear a threshold. I also want to know whether its interval would exclude effects too small to matter. With a region of practical equivalence (ROPE; Kruschke, 2018) of plus or minus 0.02 on the log scale, about 8 ms here, precision becomes a separate design criterion. Across samples of 16 to 200 participants, the left panel of Figure 4 shows how often the 95% confidence interval excludes the ROPE and the right panel shows the interval’s mean width. The dashed line in the right panel marks a width of 0.06, below which an interval centred on the true effect of 0.05 would exclude the ROPE.

Figure 4: Precision, as the Probability That an Interval Excludes a Negligible Effect and as the Mean Width of That Interval, Across Participant Counts. 12 replicates per design. All simulated designs contain 24 items.

The Significant Estimates Are the Misleading Ones

A low-powered study misses real effects, and it also distorts the ones it finds. When power is low, only unusually large estimates pass the significance threshold, so significance selects for exaggeration. To see that selection directly, I varied the true effect with 30 participants and set the result, in Figure 5, beside the sample-size curve from above, in which the effect stays at 0.05.

Figure 5: Magnitude Exaggeration Among the Statistically Significant Estimates, Against Power. 12 replicates per design.

Type M error falls as power rises in both series, from about 1.2 and 1.4 at the lowest power to roughly 1 at the highest. With twelve replicates it also wanders below 1, which a planning run with several hundred replicates would not. Together, these diagnostics cover what a planning run has to examine: detection, precision, magnitude error and singular fits. The frequent singular fits in this example call for a second look at the random-effects structure before anyone is recruited. Keeping that structure maximal protects the Type I error rate (Barr et al., 2013), and trimming it when the data cannot support it protects power (Matuschek et al., 2017). A planning run lets me weigh that trade-off while the design can still change.