Level With Me: Can the WGI Predict the WJP’s Rule of Law Scores?

TLDR: The first post established that the WGI and WJP rule of law measures agree on how they rank countries. This follow-up tests the stricter question of whether the WGI can predict the WJP’s levels, rather than just the relative ranking / relative position of countries. It can: forecasting errors appear to be minimal and don’t vary significantly over time.

This is second post in a series examining relationships between the rule of law and economic and social outcomes.

In the previous post in the series I tested whether the World Governance Indicator’s (WGI) rule of law index (RoL) was a worthy substitute for the index produced by the World Justice Project when needing to take advantage of the WGI’s wider and longer coverage.

When I asked Claude for feedback it rudely provided it, arguing that my analysis was fine for confirming general agreement as to how the two measures ranked countries, but additional analysis was needed to test whether the WGI could reliably be used to predict RoL levels.

My first response was to tell Claude that nobody should be using composite indices in this way in the first place as it implied a level of precision they don’t have, but I decided it would be a useful addendum to the first post that I could post during my holiday in the Northern Territory.

Project Setup and Data

Once again, data used in this post can be downloaded here for the WGI and here for the WJP’s RoL index. These were current as of July 2025.

#load the packages we'll probably need
library(tidyverse)
library(readxl)
library(janitor)
library(countrycode)

#import WGI data
dta_wgi_2025<-read_excel("./Data/wgidataset_with_sourcedata-2025.xlsx",
                         sheet="rl") |> 
  clean_names()

#import World Justice Project RoL data
dta_wjp_rol<-read_excel('./Data/2025_wjp_rule_of_law_index_HISTORICAL_DATA_FILE.xlsx', sheet="Historical Data")|> 
  clean_names()

Data Cleaning

Data cleaning is identical to the last post and revolves around making sure that countries names and regions are consistently applied across the two indices.

#standardize column names
dta_wgi_2025<-dta_wgi_2025 |> 
  rename(country=economy_name,
         iso3c=economy_code,
         wgi_rol=governance_estimate_approx_2_5_to_2_5)

dta_wjp_rol<-dta_wjp_rol |> 
  rename(iso3c=country_code) |> 
  rename_with(~ str_replace(., "^x", "factor_"), starts_with("x"))

#change wjp's year variable to YYYY format and convert to numeric 
#(adopts the first 4 digit year when in YYYY-YYYY format)
dta_wjp_rol <- dta_wjp_rol |> 
  mutate(year = as.numeric(str_sub(year, 1, 4)))

#cold-heartedly drop columns I'm not interested in 
dta_wgi_2025<-dta_wgi_2025 |> 
  select(iso3c, income_classification, year,wgi_rol)
#drop country and region name labels so these can be standardized 
dta_wjp_rol<-dta_wjp_rol |> 
  select(-country_year,-country,-region) |> 
  rename(wjp_rol=wjp_rule_of_law_index_overall_score)


#merge dataframes
dta_rol_unified<-left_join(dta_wgi_2025,
                           dta_wjp_rol,
                           by = join_by(year, iso3c), 
                           keep=FALSE) 

#add standardized and region names country names

#define country code assignments for legacy / ambigious codes
#(Note: matches devised by Claude) 
ref_iso3c_custom_names <- c(ADO = "Andorra",
                            ANT = "Netherlands Antilles",
                            PRI = "Puerto Rico",
                            REU = "Réunion",
                            XKX = "Kosovo")

ref_iso3c_custom_regions <- c(ADO = "Europe & Central Asia",
                              ANT = "Latin America & Caribbean",
                              PRI = "Latin America & Caribbean",
                              REU = "Sub-Saharan Africa",
                              XKX = "Europe & Central Asia")

#assign country names and regions:
dta_rol_unified<-dta_rol_unified |> 
  mutate(country_name=countrycode(iso3c, 
                                  origin='iso3c',
                                  destination = 'country.name.en',    
                                  custom_match = ref_iso3c_custom_names),
         region=countrycode(iso3c, 
                            origin='iso3c',
                            destination = 'region',    
                            custom_match =ref_iso3c_custom_regions))

Testing Cardinal Substitution

The code below basically asks whether it’s possible to estimate the level of WJP’s RoL index using the WGI’s RoL measure using a simple linear model. At the outset, the model’s fit is quite high at 97 percent, but we could have guessed this from the cross-country scatter we produced in our last post.

# Cardinal substitution: does a WGI->WJP mapping reproduce WJP levels? -----
sum_rol_country <- dta_rol_unified |>
  filter(!is.na(wgi_rol), !is.na(wjp_rol)) |>
  summarise(wgi_rol_mean = mean(wgi_rol),
            wjp_rol_mean = mean(wjp_rol),
            .by = country_name)

mod_rol_calib <- lm(wjp_rol_mean ~ wgi_rol_mean, data = sum_rol_country)

#output the R squared
summary(mod_rol_calib)$r.squared |> round(2)

The second chunk calculates the Leave-One-Out Cross-Validation Root Mean Squared Error (LOO RMSE), which estimates prediction error for each country when it’s held out of the sample: a smaller LOO RMSE implies smaller forecast errors when predicting outside the sample. This is then scaled by the standard deviation of the country-level WJP means to make it easier to interpretat.

The results are encouraging: the LOO RMSE indicates that using the WGI to estimate WJP levels carries a typical error of less than three percent, which is roughly 19% of the cross-country spread in WJP scores. Suggesting the WGI can provide a reasonable proxy when the levels are important too.

# Leave-One-Out Cross-Validation Root Mean Squared Error (LOO RMSE)
rlt_rol_loo_rmse   <- sqrt(mean((residuals(mod_rol_calib) /
                                   (1 - hatvalues(mod_rol_calib)))^2))

#divide by standard deviation of WJP index means to put the figure in the context fo the data
rlt_rol_rmse_ratio <- rlt_rol_loo_rmse / sd(sum_rol_country$wjp_rol_mean)

Systematic Bias

Of course, because the LOO RMSE measures is calculated across the entire dataset, it might hide systematic bias, such as if the forecast accuracy varies depending on the WGI score.

This is tested in the plot below, by comparing the prediction error (residual) with the predicted value for the WJP’s index (using the WGI RoL measure). If there no systematic bias, the residuals should remain relatively stable across the predicted WJP values.

Alas, this doesn’t appear to be the case, with the model under-predicting the WJP’s RoL index at both ends. Although this bias is relatively mild at 0.05 (in WJP index units), it does suggest the WGI is likely to be a poorer substitute for the WJP’s index for countries at either end of the RoL spectrum.

dta_plt_rol_calib <- sum_rol_country |>
  mutate(wjp_pred = fitted(mod_rol_calib),
         resid    = wjp_rol_mean - wjp_pred)

plt_rol_calib <- ggplot(dta_plt_rol_calib, aes(wjp_pred, resid)) +
  geom_hline(yintercept = 0, linewidth = 0.3, col = "grey70") +
  geom_point(alpha = 0.6) +
  geom_smooth(method = "loess", se = FALSE, col = "grey40", linewidth = 0.5) +
  labs(x = "Predicted WJP (from WGI)", y = "Residual (WJP units)") +
  theme_classic()

plt_rol_calib

However, being an applied economist means I know a simple trick: add non-linear terms to everything and see if the smile disappears.

And it does, the smile is now a grimace: as is shown in the plot below adding a non-linear term seems to take care of the bias at either end. Notice the average errors no longer increase at either end, making the prediction relatively constant (and small) across the RoL spectrum. Suggesting the WGI can be trusted to predict the WJP index at either end of the spectrum provided we account for the non-linearity.

#reestimate the model with a non-linear WGI term 
mod_rol_calib_non_linear <- lm(wjp_rol_mean ~ wgi_rol_mean + I(wgi_rol_mean^2),
                               data = sum_rol_country)


dta_plt_rol_calib_nl <- sum_rol_country |>
  mutate(wjp_pred = fitted(mod_rol_calib_non_linear),
         resid    = wjp_rol_mean - wjp_pred)

plt_rol_calib_nl <- ggplot(dta_plt_rol_calib_nl, aes(wjp_pred, resid)) +
  geom_hline(yintercept = 0, linewidth = 0.3, col = "grey70") +
  geom_point(alpha = 0.6) +
  geom_smooth(method = "loess", se = FALSE, col = "grey40", linewidth = 0.5) +
  labs(x = "Predicted WJP (from WGI, quadratic)", y = "Residual (WJP units)") +
  theme_classic()

plt_rol_calib_nl

But does it drift?

The final question Claude suggested I ask was whether the WGI’s ability to predict the WJP’s index changes over time. The code below tests this by comparing a simple linear prediction model with one where the influence of the WGI changes over time.

The anova() function then compares results of both models to test whether allowing the WGI’s influence to change over time improves the model’s forecasting accuracy. If it does (indicated by a significant p-value), the WGI’s ability to predict the WJP’s index might not be stable over time.

The result: a p value of 9%, which points to the time invariant model being good enough for predicting the WJP’s index.

#estimate a simple linear model with WGI's influence kept constant over time
mod_rol_calib_flat <- lm(wjp_rol ~ wgi_rol, data = dta_rol_unified)

#estimate a model where WGI's influence varies over time
mod_rol_calib_yr   <- lm(wjp_rol ~ wgi_rol * factor(year), data = dta_rol_unified)

#compare the prediction capacity of both models 
anova(mod_rol_calib_flat, mod_rol_calib_yr)

Look what Claude made me do…

So there you have it: the WGI can serve as a useful substitute for the WJP’s rule of law index whether you’re interested in RoL rankings or levels, provided you account for the non-linearity at either end of the RoL scale.

How AI was used for this post: As this additional analysis was Claude’s suggestion and I was hours away from a holiday I leaned heavily on Claude for the first draft of the code. Much of the text is mine, but I did ask AI for suggestions to make sure I was explaining the analysis accurately.

The post Level With Me: Can the WGI Predict the WJP’s Rule of Law Scores? appeared first on Giles.