Site icon R-bloggers

O*NET ratings, job descriptions and classification codes as measures of occupational similarity

Giles Dickenson-Jones
[This article was first published on Data Analytics and AI Archives - Giles, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

TLDR: The last post identified potential transition pathways for truck drivers based on how closely other occupations matched them on skills, abilities and knowledge. This post follows the same basic approach across a wider range of occupations and asks whether two proxies could stand in for the ratings where no O*NET exists: a small language model comparing the meaning of job descriptions and the distance between two jobs’ US Standard Occupational Classification (SOC) codes. Both agree with the skill-based scores on average and appear to complement each other. The SOC code helps separate jobs in different classes of work and the descriptions are better at ranking occupations within similar groups. Together they explain about a third of the variation in the skill-based score overall, and 10 to 15 percent within a major group, which is where a real job seeker is likely to look. Neither is a substitute for the O*NET’s worker characteristic data, but both provide a reasonable sense check of transition pathway estimates which can be useful when it’s not possible to use data from the O*NET.

Background

In the last post I reproduced the similarity scores estimated by Bratanova et al. (2026). They used worker characteristic data from the O*NET to do this, with the basic idea being that two jobs that demand similar things of their workers should on average be easier to transition between. There’s of course more to their paper than this, but this is the crux of the approach.

They’re certainly not the first to use the O*NET data in this way.1 There are also some great interactive tools2 that intuitively illustrate how the approach works. Unfortunately, only found these after I’d pored over the literature as an adviser on a similar project in India. Because the SOC didn’t map cleanly to India’s labour market3, we decided to dropped the O*NET in any case. But, the project left me with three questions that this post tries to answer:

Occupational Similarity: Three Ways

Had we used the O*NET, answering these questions would have been relatively simple: calculate similarity scores using occupation characteristic from the O*NET and compare these with scores calculated from text-based measures and a numeric indicator of classification similarity, which is more or less what I’m going to do here.

A primer on skill similarity and the O*NET

If you’re unfamiliar with the territory I’d recommend taking a look at my previous post where I explain why skill similarity measures are used for identifying transition pathways and the basic structure of the O*NET data that is drawn on. But, for those unwilling to leave this page:

Three ways to test whether two jobs are alike

Because I’m interested in testing the ideas rather than describing them, I’m not going to cover each approach in too much detail, but the TLDR summary of each approach is below:

Project set-up

Data:

Like the previous O*NET post, analysis uses version 31.0 of the O*NET database. You can download the data used in this post here.

Everything a reader might want to change sits in this one chunk: packages, paths, the rating scale bounds, the embedding model, the labels and the chart theme.

library(tidyverse)
library(janitor)
library(readxl)
library(text)
library(scales)

#where the data lives and where the figures get written
ref_dir_data   <- file.path(".", "Data")
ref_dir_images <- file.path(".", "images")
dir.create(ref_dir_images, showWarnings = FALSE)

#the O*NET rating files. The NAME of each entry becomes the element set label
#in the combined table, which is how essential and transferable skills stay apart.
ref_files_onet <- c(
  abilities           = "Abilities.xlsx",
  skills_essential    = "Essential Skills.xlsx",
  skills_transferable = "Transferable Skills.xlsx",
  knowledge           = "Knowledge.xlsx"
)

#the O*NET file holding one plain-English description per occupation
ref_file_desc <- "Occupation_Data.xlsx"

#O*NET rating scale bounds, used to put both ratings on a 0-100 scale.
#Importance is collected on a 1-5 scale, level on a 0-7 scale.
ref_scale_level_max <- 7
ref_scale_imp_min   <- 1
ref_scale_imp_max   <- 5

#For semantic similarity, all-MiniLM-L6-v2 has been used. 
ref_model_embed <- "sentence-transformers/all-MiniLM-L6-v2"

#readable labels for the five ratings we're comparing 
ref_labels_set <- c(
  abilities           = "Abilities",
  skills_essential    = "Essential skills",
  skills_transferable = "Transferable skills",
  knowledge           = "Knowledge",
  all                 = "Average Across Worker Characteristics"
)

#for chart labels
ref_labels_measure <- c(ref_labels_set, semantic = "Semantic (descriptions)")


#import SOC classification structure
ref_path_soc <- file.path("Data", "soc_structure_2018.xlsx")

# Import data
dta_soc_raw <- read_excel(
  ref_path_soc,
  sheet = "2018 Structure",
  skip = 8,
  col_names = c("major", "minor", "broad", "detailed", "soc_title"),
  col_types = "text"
)

# Data wrangling
# One row per code, with its level
dta_soc_long <- dta_soc_raw |>
  pivot_longer(major:detailed, names_to = "soc_level", values_to = "soc_code", values_drop_na = TRUE)

# One row per detailed occupation with parent codes and titles
lkp_soc_title <- setNames(dta_soc_long$soc_title, dta_soc_long$soc_code)

dta_soc_hier <- dta_soc_raw |>
  fill(major, minor, broad) |>
  filter(!is.na(detailed)) |>
  mutate(
    major_title = lkp_soc_title[major],
    minor_title = lkp_soc_title[minor],
    broad_title = lkp_soc_title[broad]
  ) |>
  select(major, major_title, minor, minor_title, broad, broad_title, detailed, detailed_title = soc_title)


#convert the SOC classification to format for plots
lkp_soc_major <- dta_soc_long |>
  filter(soc_level == "major") |>
  transmute(soc_major       = str_sub(soc_code, 1, 2),
            soc_major_label = str_c(soc_major, " ", str_remove(soc_title, " Occupations$")))

#colours for figures.
ref_col_desc        <- "#D4A017"
ref_col_soc_rank    <- "#1F3A5F"
ref_col_soc_raw     <- "#7A94B8"
ref_col_combo       <- "#2F7E6E"
ref_col_combo_light <- "#86B8AB"
ref_col_primary <- "#1E298D"  # dark blue
ref_col_accent  <- "#26CDDA"  # cyan
ref_col_grid    <- "#E5E7EB"
ref_col_text    <- "#121212"

#a shared theme so every chart in the post looks the same
ref_theme_post <- theme_minimal(base_size = 11) +
  theme(
    panel.grid.minor    = element_blank(),
    panel.grid.major    = element_line(colour = ref_col_grid),
    axis.text           = element_text(colour = ref_col_text),
    axis.title          = element_text(colour = ref_col_primary, face = "bold"),
    strip.text          = element_text(colour = ref_col_text, face = "bold", hjust = 0),
    legend.position     = "bottom",
    plot.title.position = "plot"
  )

#the extra theme settings the three heatmaps share: no gridlines, small
#rotated axis text and a wide legend bar
ref_theme_heatmap <- ref_theme_post +
  theme(
    panel.grid       = element_blank(),
    axis.text.x      = element_text(angle = 90, vjust = 0.5, hjust = 1, size = 7),
    axis.text.y      = element_text(size = 7),
    legend.key.width = unit(1.5, "cm")
  )

Import the data

The code below reads O*NET data adding source labels.

#read each O*NET rating file, tag it with its element set, and stack them
dta_onet_raw <- imap(ref_files_onet, \(ref_file, ref_set) {
  read_excel(file.path(ref_dir_data, ref_file)) |>
    clean_names() |>
    mutate(element_set = ref_set)
}) |>
  list_rbind() |>
  rename(soc_code  = o_net_soc_code,
         soc_label = title)

#one row per occupation: code, title and description
dta_desc_raw <- read_excel(file.path(ref_dir_data, ref_file_desc)) |>
  clean_names() |>
  rename(soc_code  = o_net_soc_code,
         soc_label = title)

#a lookup of code to title, taken from the ratings files
lkp_soc_labels <- dta_onet_raw |>
  distinct(soc_code, soc_label)

Exploratory analysis

Checking coverage

The code below does a simple sense check of the imported data by presenting the number of occupations covered and the number of characteristics measured across O*NET abilities, knowledge and skills. Each characteristic (or element set) should have the same number of values for importance and level.

Element setImportanceLevel
Abilities5252
Knowledge3333
Skills (essential)1010
Skills (transferable)2525
Total120120
#characteristic and description counts
sum_onet_coverage <- tibble(
  occupations_rated      = n_distinct(dta_onet_raw$soc_code),
  occupations_described  = n_distinct(dta_desc_raw$soc_code),
  rated_with_description = n_distinct(intersect(dta_onet_raw$soc_code,
                                                dta_desc_raw$soc_code))
)
#
sum_onet_coverage


#characteristic counts 
sum_onet_element_counts <- dta_onet_raw |>
  distinct(element_set, element_name, scale_name) |>
  count(element_set, scale_name, name = "nmb_elements") |>
  pivot_wider(names_from = scale_name, values_from = nmb_elements)

sum_onet_element_counts

Calculating skill-based similarity

We’ll follow a similar approach to the previous post when calculating skill-based similarity scores, with two main differences:

  1. We use Manhattan distance to calculate skill gaps via R’s dist() function. This is essentially the same calculation, but more memory efficient.
  2. Average skill gaps are calculated for each worker characteristic and then averaged, so abilities, knowledge, essential and transferable skills are weighted equally.

Put level and importance on a common scale

The O*NET rates level from 0 to 7 and importance from 1 to 5, so the code below scales both ratings so they are on a common scale:

dta_onet_scaled <- dta_onet_raw |>
  mutate(scale_name = str_to_lower(scale_name)) |>
  mutate(value_0_100 = case_when(
    scale_name == "level"      ~ 100 * data_value / ref_scale_level_max,
    scale_name == "importance" ~ 100 * (data_value - ref_scale_imp_min) /
                                       (ref_scale_imp_max - ref_scale_imp_min)
  )) |>
  select(soc_code, soc_label, element_set, element_name, scale_name, value_0_100)

Calculating skill gaps

In the last post, each origin occupation was paired with each potential destination job using a join before skill gaps were calculated. This worked, but required creating a large table with a lot of redundant information before making the calculation. The code below uses dist(), which is a more memory efficient way to make the same calculation by:

#split the rescaled ratings into the four sets, then add a fifth entry holding
#all of them together
dta_onet_by_set <- split(dta_onet_scaled, dta_onet_scaled$element_set)


# Stage 1: widen a set of ratings to one row per occupation, one column per
# element and scale (with columns names based on the element set, name and scale ).
fnc_widen_ratings <- function(dta_ratings) {
  dta_ratings |>
    select(soc_code, element_set, element_name, scale_name, value_0_100) |>
    pivot_wider(names_from  = c(element_set, element_name, scale_name),
                values_from = value_0_100)
}

#create wide dataframe with one row per occupation 
dta_onet_wide <- dta_onet_by_set |>
  map(fnc_widen_ratings)

#to see an example of the input data:
# tmp_dta_onet_wide_example<-dta_onet_wide$skills_transferable |> head()
# rm(tmp_dta_onet_wide_example)

#check that each worker requirement has the expected number of columns and rows
# (894 occupations and 1 column for each element_name and level e.g 52 abilities and two ratings for each (level and importance)  (52*2=104)) 
chk_wide_dimensions <- dta_onet_wide |>
  map(\(dta_wide) tibble(nmb_occupations = nrow(dta_wide),
                         nmb_columns     = ncol(dta_wide) - 1)) |>
  list_rbind(names_to = "element_set")

chk_wide_dimensions

# Stage 2: Calculate the Manhattan distance between every pair of rows, divided by the
# number of columns to make it an average gap. 
fnc_pair_gaps <- function(dta_wide) {
  
  tmp_matrix <- dta_wide |>
    #move the occupation code from a column to the row labels
    column_to_rownames("soc_code") |>
    #dist() needs a matrix of numbers, not a data frame
    as.matrix()
  
  #dist() calculates the total absolute skill gap between occupations across all characteristics and then takes the average.  
  #as.matrix() lays the input data out as a square with one row and one column
  #per occupation, presenting all job pair transiton paths 
  tmp_gaps <- as.matrix(dist(tmp_matrix, method = "manhattan")) / ncol(tmp_matrix)
  
  tmp_gaps |>
    #back to a data frame, with the row labels as a column of origin codes
    as_tibble(rownames = "soc_code_a") |>
    #stack the destination columns into rows: one row per pair
    pivot_longer(-soc_code_a, names_to = "soc_code_b", values_to = "gap_mean")
}

#average gap within each of the four sets
dta_gaps_by_set <- dta_onet_wide |>
  map(fnc_pair_gaps) |>
  list_rbind(names_to = "element_set")

#the combined score: average the four set-level gaps, so each set counts
#equally whether it has 33 characteristics or 52
dta_gaps_all <- dta_gaps_by_set |>
  summarise(gap_mean = mean(gap_mean), .by = c(soc_code_a, soc_code_b)) |>
  mutate(element_set = "all")

dta_gaps <- bind_rows(dta_gaps_by_set, dta_gaps_all)

#every set should produce the same number of pairs: occupations squared
chk_pairs_per_set <- dta_gaps |>
  count(element_set, name = "nmb_pairs")

chk_pairs_per_set

Turn gaps into scores

The code below rescales skill gaps so each job pair gets a similarity score between 0 and 100. Job pairs that require similar worker characteristics (and have smaller skill gaps) receive scores closer to 100, while job pairs with larger skill gaps receive similarity scores closer to zero. Because job pairs include moving to the same occupation, the minimum skill gap is zero.

#| label: similarity-scores

rlt_sim_ratings <- dta_gaps |>
  mutate(
    sim_score = 100 * (1 - (gap_mean - min(gap_mean)) / (max(gap_mean) - min(gap_mean))),
    .by = element_set
  )

#every occupation against itself, which should be 100 in every set
chk_self_pairs <- rlt_sim_ratings |>
  filter(soc_code_a == soc_code_b) |>
  summarise(sim_min = min(sim_score), sim_max = max(sim_score), .by = element_set)

chk_self_pairs

Scores seem to cluster by broad work classes

The heatmap above presents the average similarity score by each SOC major group excluding identical job pairs.

Because self-comparisons have been removed, the downward sloping diagonal line shows the average similarity score for jobs pairs in the same major group. As one might expect, similarity scores tend to be higher for occupation pairs that are both in the same major group. There is also some visual support for similarity scores being lower for larger differences in SOC codes, with lower average similarity scores in the bottom left and upper right third of the heatmap (where the SOC classification differences are highest).

However, neither pattern is unambiguously dominant, with a number of clusters being apparent in the heatmap. For instance, major groups 41 and 42 share unusually high average similarity scores across most other major groups up to 45 to 53. Correspondingly, the average similarity scores appear to be higher for occupation pairs within groups 45 to 53 than outside them. Major groups 11 to 19 exhibit a similar pattern.

Although exploring the source of this is worthy of another post, one explanation for the observed clustering might be that some worker requirements aren’t as widely shared as others, which might point to some inter-group transitions being more difficult than others. This clustering also isn’t all that surprising given the SOC has been designed to group similar worker groups together.

#average score between every pair of major groups, self-pairs excluded
sum_sim_ratings_major <- rlt_sim_ratings |>
  filter(soc_code_a != soc_code_b,
         element_set=="all") |>
  mutate(major_a = str_sub(soc_code_a, 1, 2),
         major_b = str_sub(soc_code_b, 1, 2)) |>
  summarise(value = mean(sim_score), .by = c(element_set, major_a, major_b)) |>
  mutate(panel = factor(ref_labels_set[element_set], levels = ref_labels_set))

plt_sim_ratings_major <- sum_sim_ratings_major |>
  #swap the two-digit codes for the readable major group names
  mutate(major_a = factor(major_a, levels = lkp_soc_major$soc_major,
                          labels = lkp_soc_major$soc_major_label),
         major_b = factor(major_b, levels = lkp_soc_major$soc_major)) |>
  ggplot(aes(x = major_b, y = major_a, fill = value)) +
  geom_tile(colour = "white", linewidth = 0.3) +
  #darker always means more alike
  scale_fill_gradient(low = ref_col_accent, high = ref_col_primary,
                      name = "Average similarity score (0-100)") +
  scale_y_discrete(limits = rev) +
  coord_equal() +
  facet_wrap(vars(panel)) +
  ref_theme_heatmap +
  labs(title ="Average similarity score by SOC major group", 
       x = "SOC major group", y = "SOC Major Group")

plt_sim_ratings_major

Estimate similarity from job descriptions

To estimate similarity scores from job descriptions, the O*NET’s occupation description data is used. For instance, the description for Emergency Management Directors (11-9161.00) is:
Plan and direct disaster response or crisis management activities, provide disaster preparedness training, and prepare emergency plans and procedures for natural (e.g., hurricanes, floods, earthquakes), wartime, or technological (e.g., nuclear power plant emergencies or hazardous materials spills) disasters or hostage situations.

These descriptions aren’t really designed to describe what a job requires in detail and is better thought of as narrative text meant to support the SOC taxonomy. Although this makes them a poor substitute for real job posting data, they do resemble the narrative descriptions used by other occupational classification systems, which is a good worst case scenario to test.

A sentence-embedding model is used to compare how similarly the meaning of two job descriptions are. These are neural network models trained on a large amount of text with the aim of turning a paragraph into a list of numbers designed to encode its underlying meaning. Because it isn’t simply comparing whether each paragraph share similar words, phrases like “drive a truck” and “operate a heavy vehicle” will receive similar scores that indicate they mean something similar. We’ll use cosine similarity, which gives a score of 1 when the meaning of seem to be pointing the same direction, 0 when they’re unrelated and <0 when they have opposite meanings.

I’ve arbitrarily chosen the all-MiniLM-L6-v2 transformer model to do compare job descriptions via the text package. I’m not going to claim a particularly rigorous process was used when selecting the model6, but it holds two advantages for this post:

  1. It’s small enough to be run locally on a laptop; and
  2. The model has been successfully used enough by researchers in adjacent fields7.

The code below has the model process the text of each O*NET job description that has a similarity score. Be warned, this can take time to run as it requires having the model process each of the 894 occupations

Note: The first run also downloads the model, which is about 100MB (excluding additional library and Python-related requirements).

textrpp_initialize()

#keep the descriptions of rated occupations only, in code order
dta_desc <- dta_desc_raw |>
  semi_join(lkp_soc_labels, by = join_by(soc_code)) |>
  arrange(soc_code)

#turn each description into a 384-number embedding. The last layer, averaged
#over tokens, is how this model's own authors produce sentence embeddings.
dta_desc_embed <- textEmbed(
  texts  = dta_desc |> select(description),
  model  = ref_model_embed,
  layers = -1,
  aggregation_from_layers_to_tokens = "concatenate",
  aggregation_from_tokens_to_texts  = "mean",
  keep_token_embeddings = FALSE)

#one row per occupation, one column per embedding dimension
dta_desc_vectors <- dta_desc_embed$texts$description

Compare every description with every other

textSimilarityMatrix() computes the cosine similarity between every job description pair to provide a standardized similarity measure across job description pairs. The argument center=TRUE tells the function to subtract values by their corresponding column means. This is the function’s default behaviour and is meant to reduce the chance of scores reflecting paragraphs looking similar as a result of sharing a common format and/or style e.g. all being job descriptions with a comparable style and format.

#cosine similarity between every pair of descriptions, after centring
mat_desc_sim <- as.matrix(
  textSimilarityMatrix(dta_desc_vectors, method = "cosine", center = TRUE)
)
dimnames(mat_desc_sim) <- list(dta_desc$soc_code, dta_desc$soc_code)

#from a matrix to one row per pair, same shape as the ratings table
rlt_sim_semantic <- mat_desc_sim |>
  as_tibble(rownames = "soc_code_a") |>
  pivot_longer(-soc_code_a, names_to = "soc_code_b", values_to = "sim_semantic")

#a description is identical to itself, so the diagonal should be 1
chk_semantic_self <- rlt_sim_semantic |>
  filter(soc_code_a == soc_code_b)

Descriptions show higher average similarity within the same group

The code below produces a heatmap of average semantic similarity scores for each job description pair once identical job pairs are dropped.

Once again, the downward sloping diagonal follows a similar pattern to the skill-based similarity scores: average similarity scores are higher for job pairs within the same SOC major group. The skill-based clustering of groups into wider families of work like the skill-based heatmap isn’t as apparent. Exploring why would make a good excuse for another post, but if I had to guess it’s probably a result of some combination of the following:

Note: It’s also likely that the descriptions of jobs in the same group leaned more on one another than jobs from other groups when they were drafted.

sum_sim_semantic_major <- rlt_sim_semantic |>
  filter(soc_code_a != soc_code_b) |>
  mutate(major_a = str_sub(soc_code_a, 1, 2),
         major_b = str_sub(soc_code_b, 1, 2)) |>
  summarise(value = mean(sim_semantic), .by = c(major_a, major_b))

plt_sim_semantic_major <- sum_sim_semantic_major |>
  mutate(major_a = factor(major_a, levels = lkp_soc_major$soc_major,
                          labels = lkp_soc_major$soc_major_label),
         major_b = factor(major_b, levels = lkp_soc_major$soc_major)) |>
  ggplot(aes(x = major_b, y = major_a, fill = value)) +
  geom_tile(colour = "white", linewidth = 0.3) +
  scale_fill_gradient(low = ref_col_accent, high = ref_col_primary,
                      name = "Average cosine similarity") +
  scale_y_discrete(limits = rev) +
  coord_equal() +
  ref_theme_heatmap +
  labs(x = "SOC major group", y = NULL)


plt_sim_semantic_major

Use the SOC code distance as a “measure” of similarity

I’ve used double quotes around measure to signify I’m making air quotes here, but to be clear: by measure I mean proxy. As even if similar jobs have been placed in the same group and groups are designed to correspond with wider groupings, such as white-collar, service, blue-collar and members of the military, code differences are not designed to precisely quantify where a job sits on some comparable continuum.

To provide an example of what I mean by this: Surgical Assistants (29-9093) and Healthcare Practitioners and Technical Workers (29-9099) are next to each other on the SOC, with codes that are six apart from one another. Yet, Home Health Aides (31-1121) has a code that’s a little over 12 thousand apart from both jobs, despite the three jobs being broadly comparable.

Although this is an extreme example, it does illustrate one of the core problems with relying on raw SOC codes as a proxy: small differences in assignments can result in large code differences that don’t correspond to how similar (or different) two jobs are. Added to this, even when these differences do provide a reasonable proxy for how similar (or different) two jobs are, this is unlikely to be constant across job pairs.

Knowing all this, a reasonable person might ask why look at SOC codes at all. Four reasons:

  1. Data availability: Data as rich as the O*NET sometimes isn’t available, can’t be drawn on, or isn’t appropriate to use for the labour market being examined.
  2. A bad proxy can still have useful information: Occupation codes might get the ordering of similar jobs right, even when it gets the size of differences wrong.
  3. To complement other similarity measures: When data is scarce, classification code differences might provide useful information that other data sources lack, making it useful as part of a wider portfolio of measures, such as part of a composite index.
  4. As an intuitive sense-check: When adjacent codes tend to be used for similar jobs, score differences can provide a simple sanity check of other similarity scores.

To limit the effect of adjacent occupations being assigned to different groups, the code below creates a difference measure based on an occupation’s ranking in the SOC, rather than raw code differences. This assumes that adjacent occupations are equally alike across the SOC, regardless of which groups they fall in. This approach was chosen as it feels easier to justify than relying on the raw codes, which sometimes exhibit large differences across classifications that are unlikely to correspond to how similar two jobs are.

Given differences in SOC code rankings are linear, it’s not surprising that the heatmap presents limited variation. However, it does present two patterns that you might expect real similarity scores should follow* when mapped on a classification system like the SOC:

  1. Occupations pairs from the same group should exhibit higher similarity with one another; and
  2. Similarity scores should decrease as the distance between groups increase.

*(Provided the classification system groups jobs in a way that aligns with the nature of the work.)

Note: The code also estimates raw SOC distance to allow testing whether within-group code differences have predictive power within the same minor group.

#position of each occupation in the SOC list: 1 = first code, 894 = last
lkp_soc_rank <- lkp_soc_labels |>
  arrange(soc_code) |>
  mutate(soc_rank = row_number()) |>
  select(soc_code, soc_rank)

#two readings of the SOC for every pair:
#1.) differences in code rankings
#2.) differences in raw codes
sum_soc_distance <- rlt_sim_ratings |>
  distinct(soc_code_a, soc_code_b) |>
  left_join(lkp_soc_rank, by = join_by(soc_code_a == soc_code)) |>
  left_join(lkp_soc_rank, by = join_by(soc_code_b == soc_code),
            suffix = c("_a", "_b")) |>
  mutate(
    soc_rank_distance = abs(soc_rank_a - soc_rank_b),
    #the full code as one integer, suffix included: "11-9199.11" -> 11919911.
    #Removing the point as well as the hyphen keeps the suffix as whole digits,
    #so two specialisations of one code are 1 apart rather than 0.01, which
    #rounding and floating point would otherwise erase.
    soc_numeric_a     = as.numeric(str_remove_all(soc_code_a, "[-.]")),
    soc_numeric_b     = as.numeric(str_remove_all(soc_code_b, "[-.]")),
    soc_code_distance = abs(soc_numeric_a - soc_numeric_b),
    major_a           = str_sub(soc_code_a, 1, 2),
    major_b           = str_sub(soc_code_b, 1, 2)
  )
#average list-position distance between every pair of major groups, self-pairs
#excluded. The raw code is kept for later; drawn up here it would show the same
#bands with different numbers.
sum_soc_distance_major <- sum_soc_distance |>
  filter(soc_code_a != soc_code_b) |>
  summarise(value = mean(soc_rank_distance), .by = c(major_a, major_b))

plt_soc_distance_major <- sum_soc_distance_major |>
  mutate(major_a = factor(major_a, levels = lkp_soc_major$soc_major,
                          labels = lkp_soc_major$soc_major_label),
         major_b = factor(major_b, levels = lkp_soc_major$soc_major)) |>
  ggplot(aes(x = major_b, y = major_a, fill = value)) +
  geom_tile(colour = "white", linewidth = 0.3) +
  #the ramp is flipped here, so darker still means closer
  scale_fill_gradient(low = ref_col_primary, high = ref_col_accent,
                      name = "Average difference in list position (darker = closer)") +
  scale_y_discrete(limits = rev) +
  coord_equal() +
  ref_theme_heatmap +
  labs(x = "SOC major group", y = NULL,
       title = "Average difference in SOC code ranking by major group")

plt_soc_distance_major

Do the measures agree?

But, do the three measures blend agree with one another?

The code below takes all three measures and combines them into a single dataframe so individual similarity measures can be compared against one another. Groupings are specified to help communicate insights from exploratory analysis (and arguments with Claude) that aren’t shown here for the sake of brevity. In short, the pairs cover:

The second group comes from the exploratory analysis left out of the post, which pointed to the SOC distance measure’s explanatory power varying based on how differently two jobs were classified. Because the ranking ignores the size of classification differences within an occupational group, raw SOC code differences might be better at separating jobs in the same group than the ranking, which I wanted to test.

#each pair is labelled by the first level at which its two codes differ. A pair
#that differs at the minor group shares the major group above it; a pair that
#differs only in the suffix is two specialisations of one detailed occupation.
ref_labels_differ <- c(
  "0" = "Differ at major group",
  "1" = "Differ at minor group",
  "2" = "Differ at broad occupation",
  "3" = "Differ at detailed occupation",
  "4" = "Differ in suffix only"
)

#labels for the second cut: the group both codes share
ref_labels_share <- c(
  "All pairs",
  "Same major group",
  "Same minor group",
  "Same broad occupation",
  "Same detailed occupation"
)

#combine ratings in a single dataframe
rlt_pairs <- rlt_sim_ratings |>
  #the combined ratings score only; the four sets had their turn above
  filter(element_set == "all") |>
  select(soc_code_a, soc_code_b, sim_ratings = sim_score) |>
  inner_join(rlt_sim_semantic, by = join_by(soc_code_a, soc_code_b)) |>
  inner_join(sum_soc_distance |>
               select(soc_code_a, soc_code_b, soc_rank_distance, soc_code_distance),
             by = join_by(soc_code_a, soc_code_b)) |>
  #each pair once, lower code first, self-pairs dropped
  filter(soc_code_a < soc_code_b) |>
  mutate(
    #distances flipped to closeness, so every measure reads higher = more alike
    soc_closeness      = -soc_rank_distance,
    soc_code_closeness = -soc_code_distance,
    #"53-3032.00": major "53", minor "53-3", broad "53-303", detailed "53-3032"
    shared_depth = case_when(
      str_sub(soc_code_a, 1, 7) == str_sub(soc_code_b, 1, 7) ~ 4,
      str_sub(soc_code_a, 1, 6) == str_sub(soc_code_b, 1, 6) ~ 3,
      str_sub(soc_code_a, 1, 4) == str_sub(soc_code_b, 1, 4) ~ 2,
      str_sub(soc_code_a, 1, 2) == str_sub(soc_code_b, 1, 2) ~ 1,
      TRUE                                                   ~ 0
    ),
    differ_at = factor(ref_labels_differ[as.character(shared_depth)],
                       levels = ref_labels_differ)
  )

Examining the number in each group

SubsetPairs
By the group the codes share
All pairs399,171
Same major group24,526
Same minor group7,851
Same broad occupation1,097
Same detailed occupation233

The code below outputs a frequency table for each group. An important characteristic of the groups shown in the table above is that most job pairs span different major groups.

#one row per cut and level, holding the pairs that belong to it
dta_pairs_subsets <- bind_rows(
  tibble(cut_by = "By where the codes first differ", depth = 0:4, subset = ref_labels_differ),
  tibble(cut_by = "By the group the codes share",    depth = 0:4, subset = ref_labels_share)
) |>
  mutate(
    cut_by    = fct_inorder(cut_by),
    subset = factor(subset, levels = c(ref_labels_differ, ref_labels_share)),
    pairs  = map2(cut_by, depth, \(ref_cut, ref_depth) {
      if (ref_cut == "By where the codes first differ") {
        filter(rlt_pairs, shared_depth == ref_depth)
      } else {
        filter(rlt_pairs, shared_depth >= ref_depth)
      }
    }),
    nmb_pairs = map_int(pairs, nrow)
  )

dta_pairs_subsets |>
  select(cut_by, subset, nmb_pairs)

Job description vs. skill-based similarity

The plot above compares similarity scores estimated from job descriptions and the skill-based scores. The relationship is positive, indicating that on average the same job pairs tend to have higher similarity scores using either measure. But, the fit isn’t great, particularly for job pairs with low similarity scores (which account for a large share of job pairs). The glib conclusion is that both measures tend to agree when the similarity of two jobs can be picked up by both measures, but not so much otherwise.

#20,000 random pairs, fixed by the seed so the figure is the same every render.
#The same sample feeds the SOC figure below.
set.seed(20260918)
dta_sample_pairs <- rlt_pairs |>
  slice_sample(n = 20000)

plt_scatter_descriptions <- dta_sample_pairs |>
  ggplot(aes(x = sim_semantic, y = sim_ratings)) +
  geom_point(colour = ref_col_desc, alpha = 0.1, size = 0.6) +
  geom_smooth(method = "loess", se = FALSE, colour = ref_col_text, linewidth = 1.1) +
  ref_theme_post +
  labs(x = "Semantic similarity of descriptions (cosine; more alike to the right)",
       y = "Ratings similarity (0-100)",
       title = "Ratings similarity against semantic similarity of descriptions, 20,000 random pairs")

plt_scatter_descriptions

SOC distance vs. skill-based similarity

The code below uses a scatter plot to compare SOC code distance measures for job pairs with the similarity scores estimated from worker characteristics.

Although I spent quite a bit of time digging into how the two measures correspond with one another (or don’t), I’ve dropped quite a lot of this analysis to keep the post short. Most of the analysis told the same basic story: both distance measures provide some* explanatory power across occupations, but most of their power comes from separating distinct classes of work from one another. The loess curve flattening at greater distances says the rest: once two jobs have SOC list differences of 300 or more, it has little to say about how alike they might be.

*(I was actually surprised how much explanatory power the measure had.)

#ggplot2 apparently can't give two facets different axis transformations, so the two
#panels are drawn separately and placed side by side with patchwork
library(patchwork)

plt_soc_position <- dta_sample_pairs |>
  ggplot(aes(x = soc_rank_distance, y = sim_ratings)) +
  geom_point(colour = ref_col_soc_rank, alpha = 0.1, size = 0.6) +
  geom_smooth(method = "loess", se = FALSE, colour = ref_col_text, linewidth = 1.1) +
  #reversed so closer pairs sit on the right
  scale_x_reverse(labels = label_comma()) +
  ref_theme_post +
  labs(x = "Difference in list position (closer pairs to the right)",
       y = "Ratings similarity (0-100)",
       subtitle = "SOC list position")

plt_soc_raw <- dta_sample_pairs |>
  ggplot(aes(x = soc_code_distance, y = sim_ratings)) +
  geom_point(colour = ref_col_soc_raw, alpha = 0.1, size = 0.6) +
  geom_smooth(method = "loess", se = FALSE, colour = ref_col_text, linewidth = 1.1) +
 
  scale_x_continuous(trans = compose_trans( "reverse"),
                     labels = label_comma()) +
  ref_theme_post +
  labs(x = "Difference in raw code (closer pairs to the right)",
       y = NULL,
       subtitle = "SOC raw code")

plt_scatter_soc <- plt_soc_position + plt_soc_raw +
  plot_annotation(title = "Ratings similarity against the SOC code, (20,000 random pairs)")

plt_scatter_soc

Correlation plot

The code above summarizes the basic message as the scatter plots using a correlation matrix. All measures agree with one another, but the SOC distance measures agrees more closely with the skill-based scores than the descriptions do. This doesn’t necessarily make distance measures a better proxy, but might just point to a similar set of worker characteristics being embodied by both the structure of the SOC and the O*NET’s worker characteristic ratings (more on this below).

#readable names for the four measures, in the order they were introduced
ref_labels_measure <- c(
  sim_ratings        = "Skill-based Ratings",
  sim_semantic       = "Descriptions",
  soc_closeness      = "SOC list position",
  soc_code_closeness = "SOC raw code"
)

#rank correlation between every pair of measures, all pairs
sum_cor <- rlt_pairs |>
  select(all_of(names(ref_labels_measure))) |>
  cor(method = "spearman") |>
  as_tibble(rownames = "measure_a") |>
  pivot_longer(-measure_a, names_to = "measure_b", values_to = "correlation") |>
  mutate(
    measure_a = factor(ref_labels_measure[measure_a], levels = ref_labels_measure),
    measure_b = factor(ref_labels_measure[measure_b], levels = ref_labels_measure)
  ) |>
  #the matrix is symmetric, so show each pair once, below the diagonal
  filter(as.integer(measure_a) > as.integer(measure_b))

plt_cor <- sum_cor |>
  ggplot(aes(x = measure_b, y = measure_a, fill = correlation)) +
  geom_tile(colour = "white", linewidth = 1) +
  geom_text(aes(label = round_half_up(correlation, 2),
                #dark text on light tiles, light text on dark ones
                colour = correlation > 0.5),
            size = 4.5) +
  scale_fill_gradient(low = "white", high = ref_col_soc_rank,
                      limits = c(0, 1), name = "Rank correlation") +
  scale_colour_manual(values = c(`TRUE` = "white", `FALSE` = ref_col_text),
                      guide = "none") +
  scale_y_discrete(limits = rev) +
  coord_equal() +
  ref_theme_post +
  theme(panel.grid = element_blank(),
        axis.text.x = element_text(angle = 30, hjust = 1)) +
  labs(x = NULL, y = NULL,
       title = "Rank correlation between the four measures, all pairs")

plt_cor

Three measures of the same hierarchy(?)

The code below produces a table of average similarity scores by each measure based on job pair grouping differences.

The table points to the SOC hierarchy doing a reasonable job of grouping similar jobs together. It also shows that most job pairs sit in different major groups and score a little above 50. It’s tempting to interpret this as indicating that the scores aren’t particularly good at differentiating jobs. But, this is actually what you’d expect of a database that’s designed to cover everything, rather than provide a representative snapshot of the labour market.

Where the codes first differPairsRatings similaritySemantic similaritySOC rank distance
Major group374,645550.20316
Minor group16,675700.3429
Broad occupation6,754740.3911
Detailed occupation864770.483
Suffix only233760.442
#average of each measure by where the codes first differ
sum_by_level <- rlt_pairs |>
  summarise(
    nmb_pairs         = n(),
    sim_ratings       = mean(sim_ratings),
    sim_semantic      = mean(sim_semantic),
    soc_rank_distance = mean(soc_rank_distance),
    .by = differ_at
  ) |>
  arrange(differ_at) |>
  mutate(sim_ratings       = round_half_up(sim_ratings, 0),
         sim_semantic      = round_half_up(sim_semantic, 2),
         soc_rank_distance = round_half_up(soc_rank_distance, 0))

sum_by_level

But the table also points to an uncomfortable truth about all three measures: it becomes increasingly difficult to use similarity scores to differentiate jobs once they share the same major group. Whether a job pair is in the same minor group, broad occupation, or have differences in the last two digits of the SOC code, the average similarity score shows limited movement. The plot below illustrates this by presenting the entire distribution of each measure by each group. Although the average score rises as the classification narrows, the boxes overlap so much that the classification tells you little about how alike two jobs are once they share a major group. The score still varies inside each grouping, which points to similarity scores being best used as a means to rank alternative pathways, rather than to put a precise number on how similar two jobs are.

Note: the association between similarity measures and SOC groups is not new, it’s a design principle of the system. The SOC classifies occupations based upon work performed, skills, education, training, and credentials8 , which are many of the same characteristics the skill based scores were based on. In addition, because managers are intentionally grouped with workers they manage and/or supervise, skill similarity is likely to be deeply nested in the hierarchy. In short, a skills-based score agreeing with a system designed around skills is exactly what you’d expect. It’s also why I felt a SOC code distance measure might provide a plausible proxy for similarity scores in the first place.

dta_plt_level_dist <- rlt_pairs |>
  transmute(differ_at,
            `Skill-based ratings`   = percent_rank(sim_ratings),
            `Job descriptions`      = percent_rank(sim_semantic),
            `SOC list position`     = percent_rank(soc_closeness)) |>
  pivot_longer(-differ_at, names_to = "measure", values_to = "percentile") |>
  mutate(measure = fct_inorder(measure))

plt_level_dist <- dta_plt_level_dist |>
  ggplot(aes(x = differ_at, y = percentile)) +
  geom_boxplot(fill = ref_col_grid, colour = ref_col_text, outlier.alpha = 0.03,
               outlier.size = 0.3, width = 0.6) +
  #the level means, which are the numbers in the table
  stat_summary(fun = mean, geom = "point", colour = ref_col_desc, size = 2.5) +
  scale_x_discrete(labels = \(x) str_remove(x, "^Differ ")) +
  facet_wrap(vars(measure), nrow = 1) +
  ref_theme_post +
  theme(axis.text.x = element_text(angle = 25, hjust = 1)) +
  labs(x = "Where the two codes first differ", y = "Percentile of the measure",
       subtitle = "All three measures rise with the hierarchy on average (gold points), and all three overlap heavily below the major group")

 
plt_level_dist

The explanatory power of descriptions appears to be steady across the hierarchy

To prove that I’m not siding with SOC distance measures as a proxy for similarity paths, the plot below presents insights resulting from a long series of arguments with Claude.

Presented in the plot are rank correlation of each measure with skill-based scores based on how job pairs are grouped within the SOC. Notice that across the largest set of pairs, SOC distance measures agree more closely with the skill-based rankings. But, in every other group job descriptions do a better job of predicting skill-based rankings. Notice also that the raw SOC code doesn’t beat the list measure either, which points to the information lost by using the ranking being minimal, even when jobs are in the same narrow classification.

Aside from this supporting the idea that the distance measure is proxying characteristics embodied by the SOC, it also points to job descriptions providing a more reliable proxy for skill similarity the deeper in the hierarchy the jobs being compared sit. And since that is where most workers are likely to look when they think about changing jobs, descriptions are likely to be the more useful measure for sense checking potential transition pathways.

# Rank correlation of the ratings score with each measure, on one set of
# pairs. Written as a function because it runs once per subset.
fnc_cor_with_ratings <- function(dta) {
  summarise(
    dta,
    `Descriptions`  = cor(sim_ratings, sim_semantic,       method = "spearman"),
    `List position` = cor(sim_ratings, soc_closeness,      method = "spearman"),
    `Raw code`      = cor(sim_ratings, soc_code_closeness, method = "spearman")
  )
}

sum_cor_by_subset <- dta_pairs_subsets |>
  mutate(result = map(pairs, fnc_cor_with_ratings)) |>
  select(cut_by, subset, nmb_pairs, result) |>
  unnest(result)

sum_cor_by_subset |>
  mutate(across(where(is.numeric), \(x) round_half_up(x, 2)))

plt_cor_by_subset <- sum_cor_by_subset |>
  ggplot(aes(y = subset)) +
  geom_vline(xintercept = 0, colour = ref_col_grid) +
  #the gap between the two measures in each row is the story
  geom_segment(aes(x = `List position`, xend = Descriptions, yend = subset),
               colour = ref_col_grid, linewidth = 2) +
  geom_point(aes(x = `Raw code`, colour = "Raw code"),
             shape = 21, fill = "white", size = 3.2, stroke = 1.1) +
  geom_point(aes(x = `List position`, colour = "List position"), size = 4) +
  geom_point(aes(x = Descriptions, colour = "Descriptions"), size = 4) +
  geom_text(aes(x = Descriptions, label = round_half_up(Descriptions, 2)),
            vjust = -1.2, size = 3, colour = ref_col_desc) +
  geom_text(aes(x = `List position`, label = round_half_up(`List position`, 2)),
            vjust = 2.1, size = 3, colour = ref_col_soc_rank) +
  #the pair count, at the left edge of each row
  geom_text(aes(x = -Inf, label = paste0(format(nmb_pairs, big.mark = ",", trim = TRUE),
                                         " pairs")),
            hjust = -0.1, size = 2.8, colour = "grey50") +
  scale_colour_manual(
    values = c(Descriptions = ref_col_desc, `List position` = ref_col_soc_rank,
               `Raw code` = ref_col_soc_raw),
    name   = NULL,
    guide  = guide_legend(override.aes = list(shape = c(16, 16, 21), fill = "white"))
  ) +
  scale_y_discrete(limits = rev) +
  scale_x_continuous(expand = expansion(mult = c(0.25, 0.08))) +
  #one panel per cut_by, each with its own five rows
  facet_wrap(vars(cut_by), scales = "free_y") +
  ref_theme_post +
  theme(panel.grid.major.y = element_blank()) +
  labs(x = "Rank correlation with the ratings score", y = NULL,
       title = "How each measure tracks the ratings, within each subset of pairs")

plt_cor_by_subset

Does the agreement hold across occupation groups?

The plot below presents the rank correlation of each measure with skill-based scores by occupational group, where job pairs are in the same major group.

Once again, description-based scores outperform the SOC distance measure in most groups, and where they don’t the gap is small. The clear exception is Architecture and Engineering, which plausibly stems from the SOC hierarchy separating engineers from the technicians and drafters who support them, a technical grouping that the description-based similarity scores largely miss.

#pairs where both occupations sit in the same major group. Each pair belongs to
#exactly one group, so no need to count both directions.
sum_cor_by_group <- rlt_pairs |>
  filter(shared_depth >= 1) |>
  mutate(major = str_sub(soc_code_a, 1, 2)) |>
  summarise(
    nmb_pairs        = n(),
    `Descriptions`   = cor(sim_ratings, sim_semantic,  method = "spearman"),
    `List position`  = cor(sim_ratings, soc_closeness, method = "spearman"),
    .by = major
  ) |>
  pivot_longer(c(Descriptions, `List position`),
               names_to = "measure", values_to = "correlation") |>
  mutate(major = factor(major, levels = lkp_soc_major$soc_major,
                        labels = lkp_soc_major$soc_major_label))

plt_cor_by_group <- sum_cor_by_group |>
  ggplot(aes(y = major)) +
  geom_vline(xintercept = 0, colour = ref_col_grid) +
  geom_point(aes(x = correlation, colour = measure), size = 2.8) +
  #pair count at the left edge of each row, so the thin groups announce themselves
  geom_text(data = \(d) distinct(d, major, nmb_pairs),
            aes(x = -Inf, label = paste0(format(nmb_pairs, big.mark = ",", trim = TRUE),
                                         " pairs")),
            hjust = -0.1, size = 2.6, colour = "grey50") +
  scale_colour_manual(values = c(Descriptions    = ref_col_desc,
                                 `List position` = ref_col_soc_rank),
                      name = NULL) +
  scale_y_discrete(limits = rev) +
  scale_x_continuous(expand = expansion(mult = c(0.3, 0.05))) +
  ref_theme_post +
  labs(x = "Rank correlation with the skill-based score", y = NULL,
       title = "How well each proxy measure tracks the skill-based score for pairs within a SOC major group")

plt_cor_by_group

Comparing the predictive power of similarity measures

The code below attempts to answer the question that inspired this post in the first place: if you don’t have detailed job characteristic data like the O*NET, how far might you get using national classification structures and job descriptions?

In an attempt to answer this, the explanatory power of each measure is compared across the same five job pair groupings and a group with all job pairs. For each group, a linear regression model is used to estimate the explanatory power of measures individually and when combined together. A model with a higher adjusted R squared score explains a larger share of differences in skill-based score. The gap between the best measure and the combined model can be thought of as the marginal predictive value of adding the other measure.

The results support the same stories that were presented before:

Note: p-values and standard errors have not been reported as the observations used in the model aren’t independent e.g. as the same job is included multiple times across job pairs. This isn’t expected to impact the r-squared scores, but will influence anything that relies on the number of independent observations.

#the models, defined once: names, formulas and colours in the same order
ref_models <- tribble(
  ~model,                             ~formula,
  "Descriptions only",                sim_ratings ~ sim_semantic,
  "SOC list position only",           sim_ratings ~ soc_closeness,
  "Descriptions + SOC list position", sim_ratings ~ sim_semantic + soc_closeness
)

ref_col_models <- c(
  `Descriptions only`                = ref_col_desc,
  `SOC list position only`           = ref_col_soc_rank,
  `Descriptions + SOC list position` = ref_col_combo
)

#the adjusted R-squared of each model on one set of pairs. Written as a function
#because it runs once per subset.
fnc_r_squared <- function(dta) {
  ref_models |>
    mutate(r_squared = map_dbl(formula,
                               \(ref_formula) summary(lm(ref_formula, data = dta))$adj.r.squared)) |>
    select(model, r_squared)
}

#all pairs first, as the whole-dataset benchmark, then the pairs cut by where
#their codes first differ
sum_models <- dta_pairs_subsets |>
  filter(subset == "All pairs" | cut_by == "By where the codes first differ") |>
  mutate(result = map(pairs, fnc_r_squared)) |>
  select(subset, nmb_pairs, result) |>
  unnest(result) |>
  mutate(subset = fct_relevel(subset, "All pairs"),
         model  = factor(model, levels = ref_models$model))

sum_models |>
  mutate(r_squared = round_half_up(r_squared, 3)) |>
  select(-nmb_pairs) |>
  pivot_wider(names_from = model, values_from = r_squared)

plt_models <- sum_models |>
  ggplot(aes(y = subset)) +
  geom_vline(xintercept = 0, colour = ref_col_grid) +
  #the gap between the best single measure and the combined model is what the
  #second measure adds
  geom_segment(data = \(d) d |> summarise(x = max(r_squared[model != "Descriptions + SOC list position"]),
                                          xend = r_squared[model == "Descriptions + SOC list position"],
                                          .by = subset),
               aes(x = x, xend = xend, yend = subset), colour = ref_col_grid, linewidth = 2) +
  geom_point(aes(x = r_squared, colour = model), size = 4) +
  geom_text(data = \(d) distinct(d, subset, nmb_pairs),
            aes(x = -Inf, label = paste0(format(nmb_pairs, big.mark = ",", trim = TRUE), " pairs")),
            hjust = -0.1, size = 2.8, colour = "grey50") +
  scale_colour_manual(values = ref_col_models, name = NULL) +
  scale_y_discrete(limits = rev) +
  scale_x_continuous(expand = expansion(mult = c(0.4, 0.05))) +
  ref_theme_post +
  theme(panel.grid.major.y = element_blank()) +
  labs(x = "Share of variation in the skill-based score explained (adjusted R²)", y = NULL,
       title = "How much of the skill-based score each proxy recovers, for all pairs and by where the two codes first differ")

plt_models

What each measure brings

The upshot of the regression results is that neither one of the measures can stand in as a substitute for skill-based similarity scores, but both measures appear to hold value in the absence of occupational data as detailed as the O*NET. A classification distance measure might say little about transition pathways for jobs that resemble each other, but might provide a useful metric for splitting occupations into distinct groups between which transition is less likely. From there, similarity scores based on job descriptions might be useful for ranking potential transition paths between similarly grouped job pairs.

The analysis also points to the measures being practically useful for sense-checking similarity scores, which was what I wanted to test in the first place. For instance, where a country’s occupational classification standards align with the ILO’s International Standard Classification of Occupations, the broad structure of occupation codes should behave in a similar way to the SOC. And classification systems that include text-based information of each job and occupational group provide a means for producing semantic similarity scores. Neither measure is likely to be as rich as the O*NET. But, when the O*NET can’t (or shouldn’t be) used for analysis, both provide accessible metrics for validating transition pathways estimated from non-traditional and unstructured data sources, such as online job postings, vocational curricula and survey data.9

Summing up

The post has its origins in a project to identify potential occupational pathways for a country with limited data and a labour force that looked nothing like most of the OECD. Neither problem is unique, but the comparability of local occupations to their US counterparts is a critically important consideration when deciding whether to use the O*NET, particularly when the pathways are intended to inform public policy.10

One solution to this is to leverage local data sources to develop a local database of occupational characteristics as a substitute to the O*NET. But, even if sufficient data and money exist to make this possible, it can be hard to know whether the identified pathways make sense. This post was meant to test whether description-based similarity measures and classification distance scores might serve as a basic sense check, which the analysis indicates they can. The classification distance measure might point to whether a pathway crosses a major occupational boundary that doesn’t make sense, the descriptions should help check if the rankings of transition pathways within a group look sane. Neither is likely to tell you that a particular pathway is correct, but together they will hopefully point to job transitions that are implausible.

One of the things I argued with Claude about while finalizing this post, was its use of the word “cheap” to describe the two proxy measures. I didn’t like the phrasing (and I still don’t), but Claude is right that both measures are cheap. One is produced by subtracting one classification code from the another. The other comes from a transformer model that runs on my laptop, takes <100MB of space and was deployed across a set of job descriptions that were never meant to be comprehensive outlines of a job. The measures are cheap, which makes it rather extraordinary that they can explain so much of a far richer dataset.

Another point that came to mind while writing this is that by making the skill-based score the thing to predict, I’ve implicitly assumed it’s the standard other measures should be judged against. However, it’s also possible that all three measures carry valuable information about an occupation that isn’t mutually shared. This is testable with the right data, but it’s a point worth keeping in mind as the skill-based score not aligning with either measure might also reflect it lacking important information. If so, the richest and most expensive dataset in the room comes with its own blind spots.

How AI was used to write this post: AI produced the first draft of the code and based on the code used in my last set of analysis of the O*NET. I then proceeded to heavily edit this until it answered the questions a human being (me) might be interested in. The bulk of the writing is my own, with AI only used when I needed inspiration for improving how some points were communicated.

  1. Raimi, D. and Greenspon, J., 2025. Finding the Right Fit: What Jobs Offer a Good Match for Fossil Fuel Workers’ Skills? (No. 25-06). Resources for the Future. ︎
  2. Resources for the Future, Skills Matching Explorer, https://www.rff.org/publications/data-tools/skills-matching-explorer/ ︎
  3. Nor was it necessarily appropriate: Lo Bello, S., Sanchez Puerta, M.L. and Winkler, H., 2019. From Ghana to America: The skill content of jobs and economic development (No. 12259). IZA Discussion Papers. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3390249 ︎
  4. As noted in the previous post, other factors matter too, such as wage differentials, the availability (and proximity) of jobs and the intrinsic benefits of occupations being compared. ︎
  5. U.S. Bureau of Labor Statistics (2018), 2018 SOC User Guide: Classification Principles and Coding Guidelines, https://www.bls.gov/soc/2018/soc_2018_class_prin_cod_guide.pdf ︎
  6. Claude suggested it and I checked if it made sense ︎
  7. See: Saroglou, S., Diamantaras, K., Preta, F., Delianidi, M., Benisis, A. and Meyer, C.J., 2025. Enhancing job matching: occupation, skill and qualification linking with the ESCO and EQF taxonomies. arXiv preprint arXiv:2512.03195. ︎
  8. U.S. Bureau of Labor Statistics, Standard Occupational Classification: User Guide, “Classification Principles”. https://www.bls.gov/soc/soc-user-guide.htm. Accessed 19 September 2026. ︎
  9. For instance, see: World Economic Forum, 2018. Towards a reskilling revolution: A future of jobs for all. Report, (link); Lassébie, J., Marcolin, L., Vandeweyer, M. and Vignal, B., 2021. Speaking the same language: A machine learning approach to classify skills in Burning Glass Technologies data. OECD Social, Employment and Migration Working Papers, (link); and Granata, J., Posadas, J. and Testaverde, M., 2021. Indonesia’s Online Vacancy Outlook: From Online Job Postings to Labor Market Intelligence 2020. World Bank: Washington, DC, USA. (link). ︎
  10. For instance, Lo Bello, S., Sanchez Puerta, M.L. and Winkler find large differences between non-routine and manual tasks when comparing developed and developing countries. See: Lo Bello, S., Sanchez Puerta, M.L. and Winkler, H., 2019. From Ghana to America: The skill content of jobs and economic development (No. 12259). IZA Discussion Papers. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3390249 ︎

The post O*NET ratings, job descriptions and classification codes as measures of occupational similarity appeared first on Giles.

To leave a comment for the author, please follow the link and comment on their blog: Data Analytics and AI Archives - Giles.
R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.
Exit mobile version