Speeding up tidySummarizedExperiment through query optimisation and the plyxp backend

Stefano Mangiola

13 hours ago

[This article was first published on tidyomicsBlog, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

Contributors: Michael Love, Justin Landis, Pierre-Paul Axisa

The generality of tidySummarizedExperiment makes it easy to interface with several tidyverse packages (e.g. dplyr, tidyr, ggplot2, purrr, plotly). This is possible thanks to its approach of converting SummarizedExperiment objects to tibbles, performing operations, and converting back to the original format. This conversion process introduces substantial overhead when working with large-scale datasets. Each operation requires multiple data transformations, with the conversion to tibble format creating memory copies of the entire dataset, followed by the reverse conversion back to SummarizedExperiment. For datasets containing hundreds of samples and tens of thousands of genes, these repeated conversions can consume memory and add significant computational overhead to even simple operations such as filtering or grouping.

With the new tidySummarizedExperiment release (v1.19.7), we have introduced new optimisations that address these performance limitations. This optimisation is powered by:

Check for the query domain (assay, colData, rowData), and execute specialised operation.
Use of plyxp for complex domain-specific queries.

plyxp is a tidyomics package developed by Justin Landis, and first released as part of Bioconductor 3.20 in October 2024. It uses data-masking functionality from the rlang package to perform efficient operations on SummarizedExperiment objects.

< section id="motivation-and-design-principles" class="level3">

Motivation and design principles

This benchmark supports ongoing work to improve the performance of tidySummarizedExperiment. In this benchmark, we show up to 30x improvement in operations such as mutate().

The current optimisation is grounded in three principles:

Decompose operation series: break mutate(a=..., b=..., c=...) into single operations for simpler handling and clearer routing. Reference implementation in R/mutate.R (decomposition step) at L146.
Analyse scope: infer whether each expression targets colData, rowData, assays, or a mix (noting that the current analyser is likely over-engineered and could be simplified). See L149.
Route mixed operations via plyxp: when an expression touches multiple slots, prefer the plyxp path for correctness and performance. See L155.

These design choices aim to preserve dimnames, avoid unnecessary tibble round-trips, and provide predictable performance across simple and mixed-slot scenarios.

< section id="example-of-code-optimisation" class="level3">

Example of code optimisation

This was the mutate() method before optimisation. The previous implementation relied on as_tibble() |> dplyr::mutate() |> update_SE_from_tibble(.data)

The function update_SE_from_tibble interprets the input tibble and converts it back to a SummarizedExperiment. Although this step provides great generality and flexibility, it is particularly expensive because it must infer whether columns are sample-wise or feature-wise.

< details class="code-fold"> < summary>Show pre-optimization source

mutate.SummarizedExperiment <- function(.data, ...) {
    # Legacy implementation of mutate() for SummarizedExperiment:
    # - Validates requested edits against special/view-only columns
    # - Performs mutate() via tibble round-trip, then reconstructs the SE
    # Check that we are not modifying a key column
    cols <- enquos(...) |> names()
    
    # Deprecation of special column names:
    # capture all quoted args to detect deprecated special-column usage
    .cols <- enquos(..., .ignore_empty="all") %>% 
        map(~ quo_name(.x)) %>% unlist()
    if (is_sample_feature_deprecated_used(.data, .cols)) {
        # Record deprecated usage into metadata for backward compatibility
        .data <- ping_old_special_column_into_metadata(.data)
    }
    
    # Identify view-only/special columns (sample/feature keys, etc.)
    # Use a small slice to reduce overhead while probing structure
    special_columns <- get_special_columns(
        # Decrease the size of the dataset
        .data[1:min(100, nrow(.data)), 1:min(20, ncol(.data))]
    ) |> c(get_needed_columns(.data))
    
    # Are any requested targets among special/view-only columns?
    tst <-
        intersect(
            cols,
            special_columns
        ) |> 
        length() |>
        gt(0)

    if (tst) {
        columns <-
            special_columns |>
                paste(collapse=", ")
        stop(
            "tidySummarizedExperiment says:",
            " you are trying to rename a column that is view only",
            columns,
            "(it is not present in the colData).",
            " If you want to mutate a view-only column,",
            " make a copy and mutate that one."
        )
    }

    # If Ranges column not in query, prefer faster tibble conversion
    # Skip expanding GRanges columns when not referenced
    skip_GRanges <-
        get_GRanges_colnames() %in% 
        cols |>
        not()
    
    # Round-trip: SE -> tibble -> dplyr::mutate -> SE
    .data |>
        as_tibble(skip_GRanges=skip_GRanges) |>
        dplyr::mutate(...) |>
        update_SE_from_tibble(.data)
}

The new implementation captures all easy cases, such as sample-only and feature-only metadata mutate(). If mutate() is a mixed operation that can be factored out to sample- and feature-wise operation it is handled by plyxp. Otherwise, the general solution is used.

Key components to compare: – The pre-optimization code always uses a tibble round-trip (as_tibble() |> dplyr::mutate() |> update_SE_from_tibble()). – The optimized code first analyzes scope (colData, rowData, assay, or mixed) and dispatches to specialized paths. – The fallback still exists (mutate_via_tibble) for complex cases, preserving generality.

< details class="code-fold"> < summary>Show post-optimization source

mutate.SummarizedExperiment <- function(.data, ...) {

       # Check if query is composed (multiple expressions)
    if (is_composed("mutate", ...)) return(decompose_tidy_operation("mutate", ...)(.data))

        # Check for scope and dispatch elegantly
        scope_report <- analyze_query_scope_mutate(.data, ...)
        scope <- scope_report$scope

        result <-
        if(scope == "coldata_only") modify_samples(.data, "mutate", ...)
        else if(scope == "rowdata_only") modify_features(.data, "mutate", ...)
        else if(scope == "assay_only") mutate_assay(.data, ...)
        else if(scope == "mixed") modify_se_plyxp(.data, "mutate", scope_report, ...)
        else mutate_via_tibble(.data, ...)

        # Record latest mutate scope into metadata for testing/introspection
        <- S4Vectors::metadata(result)
        if (is.null(meta)) <- list()
        meta$latest_mutate_scope_report <- scope_report
        S4Vectors::metadata(result) <- meta

        return(result)

}

< section id="benchmarking-overview" class="level1">

Benchmarking Overview

This vignette benchmarks a set of mutate(), filter(), select(), and distinct() scenarios (see documentation) comparing performance before and after optimisation, by explicitly checking out specific commits via git worktree, loading each commit’s code with devtools::load_all(), running the same scenarios multiple times, and comparing the runtimes with ggplot boxplots.

Before optimisation: commit 87445757d2d0332e7d335d22cd28f73568b7db66
After optimisation: commit 9f7c26e0519c92f9682b270d566da127367bcbc0

< section id="setup-helper-functions" class="level3">