Seeing nucleotides as images

[This article was first published on Data Science Guts, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

In 2022 I spent a few months on a problem far from anything I had done before: finding single letters in the tails of RNA molecules, from the electric current of a nanopore sequencer. The network I built then is the classifier inside Ninetails, an R package from Natalia Gumińska and her colleagues at the IIMCB in Warsaw. The paper describing it came out in 2025 (Gumińska et al., Nature Communications).

I am not a geneticist, and this post does not pretend otherwise. What hooked me was the computer vision: the signal is a time series, the trick that made it work was to turn it into images, and that trick is worth knowing well beyond biology.

This is the first post of a short series. Here: just enough biology to follow along, what the images are, and the whole pipeline again – from public raw data to a trained network – to check that the 2022 result holds up.

The biology, briefly

A gene’s instructions travel from DNA to the protein-making machinery of a cell as messenger RNA (mRNA): a chain of four kinds of nucleotides, written A, C, G and U. Most mRNAs end in a poly(A) tail – tens to a couple of hundred As in a row. The tail is not part of the message; it is closer to a fuse. It protects the molecule and influences how long it lives and how much protein is made from it.

The tail is not always pure A. It can also contain nucleotides other than adenosine – for example C, G or U, slipped in by the enzymes that build and rebuild it – and these non-adenosines are linked to how long an mRNA survives. That makes them interesting far beyond basic research: the Ninetails paper follows, among others, the tails of the Moderna mRNA-1273 vaccine after it enters immune cells.

To count them, one needs to read the tail itself, letter by letter. That is where the nanopore comes in.

How a nanopore reads RNA

In direct RNA sequencing (Oxford Nanopore), a single RNA molecule is pulled through a protein pore in a membrane, tail first. A voltage drives ions through the same pore, and the nucleotides sitting in it – about five at a time in the pore used for this data; Oxford Nanopore says about nine for its newer pores, without publishing the details – partly block that ionic current. The instrument records the current a few thousand times per second. A basecaller, itself a neural network, translates the current back into letters.

Inside a long run of As, the current is nearly flat: the pore holds As, then more As. A single C, G or U disturbs it for the few moments it passes through. Here is what that looks like in four molecules from the training data, one per construct: a pure-A tail, and tails with one C, G or U.

Plot code
reads <- read.delim("reads.tsv")
signals <- read.csv("signals.csv")
hz <- 3012                                     # samples per second
signals$t <- signals$sample / hz
reads$t0 <- reads$polya_start / hz
reads$t1 <- reads$transcript_start / hz
reads$tx <- reads$x_pos / hz

ggplot(signals, aes(t, pA)) +
  geom_rect(aes(xmin = t0, xmax = t1, ymin = -Inf, ymax = Inf,
                fill = class), data = reads, inherit.aes = FALSE,
            alpha = 0.15) +
  geom_line(linewidth = 0.25, colour = "grey25") +
  geom_point(aes(tx, 140, fill = class), data = reads[!is.na(reads$tx), ],
             shape = 25, size = 2.6, colour = "white", inherit.aes = FALSE) +
  facet_wrap(~ class, ncol = 1, scales = "free_x",
             labeller = as_labeller(c(A = "pure A", C = "one C",
                                      G = "one G", U = "one U"))) +
  scale_fill_manual(values = cls_col, guide = "none") +
  coord_cartesian(ylim = c(40, 145)) +
  labs(x = "time (s)", y = "current (pA)")
Figure 1: Raw current of four molecules from the training data. The shaded band is the poly(A) tail; it comes first because the molecule enters the pore tail first. The triangle marks where the basecaller placed the C, G or U.

In each panel the shaded part is the tail, and it is the calmest stretch of the read. The non-A is visible as a short spike or dip in the middle of it - in this case at the same place in every construct, because the molecules were made to order (more on that below).

The basecaller is of little help here. Long runs of a single letter are its weak spot, and it was never trained to look for one odd letter inside them. Ninetails looks at the current instead.

From a current to fragments

Ninetails does not classify a whole tail. It simplifies the tail’s signal (outliers clipped, then every five samples merged into one point), looks for places where the current jumps away from its running mean, and cuts a 100-point fragment centred on each such place. The network only ever sees these fragments.

Plot code
tails <- read.csv("tails.csv")
frags <- read.csv("fragments.csv")
win <- unique(frags[c("class", "start")])
win$end <- win$start + 99

ggplot(tails, aes(i, value)) +
  geom_rect(aes(xmin = start, xmax = end, ymin = -Inf, ymax = Inf,
                fill = class), data = win, inherit.aes = FALSE,
            alpha = 0.2) +
  geom_line(linewidth = 0.3, colour = "grey25") +
  geom_point(data = tails[tails$pseudomove != 0, ], size = 0.9,
             colour = "grey5") +
  facet_wrap(~ class, ncol = 2, scales = "free") +
  scale_fill_manual(values = cls_col, guide = "none") +
  labs(x = "point in the simplified tail", y = "signal (raw units)")
Figure 2: The four tails as Ninetails sees them. Black dots are points where the signal leaves its running mean by more than 3.5 standard deviations; a run of at least five of them gives a fragment (shaded, 100 points). The pure-A tail has no such run; its fragment here is simply the middle of the tail, as the training set uses fragments like it.

A 100-point fragment is a tiny time series: a flat line with, in three of the four classes, a bump somewhere near the middle. A 1D model could read that directly. In 2022 I went another way.

A time series as an image

The Gramian Angular Field (GAF; Wang and Oates, 2015) is a recipe for turning a series of values into an image, so that a 2D convolutional network can look at it. “GAF” is the name of the recipe; it comes in two flavours, GASF and GADF, which share the first two steps and differ only in the last one:

  1. Rescale the series to ; call the result .
  2. Read each value as the cosine of an angle: . Together with time as the radius, this places the series on a half circle (Figure 3).
  3. Fill an image: pixel combines the angle of moment with the angle of moment .
    • GASF (Summation field):
    • GADF (Difference field):

So from one fragment you get two images, built from the same angles. Every pixel compares two moments of the series. In a flat stretch all moments look alike, and so do their pixels; the one odd moment differs from all the others, so its whole row and column stand out - a cross. Ninetails computes both images and gives them to the network together.

Plot code
rescale <- function(x) (2 * x - max(x) - min(x)) / (max(x) - min(x))
g <- frags[frags$class == "G", ]
phi <- acos(rescale(g$value))                  # angle
polar <- data.frame(x = g$i * cos(phi), y = g$i * sin(phi))
rings <- expand.grid(r = c(25, 50, 75, 100),
                     a = seq(0, pi, length.out = 181))

ggplot(polar, aes(x, y)) +
  geom_path(aes(r * cos(a), r * sin(a), group = r), data = rings,
            colour = "grey85") +
  annotate("text", x = c(25, 50, 75, 100), y = -5, size = 3.2,
           label = c(25, 50, 75, 100), colour = "grey40") +
  annotate("text", x = c(110, 0, -110), y = c(0, 108, 0), size = 4,
           label = c("0", "π/2", "π"), colour = "grey30") +
  geom_path(colour = cls_col[["G"]], linewidth = 0.3, alpha = 0.35) +
  geom_point(colour = cls_col[["G"]], size = 1.4) +
  coord_equal() +
  theme_void()
Figure 3: The G fragment in polar coordinates: the angle comes from the rescaled value, the distance from the centre from the position (grey rings: points 25 to 100). The A stretches stay in one narrow band of angles; around point 50 the G throws the series out to the edges of the half circle and back.

In code, both are a few lines. Here they are as they ship in Ninetails (ninetails::create_gaf), written out. The second function is not quite the textbook GADF - more on that below, once it matters:

rescale <- function(x) (2 * x - max(x) - min(x)) / (max(x) - min(x))
minmax <- function(m) round((m - min(m)) / (max(m) - min(m)), 4)

gasf <- function(x) {
  phi <- acos(rescale(x))
  minmax(cos(outer(phi, phi, "+")))
}
gadf <- function(x) {
  psi <- asin(rescale(x))
  minmax(sin(outer(psi, psi, "+")))
}

Each 100-point fragment becomes a pair of 100 x 100 images - the GASF and the GADF - stacked as two channels, the same shape as a small two-colour photograph:

Plot code
frags <- read.csv("fragments.csv")
as_long <- function(m, cls, field) {
  data.frame(i = rep(seq_len(nrow(m)), ncol(m)),
             j = rep(seq_len(ncol(m)), each = nrow(m)),
             value = as.vector(m), class = cls, field = field)
}
gafs <- do.call(rbind, lapply(c("A", "C", "G", "U"), function(k) {
  x <- frags$value[frags$class == k]
  rbind(as_long(gasf(x), k, "GASF"), as_long(gadf(x), k, "GADF"))
}))
gafs$field <- factor(gafs$field, c("GASF", "GADF"))

ggplot(gafs, aes(j, i, fill = value)) +
  geom_raster() +
  facet_grid(field ~ class) +
  seq_fill +
  scale_y_reverse() +
  coord_equal() +
  theme_void(base_size = 13) +
  theme(strip.text = element_text(margin = margin(4, 4, 4, 4)))
Figure 4: The four fragments as GASF (top) and GADF (bottom) images. A pure-A fragment is noise; a non-A draws a cross through the image at its position.

The non-A is no longer a blip in a line: it is a cross through the image, at the row and column of its position, and its texture differs between C, G and U.

Why two images

A G pushes the current up; a C or a U pulls it down (Figure 1). Their fragments are close to mirror images of each other - and GASF cannot tell a series from its mirror. Flipping the sign of turns every into , and : the image does not change. The second channel is there to break that symmetry.

Plot code
x <- frags$value[frags$class == "G"]
mirror <- rbind(
  as_long(gasf(x), "G fragment", "GASF"),
  as_long(gasf(-x), "the same, mirrored", "GASF"),
  as_long(gadf(x), "G fragment", "GADF"),
  as_long(gadf(-x), "the same, mirrored", "GADF"))
mirror$field <- factor(mirror$field, c("GASF", "GADF"))

ggplot(mirror, aes(j, i, fill = value)) +
  geom_raster() +
  facet_grid(field ~ class) +
  seq_fill +
  scale_y_reverse() +
  coord_equal() +
  theme_void(base_size = 13) +
  theme(strip.text = element_text(margin = margin(4, 4, 4, 4)))
Figure 5: The G fragment and its mirror image. GASF (top) is identical; the second channel (bottom) is inverted.
x <- frags$value[frags$class == "G"]
identical(gasf(x), gasf(-x))
#> [1] TRUE
all.equal(gadf(-x), 1 - gadf(x))
#> [1] TRUE

Now the promised detail about Ninetails’ second channel. The textbook GADF is ; Ninetails computes with instead. It is a slightly different image: symmetric, where the textbook one is antisymmetric, and with a non-zero diagonal. But it does the one job the second channel is there for - a mirrored fragment gives the inverted image, so G and C/U look different, as the check above shows.

Other ways to draw a series

GAFs are not the only option. Two others are common, and each has its own recipe.

The Markov Transition Field (MTF; from the same paper as GAF) does not look at the values themselves, but at how the series moves between levels:

  1. Split the range of values into bins by quantiles (here ), so that every moment gets a state from 1 to 8.
  2. Count how often the series goes from state to state between one moment and the next, and turn the counts into probabilities. This is a Markov transition matrix - the parent of the image. It is a compact summary, but it has forgotten when anything happened.
  3. Spread back over time: pixel is , the probability of moving from the state of moment to the state of moment .

The recurrence plot (Eckmann et al., 1987) is older and simpler still:

  1. Rescale the series to .
  2. Pixel is 1 if moments and have nearly the same value (, here ), and 0 otherwise.
states <- function(x, q = 8) {
  cut(x, unique(quantile(x, 0:q / q)), include.lowest = TRUE,
      labels = FALSE)
}
transitions <- function(s) {           # Q x Q, rows sum to 1
  k <- max(s)
  w <- table(factor(head(s, -1), 1:k), factor(tail(s, -1), 1:k))
  unclass(w / pmax(rowSums(w), 1))
}
mtf <- function(x, q = 8) {
  s <- states(x, q)
  minmax(transitions(s)[s, s])         # pixel (i, j) = W[s_i, s_j]
}
recurrence <- function(x, eps = 0.1) {
  z <- (x - min(x)) / (max(x) - min(x))
  (abs(outer(z, z, "-")) < eps) * 1
}

Here are the first two MTF steps for the C fragment:

Plot code
x <- frags$value[frags$class == "C"]
s <- states(x)
breaks <- unique(quantile(x, 0:8 / 8))
ggplot(data.frame(i = seq_along(x), x = x, state = s), aes(i, x)) +
  geom_hline(yintercept = breaks, colour = "grey85") +
  geom_line(linewidth = 0.3, colour = "grey60") +
  geom_point(aes(colour = state), size = 1.6) +
  scale_colour_gradient(low = "#9ec5f4", high = "#123f78",
                        breaks = 1:8) +
  labs(x = "point in the fragment", y = "signal") +
  theme(panel.grid = element_blank())

w <- transitions(s)
wd <- data.frame(from = factor(rep(1:8, 8)), to = factor(rep(1:8, each = 8)),
                 p = as.vector(w))
wd$from <- factor(wd$from, 8:1)
ggplot(wd, aes(to, from, fill = p)) +
  geom_tile(colour = "white", linewidth = 0.8) +
  geom_text(aes(label = ifelse(p > 0, sprintf("%.2f", p), ""),
                colour = p > 0.5), size = 3) +
  scale_colour_manual(values = c("grey20", "white"), guide = "none") +
  seq_fill +
  coord_equal() +
  labs(x = "to state", y = "from state") +
  theme(legend.position = "none", panel.grid = element_blank())
(a) Step 1: every point gets a state, from its quantile bin (grey lines: bin edges). The flat A stretches spread over the upper states; the C dip is state 1.
(b) Step 2: the transition matrix. Noise makes the upper states jump between each other; once the series is in the dip, it tends to stay there.
Figure 6: The first two MTF steps for the C fragment.

Step 3 turns this 8 x 8 matrix back into a 100 x 100 image. Here is the C fragment in all four forms:

Plot code
x <- frags$value[frags$class == "C"]
forms <- rbind(as_long(gasf(x), "C", "GASF"),
               as_long(gadf(x), "C", "GADF (Ninetails)"),
               as_long(mtf(x), "C", "Markov transition field"),
               as_long(recurrence(x), "C", "recurrence plot"))
forms$field <- factor(forms$field, unique(forms$field))

ggplot(forms, aes(j, i, fill = value)) +
  geom_raster() +
  facet_wrap(~ field, nrow = 1) +
  seq_fill +
  scale_y_reverse() +
  coord_equal() +
  theme_void(base_size = 12) +
  theme(legend.position = "none",
        strip.text = element_text(margin = margin(4, 4, 4, 4)))
Figure 7: One C fragment, four images. All of them mark the position of the C; they differ in what else they encode.

All four light up where the C is. Which of them works best for which signal is a question of its own, and one I want to measure properly some day. Ninetails uses GASF + GADF.

The whole pipeline, from public data

The molecules in Figure 1 are not from cells. They were made in the lab to a known design: a piece of a luciferase gene followed by a tail of about 60 nucleotides that is either all A or has a single C, G or U in the middle. With a known answer for every molecule, they are the training set. The raw data is public in the European Nucleotide Archive (run ERR12257477, 1.4 GB).

The heavy steps need a GPU and an afternoon, so they are not run in this post; the code is what I ran.

# raw current, FAST5 files in four folders (A, C, G, U constructs)
curl -O ftp.sra.ebi.ac.uk/vol1/run/ERR122/ERR12257477/syntheticRNA_training_CNN.tar.gz
tar xzf syntheticRNA_training_CNN.tar.gz

# FAST5 -> POD5, then basecall with move tables (which sample is which base)
pod5 convert fast5 spikes_training_cnn/C_Rluc_CNN/*.fast5 --output C.pod5
dorado basecaller rna002_70bps_hac@v3 C.pod5 --emit-moves > C.bam

Two practical notes:

  • The data was sequenced with the RNA002 chemistry, which current Dorado no longer supports. Version 0.9.6 is the last that does.
  • Ninetails expects the inputs of the older toolchain (Guppy-basecalled FAST5 files and tail positions from nanopolish). I found the tail in each read from Dorado’s move table and wrote those inputs myself.

From there, Ninetails’ own functions cut the fragments and build the images:

library(ninetails)
gafs_C <- prepare_trainingset(
  "C",
  nanopolish = "C/nanopolish.tsv",
  sequencing_summary = "C/sequencing_summary.txt",
  workspace = "C/fast5"
)

The counts per class are not the paper’s, and should not be expected to be. The paper’s training set was built with Guppy and nanopolish and sampled to 9,440 fragments per class; here a different basecaller and my own tail boundaries decide which reads pass and where each tail starts and ends:

Plot code
counts <- data.frame(class = c("A", "C", "G", "U"),
                     n = c(27773, 8376, 10969, 6670))
ggplot(counts, aes(class, n, fill = class)) +
  geom_col(width = 0.6) +
  geom_hline(yintercept = 9440, linetype = "dashed", colour = "grey35") +
  annotate("text", x = 1.5, y = 9440, label = "paper: 9,440 per class",
           vjust = -0.6, hjust = 0, colour = "grey25", size = 3.6) +
  scale_fill_manual(values = cls_col, guide = "none") +
  scale_y_continuous(labels = scales::comma) +
  labs(x = NULL, y = "fragments")
Figure 8: Fragments per class from the public run, with Dorado and my tail boundaries. The paper’s training set, built with Guppy and nanopolish, was sampled to 9,440 per class.

For training I balanced the classes to the smallest one.

The network

The model is small on purpose. A more elaborate network, tried first, overfit the modest amount of data; a VGG-style network of four convolutional blocks did better. Written out in R keras (the original was built in R):

library(keras)
block <- function(x, filters, kernel, padding) {
  x |>
    layer_conv_2d(filters, kernel, padding = padding) |>
    layer_batch_normalization() |>
    layer_activation_relu() |>
    layer_max_pooling_2d(2) |>
    layer_dropout(0.5)
}
input <- layer_input(c(100, 100, 2))
output <- input |>
  block(32, 5, "valid") |>
  block(32, 5, "valid") |>
  block(64, 3, "same") |>
  block(128, 3, "same") |>
  layer_flatten() |>
  layer_dense(4, activation = "softmax")
model <- keras_model(input, output)      # 133,444 parameters

I retrained it from scratch, with the architecture cloned from the model file that ships with Ninetails (in Python, tf.keras.models.clone_model): Adam with learning rate 0.001, batches of 64, early stopping on validation loss. The split is by read: all fragments of one molecule go to training, validation or test together, so the test set holds molecules the network has never seen.

Does it hold up?

Plot code
cm <- matrix(c(727,   0,   4,   0,
                 4, 697,   5,  25,
                 0,   0, 731,   0,
                 5,  17,   1, 708), 4, byrow = TRUE,
             dimnames = list(true = c("A", "C", "G", "U"),
                             predicted = c("A", "C", "G", "U")))
d <- as.data.frame(as.table(cm), responseName = "n")
d$share <- d$n / rowSums(cm)[d$true]
d$true <- factor(d$true, rev(levels(d$true)))

ggplot(d, aes(predicted, true, fill = share)) +
  geom_tile(colour = "white", linewidth = 1) +
  geom_text(aes(label = n, colour = share > 0.5), size = 4.2) +
  scale_colour_manual(values = c("grey20", "white"), guide = "none") +
  seq_fill +
  coord_equal() +
  labs(x = "predicted", y = "true") +
  theme(legend.position = "none", panel.grid = element_blank())
Figure 9: Test fragments from molecules the network never saw (731 per class). Almost every mistake is C taken for U or U for C.

Balanced accuracy on the test set is 0.979: A 0.995, C 0.954, G 1.000, U 0.969. Three repeats with a random split of fragments give 0.979-0.984, so splitting by molecule costs nothing here. The weak spot is the same as in the paper: C and U, whose signals are the most alike.

The paper reports more than 93% overall and above 85% for C and U. The two sets of numbers are not directly comparable - they come from different fragments, found with a different basecaller and different tail boundaries - so read mine as “the 2022 network still does its job with today’s tools”, not as an improvement on it.

The series so far

  1. Seeing nucleotides as images (this post) - the biology in brief, GASF and GADF, and the 2022 network rebuilt from public data.

The next parts stay with nanopore signals seen through a computer vision lens; I will link them here as they come out.

To leave a comment for the author, please follow the link and comment on their blog: Data Science Guts.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

Never miss an update!
Subscribe to R-bloggers to receive
e-mails with the latest R posts.
(You will not see this message again.)

Click here to close (This popup will not appear again)