Understanding the Tucker decomposition, and compressing tensorvalued data (with R code)
Want to share your content on Rbloggers? click here if you have a blog, or here if you don't.
In many applications, data naturally form an nway tensor with n > 2, rather than a “tidy” table.
As mentioned in the beginning of my last blog post, a tensor is essentially a multidimensional array:
 a tensor of order one is a vector, which simply is a column of numbers,
 a tensor of order two is a matrix, which is basically numbers arranged in a rectangle,
 a tensor of order three looks like numbers arranged in rectangular box (or a cube, if all modes have the same dimension),
 an nth order (or nway) tensor looks like numbers arranged in an nhyperrectangle… you get the idea…
In this post I introduce the Tucker decomposition (Tucker (1966) “Some mathematical notes on threemode factor analysis”). The Tucker decomposition family includes methods such as
 the higherorder SVD, or HOSVD, which is a generalization of the matrix SVD to tensors (De Lathauwer, De Moor, and Vanderwalle (2000) “A multilinear singular value decomposition”),
 the higher order orthogonal iteration, or HOOI, which delivers the best approximation to a given tensor by another tensor with prescribed mode1 rank, mode2 rank, etc. (De Lathauwer, De Moor, and Vanderwalle (2000) “On the Best Rank1 and Rank(R1,R2,…,RN) Approximation of HigherOrder Tensors”).
I introduce both approaches, and in order to demonstrate the usefulness of these concepts, I present a simple data compression example using The World Bank’s World Development Indicators dataset (though I use the version available on Kaggle).
However, before we can get started with the decompositions, we need to look at and understand the kmode tensor product.
Throughout this post, I will also introduce the R functions from the package rTensor
, which can be used to perform all of the presented computations.
Tensor times matrix: the kmode product
The mode product of a tensor with a matrix is written as
The resulting tensor is of size , and contains the elements
It can be hard, at first, to understand what that definition really means, or to visualize it in your mind. I find that it becomes easier once you realize that the kmode product amounts to multiplying each modek fiber of by the matrix .
We can demonstrate that in R:
<span class="n">library</span><span class="p">(</span><span class="n">rTensor</span><span class="p">)</span><span class="w">
</span><span class="n">tnsr</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="n">as.tensor</span><span class="p">(</span><span class="n">array</span><span class="p">(</span><span class="m">1</span><span class="o">:</span><span class="m">12</span><span class="p">,</span><span class="w"> </span><span class="n">dim</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nf">c</span><span class="p">(</span><span class="m">2</span><span class="p">,</span><span class="w"> </span><span class="m">2</span><span class="p">,</span><span class="w"> </span><span class="m">3</span><span class="p">)))</span><span class="w">
</span><span class="n">mat</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="n">matrix</span><span class="p">(</span><span class="m">1</span><span class="o">:</span><span class="m">6</span><span class="p">,</span><span class="w"> </span><span class="m">3</span><span class="p">,</span><span class="w"> </span><span class="m">2</span><span class="p">)</span><span class="w">
</span><span class="c1"># 1mode product performed via the function ttm in rTensor</span><span class="w">
</span><span class="n">tnsr_times_mat</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="n">ttm</span><span class="p">(</span><span class="n">tnsr</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">tnsr</span><span class="p">,</span><span class="w"> </span><span class="n">mat</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">mat</span><span class="p">,</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">1</span><span class="p">)</span><span class="w">
</span>
Now, for example, the first slice of tnsr_times_mat
is the same as the matrix product of mat
with the first slice of tnsr
:
<span class="n">tnsr_times_mat</span><span class="o">@</span><span class="n">data</span><span class="p">[</span><span class="w"> </span><span class="p">,</span><span class="w"> </span><span class="p">,</span><span class="w"> </span><span class="m">1</span><span class="p">]</span><span class="w">
</span><span class="c1"># [,1] [,2]</span><span class="w">
</span><span class="c1"># [1,] 9 19</span><span class="w">
</span><span class="c1"># [2,] 12 26</span><span class="w">
</span><span class="c1"># [3,] 15 33</span><span class="w">
</span><span class="n">mat</span><span class="w"> </span><span class="o">%*%</span><span class="w"> </span><span class="n">as.matrix</span><span class="p">(</span><span class="n">tnsr</span><span class="o">@</span><span class="n">data</span><span class="p">[</span><span class="w"> </span><span class="p">,</span><span class="w"> </span><span class="p">,</span><span class="w"> </span><span class="m">1</span><span class="p">])</span><span class="w">
</span><span class="c1"># [,1] [,2]</span><span class="w">
</span><span class="c1"># [1,] 9 19</span><span class="w">
</span><span class="c1"># [2,] 12 26</span><span class="w">
</span><span class="c1"># [3,] 15 33</span><span class="w">
</span>
You might want to play around some more with the function ttm
in R to get a better understanding of the kmode product.
A few important facts about the kmode product:
 if ,
 but (in general ).
Tucker decomposition
The Tucker decomposition (Tucker (1966)) decomposes a tensor into a core tensor multiplied by a matrix along each mode (i.e., transformed via a mode product for every ):
Note that might be much smaller than the original tensor if we accept an approximation instead of an exact equality.
In case of threeway tensors, we can hold on to the following mental image:
It is interesting to note that the CP decomposition, that I introduced in a previous blog post, is a special case of the Tucker decomposition, where the core tensor is constrained to be superdiagonal.
Higherorder SVD (HOSVD)
So, how do you compute the Tucker decomposition?
Many algorithms rely on the following fundamental equivalence:
The above equation uses some notation that was not introduced yet:
 denotes the Kronecker product.

is the mode unfolding (or mode matricization) of the tensor . The mode unfolding arranges the mode fibers (a fiber is a generalization of column to tensors) of as columns into a matrix. The concept may be easiest to understand by looking at an example. The following R code shows a 3way tensor and all three of its mode unfoldings (using the
k_unfold
function from therTensor
package):<span class="n">tnsr</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="n">as.tensor</span><span class="p">(</span><span class="n">array</span><span class="p">(</span><span class="m">1</span><span class="o">:</span><span class="m">12</span><span class="p">,</span><span class="w"> </span><span class="n">dim</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nf">c</span><span class="p">(</span><span class="m">2</span><span class="p">,</span><span class="w"> </span><span class="m">3</span><span class="p">,</span><span class="w"> </span><span class="m">2</span><span class="p">)))</span><span class="w"> </span><span class="n">tnsr</span><span class="o">@</span><span class="n">data</span><span class="w"> </span><span class="c1"># , , 1</span><span class="w"> </span><span class="c1">#</span><span class="w"> </span><span class="c1"># [,1] [,2] [,3]</span><span class="w"> </span><span class="c1"># [1,] 1 3 5</span><span class="w"> </span><span class="c1"># [2,] 2 4 6</span><span class="w"> </span><span class="c1">#</span><span class="w"> </span><span class="c1"># , , 2</span><span class="w"> </span><span class="c1">#</span><span class="w"> </span><span class="c1"># [,1] [,2] [,3]</span><span class="w"> </span><span class="c1"># [1,] 7 9 11</span><span class="w"> </span><span class="c1"># [2,] 8 10 12</span><span class="w"> </span><span class="c1"># mode1 unfolding:</span><span class="w"> </span><span class="n">k_unfold</span><span class="p">(</span><span class="n">tnsr</span><span class="p">,</span><span class="w"> </span><span class="m">1</span><span class="p">)</span><span class="o">@</span><span class="n">data</span><span class="w"> </span><span class="c1"># [,1] [,2] [,3] [,4] [,5] [,6]</span><span class="w"> </span><span class="c1"># [1,] 1 3 5 7 9 11</span><span class="w"> </span><span class="c1"># [2,] 2 4 6 8 10 12</span><span class="w"> </span><span class="c1"># mode2 unfolding:</span><span class="w"> </span><span class="n">k_unfold</span><span class="p">(</span><span class="n">tnsr</span><span class="p">,</span><span class="w"> </span><span class="m">2</span><span class="p">)</span><span class="o">@</span><span class="n">data</span><span class="w"> </span><span class="c1"># [,1] [,2] [,3] [,4]</span><span class="w"> </span><span class="c1"># [1,] 1 2 7 8</span><span class="w"> </span><span class="c1"># [2,] 3 4 9 10</span><span class="w"> </span><span class="c1"># [3,] 5 6 11 12</span><span class="w"> </span><span class="c1"># mode3 unfolding:</span><span class="w"> </span><span class="n">k_unfold</span><span class="p">(</span><span class="n">tnsr</span><span class="p">,</span><span class="w"> </span><span class="m">3</span><span class="p">)</span><span class="o">@</span><span class="n">data</span><span class="w"> </span><span class="c1"># [,1] [,2] [,3] [,4] [,5] [,6]</span><span class="w"> </span><span class="c1"># [1,] 1 2 3 4 5 6</span><span class="w"> </span><span class="c1"># [2,] 7 8 9 10 11 12</span><span class="w"> </span>
A straightforward approach to solve the Tucker decomposition would be to solve each mode matricized form of the Tucker decomposition (shown in the equivalence above) for . This approach is known as higher order SVD, or HOSVD. It can be regarded as a generalization of the matrix SVD, because the matrices are orthogonal, while the tensor is “ordered” and “allorthogonal” (see De Lathauwer et. al. (2000) for detail). The resulting algorithm is shown below.
In R we can perform HOSVD using the function hosvd
from rTensor
:
<span class="n">tnsr</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="n">rand_tensor</span><span class="p">(</span><span class="n">modes</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nf">c</span><span class="p">(</span><span class="m">30</span><span class="p">,</span><span class="w"> </span><span class="m">40</span><span class="p">,</span><span class="w"> </span><span class="m">50</span><span class="p">))</span><span class="w">
</span><span class="n">hosv_decomp</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="n">hosvd</span><span class="p">(</span><span class="n">tnsr</span><span class="p">)</span><span class="w">
</span>
Now hosv_decompGU
is a list containing all the matrices . We can use the function ttl
, which performs multiple kmode products on multiple modes successively given a tensor and a list of matrices, to check that up to numerical error the equation
is satisfied:
<span class="n">HOSVD_prod</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="n">ttl</span><span class="p">(</span><span class="n">hosv_decomp</span><span class="o"><img src="http://latex.codecogs.com/png.latex?%3C/span%3E%3Cspan%20class="n">Z</span><span class="p">,</span><span class="w"> </span><span class="n">hosv_decomp</span><span class="o">\inline"/></span><span class="n">U</span><span class="p">,</span><span class="w"> </span><span class="m">1</span><span class="o">:</span><span class="m">3</span><span class="p">)</span><span class="w">
</span><span class="n">error</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="n">tnsr</span><span class="w"> </span><span class="o"></span><span class="w"> </span><span class="n">HOSVD_prod</span><span class="w">
</span><span class="n">table</span><span class="p">(</span><span class="nf">abs</span><span class="p">(</span><span class="n">error</span><span class="o">@</span><span class="n">data</span><span class="p">)</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="m">1e12</span><span class="p">)</span><span class="w">
</span><span class="c1">#</span><span class="w">
</span><span class="c1"># TRUE</span><span class="w">
</span><span class="c1"># 60000</span><span class="w">
</span>
Higher order orthogonal iteration (HOOI)
Note that we can also use HOSVD to compress by truncating the matrices . The truncated HOSVD, however, is known to not give the best fit, as measured by the norm of the difference
The higher order orthogonal iteration, or HOOI, algorithm finds the optimal approximation (with respect to the Frobenius norm loss) by, essentially, iterating the alternating truncation and SVD until convergence. If we truncate to have columns, then the HOOI solution can be obtained by the following algorithm.
Application of HOOI to data compression
The example considered below is somewhat silly, given that the tensor I’m compressing isn’t very big, and thus there isn’t much of a point in compressing it. However, I think that the example still shows off very well how the algorithm can be very useful when the data size is much bigger (or the available storage much smaller).
I have downloaded from Kaggle the World Development Indicators dataset, originally collected and published by The World Bank (the original dataset is available here).
The data can be arranged into a threeway tensor with the three modes corresponding to country (list of available countries), indicator (list of available indicators), and year (19602014). Since I didn’t have any time to deal with NA values in any creative way, I have kept only three indicators in the dataset. And I have replaced the remaining NAs with a countrywise average value for each particular indicator. Also, I have forgotten to normalize the data :disappointed:. The preprocessing resulted in a tensor of size 247countriesby3indicatorsby55years, that looks sort of like this:
In particular, large stretches of the data within a given country tend to be nearly constant, or nearly piecewise constant.
We use the function tucker
from rTensor
to obtain a Tucker decomposition via HOOI, where we set the ranks to the value 3 at each mode.
<span class="nf">dim</span><span class="p">(</span><span class="n">wdi_tnsr</span><span class="p">)</span><span class="w">
</span><span class="c1"># [1] 247 3 55</span><span class="w">
</span><span class="n">tucker_decomp</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="n">tucker</span><span class="p">(</span><span class="n">wdi_tnsr</span><span class="p">,</span><span class="w"> </span><span class="n">ranks</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nf">c</span><span class="p">(</span><span class="m">3</span><span class="p">,</span><span class="w"> </span><span class="m">3</span><span class="p">,</span><span class="w"> </span><span class="m">3</span><span class="p">))</span><span class="w">
</span><span class="n">str</span><span class="p">(</span><span class="n">tucker_decomp</span><span class="p">)</span><span class="w">
</span><span class="c1"># List of 7</span><span class="w">
</span><span class="c1"># $ Z :Formal class 'Tensor' [package "rTensor"] with 3 slots</span><span class="w">
</span><span class="c1"># .. [email protected] num_modes: int 3</span><span class="w">
</span><span class="c1"># .. [email protected] modes : int [1:3] 3 3 3</span><span class="w">
</span><span class="c1"># .. [email protected] data : num [1:3, 1:3, 1:3] 6.60e+10 1.13e+05 6.24e+05 7.76e+05 1.93e+08 ...</span><span class="w">
</span><span class="c1"># $ U :List of 3</span><span class="w">
</span><span class="c1"># ..$ : num [1:247, 1:3] 0.02577 0.00065 0.01146 0.19637 0.17317 ...</span><span class="w">
</span><span class="c1"># ..$ : num [1:3, 1:3] 1.00 6.97e10 2.08e02 2.08e02 4.70e08 ...</span><span class="w">
</span><span class="c1"># ..$ : num [1:55, 1:3] 0.0762 0.0772 0.0785 0.0802 0.082 ...</span><span class="w">
</span><span class="c1"># $ conv : logi TRUE</span><span class="w">
</span><span class="c1"># $ est :Formal class 'Tensor' [package "rTensor"] with 3 slots</span><span class="w">
</span><span class="c1"># .. [email protected] num_modes: int 3</span><span class="w">
</span><span class="c1"># .. [email protected] modes : int [1:3] 247 3 55</span><span class="w">
</span><span class="c1"># .. [email protected] data : num [1:247, 1:3, 1:55] 9.83e+07 4.44e+06 8.81e+07 1.05e+09 8.97e+08 ...</span><span class="w">
</span><span class="c1"># $ norm_percent: num 99.4</span><span class="w">
</span><span class="c1"># $ fnorm_resid : num 3.9e+08</span><span class="w">
</span><span class="c1"># $ all_resids : num [1:2] 3.9e+08 3.9e+08</span><span class="w">
</span><span class="c1"># NULL</span><span class="w">
</span>
To see how well the tensor decomposition approximates the original tensor, we can look at the relative error
<span class="n">wdi_tnsr_approx</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="n">ttl</span><span class="p">(</span><span class="n">tucker_decomp</span><span class="o"><img src="http://latex.codecogs.com/png.latex?%3C/span%3E%3Cspan%20class="n">Z</span><span class="p">,</span><span class="w"> </span><span class="n">tucker_decomp</span><span class="o">\inline"/></span><span class="n">U</span><span class="p">,</span><span class="w"> </span><span class="m">1</span><span class="o">:</span><span class="m">3</span><span class="p">)</span><span class="w">
</span><span class="n">fnorm</span><span class="p">(</span><span class="n">wdi_tnsr</span><span class="w"> </span><span class="o"></span><span class="w"> </span><span class="n">wdi_tnsr_approx</span><span class="p">)</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="n">fnorm</span><span class="p">(</span><span class="n">wdi_tnsr</span><span class="p">)</span><span class="w">
</span><span class="c1"># [1] 0.005908934</span><span class="w">
</span>
and at the percentage of the norm of the original tensor explained by the Tucker decomposition
<span class="n">tucker_decomp</span><span class="o">$</span><span class="n">norm_percent</span><span class="w">
</span><span class="c1"># [1] 99.40911</span><span class="w">
</span>
We, observe that we indeed achieve a recovery with an accuracy of over 99%. For comparison, the original tensor contains 247 * 3 * 55 = 40755
entries, while the computed Tucker decomposition consists of only 127 * 3 + 3 * 3 + 55 * 3 + 3 * 3 * 3 = 582
numbers. That’s a reduction in size by a factor greater than 70.
Even though data compression does not make much sense for the size of the dataset considered here, it clearly shows potential to be very useful for purposes of data distribution and data storage, when the data size far exceeds the terabyte range.
Rbloggers.com offers daily email updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/datascience job.
Want to share your content on Rbloggers? click here if you have a blog, or here if you don't.