Site icon R-bloggers

Notes on Linear Algebra Part 1

[This article was first published on Chemometrics and Spectroscopy Using R, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

If you are already familiar with much of linear algebra, as well as the relevant functions in R, read no further and do something else!

If you are like me, you’ve had no formal training in linear algebra, which means you learn what you need to when you need to use it. Eventually, you cobble together some hard-won knowledge. That’s good, because almost everything in chemometrics involves linear algebra.

This post is essentially a set of personal notes about the dot product and the cross product, two important manipulations in linear algebra. I’ve tried to harmonize things I learned way back in college physics and math courses, and integrate information I’ve found in various sources I have leaned on more recently. Without a doubt, the greatest impediment to really understanding this material is the use of multiple terminology and notations. I’m going to try really hard to be clear and to the point in my dicussion.

The main sources I’ve relied on are:

Let’s get started. For sanity and consistency, let’s define two 3D vectors and two matrices to illustrate our examples. Most of the time I’m going to write vectors with an arrow over the name, as a nod to the treatment usually given in a physics course. This reminds us that we are thinking about a quantity with direction and magnitude in some coordinate system, something geometric. Of course in the R language a vector is simply a list of numbers with the same data type; R doesn’t care if a vector is a vector in the geometric sense or a list of states.

< section id="dot-product" class="level2">

Dot Product

< section id="terminology" class="level3">

Terminology

The dot product goes by these other names: inner product, scalar product. Typical notations include:1

< section id="formulas" class="level3">

Formulas

There are two main formulas for the dot product with vectors, the algebraic formula (Equation 5) and the geometric formula (Equation 6).

refers to the or Euclidian norm, namely the length of the vector:2

The result of the dot product is a scalar. The dot product is also commutative: .

Watch out when using row or column vectors

From the perspective of matrices, if we think of and as column vectors with dimensions 3 x 1, then transposing gives us conformable matrices and we find the result of matrix multiplication is the dot product (compare to Equation 5):

Even though this is matrix multiplication, the answer is still a scalar.

Now, rather confusingly, if we think of and as row vectors, and we transpose ,then we get the dot product:

Equations Equation 8 and Equation 9 can be a source of real confusion at first. They give the impression that the dot product can be either or . However, this is only true in the limited contexts defined above. To summarize:

  • Thinking of the vectors as column vectors with dimensions then one can use
  • Thinking of the vectors as row vectors with dimensions then one can use

Unfortunately I think this distinction is not always clearly made by authors, and is a source of great confusion to linear algebra learners. Be careful when working with row and column vectors.

< section id="matrix-multiplication" class="level3">

Matrix Multiplication

Suppose we wanted to compute .3 We use the idea of row and column vectors to accomplish this task. In the process, we discover that matrix multiplication is a series of dot products:

The red color shows how the dot product of the first row of and the first column of gives the first entry in . Every entry in results from a dot product. Every entry is a scalar, embedded in a matrix.

< section id="what-can-we-do-with-the-dot-product" class="level3">

What Can We Do With the Dot Product?

< section id="cross-product" class="level2">

Cross Product

< section id="terminology-and-notation" class="level3">

Terminology and Notation

The cross product goes by these other names: outer product4, tensor product, vector product.

< section id="formulas-1" class="level3">

Formulas

The cross product of two vectors returns a vector rather than a scalar. Vectors are defined in terms of a basis which is a coordinate system. Earlier, when we defined it was intrinsically defined in terms of the standard basis set (in some fields this would be called the unit coordinate system). Thus a fuller definition of would be:

In terms of vectors, the cross product is defined as:

In my opinion, this is not exactly intuitive, but there is a pattern to it: notice that the terms for don’t involve the component. The details of how this result is computed relies on some properties of the basis set; this Wikipedia article has a nice explanation. We need not dwell on it however.

There is also a geometric formula for the cross product:

where is the unit vector perpendicular to the plane defined by and . The direction of is defined by the right-hand rule. Because of this, the cross product is not commutative, i.e. . The cross product is however anti-commutative:

Cross product using column vectors

As we did for the dot product, we can look at the cross product from the perspective of column vectors. Instead of transposing the first matrix as we did for the dot product, we transpose the second one:

Interestingly, we are using the dot product to compute the cross product.

The case where we treat and as row vectors is left to the reader.5

Finally, there is a matrix definition of the cross product as well. Evaluation of the following determinant gives the cross product:

< section id="what-can-we-do-with-the-cross-product" class="level3">

What Can We Do With the Cross Product?

< section id="r-functions" class="level2">

R Functions

< section id="section" class="level3">

%*%

The workhorse for matrix multiplication in R is the %*% function. This function will accept any combination of vectors and matrices as inputs, so it is flexible. It is also smart: given a vector and a matrix, the vector will be treated as row or column matrix as needed to ensure conformity, if possible. Let’s look at some examples:

# Some data for examples
p <- 1:5
q <- 6:10
M <- matrix(1:15, nrow = 3, ncol = 5)
M
     [,1] [,2] [,3] [,4] [,5]
[1,]    1    4    7   10   13
[2,]    2    5    8   11   14
[3,]    3    6    9   12   15
# A vector times a vector
p %*% q
     [,1]
[1,]  130

Notice that R returns a data type of matrix, but it is a matrix, and thus a scalar value. That means we just computed the dot product, a descision R made internally. We can verify this by noting that q %*% p gives the same answer. Thus, R handled these vectors as column vectors and computed .

# A vector times a matrix
M %*% p
     [,1]
[1,]  135
[2,]  150
[3,]  165

As M had dimensions , R treated p as a column vector in order to be conformable. The result is a vector, so this is the cross product.

If we try to compute p %*% M we get an error, because there is nothing R can do to p which will make it conformable to M.

p %*% M
Error in p %*% M: non-conformable arguments

What about multiplying matrices?

M %*% M
Error in M %*% M: non-conformable arguments

As you can see, when dealing with matrices, %*% will not change a thing, and if your matrices are non-conformable then it’s an error. Of course, if we transpose either instance of M we do have conformable matrices, but the answers are different, and this is neither the dot product or the cross product, just matrix multiplication.

t(M) %*% M
     [,1] [,2] [,3] [,4] [,5]
[1,]   14   32   50   68   86
[2,]   32   77  122  167  212
[3,]   50  122  194  266  338
[4,]   68  167  266  365  464
[5,]   86  212  338  464  590
M %*% t(M)
     [,1] [,2] [,3]
[1,]  335  370  405
[2,]  370  410  450
[3,]  405  450  495

What can we take from these examples?

< section id="other-functions" class="level3">

Other Functions

There are other R functions that do some of the same work:

The first two functions will accept combinations of vectors and matrices, as does %*%. Let’s try it with two vectors:

crossprod(p, q)
     [,1]
[1,]  130

Huh. crossprod is returning the dot product! So this is the case where “the cross product is not the cross product.” From a clarity perspective, this is not ideal. Let’s try the other function:

tcrossprod(p, q)
     [,1] [,2] [,3] [,4] [,5]
[1,]    6    7    8    9   10
[2,]   12   14   16   18   20
[3,]   18   21   24   27   30
[4,]   24   28   32   36   40
[5,]   30   35   40   45   50

There’s the cross product!

What about outer? Remember that another name for the cross product is the outer product. So is outer the same as tcrossprod? In the case of two vectors, it is:

identical(outer(p, q), tcrossprod(p, q))
[1] TRUE

What about a vector with a matrix?

tst <- outer(p, M)
dim(tst)
[1] 5 3 5

Alright, that clearly is not a cross product. The result is an array with dimensions , not a matrix (which would have only two dimensions). outer does correspond to the cross product in the case of two vectors, but anything with higher dimensions gives a different beast. So perhaps using “outer” as a synonym for cross product is not a good idea.

< section id="advice" class="level2">

Advice

Given what we’ve seen above, make your life simple and stick to %*%, and pay close attention to the dimensions of the arguments, especially if row or column vectors are in use. In my experience, thinking about the units and dimensions of whatever it is you are calculating is very helpful. Later, if speed is really important in your work, you can use one of the faster alternatives.

< section id="footnotes" class="footnotes footnotes-end-of-document">

Footnotes

  1. An extensive dicussion of notations can be found here.↩︎

  2. And curiously, the norm works out to be equal to the square root of the dot product of a vector with itself: ↩︎

  3. To be multiplied, matrices must be conformable, namely the number of columns of the first matrix must match the number of rows of the second matrix. The reason is so that the dot product terms will match. In the present case we have .↩︎

  4. Be careful, it turns out that “outer” may not be a great synonym for cross product, as explained later.↩︎

  5. OK fine, here is the answer when treating and as row vectors: which expands exactly as the right-hand side of Equation 14.↩︎

< section class="quarto-appendix-contents">

Reuse

https://creativecommons.org/licenses/by/4.0/
< section class="quarto-appendix-contents">

Citation

BibTeX citation:
@online{hanson2022,
  author = {Bryan Hanson},
  editor = {},
  title = {Notes on {Linear} {Algebra} {Part} 1},
  date = {2022-08-14},
  url = {http://chemospec.org/posts/2022-08-14-Linear-Alg-Notes/2022-08-14-Linear-Alg-Notes.html},
  langid = {en}
}
For attribution, please cite this work as:
Bryan Hanson. 2022. “Notes on Linear Algebra Part 1.” August 14, 2022. http://chemospec.org/posts/2022-08-14-Linear-Alg-Notes/2022-08-14-Linear-Alg-Notes.html.
To leave a comment for the author, please follow the link and comment on their blog: Chemometrics and Spectroscopy Using R.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.