Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

# BDA3 Chapter 14 Exercise 3

Here’s my solution to exercise 3, chapter 14, of Gelman’s Bayesian Data Analysis (BDA), 3rd edition. There are solutions to some of the exercises on the book’s webpage.

We need to reexpress $$(y – X\beta)^T (y – X\beta)$$ as $$(\mu – \beta)^T \Sigma^{-1} (\mu – \beta)$$, for some $$\mu$$, $$\Sigma$$. Using the QR-decomposition of $$X = QR$$, we see

\begin{align} (y – X\beta)^T(y – X\beta) &= (Q^T(y – X\beta))^TQ^T(y – X\beta) \\ &= (Q^Ty – Q^TX\beta)^T (Q^Ty – Q^TX\beta) \\ &= (Q^Ty – R\beta)^T (Q^Ty – R\beta) , \end{align}

where $$Q$$ is orthogonal and $$R$$ an invertible upper triangular matrix. We can read off the minimum of this quadratic form as

$\hat\beta = R^{-1}Q^Ty ,$

which shows that $$\mu = \hat\beta = R^{-1}Q^Ty$$. Note that

\begin{align} (X^TX)^{-1}X^T &= (R^TR)^{-1}R^T Q^T \\ &= R^{-1}R^{-T}R^T Q^T \\ &= R^{-1}Q^T \end{align}

so that $$\hat\beta = (X^TX)^{-1}X^Ty$$.

Expanding the brackets of both quadratic form expressions and comparing the quadratic coefficients, we see that

$\Sigma^{-1} = R^T R = X^T X ,$

which shows that $$V_\beta = (X^T X)^{-1}$$, in the notation of page 355.