Conditional and marginal distributions of a multivariate Gaussian

While reading up on Gaussian Processes (GPs), I decided it would be useful to be able to prove some of the basic facts about multivariate Gaussian distributions that are the building blocks for GPs.  Namely, how to prove that the conditional distribution and marginal distribution of a multivariate Gaussian is also Gaussian, and to give its form.

Preliminaries

First, we know that the density of a multivariate normal distribution with mean and covariance is given by

.

For simplicity of notation, I’ll now assume that the distribution has zero-mean, but everything should carry over in a straightforward manner to the more general case.

Writing out as two components , we are now interested in two distributions, the conditional and the marginal .

Separate the components of the covariance matrix into a block matrix , such that corresponds to the covariance for , similarly for , and contains the cross-terms.

Rewriting the Joint

We’d now like to be able to write out the form for the inverse covariance matrix .  We can make use of the Schur complement and write this as

.

I’ll explain below how this can be derived.

Now, we know that the joint distribution can be written as

.

We can substitute in the above expression of the inverse of the block covariance matrix, and if we simplify by multiplying the outer matrices, we obtain

.

Using the fact that the center matrix is block diagonal, we have

.

Wrapping up

At this point, we’re pretty much done.  If we condition on , the second exponential term drops out as a constant, and we have

.

Note that if and are uncorrelated, , and we just get the marginal distribution of .

If we marginalize over , we can pull the second exponential term outside the integral, and the first term is just the density of a Gaussian distribution, so it integrates to 1, and we find that

.

Schur complement

Above, I wrote that you could use the Schur complement to get the block matrix form of the inverse covariance matrix.  How would one actually derive that?  As mentioned in the wikipedia page, the expression for the inverse can be derived using Gaussian elimination.

If you right-multiply the covariance by the left-most matrix in the expression, you obtain

zero-ing out the bottom right matrix.  Multiplying by the center matrix gives you the identity in the diagonal components, and the right-most matrix zeros out the top left matrix, giving you the identity, so the whole expression is the inverse of the covariance matrix.

Further Reading

I got started on this train of thought after reading the wikipedia page on Gaussian processes.  The external link on the page to a gentle introduction to GPs was somewhat helpful as a quick primer.  The video lectures by MacKay and Rasmussen were both good and helped to give a better understanding of GPs.

MacKay also has a nice short essay on the Humble Gaussian distribution, which gives more information on the covariance and inverse covariance matrices of Gaussian distributions.  In particular, the inverse covariance matrix tells you the relationship between two variables, conditioned on all other variables, and therefore changes if you marginalize out some of the variables.  The sign of the off diagonal elements in the inverse covariance matrix is opposite the sign of the correlation between the two variables, conditioned on all the other variables.

To go deeper into Gaussian Processes, one can read the book Gaussian Processes for Machine Learning, by Rasmussen and Williams, which is available online.  The appendix contains useful facts and references on Gaussian identities and matrix identities, such as the matrix inversion lemma, another application of Gaussian elimination to determine the inverse, in this case the inverse of a matrix sum.