1.3
Matrix Diagonalization and Spectral Theorem
[MIT 18.05] Matrix Methods in Data Analysis, Signal Processing, and Machine Learning
Junseo Lee

Eigenvalues and Eigenvectors

For a matrix
A
, we have
Axλx
. We call
x
an eigenvector of
A
and
λ
an eigenvalue of
A
.
We can immediately see that the eigenvector
x
doesn’t change direction when multiplied by the matrix
A
.
In other words, the output
Ax
is on the same line as the input vector
x
, since it is just multiplied by its eigenvalue
λ
.

Computing the Eigenvalues

Note that
Axλx
is equivalent to
(A-λI)x0
.
In order for this homogeneous linear system to have nontrivial solutions (since we are mostly interested in the nonzero eigenvectors), the coefficient matrix
A-λI
must be singular, which means its determinant must be zero. The equation
det(A-λI)0
is an
n
th degree equation for
λ
and has
n
roots, which means we have
n
eigenvalues. We call this equation the characteristic equation for matrix
A
.
​

Example

Find the eigenvalues and eigenvectors of

8
3
2
7

In[]:=
A = {{8, 3}, {2, 7}};​​% // MatrixForm
Out[]//MatrixForm=

8
3
2
7

In[]:=
(* Create a 2 by 2 identity matrix *)​​I2 = IdentityMatrix[2];​​% // MatrixForm
Out[]//MatrixForm=

1
0
0
1

Characteristic equation and eigenvalues
We can find the characteristic equation of our matrix
A
by computing the determinant of
A-λI
In[]:=
fλ = Det[A - λ I2] == 0
Out[]=
50-15λ+
2
λ
0
We can see that the characteristic equation is a 2nd degree equation for
λ
.
Now, let us solve this equation to find the eigenvalues of
A
In[]:=
Solve[fλ, λ]
Out[]=
{{λ5},{λ10}}
Thus, there are two eigenvalues of
λ5
and
λ10
.
Alternatively, we can use the built-in Wolfram function
Eigenvalues[]
to find the eigenvalues of
A
directly.
In[]:=
Eigenvalues[A]
Out[]=
{10,5}
​
Eigenvectors corresponding to each eigenvalue
Next, we can find the corresponding eigenvectors of
A
by solving the system
(A-λI)x0
for
x
.
First for
λ5
,
In[]:=
NullSpace[A - 5 I2]
Out[]=
{{-1,1}}
We have the eigenvector
x
1

-1
1

corresponding to
λ5
.
Next, for
λ10
,
In[]:=
NullSpace[A - 10 I2]
Out[]=
{{3,2}}
We have the eigenvector
x
2

3
2

corresponding to
λ10
.
Alternatively, we can use the built-in Wolfram function
Eigenvectors[]
to find the eigenvectors of
A
directly.
In[]:=
Eigenvectors[A]
Out[]=
{{3,2},{-1,1}}
​

Similar Matrices and Diagonalization

Similar Matrices

For every invertible matrix
B
, the eigenvalues of
BA
-1
B
are the same as the eigenvalues of
A
. The eigenvectors
x
of
A
are multiplied by
B
to give eigenvectors
Bx
of
BA
-1
B
If
Axλx
, then
(BA
-1
B
)(Bx)BAxBλxλ(Bx)
.
We say that matrices
BA
-1
B
are “similar” to
A
.

Diagonalization Problem

Given a square matrix A, can we find an invertible matrix
X
such that
-1
X
AXD
where
D
is a diagonal matrix?
If such a matrix exists, then
A
is said to be diagonalizable and
X
is said to diagonalize A.
​
Suppose a square matrix
A
has a full set of
n
independent eigenvectors.
Put those eigenvectors
x
1
,…,
x
n
as columns of an invertible matrix
X
.
X
x
1
…
x
n

Then, multiply
AX
column by column to get the columns
λ
1
x
1
to
λ
n
x
n
, which can be split into
X
times
Λ
, which is a diagonal matrix with eigenvalues of
A
.
A
x
1
…
x
n

A
x
1
…
A
x
n

λ
1
x
1
…
λ
n
x
n

x
1
…
x
n

λ
1
⋱
λ
n
Simplifying,
AXXΛ
.
Multiply
-1
X
from left to both sides, and we have
-1
X
AXΛ
where
Λ
is a diagonal matrix.
Alternatively, we could think of it as a factorization of
A
into
AXΛ
-1
X
.

Theorem

An
nn
matrix
A
is diagonalizable if and only if
A
has
n
linearly independent eigenvectors.
​

Example

Given the matrix
A
8
3
2
7

, decompose
A
into
XΛ
-1
X
.
In[]:=
A = {{8, 3}, {2, 7}}
Out[]=
{{8,3},{2,7}}
Firstly, find the eigenvalues of
A
and form the diagonal eigenvalue matrix
Λ
.
In[]:=
Λ
= DiagonalMatrix[Eigenvalues[A]]
Out[]=
{{10,0},{0,5}}
Next, find the set of eigenvectors of
A
, and form the invertible eigenvector matrix
X
.
In[]:=
X = Transpose[Eigenvectors[A]];​​% // MatrixForm
Out[]//MatrixForm=

3
-1
2
1

Find the inverse of
X
.
In[]:=
Xinv = Inverse[X]
Out[]=

1
5
,
1
5
,-
2
5
,
3
5

Multiply
XΛ
-1
X
to get
A
In[]:=
X.
Λ
.Xinv
Out[]=
{{8,3},{2,7}}
This matrix is the same as
A
.
In[]:=
A == X.
Λ
.Xinv
Out[]=
True

Spectral Theorem

Orthogonal Matrices

An orthogonal matrix
Q
is a square matrix whose columns are orthonormal. It has the property
T
Q

-1
Q
, and therefore,
T
Q
QI
and
Q
T
Q
I
.
The columns of this orthogonal
nn
matrix are an orthonormal basis for
n
R
, and the rows of
Q
also form an orthonormal basis for
n
R
.
​

Example

As expected, the result is an identity matrix of size 2.

Remark

A matrix is orthogonally diagonalizable if and only if it is symmetric.

Example

Beautiful! We have an diagonal matrix with eigenvalues.