4. Eigenvalues and Eigenvectors

For a square matrix \(A\), its eigenvectors are vectors \(x\) such that,

\[Ax = \lambda x\]

Where \(\lambda \in R\) is an eigenvalue of \(x\). In other words, the eigenvectors of \(A\) are vectors that when transformed by \(A\) are only scaled by some factor \(\lambda\) rather than, for example, changing direction.

4.1. Finding Eigenvalues and Eigenvectors

In our equation above, there are two unknows: \(\lambda\), and \(x\). Rearanging, some of the terms, we get,

\[(A - \lambda I) x = 0\]

Where \(I\) is the identity matrix. If \(A - \lambda I\) is invertible then \((A - \lambda I) ^ {-1} (A - \lambda I) x = (A - \lambda I) ^ {-1} 0 \implies x = 0\). Therefore, for \(x\) to have a non-zero solution, \(A - \lambda I\) cannot be invertible, and thus,

\[\det (A - \lambda I) = 0\]

This gives a formula to find the eigenvalues of \(A\) which can then be substituted into \((A - \lambda I) x = 0\) to find the eigenvectors \(x\).

4.2. Properties of Eigenvalues and Eigenvectors

Several properties of eigenvalues and eigenvectors.

• Squaring a square matrix \(A\) squares its eigenvalues but its eigenvectors remain the same.

• The eigenvectors of a symmetric matrix are orthogonal.

• A square matrix \(\mathbf{A}\) is invertible IFF its columns are linearly independent.

4.3. Properties of Eigenvalues and Eigenvectors (Proofs)

• Squaring a square matrix \(A\) squares its eigenvalues but its eigenvectors remain the same.

Suppose \(\mathbf{A}\) has the eigenvector \(\mathbf{x}_i\) and eigenvalue \(\lambda_i\).

\[\begin{split}\mathbf{A} \mathbf{x}_i = \lambda_i \mathbf{x}_i \\ \mathbf{A} \mathbf{A} \mathbf{x}_i = \lambda_i \mathbf{A} \mathbf{x}_i \\ \mathbf{A}^2 \mathbf{x}_i = \lambda_i^2 \mathbf{\mathbf{x}}_i. \\\end{split}\]

Thus, \(\mathbf{x}_i\) is an eigenvector of \(\mathbf{A}\) and \(\mathbf{A}^2\). And \(\lambda_i\) is an eigenvalue of \(\mathbf{A}\) but \(\lambda_i^2\) is an eigenvalue of \(\mathbf{A}^2\).

• The eigenvectors of a symmetric matrix are orthogonal.

Suppose \(\mathbf{A}\) is a two dimensional matrix with two eigenvectors and two distinct eigenvalues.

\[\begin{split}A x_1 & = \lambda_1 x_1, \\ x_2^T A x_1 & = \lambda_1 x_2^T x_1. \\\end{split}\]

And,

\[\begin{split}A x_2 & = \lambda_2 x_2, & \\ x_1^T A x_2 & = \lambda_2 x_1^T x_2, & \\ x_2^T A^T x_1 & = \lambda_2 x_2^T x_1. & \\ x_2^T A x_1 & = \lambda_2 x_2^T x_1. & \quad \text{(Symmetric)}\\\end{split}\]

Taking both equations from each other, we get,

\[\begin{split}x_2^T A x_1 - x_2^T A x_1 & = (\lambda_2 - \lambda_1) x_2^T x_1^T \\ 0 & = (\lambda_2 - \lambda_1) x_2^T x_1^T\end{split}\]

Since \(\lambda_1\) and \(\lambda_2\) are distinct, \((\lambda_2 - \lambda_1)\) is non-zero, therefore, \(x_2^T x_1^T\) must be zero. Thus, \(x_2\) and \(x_1\) are orthogonal.

4.3.1. The determinant of \(A\) is the product of its eigenvalues

The eigenvalues \(\lambda_1, \lambda_2, \dots, \lambda_n\), are the roots of its characteristic polynomial.

\[\det (A - \lambda I) = (\lambda_1 - \lambda) (\lambda_2 - \lambda) \dots (\lambda_n - \lambda)\]

Setting \(\lambda\) to \(0\), we get,

\[\det (A) = \lambda_1 \lambda_2 \dots \lambda_n\]

4.3.2. The trace of \(A\) is equal to the sum of its eigenvalues

For this, we only prove for the \(2 \times 2\) case. Let \(A\) be the following matrix,

\[\begin{split}A = \begin{pmatrix} a & b \\ c & d \\ \end{pmatrix}\end{split}\]

Find the eigenvalues of \(A\) using its charateristic polynomial.

\[\det (A - \lambda I) = (a - \lambda) (d - \lambda) - bc = \lambda ^ 2 - (a + d) \lambda - bc\]

Using the quatratic formula, we get,

\[\lambda = \frac{a + d \pm \sqrt{\ldots}}{2}\]

Therefore,

\[\lambda_1 + \lambda_2 = a + d\]

4.3.3. \(Ax = 0\) has a non-trivial solution IFF the columns of \(A\) are linearly independent

Suppose the columns of \(A\) are linearly independent. Let the vectors \(v_1, v_2, \dots, v_n\) be the columns of \(A\).

\[\mathbf{A} \mathbf{x} = \mathbf{0} \implies \begin{pmatrix}\mathbf{v}_1 & \mathbf{v}_2 & \dots & \mathbf{v}_n\end{pmatrix} \mathbf{x} = \mathbf{0}.\]

This can be rewritten as,

\[x_1 \mathbf{v}_1 + x_2 \mathbf{v}_2 + \dots + x_n \mathbf{v}_n = \mathbf{0}.\]

Since the columns of \(A\) are linearly independent, some column \(v_i\) can be expressed as a combination of the other columns. Therefore, there must be solution to the equation above where some of the scalars \(x_1, x_2, \dots, x_n\) are non-zero.

4.3.4. \(Ax = 0\) has a non-trivial solution IFF \(A\) is invertible

Theorem

A square matrix \(\mathbf{A}\) is invertible IFF its columns are linearly independent.

Proof

This proof is taken from https://sparse-plex.readthedocs.io/en/latest/book/matrices/invertible_matrices.html.

Step 1. Suppose that \(\mathbf{A}\) is invertible. By definition, the columns of \(\mathbf{A}\) are linearly independent if \(\mathbf{A x} = \mathbf{0}\) only when \(\mathbf{x} = \mathbf{0}\). Since \(A\) has an inverse,

\[\mathbf{A}^{-1} \mathbf{A x} = \mathbf{0} \implies \mathbf{x} = \mathbf{0}\]

Step 2. If the columns of \(\mathbf{A}\) are linearly independent, then \(\mathbf{A} \mathbf{x} = \mathbf{b}\) has a unique solution. If \(\mathbf{A}\) has an inverse then there exists a matrix \(\mathbf{B}\) such that \(\mathbf{B A} = \mathbf{I}\). Suppose that \(e_i\) represents the \(i\)-th column of \(\mathbf{I}\) such that the \(i\)-th element of \(\mathbf{e}_i\) is 1.

\[\begin{split}\mathbf{AB} = \begin{bmatrix} | & | & \dots & | \\ \mathbf{AB}_1 & \mathbf{AB}_2 & \dots & \mathbf{AB}_n \\ | & | & \dots & | \\ \end{bmatrix} = \begin{bmatrix} | & | & \dots & | \\ \mathbf{e}_1 & \mathbf{e}_2 & \dots & \mathbf{e}_n \\ | & | & \dots & | \\ \end{bmatrix}\end{split}\]

Since the columns of \(A\) are linearly independent, \(\mathbf{AB}_i = \mathbf{e}_i\) has a unique solution. Thus we can find each column \(\mathbf{B}_i\) thus \(\mathbf{A}\) is invertible.

Theorem

A symmetric matrix \(\mathbf{A}\) with distinct eigenvalues has orthogonal eigenvectors.

Proof

We only prove for the \(2 \times 2\) case where \(\mathbf{A}\) has two eigenvectors.

\[\begin{split}\mathbf{A} \mathbf{x}_1 & = \lambda_1 \mathbf{x}_1 \\ \mathbf{x}_2^T \mathbf{A} \mathbf{x}_1 & = \lambda_1 \mathbf{x}_2^T \mathbf{x}_1 \\\end{split}\]

And,

\[\begin{split}\mathbf{A x}_2 & = \lambda_2 \mathbf{x}_2 \\ \mathbf{x}_1^T \mathbf{A x}_2 & = \lambda_2 \mathbf{x}_1^T \mathbf{x}_2 \\ \mathbf{x}_2^T \mathbf{A x}_1 & = \lambda_2 \mathbf{x}_2^T \mathbf{x}_1 \\\end{split}\]

Taking these away from each other, we have,

\[\begin{split}\mathbf{x}_2^T \mathbf{A x}_1 - \mathbf{x}_2^T \mathbf{A x}_1 & = (\lambda_1 - \lambda_2) \mathbf{x}_2^T \mathbf{x}_1 \\ 0 & = (\lambda_1 - \lambda_2) \mathbf{x}_2^T \mathbf{x}_1\end{split}\]

Since \(\lambda_1\) and \(\lambda_2\) are distinct, their difference is non-zero. Therefore, \(\mathbf{x}_2^T \mathbf{x}_1\) must be \(0\).