Mathematicians like to categorize things. There are some types of matrices to which mathematicians give specific names so that they can refer to them quickly without having to explain what they do in words:

 I have this matrix A whose rows are perpendicular vectors and 
 then when you multiply any vector by this matrix it doesn't change 
 the length of the vector but just kind of rotates it and stuff...

It is much simpler just to say:

 Let A be an orthogonal matrix.

Most advanced science textbooks and research papers will use terminology like “diagonal matrix”, “symmetric matrix”, and “orthogonal matrix”, so I want you to become familiar with these concepts.

This section also serves to review and reinforce what we learned about linear transformations. Recall that we can think of the matrix-vector product $A\vec{x}$ as applying a linear transformations $T_A$ to the input vector $\vec{x}$. Therefore, each of the special matrices which we will discuss here also corresponds to a special type of linear transformation. Keep this dual-picture in mind because the same terminology can be used to describe matrices and linear transformations.

• $\mathbb{R}^{m \times n}$: the set of $m \times n$ matrices
• $A,B,O,P,\ldots$: typical variable names for matrices
• $a_{ij}$: the entry in the $i$th row and $j$th column of the matrix $A$
• $A^T$: the transpose of the matrix $A$
• $A^{-1}$: the inverse of the matrix $A$. The inverse obeys $AA^{-1}=A^{-1}A=I$.
• $\lambda_1, \lambda_2, \ldots$: the eigenvalues of the matrix $A$.

For each eigenvalue $\lambda_i$ there is at least one associated eigenvector $\vec{e}_{\lambda_i}$ such that the following equation holds:

  \[
    A\vec{e}_{\lambda_i} = \lambda_i \vec{e}_{\lambda_i}.
  \]
  Multiplying the matrix $A$ by its eigenvectors $\vec{e}_{\lambda_i}$ 
  is the same scaling $\vec{e}_{\lambda_i}$ by the number $\lambda_i$.

These are matrices that only have entries on the diagonal and are zero everywhere else. For example: \[ \left(\begin{array}{ccc} a_{11} & 0 & 0 \nl 0 & a_{22}& 0 \nl 0 & 0 & a_{33} \end{array}\right). \] More generally we say that a diagonal matrix $A$ satisfies, \[ a_{ij}=0, \quad \text{if } i\neq j. \]

The eigenvalues of a diagonal matrix are $\lambda_i = a_{ii}$.

A matrix $A$ is symmetric if and only if \[ A^T = A, \qquad a_{ij} = a_{ji}, \quad \text{ for all } i,j. \] All eigenvalues of a symmetric transformation are real numbers, and the its eigenvectors can be chosen to be mutually orthogonal. Given any matrix $B\in\mathbb{M}(m,n)$, the product of $B$ with its transpose $B^TB$ is always a symmetric matrix.

Upper triangular matrices have zero entries below the main diagonal: \[ \left(\begin{array}{ccc} a_{11} & a_{12}& a_{13} \nl 0 & a_{22}& a_{23} \nl 0 & 0 & a_{33} \end{array}\right), \qquad a_{ij}=0, \quad \text{if } i > j. \]

A lower triangular matrix is one for which all the entries above the diagonal are zeros: $a_{ij}=0, \quad \text{if } i < j$.

The identity matrix is denoted as $I$ or $I_n \in \mathbb{M}(n,n)$ and plays the role of the number $1$ for matrices: $IA=AI=A$. The identity matrix is diagonal with ones on the diagonal: \[ I_3 = \left(\begin{array}{ccc} 1 & 0 & 0 \nl 0 & 1 & 0 \nl 0 & 0 & 1 \end{array}\right). \]

Any vector $\vec{v} \in \mathbb{R}^3$ is an eigenvector of the identity matrix with eigenvalue $\lambda = 1$.

A matrix $O \in \mathbb{M}(n,n)$ is orthogonal if it satisfies $OO^T=I=O^TO$. The inverse of an orthogonal matrix $O$ is obtained by taking its transpose: $O^{-1} = O^T$.

The best way to think of orthogonal matrices is to think of them as linear transformations $T_O(\vec{v})=\vec{w}$ which preserve the length of vectors. The length of a vector before applying the linear transformation is given by: $\|\vec{v}\|=\sqrt{ \vec{v} \cdot \vec{v} }$. The length of a vector after the transformation is \[ \|\vec{w}\| =\sqrt{ \vec{w} \cdot \vec{w} } =\sqrt{ T_O(\vec{v}) \cdot T_O(\vec{v}) } = (O\vec{v})^T(O\vec{v}) = \vec{v}^TO^TO\vec{v}. \] When $O$ is an orthogonal matrix, we can substitute $O^TO=I$ in the above expression to establish $\|\vec{w}\|=\vec{v}^TI\vec{v}=\|\vec{v}\|$, which shows that orthogonal transformations are length preserving.

The eigenvalues of an orthogonal matrix have unit length, but can in general be complex numbers $\lambda_i=\exp(i\theta) \in \mathbb{C}$. The determinant of an orthogonal matrix is either one or minus one $|O|\in\{-1,1\}$.

A good way to think about orthogonal matrices is to imagine that their columns form an orthonormal basis for $\mathbb{R}^n$: \[ \{ \hat{e}_1,\hat{e}_2,\ldots, \hat{e}_n \}, \quad \hat{e}_{i} \cdot \hat{e}_{j} = \left\{ \begin{array}{ll} 1 & \text{ if } i =j, \nl 0 & \text{ if } i \neq j. \end{array}\right. \] The resulting matrix \[ O= \begin{bmatrix} | & & | \nl \hat{e}_{1} & \cdots & \hat{e}_{n} \nl | & & | \end{bmatrix} \] is going to be an orthogonal matrix. You can verify this by showing that $O^TO=I$. We can interpret the matrix $O$ as a change of basis from the stander basis to the $\{ \hat{e}_1,\hat{e}_2,\ldots, \hat{e}_n \}$ basis.

The set of orthogonal matrices contains as special cases the following important classes of matrices: rotation matrices, refection matrices, and permutation matrices. We'll now discuss each of these in turn.

A rotation matrix takes the standard basis $\{ \hat{\imath}, \hat{\jmath}, \hat{k} \}$ to a rotated basis $\{ \hat{e}_1,\hat{e}_2,\hat{e}_3 \}$.

Consider first an example in $\mathbb{R}^2$. The counterclockwise rotation by the angle $\theta$ is given by the matrix \[ R_\theta = \begin{bmatrix} \cos\theta &-\sin\theta \nl \sin\theta &\cos\theta \end{bmatrix}. \] The matrix $R_\theta$ takes $\hat{\imath}=(1,0)$ to $(\cos\theta,\sin\theta)$ and $\hat{\jmath}=(0,1)$ to $(-\sin\theta,\cos\theta)$.

As a second example, consider the rotation by the angle $\theta$ around the $x$-axis in $\mathbb{R}^3$: \[ \begin{bmatrix} 1&0&0\nl 0&\cos\theta&-\sin\theta\nl 0&\sin\theta&\cos\theta \end{bmatrix}. \] Note this is a rotation entirely in the $yz$-plane: the $x$-component of a vector multiplying this matrix would remain unchanged.

The determinant of a rotation matrix is equal to one. The eigenvalues of rotation matrices are complex numbers with magnitude one.

If the determinant of an orthogonal matrix $O$ is equal to negative one, then we say that it is mirrored orthogonal. For example, the reflection through the line with direction vector $(\cos\theta, \sin\theta)$ is given by: \[ R= \begin{bmatrix} \cos(2\theta) &\sin(2\theta)\nl \sin(2\theta) &-\cos(2\theta) \end{bmatrix}. \]

A reflection matrix will always have at least one eigenvalue equal to minus one, which corresponds to the direction perpendicular to the axis of reflection.

Another important class of orthogonal matrices is the class permutation matrices. The action of a permutation matrix is simply to change the order of the coefficients of a vector. For example, the permutation $\hat{e}_1 \to \hat{e}_1$, $\hat{e}_2 \to \hat{e}_3$, $\hat{e}_3 \to \hat{e}_2$ can be represented as the following matrix: \[ M_\pi = \begin{bmatrix} 1 & 0 & 0 \nl 0 & 0 & 1 \nl 0 & 1 & 0 \end{bmatrix}. \] An $n \times n$ permutation contains $n$ ones in $n$ different columns and zeros everywhere else.

The sign of a permutation corresponds to the determinant $\det(M_\pi)$. We say that a permutation $\pi$ is even if $\det(M_\pi) = +1$ and odd if $\det(M_\pi) = -1$.

A matrix $P \in \mathbb{M}(n,n)$ is positive semidefinite if \[ \vec{v}^T P \vec{v} \geq 0, \] for all $\vec{v} \in \mathbb{R}^n$. The eigenvalues of a positive semidefinite matrix are all non-negative $\lambda_i \geq 0$.

If we have $\vec{v}^T P \vec{v} > 0$, for all $\vec{v} \in \mathbb{R}^n$, we say that the matrix is positive definite. These matrices have eigenvalues strictly greater than zero.

The defining property of a projection matrix is that it can be applied multiple times without changing the result: \[ \Pi = \Pi^2= \Pi^3= \Pi^4= \Pi^5 = \cdots. \]

A projection has two eigenvalues: one and zero. The space $S$ which is left invariant by the projection $\Pi_S$ corresponds to the eigenvalue $\lambda=1$. The space $S^\perp$ of vectors that get completely annihilated by $\Pi_S$ corresponds to the eigenvalue $\lambda=0$, which is also the null space of $\Pi_S$.

The matrix $A = \mathbb{M}(n,n)$ is normal if $A^TA=AA^T$. If $A$ is normal we have the following properties:

1. The matrix $A$ has a full set of linearly independent eigenvectors.

Eigenvectors corresponding to distinct eigenvalues are orthogonal

  and eigenvectors from the same eigenspace can be chosen to be mutually orthogonal.
- For all vectors $\vec{v}$ and $\vec{w}$ and a normal transformation $A$ we have: 
  \[
   (A\vec{v}) \cdot (A\vec{w}) 
    = (A^TA\vec{v})\cdot \vec{w}
    =(AA^T\vec{v})\cdot \vec{w}.
   \]
- $\vec{v}$ is an eigenvector of $A$ if and only if $\vec{v}$ is an eigenvector of $A^T$.

Every normal matrix is diagonalizable by an orthogonal matrix $O$. The eigendecomposition of a normal matrix can be written as $A = O\Lambda O^T$, where $O$ is orthogonal and $\Lambda$ is a diagonal matrix. Note that orthogonal ($O^TO=I$) and symmetric ($A^T=A$) matrices are special types of normal matrices since $O^TO=I=OO^T$ and $A^TA=A^2=AA^T$.

In this section we defined several types of matrices and stated their properties. You're now equipped with some very precise terminology for describing the different types of matrices.

TODO: add a mini concept map here More importantly, we discussed the relations. It might be a good idea to summarize these relationships as a concept map…