Note on book Deep Learning Chapter 2 - Linear Algebra

Note: This is my notes while reading Deep Learning by Ian Goodfellow

2.1 Scalars, Vectors, Matrices and Tensors

Scalars: A scalar is just a single number. s ∈ R could be the slope of the line.

Vectors: A vector is an array of numbers arranged in order. We write a vector $\vec x$ as:

$\vec x$ = $\begin{bmatrix} x_1 \\ x_2 \\ x_3 \\ ... \\ x_n \end{bmatrix}$

Matrices: A matrix is a 2-D array of numbers, so each element is identified by two indices instead of just one.

$A_{i,j} = \begin{bmatrix} A_{1,1} & ... & A_{1,j} \\ ... \\ A_{i,1} & ... & A_{i,j}\end{bmatrix}$

Tensors: A tensor is an array of numbers arranged on a regular grid with a variable number of axes. We identify the element of A at coordinates $(i,j,k)$ by writing $A_{i,j,k}$.

Transpose

Transpose of a matrix $A$ is the mirror image across it's diagonal.

$A_{i,j}^T = A_{j,i}$

Example: $A = \begin{bmatrix}A_{1,1} & A_{1,2} \\ A_{2,1} & A_{2,2} \\ A_{3,1} & A_{3,2} \end{bmatrix} \implies A^T = \begin{bmatrix}A_{1,1} & A_{2,1} & A_{3,1} \\ A_{1,2} & A_{2,2} & A_{3,2} \end{bmatrix}$

Vector as a matrix with only one column:

$A = \begin{bmatrix}x_1 \\ x_2 \\ ... \\ x_n \end{bmatrix}$

then,

$A^T = \begin{bmatrix}x_1 & x_2 & ... & x_n \end{bmatrix}$

2.2 Multiplying Matrices and Vectors

The matrix product of matrices $A$ and $B$ is a third matrix $C$.

$C_{i,j} = \sum_k A_{i,k}B_{k,j}$

Properties:

• $A(B + C) = AB + AC$
• $A(BC) = (AB)C$
• $AB \not= BA$
• $(AB)^T = B^TA^T$
• $x^Ty = (x^Ty)^T = y^Tx$

2.3 Identity and Inverse Matrices

An identity matrix is a matrix that does not change any vector when we multiply that vector by that matrix.

$\forall x \epsilon R^n, I_n\vec{x} = \vec{x}$

The matrix inverse of $A$ is denoted as $A^{-1}$, and it is defined as the matrix such that

$A^{-1}A = I$

So of $A^{-1}$ exists,

$Ax = b$

$A^{-1}Ax = A^{-1}b$

$Ix = A^{-1}b$

$x = A^{-1}b$