What Is a Matrix?

• A matrix is a rectangular array of numbers.
• Shape is written as $m \times n$ (rows by columns).
• A vector is a special case:
  • Column vector: $n \times 1$
  • Row vector: $1 \times n$

Matrix Operations

Addition and Scalar Multiplication

• You can add matrices only if shapes match.
• Scalar multiplication scales every entry:

Matrix Multiplication

• If $A \in \mathbb{R}^{m \times n}$ and $B \in \mathbb{R}^{n \times p}$, then:

• Entry form:

• Matrix multiplication is generally not commutative:

Transpose

• Transpose flips rows and columns:

• Useful identities:

Geometry of Matrices

Matrix as a Linear Transformation

• A matrix $A$ maps vectors: $x \mapsto Ax$.
• Geometrically, this can rotate, scale, shear, reflect, or project space.

Column-Space View

• $Ax$ is a linear combination of columns of $A$.
• The set of all outputs is the column space of $A$.

Dot Product and Orthogonality

• Dot product:

• Orthogonal vectors satisfy $x^\top y = 0$.

Determinant

Meaning

• For square $A \in \mathbb{R}^{n \times n}$, $\det(A)$ measures signed volume scaling.
• In 2D, absolute determinant is area scale factor.
• In 3D, absolute determinant is volume scale factor.

Key Facts

• $\det(A)=0$ means dimensions collapse (matrix is singular).
• $\det(AB)=\det(A)\det(B)$
• $\det(A^\top)=\det(A)$

For a $2 \times 2$ matrix:

Invertibility

Inverse Matrix

• $A^{-1}$ satisfies:

• Only square matrices can be invertible.

Equivalent Conditions

For square $A$, these are equivalent:

• $A$ is invertible
• $\det(A)\ne 0$
• $\text{rank}(A)=n$
• Columns of $A$ are linearly independent
• $Ax=b$ has a unique solution for every $b$

Rank and Linear Independence

• Rank = number of linearly independent columns (or rows).
• Full column rank means no redundant features among columns.
• In ML, low-rank structure is common and exploited by PCA, SVD, and embeddings.

Systems of Linear Equations

• A linear system can be written as:

• Cases:
  • Unique solution: full rank square system
  • Infinite solutions: underdetermined / dependent equations
  • No solution: inconsistent system

Matrices in Machine Learning

• Dataset: $X \in \mathbb{R}^{m \times d}$ (samples by features)
• Linear model:

• Normal equation (least squares):

• Covariance matrix:

Practical Notes

Check shapes before multiplying.

• Most bugs in linear algebra code are shape mismatches.

Avoid explicit matrix inverse when solving systems.

• Prefer linear solvers (solve) for numerical stability and speed.

Standardise features for distance-based and regularised methods.

• Unscaled features can dominate matrix operations and degrade conditioning.

Why This Matters for ML

• Feature matrices and parameter matrices are the core objects in ML pipelines.
• Linear regression, logistic regression, and neural network forward passes are matrix-based computations.
• Conditioning, rank, and invertibility directly affect stability, optimization, and generalization.
• Dimensionality reduction and covariance modeling are impossible to understand without matrix geometry.