The Math Toolkit of AI
Vectors, probability, calculus and optimization in one place.
Train any model in this course and the same five tools are at work: vectors hold the data, derivatives measure sensitivity, probability models the noise, optimization fits the parameters, and cross-entropy scores the predictions. Here you derive each tool once, together with the results you will lean on for the rest of the course.
What you’ll learn
- Work with vectors and matrices as a practitioner does: dot products, norms, linear layers and the eigendecomposition of symmetric matrices.
- Compute gradients with the multivariable chain rule and derive the least-squares normal equations.
- Derive a loss function from a probabilistic model by maximum likelihood.
- Prove the descent lemma and use it, with the condition number, to choose a stable learning rate.
- Relate entropy, cross-entropy and KL divergence, and derive the softmax cross-entropy gradient .
Vectors, matrices and the geometry of data
A data point with features is a vector , and a dataset of points stacks into a matrix , one example per row. Two operations do most of the work in AI. The dot product scores how well lines up with a weight vector . A matrix-vector product computes many dot products at once: a linear layer with maps input features to outputs, one row of per output.
Sizes are measured with norms, and . The Cauchy–Schwarz inequality keeps the cosine similarity inside , which is how embeddings are compared later in the course.
Symmetric matrices are everywhere in AI: covariance matrices, Gram matrices , kernel matrices and Hessians. A Gram matrix is always positive semidefinite, because . For centered data, is the covariance matrix and its eigenvectors are the principal components (PCA). Eigenvalues of this kind will decide how fast gradient descent converges in step 5.
A linear layer maps 784-pixel images to 10 class scores, . What are the shapes of and , and how many parameters does the layer have?
- is and : 8,624 parameters
- is and : 614,666 parameters
- is and : 7,850 parameters
- is and : 8,624 parameters
Show the answer
C. is and : 7,850 parameters
must be a 10-vector, so has 10 rows (one per class) and 784 columns (one per pixel), and matches the output: parameters. Each row of is a template that the image is dot-producted with.
Gradients, Jacobians and the chain rule
For , the gradient collects the partial derivatives and gives the best linear approximation, . By Cauchy–Schwarz, among all small steps of a given length the one along decreases fastest: steepest descent. For a vector-valued , the Jacobian has entries . The second derivatives of a scalar form the symmetric Hessian , which measures curvature:
Gradients of compositions follow the multivariable chain rule: Jacobians multiply. For with ,
Backpropagation is this formula applied to a network layer by layer, from the loss back to the inputs.
| Where it appears | ||
|---|---|---|
| linear models, logits | ||
| , which is for symmetric | quadratic losses, Gaussian log-densities | |
| least squares | ||
| weight decay | ||
| softmax classifiers |
Fit a linear model to targets by minimizing the squared error. Expand the loss:
Differentiate term by term with the table ( is symmetric):
Set the gradient to zero. These are the normal equations. When has full column rank, is positive definite and hence invertible:
The Hessian is positive semidefinite everywhere, so is convex and the stationary point is a global minimum. Geometrically, the residual is orthogonal to every column of : the fit is the orthogonal projection of onto the column space of .
Think first Why does the gradient of involve rather than ?
Write . The variable appears in row (the terms ) and in column (the terms ), so . Only the symmetric part of affects a quadratic form, so you can always replace by .
Probability: from noise models to losses
A random variable has a distribution, its expectation is the probability-weighted average, and its variance measures spread. Two facts carry a surprising amount of weight. Expectation is linear, , whether or not and are independent. Variances add only for uncorrelated variables, so the mean of independent draws with variance has .
Conditional probability and Bayes' rule, , turn a model of how data arise into beliefs about the parameters: the posterior is proportional to the likelihood times the prior. Most of machine learning uses the likelihood alone. Maximum likelihood picks the parameters under which the observed data are most probable. For independent examples the likelihood is a product, so you maximize its logarithm, a sum, or equivalently minimize the negative log-likelihood:
Model each target as a linear prediction plus Gaussian noise, with independent . Then
Take the negative logarithm and sum over the data:
The second term does not depend on . So maximum likelihood under Gaussian noise is least squares, the mean squared error loss of the previous step:
Change the model and the loss changes with it. Laplace noise gives absolute error, and a Bernoulli model of binary labels gives the cross-entropy loss of logistic regression (step 6). A Gaussian prior adds a ridge penalty: the MAP estimate is
Think first You average independent measurements, each with standard deviation . How many do you need before the average has standard deviation ?
, so the standard deviation of the average is . From you get , so . Halving the error costs four times the data. The same law governs mini-batch gradient noise, Monte Carlo estimates and the error bars of a test set; the Deep Layer turns it into high-probability guarantees with concentration inequalities.
Convexity and the descent lemma
Most losses have no closed-form minimizer, so you descend instead: with learning rate . When is a step guaranteed to make progress, and when does gradient descent find the global minimum rather than a local one? Two properties of the loss answer those questions.
Convexity turns local search into global optimization. If , the tangent-plane inequality at reads for every : every stationary point of a convex function is a global minimizer. Least squares and logistic regression have convex losses; neural networks do not, which is why their training is analyzed so differently. Smoothness, for its part, is what makes a gradient step safe:
Write as the integral of the gradient along the segment , , and split off the gradient at :
By Cauchy–Schwarz and the Lipschitz condition the integrand is at most , and . So lies below a quadratic that touches it at :
Substitute the gradient step :
The bracket is positive exactly when , and the guaranteed decrease is largest at . Summing the decreases over steps (they telescope) bounds the gradient even for nonconvex with minimum value :
Learning rates and conditioning
Why exactly , and what decides the speed? Take a quadratic with positive definite, so and . A gradient step is , and in the eigenbasis the coordinates decouple:
The largest eigenvalue caps the learning rate, while the smallest, , sets the pace along the flattest direction. The best constant step, , balances the two and contracts the error by a factor per step, where is the condition number. Long, thin valleys (large ) are slow for plain gradient descent, and much of the optimizer zoo, from momentum to Adam, exists to fix exactly this. The Deep Layer proves rates of this kind for every smooth, strongly convex function.
The valley in the widget below is . Gradient descent converges from every start exactly when the learning rate is below some threshold. What is the threshold?
Show the answer
0.5
Here with , so , and the steep -direction needs , i.e. . The flat -direction alone would tolerate any . Even the best constant step, , shrinks the error only by per step, because .
Entropy, cross-entropy and KL divergence
How should you score a predicted distribution when outcomes actually follow ? Information theory starts from surprise: an outcome to which you gave probability carries of it. The entropy is the average surprise of about its own outcomes, and the cross-entropy is the average surprise of a model when outcomes follow :
The KL divergence is never negative and is zero only when (Gibbs' inequality, proved in the Deep Layer). For a fixed data distribution , minimizing the cross-entropy over models therefore minimizes , the mismatch between model and truth. Let be the empirical distribution of a training set and the cross-entropy becomes the average negative log-likelihood of step 3: minimizing cross-entropy is maximum likelihood.
A classifier outputs scores (logits) and turns them into probabilities with the softmax. For a one-hot label with true class , the cross-entropy loss simplifies to a log-sum-exp:
Differentiate with respect to each logit. The first term contributes , and the log-sum-exp contributes the softmax itself (see the table in step 2):
The gradient is prediction minus target. Each entry lies in , the gradient vanishes only when the prediction matches the label, and a descent step raises the true logit in proportion to and lowers every other logit in proportion to :
With two classes the softmax reduces to the sigmoid , and the same computation gives . For logistic regression, , so by the chain rule
A classifier gives probability to the true class of an example. What is that example's cross-entropy loss, in bits?
Show the answer
C.
Only the probability of the true class enters: bits. A confident correct prediction () costs almost nothing, while a confident wrong one () costs without bound. That asymmetry is why cross-entropy rewards honest probabilities.
import numpy as np
def sigmoid(z):
return 1.0 / (1.0 + np.exp(-z))
def fit_logistic(X, y, eta=0.1, steps=2000):
"""Gradient descent on the mean cross-entropy.
X: (N, n) features, y: (N,) labels in {0, 1}."""
N, n = X.shape
w = np.zeros(n)
for _ in range(steps):
q = sigmoid(X @ w) # predicted P(y = 1 | x)
grad = X.T @ (q - y) / N # mean of (q_i - y_i) x_i
w -= eta * grad
return wLogistic regression uses the whole kit: a matrix-vector product, a probabilistic model, the cross-entropy gradient pushed back through , and a gradient step. The mean loss is convex and -smooth with , because , so any is safe. If the classes are linearly separable, no minimizer exists: the loss keeps falling as .
Key takeaways
- Data are vectors and layers are matrices. Symmetric matrices diagonalize as , and the eigenvalues of Gram matrices, covariances and Hessians govern both geometry and optimization.
- Gradients compose by the chain rule, ; least squares has the closed form .
- Loss functions are negative log-likelihoods: Gaussian noise gives squared error, Bernoulli or categorical labels give cross-entropy, and a Gaussian prior adds a ridge penalty.
- For -smooth , gradient descent decreases the loss whenever ; for convex every stationary point is a global minimum, and the condition number sets the pace.
- with , so minimizing cross-entropy is maximum likelihood, and the softmax cross-entropy gradient is .
Check yourself
In the interactive lesson these are scored with your confidence, and what you miss comes back for review.
Gradient descent on a differentiable convex function reaches a point where . What can you conclude?
- It is a local minimizer, but possibly not a global one
- It is a global minimizer
- It could be a saddle point
- Nothing, until you check the Hessian
Show the answer
B. It is a global minimizer
For convex differentiable , the tangent plane lies below the graph: for every . A stationary point is therefore globally optimal, though the minimizer need not be unique.
Targets are modeled as with Laplace noise, whose density is proportional to . Which loss does maximum likelihood minimize?
- Squared error,
- Cross-entropy
- Absolute error,
- Squared error plus
Show the answer
C. Absolute error,
The negative log-likelihood is plus a constant. Heavier-tailed noise gives a loss that grows only linearly in the residual, so outliers pull on the fit less than under squared error.
You run gradient descent on , where has eigenvalues and . Which learning rates make it converge from every starting point?
- Only exactly
- Any between and
Show the answer
A.
The component along eigenvalue is multiplied by at every step and must shrink, so forces . The flat direction then crawls with factor per step: the price of the condition number .
Which statement about the KL divergence is true?
- It is symmetric:
- It is negative when fits the data better than does
- It is finite for every pair of distributions
- It equals , so for fixed , minimizing the cross-entropy over minimizes it
Show the answer
D. It equals , so for fixed , minimizing the cross-entropy over minimizes it
, and does not depend on . KL is not symmetric, never negative (Gibbs' inequality), and infinite when for some with .
A 3-class softmax classifier outputs on an example whose true class is the second one. What is the gradient of the cross-entropy loss with respect to the logits ?
Show the answer
B.
. A descent step raises the true class's logit and lowers the others. The entries sum to zero because adding the same constant to every logit leaves the softmax unchanged.
Go deeper
- Probability Essentials · Random variables, Bayes’ rule, expectations and concentration.
- Optimization & Convexity · Gradients, convex sets, Lagrange multipliers and duality.
- Information Theory for AI · Entropy, cross-entropy and KL divergence as the language of learning.
Further reading
- I. Goodfellow, Y. Bengio & A. Courville, Deep Learning (MIT Press, 2016), chapters 2–4.
- M. P. Deisenroth, A. A. Faisal & C. S. Ong, Mathematics for Machine Learning (Cambridge University Press, 2020).
- S. Boyd & L. Vandenberghe, Convex Optimization (Cambridge University Press, 2004), chapters 2, 3 and 9.
- T. M. Cover & J. A. Thomas, Elements of Information Theory, 2nd ed. (Wiley, 2006), chapter 2.