Simple Algorithm
Artificial Intelligence · Input Layer · 17 min

Machine Learning Essentials

Learning patterns from data instead of programming rules.

Nobody can write down the rules for recognizing a cat, a spam email or a fraudulent payment, yet machines do all three every day. Instead of being programmed with rules, they are shown examples and left to find the pattern. The catch: a pattern that fits every example perfectly can still fail on the next one.

What you’ll learn

  • Distinguish supervised, unsupervised and reinforcement learning, and name the parts of a supervised problem.
  • Explain how a model learns by minimizing a loss on its training data.
  • Recognize overfitting and underfitting from training and test error, and name the remedies.
  • Describe what k-means clustering finds without any labels.
  • Evaluate a classifier with a confusion matrix, precision, recall and ROC, and explain why accuracy can mislead.

Rules versus examples

Try writing rules that catch spam. "Contains free"? Plenty of real mail does. "Contains winner"? Spammers misspell it. Every rule has exceptions, and spammers adapt. Machine learning flips the approach: collect examples, and let an algorithm find the rule.

In supervised learning every example comes with the right answer. The inputs are features (word counts, pixel values, a house's floor area), the answer is a label (spam or not, a price), and the examples together form the training set. The goal is a function from features to labels that also works on examples it has never seen.

ParadigmLearns fromExample
SupervisedExamples with correct answersPredict a house's price from its size and location
UnsupervisedExamples without answersGroup customers into segments nobody defined in advance
ReinforcementRewards for its own actionsLearn a game from wins and losses
Self-supervisedAnswers hidden in the data itselfPredict the next word of a sentence
Four ways to learn from experience. This topic covers the first two; reinforcement learning and self-supervised language models get topics of their own.
Check your understanding

A bank has millions of past payments, each marked "fraud" or "legitimate", and wants to flag fraud in new payments. What kind of learning is this?

  1. Unsupervised learning
  2. Reinforcement learning
  3. Supervised learning
  4. None: the bank should write the rules by hand
Show the answer

C. Supervised learning

Each example has features (amount, place, time and so on) and a label (fraud or not). Learning a mapping from features to labels is supervised learning; with a yes-or-no label it is called classification.

Learning a boundary

A classifier maps features to a class. With two features you can draw it as a map: every point of the plane is colored by the class the model would predict there. The border between colors is the decision boundary, and different models draw it differently.

The simplest model is k-nearest neighbors: to classify a new point, find the kk closest training points and let them vote. Logistic regression draws a single straight line instead.

How do models like logistic regression find their parameters? You define a loss that measures how wrong the predictions are on the training set, then minimize it, usually with gradient descent. For predicting numbers, the classic loss is the mean squared error:

MSE=1n∑i=1n(yi−y^i)2\text{MSE} = \frac{1}{n} \sum_{i=1}^{n} \big(y_i - \hat y_i\big)^2
yiy_i is the true value for example ii and y^i\hat y_i the model's prediction. Squaring makes every error count positively, and big errors count a lot.

Memorizing is not learning

Low error on the training set is easy to get: memorize it. What matters is generalization, low error on new data. So you hold back a test set that the model never trains on, and judge it there.

Below, the points come from a smooth wave plus noise. A polynomial of degree 1 is a straight line; higher degrees can bend more.

Check your understanding

You fit a polynomial of degree 9 to 10 training points, all at different xx. What will its training error be?

  1. About the noise level of the data
  2. Zero, which says little about how it will do on new points
  3. Larger than for a straight line
  4. Impossible to say without seeing the data
Show the answer

B. Zero, which says little about how it will do on new points

A degree-9 polynomial has 10 coefficients, enough to pass exactly through any 10 points with distinct xx, so the training error is zero. It has fit the noise along with the signal, and its test error is typically large.

Structure without labels

Often there are no labels at all: customer records, gene measurements, millions of photos. Unsupervised learning looks for structure in the data itself. The best-known task is clustering, and its workhorse is k-means:

  1. Place kk centroids somewhere.
  2. Assign: give each point to its nearest centroid.
  3. Update: move each centroid to the mean of its points.
  4. Repeat until nothing changes.
Think first k-means needs you to choose kk. Without labels, how could you tell whether 3 or 4 clusters is right?

There is no single right answer. The best achievable inertia only falls as kk grows (with one cluster per point it is zero), so you look for an "elbow" where extra clusters stop helping much, check that clusters are stable across restarts, or ask whether they are useful for your purpose. The Hidden Layer covers these tools, along with dimensionality reduction methods such as PCA.

Is the model any good?

Accuracy, the fraction of correct predictions, sounds like the obvious score. It can be badly misleading.

Predict before you look

Of a bank's card payments, 1% are fraudulent. A lazy "model" labels every payment as legitimate. What accuracy does it score?

Show the answer

99 %

It is right on every legitimate payment, 99% of them, so its accuracy is 99%. Yet it catches no fraud at all. When one class is rare, you have to look at the errors on each class separately.

A confusion matrix counts the four outcomes: true and false positives (TP, FP), true and false negatives (TN, FN). Two rates summarize it. Precision asks: of the cases I flagged, how many were real? Recall asks: of the real cases, how many did I flag?

precision=TPTP+FPrecall=TPTP+FN\text{precision} = \frac{TP}{TP + FP} \qquad \text{recall} = \frac{TP}{TP + FN}

Where to go next

Every learning system in this course, up to the largest neural networks, rests on the ideas of this lesson: features and labels, a loss to minimize, honest evaluation on held-out data, and the constant tension between fitting and generalizing.

Hidden Layer lessonWhat you will do there
Supervised LearningDerive least squares and logistic regression, train them by gradient descent, and meet kernels, support vector machines, decision trees and ensembles.
Generalization & OverfittingMake bias and variance precise, regularize with ridge and lasso, validate honestly, and ask why huge networks still generalize.
Unsupervised LearningCluster with k-means and mixture models, find low-dimensional structure with PCA, and estimate densities.
Evaluating ModelsRead ROC and precision–recall curves, check calibration, test significance honestly, and cope with distribution shift.
In the Hidden Layer, each idea gets its math and its algorithms.

Key takeaways

  • Machine learning finds patterns in examples instead of following hand-written rules. Supervised learning uses labels; unsupervised learning finds structure without them.
  • Training minimizes a loss on the training set, but the real goal is generalization: low error on data the model has never seen.
  • Too simple underfits and too flexible overfits. Regularization, more data and validation keep the balance; the test set is used once, at the end.
  • k-means groups unlabeled points by alternating assign and update steps, and the result depends on kk and on where it starts.
  • Accuracy misleads when classes are imbalanced. Confusion matrices, precision, recall and ROC curves show what a classifier really does, and the threshold sets the trade-off.

Check yourself

In the interactive lesson these are scored with your confidence, and what you miss comes back for review.

Question 1

A model has training error 0.01 and test error 0.90. What is happening, and what helps?

  1. Underfitting: use a more flexible model
  2. Nothing is wrong: test error is always higher
  3. Overfitting: regularize, simplify the model or get more data
  4. Training was too short: train for longer
Show the answer

C. Overfitting: regularize, simplify the model or get more data

A huge gap between training and test error is the signature of overfitting: the model has learned the training set's noise. Anything that limits its flexibility, or gives it more data, narrows the gap.

Question 2

You try 50 settings of a model and report the best test-set score among them. Why is that score too optimistic?

  1. You have effectively tuned on the test set, so it no longer measures performance on unseen data
  2. 50 settings are too few to find a good one
  3. It is not: test scores are always pessimistic
  4. The training set was too large
Show the answer

A. You have effectively tuned on the test set, so it no longer measures performance on unseen data

Picking the best of 50 by test score partly selects for luck on those particular test examples. Choose settings with a validation set or cross-validation, and use the test set once, at the end.

Question 3

Which statement about k-means is true?

  1. It always finds the best possible clustering
  2. It needs labeled examples to work
  3. It works out the right number of clusters by itself
  4. Different starting centroids can lead to different final clusterings
Show the answer

D. Different starting centroids can lead to different final clusterings

No step raises the inertia, so k-means always stops, but at a local optimum that depends on the start. Several restarts, or k-means++ initialization, help.

Question 4

A cancer screening test should miss as few real cases as possible, even at the cost of some false alarms. Which metric matters most?

  1. Precision
  2. Recall
  3. Accuracy
  4. The number of negatives predicted
Show the answer

B. Recall

Recall, TP/(TP+FN)TP/(TP+FN), is the fraction of real cases that get flagged, so maximizing it means few missed cancers. Follow-up tests then sort out the false alarms.

Question 5

A k-nearest-neighbors classifier with k=1k = 1 is scored on its own training set, which has no duplicate points. What accuracy does it get?

  1. About 50%
  2. About the same as on the test set
  3. 100%
  4. It depends on the learning rate
Show the answer

C. 100%

Every training point is its own nearest neighbor, so it gets its own label back. Training accuracy is perfect, which is exactly why training accuracy is no guide to quality.

Go deeper

Further reading

  • G. James, D. Witten, T. Hastie & R. Tibshirani, An Introduction to Statistical Learning, 2nd ed. (Springer, 2021).
  • S. Russell & P. Norvig, Artificial Intelligence: A Modern Approach, 4th ed. (Pearson, 2020), chapter 19.