Perception & Generative AI
Seeing, hearing and creating images, sound and video.
To you, a photo of a cat is a cat. To a computer it is a grid of numbers, and getting from one to the other took AI decades. This lesson shows how networks learn to see, how easily they can be fooled, and how the same tools now create images and sound.
What you’ll learn
- Describe how images are represented as numbers and how a convolution detects local patterns.
- Explain why CNNs stack learned convolutions, and what changed with ImageNet in 2012.
- Explain how a tiny adversarial perturbation can flip a classifier's prediction.
- Describe how a diffusion model generates data, and how multimodal models connect images and text.
A picture is a grid of numbers
A camera hands the computer only numbers: one brightness value per pixel for a grayscale image, three per pixel (red, green, blue) for color. Computer vision is the job of turning those numbers into descriptions: what is in the picture, and where.
Seeing feels effortless to us, which hides how hard it is. Hans Moravec observed in 1988 that skills we find easy, like perception and movement, are often the hardest to build into machines.
How many numbers does a computer receive for a 100 × 100 pixel color photo?
Show the answer
30000
pixels color channels numbers, and a phone photo has tens of millions. Shift the cat a few pixels and almost every number changes, yet it is the same cat. That is why rules written by hand never worked and learning from examples did.
Convolution: a sliding pattern detector
The basic tool is the convolution. A small grid of weights, the kernel, slides across the image. At each position it multiplies the pixels under it by its weights and adds them up, so the output is large wherever the patch matches the kernel's pattern. The result is a feature map: where in the image that pattern occurs.
A kernel slides over a 1,000 × 1,000 grayscale image. How many weights does this kernel have to learn?
- 1,000,000: one per pixel
- 9
- 9,000,000: nine per pixel
- It depends on what the image shows
Show the answer
B. 9
The same 9 weights (plus, usually, one bias) are reused at every position. A fully connected layer from a million pixels to a million outputs would need a trillion weights. Weight sharing also means a pattern learned in one place is found everywhere.
A convolutional neural network learns its kernels by backpropagation instead of using hand-designed ones like Sobel. Stacking layers, with pooling to shrink the maps in between, lets each layer combine the previous layer's features over a growing receptive field: edges, then textures, then parts, then objects. Beyond labeling whole images, networks draw boxes around objects (object detection) and label every pixel (segmentation).
Fooled by invisible changes
Here is a surprise. In 2013 researchers found that adding a carefully chosen, nearly invisible pattern to an image can make an accurate classifier output a confidently wrong label. These are adversarial examples. The fast gradient sign method (Goodfellow et al., 2015) builds one in a single step: nudge every pixel by a tiny amount in whichever direction increases the model's loss.
Why does the FGSM perturbation flip the prediction when random noise of the same size usually does not?
- The FGSM perturbation is larger
- The model filters random noise out before classifying
- Each pixel moves the way that hurts the model most, so thousands of tiny effects add up instead of canceling
- FGSM secretly redraws the digit as the other class
Show the answer
C. Each pixel moves the way that hurts the model most, so thousands of tiny effects add up instead of canceling
For a roughly linear model the score changes by the sum of weight × pixel change. FGSM matches every change's sign to its pixel's influence, so all the contributions add, about . Random signs mostly cancel. In high dimensions, many tiny aligned changes make a big difference.
Learning to create
Recognition maps an image to a label. A generative model goes the other way: it learns the distribution of the data well enough to produce new samples. Three families stand out: variational autoencoders; GANs, in which a generator learns to fool a discriminator; and diffusion models, behind many of the best-known image generators as of the mid-2020s.
Diffusion training is surprisingly simple. Take real data, add noise step by step until only random static is left, and train a network to undo one small step of noise. To generate, start from pure noise and apply the denoiser again and again.
Think first Every generated sample starts as random noise. Why don't all samples come out the same?
Different starting noise leads to different end points: the learned denoiser maps the whole cloud of noise onto the whole data distribution. That randomness (plus fresh noise added at each step in many samplers) is where the variety comes from.
Hearing, and connecting the senses
Sound yields to the same tools. A spectrogram turns audio into a picture of frequency against time, so speech recognition can borrow from vision and language models, and speech synthesis can generate audio much as image models generate pictures.
Multimodal models connect the senses. CLIP (Radford et al., 2021) trained an image encoder and a text encoder on hundreds of millions of image–caption pairs so that matching pairs land close together in one shared space (contrastive learning). That space lets a model sort images into categories it was never explicitly trained on, and it helps text-to-image systems steer toward a prompt.
In the Hidden Layer you'll study each area: Computer Vision (images as tensors, filters, recognition and detection), Generative Models (VAEs, GANs and diffusion), Speech & Audio (recognizing and synthesizing sound), and Multimodal Models that connect text, images, audio and video.
Key takeaways
- Images are grids of numbers, and the same object can produce very different numbers, which is why vision has to be learned.
- A convolution slides a small kernel over an image; CNNs learn their kernels and stack them to find edges, textures, parts and objects.
- Adversarial examples flip predictions with tiny, carefully aligned changes; full robustness is still an open problem.
- Generative models learn the data distribution; diffusion models create samples by denoising random noise step by step.
- Spectrograms let audio reuse vision tools, and contrastive models like CLIP put images and text in one shared space.
Check yourself
In the interactive lesson these are scored with your confidence, and what you miss comes back for review.
A kernel has weights in every row. What does its feature map highlight?
- Regions of uniform brightness
- Horizontal edges, where brightness changes from top to bottom
- The darkest pixels in the image
- Vertical edges, where brightness changes from left to right
Show the answer
D. Vertical edges, where brightness changes from left to right
On a uniform patch the s and s cancel to zero. The output is large only where the right column differs from the left: a vertical edge. Sobel-x is a refined version of this kernel.
What did AlexNet's 2012 ImageNet win mainly demonstrate?
- That deep CNNs trained on big labeled datasets with GPUs could far outperform hand-engineered vision systems
- That convolution had just been invented
- That vision no longer needs labeled data
- That networks see images the way humans do
Show the answer
A. That deep CNNs trained on big labeled datasets with GPUs could far outperform hand-engineered vision systems
CNNs date back to the 1980s. What was new was the combination of ImageNet-scale data, GPU training and tricks like ReLU and dropout, which beat the best hand-designed features by a wide margin.
How does a diffusion model generate a new image?
- It retrieves the closest image from its training set
- It paints the image one pixel at a time, left to right
- It starts from random noise and repeatedly applies a learned denoising step
- A discriminator edits a random training image until it looks new
Show the answer
C. It starts from random noise and repeatedly applies a learned denoising step
Training teaches a network to remove a little noise at a time; generation starts from pure noise and applies that step over and over until a sample emerges. Different starting noise gives different images.
CLIP was trained on hundreds of millions of image–caption pairs. What does that training produce?
- A model that writes captions word by word
- An image encoder and a text encoder whose outputs land close together for matching pairs, in one shared space
- A generator that turns captions into images
- A database that stores every caption with its image
Show the answer
B. An image encoder and a text encoder whose outputs land close together for matching pairs, in one shared space
Contrastive training pulls each image's vector toward its own caption's and away from the other captions. In the shared space you can classify an image by finding which description ("a photo of a dog") lies closest, with no task-specific training.
Further reading
- A. Krizhevsky, I. Sutskever & G. E. Hinton, "ImageNet Classification with Deep Convolutional Neural Networks" (NeurIPS, 2012).
- J. Ho, A. Jain & P. Abbeel, "Denoising Diffusion Probabilistic Models" (NeurIPS, 2020).