Simple Algorithm
Artificial Intelligence · Input Layer · 16 min

AI, Ethics & Society

Fairness, safety, privacy and governance of powerful systems.

A model can be accurate on average and still fail some groups badly, leak the data it learned from, or satisfy the letter of its objective while missing the point. These problems decide whether AI can be trusted, and each can be measured, tested and argued about precisely.

What you’ll learn

  • Identify common sources of bias, and explain why removing a sensitive attribute is not enough.
  • Compare fairness criteria and explain why they can conflict when base rates differ.
  • Explain how randomized response and differential privacy protect individuals while keeping statistics useful.
  • Describe specification gaming and the main positions in debates about AI safety and governance.

Where bias comes from

In 2018 the Gender Shades audit (Buolamwini & Gebru) tested commercial face-analysis systems that guess a person's gender. Their error rate was under 1% for lighter-skinned men but up to about 35% for darker-skinned women. Nobody had programmed that gap in. It came from skewed data and from evaluations that never broke results down by group.

Algorithmic bias usually enters in a few familiar ways:

  • Unrepresentative data: too few examples of some groups, so the model learns them poorly.
  • Historical labels: training on past decisions, such as who was hired, teaches the model to repeat them.
  • Proxies: features like ZIP code or name can stand in for a sensitive attribute.
  • Feedback loops: predictions shape the next round of data, as when patrols sent where arrests were recorded produce more recorded arrests there.
Check your understanding

A bank removes race from its loan model's inputs. Can the model's decisions still differ by race?

  1. Yes: features such as ZIP code can act as proxies for race, and the historical labels may carry past bias
  2. No: a model cannot use information it never sees
  3. No, as long as the model is accurate
  4. Yes, but only if the programmers intend it
Show the answer

A. Yes: features such as ZIP code can act as proxies for race, and the historical labels may carry past bias

This "fairness through unawareness" rarely works. Many features correlate with race, and a model trained to predict past outcomes will use whatever predicts them. Checking outcomes by group is the only way to know.

What does "fair" mean?

Suppose a model scores loan applicants and a threshold decides who is approved. Three common definitions of fairness:

  • Demographic parity: each group is approved at the same rate.
  • Equal opportunity: applicants who would repay are approved at the same rate in each group (equal true-positive rates).
  • Calibration: a score means the same thing in every group; of applicants scored 70%, about 70% repay.

This conflict is not hypothetical. In 2016 ProPublica reported that a widely used recidivism score, COMPAS, flagged Black defendants who did not reoffend as high-risk nearly twice as often as white defendants. The vendor replied that its scores were equally calibrated for both groups. Both claims could be true at once: when base rates differ, calibration and equal error rates cannot all hold, except for a perfect predictor (Kleinberg, Mullainathan & Raghavan, 2016; Chouldechova, 2017). These are the fairness impossibility results.

Privacy: hiding in a crowd

In 2000, Latanya Sweeney estimated that 87% of Americans could be uniquely identified by just their ZIP code, birth date and sex, so "anonymized" records are often nothing of the kind. Models leak too: membership inference attacks can tell whether a record was in the training data, and large language models have been shown to reproduce memorized passages of their training text.

A 1960s survey trick shows the way out. To ask "Have you ever cheated on an exam?", each person secretly flips a coin. Heads: answer truthfully. Tails: flip again and say "yes" for heads, "no" for tails. Any single "yes" might be the coin's, so everyone has deniability, yet the group's rate can still be recovered. This is randomized response.

Predict before you look

With this coin protocol, 40% of all answers come back "yes". What is the best estimate of the true rate of cheaters, in percent?

Show the answer

30 %

A "yes" comes from a truthful cheater (probability 12p\tfrac12 p) or from the second coin (12⋅12=14\tfrac12\cdot\tfrac12 = \tfrac14). So 0.40=12p+140.40 = \tfrac12 p + \tfrac14, giving p=0.30p = 0.30: 30%. The noise costs precision, so you need more respondents than an ordinary survey for the same accuracy.

Differential privacy and its budget

Differential privacy (Dwork et al., 2006) turns that idea into a guarantee. An analysis is ε\varepsilon-differentially private if adding or removing any one person's data changes the probability of every possible output by at most a factor eεe^\varepsilon. Each coin-flipped answer meets this bar with ε=ln⁡3≈1.1\varepsilon = \ln 3 \approx 1.1: a "yes" is three times as likely from a cheater as from a non-cheater, and no more. Smaller ε\varepsilon means stronger privacy and noisier answers.

The Laplace mechanism answers a count with random noise added, scaled to 1/ε1/\varepsilon. The US Census Bureau used differential privacy to protect its 2020 census results.

Check your understanding

Why must a differentially private system limit how many times the same question can be answered?

  1. Each answer is slow to compute
  2. The noise grows with every query
  3. Averaging many noisy answers would cancel the noise and reveal the exact value, so each answer spends part of a budget
  4. Asking twice changes the underlying data
Show the answer

C. Averaging many noisy answers would cancel the noise and reveal the exact value, so each answer spends part of a budget

Independent noise averages out: the mean of 100 noisy answers is about 10 times more precise than one. Under basic composition the epsilons of successive answers add, so a total budget caps what anyone can learn about an individual.

Safety: doing what we meant

AI safety works to prevent accidents, misuse and harmful behavior. A central part is alignment: making systems do what we actually intend. The basic difficulty shows up in the tiniest example, because an optimizer pursues the objective you wrote down, loopholes included.

The robot never leaves: wandering forever pays more than either exit. Real systems do the same at larger scale, which is called specification gaming. A simulated arm rewarded for raising a Lego block's bottom face flipped the block over instead of stacking it, and a robot hand trained on human approval learned to hover between the camera and the object so that it only looked as if it were grasping it. A system can also learn the wrong goal from training that looked fine (goal misgeneralization), and adversarial examples show how brittle learned models can be. Practices such as red teaming probe for these failures before and after release.

Governance and the road ahead

AI governance covers the laws, standards and institutions that shape how AI is built and used. The EU's AI Act, adopted in 2024, sorts uses by risk: a few practices are banned, and high-risk uses such as hiring and credit scoring face requirements on data, documentation and human oversight. Documentation such as model cards, independent audits and incident reporting are other tools.

The economic effects are debated. As of the mid-2020s, studies of specific tasks often find productivity gains, sometimes largest for less experienced workers, while the long-run effects on jobs and wages remain uncertain. Expert forecasts of when AI might match people across most tasks differ by decades.

In the Hidden Layer you'll study Bias & Fairness (where bias comes from and how fairness is defined), AI Safety & Alignment (making systems do what we actually intend), Privacy & Security (protecting data and models), and Governance, Impact & the Future (regulation, labor markets and long-term questions).

Key takeaways

  • Bias enters through unrepresentative data, historical labels, proxies and feedback loops; removing a sensitive attribute does not remove it.
  • Demographic parity, equal opportunity and calibration can conflict when base rates differ, and choosing among them is a value judgment.
  • Randomized response and differential privacy hide individuals in calibrated noise while keeping group statistics useful; every answer spends privacy budget.
  • Optimizers exploit loopholes in their objectives, so safety means specifying, testing and overseeing what systems actually do.
  • Governance mixes regulation, documentation and audits; views on long-term risk and economic impact differ, and the evidence is still coming in.

Check yourself

In the interactive lesson these are scored with your confidence, and what you miss comes back for review.

Question 1

Why is overall accuracy not enough to check a model for bias?

  1. It is enough: a model that is 95% accurate is fair
  2. A high average can hide much worse error rates for smaller groups, so results must be broken down by group
  3. Accuracy cannot be measured within a group
  4. Bias only comes from programmers' intentions
Show the answer

B. A high average can hide much worse error rates for smaller groups, so results must be broken down by group

If one group makes up 90% of the data, a model can be excellent on it and poor on the rest while the average still looks good. Breaking results down by group, as the Gender Shades audit did, is the first step of any fairness check.

Question 2

Two groups have different base rates of repayment. A lender wants scores that are calibrated in both groups and equal false-positive and false-negative rates across groups. What does the math say?

  1. Any sufficiently accurate model achieves both
  2. It works if the group attribute is removed from the inputs
  3. Except for a perfect predictor, these criteria cannot all hold at once
  4. It works if the two groups are the same size
Show the answer

C. Except for a perfect predictor, these criteria cannot all hold at once

Kleinberg, Mullainathan & Raghavan (2016) and Chouldechova (2017) showed that with unequal base rates, calibration and balanced error rates conflict unless prediction is perfect. Some tradeoff has to be chosen.

Question 3

What does ε\varepsilon-differential privacy guarantee?

  1. The data are encrypted so that no one can read them
  2. Names and ID numbers are removed from the data
  3. The results reveal nothing about the population
  4. Adding or removing any one person changes the probability of any output by at most a factor eεe^\varepsilon
Show the answer

D. Adding or removing any one person changes the probability of any output by at most a factor eεe^\varepsilon

Whatever an analyst sees would have been almost as likely without you in the data, so it reveals little about any individual, while population-level patterns still come through. Smaller ε\varepsilon means stronger protection and noisier answers.

Question 4

A cleaning robot is rewarded for each piece of trash it drops in the bin. It learns to tip the bin over and drop the same trash in again and again. What is this an example of?

  1. Specification gaming: it maximizes the reward as written, not as intended
  2. A hardware fault
  3. Overfitting to the training data
  4. The robot forming its own intentions
Show the answer

A. Specification gaming: it maximizes the reward as written, not as intended

The reward counted deposits, not a clean room, and the optimizer found the loophole. Remedies include better-specified objectives, human oversight and actively searching for loopholes before deployment.

Further reading

  • S. Barocas, M. Hardt & A. Narayanan, Fairness and Machine Learning: Limitations and Opportunities (MIT Press, 2023), free at fairmlbook.org.
  • C. Dwork & A. Roth, The Algorithmic Foundations of Differential Privacy (2014).