Intelligent Agents & Environments
Percepts, actions, rationality and the PEAS framework.
A chess engine, a spam filter and a Mars rover all compute the same kind of object: a function from everything they have perceived to what they do next. This lesson makes that object precise, classifies the environments it must cope with, builds the four classic agent designs, and turns "rational" into something you can calculate.
What you’ll learn
- Define agent functions and agent programs, and show by a counting argument why table-driven agents are hopeless.
- Specify a task environment with PEAS and classify it along the seven standard dimensions.
- Write pseudocode for reflex, model-based, goal-based and utility-based agents, and derive the belief update a model-based agent needs.
- State the definition of a rational agent and apply the principle of maximum expected utility.
- Explain why a rational agent in an unknown environment must explore, and how a learning agent is organized.
Agent functions and agent programs
Fix a set of possible percepts and a set of actions . A percept sequence is everything the agent has perceived so far, . An agent's behavior is completely described by its agent function, which maps every finite percept sequence to an action:
The agent function is an abstract mathematical object. What actually runs is an agent program on a physical architecture: agent = architecture + program. The program receives only the current percept, so if the right action depends on the past, the program must remember it.
function TABLE-DRIVEN-AGENT(percept) returns an action
persistent: percepts, the percept sequence so far (initially empty)
table, an action for every possible percept sequence
append percept to percepts
return table[percepts]The most literal implementation of an agent function: look the whole history up in a table. Correct by construction, and hopeless in practice.
An agent that acts for steps needs one entry per percept sequence of length :
Take one video camera: 30 frames per second, pixels, 24 bits of color each. The percept stream, in bits per second, is
An hour of driving is about bits, and every distinct bit string is a distinct percept sequence:
The observable universe holds roughly atoms. The design problem of AI is therefore to produce rational behavior from a small program, the way a few lines of Newton's method replaced printed tables of square roots.
An agent has two possible percepts and acts for 10 steps. How many entries does its lookup table need?
Show the answer
C.
One entry per sequence of length : . The longest sequences dominate: half of all entries are for length 10 alone.
Task environments
A task environment is specified by PEAS: the performance measure, the environment, the actuators and the sensors. The performance measure scores environment histories, not the agent's opinion of itself: a vacuum robot is judged by how clean the floor is, not by how hard it thinks it worked.
Seven properties of the environment then decide which techniques can work.
| Property | Question | The harder case |
|---|---|---|
| Observable | Do the sensors reveal the whole state that matters for the decision? | Partially observable (poker, driving) |
| Agents | Is anyone else's behavior best described as maximizing a measure that depends on yours? | Multiagent: competitive (chess) or cooperative (cars sharing a road) |
| Deterministic | Is the next state fixed by the current state and action? | Nondeterministic; stochastic when outcomes have probabilities |
| Episodic | Does each decision stand alone, with no effect on later ones? | Sequential: actions have long-term consequences |
| Static | Does the world stand still while the agent deliberates? | Dynamic (semidynamic if only the score changes with time, as in chess with a clock) |
| Discrete | Are states, time, percepts and actions discrete? | Continuous (steering angles, sensor voltages) |
| Known | Does the agent know the rules: what its actions do, and with what probabilities? | Unknown: it must learn how the world works |
| Task | Observable | Agents | Outcomes | Episodic? | Static? | Discrete? |
|---|---|---|---|---|---|---|
| Crossword | Fully | Single | Deterministic | Sequential | Static | Discrete |
| Chess with a clock | Fully | Multi | Deterministic | Sequential | Semi | Discrete |
| Poker | Partially | Multi | Stochastic | Sequential | Static | Discrete |
| Taxi driving | Partially | Multi | Stochastic | Sequential | Dynamic | Continuous |
| Medical diagnosis | Partially | Single | Stochastic | Sequential | Dynamic | Continuous |
| Picking parts off a conveyor | Partially | Single | Stochastic | Episodic | Dynamic | Continuous |
A program learns to play a new video game from the raw screen, without being told the rules. The screen shows the entire game state. How should you classify this environment?
- Partially observable and unknown
- Fully observable but unknown
- Fully observable and known
- Unknown, and therefore partially observable
Show the answer
B. Fully observable but unknown
Observability is about what the sensors reveal now; knownness is about whether the agent knows the rules, i.e. what its actions do. Here the agent sees everything but must learn the dynamics. Solitaire is the mirror image: the rules are known, but face-down cards make it partially observable.
Reflex agents and internal state
The simplest program ignores history. A simple reflex agent applies condition–action rules to the current percept: "if the car in front is braking, then brake". It is correct only when the right action is a function of the current percept, which in practice means a fully observable environment.
Without that, reflexes can trap an agent. A vacuum robot with no location sensor that perceives only "clean" must always do the same thing on a clean square; if that action is Left and it is already in the leftmost square, it pushes against the wall forever. Randomizing the action breaks such loops, at a price. The better fix is memory.
A model-based agent keeps an internal state, a summary of the history, updated with two models: a transition model (how the world evolves and what the agent's actions do) and a sensor model (how states show up in percepts).
function MODEL-BASED-AGENT(percept) returns an action
persistent: belief, the current estimate of the world state
transition_model, sensor_model, rules
action, the most recent action (initially none)
belief ← UPDATE(belief, action, percept, transition_model, sensor_model)
action ← RULE-MATCH(belief, rules).action
return actionThe program stores a compact belief instead of the whole percept sequence. Under uncertainty the belief is a probability distribution over states.
Let the belief be the distribution of the current state given all percepts and actions so far:
Apply Bayes' rule to the newest percept, with a normalizing constant:
Sensor Markov assumption: given the current state, the percept is independent of the past, so the first factor is the sensor model . For the second factor, sum over the previous state and use the transition Markov assumption (the Markov property): depends only on and .
Together: predict with the transition model, update with the sensor model, normalize. This is filtering, and for a hidden Markov model it is the forward algorithm. Each step costs in general, and the agent never stores more than one distribution.
Goals, utilities and rationality
Knowing the state is not enough; the agent must also know what it is trying to achieve. A goal-based agent asks "what will happen if I do this, and does it reach the goal?", which is the job of search and planning. Goals are binary, though: they cannot say that one route is faster, safer or cheaper than another.
A utility-based agent ranks outcomes with a utility function and, when outcomes are uncertain, picks the action with the highest expected utility:
Change one thing in the vacuum world: each move now costs one point. Why is the reflex agent no longer rational?
- Once both squares are clean it keeps moving, and its percept (say "in A, clean") cannot tell it whether B is clean, so no reflex rule knows when to stop
- It should never move at all, since every move now costs a point and the other square may already be clean
- It should suck on every step, because sucking is now the only action that cannot lose points
- A rational agent must know in advance where the dirt is, and the reflex agent does not
Show the answer
A. Once both squares are clean it keeps moving, and its percept (say "in A, clean") cannot tell it whether B is clean, so no reflex rule knows when to stop
The best behavior is now: clean, check the other square once, then stop. Stopping requires remembering that the other square is already clean, but a reflex agent sees "in A, clean" both before and after visiting B. It needs internal state, a model-based design. The world did not change; only the performance measure did.
Learning agents and unknown environments
In an unknown environment no designer can supply the right program in advance, so the agent must improve itself. Turing proposed exactly this in 1950: build a "child machine" and educate it. A learning agent has four parts:
- the performance element, any of the designs above, which chooses actions;
- the critic, which scores behavior against a fixed performance standard (the agent must not be able to lower the bar in order to pass it);
- the learning element, which uses the critic's feedback to improve the performance element;
- the problem generator, which proposes exploratory actions that may be worse now but teach the agent something.
The problem generator embodies a point that surprises people: rationality includes gathering information. An agent that always takes the action that looks best so far can lock onto a mediocre option whose rivals were merely unlucky early on.
The cleanest model is the multi-armed bandit: slot machines with unknown payout rates. UCB1 adds an optimism bonus to each arm's average reward , after pulls of that arm in rounds:
The bonus shrinks as an arm is tried, so every arm keeps being sampled for as long as it might still be the best.
Key takeaways
- An agent is a function from percept sequences to actions, implemented by an agent program. Tables are hopeless: one hour of video already gives about percept sequences.
- PEAS plus seven dimensions (observable, agents, deterministic, episodic, static, discrete, known) determine which methods apply; observability and knownness are independent.
- Reflex agents suffice only when the current percept determines the right action. Model-based agents keep a belief, updated by the Bayes filter: predict with the transition model, update with the sensor model.
- Goal-based agents plan; utility-based agents maximize expected utility. A rational agent maximizes expected performance given its percepts and knowledge, so what is rational depends on the performance measure.
- In unknown environments rationality includes exploration: learning agents add a critic, a learning element and a problem generator.
Check yourself
In the interactive lesson these are scored with your confidence, and what you miss comes back for review.
Two agents run completely different programs, but for every possible percept sequence they choose the same action. Which statement is true?
- They are the same program written in two languages
- They implement the same agent function, so under any performance measure they are equally rational
- They must run on the same architecture
- At least one of them must be table-driven
Show the answer
B. They implement the same agent function, so under any performance measure they are equally rational
Rationality is a property of behavior, that is, of the agent function. Programs can differ in speed, memory and elegance while realizing the same function.
A robot is equally likely to be at a door or at a wall. Its sensor says "door" with probability 0.9 at a door and 0.1 at a wall. It reads "door". What is its belief that it is at a door?
Show the answer
C.
Update and normalize: . With a uniform prior, the posterior is the normalized likelihood.
Why does a model-based agent need a transition model and not just a sensor model?
- To predict how the state changed since the last percept, including the effect of its own action, before weighing the new percept
- Because a sensor model is only defined for fully observable environments
- To learn the rules of the environment from scratch
- To choose the action with the highest utility
Show the answer
A. To predict how the state changed since the last percept, including the effect of its own action, before weighing the new percept
The belief update is predict-then-correct: carries the old belief forward through the action, and only then does reweight it. Without the prediction, the corridor robot could not use the fact that it moved.
In an unknown environment, an agent always takes the action with the best average reward so far. Why can this be irrational?
- The average is the wrong statistic; it should use the median reward
- Rational agents must randomize all of their actions
- It is rational, because it maximizes expected reward under its current estimates
- It may never retry actions whose estimates are low only because of early bad luck, and so lose reward forever; information has value
Show the answer
D. It may never retry actions whose estimates are low only because of early bad luck, and so lose reward forever; information has value
Maximizing expected performance over the agent's lifetime includes the value of what exploration teaches. Greedy behavior can incur regret that grows linearly with time, while UCB1 keeps it logarithmic.
Go deeper
- Rationality & Expected Utility · Von Neumann–Morgenstern axioms and what “rational” really means.
- Measuring Intelligence & AGI · Benchmarks, the ARC challenge and formal definitions of intelligence.
- Minds, Machines & Meaning · The Turing test, the Chinese room and the symbol grounding problem.
Further reading
- S. Russell & P. Norvig, Artificial Intelligence: A Modern Approach, 4th ed. (Pearson, 2020), chapter 2 (agents), chapter 14 (filtering) and chapter 16 (utility).
- S. Thrun, W. Burgard & D. Fox, Probabilistic Robotics (MIT Press, 2005), chapter 2 (the Bayes filter).
- P. Auer, N. Cesa-Bianchi & P. Fischer, "Finite-time analysis of the multiarmed bandit problem," Machine Learning 47, 235–256 (2002).