Rationality & Expected Utility
Von Neumann–Morgenstern axioms and what “rational” really means.
Why should an agent maximize expected utility, rather than guard against the worst case or follow a rule of thumb? Von Neumann and Morgenstern gave a startling answer: if its preferences satisfy four innocent-looking axioms, it already behaves as an expected-utility maximizer. This lesson proves the theorem, then probes where people, and practical agents, depart from it.
What you’ll learn
- State the von Neumann–Morgenstern axioms and prove the expected utility theorem, including uniqueness up to positive affine transformations.
- Use a money pump to argue for transitivity, and show which axiom the Allais paradox violates.
- Characterize risk attitudes by the curvature of utility, and derive the Arrow–Pratt approximation to the risk premium.
- Explain why stationary preferences over reward sequences lead to exponential discounting, and predict how discounting changes an optimal policy.
- Distinguish perfect rationality from bounded optimality, and prove that the expected value of information is never negative.
Lotteries and the axioms
Independence says that a component both options share, received with the same probability, should not affect the ranking: if a coin decides between and your choice, only your choice matters. Russell and Norvig list six "constraints" instead (orderability, transitivity, continuity, substitutability, monotonicity, decomposability), a presentation in the same spirit.
Why transitivity? An agent with cyclic strict preferences can be money-pumped. Holding , it pays a cent to swap for , another to swap for , and another to swap for : it is back where it started, three cents poorer, and ready to go round again.
Allais (1953). Choose within each pair (amounts in millions of dollars):
- : 1 for sure, or : 5 with 10%, 1 with 89%, 0 with 1%.
- : 1 with 11%, 0 with 89%, or : 5 with 10%, 0 with 90%.
Most people choose and . Which axiom does that pattern violate?
- Completeness
- Transitivity
- Continuity
- Independence
Show the answer
D. Independence
Let be the lottery "5 with probability , else 0". Then , , and . Each pair differs only in versus ; the shared 89% component is in the first pair and in the second. Independence forces . In expected-utility terms, says , and says the reverse.
The expected utility theorem
Necessity is easy: expected utility is linear in , which gives independence, and continuous in mixing weights, which gives continuity. For sufficiency, pick a best outcome and a worst (finitely many outcomes, complete and transitive preferences). Repeated use of independence shows that every lottery lies between them. Assume (otherwise everything is indifferent) and write for " with probability , else ".
More chance of the best is better. Independence gives for (mix with ). For , write as a mixture of and , and apply independence once more to get :
Calibrate. By continuity each is indifferent to some , and by the previous step that is unique. Call it . Then .
Linearity. Independence lets you replace and by their calibrated equivalents inside a mixture, and a mixture of two 's is again an :
So is affine in mixtures. Writing and setting gives the representation.
Uniqueness. If is another expected-utility representation, apply it to , where is normalized by and :
The theorem constrains consistency, not content. It says nothing about which outcomes an agent should value, only that coherent preferences over gambles act as if a utility were being averaged. Utilities live on an interval scale: "utility 10 versus 5" does not mean twice as good, but ratios of utility differences are meaningful, because they survive every map .
Reinforcement learning has the same freedom to rescale rewards. Potential-based reward shaping (Ng, Harada and Russell, 1999) is a sequential cousin: adding to every reward leaves optimal policies unchanged.
Think first You normalize and , and find you are indifferent between outcome and a 70% chance of (else ). On a new scale where is worth and is worth , what is worth?
, and the new scale is , so ; equivalently . Every positive affine rescaling preserves every choice between lotteries.
Risk attitudes and the utility of money
When outcomes are amounts of money, is an increasing function on . The certainty equivalent of a gamble is the sure amount valued equally, , and the risk premium is . By Jensen's inequality, is concave exactly when for every gamble: concave utility means risk aversion (), linear utility risk neutrality, convex utility risk seeking.
Let with and a small variance . Expand both sides of around :
Solve for the premium. It is proportional to the variance, with the coefficient of absolute risk aversion (Pratt, 1964; Arrow):
Exponential utility has constant absolute risk aversion; logarithmic utility has , so the same gamble matters less to a richer agent.
Think first With , what is the certainty equivalent of a fair coin flip paying 0 or 100 dollars?
, so dollars, and the risk premium is dollars. The Arrow–Pratt approximation is poor here, because the gamble is not small relative to wealth.
Utilities over time
An agent acting over time chooses among sequences of rewards. Suppose its preferences over sequences are stationary: if two sequences begin with the same reward, they are ranked the same way as the sequences with that first reward removed. Koopmans (1960) and later work showed that stationarity, together with separability across periods and mild regularity conditions, forces the additive discounted form
the return of reinforcement learning, with discount factor ( only when every sequence is finite).
Think first Many human choices are described well by hyperbolic discounting, . Why can't a time-consistent agent use it?
Compare 100 dollars on day with 110 dollars on day . At : versus , so you take the 100 now. At : versus , so you plan to wait. But when day 30 arrives, you face the first comparison again and switch. Only exponential discounting makes the ratio independent of , so plans are never reversed (Strotz, 1950s).
In the bridge world below, the start cell has a +1 exit one step to the west and a +10 exit five steps to the east, across a bridge over a chasm. Steps cost nothing and, with the noise at 0, moves always succeed. Below which discount factor does the rational agent at the start stop crossing?
Show the answer
0.56
Going west is worth ; crossing is worth , because the big reward arrives five steps later. Crossing wins iff , that is , or .
Rationality with limited computation
The definition of a rational agent ignores the cost of thinking. Perfect rationality, always outputting the expected-utility-maximizing action at once, is usually out of reach: planning over a finite horizon in a partially observable MDP is PSPACE-complete (Papadimitriou and Tsitsiklis, 1987), and for some infinite-horizon criteria even deciding whether a policy can reach a given value is undecidable (Madani, Hanks and Condon, 2003). Herbert Simon's bounded rationality (1955) replaced optimizing with satisficing. Russell's later distinctions are sharper:
- Calculative rationality: the program would eventually compute the optimal decision for the situation it started in, by which time the world may have moved on.
- Metalevel rationality: treat computations as actions, and deliberate only while the expected improvement in the decision exceeds the cost of the time spent (Russell and Wefald, 1991).
- Bounded optimality (Russell and Subramanian, 1995): among all programs that run on the agent's machine , choose one with the best expected performance in its class of environments .
For any fixed action , the law of total expectation and the definition of a maximum give
Take the maximum over on the left. Equality holds iff a single action attains the inner maximum for almost every observation, i.e. the information never changes the decision. The value of information is the convexity of the maximum.
The hypotheses matter. Information that is costly or arrives late can lower expected utility, and in games even free information can hurt: a player known to be informed may be exploited, or lose the ability to commit credibly. Those are exactly the settings in which agents interact with other agents.
Key takeaways
- Von Neumann–Morgenstern: preferences over lotteries that are complete, transitive, continuous and independent are exactly those that maximize expected utility, with unique up to , .
- Intransitive agents can be money-pumped; the Allais pattern violates independence. Expected utility is a normative standard, not a description of people.
- Concave utility means risk aversion, with risk premium ; the St. Petersburg game shows why utilities over unbounded gambles must be bounded.
- Stationary, separable preferences over reward sequences force exponential discounting; encodes time preference and can flip optimal policies (on the bridge, at ).
- Perfect rationality is intractable; bounded optimality optimizes over feasible programs; free information never lowers a single decision maker's expected utility.
Check yourself
In the interactive lesson these are scored with your confidence, and what you miss comes back for review.
is a von Neumann–Morgenstern utility for an agent's preferences. Which function is guaranteed to represent the same preferences over lotteries?
Show the answer
C.
Only positive affine transformations preserve every expected-utility comparison. Monotone but nonlinear maps such as or preserve the ranking of sure outcomes but can reverse comparisons between gambles.
An agent with concave utility of money is offered a gamble, or the gamble's expected value for sure. What does it choose?
- The gamble
- The sure amount (or it is indifferent)
- It is always indifferent
- It depends only on the gamble's variance
Show the answer
B. The sure amount (or it is indifferent)
Jensen's inequality: for concave , so the sure expected value is weakly preferred and the risk premium is non-negative. Variance alone determines the premium only approximately, for small gambles.
Which discount function guarantees that an agent never reverses a plan merely because time has passed?
- for steps from now, else
Show the answer
A.
Only with exponential discounting is the relative weight of versus , , the same at every . That stationarity is what makes plans time-consistent.
What distinguishes bounded optimality from perfect rationality?
- It accepts any action that is good enough, without optimizing
- It allows the agent to violate the von Neumann–Morgenstern axioms
- It applies only to deterministic environments
- It optimizes over the programs that can run on the agent's actual machine, so the cost of deliberation counts
Show the answer
D. It optimizes over the programs that can run on the agent's actual machine, so the cost of deliberation counts
Perfect rationality demands the optimal action regardless of computation. Bounded optimality asks for the best program given the machine and the class of environments, which always exists.
Further reading
- J. von Neumann & O. Morgenstern, Theory of Games and Economic Behavior, 2nd ed. (Princeton University Press, 1947).
- M. Allais, "Le comportement de l'homme rationnel devant le risque: critique des postulats et axiomes de l'école américaine," Econometrica 21, 503–546 (1953).
- T. C. Koopmans, "Stationary ordinal utility and impatience," Econometrica 28, 287–309 (1960).
- S. J. Russell & D. Subramanian, "Provably bounded-optimal agents," Journal of Artificial Intelligence Research 2, 575–609 (1995).
- S. Russell & P. Norvig, Artificial Intelligence: A Modern Approach, 4th ed. (Pearson, 2020), chapters 16–17.