Skip to main content

Reinforcement Learning: An Introduction — Sutton & Barto

Review of Sutton & Barto’s RL textbook: the foundational treatment of reinforcement learning, its pedagogical approach, and how it holds up in the era of RLHF and large-scale RL.

finished certainty: likely importance: 7

Reinforcement Learning: An Introduction (1st ed. 199828ya; 2nd ed. 2018) by Richard Sutton and Andrew Barto is universally known as “the RL book” or “Sutton & Barto.” It is the standard introductory textbook for reinforcement learning, and it has held this position for over two decades. The second edition, freely available as a PDF from Sutton’s website, is the version you should read.

What It Covers

The book proceeds through RL from first principles. Part I (chapters 1–8) covers tabular methods: multi-armed bandits, finite Markov decision processes (MDPs), dynamic programming, Monte Carlo methods, temporal-difference learning (TD), n-step bootstrapping, and planning with tabular models. Part II (chapters 9–13) covers approximate solution methods: on-policy and off-policy approximation with function approximation, eligibility traces, and policy gradient methods. Part III (chapters 14–17) looks at frontiers: psychology connections, neuroscience, applications, and the “AI perspective.”

The pedagogical approach is distinctive and effective. Sutton and Barto build everything from the ground up, starting with the simplest possible settings (bandits, small gridworlds) and gradually adding complexity. Each concept is motivated by its limitations: tabular methods can’t scale, so we need function approximation; value-based methods struggle with continuous actions, so we need policy gradients. This makes the book feel like a guided intellectual journey rather than a catalog of algorithms.1

What’s Great

The treatment of temporal-difference learning (chapters 6–7) is the book’s crown jewel. TD learning is Sutton’s most important contribution to the field—the idea that you can learn from incomplete episodes by bootstrapping from your own estimates is both simple and profound. The book develops this idea with extraordinary care, from TD(0) through the full TD(λ) spectrum, making clear exactly what bootstrapping buys you and what it costs.2

The multi-armed bandit chapter (chapter 2) is the best introduction to the exploration-exploitation tradeoff that I know of. It covers ε-greedy, UCB, gradient bandits, and Thompson sampling in a compact, clear treatment that makes the fundamental tension vivid. This chapter alone is worth reading even if you never touch the rest of the book—the exploration-exploitation tradeoff appears everywhere in decision-making, not just in RL.3

The dynamic programming chapter (chapter 4) does an excellent job of connecting RL to classical optimal control theory. Policy iteration and value iteration are derived cleanly, and the relationship between the Bellman equations and the RL update rules is made explicit. This bridges two communities—control theory and machine learning—that often talk past each other.

Chapter 13 on policy gradient methods is a significant addition in the second edition and covers REINFORCE, actor-critic methods, and the policy gradient theorem. Given that policy gradients are the foundation of modern RL applications (including RLHF), this chapter is essential. The derivation of the policy gradient theorem is particularly clean.4

What’s Missing or Dated

The elephant in the room is deep RL. The book covers function approximation in chapters 9–11, but primarily with linear methods—tile coding, radial basis functions, and the like. There is a brief mention of neural network function approximators, but DQN (Mnih et al. 201511ya), AlphaGo, PPO, and the deep RL revolution that made reinforcement learning famous are not covered in any depth.5 This is a significant gap: most RL work in 2026 uses deep neural networks as function approximators, and the specific challenges of deep RL—instability, sample inefficiency, reward hacking—are largely absent.

RLHF (reinforcement learning from human feedback), which has become enormously important through its role in training ChatGPT and other large language models, is not covered at all. This is unsurprising—RLHF’s prominence postdates the second edition—but it means the book doesn’t address what is arguably the most impactful application of RL ideas in practice.

The book also says relatively little about model-based RL beyond tabular Dyna (chapter 8). Modern model-based methods—world models, MuZero, decision transformers—are absent. The multi-agent RL setting, which has become important for game-playing AI and real-world applications, gets only a brief mention.

Reading Guide and Practical Recommendations

  • Everyone in ML should read: Chapters 1–3 (bandits, MDPs), Chapter 6 (TD learning). These are foundational concepts that appear throughout ML and decision theory.

  • Aspiring RL researchers: Read the whole book front-to-back. It is well-sequenced and builds cumulatively. Don’t skip Part I even if you think tabular methods are “too simple”—the intuitions developed there carry directly into the deep RL setting.

  • Practitioners who need to use RL: Read chapters 1–7 and 13, then supplement with Spinning Up in Deep RL (OpenAI) for practical deep RL implementations. The book gives you the theory; Spinning Up gives you the code.

  • People interested in RLHF specifically: Read chapters 1–3 and 13 (policy gradients), then go directly to the RLHF literature. The core RL concepts in the book map directly onto the RLHF pipeline.

  • Supplement with: David Silver’s RL course (YouTube lectures), Spinning Up, and Lilian Weng’s RL blog posts.

How It Holds Up

Sutton & Barto occupies an unusual position: it is simultaneously foundational and incomplete. The tabular theory it develops is clean, beautiful, and permanent—the Bellman equations, TD learning, and the policy gradient theorem are not going to be superseded. The intuitions it builds are exactly the right ones. But the practice of RL has moved so far beyond what the book covers that reading it alone would leave you unprepared for actual RL research or engineering in 2026.6

This is the fate of any textbook in a fast-moving field, and it is less a criticism than a statement of fact. The book is aware of its limitations—Sutton’s research page and blog (including the famous “Bitter Lesson”) make clear that he views scaling compute and learning-based approaches as the future, even though the book focuses on the conceptual foundations.

Rating: 7/10. The right first book on reinforcement learning. It builds the intuitions and mathematical framework that everything else rests on. Just don’t stop here—the gap between what the book covers and modern practice is large and growing.7

Further Reading