Reinforcement Learning
How agents learn from rewards and punishments through trial and error: Markov decision processes, value functions, and policy learning.
Reinforcement learning (RL) is the branch of machine learning in which an agent learns what to do — which actions to take in which situations — by interacting with an environment and receiving rewards or penalties. Unlike supervised learning, RL has no labeled dataset of correct answers: the agent must discover good behavior through trial and error, and rewards may arrive long after the actions that caused them. This makes RL the natural framework for sequential decision problems: games, robotics, navigation, recommendation, and resource allocation.
The central dilemma is exploration versus exploitation: the agent must try actions whose value is uncertain (exploration) to learn, while also using what it already knows (exploitation) to collect reward. Simple mechanisms such as ε-greedy — take the best-known action most of the time, a random action occasionally — remain the standard baseline, and more sophisticated methods such as upper confidence bounds (UCB) allocate exploration to the most promising unknowns.
Algorithms fall into families. Value-based methods learn value functions — Q-learning and SARSA learn the expected return of each state–action pair from experience (temporal-difference learning), and deep Q-networks (DQN) combine Q-learning with neural networks and experience replay, famously learning to play Atari games from pixels in 2015. Policy-based methods optimize the policy directly by gradient ascent on expected return (REINFORCE, policy gradients). Actor-critic methods combine both, and proximal policy optimization (PPO) is the practical workhorse of modern deep RL. Model-based methods learn or use a model of the environment to plan, gaining sample efficiency at the cost of model accuracy.
RL's landmark results are measured in games and control: AlphaGo defeating the world champion at Go in 2016 and its successor AlphaZero mastering Go, chess, and shogi from self-play alone; agents reaching superhuman level in Dota 2 and StarCraft II; robots learning locomotion and manipulation in simulation. In industry, RL schedules resources, optimizes ad and recommendation systems, designs chip layouts, and — through reinforcement learning from human feedback (RLHF) — aligns large language models with human preferences, the technique behind modern instruction-following assistants.
Open problems remain: RL is sample-hungry compared with human learning; reward specification is treacherous, and agents reliably find loopholes in misspecified rewards (specification gaming); safety in real environments is unresolved; and reproducing published results is notoriously difficult. RL is best understood as a framework for learning from interaction — powerful, general, and still far from solved.
Tags
agents machine learning reinforcement learning