How Rewards
Guide Actions

1. 1How can we learn from actions?
2. 2Operant Conditioning
3. 3Pavlovian Conditioning
4. 4Sequential Decisions
5. 5Temporal Difference Learning

Reinforcement Learning is the art of learning without being told

Of course, the reward function is ‘telling’ in a similar way to data. So what do you think the difference truly is?

The power to change expectations comes from violating them.

Acquisition and extinction are the two faces of that violation. Pair a cue with food and the probability of salivating rises, fastest on the first trials. Pair the same cue with nothing and it falls, again fastest at the start. The steps get smaller as the prediction catches up to what actually arrives.

T(s, a) = s_NEXT — recursion. Sequential decision making and dynamic programming.

The value of a state is the best immediate reward plus the discounted value of wherever that action sends you. Once you can write that down, following the value function is what maximising reward looks like as a procedure.

Bonus slide

Blockers as causal cues — any effect only has one cause?

Other ideas for the prediction error learning reasons?

How do we learn the value function? Temporal difference learning turns the Bellman backup into an update you can run from one step of experience. The quantity in brackets is the surprise: reward plus what you now think the next state is worth, minus what you thought this state was worth.

Dopamine as a signal for both positive and negative prediction errors.

Striatum region as a central area to reward learning. In this region changes only occur in the presence of dopamine (there is actually degradations of pathways — unlearning — in extreme absence of dopamine).

Bonus slide

Blocking still occurs in drug use. Why is this if drugs create errors forcefully (by increasing dopamine)?

An explanation encompassing all of Marr’s levels is an aspiration towards which many neuroscientists strive. The systems that carry out reinforcement learning are a rare case where they can come within striking distance of this high bar.

At the computational level reinforcement learning has a simple answer: maximise reward. This is what Bellman recognised as the goal of sequential decision processes and what following the value function should get you. But how do we learn the value function? That’s where temporal difference learning comes in. The work of Bush, Mosteller, Rescorla, Wagner and Sutton all turned stacks of data from conditioning experiments into strings of symbols that could describe the algorithm needed to do the learning part of reinforcement learning. On the implementation level, dopamine neurons take on the task of calculating prediction error and the signals they send to other brain areas control the associations learned there. In this way, a satisfying understanding of a fundamental ability – to learn from rewards – was achieved by tunnelling towards the topic from many different angles.