Home on Decisions & Dragons

Should we abandon RL? Is it the right approach?

Wed, 02 Jul 2025 23:43:37 -0400

No, we should not abandon reinforcement learning. I get it though — RL algorithms are brittle, difficult to scale, and complicated. However, this question is predicated on a misconception. RL is not an approach. RL is a problem definition.

RL is the problem of determining how an agent should make decisions in an unfamiliar environment. It must act to both learn about its environment and pursue its objective. It must learn from its experiences, rather than a human-curated dataset.

We don’t abandon problems. They are imposed upon us and we ignore them at our peril. Asking if a problem is “right” isn’t a coherent question. What is a coherent question is whether a problem is important or relevant.

RL is an important problem. If you care about building systems that can act and learn in the world like people do, then the RL problem is impossible to avoid. The question is not whether we should abandon RL, but how we can make better algorithms to solve it.

Why is it better to subtract a baseline in REINFORCE?

Thu, 08 May 2025 15:06:56 -0400

Suppose we have a softmax policy over five actions. In the current state, an oracle tells us that the true Q-values are -1, -2, -3, -4, and -5. The first action is the best, even though it is negative, so we should increase the probability of it. We sample an action from our policy. It’s the first! What a stroke of luck!

Or is it? Unfortunately, our algorithm is REINFORCE. It decreases the probability of selecting the winning action. Why? Because the REINFORCE stochastic gradient is the product of the Q-value estimate and the gradient of the log probability of the action: $\nabla \log \pi(a | s) Q^\pi(s, a)$. Since the Q-value is negative, REINFORCE decreases the action’s probability.

Unless we’re using a baseline. For example, if we subtracted a baseline of -3, then we would multiply the gradient of the log probability by $(-1 - -3) = 2$: a positive weight. In this case, REINFORCE correctly increases the probability of the action.

Why does experience replay require off-policy learning and how is it different from on-policy learning?

Sat, 04 May 2024 11:39:36 -0400

When you use an experience replay buffer, you save the most recent $k$ experiences of the agent, and sample data from that buffer for training. Typically, the agent does a step of training to update its policy for every step in the environment. At any moment in time, the vast majority of experiences in the buffer are generated with a different – earlier – policy than the current policy. And if the policy used to collect data is different than the policy being evaluated or improved, then you need an off-policy method.

What is the "horizon" in reinforcement learning?

Sun, 21 Apr 2024 16:49:00 -0400

In reinforcement learning, an agent receives reward on each time step. The goal, loosely speaking, is to maximize the future reward received. But that doesn’t fully define the goal, because each decision can affect what reward the agent can receive the future. Consequently, we’re left with the question “how does potential future reward affect our decision right now?” The “horizon” refers to how far into the future the agent will optimize its reward. You can have finite-horizon objectives, or even infinite-horizon objectives.

Why doesn't Q-learning work with continuous actions?

Sun, 21 Apr 2024 15:36:15 -0400

Q-learning requires finding the action with the maximum Q-value in two places: (1) In the learning update itself; and (2) when extracting the policy from the learned Q-values. When there are a small number of discrete actions, you can simply enumerate the Q-values for each and pick the action with the highest value. However, this approach does not work with continuous actions, because there are an infinite number of actions to evaluate!

Why is the DDPG gradient the product of the Q-function gradient and policy gradient?

Sat, 20 Apr 2024 12:51:21 -0400

The DDPG and DPG paper before it express the gradient of the objective $J(\pi)$ as the product of the policy and Q-function gradients:

$$ \nabla_\theta J(\pi) = E_{s \sim \rho^\pi} \left[\nabla_\theta \pi_\theta(s) \nabla_a Q(s, a) \rvert_{a \triangleq \pi_\theta(s)} \right]. $$

This expression looks a little scary, but it’s conveying a straightforward concept: the gradient is the average of the Q-function’s gradient with respect to the policy parameters, evaluated at the policy’s selected action. That may not be obvious because the product of “gradients” (spoiler: there is some notation abuse) is the result of applying the multivariable chain rule of differentiation. If we were to reverse this step, the expected value would simplify to the more explicit expression $\nabla_\theta Q(s, \pi_\theta(s))$.

If Q-learning is off-policy, why doesn't it require importance sampling?

Tue, 02 Apr 2024 23:43:37 -0400

In off-policy learning, we evaluate the value function for a policy other than the one we are following in the environment. This difference creates a mismatch in state-action distributions. To account for this difference, some actor-critic methods use importance sampling. However, Q-learning does not. There is a simple reason for that: In Q-learning, we only use samples to tell us about the effect of actions on the environment, not to estimate how good the policy action selection is. Let’s make that more concrete with a simple example and re-derive the Q-learning and importance sampling approaches.

What is the difference between V(s) and Q(s,a)?

Sat, 30 Mar 2024 23:24:52 -0400

State value function $V(s)$ expresses how well the agent expects to do when it acts normally. $Q(s, a)$ is a counterfactual function that expresses how well the agent expects to do if it first takes some potentially alternative action before acting normally.

Why does the policy gradient include a log probability term?

Fri, 29 Mar 2024 20:07:54 -0400

Actually, it doesn’t! What you’re probably thinking of is the REINFORCE estimate of the policy gradient. How we derive the REINFORCE estimate you’re familiar with and why we use it is something I found to be poorly explained in literature. Fortunately, it is not a hard concept to learn!

What is the difference between model-based and model-free RL?

Fri, 29 Mar 2024 19:50:17 -0400

In reinforcement learning, the agent is not assumed to know how the environment will be affected by its actions. Model-based and model-free reinforcement learning tackle this problem in different ways. In model-based reinforcement learning, the agent learns a model of how the environment is affected by its actions and uses this model to determine how to act. In model-free reinforcement learning, the agent learns how to act without ever learning to precisely predict how the environment will be affected by its actions.

About

Tue, 26 Mar 2024 20:17:59 -0400

Reinforcement learning (RL) is an elegant problem definition for autonomous agents that learn from their own interactions with an environment. But the methods to solve this simple problem definition are not so simple. To solve this problem you must simultaneously tackle many subproblems that are all complex enough to warrant their own subfields in AI, such as perception, prediction, planning, and memory.

Unlike other forms of machine learning, we cannot feed an algorithm a well curated dataset. An agent must form its own data from interactions. Even worse, this data is temporally correlated and does not explicitly include the correct response. The agent must reason about its data to determine the correct response and it must actively explore the environment to ensure it has good data coverage. If that wasn’t hard enough, it must do all this even when it doesn’t have much data to go on. The world waits for no ~~man~~ agent, and it must make do with what it has.

Math Notation Cheatsheet

Tue, 26 Mar 2024 20:17:59 -0400

In this section I outline the meaning of the mathematical notation I use. When appropriate (and possible), I also describe the meaning in simple python.

General math and statistics

$f(x) \triangleq mx + b$

The $\triangleq$ indicates that the expression on the left is defined to be the expression on the right, rather than an equivalence that is derived from mathematical rules.

$(a, b)$

The set of real numbers between $a$ and $b$, excluding those values.

Revisions

Tue, 26 Mar 2024 20:17:59 -0400

I have a tendency to make a lot of typos. My brain reads what I meant, not what I typed. On an occasion, I may also make an error (:gasp:) or state something too imprecisely.

I don’t have collaborators for this site to check for any kind of errors. Therefore, if you spot an error of any sort, please let me know by reporting it on my Github issues page or contact me some other way (see the about page).