Deep reinforcement learning (DRL) has emerged as a powerful framework for solving sequential decision-making problems, achieving remarkable success in a wide range of applications, including game AI, autonomous driving, biomedicine, and large language models. However, the diversity of algorithms and the complexity of theoretical foundations often pose significant challenges for beginners seeking to enter the field. This tutorial aims to provide a concise, intuitive, and practical introduction to DRL, with a particular focus on the Proximal Policy Optimization (PPO) algorithm, which is one of the most widely used and effective DRL methods. To facilitate learning, we organize all algorithms under the Generalized Policy Iteration (GPI) framework, offering readers a unified and systematic perspective. Instead of lengthy theoretical proofs, we emphasize intuitive explanations, illustrative examples, and practical engineering techniques. This work serves as an efficient and accessible guide, helping readers rapidly progress from basic concepts to the implementation of advanced DRL algorithms.

We begin our introduction with a toy example in which an agent needs to cross a frozen lake from its starting point to the goal while avoiding falling into any holes [10]. As illustrated in Figure 2, the agent begins in an initial state \( \boldsymbol{s}_0\) . At each time step \( t\) , the agent observes the current state \( \boldsymbol{s}_t\) of the environment and selects an action \( \boldsymbol{a}_t\) based on this observation. One time step later, due to its action, the agent transitions to a new state \( \boldsymbol{s}_{t+1}\) . Additionally, the environment provides the agent with a feedback signal \( r_{t+1}\) , known as the reward, which indicates the quality of the action taken. The agent interacts with the environment in this way until it reaches a final state \( \boldsymbol{s}_T\) (for example, the agent may reach the goal) at time step \( T\) . This entire process, from the initial state to the final state, is defined as an episode. An episode encompasses all the actions, states, and rewards experienced by the agent during its journey. Throughout the episode, we can record a sequence of state-action-reward pairs, referred to as a trajectory. A trajectory captures the complete history of the agent’s interactions with the environment and is often denoted as \( \tau\) :

In reinforcement learning, we aim to find the optimal policy \( \pi^{*}\) to maximize the accumulated reward collected along the trajectory. The following provides a detailed explanation of this process using the frozen lake environment as an example.

2.1.1   State

2.1.2   Action

2.1.3   Policy

2.1.4   Reward

2.1.5   Return

2.1.6   Transition

2.1.7   Termination of an episode

In reinforcement learning, we utilize value functions to characterize the quality of a particular state or state-action pair under a specific policy \( \pi\) in terms of the expected returns. In other words, value functions provide a way for evaluating a specific policy \( \pi\) . Specifically, the state-value function represents the value of a state, while the action-value function reflects the value of an action taken in a given state, or the value of a state-action pair.

Mathematically, the state-value function for a given state \( \boldsymbol{s} \in \mathcal{S}\) under a policy \( \pi\) , denoted \( V_\pi(\boldsymbol{s})\) , represents the expected return when the agent starts in state \( \boldsymbol{s}\) and subsequently follows policy \( \pi\) :

A higher state-value indicates that the state \( \boldsymbol{s}_t\) is more favorable under the current policy. For example, under the optimal policy, a state closer to the goal state is more likely to have a higher state-value.

Similarly, we define the action-value function for policy \( \pi\) , denoted as \( Q_\pi(\boldsymbol{s}, \boldsymbol{a})\) , which represents the expected return when taking action \( \boldsymbol{a}\) in state \( \boldsymbol{s}\) and following policy \( \pi\) thereafter:

The higher the value of a state-action pair, the better the agent selects the action \( \boldsymbol{a}_t\) in state \( \boldsymbol{s}_t\) under the current policy. For example, under the optimal policy, for a given state, the action that is more likely to lead the agent toward the goal state is more likely to have a higher action-value.

Relationships between the state-value and action-value: State-values and action-values are closely interconnected and can be expressed in terms of each other. As illustrated in the yellow block of Figure 3, the agent is currently in state \( s_t = 14\) . Under the policy \( \pi\) , the agent can select one of three possible actions, with probabilities 0.5, 0.25, and 0.25, respectively. Each action corresponds to a unique state-action pair \( (s_t, a_t)\) , which is associated with a specific action-value \( Q_\pi(s_t, a_t)\) . The state-value \( V_\pi(s_t)\) is the weighted average of the action-values \( Q_\pi(s_t, a_t)\) , weighted by the probabilities of selecting each action according to the policy, \( \pi(a_t | s_t)\) . Mathematically, this relationship is given by:

The green, orange, and blue blocks of Figure 3 illustrate that action-values can also be expressed in terms of state-values. When the agent selects an action \( a_t\) in state \( s_t\) , it transitions to a next state \( s_{t+1}\) with a probability described by the transition model \( p(s_{t+1}, r_{t+1} | s_t, a_t)\) . Each transition is associated with a reward \( r_{t+1}\) , and the subsequent trajectory depends on the next state. The action-value \( Q_\pi(s_t, a_t)\) is the weighted average of the expected cumulative rewards over all possible subsequent trajectories. This is computed as the immediate reward \( r_{t+1}\) plus the discounted state-value of the next state \( \gamma V_\pi(s_{t+1})\) . Mathematically, this relationship is given by:

The Bellman equation: State-values and action-values possess an important recursive property, which gives rise to the Bellman equation. The Bellman equation can be derived by alternately applying the relationships between state-values and action-values, as outlined in Equations (5) and (6). We begin by introducing the Bellman equation for state-values, which expresses the relationship between the value of a state \( s_t\) and the values of its successor states \( s_{t+1}\) :

Similarly, the Bellman equation for action-values expresses the relationship between the value of a state-action pair \( (s_t, a_t)\) and the values of its successor state-action pairs \( (s_{t+1}, a_{t+1})\) :

Advantage function: For a given state-action pair \( (\boldsymbol{s}_t, \boldsymbol{a}_t)\) , the advantage function \( A_\pi(\boldsymbol{s}_t, \boldsymbol{a}_t)\) is defined as the difference between the action-value of the state-action pair \( Q_\pi(\boldsymbol{s}_t, \boldsymbol{a}_t)\) and the state-value of the state \( V_\pi(\boldsymbol{s}_t)\) :

Intuitively, the advantage function measures how much better or worse a particular action \( \boldsymbol{a}_t\) is compared to the average action in the given state \( \boldsymbol{s}_t\) . Specifically, if \( A_\pi(\boldsymbol{s}_t, \boldsymbol{a}_t) > 0\) , selecting action \( \boldsymbol{a}_t\) yields higher expected returns than the average return obtained by selecting any other action in state \( \boldsymbol{s}_t\) . Conversely, if \( A_\pi(\boldsymbol{s}_t, \boldsymbol{a}_t) < 0\) , action \( \boldsymbol{a}_t\) results in lower expected returns than the average.

In reinforcement learning, Monte-Carlo (MC) methods offer a simple yet effective approach for evaluating a given policy. The core idea is to compute the average of observed returns after visiting each state. By the law of large numbers, as more episodes are sampled, the average of these returns converges to their true values under the given policy. Specifically, when estimating the state-value \( V_\pi(\boldsymbol{s}_i)\) for each state \( \boldsymbol{s}_i\) , we maintain a list \( \mathcal{G}(\boldsymbol{s}_i)\) for each state, which stores the observed accumulated rewards obtained when visiting that state. The final estimation of the state-value \( V_\pi(\boldsymbol{s}_i)\) under policy \( \pi\) is the average of all observations in \( \mathcal{G}(\boldsymbol{s}_i)\) . This procedure is formally described in Algorithm 1.

Incremental implementation: In Algorithm 1, we need to store every observation in \( \mathcal{G}(\boldsymbol{s}_i)\) . If the number of samples is very large, this approach can be space-intensive. To address this issue, we can use an incremental mean calculation method:

where \( G_t^{[m]}\) represents the observed return the \( m\) -th time we visit state \( \boldsymbol{s}_t\) , and \( \hat{V}_\pi^{[m]}(\boldsymbol{s}_t)\) denotes the estimate of the state-value function \( V_\pi(\boldsymbol{s}_t)\) after \( m\) updates. Equation (12.b) can be written in a more general form:

where \( \alpha = \frac{1}{m+1}\) . Generally, Equation (13) allows us to adjust the previous estimation by a fraction of the difference between the new observation and the old estimation, with \( \alpha\) controlling the speed of the update. Using this incremental manner, we only need to store the previous estimation and the current observation, which significantly reduces the memory requirements and improves the computational efficiency of Algorithm 1.

MC methods for estimating action-values: Given a policy \( \pi\) to evaluate, the procedure for estimating action-values using the Monte Carlo method is essentially the same as that for estimating state-values. The only difference is that we need to maintain a list for each state-action pair, rather than for each state, to store observations. Similarly, action-value estimates can also be updated incrementally:

where \( G_t^{[m]}\) represents the observed return the \( m\) -th time we visit state-action pair (\( \boldsymbol{s}_t, \boldsymbol{a}_t\) ), and \( \hat{Q}_\pi^{[m]}(\boldsymbol{s}_t, \boldsymbol{a}_t)\) denotes the estimate of the action-value function \( Q_\pi(\boldsymbol{s}_t, \boldsymbol{a}_t)\) after \( m\) updates.

While Monte Carlo (MC) methods can effectively estimate values, they have a significant limitation: updates occur only at the end of an episode. In applications with very long episodes, delaying updates until completion can be inefficient. Moreover, some applications involve episodes of infinite length, making MC methods impractical for value updates. MC methods are also unsuitable for online learning. To address these limitations, Temporal-Difference (TD) methods have been introduced. Like MC methods, TD methods build upon Equation (13), but unlike MC, TD methods update values at each time step. This section provides a comprehensive introduction to TD methods.

In MC methods, updates must wait until the end of an episode because calculating the return requires summing all rewards obtained after visiting a particular state. To avoid this delay, alternative methods for calculating returns are needed. Specifically, when evaluating a given policy \( \pi\) via state-values, we consider the computation of the return from a trajectory \( \tau = \{\boldsymbol{s}_0, \boldsymbol{a}_0, r_1, \dots, \boldsymbol{s}_{T-1}, \boldsymbol{a}_{T-1}, r_T, \boldsymbol{s}_T\} \) . The return for an agent starting in state \( \boldsymbol{s}_t \) is defined by Equation (15.a). Notably, the term \( r_{t+2} + … + \gamma^{T-t-2}r_T \) represents the return for the agent starting in state \( \boldsymbol{s}_{t+1} \) . Based on this observation, we can express the return \( G_t \) as \( G_t = r_{t+1} + \gamma G_{t+1} \) . As shown in Equation (15.b), TD methods approximate \( G_{t+1} \) using the previous estimate of \( V_\pi (\boldsymbol{s}_{t+1}) \) , denoted as \( \hat{V}_\pi (\boldsymbol{s}_{t+1}) \) , thereby avoiding the need to compute \( G_{t+1}\) through the sequence \( r_{t+2}, …, r_T\) . Consequently, this approach elininates the need to wait until the end of an episode for updates.

We can see that the TD method allows us to compute a new estimate of \( V_\pi(\boldsymbol{s}_t)\) by observing only \( r_{t+1} \) , i.e., the reward at the next time step. By substituting Equation (15.b) into Equation (12.b), we derive the TD update rule for estimating state values:

where \( r_{t+1} + \gamma \hat{V}_\pi(\boldsymbol{s}_{t+1}) \) is referred to as the TD target, and \( r_{t+1} + \gamma \hat{V}_\pi(\boldsymbol{s}_{t+1}) - \hat{V}_\pi^{[m]}(\boldsymbol{s}_t) \) is known as the TD error.

Using \( n\) -step return: Equation (15.b) is also known as the 1-step return because it approximates \( G_t\) using a linear combination of the reward over only the next 1 step and the estimated state-value of \( \boldsymbol{s}_{t+1}\) . This approach can be generalized to the \( n\) -step return, where we approximate \( G_t\) using the linear combination of rewards over the next \( n\) steps and the estimated state-value of \( \boldsymbol{s}_{t+n}\) :

TD methods that use \( n\) -step returns to approximate the new observation in Equation (13) are referred to as \( n\) -step TD methods. These methods form a spectrum, with 1-step TD methods at one end and Monte Carlo (MC) methods at the other. In this context, MC methods could be considered as \( \infty\) -step TD methods.

TD methods for estimating action-values: Similar as we did in MC methods, the TD update rule for action-values is as follows:

where \( \hat{Q}_\pi(\boldsymbol{s}_{t}, \boldsymbol{a}_{t})\) denotes an estimate of \( Q_\pi(\boldsymbol{s}_{t}, \boldsymbol{a}_{t})\) . We can also leverage the \( n\) -step return to approximate the new observation in Equation (13):

Previously, we have introduced two approaches, MC and TD methods, for policy evaluation by estimating value functions under a given policy. Generally, MC methods provide unbiased but relatively high-variance estimates, while TD methods offer lower variance at the cost of increased bias. Specifically, MC methods yield an unbiased estimate of the expected return for a given state by calculating returns as the sum of all future rewards. However, this approach results in high variance, as it depends on the cumulative sum of numerous random variables \( r_{t+1}, r_{t+2}, \dots, r_T \) , each affected by the policy’s action choices and the environment’s state-transition dynamics. In contrast, the 1-step TD method reduces variance by involving only two random variables (\( r_{t+1}\) and \( \boldsymbol{s}_{t+1}\) ) for state-values estimation, or three random variables (\( r_{t+1}\) , \( \boldsymbol{s}_{t+1}\) , and \( \boldsymbol{a}_{t+1}\) ) for action-values estimation. Despite this advantage, TD methods introduce bias due to the use of the biased estimator—specifically, \( \hat{V}_\pi(\boldsymbol{s}_{t+1}) \) for state-value estimation and \( \hat{Q}_\pi(\boldsymbol{s}_{t+1}, \boldsymbol{a}_{t+1}) \) for action-value estimation—in place of the full return. In TD methods that use \( n \) -step returns, as shown in Equations (17.a)(19), the choice of \( n \) serves as a trade-off between bias and variance in the value estimation. In this section, we introduce the \( \lambda\) -return, which provides another method to trade off between the bias and variance of the estimator. Estimating the value functions using the TD method computed with the \( \lambda\) -return results in the TD(\( \lambda\) ) algorithm [9]. Empirical studies suggest that the \( \lambda\) -return can achieve better performance than using a fixed \( n \) -step return [9, 11].

The \( \lambda\) -return, denoted by \( G_t^\lambda\) , is calculated as an exponentially-weighted average of \( n\) -step returns with a decay parameter \( \lambda \in [0, 1]\) , along with the normalization factor \( 1 - \lambda\) to ensure the sum of all weights equals 1.

where \( G_{t:t+n}\) represents the \( n\) -step return. Simplifying Equation (20) yields:

where we observe two special cases by setting \( \lambda = 0\) and \( \lambda = 1\) :

It is evident that when \( \lambda = 1\) , estimating values using \( \lambda\) -return leads to the Monte Carlo methods, while \( \lambda\) -return reduces to the 1-step return when \( \lambda = 0\) . For values \( 0 < \lambda < 1\) , the \( \lambda\) -return offers a compromise between bias and variance, controlled by the parameter \( \lambda\) .

Now we consider episodes with a finite length \( T\) . Corresponding to the previous infinite-horizon case, we can assume that all rewards after step \( T\) are 0 so that we have \( G_{t:t+n} = G_{t:T}\) for all \( n \geq T - t\) . Consequently, we have

The 1-step return is given the largest weight, \( 1 - \lambda\) ; the 2-step return is given the next largest weight; and so on.

The methods introduced previously are referred to as tabular-based methods. These methods are not practical for high-dimensional cases, where the state or action spaces are extremely large, making tabular storage infeasible. To address this limitation, we can use a function approximator \( f(\cdot; \boldsymbol{w})\) to approximate the value function. In this context, the value functions are parameterized by \( \boldsymbol{w}\) . For instance, when parameterizing the state-value functions, the function approximator takes a state \( \boldsymbol{s}\) as input and outputs the corresponding state-value under a specific policy \( \pi\) , i.e. \( f(\cdot; \boldsymbol{w}) = \hat{V}_\pi(\boldsymbol{s}; \boldsymbol{w})\) .

A commonly used function approximator is a deep neural network. The goal is to optimize the parameters \( \boldsymbol{w}\) to achieve the best approximation of the true value funtions. Specifically, for approximating the state-value function under a given policy \( \pi\) . We aim to minimize the difference between the output of the state-value function approximator and the true state-value \( y_t\) for any given state \( \boldsymbol{s}_t \in \mathcal{S}\) :

In practical applications, the true state-value is not directly accessible. Instead, we use methods introduced in previous sections to define the target \( y_t\) . One common choice is the TD target. The objective with the TD target becomes

From the perspective of deep learning, this involves training a neural network \( \hat{V}_\pi(\boldsymbol{s}; \boldsymbol{w})\) within a supervised learning framework, where the task is framed as a regression problem. The training data is generated by having the agent interact with the environment following the policy \( \pi\) . Specifically, suppose the current network parameters are \( \boldsymbol{w}_c\) , a sampled trajectory \( \tau = \{\boldsymbol{s}_t, \boldsymbol{a}_t, r_{t+1}\}_{t=0}^{T-1}\) generates training pairs \( (r_{t+1} + \gamma \hat{V}_\pi(\boldsymbol{s}_{t+1}; \boldsymbol{w}_c), \hat{V}_\pi(\boldsymbol{s}_t; \boldsymbol{w}_c))\) . Using this dataset, we train the neural network by minimizing the mean squared error (MSE) loss:

where \( N\) denotes the total number of training pairs, and \( \boldsymbol{s}_t^{(i)}\) and \( r_{t+1}^{(i)}\) represent the \( i\) -th occurrence of the state \( \boldsymbol{s}_t\) and the corresponding rewards in the samples, respectively.

The primary objective of this subsection is to derive a practical gradient estimator for \( \nabla_{\boldsymbol{\theta}_\text{old}} \mathbb{E}_{\pi_{\boldsymbol{\theta}_\text{old}}}[G(\tau)]\) . Additionally, we introduce variance reduction techniques to enhance the overall performance and stability of the algorithm. The resulting method is commonly referred to as the Vanilla Policy Gradient (VPG) algorithm.

We begin our discussion be treating the trajectory \( \tau = \{\boldsymbol{s}_t, \boldsymbol{a}_t, r_{t+1}\}_{t=0}^{T-1}\) as a random variable, with \( p_{\boldsymbol{\theta}}(\tau)\) representing the probability of observing a specific trajectory under the policy parameterized by \( \boldsymbol{\theta}\) . We begin our discussion by considering the undiscounted case where the total accumulated reward is given by \( G(\tau) = \sum_{t=1}^T r_t\) . Our goal is to optimize the policy parameters \( \boldsymbol{\theta}\) to maximize the expected return. To achieve this, we focus on computing the gradient \( \nabla_{\boldsymbol{\theta}} \mathbb{E}_{\tau \sim p_{\boldsymbol{\theta}}(\tau)}[G(\tau)]\) :

Next, we expand the \( \nabla_{\boldsymbol{\theta}} \log p_{\boldsymbol{\theta}} (\tau)\) to derive a practical formula:

where \( p(\boldsymbol{s}_0)\) represents the distribution of the initial state, and \( p(\boldsymbol{s}_{t+1}, r_{t+1} | \boldsymbol{s}_t, \boldsymbol{a}_t)\) denotes the environment’s transition dynamics. These terms are independent with the policy’s parameters \( \boldsymbol{\theta}\) so that their gradients equal to zero. Substituing \( G(\tau) = \sum_{t=1}^T r_t\) and Equation (29) into Equation (28), we obtain

Intuitively, \( \log \pi(\boldsymbol{a}_t | \boldsymbol{s}_t; \boldsymbol{\theta})\) describes the log-probability of taking an action \( \boldsymbol{a}_t\) when observing the state \( \boldsymbol{s}_t\) under the current policy’s parameter \( \boldsymbol{\theta}\) . By multiplying this log-probability with the accumulated rewards and taking the gradient with respect to the \( \boldsymbol{\theta}\) , we are pushing up the log-probabilities of each action in proportion to \( G(\tau)\) through a gradient ascent step. However, this approach overlooks the fact that rewards obtained prior to taking an action have no influence on the quality of that action—the rewards that occur after the action are the relevant ones. To address this, people have introduced the concept of causality [12, 13], which states that the policy cannot influence rewards that have already been obtained. By exploiting this causality, we can modify the objective to account only for future rewards, resulting in the following adjusted form:

where the term \( \mathbb{E}_{\tau \sim p_{\boldsymbol{\theta}}(\tau)} \left[ \sum_{t=0}^{T-1} \nabla_{\boldsymbol{\theta}} \log \pi(\boldsymbol{a}_t|\boldsymbol{s}_t; \boldsymbol{\theta}) \sum_{t^\prime = 1}^t r_{t^\prime} \right]\) vanishes due to the causality assumption. Furthermore, it can be shown mathematically that the term \( \sum_{t=0}^{T-1} \nabla_{\boldsymbol{\theta}} \log \pi(\boldsymbol{a}_t|\boldsymbol{s}_t; \boldsymbol{\theta}) \sum_{t^\prime = 1}^t r_{t^\prime}\) has zero mean but nonzero variance [14]. Therefore, leveraging causality is an effective way to reduce variance in policy gradient methods.

It is worth noting that a discount factor \( \gamma\) is typically introduced to assign greater importance to immediate rewards. With this, the gradient estimator is given by

In practice, we often collect a batch of trajectories and estimate the gradient of the expected return by averaging over the batch of trajectories. The algorithm is summarized in Algorithm 2.

We can reduce the variance of the policy gradient estimator by introducing a baseline term \( b(\boldsymbol{s}_t)\) that have no effect on the expectation.

A near-optimal choice of baseline is the state-value function [15]. By the definition of advantage function, we rewrite the objective and its gradient as:

To demonstrate the advantage of introducing a baseline, consider the following example. For a given agent’s state \( \boldsymbol{s}_t\) , two different actions, \( \boldsymbol{a}_t^{[1]}\) and \( \boldsymbol{a}_t^{[2]}\) , lead to distinct trajectories with accumulated rewards \( G^{[1]}\) and \( G^{[2]}\) , respectively. Let \( \bar{G}\) denote the average accumulated reward across various actions. Assume \( G^{[1]} > \bar{G} > G^{[2]} > 0\) . Without a baseline, the gradient ascent step would lead to an increase in the selection probabilities of both actions, with a relatively greater increment for \( \boldsymbol{a}_t^{[1]}\) compared to \( \boldsymbol{a}_t^{[2]}\) . However, by introducing a baseline, we calculate the advantages as \( A^{[1]} = G^{[1]} - \bar{G} > 0\) and \( A^{[2]} = G^{[2]} - \bar{G} < 0\) . As a result, the gradient ascent step increases the probability of selecting \( \boldsymbol{a}_t^{[1]}\) while decreasing the probability of selecting \( \boldsymbol{a}_t^{[2]}\) . This method ensures that policy updates focus on the relative performance of actions compared to the baseline, rather than being influenced by absolute rewards, which can vary widely. When the state-value function is used as the baseline, the policy gradient step increases the probability of actions that are better than average and decreases the probability of those that are worse than average [11]. This leads to more stable and focused policy updates, resulting in more efficient learning and improved convergence properties in reinforcement learning algorithms.

We know that in policy gradient methods, evaluating a specific policy typically involves two steps: first, estimating the expected return \( G(\tau)\) through sampling, where \( G(\tau)\) can take various forms, such as \( Q_\pi(\boldsymbol{s}_t, \boldsymbol{a}_t)\) and \( A_\pi(\boldsymbol{s}_t, \boldsymbol{a}_t)\) . Then, we compute its gradient with respect to the policy parameters \( \boldsymbol{\theta}\) . This gradient provides information about the direction of the greatest change in the expected return. It is worth noting that in the first step, the expected return can be estimated using both the Monte Carlo method and the Temporal Difference method. If the expected return is estimated using the Monte Carlo method, the resulting algorithm is called the Monte Carlo policy gradient, or REINFORCE, which we have discussed previously. In this section, we will discuss the algorithms where the expected return is estimated using Temporal Difference methods, and the resulting algorithms are often referred to as Actor-Critic methods.

Recall that the key idea of TD methods is to leverage previous value estimates when calculating returns, as shown in Equation (17.a) and Equation (19). In Section 4.4, we also introduced the value parameterization technique. Suppose we use a neural network to approximate the state-value function, denoted as \( \hat{V}(\boldsymbol{s}; \boldsymbol{w})\) , where \( \boldsymbol{w}\) represents the network parameters. Using 1-step TD, the expected return is estimated by \( r_{t+1} + \gamma \hat{V}(\boldsymbol{s}_{t+1}; \boldsymbol{w})\) . In actor-critic methods, this value network is referred to as the critic, while the policy network is known as the actor. A basic actor-critic algorithm is presented in Algorithm 3.

It is worth noting that, compared to Monte Carlo policy gradient methods, actor-critic methods have the advantage of not requiring the episode to terminate before updating the policy. Additionally, due to the nature of TD methods, they typically exhibit lower variance.

In this section, we present a widely used technique for estimating the advantage function, which utilizes the \( \lambda\) -return to compute advantage estimates. Similar to the introduction of TD\( (\lambda)\) , where TD\( (\lambda)\) is formulated as an exponentially weighted average of \( n\) -step returns, Generalized Advantage Estimation (GAE), or GAE(\( \lambda\) ) [11], is defined as an exponentially weighted average of \( n\) -step advantage estimators.

Consider a trajectory \( \tau = \{\boldsymbol{s}_t, \boldsymbol{a}_t, r_{t+1}\}_{t=0}^{T-1}\) . When the agent is in state \( \boldsymbol{s}_t\) and takes action \( \boldsymbol{a}_t\) , the \( n\) -step return can be used to estimate the value of the state-action pair \( (\boldsymbol{s}_t, \boldsymbol{a}_t)\) , denoted as \( \hat{Q}_{t:t+n}\) . By definition, the difference between the estimated action value \( \hat{Q}_{t:t+n}\) and the state value \( \hat{V}_\pi(\boldsymbol{s}_t)\) provides a \( n\) -step advantage estimator, represented as \( \hat{A}_{t:t+n}\) :

The \( n\) -step advantage estimator can be shown to be equivalent to the discounted sum of TD errors, which can be derived by demonstrating the equivalence between Equation (36.a) and Equation (35). Here, the TD error is defined as \( \delta_{t+1}^V = r_{t+1} + \gamma \hat{V}_\pi(\boldsymbol{s}_{t+1}) - \hat{V}_\pi(\boldsymbol{s}_t)\) :

GAE(\( \lambda\) ) is defined as the exponentially-weighted average of the \( n\) -step advantage estimators:

where we observe two special cases by setting \( \lambda = 0\) and \( \lambda = 1\) :

When \( \lambda = 1\) , GAE(\( \lambda\) ) corresponds to an \( \infty\) -step advantage estimator, equivalent to the Monte Carlo method, which is characterized by high variance due to the summation of multiple terms. Conversely, when \( \lambda = 0\) , GAE(\( \lambda\) ) reduces to the 1-step TD residual, which has lower variance but introduces higher bias. For intermediate values \( 0 < \lambda < 1\) , GAE(\( \lambda\) ) strikes a balance between bias and variance, with the trade-off controlled by the parameter \( \lambda\) .

In previous sections, we estimated the gradient of the expected return with respect to the current policy parameters, as illustrated in Figure 13(a). This approach is referred to as using on-policy samples, where the policy used for sampling and evaluation is the same. However, this training procedure has a significant limitation: samples collected using the current policy can only be used for a single gradient step. Afterward, a new batch of samples must be collected to perform the gradient step for the updated policy. This process results in low data efficiency. To address this issue, we can leverage off-policy samples, as illustrated in Figure 13(b). Instead of using the entire batch of collected data for just one gradient step, off-policy training allows us to reuse the same batch by generating mini-batches for multiple gradient updates. Let the policy used for sampling be denoted as \( \pi(\boldsymbol{a} | \boldsymbol{s}; \textcolor{blue}{ \boldsymbol{\theta}_{\text{old}}})\) and the policy used to perform the \( k\) -th gradient step as \( \pi(\boldsymbol{a} | \boldsymbol{s}; \textcolor{red}{ \boldsymbol{\theta}_k})\) . At each gradient step, we compute:

where \( \textcolor{red}{ \boldsymbol{\theta}_k} \neq \textcolor{blue}{ \boldsymbol{\theta}_{\text{old}}}\) after the first gradient step for each episode. The challenge arises because the expectation must now be computed over the sampling distribution \( p_\textcolor{blue}{}(\tau)\) , even though the policy being updated is \( \pi(\boldsymbol{a}|\boldsymbol{s}; \textcolor{red}{ \boldsymbol{\theta}_k})\) . To resolve this, the remaining part of this section introduces a technique called importance sampling, which enables us to compute expectations when the sampling distribution differs from the distribution under policy evaluation.

Importance sampling is a general technique for estimating the expectation of a function \( f(\boldsymbol{x})\) of a random variable \( \boldsymbol{x}\) under one probability distribution \( p(\boldsymbol{x})\) , using samples drawn from a different distribution \( q(\boldsymbol{x})\) . We begin our detailed introduction by considering the problem of estimating \( \mathbb{E}_{\boldsymbol{x} \sim p(\boldsymbol{x})}[f(\boldsymbol{x})]\) , the expectation of \( f(\boldsymbol{x})\) with respect to \( \boldsymbol{x} \sim p(\boldsymbol{x})\) , where \( p(\boldsymbol{x})\) is the true distribution of \( \boldsymbol{x}\) . A straightforward approach would be to generate independent and identically distributed (i.i.d.) samples from \( p(\boldsymbol{x})\) and compute their average to obtain an unbiased estimate of \( \mathbb{E}_{\boldsymbol{x} \sim p(\boldsymbol{x})}[f(\boldsymbol{x})]\) . However, this approach is not feasible when \( p(\boldsymbol{x})\) is unknown and we only have i.i.d. samples from another distribution \( q(\boldsymbol{x})\) . In this case, importance sampling can be applied as follows:

Using the importance sampling technique, we can conveniently compute the objective and its gradient when leveraging off-policy samples:

where \( \mathcal{L}^{\text{IS}}(\boldsymbol{\theta})\) represents the loss function for policy gradient incorporating the importance sampling technique. Equations (41.a) and (41.b) are derived by directly applying the importance sampling technique to Equations (34.a) and (34.b), respectively. Furthermore, Equation (41.b) can be derived by directly taking the gradient of Equation (41.a):

By leveraging the importance sampling technique, the same batch of data can be reused for multiple gradient steps, thereby significantly improving data efficiency.

Empirically, the policy gradient methods discussed so far often suffer from instability, where even a single poorly chosen update step can significantly degrade the performance. The Proximal Policy Optimization (PPO) algorithm addresses this limitation by introducing a clipped objective function that restricts the magnitude of policy updates. This mechanism prevents excessive changes to the policy, ensuring updates remain controlled and the policy evolves incrementally. By stabilizing the training process, PPO reduces the risk of divergence and enhances overall robustness. It has become one of the most popular deep reinforcement algorithms in diverse applications.

PPO optimizes a clipped loss defined as:

where the \( \text{clip}(\cdot)\) operator constrains the ratio \( \frac{\pi(\boldsymbol{a}|\boldsymbol{s};\boldsymbol{\theta})}{\pi(\boldsymbol{a}|\boldsymbol{s};\boldsymbol{\theta}_\text{old})}\) to lie within the range \( [1 - \epsilon, 1 + \epsilon]\) , and \( \epsilon\) is a hyperparameter that controls the extent of the policy update. Although the above equation appears complex, its intuition becomes clearer through its simplified version:

We offer the following intuitive explanation based on the simplified form of the objective:

• when \( A_{\pi_{\boldsymbol{\theta}_{\text{old}}}}(\boldsymbol{s}, \boldsymbol{a}) \geq 0\) , it implies that the specific action \( \boldsymbol{a}\) is at least as good as the average action under the state \( \boldsymbol{s}\) . In this case, we aim to make that action more likely to be chosen, i.e., increase \( \pi(\boldsymbol{a}|\boldsymbol{s}; \boldsymbol{\theta})\) , to improve the objective. However, the use of the min operator imposes a limit on how much the objective can increase. Once \( \pi(\boldsymbol{a}|\boldsymbol{s};\boldsymbol{\theta})> (1 + \epsilon)\pi(\boldsymbol{a}|\boldsymbol{s};\boldsymbol{\theta}_\text{old})\) , the min term activates, and this part of the objective reaches a maximum of \( (1 + \epsilon)\pi(\boldsymbol{a}|\boldsymbol{s};\boldsymbol{\theta}_\text{old})\) .
• when \( A_{\pi_{\boldsymbol{\theta}_{\text{old}}}}(\boldsymbol{s}, \boldsymbol{a}) < 0\) , it indicates that the specific action \( \boldsymbol{a}\) is worse than the average action under the state \( \boldsymbol{s}\) . In this scenario, we aim to make that action less likely to be chosen, i.e., reduce \( \pi(\boldsymbol{a}|\boldsymbol{s}; \boldsymbol{\theta})\) , to improve the objective. However, the max operator imposes a limit on how much the objective can increase. Once \( \pi(\boldsymbol{a}|\boldsymbol{s};\boldsymbol{\theta})> (1 + \epsilon)\pi(\boldsymbol{a}|\boldsymbol{s};\boldsymbol{\theta}_\text{old})\) , the max term activates, and this part of the objective reaches a floor of \( (1 + \epsilon)\pi(\boldsymbol{a}|\boldsymbol{s};\boldsymbol{\theta}_\text{old})\) .

In practice, we employ an actor-critic approach, utilizing a learned state-value function to compute variance-reduced advantage estimates. To encourage sufficient exploration, the overall objective is further augmented with an entropy bonus term. The detailed procedure is presented in Algorithm 4.

\IncMargin{1.2em}

Standard implementation fits the critic with a clipped objective:

where \( \hat{V}_\pi(\boldsymbol{s})\) is clipped around the previous value estimates and \( \epsilon\) is fixed to the same value as the value used to clip probability ratios in 43.

For continuous tasks, such as robotic control, there is a clear failure termination criterion but no explicit success criterion. During training, a maximum episode length is often imposed, causing the interaction between the agent and the environment to terminate when this limit is reached. This termination is referred to as a timeout. To ensure the value estimation aligns with the original task objectives, the rewards associated with timeout states should be appropriately handled:

In certain cases where \( \boldsymbol{s}_{T+1}\) cannot be obtained due to environment resets, \( \boldsymbol{s}_{T}\) can be used as an approximation.

To reduce the variance of policy gradients and improve training stability, batch normalization can be applied to the advantage function:

Applying batch normalization to neural network inputs can improve training performance in supervised learning. However, in reinforcement learning, batch normalization is generally not suitable. Instead, a dynamic mean and variance of the states are typically maintained to normalize inputs during training:

\( \hat{\mu}\) and \( \hat{\sigma}^2\) represent the mean and variance of the current input batch.

To maintain policy stochasticity for continued exploration, an entropy bonus term can be incorporated into the loss function. Thus, the complete loss function is defined as:

which maximizes an entropy bonus term.

To achieve more efficient training, the learning rate can be dynamically adjusted based on changes in the KL divergence throughout the training process. It implements a dynamic strategy for controlling the learning rate \( \alpha\) through a comparative analysis of the current KL divergence \( kl\) against a target threshold \( kl^*\) .

Gradient clipping is a technique introduced to prevent gradient explosion during training. This trick serves to stabilize the training process:

[1] Hua Wei and Jingxiao Chen and Xiyang Ji and Hongyang Qin and Minwen Deng and Siqin Li and Liang Wang and Weinan Zhang and Yong Yu and Liu Linc and others Honor of Kings Arena: an Environment for Generalization in Competitive Reinforcement Learning Advances in Neural Information Processing Systems (NeurIPS) 2022 35 11881–11892

[2] Gengrui Zhang and Yao Wang and Xiaoshuang Chen and Hongyi Qian and Kaiqiao Zhan and Ben Wang UNEX-RL: Reinforcing Long-Term Rewards in Multi-Stage Recommender Systems with UNidirectional EXecution Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) 2024 38 8 9305–9313

[3] Zuojin Tang and Xiaoyu Chen and YongQiang Li and Jianyu Chen Efficient and Generalized end-to-end Autonomous Driving System with Latent Deep Reinforcement Learning and Demonstrations arXiv preprint arXiv:2401.11792 2024

[4] Jialin Jiang and Lidong Yang and Shihao Yang and Li Zhang A Deep Learning-based Framework for Environment-adaptive Navigation of Size-adaptable Microswarms Engineering 2024

[5] Aixin Liu and Bei Feng and Bing Xue and Bingxuan Wang and Bochao Wu and Chengda Lu and Chenggang Zhao and Chengqi Deng and Chenyu Zhang and Chong Ruan and others DeepSeek-V3 Technical Report arXiv preprint arXiv:2412.19437 2024

[6] Molei Qin and Shuo Sun and Wentao Zhang and Haochong Xia and Xinrun Wang and Bo An EarnHFT: Efficient Hierarchical Reinforcement Learning for High Frequency Trading Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) 2024 38 13 14669–14676

[7] Kaiwen Jiang and Zhen Fu and Junde Guo and Wei Zhang and Hua Chen Learning Whole-Body Loco-Manipulation for Omni-Directional Task Space Pose Tracking with a Wheeled-Quadrupedal-Manipulator IEEE Robotics and Automation Letters (RA-L) 2024

[8] Xue Bin Peng and Ze Ma and Pieter Abbeel and Sergey Levine and Angjoo Kanazawa AMP: Adversarial Motion Priors for Stylized Physics-Based Character Control ACM Transactions on Graphics (ToG) 2021 40 4 1–20

[9] Richard S. Sutton and Andrew G. Barto Reinforcement Learning: An Introduction The MIT Press 2018 Second

[10] Ariel Kwiatkowski and Mark Towers and Jordan Terry and John U. Balis and Gianluca De Cola and Tristan Deleu and Manuel Goulão and Andreas Kallinteris and Markus Krimmel and Arjun KG and Rodrigo Perez-Vicente and Andrea Pierré and Sander Schulhoff and Jun Jet Tai and Hannah Tan and Omar G. Younis Gymnasium: A Standard Interface for Reinforcement Learning Environments 2024

[11] John Schulman and Philipp Moritz and Sergey Levine and Michael I. Jordan and Pieter Abbeel High-Dimensional Continuous Control Using Generalized Advantage Estimation International Conference on Learning Representations (ICLR) 2016

[12] John Schulman Optimizing expectations: From deep reinforcement learning to stochastic computation graphs UC Berkeley 2016

[13] Sergey Levine CS285: Deep Reinforcement Learning Course at University of California, Berkeley 2024

[14] Joshua Achiam Spinning Up in Deep Reinforcement Learning 2018

[15] Shiyu Zhao Mathematical Foundations of Reinforcement Learning Springer Verlag, Singapore 2025