Inspired by DAC [31], we reformulate the training as two augmented Markov Decision Processes (MDPs): high-MDP \( {\cal M}^h\) and low-MDP \( {\cal M}^l\) . The high-level policy \( \pi^h\) and low-level policy \( \pi^l\) make decisions within their respective MDPs and are optimized jointly.

The \( {\cal M}^h\) can be defined by a tuple \( <{{\cal S}^h},{{\cal H}^h},{{\cal R}^h},{{\cal T}^h},\gamma >\) , where: 1) \( {\cal S}^h\) is the high-level state space, derived from the environment; 2) \( {\cal H}^h\) is the hybrid action space, composed of discrete and continuous subspaces, \( {\cal O}\) and \( {\cal A}^h\) ; 3) \( {\cal R}^h\) is the high-level reward function, determined by low-level rewards and agent violations; and 4) \( {\cal T}^h\) and \( \gamma\) are the high-level transition function and discount factor.

Similarly, the \( {\cal M}^l\) is defined by \( <{{\cal Z}^l},{{\cal A}^l},{{\cal R}^l},{{\cal T}^l},\gamma>\) , where: 1) \( {\cal Z}^l\) is the low-level state space, combining the original low-level state space \( {\cal S}^l\) and all motion guidance mapping to \( {\cal H}^l\) ; 2) \( {\cal A}^l\) is the low-level continuous action space; 3) \( {\cal R}^l\) is the low-level reward function derived from environmental feedback; and 4) \( {\cal T}^l\) is the transition function.

At each long timestep \( {T^h}\) , \( {\pi ^h}( o, {a^h} \mid {s^h} )\) outputs guidance-action \( (o, {a^h}) \in {{\cal H}^h}\) , with \( {s^h} \in {{\cal S}^h}\) . Each guidance-action explicitly represents a motion guidance \( {\cal G}\) via a bi-directional mapping: \( {\cal G} \leftrightarrow (o, {a^h})\) . Hereafter, \( {\cal G}(o, a^h)\) denotes the motion guidance corresponding to \( (o, a^h)\) . Then, at each short timestep \( {T^l}\) , \( {\pi ^l}( {a^l} \mid )\) receives extend-state \( {z^l} = ({s^l}, {\cal G}(o, {a^h}))\) from the environment and the high-level policy \( \pi^h\) , where \( {z^l} \in {{\cal Z}^l}\) and \( {s^l} \in {{\cal S}^l}\) , to output a execution-action \( a^l \in {{\cal A}^l}\) . The timesteps are related by \( {T^h} = n{T^l}\) , where \( n\) is determined by the termination function \( \beta\) , i.e., \( {\arg _i}\left[ {\beta ( {z_{t + i{T^l}}^l} ) = 1} \right]\) . Here, \( z_{t + i{T^l}}^l\) is required for the \( i\) -th output of \( {\pi ^l}\) after receiving \( \cal G\) . Thus, each high-level motion guidance corresponds to a variable-length sequence of low-level control commands.

The safety mechanism for the hierarchical policy includes: 1) safety evaluation module, 2) high- and low-level independent safety correction module, and 3) the safety-aware termination function. The safety evaluation module crosses both levels and generates risk severity of motion guidance \( {\cal K}( {{\cal G}( {o,{a^h}} ),s} )\) at different timescales. The safety correction module fuses \( {\cal K}\) with action values to produce safer actions \( o^{[s]}\) , \( a^{h,[s]}\) , and \( a^{l,[s]}\) . The safety-aware termination function \( \beta\) combines the safety evaluation results from both levels, prioritizing high-level correction to further enhance safety.

In high-MDP, the state-action value function for the optimal high-level policy is defined by the following Bellman optimality equation:

where \( r_{t + {T^h}}^h \in {\cal R}^h\) is given by:

where \( r_{t + i{T^l}}^l \in {\cal R}^l\) is the feedback reward from the environment for the \( i\) -th action of \( \pi^l\) . The violation flag function \( f_v\) is set to 1 in case of agent violations (e.g., vehicle collisions) and 0 otherwise. Introducing \( f_v\) prevents the dilution of violation-related rewards \( {\cal R}_{vio}\) by expectation-seeking operations, ensuring that the high-level policy maintains a strong emphasis on violations. The definition of \( {\cal R}_{vio}\) is provided in Sec. 4.1.

However, finding the optimal \( a^h\) in a hybrid action space is challenging. Following the idea of parameterized actor-critic, the high-level policy \( \pi^h\) outputs \( a^h\) through the cooperation between a deterministic policy network \( \mu^h(s^h; \theta^h)\) and a value network \( Q^h(s^h, o, a^h; \omega^h)\) . Details of this cooperation can be found in our previous work [13]. Thus, with \( {\pi ^h} = ( { \cdot \left| {{\mu ^h}( { \cdot {\rm{ }};{\theta ^h}} ),{Q^h}( { \cdot {\rm{ }};{\omega ^h}} ),{s^h}} \right.} )\) , the optimal state-action value function can be rewritten as:

This function’s solution is the optimal guidance-action \( (o, a^h)\) .

The \( (o, a^h)\) explicitly represents the motion guidance \( {\cal G}\) through a bi-directional mapping:

where \( \Psi\) is an explicit representation function, depending on the practical significance of \( (o, a^h)\) and \( \cal G\) . A generalized example for AD is provided in Sec. 4.1.

In low-MDP, \( {\cal G}(o, a^h)\) is provided every \( T^h\) , while the low-level policy \( \pi^l\) acquires the observed extend-state \( z_{t + i{T^l}}^l\) at much shorter timestep \( T^l\) . As a result, the high-level output observed from the low-level perspective may change, as illustrated in Figure 2 with a lane-change scenario. Directly using the fixed \( {\cal G}(o, a^h)\) in constructing \( z_{t + i{T^l}}^l\) would introduce state inconsistencies, hindering stable training of \( \pi^l\) . To address this, motion guidance is incrementally updated at each short timestep using physical information, allowing the guidance-action to be naturally updated:

where the superscript \( u\) denotes the variable has undergone incremental updating. Accordingly, the actual low-level extend-state becomes \( z_{t + i{T^l}}^l = ( {s_{t + i{T^l}}^l,{\cal G}_{t + i{T^l}}^{\left[ u \right]}} )\) . Since motion guidance is environment-specific, incremental updating using physical information is straightforward to implement. An example is provided in Sec. 4.1.

Therefore, in low-MDP, the state-action value function for the optimal low-level policy is given by the following Bellman optimality equation:

where \( U( {z_{t + {T^l}}^l} )\) is the optimal state value function. Since \( \pi^l\) outputs continuous actions and is compatible with any optimization algorithm, it can be directly approximated using a policy network \( {\mu ^l}( {{z^l}{\rm{ }};{\theta ^l}} )\) . Meanwhile, a value network \( {Q^l}( {z_t^l,a_t^l;{\omega ^l}} )\) is introduced to estimate the state-action value.

In lanes that are discrete laterally but continuous longitudinally, the hybrid action space \( {\cal H}^h\) of the high-level policy is defined as follows:

where \( w_r\) is the lane width, \( {R_0}\) is the minimum turning radius, \( {2a_{\max }^-}\) is the maximum braking acceleration, and \( v_e\) is the EV’s speed. The \( {\cal O}\) allows \( o\) to represent lane selection, restricting the target to the current or adjacent lane. Given the EV’s state and kinematics, the \( {\cal A}^h\) allows \( a^h\) to represent the selection of a feasible target location within the chosen lane. Together, \( (o, a^h)\) specifies a target point without further processing, which can be considered as a coarse-grained motion guidance.

To provide more comprehensive and detailed guidance for low-level policy, a finer-grained motion guidance is designed using polynomial curve: \( {\cal G} = {\arg _{(x_j, y_j)}}[ y_j = \sum_{m=0}^5 c_m x_j^m ]\) , where \( j \in [1, …, g]\) . Here, \( \cal G\) is a set of \( g\) target points lying on a fifth-degree polynomial parameterized by coefficients \( c_m\) . In the Frenet frame, with the EV at the origin, \( x_j\) and \( y_j\) are the longitudinal and lateral coordinates of the \( j\) -th target point. The start point is set by the EV’s current state: \( (x_1, y_1) = (x_e, y_e)\) , with heading angle \( \varphi_1 = \varphi_e\) . The end point is given by \( \pi^h\) : \( (x_g, y_g) = (o, a^h)\) , with \( \varphi_g\) from lane information. Solving the resulting linear system yields the coefficients \( c_m\) [13]. The mapping \( (o, a^h) \equiv (x_g, y_g) \in {\cal G}\) defines an explicit representation function \( \Psi\) , which satisfies Eq. 4 given the current EV and road states.

In timestep \( T^h\) , the frenet frame moves with EV. Since the points in \( \cal G\) are defined relative to previous frenet frame, their coordinates must be updated. A simple coordinate transformation yields the updated set \( {{\cal G}^{[u]}}\) , corresponding to Eq. 5.

The control commands required for EV driving are acceleration and steering angle. Thus, the execution action output by \( \pi^l\) consists of two items: \( a^l = (\delta_e, a_e)\) . According to general vehicle kinematics, the low-level action space is defined as:

Both \( \pi^h\) and \( \pi^l\) should account for the states of the EV and surrounding vehicles (SVs) in adjacent lanes. Thus, the state space is defined as:

The EV state includes lane ID, longitudinal and lateral positions, heading angle, and longitudinal and lateral speeds. SVs ahead of and behind the EV in the current and adjacent lanes are considered, with up to six SVs in total. Each SV state includes a presence flag, relative longitudinal and lateral positions, relative heading angle, and relative longitudinal and lateral speeds. The EV only considers SVs within the observation range \( \Delta x \in [-80\,\mathrm{m},\,160\,\mathrm{m}]\) .

The reward function is designed to consider driving safety, efficiency, and action consistency:

where the weights for each term reflect its relative importance. In safety reward \( {{\cal R}_s}\) , \( f_v = 1\) indicates an EV violation, such as road departure or collision with an SV. The results from the safety evaluation module are also included. In efficiency reward \( {{\cal R}_e}\) , the EV is encouraged to reach the target speed \( v_*\) , while speeds below the threshold \( v_p\) are penalized. In action consistency reward \( {{\cal R}_c}\) , fluctuations in steering and acceleration commands are penalized to promote smoothness.

Artificial Potential Fields (APF) integrate discrete events into a unified field over high-dimensional observations [26], providing reliable and expressive measures of driving risk severity. Based on APF, we propose a risk severity evaluation model for \( \cal G\) :

where \( {\rho _j^k}\) is the risk potential field at the \( j\) -th point relative to the \( k\) -th SV. Since points closer to the EV are more critical, the importance at the \( i\) -th point is: \( {{\cal I}_j} = 1 - {e^{{K_r}\left( {j - g} \right)}}\) , where \( K_r\) is a decay rate coefficient. Additionally, \( {\rho _j^k}\) is defined as:

where \( {w_1} \in [0.5,1]\) and \( {w_1} + {w_2} = 1\) . In addition, \( {X_s}\) and \( {Y_s}\) are the minimum safe distances in the longitudinal and lateral directions, respectively. The \( \Delta x_j^k\) and \( \Delta y_j^k\) are the longitudinal and lateral distances of the \( j\) -th point relative to the \( k\) -th SV.

To obtain \( a^{l,r}\) and simplify the design, we combine the widely used Intelligent Driver Model (IDM) and Stanley path-tracking algorithm as a conservative prior control model. Specifically, IDM determines the acceleration based on environment, while Stanley algorithm computes the steering angle according to motion guidance.

The training scenario is built in Highway-Env with three defined lanes [33]. At the start of each episode, the EV and SVs are randomly placed in any lane with random initial speeds. SVs follow the IDM and MOBIL models, allowing lane changes to reach their target speeds, which may interfere with the EV. The vehicle capacity (V/C) ratio, representing traffic density.

With this setup, training is conducted for 2,000 episodes using five seeds. Testing is then performed over 100 episodes on both Highway-Env and the HighD dataset [34], with each episode limited to 100s. Notably, the traffic density in Highway-Env is set to 0.3, presenting a challenging scenario.

We select several popular RL-based AD methods as baselines, which have different policy structures. PPO directly outputs control commands. The General Hierarchical method (Gen-H-RL) uses a high-level RL policy to select discrete behaviors for a rule-based low-level controller. Classical Hierarchical RL (Class-HRL) combines a high-level value-based policy with a low-level actor-critic structure. RL-PTA have an implicit hierarchical policy structure with fixed timescale. MHRL-I and Skill-Critic are preliminary methods exploring multi-timescale RL policy training, with discrete and continues high-level outputs, respectively. In contrast, our method features a more advanced hierarchical policy structure: P-AC is used to generate Hybrid action-based motion guidance (path points), and a supporting Safety mechanism is also included. The details of all methods are shown in Table 1. For all MT-based methods, the high- and low-level outputs operate at timesteps of 1s and 0.1s, respectively, as this has proven effective [6].

To comprehensively evaluate the performance of each driving policy, we select key metrics across four aspects:

The total reward curves for all methods during training are shown in Fig. 3. All methods converge after 1,800 episodes. Our proposed MTHRL-HS achieves the highest reward, indicating superior driving performance.

In Fig. 3(a), PPO converges more slowly and achieve significantly lower rewards than methods using hierarchical techniques, suggesting that direct control output hinders effective policy learning. Gen-H-RL, which applies RL only at the high level, converges fastest but yields relatively lower rewards. In contrast, Class-HRL uses RL at both levels, leading to marginally improved performance yet slightly slower convergence. The methods in Fig. 3(b) generally benefit from more advanced hierarchical structures, resulting in better policies. RL-PTA, while implicitly hierarchical with a fixed timescale, achieves strong performance with less fluctuation across different seeds. MHRL-I/Skill-Critic perform significantly worse than MTHRL-H, demonstrating that hybrid action-based motion guidance, which better aligns with road structure, leads to superior policies compared to purely continuous or discrete actions. Furthermore, MTHRL-HS, with its safety mechanism, further enhances policy performance.

The testing results further show that different methods produce distinct driving behaviors, as detailed in Table 2. Notably, our method achieves the highest TR, consistent with the training results. Compared to RL-PTA, which has the second-highest TR, MTHRL-H improves TR by 4.4%, and with the safety mechanism ‘S’, this improvement increases to 9.5%. After analyzing all the metrics, we provide additional details for some key metrics of the well-performing methods. The joint distribution of acceleration and speed is shown in Fig. 4, while the joint distribution of steering angle and CDD is shown in Fig. 5. The Fig. 6 further presents the distributions of TTC-C and TTC-T as boxplots.

For driving efficiency, MTHRL-H achieves the highest DS and TLC, with DS improved by 13.1% over the suboptimal method. This indicates that the multi-timescale hierarchical policy enables more flexible lane-changing behaviors to enhance driving efficiency, while the introduction of safety mechanism does not significantly compromise this performance. More specifically, under the same testing environment, Fig. 4 shows that the peak speed distributions for each method correspond to two potential cases: one near 20 m/s, representing opportunities for overtaking to reach the target speed; and another near 10 m/s, indicating the EV must follow SVs due to traffic congestion. Notably, MTHRL-HS achieves better lane-change timing in the former case, yielding the highest peak speed distribution. In the latter case, it effectively chooses to follow a faster SV, shifting the peak speed distribution upward. In contrast, other methods show lower driving efficiency, with worse DS, TLC, and speed distributions. In particular, PPO, lacking hierarchical structures, tend to adopt overly conservative following policies.

For driving action consistency, MTHRL-H shows the lowest means and standard deviations for AS, AA, and CDD, which are 50.0%, 26.2%, and 22.2% lower than those of the suboptimal method, respectively. The safety mechanism does not adversely affect these metrics. As shown in Fig. 4 and Fig. 5, our method produces acceleration, steering angle, and CDD values more tightly centered around 0. Even during lane changes, while CDD increases, steering angles remain smaller than with other methods. This suggests that high-level hybrid action-based motion guidance improves action consistency. It reduces fluctuations in control commands, resulting in smoother longitudinal and lateral driving behaviors. In contrast, PPO, which directly outputs control commands, lead to more fluctuating behaviors and make it harder to keep the EV on the centerline. Among hierarchical methods, generating finer-grained path points at the high level yields greater action consistency than discrete behavior generation.

For driving safety, excluding the over-conservative PPO, MTHRL-H reduces the CR by 22.2% compared to the suboptimal method. With the safety mechanism, the reduction in CR increases to 66.7%, clearly demonstrating its effectiveness in improving safety. Additionally, TTC-C and TTC-T reflect the policy’s ability in ensuring safer driving during interactions with SVs. As shown in Table 2 and Fig. 6, MTHRL-HS achieves the highest TTC-C and TTC-T, indicating a more cautious driving style. Meanwhile, the safety mechanism results in a significantly higher TTC-T than TTC-C, highlighting its ability to guide the EV into safer lanes.

The metrics in Table 3 show the driving performance of each method on the HighD dataset. Compared to the Highway-Env scenario, all methods perform better due to lower traffic density in HighD. MTHRL-H and MTHRL-HS remain the top performers overall, despite a smaller TR gap with other methods, demonstrating strong adaptability and robust policy performance in real traffic. Benefiting from the multi-timescale hierarchical architecture and hybrid action-based motion guidance, they lead in both efficiency and action consistency. Specifically, DS remains the highest, with only minor impact from the safety mechanism, while AS, AA, and CDD stay at the lowest levels. For safety, excluding the over-conservative PPO, MTHRL-HS further reduces the CR to 0.02% by leveraging the hierarchical safety mechanism. Therefore, above results confirm the superiority of our method across all driving metrics and its strong potential for real-world deployment.