Table 2. The algorithm process of... | Emerald Publishing

Table 2.

The algorithm process of collision-avoidance strategy for ASs based on the DDPG algorithm


Initialise critic and actor network parameters: θ^Q and θ^μ
Initialise the critic and actor network target networks and directly copy parameters: $θ^{Q′} \leftarrow θ^{Q}, θ^{μ′} \leftarrow θ^{μ}$
Initialise the experience pool R
1: For episode = 1, M:
2: Initialise random process N
3: Obtain the initial state of the AS s₁
4: For t = 1, T:
5: Choose an action based on the current policy and random noise: $a_{t} = μ (s_{t} \| θ^{μ}) + N_{t}$
6: Perform action a_t, get instant reward r_t and next state s_t₊₁
7: Store state/action pairs (s_t, a_t, r_t, s_t₊₁) into experience pool R
8: Randomly sample K state/action pairs (s_t, a_t, r_t, s_t₊₁) from the experience pool R
9: Compute the ‘label value’ of the critical network: $y_{i} = r_{i} + γ Q' (s_{i + 1}, μ' (s_{i + 1} ∣ θ^{μ′}) ∣ θ^{Q′})$
10: Calculate the error of the critical network $L = \frac{1}{K} \sum_{i} {(y_{i} - Q (s_{i}, a_{i} ∣ θ^{Q}))}^{2}$ and use gradient descent to update critical network parameters
11: Calculate the gradient of the actor network and update the actor network parameters using gradient ascent: ${{\nabla_{θ^{μ}} J \approx \frac{1}{K} \sum_{i} \nabla_{a} Q (s, a ∣ θ^{Q}) \|}_{s = s_{i}, a = μ (s_{i})} \nabla_{θ^{μ}} μ (s ∣ θ^{μ}) \|}_{s_{i}}$
12: Update the parameters of the critic and actor network target network: $\begin{array}{l} θ^{Q′} \leftarrow τ θ^{Q} + (1 - τ) θ^{Q′} \\ θ^{π′} \leftarrow τ θ^{π} + (1 - τ) θ^{π′} \end{array}$
13: End for
14: End for


Initialise critic and actor network parameters: θ^Q and θ^μ
Initialise the critic and actor network target networks and directly copy parameters: $θ^{Q′} \leftarrow θ^{Q}, θ^{μ′} \leftarrow θ^{μ}$
Initialise the experience pool R
1: For episode = 1, M:
2: Initialise random process N
3: Obtain the initial state of the AS s₁
4: For t = 1, T:
5: Choose an action based on the current policy and random noise: $a_{t} = μ (s_{t} \| θ^{μ}) + N_{t}$
6: Perform action a_t, get instant reward r_t and next state s_t₊₁
7: Store state/action pairs (s_t, a_t, r_t, s_t₊₁) into experience pool R
8: Randomly sample K state/action pairs (s_t, a_t, r_t, s_t₊₁) from the experience pool R
9: Compute the ‘label value’ of the critical network: $y_{i} = r_{i} + γ Q' (s_{i + 1}, μ' (s_{i + 1} ∣ θ^{μ′}) ∣ θ^{Q′})$
10: Calculate the error of the critical network $L = \frac{1}{K} \sum_{i} {(y_{i} - Q (s_{i}, a_{i} ∣ θ^{Q}))}^{2}$ and use gradient descent to update critical network parameters
11: Calculate the gradient of the actor network and update the actor network parameters using gradient ascent: ${{\nabla_{θ^{μ}} J \approx \frac{1}{K} \sum_{i} \nabla_{a} Q (s, a ∣ θ^{Q}) \|}_{s = s_{i}, a = μ (s_{i})} \nabla_{θ^{μ}} μ (s ∣ θ^{μ}) \|}_{s_{i}}$
12: Update the parameters of the critic and actor network target network: $\begin{array}{l} θ^{Q′} \leftarrow τ θ^{Q} + (1 - τ) θ^{Q′} \\ θ^{π′} \leftarrow τ θ^{π} + (1 - τ) θ^{π′} \end{array}$
13: End for
14: End for