AC ＆ A2C ＆ A3C

Basic concepts

Actor-Critic（AC）

AC The algorithm is based on both value function 、 Algorithm based on strategy function . The algorithm based on value function , It mainly refers to the value of all actions output by the algorithm itself , Choose the action according to the highest value , This kind of algorithm cannot select continuous actions . The algorithm based on value function refers to the probability that the algorithm itself outputs various actions to be taken in the next step , Then choose the action according to the probability .

A simple AC Algorithm （QAC） Strategy gradient learning is divided into two parts ：
1、Critic： Use state action value function $q_{\omega }(s,a)$, Using linear Q- Value function （$\varphi (s,a{)}^T\omega$） Close to . And use $TD(\lambda )$ To update the parameters $\omega$;
2、Actor： Use Critic Obtained value function $q_{\omega }(s,a)$ Boot policy function parameters θ Update .

QAC The pseudo code of the algorithm is shown in the figure below ：

Advanced Actor-Critic（A2C）

stay AC in ,Critic Use the state action value function $q_{\omega }(s,a)$ The corresponding policy gradient is ：

stay A2C in , Subtract the baseline function when defining the objective function $B(s)$, This can reduce the variance . At this point, the policy gradient is ：

take $B(s)=v_{{\pi }_{\theta }}(s)$,$A^{{\pi }_{\theta }}(s,a)=q_{\omega }(s,a)-v_{{\pi }_{\theta }}(s)$, among $A^{{\pi }_{\theta }}(s,a)$ It's called the dominance function , It represents a state - Action pairs relative to the average state - How good or bad the action is . If the dominance function is greater than zero , It means that the action is better than the average action , If the dominance function is less than zero , It means that the current action is not as good as the average action .

Actor In the parameter $\theta$ The update method of is ：

Critic Parameters are no longer used in $\omega$, Instead, use parameters $v$. Parameters $v$ The update method of is :

among :
,

Asynchronous Advantage Actor-Critic（A3C)