Discrete-time dynamic graphical games:model-free reinforcement learning solution
Mohammed I ABOUHEAF
Frank L LEWIS
Magdi S MAHMOUD
Dariusz G MIKULSKI
摘要：This paper introduces a model-free reinforcement learning technique that is used to solve a class of dynamic games known as dynamic graphical games. The graphical game results from multi-agent dynamical systems, where pinning control is used to make all the agents synchronize to the state of a command generator or a leader agent. Novel coupled Bellman equations and Hamiltonian functions are developed for the dynamic graphical games. The Hamiltonian mechanics are used to derive the necessary conditions for optimality. The solution for the dynamic graphical game is given in terms of the solution to a set of coupled Hamilton-Jacobi-Bellman equations developed herein. Nash equilibrium solution for the graphical game is given in terms of the solution to the underlying coupled Hamilton-Jacobi-Bellman equations. An online model-free policy iteration algorithm is developed to learn the Nash solution for the dynamic graphical game. This algorithm does not require any knowledge of the agents’ dynamics. A proof of convergence for this multi-agent learning algorithm is given under mild assumption about the inter-connectivity properties of the graph. A gradient descent technique with critic network structures is used to implement the policy iteration algorithm to solve the graphical game online in real-time.
机标关键词：iteration algorithmreinforcement learningnecessary conditionsequilibrium solutionlearning algorithmdynamical systemsgradient descentdynamic games
资助基金：the Deanship of Scientific Research at King Fahd University of Petroleum & Minerals Project(JF141002)the National Science Foundation(ECCS-1405173)the Office of Naval Research()the Office of Naval Research( N000141310562)the Office of Naval Research( N000141410718)the U.S. Army Research Office(W911NF-11-D-0001)the National Natural Science Foundation of China(61120106011)the Project 111 from the Ministry of Education of China(B08015)
论文发表日期：2015-01-01
在线出版日期：2025-08-15（本平台首次上网日期，不代表文献的发表时间）
页数：15( 55-69 )
英文信息