Title: Discrete-Time Deterministic Q-Learning: A Novel Convergence Analysis
|
| Authors: Wei, QL; Lewis, FL; Sun, QY; Yan, PF; Song, RZ
|
| Author Full Names: Wei, Qinglai; Lewis, Frank L.; Sun, Qiuye; Yan, Pengfei; Song, Ruizhuo
|
| Source: IEEE TRANSACTIONS ON CYBERNETICS, 47 (5):1224-1237; 10.1109/TCYB.2016.2542923 MAY 2017
|
| Language: English
|
| Abstract: In this paper, a novel discrete-time deterministic Q-learning algorithm is developed. In each iteration of the developed Q-learning algorithm, the iterative Q function is updated for all the state and control spaces, instead of updating for a single state and a single control in traditional Q-learning algorithm. A new convergence criterion is established to guarantee that the iterative Q function converges to the optimum, where the convergence criterion of the learning rates for traditional Q-learning algorithms is simplified. During the convergence analysis, the upper and lower bounds of the iterative Q function are analyzed to obtain the convergence criterion, instead of analyzing the iterative Q function itself. For convenience of analysis, the convergence properties for undiscounted case of the deterministic Q-learning algorithm are first developed. Then, considering the discounted factor, the convergence criterion for the discounted case is established. Neural networks are used to approximate the iterative Q function and compute the iterative control law, respectively, for facilitating the implementation of the deterministic Q-learning algorithm. Finally, simulation results and comparisons are given to illustrate the performance of the developed algorithm.
|
| ISSN: 2168-2267
|
| eISSN: 2168-2275
|
| IDS Number: ES8HN
|
| Unique ID: WOS:000399797000009
*Click Here to View Full Record
