logo
banner

Journals & Publications

Journals Publications Papers

Papers

Discrete-Time Deterministic Q-Learning: A Novel Convergence Analysis
Jul 18, 2017Author:
PrintText Size A A

Title: Discrete-Time Deterministic Q-Learning: A Novel Convergence Analysis

 Authors: Wei, QL; Lewis, FL; Sun, QY; Yan, PF; Song, RZ

 Author Full Names: Wei, Qinglai; Lewis, Frank L.; Sun, Qiuye; Yan, Pengfei; Song, Ruizhuo

 Source: IEEE TRANSACTIONS ON CYBERNETICS, 47 (5):1224-1237; 10.1109/TCYB.2016.2542923 MAY 2017

 Language: English

 Abstract: In this paper, a novel discrete-time deterministic Q-learning algorithm is developed. In each iteration of the developed Q-learning algorithm, the iterative Q function is updated for all the state and control spaces, instead of updating for a single state and a single control in traditional Q-learning algorithm. A new convergence criterion is established to guarantee that the iterative Q function converges to the optimum, where the convergence criterion of the learning rates for traditional Q-learning algorithms is simplified. During the convergence analysis, the upper and lower bounds of the iterative Q function are analyzed to obtain the convergence criterion, instead of analyzing the iterative Q function itself. For convenience of analysis, the convergence properties for undiscounted case of the deterministic Q-learning algorithm are first developed. Then, considering the discounted factor, the convergence criterion for the discounted case is established. Neural networks are used to approximate the iterative Q function and compute the iterative control law, respectively, for facilitating the implementation of the deterministic Q-learning algorithm. Finally, simulation results and comparisons are given to illustrate the performance of the developed algorithm.

 ISSN: 2168-2267

 eISSN: 2168-2275

 IDS Number: ES8HN

 Unique ID: WOS:000399797000009

*Click Here to View Full Record