Performance evaluation of direct heuristic dynamic programming using control-theoretic measures

Lei Yang, Jennie Si, Konstantinos Tsakalis, Armando Rodriguez

Research output: Contribution to journalArticlepeer-review

14 Scopus citations


Approximate dynamic programming (ADP) has been widely studied from several important perspectives: algorithm development, learning efficiency measured by success or failure statistics, convergence rate, and learning error bounds. Given that many learning benchmarks used in ADP or reinforcement learning studies are control problems, it is important and necessary to examine the learning controllers from a control-theoretic perspective. This paper makes use of direct heuristic dynamic programming (direct HDP) and three typical benchmark examples to introduce a unique analytical framework that can be applied to other learning control paradigms and other complex control problems. The sensitivity analysis and the linear quadratic regulator (LQR) design are used in the paper for two purposes: to quantify direct HDP performances and to provide guidance toward designing better learning controllers. The use of LQR however does not limit the direct HDP to be a learning controller that addresses nonlinear dynamic system control issues. Toward this end, applications of the direct HDP for nonlinear control problems beyond sensitivity analysis and the confines of LQR have been developed and compared whenever appropriate to an LQR design.

Original languageEnglish (US)
Pages (from-to)177-201
Number of pages25
JournalJournal of Intelligent and Robotic Systems: Theory and Applications
Issue number2-3
StatePublished - Jul 2009


  • Approximate dynamic programming (ADP)
  • Direct heuristic dynamic programming (direct HDP)
  • Linear quadratic regulator (LQR)
  • On-line learning control
  • Sensitivity and complementary sensitivity

ASJC Scopus subject areas

  • Software
  • Control and Systems Engineering
  • Mechanical Engineering
  • Industrial and Manufacturing Engineering
  • Artificial Intelligence
  • Electrical and Electronic Engineering


Dive into the research topics of 'Performance evaluation of direct heuristic dynamic programming using control-theoretic measures'. Together they form a unique fingerprint.

Cite this