Milad Farsi, Jun Liu

Model-Based Reinforcement Learning

From Data to Continuous Actions with a Python-Based Toolbox
Herausgegeben:Di Benedetto, Maria Domenica

• Gebundenes Buch

Produktbeschreibung

Model-Based Reinforcement Learning

Explore a comprehensive and practical approach to reinforcement learning

Reinforcement learning is an essential paradigm of machine learning, wherein an intelligent agent performs actions that ensure optimal behavior from devices. While this paradigm of machine learning has gained tremendous success and popularity in recent years, previous scholarship has focused either on theory--optimal control and dynamic programming - or on algorithms--most of which are simulation-based.

Model-Based Reinforcement Learning provides a model-based framework to bridge these two aspects, thereby creating a holistic treatment of the topic of model-based online learning control. In doing so, the authors seek to develop a model-based framework for data-driven control that bridges the topics of systems identification from data, model-based reinforcement learning, and optimal control, as well as the applications of each. This new technique for assessing classical results will allow for a more efficient reinforcement learning system. At its heart, this book is focused on providing an end-to-end framework--from design to application--of a more tractable model-based reinforcement learning technique.

Model-Based Reinforcement Learning readers will also find:
_ A useful textbook to use in graduate courses on data-driven and learning-based control that emphasizes modeling and control of dynamical systems from data
_ Detailed comparisons of the impact of different techniques, such as basic linear quadratic controller, learning-based model predictive control, model-free reinforcement learning, and structured online learning
_ Applications and case studies on ground vehicles with nonholonomic dynamics and another on quadrator helicopters
_ An online, Python-based toolbox that accompanies the contents covered in the book, as well as the necessary code and data

Model-Based Reinforcement Learning is a useful reference for senior undergraduate students, graduate students, research assistants, professors, process control engineers, and roboticists.

---

Hinweis: Dieser Artikel kann nur an eine deutsche Lieferadresse ausgeliefert werden.

Produktdetails

• Wiley-IEEE Press Book Series on Control Systems Theory and Applications
• Verlag: Wiley & Sons / Wiley-IEEE Press
• Artikelnr. des Verlages: 1W119808570
• 1. Auflage
• Seitenzahl: 272
• Erscheinungstermin: 28. Dezember 2022
• Englisch
• Abmessung: 235mm x 157mm x 19mm
• Gewicht: 520g
• ISBN-13: 9781119808572
• ISBN-10: 111980857X
• Artikelnr.: 62234376

Autorenporträt

Milad Farsi received the B.S. degree in Electrical Engineering (Electronics) from the University of Tabriz in 2010. He obtained his M.S. degree also in Electrical Engineering (Control Systems) from the Sahand University of Technology in 2013. Moreover, he gained industrial experience as a Control System Engineer between 2012 and 2016. Later, he acquired the Ph.D. degree in Applied Mathematics from the University of Waterloo, Canada, in 2022, and he is currently a Postdoctoral Fellow at the same institution. His research interests include control systems, reinforcement learning, and their applications in robotics and power electronics. Jun Liu received the Ph.D. degree in Applied Mathematics from the University of Waterloo, Canada, in 2010. He is currently an Associate Professor of Applied Mathematics and a Canada Research Chair in Hybrid Systems and Control at the University of Waterloo, Canada, where he directs the Hybrid Systems Laboratory. From 2012 to 2015, he was a Lecturer in Control and Systems Engineering at the University of Sheffield. During 2011 and 2012, he was a Postdoctoral Scholar in Control and Dynamical Systems at the California Institute of Technology. His main research interests are in the theory and applications of hybrid systems and control, including rigorous computational methods for control design with applications in cyber-physical systems and robotics.

Inhaltsangabe

About the Authors xi

Preface xiii

Acronyms xv

Introduction xvii

1 Nonlinear Systems Analysis 1

1.1 Notation 1

1.2 Nonlinear Dynamical Systems 2

1.2.1 Remarks on Existence, Uniqueness, and Continuation of Solutions 2

1.3 Lyapunov Analysis of Stability 3

1.4 Stability Analysis of Discrete Time Dynamical Systems 7

1.5 Summary 10

Bibliography 10

2 Optimal Control 11

2.1 Problem Formulation 11

2.2 Dynamic Programming 12

2.2.1 Principle of Optimality 12

2.2.2 Hamilton-Jacobi-Bellman Equation 14

2.2.3 A Sufficient Condition for Optimality 15

2.2.4 Infinite-Horizon Problems 16

2.3 Linear Quadratic Regulator 18

2.3.1 Differential Riccati Equation 18

2.3.2 Algebraic Riccati Equation 23

2.3.3 Convergence of Solutions to the Differential Riccati Equation 26

2.3.4 Forward Propagation of the Differential Riccati Equation for Linear Quadratic Regulator 28

2.4 Summary 30

Bibliography 30

3 Reinforcement Learning 33

3.1 Control-Affine Systems with Quadratic Costs 33

3.2 Exact Policy Iteration 35

3.2.1 Linear Quadratic Regulator 39

3.3 Policy Iteration with Unknown Dynamics and Function Approximations 41

3.3.1 Linear Quadratic Regulator with Unknown Dynamics 46

3.4 Summary 47

Bibliography 48

4 Learning of Dynamic Models 51

4.1 Introduction 51

4.1.1 Autonomous Systems 51

4.1.2 Control Systems 51

4.2 Model Selection 52

4.2.1 Gray-Box vs. Black-Box 52

4.2.2 Parametric vs. Nonparametric 52

4.3 Parametric Model 54

4.3.1 Model in Terms of Bases 54

4.3.2 Data Collection 55

4.3.3 Learning of Control Systems 55

4.4 Parametric Learning Algorithms 56

4.4.1 Least Squares 56

4.4.2 Recursive Least Squares 57

4.4.3 Gradient Descent 59

4.4.4 Sparse Regression 60

4.5 Persistence of Excitation 60

4.6 Python Toolbox 61

4.6.1 Configurations 62

4.6.2 Model Update 62

4.6.3 Model Validation 63

4.7 Comparison Results 64

4.7.1 Convergence of Parameters 65

4.7.2 Error Analysis 67

4.7.3 Runtime Results 69

4.8 Summary 73

Bibliography 75

5 Structured Online Learning-Based Control of Continuous-Time Nonlinear Systems 77

5.1 Introduction 77

5.2 A Structured Approximate Optimal Control Framework 77

5.3 Local Stability and Optimality Analysis 81

5.3.1 Linear Quadratic Regulator 81

5.3.2 SOL Control 82

5.4 SOL Algorithm 83

5.4.1 ODE Solver and Control Update 84

5.4.2 Identified Model Update 85

5.4.3 Database Update 85

5.4.4 Limitations and Implementation Considerations 86

5.4.5 Asymptotic Convergence with Approximate Dynamics 87

5.5 Simulation Results 87

5.5.1 Systems Identifiable in Terms of a Given Set of Bases 88

5.5.2 Systems to Be Approximated by a Given Set of Bases 91

5.5.3 Comparison Results 98

5.6 Summary 99

Bibliography 99

6 A Structured Online Learning Approach to Nonlinear Tracking with Unknown Dynamics 103

6.1 Introduction 103

6.2 A Structured Online Learning for Tracking Control 104

6.2.1 Stability and Optimality in the Linear Case 108

6.3 Learning-based Tracking Control Using SOL 111

6.4 Simul