Continuous Action-Space Reinforcement Learning Methods Applied to the Minimum-Time Swing-Up of the Acrobot

Barry D. Nichols

doi:10.1109/SMC.2015.364

Continuous Action-Space Reinforcement Learning Methods Applied to the Minimum-Time Swing-Up of the Acrobot

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Here I apply three reinforcement learning methods to the full, continuous action, swing-up acrobot control benchmark problem. These include two approaches from the literature: CACLA and NM-SARSA and a novel approach which I refer to as Nelder Mead-SARSA. Nelder Mead-SARSA, like NMSARSA, directly optimises the state-action value function for action selection, in order to allow continuous action reinforcement learning without a separate policy function. However, as it uses a derivative-free method it does not require the first or second partial derivatives of the value function. All three methods achieved good results in terms of swing-up times, comparable to previous approaches from the literature. Particularly Nelder Mead-SARSA, which performed the swing up in a shorter time than many approaches from the literature.

Original language	English
Title of host publication	2015 IEEE International Conference on Systems, Man, and Cybernetics
Publisher	Institute of Electrical and Electronics Engineers (IEEE)
Pages	2084-2089
Number of pages	6
ISBN (Print)	978-1-4799-8696-5
DOIs	https://doi.org/10.1109/SMC.2015.364
Publication status	Published - 12 Oct 2015
Event	2015 IEEE International Conference on Systems, Man, and Cybernetics - Hong Kong, China Duration: 9 Oct 2015 → 12 Oct 2015

Conference

Conference	2015 IEEE International Conference on Systems, Man, and Cybernetics
Period	9/10/15 → 12/10/15

Keywords

reinforcement learning
continuous action-space
computational intelligence
artificial neural networks
intelligent control

Access to Document

10.1109/SMC.2015.364

https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=7379496

Cite this

@inproceedings{f3a271d2426c4bd4aa4970bf88aa2dff,

title = "Continuous Action-Space Reinforcement Learning Methods Applied to the Minimum-Time Swing-Up of the Acrobot",

abstract = "Here I apply three reinforcement learning methods to the full, continuous action, swing-up acrobot control benchmark problem. These include two approaches from the literature: CACLA and NM-SARSA and a novel approach which I refer to as Nelder Mead-SARSA. Nelder Mead-SARSA, like NMSARSA, directly optimises the state-action value function for action selection, in order to allow continuous action reinforcement learning without a separate policy function. However, as it uses a derivative-free method it does not require the first or second partial derivatives of the value function. All three methods achieved good results in terms of swing-up times, comparable to previous approaches from the literature. Particularly Nelder Mead-SARSA, which performed the swing up in a shorter time than many approaches from the literature.",

keywords = "reinforcement learning, continuous action-space, computational intelligence, artificial neural networks, intelligent control",

author = "Nichols, {Barry D.}",

year = "2015",

month = oct,

day = "12",

doi = "10.1109/SMC.2015.364",

language = "English",

isbn = "978-1-4799-8696-5",

pages = "2084--2089",

booktitle = "2015 IEEE International Conference on Systems, Man, and Cybernetics",

publisher = "Institute of Electrical and Electronics Engineers (IEEE)",

address = "United States",

note = "2015 IEEE International Conference on Systems, Man, and Cybernetics ; Conference date: 09-10-2015 Through 12-10-2015",

}

Nichols, BD 2015, Continuous Action-Space Reinforcement Learning Methods Applied to the Minimum-Time Swing-Up of the Acrobot. in 2015 IEEE International Conference on Systems, Man, and Cybernetics., 7379496, Institute of Electrical and Electronics Engineers (IEEE), pp. 2084-2089, 2015 IEEE International Conference on Systems, Man, and Cybernetics, 9/10/15. https://doi.org/10.1109/SMC.2015.364

Continuous Action-Space Reinforcement Learning Methods Applied to the Minimum-Time Swing-Up of the Acrobot. / Nichols, Barry D.
2015 IEEE International Conference on Systems, Man, and Cybernetics. Institute of Electrical and Electronics Engineers (IEEE), 2015. p. 2084-2089 7379496.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Continuous Action-Space Reinforcement Learning Methods Applied to the Minimum-Time Swing-Up of the Acrobot

AU - Nichols, Barry D.

PY - 2015/10/12

Y1 - 2015/10/12

N2 - Here I apply three reinforcement learning methods to the full, continuous action, swing-up acrobot control benchmark problem. These include two approaches from the literature: CACLA and NM-SARSA and a novel approach which I refer to as Nelder Mead-SARSA. Nelder Mead-SARSA, like NMSARSA, directly optimises the state-action value function for action selection, in order to allow continuous action reinforcement learning without a separate policy function. However, as it uses a derivative-free method it does not require the first or second partial derivatives of the value function. All three methods achieved good results in terms of swing-up times, comparable to previous approaches from the literature. Particularly Nelder Mead-SARSA, which performed the swing up in a shorter time than many approaches from the literature.

AB - Here I apply three reinforcement learning methods to the full, continuous action, swing-up acrobot control benchmark problem. These include two approaches from the literature: CACLA and NM-SARSA and a novel approach which I refer to as Nelder Mead-SARSA. Nelder Mead-SARSA, like NMSARSA, directly optimises the state-action value function for action selection, in order to allow continuous action reinforcement learning without a separate policy function. However, as it uses a derivative-free method it does not require the first or second partial derivatives of the value function. All three methods achieved good results in terms of swing-up times, comparable to previous approaches from the literature. Particularly Nelder Mead-SARSA, which performed the swing up in a shorter time than many approaches from the literature.

KW - reinforcement learning

KW - continuous action-space

KW - computational intelligence

KW - artificial neural networks

KW - intelligent control

UR - https://ieeexplore.ieee.org/document/7379496/

U2 - 10.1109/SMC.2015.364

DO - 10.1109/SMC.2015.364

M3 - Conference contribution

SN - 978-1-4799-8696-5

SP - 2084

EP - 2089

BT - 2015 IEEE International Conference on Systems, Man, and Cybernetics

PB - Institute of Electrical and Electronics Engineers (IEEE)

T2 - 2015 IEEE International Conference on Systems, Man, and Cybernetics

Y2 - 9 October 2015 through 12 October 2015

ER -

Continuous Action-Space Reinforcement Learning Methods Applied to the Minimum-Time Swing-Up of the Acrobot

Abstract

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this