TY - GEN
T1 - Grounding subgoals in information transitions
AU - Van Dijk, S.G.
AU - Polani, D.
N1 - “This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. In most cases, these works may not be reposted without the explicit permission of the copyright holder."
“Copyright IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE.”
PY - 2011/1/1
Y1 - 2011/1/1
N2 - In reinforcement learning problems, the construction of subgoals has been identified as an important step to speed up learning and to enable skill transfer. For this purpose, one typically extracts states from various saliency properties of an MDP transition graph, most notably bottleneck states. Here we introduce an alternative approach to this problem: assuming a family of MDPs with multiple goals but with a fixed transition graph, we introduce the relevant goal information as the amount of Shannon information that the agent needs to maintain about the current goal at a given state to select the appropriate action. We show that there are distinct transition states in the MDP at which new relevant goal information has to be considered for selecting the next action. We argue that these transition states can be interpreted as subgoals for the current task class, and we use these states to automatically create a hierarchical policy, according to the well-established Options model for hierarchical reinforcement learning.
AB - In reinforcement learning problems, the construction of subgoals has been identified as an important step to speed up learning and to enable skill transfer. For this purpose, one typically extracts states from various saliency properties of an MDP transition graph, most notably bottleneck states. Here we introduce an alternative approach to this problem: assuming a family of MDPs with multiple goals but with a fixed transition graph, we introduce the relevant goal information as the amount of Shannon information that the agent needs to maintain about the current goal at a given state to select the appropriate action. We show that there are distinct transition states in the MDP at which new relevant goal information has to be considered for selecting the next action. We argue that these transition states can be interpreted as subgoals for the current task class, and we use these states to automatically create a hierarchical policy, according to the well-established Options model for hierarchical reinforcement learning.
UR - http://www.scopus.com/inward/record.url?scp=80052250027&partnerID=8YFLogxK
U2 - 10.1109/ADPRL.2011.5967384
DO - 10.1109/ADPRL.2011.5967384
M3 - Conference contribution
AN - SCOPUS:80052250027
SN - 978-1-4244-9887-1
T3 - Symposium Series on Computational Intelligence
SP - 105
EP - 111
BT - Procs of 2011 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning
PB - Institute of Electrical and Electronics Engineers (IEEE)
T2 - 2011 IEEE Symposium on Adaptive Dynamic Programming And Reinforcement Learning (ADPRL)
Y2 - 11 April 2011 through 15 April 2011
ER -