Grounding subgoals in information transitions

S.G. Van Dijk, D. Polani

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    16 Citations (Scopus)
    66 Downloads (Pure)

    Abstract

    In reinforcement learning problems, the construction of subgoals has been identified as an important step to speed up learning and to enable skill transfer. For this purpose, one typically extracts states from various saliency properties of an MDP transition graph, most notably bottleneck states. Here we introduce an alternative approach to this problem: assuming a family of MDPs with multiple goals but with a fixed transition graph, we introduce the relevant goal information as the amount of Shannon information that the agent needs to maintain about the current goal at a given state to select the appropriate action. We show that there are distinct transition states in the MDP at which new relevant goal information has to be considered for selecting the next action. We argue that these transition states can be interpreted as subgoals for the current task class, and we use these states to automatically create a hierarchical policy, according to the well-established Options model for hierarchical reinforcement learning.
    Original languageEnglish
    Title of host publicationProcs of 2011 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning
    Subtitle of host publicationADPRL 2011
    PublisherInstitute of Electrical and Electronics Engineers (IEEE)
    Pages105-111
    Number of pages7
    ISBN (Print)978-1-4244-9887-1
    DOIs
    Publication statusPublished - 1 Jan 2011
    Event2011 IEEE Symposium on Adaptive Dynamic Programming And Reinforcement Learning (ADPRL) - Paris, France
    Duration: 11 Apr 201115 Apr 2011

    Publication series

    NameSymposium Series on Computational Intelligence
    PublisherIEEE

    Conference

    Conference2011 IEEE Symposium on Adaptive Dynamic Programming And Reinforcement Learning (ADPRL)
    Country/TerritoryFrance
    CityParis
    Period11/04/1115/04/11

    Fingerprint

    Dive into the research topics of 'Grounding subgoals in information transitions'. Together they form a unique fingerprint.

    Cite this