Improving Learning for Embodied Agents in Dynamic--Environments by State Factorisation

A new reinforcement learning algorithm designed--specifically for robots and embodied systems--is described. Conventional reinforcement learning methods intended for learning general tasks suffer from a number of disadvantages in this domain including slow learning speed, an inability--to generalise between states, reduced performance--in dynamic environments, and a lack of scalability. Factor-Q, the new algorithm, uses factorised state and action, coupled with multiple structured rewards, to address these issues. Initial experimental results demonstrate that Factor-Q is able to learn as efficiently in dynamic as in static environments, unlike conventional methods. Further, in the specimen task,--obstacle avoidance is improved by over two orders--of magnitude compared with standard Qlearning.