Process empowerment for robust intrinsic motivation

Information processing in dynamical control systems influences the properties of the perception-action loop in natural and artificial agents. The ability to causally affect environment by agent’s actions is crucial for learning meaningful behavior and survival. Empowerment is an information-theoretic approach to intrinsically discover this causality between actions and observations without externally provided domain expertise such as a reward function. This form of artificial intrinsic motivation has been successfully demonstrated to lead to the emergence of meaningful behavior in various domains ranging from robotics to transportation. The original formulation of the empowerment principle is based on the information flow from open-loop actions to future observations. This is not robust to randomness and unpredictable perturbations in environments with structures that require careful maneuvering. In this work we define a feedback-aware empowerment variant, called process empowerment and derive a solution given by self-consistent equations which can be used for its numerical evaluation. Process empowerment proves to be a robust intrinsic motivation in a paradigmatic proof-of-concept example (‘Windy Bridge’), and in scenarios with obstacles and noisy perturbation (‘Hallway’) and with occasional adversarial action by an oracle agent (‘Race’). It demonstrates superior robustness in dealing with noisy environments in delicate situations, and allows transferring solutions for deterministic problems into a noisy, disruptive and occasionally adversarial variant of the problem, through ‘empowerment cushioning’.