Blog

The latest from Google Research

Off-Policy Estimation for Infinite-Horizon Reinforcement Learning

Friday, April 17, 2020

Posted by Ali Mousavi, AI Resident and Lihong Li, Research Scientist, Google Researchreinforcement learninglearn good policiesroboticsrecommendation systemsnewtarget policybehavior policy more critical

Agent-environment interaction in reinforcement learning. At each step, an agent takes an action based on a policy, receives a reward and makes a transition to a new state.

Pongbias

In off-policy evaluation, unlike the behavior policy, we do not have any data from the target policy. Therefore, we cannot compute the expected reward of the target policy without using information from the behavior policy.

Black-Box Off-Policy Estimation for Infinite-Horizon Reinforcement LearningICLR 2020a prioriBackgrounddata trajectoryoften grows exponentiallyinfinite-horizon problemOur Approach for Infinite-Horizon RLimportance samplingprobability distributionindividualstationary distributionour paperbackward operatorExperimental ResultsModelWin IPSroot-mean-square error

Left: The ModelWin environment with three states (s₁-s₃) and two actions (a₁, a₂). Right: RMSE of different approaches on the ModelWin problem in logarithmic scale. Our approach has converged very quickly in this simple problem, compared to the previous state-of-the-art method (IPS) that needs longer horizon length to converge.

CartPolePendulumMountainCar

Comparison of different methods on three environments: Cartpole, Pendulum, and Mountaincar. The left column shows the environments. The right column shows the log of the RMSE with respect to the target policy reward as the number of trajectories collected by the behavior policy changes. The results are based on 50 runs.