Blog

The latest from Google Research

Simulated Policy Learning in Video Models

Monday, March 25, 2019

Posted by Łukasz Kaiser and Dumitru Erhan, Research Scientists, Google AIreinforcement learningclassic Atari 2600 gamesRecent workexploration regimesMontezuma's RevengeModel-Based Reinforcement Learning for Ataricodetensor2tensorLearning a SimPLe World Modelwell establishedrecentmodel-basedreinforcementlearningmethods

Main loop of SimPLe. 1) The agent starts interacting with the real environment. 2) The collected observations are used to update the current world model. 3) The agent updates the policy by learning inside the world model.

pixel spacetrajectoriesrealfeedforward convolutional networkPongpreviouswork

One example of an issue arising from stochasticity is seen when the SimPLe model is applied to Kung Fu Master. In the animation, the left is the output of the model, the middle is the groundtruth, and the right panel is the pixel-wise difference between the two. Here the model's predictions deviate from the real game by spawning a different number of opponents.

Proximal Policy Optimization (PPO)FreewaySimPLe Efficiency Rainbow

The number of interactions needed by the respective model-free algorithms (left - Rainbow; right - PPO) to match the score achieved using our SimPLe training method. The red line indicates the number of interactions used by our method.

SimPLe SuccessFreeway

Breakout

Nearly pixel perfect predictions can be made by SimPLe, on Breakout (top) and Freeway (bottom). In each animation, the left is the output of the model, the middle is the groundtruth, and the right is the pixel-wise difference between the two.

SimPLe SurprisesAtlantisBattlezonePrivate Eye

In Battlezone, we find the model struggles with predicting small, relevant parts, such as the bullet.

ConclusionrepositorycolabAcknowledgementsThis work was done in collaboration with the University of Illinois at Urbana-Champaign, the University of Warsaw and deepsense.ai. We would like to give special recognition to paper co-authors Mohammad Babaeizadeh, Piotr Miłos, Błażej Osiński, Roy H Campbell, Konrad Czechowski, Chelsea Finn, Piotr Kozakowski, Sergey Levine, Ryan Sepassi, George Tucker and Henryk Michalewski.