Blog

The latest from Google Research

Introducing Dreamer: Scalable Reinforcement Learning Using World Models

Wednesday, March 18, 2020

Posted by Danijar Hafner, Student Researcher, Google Researchartificial agentsreinforcement learningModel-freeDQNAlphaStarreal-world scenariosmodel-basedworld modelDeep Planning NetworkDeepMindDreamerbackpropagationcompact model statessource codeHow Does Dreamer Work?typical

The three processes of the Dreamer agent. The world model is learned from past experience. From predictions of this model, the agent then learns a value network to predict future rewards and an actor network to select actions. The actor network is used to interact with the environment.

Learning the World ModelPlaNetcompact model states

Dreamer learns a world model from experience. Using past images (o₁–o₃) and actions (a₁–a₂), it computes a sequence of compact model states (green circles) from which it reconstructs the images (ô₁–ô₃) and predicts the rewards (r̂₁–r̂₃).

DeepMind Control SuiteDeepMind Lab

Predicting ahead using compact model states enables long-term predictions in complex environments. Shown here are two sequences that the agent has not encountered before. Given five input images, the model reconstructs them and predicts the future images up to time step 50.

Efficient Behavior Learning

Dreamer learns long-sighted behaviors from predicted sequences of model states. It first learns the long-term value (v̂₂–v̂₃) of each state, and then predicts actions (â₁–â₂) that lead to high rewards and values by backpropagating them through the state sequence to the actor network.

Performance on Control Tasks

Dreamer learns to solve 20 challenging continuous control tasks with image inputs, 5 of which are displayed here. The visualizations show the same 64x64 images that the agent receives from the environment.

PlaNetA3CD4PG

Dreamer outperforms the previous best model-free (D4PG) and model-based (PlaNet) methods on the benchmark of 20 tasks in terms of final performance, data efficiency, and computation time.

Dreamer learns successful behaviors on Atari games and DeepMind Lab levels, which feature discrete actions and visually more diverse scenes, including 3D environments with multiple objects.

ConclusionAcknowledgementsThis project is a collaboration with Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. We further thank everybody in the Brain Team and beyond who commented on our paper draft and provided feedback at any point throughout the project.