Blog

The latest from Google Research

An Optimistic Perspective on Offline Reinforcement Learning

Tuesday, April 14, 2020

Posted by Rishabh Agarwal, AI Resident and Mohammad Norouzi, Research Scientist, Google Research

“The potential for off-policy learning remains tantalizing, the best way to achieve it still a mystery.”

reinforcement learningan agentonline roboticsautonomous drivingunintendedbehaviorchallengingpreviouslycollectedinteraction offline RLfully off-policydistribution mismatchi.e.

RL with online interactions vs. Offline RL.

An Optimistic Perspective on Offline RLAtari 2600 gamesDQNdata-driven RLparadigm.DQN Replay Datasetcodeoffline-rl.github.ioA Primer on Off-policy and Offline RL

Online, off-policyDQNAtari 2600 gamesi.e., Qdistributional RLQR-DQNonlineofflinerecentworkpresentsremediesregularizingThe DQN Replay Dataset for Offline RLofflineDQNAtari 2600 gamessticky actions

Offline RL on Atari games using the DQN Replay Dataset.

offline ImageNetTraining Offline Agents on the DQN Replay DatasetQR-DQNi.e.,fully-trained

Offline DQN. Normalized improvement over a fully-trained DQN, per game, of offline DQN trained using DQN replay. On the normalized scale, fully-trained DQN corresponds to 100% performance while random agent corresponds to 0%.

Offline QR-DQN. Normalized performance improvement (in %) over a fully-trained DQN agent, per game, of offline QR-DQN trained offline using DQN replay.

Introducing Two Robust Offline RL Agentsgeneralizationensemble of models

Ensemble-DQN is a simple extension of DQN that trains multiple Q-value estimates and averages them for evaluation.

Random Ensemble Mixture (REM) is an easy to implement extension of DQN inspired by Dropout. The key intuition behind REM is that if one has access to multiple estimates of Q-values, then a weighted combination of the Q-value estimates is also an estimate for Q-values. Accordingly, in each training step, REM randomly combines multiple Q-value estimates and uses this random combination for robust training.

Neural Network architectures for DQN, distributional QR-DQN and the expected RL variants with the same multi-head QR-DQN architecture, i.e., Ensemble-DQN and REM. In QR-DQN, each head (red rectangles) corresponds to a specific fraction of the return distribution, while in the proposed variants, each head approximates the Q-function.

C51distributional

Offline REM vs. baselines. Median normalized scores averaged over 5 runs across 60 Atari games of offline agents trained using DQN replay for 5x iterations, compared to online DQN.

Online REM vs. baselines. Median normalized evaluation scores averaged over 5 runs (shown as traces) across stochastic 60 Atari 2600 games of online agents trained for 200 million frames. Online REM with 4 Q-networks performs comparably to online QR-DQN.

Comparison of Results: Important Factors in Offline RLpriorworkthatreports

Offline Dataset Size: We trained offline QR-DQN and REM with reduced data obtained via randomly subsampling the entire DQN Replay Dataset, maintaining the same data distribution. Analogous to supervised learning, performance tends to increase as the size of data increases. With only 10% of the entire dataset, REM and QR-DQN approximately recover the performance of fully-trained DQN.

Offline Dataset Composition: We trained offline RL agents on the first 20 million frames per game in the DQN Replay Dataset. Offline REM and QR-DQN outperform the best policy in this lower quality dataset, indicating that standard RL agents work well in the offline setting with sufficiently diverse datasets.

Offline RL with Lower Quality Dataset. REM and QR-DQN trained on offline data collected from DQN trained for 20 iterations (by using the first 20M frames from each game replay dataset). The horizontal line shows the performance of best policy in this dataset, which is significantly worse than fully-trained DQN.

Offline Algorithm Choice: There are claims that standard off-policy agents are ineffective on continuous control tasks when trained offline. However, we found that recent continuous control agents, such as TD3, perform comparably to a sophisticated offline agent when trained on large and diverse offline datasets.

Future WorkDQN replaye.g.koffline policy evaluationmodel-based RLself-supervised learning Acknowledgements
This research was conducted in collaboration with Dale Schuurmans. We’d like to thank members of the Google Research, Brain Team for valuable discussions. A prior version of this work was presented as a contributed talk at NeurIPS 2019 DRL Workshop.