Blog

The latest from Google Research

Learning to Generalize from Sparse and Underspecified Rewards

Friday, February 22, 2019

Posted by Rishabh Agarwal, Google AI Resident and Mohammad Norouzi, Research ScientistReinforcement learningplaying video gamescontinuous controlrobotic learningsparseunderspecifiedlearning agent generalizei.e.,language understandinge.g.,10

In this instruction-following task, the action trajectories a₁, a₂ and a₃ reach the goal, but the sequences a₂ and a₃ do not follow the instructions. This illustrates the issue of underspecified rewards.

sparse underspecifiedreward hackingLearning to Generalize from Sparse and Underspecified RewardsunderspecifiedMeta Reward Learning MeRLMeRLexplorationsemantic parsinglogical formse.g.,SQLweakly-supervisedWhich nation won the most silver medals?a relevant Wikipedia tablei.e.,

WikiTableQuestionsWikiSQLprior work1.2%2.4%s,ground-truthpreviousrewardlearning

Overview of the proposed approach. We employ (1) mode covering exploration to collect a diverse set of successful trajectories in a memory buffer; (2) Meta-learning or Bayesian optimization to learn an auxiliary reward that provides more refined feedback for policy optimization.

Meta Reward Learning (MeRL) generalization performance validation setmeta learning

Schematic illustration of MeRL: The RL agent is trained via the reward signal obtained from the auxiliary reward model while the auxiliary rewards are trained using the generalization error of the agent.

Learning from Sparse RewardssparseOur paperthe two directionsKullback–Leiblerbimodalmode coveringmode seeking

Left: Optimizing mode covering KL. Right: Optimizing mode seeking KL

Conclusionmodelling reward functionscredit-assignment problemAcknowledgements This research was done in collaboration with Chen Liang and Dale Schuurmans. We thank Chelsea Finn and Kelvin Guu for their review of the paper.