2018-10-23-romoff18a.md

title	abstract	keywords	layout	series	id	month	tex_title	firstpage	lastpage	page	order	cycles	bibtex_author	author	date	address	publisher	container-title	volume	genre	issued	pdf	extras
Reward Estimation for Variance Reduction in Deep Reinforcement Learning	Reinforcement Learning (RL) agents require the specification of a reward signal for learning behaviours. However, introduction of corrupt or stochastic rewards can yield high variance in learning. Such corruption may be a direct result of goal misspecification, randomness in the reward signal, or correlation of the reward with external factors that are not known to the agent. Corruption or stochasticity of the reward signal can be especially problematic in robotics, where goal specification can be particularly difficult for complex tasks. While many variance reduction techniques have been studied to improve the robustness of the RL process, handling such stochastic or corrupted reward structures remains difficult. As an alternative for handling this scenario in model-free RL methods, we suggest using an estimator for both rewards and value functions. We demonstrate that this improves performance under corrupted stochastic rewards in both the tabular and non-linear function approximation settings for a variety of noise types and environments. The use of reward estimation is a robust and easy-to-implement improvement for handling corrupted reward signals in model-free RL.	Reinforcement Learning, Uncertainty, Goal Specification	inproceedings	Proceedings of Machine Learning Research	romoff18a	0	Reward Estimation for Variance Reduction in Deep Reinforcement Learning	674	699	674-699	674	false	Romoff, Joshua and Henderson, Peter and Piche, Alexandre and Francois-Lavet, Vincent and Pineau, Joelle	given family Joshua Romoff given family Peter Henderson given family Alexandre Piche given family Vincent Francois-Lavet given family Joelle Pineau	2018-10-23		PMLR	Proceedings of The 2nd Conference on Robot Learning	87	inproceedings	date-parts 2018 10 23	http://proceedings.mlr.press/v87/romoff18a/romoff18a.pdf	label link Source code http://github.com/facebookresearch/reward-estimator-corl