Timezone: »

 
Neighborhood Mixup Experience Replay: Local Convex Interpolation for Improved Sample Efficiency in Continuous Control Tasks
Ryan Sander · Wilko Schwarting · Tim Seyde · Igor Gilitschenski · Sertac Karaman · Daniela Rus
Event URL: https://openreview.net/forum?id=jp9NJIlTK-t »

The human brain is remarkably sample efficient, capable of learning complex behaviors by meaningfully combining previous experiences to simulate novel ones, even when few experiences are available. To improve sample efficiency in continuous control tasks, we take inspiration from this learning phenomenon. We propose Neighborhood Mixup Experience Replay (NMER), a modular replay buffer that interpolates transitions with their closest neighbors in normalized state-action space. NMER preserves a locally linear approximation of the transition manifold by only interpolating transitions with similar state-action features. Under NMER, a given transition’s set of state-action neighbors is dynamic and episode agnostic, in turn encouraging greater policy generalizability via cross-episode interpolation. We combine our approach with recent off-policy reinforcement learning algorithms and evaluate on several continuous control environments. We observe that NMER improves sample efficiency over other state-of-the-art replay buffers, enabling agents to effectively recombine previous experience and learn from limited data.

Author Information

Ryan Sander (Massachusetts Institute of Technology)

Ryan Sander recently finished his Master's of Engineering in Artificial Intelligence from the Massachusetts Institute of Technology, advised by Professor Daniela Rus and Professor Sertac Karaman in MIT's CSAIL Distributed Robotics Laboratory. Ryan's Master's research focused primarily on investigating and applying novel deep reinforcement learning algorithms to autonomous vehicles. Prior to his Master's, Ryan completed his Bachelor's of Science in Electrical Engineering and Computer Science and Mathematical Economics from the Massachusetts Institute of Technology in 2020.

Wilko Schwarting (Massachusetts Institute of Technology)
Tim Seyde (MIT CSAIL)
Igor Gilitschenski (University of Toronto)
Sertac Karaman (MIT)
Daniela Rus (Massachusetts Institute of Technology)

More from the Same Authors