“Is Curiosity All You Need? On the Utility of Emergent Behaviors from Curious Exploration”, 2021-09-17 (; similar):
Curiosity-based reward schemes can present powerful exploration mechanisms which facilitate the discovery of solutions for complex, sparse or long-horizon tasks. However, as the agent learns to reach previously unexplored spaces and the objective adapts to reward new areas, many behaviors emerge only to disappear due to being overwritten by the constantly shifting objective.
We argue that merely using curiosity for fast environment exploration or as a bonus reward for a specific task does not harness the full potential of this technique and misses useful skills. Instead, we propose to shift the focus towards retaining the behaviors which emerge during curiosity-based learning. We posit that these self-discovered behaviors serve as valuable skills in an agent’s repertoire to solve related tasks.
Our experiments demonstrate the continuous shift in behavior throughout training and the benefits of a simple policy snapshot method to reuse discovered behavior for transfer tasks.