“ODT: Online Decision Transformer”, 2022-02-11 (; similar):
Recent work has shown that offline reinforcement learning (RL) can be formulated as a sequence modeling problem ( et al 2021; et al 2021) and solved via approaches similar to large-scale language modeling. However, any practical instantiation of RL also involves an online component, where policies pretrained on passive offline datasets are finetuned via task-specific interactions with the environment.
We propose Online Decision Transformers (ODT), an RL algorithm based on sequence modeling that blends offline pretraining with online finetuning in an unified framework. Our framework uses sequence-level entropy regularizers in conjunction with autoregressive modeling objectives for sample-efficient exploration and finetuning.
Empirically, we show that ODT is competitive with the state-of-the-art in absolute performance on the D4RL benchmark but shows much more gains during the finetuning procedure.
View PDF: