I'm starting off with RL and these might be very trivial questions but I want to wrap my head around everything as best as I can. If you have any resources that would provide good intuitions behind applications of RL, please provide them in the comments too :) Thanks.
Questions:
- In which scenarios do we prefer supervised learning over offline reinforcement learning?
- How does the number of samples affect the training for each case? Does supervised learning converge faster?
- What are the examples where both of them have been used and compared for comparative analysis?
Intuition:
- Supervised Learning can be good for predicting a reward given a state but we cannot depend on it for maximizing future rewards. Since it does not use rollouts to maximize rewards, and it does not do planning, we cannot expect to use it in cases where delayed rewards would be expected.
- Also, in a dynamic environment that is non-iid, each action affects the state and then affects further actions taken. So, for continual settings, we accounted for distributional shift in most cases for RL.
- Supervised Learning tries to find the best action for each state, which may be correct in most of the cases but it is a very rigid and dumb approach for ever changing environments. Reinforcement Learning learns for itself and is more adaptable.
For the answers, if possible, provide with a single-liner and then any detail and source of answer would be appreciated too. I want this post to be a nice guideline for anyone trying to apply RL. I'll edit and update answers to any questions answered below to compile all the information I get. If you feel like I should be thinking about any other major questions and concerns, mention them as well please. Thank you!
[EDIT]: Resources I found regarding this:
RAIL Lecture by Sergey Levine: Imitation Learning vs. Offline Reinforcement Learning
Medium post by Sergey Levine: Decisions from Data: How Offline Reinforcement Learning Will Change How We Use Machine Learning
Medium post by Sergey Levine: Understanding the World Through Action: RL as a Foundation for Scalable Self-Supervised Learning
Research Paper by Sergey Levine: When Should We Prefer Offline Reinforcement Learning over Behavioral Cloning?
Research Paper by Sergey Levine: RVS: What is Essential for Offline RL via Supervised Learning?
