Conservative Q learning for Offline Reinforcement Learning
We develop a conservative Q-learning (CQL) algorithm, such that the expected value of a policy under the learned Q-function lower-bounds its true value. A lower bound on the Q-value prevents the over-estimation that is common in offline RL settings due to OOD actions and function approximation error. We start by focusing on policy evaluation step in CQL, which could be used by itself as an off-policy evaluation procedure, or integrated into a complete offline RL algorithm.