“Conservative Objective Models for Effective Offline Model-Based Optimization”, Brandon Trabucco, Aviral Kumar, Xinyang Geng, Sergey Levine2021-07-14 (offline RL)⁠:

Computational design problems arise in a number of settings, from synthetic biology to computer architectures.

In this paper, we aim to solve data-driven model-based optimization (MBO) problems, where the goal is to find a design input that maximizes an unknown objective function provided access to only a static dataset of prior experiments. Such data-driven optimization procedures are the only practical methods in many real-world domains where active data collection is expensive (eg. when optimizing over proteins) or dangerous (eg. when optimizing over aircraft designs).

Typical methods for MBO that optimize the design against a learned model suffer from distributional shift: it is easy to find a design that “fools” the model into predicting a high value. To overcome this, we propose conservative objective models (COMs), a method that learns a model of the objective function that lower bounds the actual value of the ground-truth objective on out-of-distribution inputs, and uses it for optimization.

Structurally, COMs resemble adversarial training methods used to overcome adversarial examples. COMs are simple to implement and outperform a number of existing methods on a wide range of MBO problems, including optimizing protein sequences, robot morphologies, neural network weights, and superconducting materials.