MARTINI

Learning to mix surrogate rewards for inference-time steering of generative models.

MARTINI (Mixing And Reward Tilting for INference-time Improvement) learns to combine cheap surrogate rewards using a small set of oracle rankings or numerical rewards. The learned composition steers a pretrained generative model without querying the oracle at inference time.

Figure 1: MARTINI combines surrogate rewards to steer generation toward the oracle reward tilt.
Figure 1 from Steered Not Stirred: a learned surrogate composition guides generation toward the desired reward-tilted distribution.

The paper studies language generation, ImageNet alignment, and protein-ligand cofolding, with the learned reward composition used in existing inference-time steering methods.