Proposal of a method to train foundation models with a single algorithm

Foundation models are typically trained in stages: pretraining, midtraining, supervised fine-tuning (SFT), and reinforcement learning (RL), with changes in both data and learning algorithms. However, this research shows that this algorithmic separation is not fundamental.

By deriving suitable RL objectives as maximum-likelihood fitting to model-generated target distributions, all of these stages can be placed under a single likelihood-based training algorithm, called unified training. This perspective turns the foundation-model training lifecycle into a problem of choosing which data to train on, in what proportions, and when, with conventional staged training being a special case. The paper received 5 upvotes on Hugging Face Daily Papers.