The Delta Learning Hypothesis: Preference Tuning on Weak Data can Yield Strong Gains

The preprint operates within the dominant paradigm of Supervised Fine-tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF) for Large Language Model (LLM) optimization.

This preprint presents the "delta learning hypothesis," which suggests that improvements to LLMs can be achieved by leveraging the relative difference in quality between pairs of data points, even if the individual data points are of low quality. This hypothesis challenges the prevailing assumption in the field that high-quality training data is essential for building strong models. However, it does not fundamentally challenge the existing paradigms of SFT and RLHF; rather, it offers a novel approach within these paradigms. The core techniques used in the research, such as preference tuning and the use of weaker models for data generation, are consistent with current practices in LLM optimization. The proposed delta learning approach could be considered an extension or refinement of existing techniques, rather than a radical departure. Therefore, the preprint's contribution is best classified as normal science, as it builds upon the established paradigm and explores novel solutions within its framework. While the delta learning hypothesis presents a new perspective, it does not constitute a paradigm shift or model revolution, as it does not propose a fundamentally different framework for understanding or optimizing LLMs. It could potentially be considered Model Drift if the described improvements are only marginal and not game-changing. More research will be needed to fully validate this hypothesis and assess its long-term impact on the field.