18 August 2026

Repeating quality training data scales slightly with model size

• Researchers found that the best amount of times to repeat high-quality data during training increases modestly as models grow larger, when keeping the total training volume constant.
• Smaller test models can predict the optimal repetition strategy for much larger models, potentially saving computation time and resources during development.
• The relationship between model size and data repetition is mild rather than dramatic, suggesting diminishing returns as models scale up.

How it was covered

TLDR AITLDR editorial team

The optimal amount of high-quality domain data repetition increased mildly with model size at a fixed tokens-per-parameter ratio. Smaller proxy models could help estimate repetition schedules for larger models.