Bi-Phase Training: Efficient Pre-Training for Large AI Models

Paper: Bi-Phase Training: Learning Efficiently From Scratch

Authors: Abel Gurung, Joseph Campbell

Executive Summary

The research presents a novel approach called Bi-Phase Training (BPT), which aims to make the pre-training of foundational AI models more efficient, especially as these models grow in size. As companies increasingly rely on large language models, the cost and resources required for training these models can be significant. BPT addresses this issue by reducing the number of parameters that need to be trained while still maintaining effective learning outcomes.

BPT operates by using a combination of high-rank transformations and low-rank adapters in its training process. It updates model weights through a constrained method that involves periodic merging of updates into a base model that remains largely unchanged. This means that, instead of adjusting every parameter, BPT focuses on a smaller, efficient set of transformations. As a result, the method could allow organizations to achieve similar performance levels to full-parameter training but with significantly fewer resources.

In practical terms, the research demonstrates that BPT can be applied to various language models, specifically tested on models ranging from 100 million to 1.5 billion parameters. When evaluated using a large dataset, FineWeb-Edu, containing billions of tokens, BPT achieved validation results that are comparable to those of traditional full-parameter models. For example, at the 1.5 billion parameter scale, BPT utilized 66% fewer trainable parameters, yet the gap in performance—a measure known as validation perplexity—was minimal. This suggests that businesses may be able to implement BPT to train language models more cost-effectively without sacrificing performance.

Moreover, the research included comprehensive evaluations across a variety of downstream tasks. These tasks are essential for assessing how well a model can perform in real-world applications, such as text generation, understanding, or summarization. BPT was shown to retain much of the performance characteristics of models trained with a complete set of parameters, indicating that it could be a viable option for organizations looking to deploy AI solutions quickly and efficiently.

While the findings are based on benchmarks and simulations rather than real-world applications, the implications are clear. Companies could leverage BPT to reduce training costs and resource consumption while still developing high-performing AI models. This advancement may be particularly beneficial for enterprises with limited computational resources or those looking to accelerate their AI initiatives without compromising quality.

Academic Abstract

Pre-training foundational models is computationally expensive as models scale. In this work, we introduce Bi-Phase Training (BPT), a parameter-efficient pre-training method that preserves the high-rank learning dynamics observed in full-parameter optimization with substantially reduced trainable parameters. BPT parameterizes each weight update as a composition of constrained high-rank left/right transformations via trainable diagonal matrices and low-rank adapters whose updates are periodically merged into the frozen base weights, resulting in a compounding expansion of accessible directions during training. Across language-model pre-training from scratch (Qwen2.5 models from 100M to 1.5B parameters and OLMo-2-1B) on FineWeb-Edu (10B and 100B tokens), BPT achieves validation perplexity close to full-parameter baselines while using far fewer trainable parameters; e.g., at the 1.5B scale, BPT uses 66% fewer trainable parameters with only a small validation-perplexity gap. Additionally, a comprehensive evaluation across diverse downstream tasks shows that BPT preserves much of the full-parameterized model's downstream performance.