More effective training of Large language models. Computational expenses can be lowered by up to 91% by training distinct models on various datasets and then combining them. Tailoring a pretrained model to varied data distributions improves its adaptability and task performance. Distribution vectors use fine-tuned version differences to provide a more flexible and optimized model.
Large language models (LLMs) undergo a number of training phases, including pretraining, instruction tailoring, and reinforcement learning from human input, on heterogeneous datasets with varying distributions. Building successful models needs determining the best combination of data distributions across datasets, which usually necessitates repeatedly training and assessing the model on a vast array of combinations.
At the most recent Conference on Empirical Methods in Natural-Language Processing (EMNLP) that can save up to 91% of the computational cost of training LLMs or other neural-network-based models with mixed data distributions. In addition, the technique really raises the calibre of the models that are produced.
It train a different model on each dataset and then weight the models to create a composite model, as opposed to the conventional method of optimising data distributions, which weights the several datasets used to train a single model.
This novel method has the potential to greatly increase the effectiveness and accessibility of large-model training and was recognised with a special prize for “efficient modelling, training, and inference” at EMNLP.
Distribution-edited models
Conventional training techniques (such instruction tuning) use grid search, an exhaustive-search technique that only compares results for a large range of weight values, to choose the best combination of training data distributions. This requires a great deal of time, money, and flexibility because, once trained, the model cannot be altered without paying further expenses.
To suggest fine-tuning a pretrained model on data distributions that correspond to various tasks in order to overcome these restrictions. Then, the parameter values of the original model are subtracted from those of the fine-tuned models. The variations in parameter values are referred to as distribution vectors, and a weighted sum of the distribution vector is added to the original model’s parameters to create a composite model.
To emphasise the use of weight vector arithmetic for model editing, it refer to the final model as a distribution-edited model (DEM). Each fine-tuned model’s perplexity that is, the likelihood that its parameter values can be predicted from those of the original model is the basis for the weights.
This method is based on two important findings: perplexity can be calculated in a single forward pass on validation data, which is far more efficient than grid search; and training the model independently on each dataset enables better modelling of each dataset’s underlying properties because other data distributions are not interfered with during the training process. The first point enhances the quality of the model, and the second point greatly increases the efficiency of training.
The steps in the approach are as follows, in greater detail:
- Individual-distribution training: Using conventional training techniques, the initial model is trained on individual data distributions. For use in later stages, checkpoints snapshots of the model state following training on a specific dataset are saved.
- Calculation of distribution vectors: Distribution vectors are calculated by deducting the parameters of the pretrained model from the parameters of the refined models. The distinct qualities of every dataset are captured by these vectors.
- Optimization of merging coefficients: One forward pass per combination is used to get the best coefficients for combining the data distribution vectors based on perplexity on the validation set.
- Distribution vector merging: A unified model that accurately depicts the joint distribution of various datasets is produced by linearly combining the distribution vectors with adjustable weights.
- The flexibility and scalability that result from DEM allow for incremental changes when new datasets are added without necessitating complete retraining. For dynamic and extensive training scenarios, this makes it perfect.
A pretrained model is fine-tuned on data distributions that correspond to various tasks (ΘD1 – ΘDn) using distribution-edited models (DEMs). A set of distribution vectors (ΔΘD1 – ΔΘDn) is then obtained by subtracting the parameter values of the original model (Θ) from those of the refined models. A weighted sum of distribution vectors (Σ) is added to the original model’s parameters to create the DEM, which is a composite (ΘD).
Evaluation and future work
It evaluated the method by training LLMs with increasing sizes throughout the instruction-tuning step, ranging from 3 billion parameters to 13 billion parameters. To research demonstrated that DEM can achieve up to 16.1% quality improvement and reduce training costs by up to 91% when compared to traditional data-mixing strategies. This highlights DEM’s potential to democratise access to cutting-edge training methods and provide organisations using neural models at scale with game-changing advantages. Furthermore, DEM’s adaptability guarantees that practitioners and researchers may promptly adjust to new data requirements without sacrificing efficiency.
The following is a summary of the study’s main conclusions:
- Outstanding performance: DEM has been verified on well-known benchmarks such as MMLU, BBH, and HELM, where it outperformed data mixing on individual jobs by up to 16.1%.
- Effectiveness in a wide range of domains: Tests on datasets like MathQA, Super-Natural Instructions (SNI), and Chain-of-Thought (CoT) show that DEM is effective in a number of areas.
- Scalability: DEM is demonstrated to enhance performance at various model sizes (3B, 7B, and 13B), offering compelling proof of this method’s scalability.
The success of DEM emphasises how crucial innovation is to increasing the effectiveness and accessibility of machine learning. Frameworks like DEM will be crucial for preserving efficiency without compromising performance as the machine learning community continues to grow models and datasets. Future studies should examine the framework’s performance in other training circumstances and its applicability to different model designs, including mixture-of-experts models or encoder-decoder frameworks.
