This is the first blog post in a three-part series about adapting open source large language models (LLMs). In this post, we’ll take a look at the various approaches available to adapt LLMs to domain data.

• In Part 2, we’ll discuss how to determine if fine-tuning is the right approach for your use case.
• In Part 3, we’ll explore some rules of thumb for curating a good training dataset.

Fine-tuning is the process of adapting a pre-trained language model using an annotated dataset in a supervised manner or using reinforcement learning-based techniques. There are two major differences compared to pre-training:

1. Supervised training on an annotated dataset—that contains the correct labels/answers/preferences—instead of self-supervised training
2. Requires fewer tokens (thousands or millions instead of the billions or trillions needed in pre-training) where the primary aim is to enhance abilities like instruction following, human alignment, task performance, etc.

There are two dimensions to understanding the current landscape of fine-tuning: percentage of parameters changed and new capabilities added as a result of the fine-tuning.

Depending on the number of parameters changed, there are two categories of algorithms:

1. Full fine-tuning: As the name suggests, this encompasses changing all parameters of the model and includes legacy fine-tuning as done on smallish models like XLMR and BERT (100 – 300M parameters) as well as fine-tuning on large models like Llama 2, GPT3 (1B+ parameters), etc.
2. Parameter-efficient fine-tuning (PEFT): Instead offine-tuning all LLM weights, PEFT algorithms only fine-tune a small number of additional parametersorupdate a subset of the pre-trained parameters, typically 1 – 6% of the total parameters.

With ICL, we adapt the LLM by placing prototype examples in the prompt. “Demonstration through examples” has been shown in multiple studies to be effective. The examples can contain different kinds of information:

• Input and output text only—that is, few-shot learning
• Reasoning traces: adding intermediate reasoning steps; see Chain-of-Thought (CoT) prompting
• Planning and reflection traces: adding information that teaches the LLM to plan and reflect on its problem solving strategy; see ReACT

Multiple other strategies to modify the prompts exist, and the Prompt Engineering Guide contains a comprehensive overview.

Pre-training, a vital part of LLM training, uses a token prediction variant as a loss function. Its self-supervised nature allows for training on extensive data. As an example, Llama 2 is trained on 2 trillion tokens. This requires massive computational infrastructure: Llama 2 70B took 1,720,320 GPU hours. Therefore, for teams with limited resources, we do not recommend pre-training as a viable approach for LLM adaptation.

With pre-training being computationally prohibitive, it stands to reason that updating the weights of a model that is already pre-trained could be an effective way to adapt an LLM for particular tasks. Any approach that updates the model weights of a pre-trained model is susceptible to a phenomenon called catastrophic forgetting, which is a term coined for the model forgetting previously learned skills and knowledge. For example, this study showed how a model fine-tuned in the medical domain degraded in performance for instruction-following and common QA tasks. Other studies have also shown that general knowledge obtained through pre-training can be forgotten in subsequent training sessions. For example, this study provided some evidence of knowledge forgetting in LLMs, from the perspectives of domain knowledge, reasoning, and reading comprehension.

Having considered catastrophic forgetting, recent developments have shown that continued pre-training (CPT) can lead to further improvements in performance at a fraction of the compute cost that pre-training would incur. CPT can be beneficial for tasks that require a new transformation skill to be acquired by the LLM. As an example, continued pre-training has been reported to be successful at adding multilingual capabilities.

But CPT is still an expensive process, requiring a substantial amount of data and computational resources. For instance, the Pythia suite underwent a second stage of pre-training, resulting in the creation of FinPythia-6.9B. This model, specifically designed for financial data, was subjected to CPT for 18 days using a dataset comprising 24 billion tokens. Furthermore, CPT is also prone to catastrophic forgetting. Therefore, for teams with limited resources, we do not recommend continued pre-training as a viable approach for LLM adaptation.

To summarize, adapting LLM using self-supervised algorithms and unannotated datasets as done in pre-training and continued pre-training is resource- and cost-intensive and not recommended as a viable approach.

Fine-tuning with smallish annotated datasets is a more cost-effective approach compared to pre-training(s) with unannotated datasets. By adapting a pre-trained model to a specific task, fine-tuned models are shown to achieve state-of-the-art results in a wide range of applications and in specialized domains, such as legal, medical, or finance.

Fine-tuning, specifically parameter-efficient fine-tuning (PEFT), requires only a fraction of the computational resources needed for pre-training/continued pre-training. Therefore, this is a viable approach for adapting LLMs for teams with limited resources. In Part 3 of this series, we dig into fine-tuning details, including full fine-tuning, PEFT, and practical guidelines for how to fine-tune.