July 1st, 2024

Large Language Models, or LLMs, have become commonplace in the tech world. The number of applications that LLMs are revolutionizing is multiplying by the day — extraction use cases, chatbots, tools for creatives and engineers. In spite of this, at its core, the LLM is a multi-purpose neural network, dozens of layers deep, designed to simply predict one word after the next. It predicts words by performing billions of matrix multiplication steps based on so-called parameter weights, which are discovered during the model training process. Almost all open-source, open-weight models are trained on a massive amount of text from every conceivable genre and topic. How, then, do researchers and engineers create novel specialized applications? The answer is fine-tuning.

In this post, we will demystify the process of fine-tuning and discuss the tradeoffs of other approaches to customizing an LLM.

The history of fine-tuning

In the ancient days of LLMs, by which we mean five years ago, the primary approaches to customizing an LLM was identical to the approaches to customizing any other deep learning model. A machine learning engineer would have two options:

1. Retrain the entire LLM. This would mean discarding the trained weights and instead only using the open source model’s architecture to train it on a specialized dataset. As long as the amount and diversity of the specialized data is comparable to what the original model was trained on, this can be the ideal method of customizing a model. However, of course, this is a massive waste of resources due to the computational power required and the difficulty of collecting such a massive dataset. Even if an organization could provision enough GPUs, the cost of training modern-day models could cost up to $190 million.
2. Retrain the last few layers of the LLM while keeping the rest of the weights frozen. This is a more efficient method in terms of time and computational power required because it significantly cuts down the number of parameters that need to be trained. However, for most tasks, this leads to subpar quality.

Of course, almost everyone chooses to retrain the last few layers. And where there is only one option, the research community saw an opportunity to step in. Soon, the LLM space saw an enormous amount of activity in fine-tuning, which leads us to today.

Modern approaches to fine-tuning

Most fine-tuning approaches today are parameter-efficient. Deep neural networks are composed of matrices and vectors (generally called tensors), which are at their core arrays of floating point numbers. By training a small subset of these tensors, while the rest of the LLM’s weights are kept frozen, practitioners achieve good enough results without having to retrain the entire model. Generally, this method requires at least a hundred or so handcrafted examples of input-output pairs for fine-tuning. This is called supervised learning.

The modern fine-tuning landscape involves an unsupervised learning step afterwards. Given a set of inputs, a practitioner gathers the various possible outputs from the LLM and casts votes among them. This preference data is then used to further train the LLM’s weights. Usually, this approach is used for LLM alignment and safety, which defends the application from malicious uses, outputs embarrassing to the organization, and prompt injection attacks.

Fine-tuning’s relationship to prompt engineering

A natural question arises: why fine-tune instead of crafting a well-considered system prompt? Wouldn’t that be easier and more efficient? The answer is no, it wouldn’t. Here’s why:

1. Advanced techniques make prompt engineering obsolete: Plum Defense’s product uses soft-prompting and other techniques to train the input layer itself. This obviates the need for prompt engineering entirely, which lets organizations avoid the time-consuming trial-and-error process to get the prompt just right. Prompt engineering has been a stopgap measure in the early days of LLM applications to convey the practitioner’s intent to the LLM. It is not the long-term solution for LLM application development.
2. The system prompt is precious: the limited budget for system prompt length is better used for up-to-date information, e.g., Retrieval-Augmented Generation (RAG). Even as context windows increase in size with each new open-source model, the system prompt is the least efficient place to provide the LLM model with verbose instructions and examples.
3. The longer the prompt, the slower the application: an LLM must attend to the entire system prompt for each token generated. This pain becomes more acute in the chatbot case, where the length of the conversation so far is also counted toward the system context. The longer the conversation, and the longer your beautifully-crafted system prompt, the slower the bot becomes. Even in cases where the model allows for system prompts that are millions of tokens long, doubling the size of the context will quadruple the latency. This means adding a few hundred words to the system prompt may result in several seconds of additional latency in production, making a chatbot impossible to use.
4. Edge case handling: the number of edge cases that the system prompt would need to consider and emphasize to the LLM is too large. The instructions would have to be too nuanced and long to cover them all. However, fine-tuning on a dataset that considers these edge cases would be more straightforward.

Do I need to fine-tune the LLM in my production application?