The large language models (LLMs) behind today’s chatbots are built in stages: a model architecture, a long pretraining run on enormous amounts of text, and further fine-tuning with human feedback. This explainer walks through each stage in plain terms, drawing on the research papers that introduced the key methods and on guidance from the US National Institute of Standards and Technology (NIST).

1. Text becomes tokens

Models do not read words the way people do. They work on tokens, and “a token could be a word, a subword (a subset of a word), or even a single character,” according to Google’s developer guide to LLMs. Most modern models split text into subwords: “unwatched,” for example, might become “un,” “watch” and “ed.” A language model, in Google’s definition, “estimates the probability of a token or sequence of tokens occurring within a longer sequence of tokens.”

2. The architecture: the transformer

GPT-3, discussed below, is built on the transformer, introduced in the 2017 paper “Attention Is All You Need” by Ashish Vaswani and colleagues, most of them at Google. Earlier language systems relied on recurrent networks, whose “inherently sequential nature precludes parallelization within training examples,” the authors wrote. The transformer is instead “based solely on attention mechanisms.”

The key ingredient is self-attention, which the paper describes as a mechanism “relating different positions of a single sequence in order to compute a representation of the sequence.” Google’s guide likewise contrasts older recurrent networks with large language models that “evaluate the whole context simultaneously.” The original model was built for translation and trained on about 4.5 million English-German sentence pairs on one machine with eight graphics processing units (GPUs). Its decoder is auto-regressive: it generates output one symbol at a time, feeding its previous output back in as input.

3. Pretraining: predicting the next token, at scale

The bulk of the work happens in pretraining. As NIST’s Generative AI Profile puts it, “LLMs predict the next token or word in a sentence or phrase,” producing outputs that approximate the statistical distribution of their training data.

The 2020 OpenAI paper “Language Models are Few-Shot Learners” shows what that looks like at scale. GPT-3, an autoregressive model with 175 billion parameters, was trained on a total of 300 billion tokens. Its largest data source was Common Crawl web data from 2016 to 2019, cut from 45 terabytes of compressed text to 570 gigabytes by filtering. The team added curated sets, including an expanded WebText dataset, two internet-based book collections and English-language Wikipedia, and sampled the higher-quality sets more often. Pretraining the largest model took “several thousand petaflop/s-days” of computing.

The authors also found that the pretrained model could pick up many tasks from a few examples placed in the prompt, without any further training, an ability they called “in-context learning.”

4. Fine-tuning with human feedback

A pretrained model is not automatically a helpful assistant. “Making language models bigger does not inherently make them better at following a user’s intent,” OpenAI researchers wrote in the 2022 InstructGPT paper. Their method, reinforcement learning from human feedback (RLHF), has three steps:

  1. Demonstrations. Human labelers write examples of the desired responses to prompts, and the pretrained model is fine-tuned on them with supervised learning.
  2. Comparisons. Labelers rank several model outputs for the same prompt, and a separate reward model is trained to predict which output they would prefer.
  3. Reinforcement learning. The fine-tuned model is optimized against that reward model using an algorithm called proximal policy optimization (PPO).

The team hired about 40 contractors and used datasets of roughly 13,000, 33,000 and 31,000 training prompts for the three stages. In human evaluations, outputs from the 1.3 billion parameter InstructGPT model were preferred to those of the 175 billion parameter GPT-3.

Common misconceptions

  • “The model looks facts up.” It predicts likely text. NIST says this is why LLMs produce confabulations, also called hallucinations: confidently stated but false content. Such errors are “a natural result of the way generative models are designed.”
  • “Bigger always means better.” The InstructGPT result, with a far smaller model preferred, shows training method matters as well as size.
  • “Human feedback teaches universal values.” The InstructGPT authors say their procedure aligns the model to the stated preferences of a specific group, mostly their labelers and researchers, rather than any broader notion of “human values.” They also report an “alignment tax”: lower performance on some public benchmarks.
  • “The training data has been checked.” NIST notes that training data may be too large for humans to vet, that most developers do not disclose specific data sources, and that models can memorize and reveal sensitive information from training data. How copyright applies is still being debated in legal fora.
  • “Training runs can simply be redone.” A bug meant the GPT-3 team’s filter missed some overlaps between the training data and benchmark test sets, and the authors wrote that “due to the cost of training it was not feasible to retrain the model.”

Corrections and tips: email the editor using the address on the Contact page.