1:44:31
Stanford CS229 I Machine Learning I Building Large Language Models (LLMs)
Stanford Online
Overview
This video provides a comprehensive overview of building Large Language Models (LLMs), focusing on the practical aspects beyond just architecture. It delves into the core components of LLM training, including pretraining and post-training phases. The discussion covers the autoregressive nature of LLMs, the importance and mechanics of tokenization (like Byte Pair Encoding), and evaluation metrics such as perplexity and benchmarks like MMLU. A significant portion is dedicated to the challenges and processes involved in data collection and preparation, emphasizing its critical role. Finally, it touches upon scaling laws, resource allocation, and the shift towards creating AI assistants through post-training alignment techniques like Supervised Fine-Tuning (SFT).
How was this?
Save this permanently with flashcards, quizzes, and AI chat
Chapters
- LLMs are neural networks, and their training involves several critical components: architecture, training loss/algorithm, data, evaluation, and systems.
- While architecture and training algorithms are often the focus in academia, industry prioritizes data, evaluation, and systems for practical LLM development.
- The training process is broadly divided into pretraining (modeling the internet) and post-training (creating AI assistants like ChatGPT).
Understanding these core components provides a framework for comprehending the entire LLM development lifecycle and the relative importance of different factors in achieving successful models.
The five key components are architecture, training loss/algorithm, data, evaluation, and systems.
- Language models predict the probability distribution of sequences of tokens (words or subwords).
- Autoregressive language models predict the next token based on the preceding sequence, using the chain rule of probability.
- The core task during pretraining is next-token prediction, optimized using cross-entropy loss, which is equivalent to maximizing the log-likelihood of the text.
- Tokenization breaks down text into manageable units (tokens) that are more general than words, handling variations and different languages effectively.
This foundational stage imbues the model with a broad understanding of language, grammar, and world knowledge by exposing it to vast amounts of text data.
For the sentence 'The mouse ate the cheese,' a language model assigns a probability, with grammatically correct and semantically plausible sentences receiving higher probabilities than incorrect ones like 'The the mouse ate cheese' or 'The cheese ate the mouse.'
- Tokenizers are crucial for converting text into numerical representations that models can process, addressing issues like typos and languages without clear word boundaries.
- Byte Pair Encoding (BPE) is a common tokenization algorithm that merges frequent pairs of tokens iteratively to build a vocabulary.
- Tokenizers aim for an average token length of 3-4 characters, balancing sequence length with vocabulary size.
- While effective, tokenizers can sometimes struggle with specific data types like numbers or code, leading to research into alternative methods like character or byte-level tokenization.
Effective tokenization is essential for efficient and accurate language processing, directly impacting the model's ability to understand and generate text.
Byte Pair Encoding starts with individual characters as tokens and iteratively merges the most frequent adjacent pairs (e.g., 't' + 'o' might merge into 'to') to create new tokens.
- Perplexity, an exponentiated average per-token loss, was a traditional metric indicating how many choices a model hesitates between for the next token.
- While perplexity is useful for development, it's less common in academic benchmarking due to its dependence on tokenization and data.
- Modern evaluation often aggregates results from various NLP benchmarks (like HELM, Hugging Face leaderboard) or uses specific tasks like multiple-choice question answering (e.g., MMLU).
- Challenges in evaluation include inconsistencies in methodology across different organizations and the critical issue of train-test contamination.
Robust evaluation methods are necessary to accurately measure an LLM's progress, compare different models, and identify areas for improvement.
The MMLU benchmark presents questions across various domains (e.g., astronomy) with multiple-choice answers, and models are evaluated on their ability to select the correct option.
- Training LLMs involves processing vast amounts of data, often starting with web crawlers like Common Crawl (petabytes of data).
- Data preparation includes extracting text, filtering undesirable content (NSFW, PII), de-duplication, and heuristic/model-based filtering for quality.
- Data is often classified into domains (code, books, entertainment) and weighted to optimize model performance, with code and books sometimes up-weighted for reasoning and quality.
- High-quality data, like Wikipedia, is often used for a final fine-tuning stage (overfitting) to enhance model performance.
- Data collection and cleaning are complex, resource-intensive processes, with significant research still needed in areas like synthetic data and multimodal data integration.
The quality, quantity, and composition of training data are paramount, significantly influencing an LLM's capabilities, biases, and overall effectiveness.
Starting with a raw web crawl (like Common Crawl), steps include removing HTML tags, filtering out harmful content, removing duplicate paragraphs, and potentially using a classifier trained on Wikipedia-linked pages to identify high-quality content.
- Scaling laws demonstrate that increasing model size (parameters) and training data consistently improves performance, defying traditional overfitting concerns.
- These laws allow prediction of performance gains based on compute, data, and parameter count, enabling efficient resource allocation.
- The optimal strategy often involves balancing model size and data quantity (e.g., Chinchilla's finding of ~20 tokens per parameter for training optimality).
- Scaling laws suggest that architectural innovations often provide smaller gains than scaling compute and data, emphasizing the importance of systems and data.
- Training state-of-the-art LLMs is extremely compute-intensive, costing millions of dollars and significant energy.
Understanding scaling laws is critical for making informed decisions about how to best utilize computational resources to train the most effective models.
If a company has a fixed amount of compute, scaling laws help determine whether it's better to train a larger model on less data or a smaller model on more data to achieve the best performance.
- Post-training refines pretrained LLMs to act as helpful AI assistants, aligning them with user instructions and desired behaviors.
- This alignment addresses limitations of pure language models, which might not directly answer questions or could generate undesirable content.
- Supervised Fine-Tuning (SFT) is a key post-training technique where the model is fine-tuned on human-curated question-answer pairs.
- The goal is to teach the model to follow instructions and provide helpful, harmless, and honest responses, even with relatively small amounts of specialized data.
Post-training transforms general-purpose language models into useful and safe AI assistants capable of interacting effectively with users.
Instead of a pretrained model completing a question like 'Explain the moon landing...' with another question, SFT trains it to provide a direct answer, and also trains it to refuse harmful requests, such as generating hateful content.
Key takeaways
- LLM development is a multi-faceted process heavily reliant on data quality, efficient systems, and robust evaluation, not just novel architectures.
- Autoregressive language modeling, predicting the next token, forms the core of pretraining, enabling models to learn language patterns from vast datasets.
- Tokenization is a critical preprocessing step that converts text into a format models can understand, with algorithms like BPE balancing vocabulary size and sequence length.
- While perplexity was an early metric, modern LLM evaluation relies on diverse benchmarks and careful consideration of data contamination.
- The quality and curation of training data are arguably the most significant factors determining an LLM's capabilities and biases.
- Scaling laws provide a predictable relationship between compute, model size, data, and performance, guiding optimal resource allocation.
- Post-training techniques like SFT are essential for aligning LLMs with human intent, transforming them into helpful and safe AI assistants.
Key terms
Large Language Model (LLM)Autoregressive Language ModelTokenizationByte Pair Encoding (BPE)Cross-Entropy LossPerplexityMMLU (Massive Multitask Language Understanding)Common CrawlScaling LawsSupervised Fine-Tuning (SFT)
Test your understanding
- What are the five key components involved in training Large Language Models, and which are prioritized in industry versus academia?
- How does an autoregressive language model predict the next token, and what is the primary loss function used during pretraining?
- Explain the purpose of tokenization and how algorithms like Byte Pair Encoding work to create a model's vocabulary.
- Why is perplexity considered less reliable for academic benchmarking compared to other evaluation methods, and what are some alternative approaches?
- Describe the major steps involved in preparing raw internet data for LLM pretraining, highlighting the importance of data quality.
- How do scaling laws help in predicting LLM performance and optimizing the allocation of training resources like compute and data?
- What is the primary goal of post-training, and how does Supervised Fine-Tuning (SFT) help achieve this goal?