Introduction to Large Language Models
In my previous post, I laid out the 8-step LLM pipeline, decoding how large language models (LLMs) process language behind the scenes. Now, let’s zoom in — starting with Step 1: Input Processing.
What is Input Processing?
Input processing is the first step in the LLM pipeline. It involves transforming raw text into structured numeric inputs that LLMs can understand. This step is crucial because the quality of input encoding directly affects the model’s output. In this post, I’ll explore exactly how raw text transforms into structured numeric inputs that LLMs can understand, diving into text cleaning, tokenization methods, numeric encoding, and chat structuring.
Text Cleaning and Normalization
The goal of text cleaning and normalization is to convert raw user input into standardized, clean text for accurate tokenization. Raw input text is often messy, with typos, casing, punctuation, and emojis. Normalization ensures consistency and reduces tokenization errors, ensuring better downstream performance.
Why Text Cleaning and Normalization?
- Raw input text is often messy and needs to be standardized.
- Normalization ensures consistency and reduces tokenization errors.
- Different models have different normalization techniques, such as GPT models preserving formatting and nuance, while BERT aggressively cleans text.
Technical Details
- Unicode normalization (NFKC/NFC) standardizes characters.
- Case folding (lowercasing) reduces vocab size and standardizes representation.
- Whitespace normalization removes unnecessary spaces, tabs, and line breaks.
- Punctuation normalization ensures consistent punctuation usage.
- Contraction handling involves splitting or keeping contractions intact based on model requirements.
Tokenization
Tokenization is the process of converting pre-processed text into tokens. The goal of tokenization is to convert raw text into tokens that can be processed by LLMs. Tokenization directly impacts model quality and efficiency.
Why Tokenization?
- Models can’t read raw text directly and must convert it to discrete units (tokens).
- Tokens are the fundamental unit that neural networks process.
Tokenizer Types
- Subword tokenization (BPE, WordPiece, Unigram) is the most common in modern LLMs.
- Byte Pair Encoding (BPE) iteratively merges frequent character pairs.
- WordPiece optimizes splits based on likelihood in the training corpus.
- Unigram removes unlikely tokens iteratively, creating an optimal set.
Numerical Encoding
The goal of numerical encoding is to convert tokens into unique numerical IDs. LLMs don’t process text directly and operate on numbers. Every token has a unique integer representation in the model’s vocabulary.
Why Numerical Encoding?
- LLMs don’t process text directly and operate on numbers.
- Token IDs enable efficient tensor operations and computations inside neural layers.
Technical Details
- Vocabulary lookup tables efficiently map tokens to unique integers (token IDs).
- Vocabulary size defines model constraints (memory usage and performance).
- Lookup tables are hash maps, allowing constant-time token-to-ID conversions.
Formatting Input for LLMs
The goal of formatting input for LLMs is to structure tokenized input for conversational models (multi-turn chat). LLMs like GPT-4, Claude, and LLaMA expect input structured into roles (system, user, assistant).
Why Formatting Input?
- LLMs expect input structured into roles (system, user, assistant).
- Formatting input provides context and helps the model distinguish between different roles.
Technical Details
- Chat templates provide role identification, context management, and structured input.
- Each message is wrapped with special tokens or structured JSON, helping the model distinguish inputs clearly.
Model Input Encoding
The goal of model input encoding is to convert numeric token IDs into structured numeric arrays (tensors) for GPU-based neural computation compatibility.
Why Model Input Encoding?
- Neural networks expect numeric arrays (tensors) with uniform dimensions.
- Token IDs alone are discrete integers, while tensor arrays add structure and context.
Technical Details
- Padding adds special tokens to shorter sequences, ensuring uniform tensor shapes.
- Truncation removes excess tokens from long inputs, ensuring compatibility with fixed context windows.
- Attention masks distinguish real tokens from padding tokens, preventing the model from attending to padding tokens during computation.
Conclusion
Input processing is a critical step in the LLM pipeline, involving text cleaning, tokenization, numerical encoding, and chat structuring. The quality of input encoding directly affects the model’s output. Understanding the different techniques and trade-offs involved in input processing can help improve the performance and efficiency of LLMs.
FAQs
- What is input processing in the context of LLMs?
Input processing is the first step in the LLM pipeline, involving transforming raw text into structured numeric inputs that LLMs can understand. - What is the goal of text cleaning and normalization?
The goal of text cleaning and normalization is to convert raw user input into standardized, clean text for accurate tokenization. - What is tokenization?
Tokenization is the process of converting pre-processed text into tokens that can be processed by LLMs. - What is numerical encoding?
Numerical encoding is the process of converting tokens into unique numerical IDs that can be processed by LLMs. - Why is formatting input for LLMs important?
Formatting input for LLMs is important because it provides context and helps the model distinguish between different roles (system, user, assistant).
