Standardized Architecture Templates for Modern Open-Source LLMs

This document summarizes a Stanford CS336 lecture by Tatsu regarding the convergence of Large Language Model (LLM) architectures. The core takeaway is that 90% of architectural design choices for open-source models have standardized, allowing developers to follow a 'default' 2026 configuration for training high-performance models. Key structural conventions include using pre-norm, RMS Norm, RoPE positional encoding, GQA (Grouped Query Attention), and SwiGLU or GeGLU activation functions. The lecture emphasizes ditching bias terms and utilizing serial transformer blocks. Important stability techniques to prevent mid-training loss spikes include Z-loss (to normalize softmax outputs), QK norm, and logit soft capping. Furthermore, the author notes that hyperparameters have largely converged, with a hidden dimension to layers ratio of 100, and vocabulary sizes for general models ranging between 100K-200K. For handling long contexts, the industry has shifted toward alternating local sliding-window and global attention patterns. Entities mentioned include Llama series, Mistral, Qwen, Gemma, T5, PaLM, GPT-J, GPT-4, Cohere Command R, Olmo, and DCLM. Technical tools and frameworks referenced include various transformer-based implementations and optimization techniques like weight decay, which is framed as essential optimizer intervention rather than just an overfitting prevention measure. Statistics provided highlight that RMS Norm saves up to 25% in runtime and GQA reduces inference costs by approximately 80%. The information serves as a practical guide for engineers to avoid reinventing the wheel when developing their own LLMs.