Small Language Model (SLM) Engineering with GPT-Style Transformers

Introduction to Small Language Models (SLMs)

• Evolution of language models: From RNNs to Transformers
• Parameter scaling trends in modern NLP
• SLMs vs high-parameter LLMs:
  • Model size vs performance trade-offs
  • Compute, memory, and latency considerations
• When and why to build smaller models
• Enterprise use cases for compact generative models
• Cost-performance optimization perspectives

End-to-End GPT-Style Training Architecture

• Complete SLM lifecycle: Dataset → Tokenization → Encoding → Training → Evaluation → Inference
• Decoder-only architecture overview
• Autoregressive next-token prediction objective
• High-level system architecture design
• Compute graph understanding in PyTorch
• Hardware considerations (GPU memory, batch sizing, throughput)

Dataset Engineering for SLM Training

• Synthetic datasets (TinyStories) and compact corporaDataset structure and splits

• Evaluating dataset quality for language modeling

• Vocabulary distribution analysis

• Token length statistics and context window sizing

• Data leakage and validation pitfalls

• Hands-on:
  • Explore and analyze TinyStories dataset
  • Build train/validation splits
  • Inspect token distribution statistics

Tokenization & Data Preprocessing

• Purpose of tokenization in autoregressive models
• Overview of GPT-2 tokenizer mechanics
• Byte Pair Encoding (BPE) fundamentals
• Converting raw text to token IDs
• Special tokens and vocabulary size considerations
• Handling unknown and rare tokens
• Preparing sequential training data
• Hands-on:
  • Implement tokenization pipeline
  • Encode dataset into token streams
  • Save binary .bin files for training

Efficient Data Loading & Batching

• Binary storage using .bin files
• Memory mapping for scalable training
• Next-token prediction objective
• Context window (block size) trade-offs
• Input-output shifting mechanism
• Batch sampling strategies
• Handling long sequences efficiently
• Hands-on:
  • Implement memory-mapped dataset loader
  • Build dynamic batch generator

Transformer Architecture Fundamentals

• Token embeddings
  • Positional embeddings (learned vs sinusoidal)
  • Transformer block components
  • Residual connections and LayerNorm
  • Feedforward networks
  • Decoder-only stack structure
• Architecture discussion:
  • Depth vs width trade-offs
  • Parameter efficiency strategies
  • Activation function selection
• Hands-on:
  • Implement embedding layer
  • Build modular Transformer block

Self-Attention & Multi-Head Attention

• Attention mechanism intuition
  • Query, Key, Value formulation
  • Scaled dot-product attention
  • Causal masking in autoregressive models
  • Multi-head attention benefits
  • Computational complexity analysis (O(n²))
  • Memory optimization considerations
• Hands-on:
  • Implement self-attention from scratch
  • Add multi-head attention module

Building the Complete GPT-Style Model

• Model configuration class design
  • Stacking Transformer blocks
  • Weight tying between embedding and output layers
  • Parameter count calculation
  • Initialization strategies
  • Dropout and regularization techniques
• Hands-on:
  • Assemble full GPT-style SLM architecture
  • Print parameter summary

Loss Function & Training Objective

• Cross-entropy loss for language modeling
  • Token-level loss aggregation
  • Masked vs unmasked loss
  • Perplexity as evaluation metric
  • Training vs inference behavior
• Hands-on:
  • Implement training loss computation
  • Compute perplexity metrics
• Real-world application:
  ● Monitoring model performance in iterative experiments

Optimization Strategies & Training Stability

• Optimizer comparison (Adam vs AdamW)
  • Learning rate scheduling (warmup + cosine decay)
  • Gradient clipping
  • Gradient accumulation
  • Mixed precision training (FP16 / AMP)
  • Memory optimization techniques
  • Detecting exploding / vanishing gradients
• Hands-on:
  • Configure optimizer and scheduler
  • Enable mixed precision training
  • Implement gradient clipping

Pre-Training the Small Language Model

• Training loop architecture
  • Evaluation intervals and validation loops
  • Logging and experiment tracking
  • Checkpointing strategies
  • Early stopping criteria
  • Convergence diagnostics
• Hands-on:
  • Train the SLM for multiple epochs
  • Save and reload checkpoints

Inference & Text Generation

• Loading trained model weights
• Switching to evaluation mode
• Generating token probabilities
• Sampling strategies:
  • Temperature scaling
  • Top-k sampling
  • Top-p (nucleus) sampling
• Greedy vs stochastic decoding
• Controlling creativity vs coherence
• Hands-on:
  • Build inference script
  • Compare different sampling strategies

Model Evaluation & Improvement Strategies

• Perplexity benchmarking
• Qualitative vs quantitative evaluation
• Failure pattern analysis
• Hallucination and repetition issues
• Fine-tuning vs retraining considerations
• Scaling strategies for larger datasets
• Architecture discussion:
  • When to scale parameters vs improve data
  • Performance vs cost trade-offs

Capstone Project: Build & Deploy a Domain-Specific Mini GPT

Participants will:

• Prepare a custom dataset
• Train a compact GPT-style model
• Optimize training configuration
• Implement inference pipeline
• Evaluate performance
• Present architecture and optimization decisions
• Outcome:
  • A fully functional small GPT-style language model trained and deployed locally.