Every LLM on Earth is the same stack — data, tokens, embeddings, attention, MLPs, an optimizer and a lot of GPUs. Pick an architecture, snap the real parts together, launch the training loop, and watch it stream tokens back. No proprietary weights, no secret sauce — just how frontier models actually work.
Frontier runs ingest 10-20T tokens: filtered web, code, books, math, multilingual, synthetic. Quality (dedup, classifier scoring, decontamination) matters more than raw size after ~1T.
For a fixed FLOP budget the loss-minimising ratio is ~20 tokens per parameter. Modern inference-optimised runs over-train (100-300 tok/param) to shrink deploy cost.
RMSNorm pre-norm, SwiGLU MLP, RoPE positions, GQA, FlashAttention. Optional MoE for sparse scaling. Very few papers change the recipe — most add data or scale.
SFT on instruction data → preference optimisation (DPO or PPO). Constitutional AI / RLAIF replaces some human labels with model critiques. Safety evals gate release.
vLLM PagedAttention + continuous batching + speculative decoding + FP8 quant. A single H100 serves 10-30 concurrent 70B chats when tuned.
Roughly: 80% pre-training compute, 10% data + eval, 5% alignment, 5% safety + serving. Inference eventually dominates once deployed at scale.
Open sources only. Content is synthesised from public research (Vaswani 2017, Chinchilla, LLaMA 1-3, Mixtral, DeepSeek, GPT-4 tech reports, o1 blog), open-source stacks (PyTorch, DeepSpeed, FSDP, vLLM, Megatron-LM, FlashAttention, tiktoken, SentencePiece) under Apache-2.0 / BSD / MIT, and Wikipedia (CC BY-SA). No proprietary weights, training data or internal docs are used or redistributed.
Trademarks. OpenAI, Anthropic, Meta, Google, Mistral, DeepSeek, xAI and other product / company names are trademarks of their respective owners, mentioned only as editorial and educational references (nominative fair use). This site is not affiliated with, endorsed by, or sponsored by any of them.