TinyStories GPT: A Language Model from Scratch
Python, PyTorch, Transformers, BPE Tokenizer, Hugging Face Datasets
Show descriptionHide description
A GPT-style decoder-only transformer written and trained from scratch on a single laptop GPU (RTX 4060, 8 GB), with no pretrained weights or pretrained tokenizer. It trains its own byte-level BPE tokenizer with 8,192 tokens on the TinyStories dataset of about 2.1 million short stories, then trains a 29.6M-parameter model (8 layers, 8 heads, 512-token context, pre-norm blocks and weight tying) with bf16, AdamW and a cosine learning-rate schedule tied to a wall-clock budget. After 6 hours and 1.44 billion tokens it reaches a validation loss of 1.285 (perplexity 3.6) and writes coherent short stories.


















