microgpt

Andrej Karpathy released an art project named microgpt, a single-file containing only 200 lines of pure Python code, which can train and infer GPT models without dependencies. The project implements all algorithmic content from dataset processing, tokenizer, automatic differentiation engine to GPT-2 style neural network architecture, Adam optimizer, training and inference loop. The code can be obtained from GitHub gist and karpathy.art, and has been released as a triptych.