Model Architecture
6 layers, 6 heads, 384 dim
Training Parameters
CPU Sharding & Hardware
4
CPU sharding splits gradient accumulation across 4 parallel threads. Effective batch = 64 × 1 × 4 = 256 tokens/step.
Resource Estimate
Calculating...
About nanoGPT
Andrej Karpathy's nanoGPT — the simplest, fastest repository for training/finetuning medium-sized GPTs.
This tool generates CPU-sharded configs that split gradient accumulation across multiple threads for model-parallel training without GPU.
Supports: scratch training, GPT-2 fine-tuning, Shakespeare/OpenWebText datasets.