nanochat: Build Your Own ChatGPT Clone - Complete Training Pipeline Guide

What is nanochat?

Full-Stack Pipeline

Complete training/inference pipeline from tokenizer to web UI in a single codebase

Train tokenizer using Rust implementation

Pretrain Transformer LLM on FineWeb

ChatGPT-like web UI included

Minimal & Clean

~8,000 lines of clean code with dependency-minimal approach

Single script execution

From scratch implementation

Dependency-minimal codebase

Capstone project of LLM101n

Complete Training Stages

Tokenization

Train custom tokenizer using new Rust implementation

Train tokenizer using new Rust implementation

Pretraining

Pretrain Transformer LLM on FineWeb dataset with CORE evaluation

Pretrain Transformer LLM on FineWeb

Evaluate CORE score across a number of metrics

Midtraining

Train on user-assistant conversations and tool use

User-assistant conversations from SmolTalk

Multiple choice questions

Tool use

Advanced Training Features

Supervised Fine-Tuning (SFT)

Evaluate chat model on world knowledge multiple choice, math, code

World knowledge multiple choice (ARC-E/C, MMLU)

Math (GSM8K)

Code (HumanEval)

Reinforcement Learning

Optional RL training with GRPO on GSM8K

RL model optionally on GSM8K with GRPO

Infrastructure & Deployment

Cloud-Ready

Boot up cloud GPU box and run single script

Boot up cloud GPU box

Run single script

As little as 4 hours later you can talk to your own LLM

Efficient Inference

KV cache, prefill/decode, and tool use in lightweight sandbox

Engine with KV cache, simple prefill/decode

Tool use (Python interpreter in lightweight sandbox)

Talk to it over CLI or ChatGPT-like WebUI

Performance Benchmarks

For as low as ~$100 in cost (~4 hours on an 8XH100 node), you can train a little ChatGPT clone that you can kind of talk to, and which can write stories/poems, answer simple questions.

About ~12 hours surpasses GPT-2 CORE metric.

As you scale up towards ~$1000 (~41.6 hours of training), it becomes more coherent and can solve simple math/code problems and take multiple choice tests.

A depth 30 model trained for 24 hours gets into 40s on MMLU and 70s on ARC-Easy, 20s on GSM8K, etc.

💰 Training Cost Breakdown

Basic ChatGPT Clone

$100 • ~4 hours

Can kind of talk to

Write stories/poems

Answer simple questions

8XH100 node

Surpasses GPT-2

~12hrs • GPT-2 Level

Surpasses GPT-2 CORE metric

About ~12 hours training

More Coherent

$1000 • ~41.6 hours

Solve simple math/code problems

Take multiple choice tests

40s on MMLU, 70s on ARC-Easy

20s on GSM8K

🎯 nanochat Goals & Features

Strong Baseline Stack

Get the full "strong baseline" stack into one cohesive, minimal, readable, hackable, maximally forkable repo.

Research Potential

Has potential to grow into a research harness, or a benchmark, similar to nanoGPT before it.

LLM101n Capstone

nanochat will be the capstone project of LLM101n (which is still being developed).

Report Cards

Write a single markdown report card, summarizing and gamifying the whole thing.

Repository Features

Capstone project of LLM101n (still being developed)
Potential to grow into research harness or benchmark, similar to nanoGPT
Cohesive, minimal, readable, hackable, maximally forkable repo
Not finished, tuned or optimized - likely quite a bit of low-hanging fruit
Generates single markdown report card, summarizing and gamifying the whole thing
Not finished, tuned or optimized - likely quite a bit of low-hanging fruit

Chris Prakoso

Augmented Humanity | Practical AI, Data & Analytics

📚 Life Apprentice 🤖 Augmented Humanity

Follow:

Connect with me for cutting-edge AI/ML insights, hands-on LLM tutorials, and the latest in open-source machine learning development

Connect on LinkedIn Send Message