Andrej Karpathy — Guide

Andrej Karpathy — User Guide

Karpathy’s DL materials.

Visit website VPN may be required Free

Strengths

Top AI researchers explain in person
Build from scratch and gain a deep understanding of the principles
The code is clear and can be run directly
The explanation is simple and in-depth, suitable for learners with a certain foundation.
Free and open to the public, quality comparable to top courses

Best for

In-depth understanding of neural network and deep learning principles
Learn to build Transformer models like GPT from scratch
Understand how LLM works
Improve the theoretical and practical capabilities of AI/ML
Understand cutting-edge thinking in the field of AI

Core course recommendations

A few of Karpathy’s most important video lessons.

Scenario

Neural Networks: Zero to Hero Series

Prompt example

This is Karpathy’s most important course series,
Build a neural network from scratch to GPT:

1. The spelled-out introduction to neural networks
   - Build a tiny neural network library from scratch
   - Understand backpropagation
   - Duration: approximately 2.5 hours

2.Building makemore
   -Character level language model
   - From bigram to MLP
   - Duration: approximately 1.5 hours

3. Building a GPT from scratch
   - Complete implementation of Transformer
   - Understand the attention mechanism
   - Duration: approximately 2 hours

4. Let's build the GPT Tokenizer
   - Understand BPE tokenization
   - Duration: approximately 2 hours

GitHub: github.com/karpathy/nn-zero-to-hero

Output / what to expect

After completing this series,

You will truly understand how GPT works,

Rather than just calling the API.

Tips

It is recommended to write the code by hand while reading. Karpathy's code is on GitHub and can be run directly.

Scenario

Key videos for understanding LLM

Prompt example

A few important videos for understanding LLM:

1. "Intro to Large Language Models" (2023)
   - 1 hour overview of LLM
   - Suitable for those with technical background
   - Explain the capabilities, limitations and future of LLM

2. "State of GPT" (Microsoft Build 2023)
   - GPT training process
   - How RLHF works
   - How to use GPT effectively

3. "Let's reproduce GPT-2" (2024)
   - Complete reproduction of GPT-2
   - Contains training process
   - Duration: approximately 4 hours

Suitable for the crowd:
- Have Python basics
- Understand basic machine learning concepts
- Want to deeply understand the principles of LLM

Output / what to expect

These videos provide the most in-depth understanding of LLM,

“Intro to LLMs” is one of the best introductory videos to LLM,

Even professional researchers recommend viewing.

Tips

You can first read "Intro to LLMs" to get the overall concept, and then study the specific implementation in depth.

Study suggestions

How to study Karpathy's courses most effectively.

Scenario

Learning path planning

Prompt example

Recommended learning path:




Prerequisite knowledge:


- Python programming basics


- Basic linear algebra (matrix operations)


- Basics of calculus (concept of derivatives)




Learning order:


1. Read "Intro to LLMs" first (understand the big picture)


2. Learn "micrograd" (understand backpropagation)


3. Learn the "makemore" series (language model basics)


4. Learn "nanoGPT" (complete Transformer)


5. In-depth understanding of "GPT Tokenizer" (understanding tokenization)




Auxiliary resources:


- 3Blue1Brown's Neural Network Series (Visualization)


- Fast.ai’s practical courses (Engineering Practice)

Output / what to expect

Follow this path to learn,

Usually requires 2-3 months of spare time,

Upon completion your understanding of LLM will be far superior to that of most practitioners.

Tips

Don't skip hands-on practice, understanding code is the core of learning, and just watching videos will have limited effect.

Compared with similar tools

Tool	Strength	Best for	Pricing
Andrej Karpathy This tool	Top researchers explain, build from scratch, and gain in-depth understanding	Learners who want to deeply understand the principles of AI	completely free
Fast.ai	Pay more attention to practice, code first	Quickly get started with deep learning applications	completely free
DeepLearning.AI	Systematic courses with certificates	System learning, certification required	Free Audition / Paid Certificate
3Blue1Brown	Visual explanation of mathematical principles	Understand the basics of mathematics	completely free

Sources & references: