Let's build the GPT Tokenizer

Why it was recommended

Worth spending 2h for the deeper understanding if you deal with LLMs.

Strong recommendationcranium · Original2024-10

What it covers

Andrej Karpathy builds the GPT tokenizer from scratch, implementing byte pair encoding along with encode and decode. He shows how many odd LLM behaviors trace back to. It is for someone who already knows the basic terms and wants the next step.

Video details

Channel

Andrej Karpathy

Length

2 hr 13 min

Year

2024

More in this topic

• Transformers, the tech behind LLMs | Deep Learning Chapter 5
• Deep Dive into LLMs like ChatGPT
• Let's build GPT: from scratch, in code, spelled out.
• A Hackers' Guide to Language Models
• [1hr Talk] Intro to Large Language Models
• Sparks of AGI: early experiments with GPT-4