Ch 2 · Tokens — How AI Reads Text · AI & LLM Literacy
Topic 2 of 7 in AI & LLMs — How They Actually Work — 2 lessons.
Words Become Numbers
A model can't read letters — it only does maths. So text is first chopped into pieces called tokens (roughly words or word-chunks), and each token is mapped to a number using a fixed dictionary called a vocabulary. Let's build a toy version with re.
Building a Tiny Vocabulary
Now turn those tokens into numbers. We build a vocab dict mapping each unique token to an integer ID, then encode the sentence. This is exactly what happens before text reaches the model.