Words Become Numbers · AI & LLM Literacy

Ch 2 · Tokens — How AI Reads Text — card 1 of 2.

A model can't read letters — it only does maths. So text is first chopped into pieces called tokens (roughly words or word-chunks), and each token is mapped to a number using a fixed dictionary called a vocabulary. Let's build a toy version with re.

import re

def tokenize(text):
    # split into words and punctuation
    return re.findall(r"[A-Za-z']+|[.,!?]", text.lower())

sentence = "Hello, AI! How are you?"
tokens = tokenize(sentence)
print("tokens:", tokens)
print("count: ", len(tokens))