Token
ELI5 — The Vibe Check
In AI-land, a token is a chunk of text — roughly 3/4 of a word. Every time you talk to an AI, your message gets chopped into tokens, processed, and you get tokens back. More tokens = more expensive. It's the currency of the AI world. 'Hello world' is 2 tokens, but emoji are weirdly expensive and non-English text uses more tokens per word.
Real Talk
A token is the fundamental unit of text processing in LLMs. Tokenizers split text into subword units using algorithms like BPE (Byte Pair Encoding). Token count determines both cost (API pricing is per-token) and context window usage. English text averages ~1.3 tokens per word; code typically has a higher token-per-character ratio due to syntax.
When You'll Hear This
"That prompt is 4,000 tokens — it's going to cost us." / "We optimized the system prompt from 2,000 to 800 tokens."
Related Terms
Context Window
A context window is how much text an AI can 'see' at once — its working memory.
LLM (Large Language Model)
An LLM is a humongous AI that read basically the entire internet and learned to predict what words come next, really really well.
Prompt Engineering
Prompt engineering is the art of talking to AI so it actually does what you want.
Tokenizer
A tokenizer chops text into pieces that the AI model can understand — but not in ways humans would expect.