Types of tokens:
- Word tokens: Individual words, such as "hello" or "AI".
- Character tokens: Individual characters, such as letters or punctuation marks.
- Subword tokens: Smaller units of words, such as word stems or word roots.
- Phrase tokens: short phrases, such as "hello world" or "AI is cool".