The difference between tokens and words (johndcook.com)
Large language models operate on tokens, not words, though tokens roughly correspond to words.
Tokens, Words
10 points by handfuloflight 46 days ago | 0 comments