The difference between tokens and words
(johndcook.com)
Large language models operate on tokens, not words, though tokens roughly correspond to words.
Large language models operate on tokens, not words, though tokens roughly correspond to words.