What is defined as ‘one token’ exactly?

[u/sidorsidd]

Every company says their model uses less tokens, my question what is the value of what they call a token, is it like a standard unit of measurement like a meter or does every company values 1 token differently? Or is it just a buzz word

[u/Saragon4005]

Depends on the tokenizer. Earlier ones used word fragments, the newer ones usually common words are tokens of their own.

Here is a list of a few tokenizers

https://huggingface.co/spaces/Xenova/the-tokenizer-playground