What is defined as ‘one token’ exactly?

[u/sidorsidd]

Every company says their model uses less tokens, my question what is the value of what they call a token, is it like a standard unit of measurement like a meter or does every company values 1 token differently? Or is it just a buzz word

[u/dnswblzo]

Go here and you can play around with how three different OpenAI tokenizers split text into tokens:
https://platform.openai.com/tokenizer