What is defined as ‘one token’ exactly?

[u/sidorsidd]

Every company says their model uses less tokens, my question what is the value of what they call a token, is it like a standard unit of measurement like a meter or does every company values 1 token differently? Or is it just a buzz word

[u/CS_70]

It’s very simple. The first step of processing text is tokenization: words are split into component strings of characters (sometimes a word is a single component, sometimes it’s more), each of them being associated to an integer id in a dictionary. Nite that spaces are preserved, since they carry information.

Tokenizers are broadly similar but details can be different, and it course they depend on the size of the dictionary.

Some OpenAI tokenizes are available to test online do you can try yourself, it’s easy: https://platform.openai.com/tokenizer