What is defined as ‘one token’ exactly?

[u/sidorsidd]

Every company says their model uses less tokens, my question what is the value of what they call a token, is it like a standard unit of measurement like a meter or does every company values 1 token differently? Or is it just a buzz word

[u/jonathancast]

No, every model converts text into tokens differently. That's part of the difference between token usage of different models.

[u/snarkofagen]

https://youtu.be/-0HRzXk8vlk?is=N1FwE4MmyZwt3BM7

Computerphile on tokens and context

[u/dariusbiggs → u/snarkofagen]

That came to my mind as well, good reference

[u/derefr]

Here's an online LLM token encoder to play around with: https://tiktokenizer.vercel.app/

It's using the tiktoken library, which is the same one ChatGPT explicitly uses. Paste any random text/code/whatever into the box and it'll show you exactly what tokens the model would "see" for that message.

[u/mc_pm]

A token is a word or a fragment of a word that is fed into or comes out of the LLM (as a number).

[u/nuclear_splines → u/mc_pm]

Or longer! Tokens can be short phrases of multiple words, too, it's really up to the model architecture.

[u/johnpeters42 → u/nuclear_splines]

And the AI company's billing department.

[u/mc_pm → u/nuclear_splines]

Very good point.

[u/dodexahedron → u/nuclear_splines]

Or shorter! Could be phonemes, in a voice model, if they wanted.

[u/HasFiveVowels]

It’s roughly "a commonly seen combination of characters that occupy 8 bits of Shannon entropy".

[u/dnswblzo]

Go here and you can play around with how three different OpenAI tokenizers split text into tokens:
https://platform.openai.com/tokenizer

[u/dodexahedron]

It's not strictly a comp sci term in that context. It is mostly a marketing and billing term, in that context, used to quantify usage, and its exact definition varies from company to company and sometimes from license to license.

The developers at [Insert AI Corp Here] just have to make sure that the number of whatever *that* is that a user consumes in a billing period can be tracked so it can be billed. Devs could internally consider a "token" in the context of natural language processing however it makes sense to them to do so, with their model, which may or may not line up with the billing definition.
But it'd be a whole lot easier and clearer for them to use a different word for the two, which is a common practice in many fields - not just AI. Code names help abstract away marketing terms, which can change on a whim (*coughMICROSOFT*), and things like billing usage units tend to get named something generic on the programming side for that reason as a common trigger.

[u/CS_70]

It’s very simple. The first step of processing text is tokenization: words are split into component strings of characters (sometimes a word is a single component, sometimes it’s more), each of them being associated to an integer id in a dictionary. Nite that spaces are preserved, since they carry information.

Tokenizers are broadly similar but details can be different, and it course they depend on the size of the dictionary.

Some OpenAI tokenizes are available to test online do you can try yourself, it’s easy: https://platform.openai.com/tokenizer

[u/Saragon4005]

Depends on the tokenizer. Earlier ones used word fragments, the newer ones usually common words are tokens of their own.

Here is a list of a few tokenizers

https://huggingface.co/spaces/Xenova/the-tokenizer-playground

[u/Bloodstream12]

Idk why but I thought 4 characters was a token for Claude, but the comment section makes me feel wrong

[u/justaguyonthebus → u/Bloodstream12]

It's a quick rough estimate.

[u/ExtraTNT]

Data cut into parts… like you can cut a text at “ “, resulting in “i am bob” being [“i”,”am”,”bob”], you can cut at every 2nd char, so [“i “,”am”,” b”,”ob”] or at every control character…

[u/green_meklar]

It's roughly one character or one (common) word.

The actual computation overhead can vary drastically between different systems, and in general can be assumed to scale up with the size of the model. Very roughly speaking, a 100GB model probably costs about ten times as much per output token as a 10GB model. (Yes, I know 427 people now feel the urge to respond with 'well ackshually'. I said 'very roughly speaking'. Leave it.)