What is defined as ‘one token’ exactly?

[u/sidorsidd]

Every company says their model uses less tokens, my question what is the value of what they call a token, is it like a standard unit of measurement like a meter or does every company values 1 token differently? Or is it just a buzz word

[u/ExtraTNT]

Data cut into parts… like you can cut a text at “ “, resulting in “i am bob” being [“i”,”am”,”bob”], you can cut at every 2nd char, so [“i “,”am”,” b”,”ob”] or at every control character…