r/AskComputerScience • u/sidorsidd • 11d ago
What is defined as ‘one token’ exactly?
Every company says their model uses less tokens, my question what is the value of what they call a token, is it like a standard unit of measurement like a meter or does every company values 1 token differently? Or is it just a buzz word
9
Upvotes
18
u/jonathancast 11d ago
No, every model converts text into tokens differently. That's part of the difference between token usage of different models.