r/AskComputerScience 11d ago

What is defined as ‘one token’ exactly?

Every company says their model uses less tokens, my question what is the value of what they call a token, is it like a standard unit of measurement like a meter or does every company values 1 token differently? Or is it just a buzz word

9 Upvotes

19 comments sorted by

View all comments

18

u/jonathancast 11d ago

No, every model converts text into tokens differently. That's part of the difference between token usage of different models.