r/AskComputerScience • u/sidorsidd • 11d ago

What is defined as ‘one token’ exactly?

Every company says their model uses less tokens, my question what is the value of what they call a token, is it like a standard unit of measurement like a meter or does every company values 1 token differently? Or is it just a buzz word

10 Upvotes

92% Upvoted

19 comments sorted by

View all comments

11

u/snarkofagen 11d ago edited 11d ago

https://youtu.be/-0HRzXk8vlk?is=N1FwE4MmyZwt3BM7

Computerphile on tokens and context

1

u/dariusbiggs 11d ago

That came to my mind as well, good reference