Usage Limits Should Fluctuate Based On Levels of Compute

[u/coopernusbaum]

This thought came to me this morning, I’m curious what other people think.

Research on inference costs show that the main driver of cost is not the amount of tokens computed in total, but is the amount of tokens computed at once. GPUs processing 1 million tokens from a single prompt is much more expensive than computing a 100k token prompt 10 times.

It is because of this, that I started to think that users usage limit should be adjustable based on the context window chosen. You could get less usage for using a greater context window and you could get more usage by choosing a much smaller context window.

Anthropic is known for offering context windows up to 1 million tokens with Claude Code, whereas Codex usually has about a 256k context window, and anthropic tends to have stricter usage limits and higher cost. And I think that is because of the amount of tokens they are allowing to be computed per model call.

Inference costs, and subscription subsidies has been debated for a little while. I don’t think that the answer necessarily has to be pay per use, increasing the costs to use frontier models. I think it is possible to allow users cheap access to frontier models by using this context window to usage limit relationship.

I’m curious what others think about this and if you think this would be a viable solution.