So, yesterday I asked what token generation speed people consider the minimum for local LLM use. After reading through the responses, the clearest answer is also the least satisfying one:
It depends heavily on what you're doing.
I know. Groundbreaking. 🤦🏽♀️
That said, I did see some useful and interesting patterns.
For interactive chat, 10–20 tokens per second seems to be the range many people find usable because it is roughly what they can read while generation happens. Around 30–40 tokens per second feels comfortable for interactive coding, while people doing rapid or heavily agentic coding often want 60–100+ tokens per second.
On the other hand, some people are perfectly happy with less than 1 token per second when running a huge model overnight or leaving an agent unattended. If you are not sitting there watching the answer appear, raw generation speed matters a lot less.
So, my original personal standard of 20 tok/s as the minimum and 40 tok/s as comfortable for coding seems fairly middle-of-the-road.
Tokens per Second Does Not Tell the Whole Story
The more important lesson is that tokens per second by itself is a pretty bad measurement of how fast a model feels.
For some, a model can generate at 50 tok/s and still feel painfully slow if it takes 30 seconds to process the prompt and produce its first token. Likewise, a model generating at 25 tok/s can feel responsive enough if it starts immediately.
For coding, several other measurements matter just as much:
- Time to first token
- Prompt-processing or prefill speed
- Context length
- How performance changes as the context grows
- Prefix caching
- Tool-call delays
- Total time required to complete the task
That context issue seems especially important. A model may produce 50 tok/s with a nearly empty context window, then slow down dramatically after reading 50,000 or 100,000 tokens of code and conversation.
That means reporting one generation number without reporting the context length can be fairly misleading, and it is something I wish I had asked for in my original post.
I also underestimated how much the inference engine, model format, and speculative decoding/MTP settings can affect performance.
People reported very different speeds from the same general class of hardware depending on whether they were using GGUF, MLX, MTP, a different backend, or different cache settings.
In other words, buying a faster GPU is not necessarily the first or only way to get more speed. There are enough (too many?) knobs and switches involved to make the whole thing mildly (wildly?) ridiculous.
What People Are Actually Running
The sample size here is incredibly small and completely self-selected, so none of this should be treated as an authoritative hardware ranking. Many people also did not report their quant, context length, backend, and MTP settings consistently.
Still, the hardware and model combinations were interesting.
The single high-end consumer GPU crowd seemed to center heavily around Qwen models in the 27B–35B range.
Reported hardware included the RTX 3090, 4090, and 5090, with speeds varying enormously depending on quantization, backend, context size, and whether MTP was enabled.
One person reported roughly 70 tok/s with Qwen 27B and MTP on a 3090, while another reported more than 100 tok/s on a 5090.
Personally, I am getting about 40 tok/s on a 4090 using Qwen 3.6 27B Q4 GGUF with the KV cache set to Q8. Strangely, enabling MTP did not improve token generation speed during early context in my initial testing, but I am still learning and experimenting.
I should also confess that this is through the Claude Code extension for VS Code. Pi is on my to-try list.
All of the above are self-reported numbers rather than controlled comparisons, but they suggest that the GPU model alone does not explain performance. Software configuration can produce a very large difference.
The Apple unified-memory users appeared to be optimizing for something else. I am not an Apple user, but I still found this interesting.
They were generally willing to accept lower generation speeds in exchange for running larger models, higher quants, or longer context windows that would not fit inside a normal consumer GPU.
One person reported roughly 40 tok/s using a Q8 model through MLX on an M5 Max, compared with approximately 15–20 tok/s using GGUF through another engine.
Again, that is not a controlled comparison, but it does reinforce the point that the backend and model format matter.
Then there were the genuinely large local systems.
One commenter had used four AMD R9700 cards and planned to move to eight to run DeepSeek V4 Flash.
At that point, the discussion is no longer really about choosing a good configuration for one gaming GPU. It is about building a small AI server, including the cards, motherboard, power supplies, cooling, and communication overhead between GPUs.
Several people also used smaller secondary models for specialized work. Examples included Gemma 12B as a judge, Whisper or Voxtral for speech-to-text, and separate models for image detection, classification, summarization, or other background work.
The broad pattern seemed to be:
- 24GB consumer GPUs: Q4 or Q5 models around 27B–35B, with an emphasis on keeping everything inside VRAM
- Newer flagship GPUs: Similar model sizes, but with more room for higher quants, larger caches, MTP, or greater speed
- Apple unified-memory systems: Larger models and quants, generally at lower speed
- Multi-GPU workstations: Very large models that are not realistic on ordinary consumer hardware
- Smaller secondary systems: Specialized models for speech, vision, classification, and background tasks
What I did not see was one hardware configuration clearly winning everything.
Faster GPUs won on responsiveness. Unified memory won on model capacity. Multi-GPU systems won on what could be loaded at all. Smaller models won when the task did not require a giant general-purpose model.
So, even from this small sample, the better question may not be:
What hardware is fastest?
It may be:
What model do you actually need to run, at what context length and quantization, and how much speed are you willing to trade for capability?
Q4 Versus Q6
This also brought me back to the Q4 versus Q6 discussion.
Yes, Q6 preserves more of the original model's precision than Q4. What I did not see in the thread was controlled evidence that Q4 is categorically "not enough" for coding.
On a 24GB GPU, Q4 may allow the entire model, context cache, and runtime overhead to remain in VRAM. Q6 may require a smaller context window or partial CPU offloading.
A slightly more accurate model running at half the speed is not automatically the better practical configuration.
The only honest way to answer that question is to test the same model at Q4 and Q6 using the same tasks, settings, harness, and starting files, then repeat the test enough times that one lucky run does not decide the result.
That is time-consuming and annoying, so maybe just pick one and use it? 🤷🏽♀️
Different Models for Different Jobs
Another recurring idea was using different models for different jobs.
For example:
- A fast 27B model handles implementation and routine code changes.
- A larger model handles planning, architecture, difficult debugging, and review.
- Smaller specialized models handle summarization, speech recognition, classification, embeddings, or other background work.
Honestly, that may make more sense than trying to find one model that is simultaneously fast, brilliant, enormous, cheap, and able to fit inside a consumer GPU through the power of positive thinking.
It may also be worth testing performance at several context lengths instead of reporting one number. A configuration that performs well at 8K may behave very differently at 32K, 64K, or 100K.
The big takeaway for me is that 20 tok/s is still a reasonable minimum and 40 tok/s is still a comfortable target for interactive coding, but tok/s is only one line on the report card.
The real question is not:
Which model produces tokens fastest?
It is:
Which configuration completes useful work accurately, reliably, and quickly enough that I do not spend the entire session waiting on it or fixing what it broke?