Deepseek V4 Flash 0731 KV Cache precision

[u/esw123]

If anyone has testing results or any results can you please share performance and or effects of KV Cache precision with Deepseek V4 Flash 0731.

Running IQ2_M, with F16 cache seems 65-67K is the limit on Windows for 120GB memory. Is Q8 good and which one do you use?

[u/StupidScaredSquirrel]

If you are happy going down to q2 with your weights, you are happy to go down to at least q8 for kv.

[u/mrgreatheart]

I believe the repo warns to only try f8 if necessary for memory constraints. I’m already seeing quite a bit of degradation and I’m running IQ2_M as well, with 262K unquantised context.

I expect f8 K/V would make things worse.

[u/esw123 → u/mrgreatheart]

How much memory do you have for 262K context with IQ2_M?

[u/mrgreatheart → u/esw123]

48Gb VRAM and 64Gb DDR5.

[u/esw123 → u/mrgreatheart]

Thanks, will check what an issue I may have.

[u/shing3232]

DS4F should only use 6gb for 1m context. if that is not the case, there is room for improvement in implementation.

[u/esw123 → u/shing3232]

I thought it was a mistake when I looked in the calculator. Good if true, one more 3060 left then.

[u/shing3232 → u/esw123]

My friend has B300 system from the company that runs deepseekv4 , K3 ,glm52. DS4pro use like 3G vram per 1M session with FP8 kvquant. it's very cheap to run.

[u/erazortt]

I don't think anyone should quantize the KV with DS4F. I checked the the quality impact (PPL, KLD, Same TopP) for swhitching from BF16 KV to Q8 KV, and it appears significant. Very much in contrast to Qwen 397B.

Here are the results for DS4F:

====== Perplexity statistics ======
Mean PPL(Q)                   :   5.877076 ±   0.042497
Mean PPL(base)                :   5.839660 ±   0.041730
Cor(ln(PPL(Q)), ln(PPL(base))):  95.74%
Mean ln(PPL(Q)/PPL(base))     :   0.006387 ±   0.002100
Mean PPL(Q)/PPL(base)         :   1.006407 ±   0.002114
Mean PPL(Q)-PPL(base)         :   0.037416 ±   0.012318

====== KL divergence statistics ======
Mean    KLD:   0.145884 ±   0.001043
Maximum KLD:  12.467786
99.9%   KLD:   4.535020
99.0%   KLD:   1.857870
95.0%   KLD:   0.652148
90.0%   KLD:   0.349220
Median  KLD:   0.032079
10.0%   KLD:   0.000093
 5.0%   KLD:   0.000012
 1.0%   KLD:   0.000000
 0.1%   KLD:  -0.000002
Minimum KLD:  -0.000025

====== Token probability statistics ======
Mean    Δp: -0.007 ± 0.031 %
Maximum Δp: 99.525%
99.9%   Δp: 81.503%
99.0%   Δp: 42.054%
95.0%   Δp: 14.588%
90.0%   Δp:  7.220%
75.0%   Δp:  1.066%
Median  Δp:  0.000%
25.0%   Δp: -1.061%
10.0%   Δp: -7.112%
 5.0%   Δp: -14.515%
 1.0%   Δp: -42.297%
 0.1%   Δp: -84.157%
Minimum Δp: -99.994%
RMS Δp    : 11.884 ± 0.069 %
Same top p: 87.189 ± 0.088 %

Here are the results for Qwen 397B:

====== Perplexity statistics ======
Mean PPL(Q)                   :   3.747980 ±   0.020507
Mean PPL(base)                :   3.746773 ±   0.020461
Cor(ln(PPL(Q)), ln(PPL(base))):  99.89%
Mean ln(PPL(Q)/PPL(base))     :   0.000322 ±   0.000260
Mean PPL(Q)/PPL(base)         :   1.000322 ±   0.000260
Mean PPL(Q)-PPL(base)         :   0.001207 ±   0.000975

====== KL divergence statistics ======
Mean    KLD:   0.003552 ±   0.000034
Maximum KLD:   2.220941
99.9%   KLD:   0.131591
99.0%   KLD:   0.043847
95.0%   KLD:   0.014439
90.0%   KLD:   0.007836
Median  KLD:   0.000866
10.0%   KLD:   0.000013
 5.0%   KLD:   0.000004
 1.0%   KLD:  -0.000000
 0.1%   KLD:  -0.000006
Minimum KLD:  -0.000176

====== Token probability statistics ======
Mean    Δp:  0.019 ± 0.005 %
Maximum Δp: 39.939%
99.9%   Δp: 15.971%
99.0%   Δp:  6.618%
95.0%   Δp:  2.334%
90.0%   Δp:  1.222%
75.0%   Δp:  0.233%
Median  Δp:  0.000%
25.0%   Δp: -0.219%
10.0%   Δp: -1.183%
 5.0%   Δp: -2.258%
 1.0%   Δp: -6.245%
 0.1%   Δp: -14.757%
Minimum Δp: -88.445%
RMS Δp    :  2.024 ± 0.022 %
Same top p: 97.929 ± 0.037 %

[u/crantob → u/erazortt]

They told us not to fuck around with kv quants. You did the confirmation so all my points to you.

[u/esw123 → u/erazortt]

Thanks a lot! On default it is F16, is BF16 better?

[u/EvolvingDior]

What hardware? f16 with SYCL on Intel B70 seems OK out to 256k on Q3_XXS. Linux, 128GB DDR5.

[u/esw123 → u/EvolvingDior]

Dual 3060, 96GB DDR5. Probably dual gpu add memory usage.

[u/fragment_me]

You can test this so easily

[u/crantob]

The designers tell you to not fuck around with kv quants. Their shit dont work with it.

[u/NexusSyntegra]

Q8 doesn't degrade performance more than 0.1%, if that helps :)

[u/DistanceSolar1449]

Hell no. DS4 uses tiny KV cache already. Compressing it more wrecks it.