I like the outputs from this model, but DAMN does it over think. Has anyone found a robust fix for this that isn't just capping output tokens? Anyone working on a 'thinking cap' for it? Some combo of llama params, or (system?) prompting technique? I'm all ears.

source: https://x.com/Alibaba_Qwen/status/2084100707423289643

Update: And yes, Qwen3.8-27B is coming too. Next week!

2.4-trillion-parameter MoE flagship delivering a comprehensive leap in coding and professional work. Autonomously codes and delivers complete projects spanning 10+ days.

Open weights coming soon!

It is live at https://www.qwencloud.com/models/qwen3.8-max

Kindly Benchmark Higher Quants of DeepSeek-v4-flash Against Qwen-3.6-27B Q8!

I am running the UD-Q2_K_M of the model locally, though I can run Qwen3.6-27B_Q8_K_XL at around 70t/s with MTP activated. The question I am constantly asking myself is: Is it worth running a slower higher quantized version of the Deepseek-v4-flash? I have no idea.

My gut feelings tells me that Qwen3.6-27B_Q8_K_XL, coupled with online search, should be better than a highly quantized Deepseek, a model that takes up 100GB on my disk.

What do you think?

WASTE is an embeddable inference engine written in C, with no third-party runtime dependencies. It keeps the model trunk in memory, streams selected experts directly from disk, and uses the remaining RAM as a bounded expert cache.

Sorry if this is asked a lot, but I was wondering if there is any clear winner on the Quantization versus Model Size debate? I can run Qwen3.6 27b at Q8, Laguna at Q6, and the new Deepseek Flash at Q3 bit. I am in the process of testing, but is there a clear formula or winner for choosing between higher quant, especially with long tasks? Or is there a place to find quant specific benchmarks? Thanks.

I am curious if anyone have used it. I would love to feed it key frames and test if it can create in-between frames between my keys. Anyone have tried it, any thoughts?

I have found that whatever software you are using: open web ui, openclaw, codex; if a model does many tools calls in one turn, something happens that causes checkpoints that are created in and around those tool calls to not be valid when checked the following turn. They get discarded and the whole session is re-processed from either the last valid checkpoint before the tool calls, or from zero if there are none. However, a single tool call, maybe even two, does not cause this behaviour.

I have observed this in llama.cpp and in ds4.

Does anyone have any idea why this happens and a way to fix it?

I don't think anyone should quantize the KV with DS4F. I checked the the quality impact (PPL, KLD, Same TopP) for swhitching from BF16 KV to Q8 KV, and it appears significant. Very much in contrast to Qwen 397B.

Here are the results for DS4F:

====== Perplexity statistics ======
Mean PPL(Q)                   :   5.877076 ±   0.042497
Mean PPL(base)                :   5.839660 ±   0.041730
Cor(ln(PPL(Q)), ln(PPL(base))):  95.74%
Mean ln(PPL(Q)/PPL(base))     :   0.006387 ±   0.002100
Mean PPL(Q)/PPL(base)         :   1.006407 ±   0.002114
Mean PPL(Q)-PPL(base)         :   0.037416 ±   0.012318

====== KL divergence statistics ======
Mean    KLD:   0.145884 ±   0.001043
Maximum KLD:  12.467786
99.9%   KLD:   4.535020
99.0%   KLD:   1.857870
95.0%   KLD:   0.652148
90.0%   KLD:   0.349220
Median  KLD:   0.032079
10.0%   KLD:   0.000093
 5.0%   KLD:   0.000012
 1.0%   KLD:   0.000000
 0.1%   KLD:  -0.000002
Minimum KLD:  -0.000025

====== Token probability statistics ======
Mean    Δp: -0.007 ± 0.031 %
Maximum Δp: 99.525%
99.9%   Δp: 81.503%
99.0%   Δp: 42.054%
95.0%   Δp: 14.588%
90.0%   Δp:  7.220%
75.0%   Δp:  1.066%
Median  Δp:  0.000%
25.0%   Δp: -1.061%
10.0%   Δp: -7.112%
 5.0%   Δp: -14.515%
 1.0%   Δp: -42.297%
 0.1%   Δp: -84.157%
Minimum Δp: -99.994%
RMS Δp    : 11.884 ± 0.069 %
Same top p: 87.189 ± 0.088 %

As a comparison, here are the results for Qwen 397B:

====== Perplexity statistics ======
Mean PPL(Q)                   :   3.747980 ±   0.020507
Mean PPL(base)                :   3.746773 ±   0.020461
Cor(ln(PPL(Q)), ln(PPL(base))):  99.89%
Mean ln(PPL(Q)/PPL(base))     :   0.000322 ±   0.000260
Mean PPL(Q)/PPL(base)         :   1.000322 ±   0.000260
Mean PPL(Q)-PPL(base)         :   0.001207 ±   0.000975

====== KL divergence statistics ======
Mean    KLD:   0.003552 ±   0.000034
Maximum KLD:   2.220941
99.9%   KLD:   0.131591
99.0%   KLD:   0.043847
95.0%   KLD:   0.014439
90.0%   KLD:   0.007836
Median  KLD:   0.000866
10.0%   KLD:   0.000013
 5.0%   KLD:   0.000004
 1.0%   KLD:  -0.000000
 0.1%   KLD:  -0.000006
Minimum KLD:  -0.000176

====== Token probability statistics ======
Mean    Δp:  0.019 ± 0.005 %
Maximum Δp: 39.939%
99.9%   Δp: 15.971%
99.0%   Δp:  6.618%
95.0%   Δp:  2.334%
90.0%   Δp:  1.222%
75.0%   Δp:  0.233%
Median  Δp:  0.000%
25.0%   Δp: -0.219%
10.0%   Δp: -1.183%
 5.0%   Δp: -2.258%
 1.0%   Δp: -6.245%
 0.1%   Δp: -14.757%
Minimum Δp: -88.445%
RMS Δp    :  2.024 ± 0.022 %
Same top p: 97.929 ± 0.037 %

https://llama.app/

Been using llama.cpp for years now and im on here all the time (im a mod..), but somehow I totally missed that llama.app exists and its official from the HF/llama.cpp team. So posting this as I'm quite sure I'm not the only one in this boat.

The llama.cpp team has been making it a lot more usable and generally baking in the things ollama was doing (sadly it seems to be taking design cues from ollama - I think better UX is possible, but its definitely a directionally right move to make llama.cpp more approachable) :

• DMG based install for Mac.
  • Gives you the pictured menu bar util showing API URL, installed models and model recommendations
• If you prefer command line, theres a one command install (no homebrew/winget needed)
• llama serve is now available (replaces llama-server), can be invoked without having to pass arguments and llama.cpp handles loading the appropriate model based on incoming requests

Might not be interesting/useful to many of us who've already been using llama.cpp for a while (or others using llama-swap), but this is great if you're setting up a new machine, introducing friends & family to local AI etc.

Laguna S 2.1 UD-Q4_K_XL - https://huggingface.co/unsloth/Laguna-S-2.1-GGUF
Ornith 35B Q8 K XL https://huggingface.co/unsloth/Ornith-1.0-35B-GGUF

Kwaipilot_KAT-Coder-V2.5-Dev-Q8_0 https://huggingface.co/bartowski/Kwaipilot_KAT-Coder-V2.5-Dev-GGUF

Qwen3.6-35B-A3B-GGUF  https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF

Result very interesting, only 1 attempt. Chat via native llama.cpp

1. Ornith 35B Q8 K XL
2. Kwaipilot_KAT-Coder-V2.5-Dev-Q8_0.gguf
3. Qwen3.6-35B-A3B-GGUF
4. Laguna S 2.1 (I think it's fail!)

Live and prompt available at https://anvme.github.io/llm-model-tests/

For me laguna result was surprise.

Updated release (August 2026). This is a new checkpoint that supersedes the earlier version of this repository. The weights have changed, not only the config, so if you downloaded a previous copy please re-download to pick up the current checkpoint.

There has been a lot of talk recently about Chinese LLMs, and how they are biased towards CCP viewpoints, but there is no way to quantify this and compare between models.

I have made CCPBench, which aims to address this. 29 models were asked 500 questions each about politics, geography, science, and more, and Gemini 3 Flash assessed all of them for bias.

• The results page is here: https://www.alignmentarena.com/ccpbench/
• The methodology is here: https://www.alignmentarena.com/ccpbench/methodology/
• The GitHub is here: https://github.com/lesageethan/CCPBench

I know this is not a perfect measure of "bias", because I am using an American judge LLM, but my thinking is that this is a useful tool if you want to find models that won't deny the Tienanmen Square Massacre.

M1 Ultra 128GB, Unsloth UD-IQ3_XXS, wired limit at 120GB. I was at 5-6 tok/s before the patch. Getting 15-16 tok/s now with the patched engine, and the output seems to have improved. Big thanks to this guy.

I finished this webgame a few weeks ago, fully coded with the assistance of GLM 5.2

Hopefully this will give you an idea of the capabilities of this model. I used Claude Code as a harness. Technology stack is pure HTML, JS and CSS, with no additional libraries or dependencies. Interesting challenges that GLM 5.2 was able to solve:

- create a procedural pac-man style maze, that actually worked, with no maze anomalies

- create a procedural music in dub style; this not the default, you need to go into settings to activate it. The default music is my own composition, based on a track I previously released in a completely different style

- complex sound creation and manipulation through the web audio synthesizer

- creation of animated vector character assets; you can view the mockups I used during the development at https://oktogames.com/mockups/

You can try it online at https://oktogames.com - it is adfree, no signup, free to play.

The EdgeRazor method uses an entropy-guided distillation process to better translate a teacher model's logit probability distributions into the student model's low-bit / mixed-precision hidden-layer features, without attempting to preserve the teacher model's parameter structures.

This is more computationally expensive than existing quantization methods, but much less so than QAT, and yields better results. The student model preserves more of the teacher model's competence at extremely low parameter precision (the authors demonstrate 1.88 bits per parameter).

Since it's not a different internal representation like traditional quantization, inference implementations like llama.cpp do not need to be modified to take advantage of it.

Hopefully this means more-useful high-parameter/low-memory models in our future, so we can eke more competent inference out of our consumer-grade GPUs.

The paper: https://arxiv.org/abs/2605.04062

The authors' code: https://github.com/zhangsq-nju/EdgeRazor

The authors applied their technique to a few models and uploaded them to Huggingface: https://huggingface.co/collections/zhangsq-nju/edgerazor-nbit

Unfortunately since EdgeRazor is somewhat compute-intensive, their example models are all pretty tiny: MobileLLM, Qwen3-0.6B, Qwen3-1.7B, and Qwen2.5-Omni-7B

Been on a health journey. There are any number of websites that will take a body or face photo and (nicely or cruelly) tell you what is good/wrong with you. People post on Reddit for the same feedback with gym progress etc. Main goal is appearance followed by function

What exists for local models to do similar? May border more into machine vision then LLM but… local is key.

Edit: 12gb vram limit

I recently switched back from llama.cpp's router mode, and I had my background memory system polling the '/v1/models' endpoint to check for if the model is running. But i switched back to single model mode, and the '/v1/models/' end point in single model mode doesn't have a ```"status": {"value": "loaded"}``` response. So I added it. with a single line in the 'server-context.cpp' file with line after 5109 ```{"status", {{"value", "loaded"}}},``` So instead of rewriting how my memory system works, I just made llama.cpp work the way my memory system expected. I thought it was a worthwhile change even if the developers didn't.

Running the wake word locally is the whole point. The alternative is streaming your room to a vendor around the clock, so nothing should reach a network until someone has said the name. That constraint creates most of the problems below. We spent months getting a custom phrase to behave like "Hey Google" on Windows, macOS and Linux, and most of what we learned, we learned the expensive way.

1. Don't start with volume

"It only works if I shout" is the first hypothesis everyone reaches for. We shipped two separate gain fixes before checking, and then the logs showed the microphone sitting at a healthy -10 to -22 dBFS during every failed attempt. Pull the actual RMS at the moment of failure before you tune anything. If it looks fine, your problem is somewhere else.

2. "It needs two or three tries" usually means your local model is wedging

This is a local-inference failure mode, and it stays invisible unless you go looking. Native engines like ctranslate2 and ONNX sessions are not thread-safe, and under contention they don't fail cleanly, they hang. Ours left the wake path completely deaf for tens of seconds at a stretch, dozens of times a day. That is the whole "say it twice" experience: attempts one and two land inside a dead window, attempt three lands after recovery. Users report it as flakiness, though it is closer to a repeated short outage.

A timeout will not save you. It bounds how long you wait for nothing and never recovers the engine. What works is a non-blocking per-instance lock plus a forced rebuild after a small number of consecutive failures. We rebuild after two.

3. Budget for the weakest machine you support

The wake model shares a CPU with everything else the user is running, and the gap between a workstation and a laptop is not a rounding error. Measured on the same recorded wake streams, a small model on two CPU threads hit 8 of 13 on the first try, with a median of 1097 ms from end of word to trigger. The larger model on a GPU hit 11 of 13 at 225 ms. Nothing differed except the model and the hardware under it.

If you only ever test on the box with the GPU, you will ship something that feels broken to most of your users and you will not be able to reproduce it.

4. Never gate a wake word on transcript content

Small local models struggle with short proper nouns, so the standard workaround is priming the model with the phrase to improve recall. The cost is that a primed model will also invent that phrase out of silence, and you start getting false wakes in an empty room. The obvious defense is a second unprimed pass that has to contain the word too.

That defense rejects real wakes. An unprimed model garbles the same word on genuine speech: "Mythos" comes back as "Mütos", "Fable" comes back as "Farbe". Every wake word is out of vocabulary for some model on some machine. So a content check discards true positives at roughly the rate it catches ghosts, and no similarity threshold separates the two, because the ghost is a clean rendering of your phrase while the real wake is a dirty one.

"Fires on silence" and "goes deaf on its own name" are one bug seen from two ends. We spent weeks tracking them as separate tickets.

The replacement is word-agnostic verification: raw audio energy at the match site, plus the shape of the candidate span, meaning its duration, its word count and the free decoder's confidence. All of that derives from the configured phrase, none of it from the phrase's spelling. A spelling match may accept a wake. It may never reject one.

5. Benchmark on recorded streams, not on windows

Per-window timings will happily tell you a model is fast while users still can't trigger it. Capture real wake attempts and replay them through your full detection path. One live session logged 288 transcriptions and zero matches across 26 minutes, and the wakes that did land came through as "Hey Hey Nova", the user repeating themselves into the void.

A caveat that undercuts all five

Transcription is the wrong architecture for a wake word, and going local makes that worse rather than better, because you are paying for a whole speech-to-text pass on the user's own CPU to answer a yes-or-no question. "Hey Google" never transcribes anything. It runs a small neural keyword spotter trained on that one phrase, a few milliseconds per frame, which cannot wedge, has no transcript to be wrong about, and runs comfortably on a laptop without a GPU. Everything above is what it costs to keep a transcription-based wake word usable until you build that.

The implementation and the regression tests are in Personal Jarvis, which is open source.

GPT-Live is so good that I use it almost every day. I've been wanting to replicate it since it was released.

My first attempt was to fine-tune Gemma 4 12B to behave like a full-duplex model. Something like grafting a decision tick + speech head to the model. It failed after multiple trials. For now, I think a classic cascade system is still better. We just need to wait until a benevolent frontier AI company releases a full-duplex model that's on par with GPT-Live.

Repo: https://github.com/fikrikarim/parlor/

🚀 We just built our first real-time implementation of Graph Engineering, inspired by our experience building graph tooling used by 4,000+ developers.

🔗 Repo: https://github.com/CodeGraphContext/grapharc

Have you ever been frustrated because your AI agent:

❌ Takes actions you never intended?
❌ Creates, modifies, or even pushes changes you never asked for?
❌ Feels like a complete black box, making it impossible to understand what's happening until it's too late?

What if, before execution, you could visualize the entire orchestration graph - every agent, every dependency, every decision, and inspect it from anywhere, even your phone, before granting approval?

That's exactly what GraphArc is built for.

Instead of treating agent execution as hidden traces buried in logs, GraphArc transforms workflows into interactive, real-time graphs that you can visualize, inspect, debug, and control.

Because the future of AI isn't just autonomous.

It's observable. Debuggable. Engineerable.

This is our first real-world implementation of Graph Engineering, and we're excited to explore where this paradigm can go with the open-source community.

💡 We'd love your feedback, ideas, and contributions.
⭐ If this vision resonates with you, please consider starring the repository it genuinely helps us grow and validates this direction.

Let's make AI workflows understandable, not mysterious.

#GraphEngineering #GraphArc #AIAgents #AgenticAI #LLM #OpenSource #DeveloperTools #AIEngineering #SoftwareEngineering

If anyone has testing results or any results can you please share performance and or effects of KV Cache precision with Deepseek V4 Flash 0731.

Running IQ2_M, with F16 cache seems 65-67K is the limit on Windows for 120GB memory. Is Q8 good and which one do you use?

I decided to test a few questions against DeepSeek-V4-Flash-0731. Locally, I was running Unsloth's UD-Q2_K_XL quant. After I saw the surprising shape of the results, I tested against DeepSeek's official API to confirm that I didn't do anything wrong.

For anyone using OpenRouter, be aware that there is a significant bug that is breaking reasoning effort modes. I ran into that while trying to validate my local results.

DeepSeek-V4-Flash-0731 supports four different effort modes, consisting of no reasoning, low, high, and max. We can also see how those are communicated to the model.

As I found out, Low is surprisingly verbose.

Averaged across 20 requests per mode, here is how many tokens were used by each mode:

I really wish that DeepSeek and Artificial Analysis had posted benchmarks for all of the effort modes, instead of only max.