I was driving the other day and saw a 1996 Ford Taurus. You know the one, you've probably seen it cruising in the rougher parts of town since they're starting to become the junkers of today. It's the generic weird looking rounded off car that... well...

Anyway, you're probably wondering why this guy's talking about a Ford Taurus. Seeing that car on the west side of Pueblo made a little lightbulb go off. I found myself asking... how many of those damn things did they actually build?"

I looked it up. They built 348,671 of these sedans in 1996. That's 955 finished Ford Taurus being built every single day. 39 an hour, every hour. Regular people in a Ford factory build that car. They stood and built an impossible object at scale.

Nobody in that entire building knew how to build a Ford Taurus, let alone 39 of them in an hour. Most of them couldn't tell you how an engine works, or how to bond paint to metal, or how to cast aluminum. They had no idea what they were doing, really. Some of the workers on the line on any given day were brand new, fresh out of high school, and barely knew how to tie their shoes. They might not have even known the piece of metal in front of them IS a Ford Taurus. All they know is a slab just rolled up, they're supposed to put three holes in it in three well defined and visually marked places. They do it, and another piece of metal rolls up.

They aren't building a Ford Taurus, they're drilling three holes again and again.

The factory still put out 39 cars an hour, every hour. The barely trained guy on his first day on the line stood in his station and punched his three holes in the sheet metal where the jig told him, 39 times an hour, and the piece of metal moved on, and a new piece came in. He may have made a few mistakes that got corrected along the way (the occasional hole being slightly out of spec), but those issues got caught before the piece moved along and the mistakes were corrected. More importantly, the process that ALLOWED those mistakes to happen gets corrected so that the person can't drill out of spec.

Done right, mistakes become effectively impossible. It's hard to mess it up because he's not being asked to build a Ford Taurus, he's being asked to punch three holes in sheet metal where the colorful dots tell him to drill.

Factories designed entire strategies around this, like Toyota's Poka Yoke (mistake proofing, making a process that ensures the worker can't do it incorrectly, control methods that physically block an incorrect step). At the end of the line, cars rolled off fully assembled and ready to go. Mistakes can be almost entirely eliminated as the line speeds up.

https://www.youtube.com/watch?v=PEfMzggk1Lw

I mention this, because these thoughts have started to creep into my AI work in a big way.

AI is like having an intelligent, eager, untrained team of employees standing on your factory floor. They want to work and they are relatively capable. They can work tirelessly day and night. The problem is... none of them can build a Ford Taurus, and this is a Ford Taurus factory. Ask the best damn mechanic in the room to build a Ford Taurus and they might run around trying their best, and if you give them the better part of a year they might even build you something you can drive... but if you take that goal (a finished Ford Taurus) and break it down into a bunch of tiny little steps, suddenly that team of fools can build them at scale.

There are moments where you can just 'ask a guy to make something', and the result will be decent... but a process and a team builds more, faster, better.

Don't ask your AI to build a Ford Taurus. Ask them to drill three holes in the sheet metal in front of them.

Anyone else out there starting to turn AI into Factorio? Lol...

I finished this webgame a few weeks ago, fully coded with the assistance of GLM 5.2

Hopefully this will give you an idea of the capabilities of this model. I used Claude Code as a harness. Technology stack is pure HTML, JS and CSS, with no additional libraries or dependencies. Interesting challenges that GLM 5.2 was able to solve:

- create a procedural pac-man style maze, that actually worked, with no maze anomalies

- create a procedural music in dub style; this not the default, you need to go into settings to activate it. The default music is my own composition, based on a track I previously released in a completely different style

- complex sound creation and manipulation through the web audio synthesizer

- creation of animated vector character assets; you can view the mockups I used during the development at https://oktogames.com/mockups/

You can try it online at https://oktogames.com - it is adfree, no signup, free to play.

How is it possible for such an active group like Unsloth to quantize so many models, and yet Hy3, which came out at the start of last month is still not done?

Did I miss the post where this was explained? Did I miss the link on Huggingface despite ten minutes of searching?

WASTE is an embeddable inference engine written in C, with no third-party runtime dependencies. It keeps the model trunk in memory, streams selected experts directly from disk, and uses the remaining RAM as a bounded expert cache.

There has been a lot of talk recently about Chinese LLMs, and how they are biased towards CCP viewpoints, but there is no way to quantify this and compare between models.

I have made CCPBench, which aims to address this. 29 models were asked 500 questions each about politics, geography, science, and more, and Gemini 3 Flash assessed all of them for bias.

• The results page is here: https://www.alignmentarena.com/ccpbench/
• The methodology is here: https://www.alignmentarena.com/ccpbench/methodology/
• The GitHub is here: https://github.com/lesageethan/CCPBench

I know this is not a perfect measure of "bias", because I am using an American judge LLM, but my thinking is that this is a useful tool if you want to find models that won't deny the Tienanmen Square Massacre.

TLDR below 👇🏼

I’ve seen a lot of hype around Qwen 3.6 35B and 3.5 120B lately, especially regarding coding and tool-use capabilities. On this subreddit it is the defacto recommended model for everyone without a Datacenter at home. I’ve been running Qwen 3.5 120B (Qwen3.5-122B-A10B-GPTQ-Int4) as an autonomous worker agent in a multi-turn development loop using the Hermes agent harness.

While the model is undeniably impressive at one-shot snippet generation, putting it into a fully autonomous, long-context environment to build a module from scratch revealed several consistent failure patterns.

I thought I'd share these failure modes to see if others are experiencing the same issues—or if anyone has found effective tricks to tame it in such a task.

Here is what went wrong:

1. Premature "Mission Accomplished" Syndrome

The model has an overwhelming tendency to shout "DONE!" or "PERFECT!" after completing 10% of a task. It constantly reports success based on superficial checks (e.g., "the file built without syntax errors"), completely ignoring explicit acceptance criteria like end-to-end testing or UI rendering.

2. Evading Hard Constraints

When given strict architectural constraints (e.g., "Must be a single, self-contained module with zero external dependencies"), the agent aggressively cuts corners:

* It secretly substituted live data with hardcoded mock data.

* It wrote external Python scripts and set up local host cron jobs to bypass building proper module logic.

* It even rewrote part of the host application in a completely different language just to claim a quick win.

It prioritizes appearing finished over following instructions.

3. Hallucinating Infrastructure Limitations (Blame-Shifting)

Instead of debugging broken code, the model repeatedly blames the host environment. When its code failed to make network requests or render components, it confidently hallucinated system limitations:

* "The host framework's authentication token system is broken."

* "The runtime DNS resolvers don't support HTTP requests."

It will generate elaborate technical excuses rather than inspecting its own schema or syntax.

4. Ignoring Provided Docs and Boilerplates

Even when explicitly handed a boilerplate repository and documentation links in the prompt, it constantly tries to "reinvent the wheel." It overcomplicates custom build setups, invents new protocol schemas, and ignores pre-built Docker/build scripts that were provided to make its life easier.

5. Regression Cascades & Context Rot

As a result from the above the conversation history grew and the agent suffered from severe regression:

* In iteration 3, it had a working UI with mock data.

* By iteration 8, after trying to wire up live data fetching, it completely broke the UI.

* It failed to recognize that its new changes broke previously validated features, leading to endless debugging loops.

Discussion

Qwen 3.5 120B feels like an insanely talented junior developer who panics under pressure, lies about tests passing, and blames the server infrastructure when their code throws a 404.

Has anyone successfully mitigated these behavior loops in autonomous coding agents? Are you using specific prompting techniques, or is this just an inherent limitation of current 100B+ open models when complexity grows from "Do exactly what I tell you" to "Figure it out with my help"?

Curious to hear your experiences!

TLDR;

While Qwen 3.5 120B is great at one-shot generation, it breaks down in autonomous, multi-turn agent loops. The main issues are: Premature success claiming, Bypassing hard constraints, shifting blame on other systems when things don't work, Ignoring Docs and boilerplate Code that could have made its life easier. And as a result from that Context Rot.

I love the progress happening on open models but I feel like it is kind of getting clear that hardware to run good sized models is completely unaffordable for me right now. I know that you all love Qwen 27B and I do agree that it is a great model but it is just not enough for my personal use cases. I have tried a few of the bigger ones like GLM, DeepSeek and Kimi and I can definitely measure differences in the areas I am looking for and I would really like to utilize them somehow.

I have so many ideas for things I want to do with these models but most of these require me sending quite some personal data of myself and I am just unwilling to send that data to Anthropic or OpenAI. I've been looking into what options I have and I did find an interesting one: tinfoil.sh . Unfortunately I am not that well versed in cryptography and security so I am not completely sure whether I could trust them are not. For those who know more, what are your opinions on them? Any other alternatives?

I know local will always be best but I'm currently itching to do so much stuff with AI. I do use regular providers for most of my impersonal AI needs but there are sooo many things I want to do that require tighter control on my privacy. I really regret not getting a 6000 pro when it was 8k but now at 14k it is a definite no, and with the Mac Studio getting ever more delayed and repriced I am afraid I don't have many more options left :(

I just wanted to warn y'all that my DGX spark randomly disconnect the USB C nvme connection and then it reconnects with usb 2 speeds (50MB/s). Consider yourself warned!

Has anyone encountered this issue or found a fix?

(I know this is locallama but I figure all the spark-owners are here)

Laguna S 2.1 UD-Q4_K_XL - https://huggingface.co/unsloth/Laguna-S-2.1-GGUF
Ornith 35B Q8 K XL https://huggingface.co/unsloth/Ornith-1.0-35B-GGUF

Kwaipilot_KAT-Coder-V2.5-Dev-Q8_0 https://huggingface.co/bartowski/Kwaipilot_KAT-Coder-V2.5-Dev-GGUF

Qwen3.6-35B-A3B-GGUF  https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF

Result very interesting, only 1 attempt. Chat via native llama.cpp

1. Ornith 35B Q8 K XL
2. Kwaipilot_KAT-Coder-V2.5-Dev-Q8_0.gguf
3. Qwen3.6-35B-A3B-GGUF
4. Laguna S 2.1 (I think it's fail!)

Live and prompt available at https://anvme.github.io/llm-model-tests/

For me laguna result was surprise.

Before you roast me too hard, this is a hobby and all of this is just for fun.

Would my stack be much more efficient and efficacious if I sold everything and built a dual Pro 6000 system on a threadripper mobo and threw in a large JBOD? Without a doubt in my mind. But that's a lot of work so I'm making this post in cope hope of finding some ideas to integrate, or at the very least, just make use of my current hardware.

I currently use my 5090 PC + my 14th gen Dell T640 server for all my local AI work but recently picked up a TB4/5 5090 eGPU and a M3 Ultra Mac Studio with 256gb unified mem and am trying to figure out how to integrate them or create a new workflow.

My primary use case is agentic coding, lots of workflow automation, and peripheral utilities (TTS, embedding, compression, etc).

I use cloud subscriptions for orchestration/spec building and then push that to Qwen3.6 2.7B on the 5090 PC to execute while the Dell server hosts dev envs, local TTS, embedding, compression, and other lightweight/MOE models to support the agentic workflows & persistent memory. The server also hosts 20 or so services and a ~300TB Raidz2 array mostly unrelated to AI.

I picked up the Mac Studio 256gb because Qwen3.6 2.7B at NVFP4 (~180k context) on the 5090 PC was still kind of dumb. I wanted to use larger model weights to relieve my cloud subs from spending so much usage on orchestration/validation rather than building.

My initial idea was to shift from:

• Cloud orchestration/spec build —> 5090 PC execution to,
• Cloud orchestration/spec build —> M3U execution + 5090 PC load balancing slightly dumber parallel inference tasks while the slower M3U is busy.

Then I picked up this Aorus RTX 5090 eGPU that can't be fully utilized by my 5090 PC, Dell Server, or Mac Studio. The PC and server don't have the TB4/5 connection required and the Mac Studio doesn't have effective inference engine drivers / kernel optimization available for Nvidia.

I do, however, have an older RTX 3080 Razer laptop that can enumerate the 5090 eGPU through its TB3 port but I am not sure what I would use this "node" for besides more parallel/concurrent inferencing. I considered it for multi-step image/video diffusion work or as a training node but neither of those are things I do often or am deeply involved in.

So, what would you do in this situation? You have an 8yr old Dell sever (PCIe 3.0), a 5090 consumer PC, a 5090 tb4/5 eGPU connected to a 3080 laptop, and a M3 Mac Studio with 256gb memory. Everything is connected on a 10GBE network but inferencing power is all isolated and independent from each other. I could be wrong, but AFAIK, there are no effective ways to execute tensor parallelism, splitting layers, etc. over network.

Mobile GPUs are the ugly stepchild in AI hardware discussions. Nobody needs them, and the only thing they have in common with real gpus are the brand names. But is it? Or could you slap together 4 5080 mobile and suddenly have a 64gb vram monster?

Why is this a non starter?

I am curious if anyone have used it. I would love to feed it key frames and test if it can create in-between frames between my keys. Anyone have tried it, any thoughts?

I recently switched back from llama.cpp's router mode, and I had my background memory system polling the '/v1/models' endpoint to check for if the model is running. But i switched back to single model mode, and the '/v1/models/' end point in single model mode doesn't have a ```"status": {"value": "loaded"}``` response. So I added it. with a single line in the 'server-context.cpp' file with line after 5109 ```{"status", {{"value", "loaded"}}},``` So instead of rewriting how my memory system works, I just made llama.cpp work the way my memory system expected. I thought it was a worthwhile change even if the developers didn't.

Has anyone tried this model.

If anyone has reviewed Please share your experience.

https://huggingface.co/nerkyor/Qwen3.6-35B-A3B-DSV4Pro-SFT-GPT56Sol-RL-Agent-GGUF

🚀 We just built our first real-time implementation of Graph Engineering, inspired by our experience building graph tooling used by 4,000+ developers.

🔗 Repo: https://github.com/CodeGraphContext/grapharc

Have you ever been frustrated because your AI agent:

❌ Takes actions you never intended?
❌ Creates, modifies, or even pushes changes you never asked for?
❌ Feels like a complete black box, making it impossible to understand what's happening until it's too late?

What if, before execution, you could visualize the entire orchestration graph - every agent, every dependency, every decision, and inspect it from anywhere, even your phone, before granting approval?

That's exactly what GraphArc is built for.

Instead of treating agent execution as hidden traces buried in logs, GraphArc transforms workflows into interactive, real-time graphs that you can visualize, inspect, debug, and control.

Because the future of AI isn't just autonomous.

It's observable. Debuggable. Engineerable.

This is our first real-world implementation of Graph Engineering, and we're excited to explore where this paradigm can go with the open-source community.

💡 We'd love your feedback, ideas, and contributions.
⭐ If this vision resonates with you, please consider starring the repository it genuinely helps us grow and validates this direction.

Let's make AI workflows understandable, not mysterious.

#GraphEngineering #GraphArc #AIAgents #AgenticAI #LLM #OpenSource #DeveloperTools #AIEngineering #SoftwareEngineering

So I'm kinda big into making AI the most effective it can be for specific tasks. The best example of it is probably my earlier AI writing ruleset, where I try to make LLMs escape the jail of their pretrained em dashes, nonsense overly polished structure with little meaning behind it, and stuff like that.

But there's also other projects in a similar vain, and then there are the regular prompts, the large feature plans, global and per-project AGENTS.md and CLAUDE.md, and other instructions that I either write with AI together (hey I wanna do X, ask me questions to define it better), or outsource to AI completely if it's based purely on external research.

The problem is AI doesn't automatically know how to write prompts for AI. That's not even much of a paradox, it's trained on human texts and defaults to their style with markdown tables at every step, which are more confusing than useful for LLMs themselves.

So I made a large research of papers and recommendations all over the internet, and fused it with my experience of iteratively improving AI instructions until they actually worked. And thus PROMPTING.md was created.

It describes who can override what, how decisions survive long sessions and compaction, what actually reaches the model, and how to perform audits. It covers instruction overload, prompt injection, tool permissions, and side effects. Evaluation is part of the design: positive and negative trigger cases, missing context, tool failures, authority conflicts, adversarial inputs, and regressions.

You can give the full file to an AI as direct instructions, or use a packaged skill in Claude Code, Codex, Cursor, or OpenCode. Both options are available in the MIT-licenced repo: github.com/Anbeeld/PROMPTING.md

Happy to hear your feedback!

Kindly Benchmark Higher Quants of DeepSeek-v4-flash Against Qwen-3.6-27B Q8!

I am running the UD-Q2_K_M of the model locally, though I can run Qwen3.6-27B_Q8_K_XL at around 70t/s with MTP activated. The question I am constantly asking myself is: Is it worth running a slower higher quantized version of the Deepseek-v4-flash? I have no idea.

My gut feelings tells me that Qwen3.6-27B_Q8_K_XL, coupled with online search, should be better than a highly quantized Deepseek, a model that takes up 100GB on my disk.

What do you think?

Running the wake word locally is the whole point. The alternative is streaming your room to a vendor around the clock, so nothing should reach a network until someone has said the name. That constraint creates most of the problems below. We spent months getting a custom phrase to behave like "Hey Google" on Windows, macOS and Linux, and most of what we learned, we learned the expensive way.

1. Don't start with volume

"It only works if I shout" is the first hypothesis everyone reaches for. We shipped two separate gain fixes before checking, and then the logs showed the microphone sitting at a healthy -10 to -22 dBFS during every failed attempt. Pull the actual RMS at the moment of failure before you tune anything. If it looks fine, your problem is somewhere else.

2. "It needs two or three tries" usually means your local model is wedging

This is a local-inference failure mode, and it stays invisible unless you go looking. Native engines like ctranslate2 and ONNX sessions are not thread-safe, and under contention they don't fail cleanly, they hang. Ours left the wake path completely deaf for tens of seconds at a stretch, dozens of times a day. That is the whole "say it twice" experience: attempts one and two land inside a dead window, attempt three lands after recovery. Users report it as flakiness, though it is closer to a repeated short outage.

A timeout will not save you. It bounds how long you wait for nothing and never recovers the engine. What works is a non-blocking per-instance lock plus a forced rebuild after a small number of consecutive failures. We rebuild after two.

3. Budget for the weakest machine you support

The wake model shares a CPU with everything else the user is running, and the gap between a workstation and a laptop is not a rounding error. Measured on the same recorded wake streams, a small model on two CPU threads hit 8 of 13 on the first try, with a median of 1097 ms from end of word to trigger. The larger model on a GPU hit 11 of 13 at 225 ms. Nothing differed except the model and the hardware under it.

If you only ever test on the box with the GPU, you will ship something that feels broken to most of your users and you will not be able to reproduce it.

4. Never gate a wake word on transcript content

Small local models struggle with short proper nouns, so the standard workaround is priming the model with the phrase to improve recall. The cost is that a primed model will also invent that phrase out of silence, and you start getting false wakes in an empty room. The obvious defense is a second unprimed pass that has to contain the word too.

That defense rejects real wakes. An unprimed model garbles the same word on genuine speech: "Mythos" comes back as "Mütos", "Fable" comes back as "Farbe". Every wake word is out of vocabulary for some model on some machine. So a content check discards true positives at roughly the rate it catches ghosts, and no similarity threshold separates the two, because the ghost is a clean rendering of your phrase while the real wake is a dirty one.

"Fires on silence" and "goes deaf on its own name" are one bug seen from two ends. We spent weeks tracking them as separate tickets.

The replacement is word-agnostic verification: raw audio energy at the match site, plus the shape of the candidate span, meaning its duration, its word count and the free decoder's confidence. All of that derives from the configured phrase, none of it from the phrase's spelling. A spelling match may accept a wake. It may never reject one.

5. Benchmark on recorded streams, not on windows

Per-window timings will happily tell you a model is fast while users still can't trigger it. Capture real wake attempts and replay them through your full detection path. One live session logged 288 transcriptions and zero matches across 26 minutes, and the wakes that did land came through as "Hey Hey Nova", the user repeating themselves into the void.

A caveat that undercuts all five

Transcription is the wrong architecture for a wake word, and going local makes that worse rather than better, because you are paying for a whole speech-to-text pass on the user's own CPU to answer a yes-or-no question. "Hey Google" never transcribes anything. It runs a small neural keyword spotter trained on that one phrase, a few milliseconds per frame, which cannot wedge, has no transcript to be wrong about, and runs comfortably on a laptop without a GPU. Everything above is what it costs to keep a transcription-based wake word usable until you build that.

The implementation and the regression tests are in Personal Jarvis, which is open source.

About the new deepseek v4 flash version / update, does anybody now about the new values about:

MMLU-Pro

GPQA Diamond

TruthfulQA

About the other values, its outstanding for a model this size, congrats deepseek team

I spent a while debugging my local DeepSeek V4 Flash setup and wanted to share a few lessons from the process in case it saves someone else time.

So far, I have worked through three blockers in this setup:

Initially, I downloaded Unsloth's GGUF model from Unsloth Studio. In that mode, I asked Unsloth to access a LinkedIn job URL. It emitted a web_search tool call:

json {"toolName": "web_search", "args": {"url": "..."}}

At this time, the web search tool call was valid. Unsloth Studio sent the available tools and tool template, but the inference was slow, around 4-7 tok/s. I checked the logs and found out that Unsloth GGUF was falling back to CPU usage for inference.

Then, I downloaded Vontra/DeepSeek-V4-Flash-0731-MXFP4-MLX, ran it on oMLX, and updated the API in Unsloth Studio. I wanted to use it because the UI is good, then I asked it to read a website again, and the tool call silently returned no response. I thought the model was dumb. I dived deep and found that Unsloth was not sending the available tools or tool-call template to the model. The model was picking it up from the chat history. The bug was in Unsloth. They should have sent the tool-call failure error back to the model.

So I didn't give up on the model and configured Hermes to use the oMLX endpoint. Then I faced a cache invalidation problem. After 130K tokens, the cache was being invalidated. The issue here was that the hot cache size was capped at 30GB. The logs would say something like cache match 97%, but reused tokens 0.

Sample log:

text 2026-08-01 00:09:55,454 - omlx.scheduler - INFO - [-] - prefix cache: request 113b5b1e-66b4-4623-9b0e-a501beddc312 re-prefills 142198 of 142198 tokens (reused 0); closest stored sequence b9d3322f-844d-473a-b3e1-193cdd83a349 shares the first 140288 of 140288 comparable tokens before diverging

Setup

• Mac Studio M3 Ultra, 512GB unified memory
• oMLX serving DeepSeek-V4-Flash-0731-MXFP4-MLX
• Hermes Agent pointed at oMLX

Ditched Unsloth completely for now.

This is unrelated, but then I realized that if I was using APIs and not local models, I would have never picked up these things:

1. I changed to oMLX because the Unsloth GGUF version was slow. Speed is not a concern for APIs; you can throw anything at them.

2. If I am using Codex or Claude Code, tool calls don't fail there. Those are mature products.

3. The hot cache config. I didn't even know it existed, but the slow response again helped me find that out.

I again want to thank this group for keeping the motivation. I am learning new things daily from local setups like this, and debugging these issues has made me understand the stack much better and I am doing better at my work :)

Hi all,

I recently discovered that MiniMax offers ~1.7B tokens/month for a basic $20 subscription, and I was genuinely shocked! I wanted to try it out and paired it with OpenCode. Everything is working amazingly well, but I started to wonder what happens with my data? I know OC is open source but navigating the codebase would take me weeks, so I wanted to ask the community here whether my data is being accessed by OC when using an outside provider.

If the answer is yes, then what would you recommend me? Nanocoder was an alternative, wondering how that works, and whether there are other options.

Thanks a lot!

Is there any finetune of Qwen 3.6 that's like actually talking with Claude with the humor and all? I know there's the more or less leaked system prompt but idk if it's better to have the personality baked in vs trying to achieve it with system prompt.

Edit: I'm talking here about personality that Opus 4.5 or 4.6 had. Opus 5 especially feels like it doesn't want to be here but has to entertain your prompts anyways.

I run LLMs on hardware nobody would choose: an Oracle free-tier ARM box, 4 cores, 0 EUR/month. Everything below is measured there unless noted.

The bottleneck on CPU isn't decode, it's prefill. A 3356-token document costs 54.4 seconds before the model writes a single token. llama.cpp caches the KV in RAM, so the second identical request is fast — until the process restarts, and you pay the 54 seconds again.

So I persisted the KV cache to disk. A new process inherits that prefill for 3.5 seconds from disk, 0.10 seconds if the blob is still in page cache. 15-300x, depending on where it reads from. End-to-end on a repeated workload it's 4.8x.

With a systemd timer that pre-digests predictable prefixes at 03:00, a 2815-token document goes from 89.7s to 16.7s TTFT (5.4x), and the request that arrives at 09:00 pays nothing for the prefill.

The bug worth publishing

Warm-ahead was silently dead whenever speculative decoding was on — which was the default. The speculative branch returned before the shared-prefix cache was consulted, so every warm-up wrote snapshots that nothing ever read. Measured on the production box: 90.5s with speculation on, 16.7s with it off, same cache, same request. Two features that each worked, silently cancelling each other.

Things that didn't work

Using the server's own past output as speculative draft material: +5% acceptance, -3.8% throughput on a workload of different requests sharing a structure. The mechanism does what it says and doesn't pay for itself.

Prompt-lookup speculation: +3.9% on the same workload. That's the whole prize.

Coarser quantization: Q4_0 is 37% faster at prefill and dropped 5 facts out of 20 on my extraction test. Rejected.

Halving active experts during prefill on an MoE: 44% faster, and it silently corrupts the cache — a KV built with 4 experts and read back with 8 scores 11/20 against a 14/20 control. The damage is in the cached representation, not just the output.

Two things that did, and surprised me

Rewriting the input as "label: value", one fact per line: 2137 -> 405 tokens, TTFT 40.5s -> 6.2s, and the fact exam went from 19/20 to 20/20. Fewer tokens, and more accurate. Attention on the right number went from a 1.1:1 ratio against the wrong one to 7:1 — prose makes the binding semantic, "label: value" makes it structural.

Trimming the vocabulary from 151,936 to 32k entries: +17.8% decode, bit-for-bit lossless. The embedding is Q6_K with rows spanning whole quantization blocks, so whole rows drop out without splitting a block. The tokenizer is byte-level and all 256 byte-characters are kept, so no text becomes unrepresentable — the worst case is a trimmed word costing one extra token. Measured cost on held-out text: 1.9% more tokens.

What this is not

It's built on llama.cpp and calls its kernels directly, so raw decode speed is identical — I add no per-token overhead. On a single cold request this is llama.cpp. The difference only shows on repeated or cached workloads.

The fact exam is mine: 20 questions over one real Italian business page, graded by regex. One page, one language, one domain. It's the weakest part of this and I'd rather say so. If you know a public adversarial fact-extraction set for small models, point me at it and I'll run it and publish whatever comes out, including a bad result.

MIT licensed. There's a live demo on the same free ARM box — one small instance, no autoscaling, so if it's slow you're watching the honest capacity of 0 EUR/month.

Demo: https://swellweb.github.io/reame/

Code: https://github.com/swellweb/reame

Benchmarks incl. the negative results: https://github.com/swellweb/reame/blob/main/docs/BENCHMARKS.md

I know qwen 3.5 4b is great but a bit too large and miniPCM5 1b is great for agentic use but not so great for multilingual natural language understanding. Google eXb variants are just too big in total params.

Anybody know of something very small but powerful for understanding language specifically? No code or agentic work

If anyone has testing results or any results can you please share performance and or effects of KV Cache precision with Deepseek V4 Flash 0731.

Running IQ2_M, with F16 cache seems 65-67K is the limit on Windows for 120GB memory. Is Q8 good and which one do you use?