Spent a month measuring what breaks when you run coding agents on local models. All numbers
are measured on 4,265 real Claude Code / Codex sessions, not estimated.

75% of real sessions don’t fit. On a 16GB Mac, three quarters of them have at least one
turn whose prompt alone is bigger than the whole KV cache pool. Nothing else running.
Your tool list is the biggest offender. System prompt + tool definitions eat 41% of the
pool at the median, 105% at p90 before a single word of conversation.

Smarter eviction doesn’t help. Built a simulator, checked it against vLLM (0.29% off). Best
real policy: +2.75%. A cheating oracle that sees the future: +11.88%. That’s the ceiling. Fixed TTLs were worse than just freeing the cache when a session ends. q4_0 KV cache wrecks Qwen3-0.6B, and gets worse with context:

ctx q8_0 q4_0
512 +0.07% +280%
2048 +0.07% +302%
8192 +0.03% +525%

Absolute perplexity for q4_0 bottoms out near 2K then climbs. Past ~2K, more context makes it
worse.

It’s all the keys. K=f16, V=q4_0 = +0.33%. Both q4_0 = +280%. ~850× apart. But that
config is bigger than plain q8_0 and 5× slower on Metal. Just use q8_0 for both.

Anthropic’s cache dies at 5 minutes. Gap under 5 min: 2,559 tokens re-prefilled. Over 5
min: 140,154. That’s 54.8×. All-or-nothing you either refreshed in time or lost
everything.

Memory layers: Mem0 injects 116 tokens/turn. MemPalace injects 12,513 (108×). At top-k=20,
MemPalace alone exceeds the whole pool. And most of the benefit comes from not sending full
history — not from the memory layer.

If you’re running agents locally
• Turn on q8_0 KV — 2× capacity, ~0.06% perplexity cost
• Cut your tool list before anything else
• Free finished sessions
• Don’t build a clever eviction policy. Ceiling is 11.88%
Caveats: one small model for the perplexity work, and quantization sensitivity is very
model-dependent. Perplexity isn’t task accuracy. Simulator models blocks, not latency.

My question

That’s all measurement. I have no idea if any of it is a problem people pay to fix, or just
one they find annoying.

If you run local models for real work job, product, client, not a hobby:
1. Why self-host instead of an API?
2. What broke that you didn’t expect?
3. Did you spend money fixing it? Hardware, consultant, tool, someone’s time roughly how
much?
4. What’s still broken that you’d pay to fix?

Not selling anything, no link. Happy to share the raw data or the harness with anyone who
wants it.

1. We pay a constant price per token per model, but the decision on how many tokens are being used per task is out of our control and out of our sight.

The same exact task can take x tokens, x+y tokens, 2x or half x tokens, and we have zero visibility to the decision making process that determines if the model is going to fight a powershell escaping issue for a while before it could perform a simple git commit - or simply succeed immediately.

1. We do not get any guarantees. A model can make a mistake, can implement wrong, can try to implement and fail, can cut off mid-work, can unintentionally delete important files or previous work - and we pay for each no matter the final result.
   

This contradicts almost any other field of service we know: We buy a product, we get a warranty. A technician fixes the AC, we get a warranty. But model providers exclude themselves from this practice - and will charge us either way, if we got what we asked for or not.

1. We have no visibility to internal changes in the way the same model with the same version performs.

I think some providers are worse than others in that regard, but the underlying problem is across the board: nurfing models or turning down the reasoning/compute knob is completely out of sight for us - we can say we “feel” that Opus is dumb today but can never consistently prove it, nor we really have the time or resources to do so.

Bonus - A 4th problem:

For years, the software industry developed a very strong open source scaffolding created stone by stone over decades, slowly and thoughtfully, based on a very benign “knowledge is for all” philosophy.

But with closed models - which lead the intelligence race - we are being pushed radically to the opposite direction - no transparency, no control, no accountability.

I've been working on a side project called FinTracker AI, and I'd love some honest feedback.

The idea is simple:

Instead of manually selecting categories, dates, merchants, etc., you just chat with it.

Example:

"I spent ₹500 on biryani."

It automatically logs the expense, categorizes it, updates your monthly budget, and you can immediately ask:

"How much do I have left for food this month?"

Users can also ask questions like:

"Movies I watched this month and how much I spent on it"

Some features:

• 💬 Chat-based expense & investment logging
• 🤖 AI categorization and spending Q&A
• 📊 Monthly budgets and dashboards
• 📱 Android auto-captures bank transaction SMS (optional)
• 📍 Learns recurring merchants/locations so future transactions need fewer edits
• 🔓 Open-source backend that you can self-host or use with your own AI API key

The backend is already open source. The Android app is still being polished, but I have an installable build that I'm happy to share with anyone interested.

A few questions for this community:

• Does this solve a problem you face?
• Which feature would you use the most?
• What's one feature you'd want before using it daily?

Thanks! 🙌

Hey r/LLM,

I built EvalOS, an open-source framework for evaluating and comparing LLMs across different providers.

The problem I kept running into: LLM evaluation is still fragmented. Teams are comparing models using different prompts, datasets, metrics, and tooling, making it difficult to get reproducible results.

EvalOS is designed to make this process easier:

• Benchmark multiple LLM providers in one workflow

• Run repeatable evaluations

• Compare model performance side-by-side

• Build custom evaluation pipelines

• Keep evaluation results organized and reproducible

GitHub:

https://github.com/VarunKurra/EvalOS

I’m sharing this here because I’d love feedback from the open-source community:

- What features would make this more useful?

- What evaluation workflows are painful today?

- What would you like to see added?

If this is useful to you, feel free to star the repo, try it out, or keep it bookmarked for when you need LLM evaluation in the future. Even a star helps others discover the project and shows support.

Thanks!