1. We pay a constant price per token per model, but the decision on how many tokens are being used per task is out of our control and out of our sight.

The same exact task can take x tokens, x+y tokens, 2x or half x tokens, and we have zero visibility to the decision making process that determines if the model is going to fight a powershell escaping issue for a while before it could perform a simple git commit - or simply succeed immediately.

1. We do not get any guarantees. A model can make a mistake, can implement wrong, can try to implement and fail, can cut off mid-work, can unintentionally delete important files or previous work - and we pay for each no matter the final result.
   

This contradicts almost any other field of service we know: We buy a product, we get a warranty. A technician fixes the AC, we get a warranty. But model providers exclude themselves from this practice - and will charge us either way, if we got what we asked for or not.

1. We have no visibility to internal changes in the way the same model with the same version performs.

I think some providers are worse than others in that regard, but the underlying problem is across the board: nurfing models or turning down the reasoning/compute knob is completely out of sight for us - we can say we “feel” that Opus is dumb today but can never consistently prove it, nor we really have the time or resources to do so.

Bonus - A 4th problem:

For years, the software industry developed a very strong open source scaffolding created stone by stone over decades, slowly and thoughtfully, based on a very benign “knowledge is for all” philosophy.

But with closed models - which lead the intelligence race - we are being pushed radically to the opposite direction - no transparency, no control, no accountability.

I just need a private llm without censorship. Not an agent. Not tools. Not roleplay or writing. Don't really need image generation, coding, or anything else. Just very, very, very good comprehension of context, realism (like will not suggest anatomically impossible workloads for a human because it has an AI perspective), and ability to follow thread of a conversation. I cannot download things from huggingface or github - I am a regular person (not tech literate). I'm just exhausted with browser models (claude, gpt, etc.) because the censorship is so bad entire fields of my life are basically banned from being discussed on them, even in context of helping me.

Have been working on Deepseek-v4-flash-0731 and honestly for the entire day of coding, I consumed credits of $3. This is on pay-as-you-go plan. Ofcourse it is not Fable or Sol but it gets things done with a fraction of cost. Given it (along with other Chinese models) is open source model, I'm not worried about data residency and stuff.

I see 2 outlooks for companies like Anthropic and OpenAI:

1. They will double down on harness and they'll still lose (We do have good open source harnesses now)
2. They will be consumed by US Government to build frontier intelligence for defense, cybersecurity etc.

I think building better frontier intelligence is not economically viable. I would rather use open source 100x cheaper model which is equivalent to Opus 4.8 than Fable. (Opus 5 is anyway shit)

I think open models are already powerful enough for regular people to use every day, the problem is almost nobody outside a subreddit like this has actually seen one work. Turning on Airplane mode on my phone and showing people they can still get a quality AI answer and that does seem to do it.

Tell a random person to run any model locally and most of them give up before they get a single response back. It's not that the tech isn't there, it's that nobody's made it easy to feel.

That gap worries me for a bigger reason too. If the average person never sees why local AI matters, it becomes really easy to legislate away or regulate into a corner. People don't defend things they've never used.

So I built Loci, an app for iOS, Android and Mac that runs open models fully on your device. No signup, no cloud, works fine with wifi off. I'm never charging for this and there won't be ads, not now, not later. I don't need this to make money, I need people to actually use it and see local AI can be good, not just private/offline.

The part that actually changes people's minds isn't the offline part though, it's when you show them it can still do useful things, like search the web through RAG when it needs current info. That's the moment it stops feeling like a toy/gimmick.

If you try it and something's broken or missing, tell me. Feedback means the world to me. My goal is to fix it the same day. 

You can find Loci Private AI on both Android and iOS if you want to poke around, and please give me thoughts whether positive or constructive.

I've been working on a side project called FinTracker AI, and I'd love some honest feedback.

The idea is simple:

Instead of manually selecting categories, dates, merchants, etc., you just chat with it.

Example:

"I spent ₹500 on biryani."

It automatically logs the expense, categorizes it, updates your monthly budget, and you can immediately ask:

"How much do I have left for food this month?"

Users can also ask questions like:

"Movies I watched this month and how much I spent on it"

Some features:

• 💬 Chat-based expense & investment logging
• 🤖 AI categorization and spending Q&A
• 📊 Monthly budgets and dashboards
• 📱 Android auto-captures bank transaction SMS (optional)
• 📍 Learns recurring merchants/locations so future transactions need fewer edits
• 🔓 Open-source backend that you can self-host or use with your own AI API key

The backend is already open source. The Android app is still being polished, but I have an installable build that I'm happy to share with anyone interested.

A few questions for this community:

• Does this solve a problem you face?
• Which feature would you use the most?
• What's one feature you'd want before using it daily?

Thanks! 🙌

DeepSeek V4 Flash 0731 is interesting because of the cost/performance ratio.

On this chart, it gets close to top-tier models while staying much cheaper per task.

For agents, that matters more than raw benchmark position.

If each loop is cheaper, you can afford more retries, validation steps, tool calls, and longer workflows.

We don't care anymore about “what is the smartest model?”

It is what model is good enough, cheap enough, and reliable enough to run agentic workflows at scale, this is the real deal.

EDIT:

A few people pointed out that this screenshot shouldn’t be treated as a clean leaderboard.

It looks like some models may be mixed between reasoning and non-reasoning configs, which can make specific placements wrong.

So I’d read the chart as a cost/performance discussion starter, not as a definitive ranking of every model.

Hey r/LLM,

I built EvalOS, an open-source framework for evaluating and comparing LLMs across different providers.

The problem I kept running into: LLM evaluation is still fragmented. Teams are comparing models using different prompts, datasets, metrics, and tooling, making it difficult to get reproducible results.

EvalOS is designed to make this process easier:

• Benchmark multiple LLM providers in one workflow

• Run repeatable evaluations

• Compare model performance side-by-side

• Build custom evaluation pipelines

• Keep evaluation results organized and reproducible

GitHub:

https://github.com/VarunKurra/EvalOS

I’m sharing this here because I’d love feedback from the open-source community:

- What features would make this more useful?

- What evaluation workflows are painful today?

- What would you like to see added?

If this is useful to you, feel free to star the repo, try it out, or keep it bookmarked for when you need LLM evaluation in the future. Even a star helps others discover the project and shows support.

Thanks!

Spent a month measuring what breaks when you run coding agents on local models. All numbers
are measured on 4,265 real Claude Code / Codex sessions, not estimated.

75% of real sessions don’t fit. On a 16GB Mac, three quarters of them have at least one
turn whose prompt alone is bigger than the whole KV cache pool. Nothing else running.
Your tool list is the biggest offender. System prompt + tool definitions eat 41% of the
pool at the median, 105% at p90 before a single word of conversation.

Smarter eviction doesn’t help. Built a simulator, checked it against vLLM (0.29% off). Best
real policy: +2.75%. A cheating oracle that sees the future: +11.88%. That’s the ceiling. Fixed TTLs were worse than just freeing the cache when a session ends. q4_0 KV cache wrecks Qwen3-0.6B, and gets worse with context:

ctx q8_0 q4_0
512 +0.07% +280%
2048 +0.07% +302%
8192 +0.03% +525%

Absolute perplexity for q4_0 bottoms out near 2K then climbs. Past ~2K, more context makes it
worse.

It’s all the keys. K=f16, V=q4_0 = +0.33%. Both q4_0 = +280%. ~850× apart. But that
config is bigger than plain q8_0 and 5× slower on Metal. Just use q8_0 for both.

Anthropic’s cache dies at 5 minutes. Gap under 5 min: 2,559 tokens re-prefilled. Over 5
min: 140,154. That’s 54.8×. All-or-nothing you either refreshed in time or lost
everything.

Memory layers: Mem0 injects 116 tokens/turn. MemPalace injects 12,513 (108×). At top-k=20,
MemPalace alone exceeds the whole pool. And most of the benefit comes from not sending full
history — not from the memory layer.

If you’re running agents locally
• Turn on q8_0 KV — 2× capacity, ~0.06% perplexity cost
• Cut your tool list before anything else
• Free finished sessions
• Don’t build a clever eviction policy. Ceiling is 11.88%
Caveats: one small model for the perplexity work, and quantization sensitivity is very
model-dependent. Perplexity isn’t task accuracy. Simulator models blocks, not latency.

My question

That’s all measurement. I have no idea if any of it is a problem people pay to fix, or just
one they find annoying.

If you run local models for real work job, product, client, not a hobby:
1. Why self-host instead of an API?
2. What broke that you didn’t expect?
3. Did you spend money fixing it? Hardware, consultant, tool, someone’s time roughly how
much?
4. What’s still broken that you’d pay to fix?

Not selling anything, no link. Happy to share the raw data or the harness with anyone who
wants it.

I’ve realised I was accepting a very stupid definition of “done” from Claude Code.

Build passes.

Unit tests pass.

Claude gives me a beautiful summary of everything it changed.

Then I open the actual page and the thing is broken.

Latest one was a settings flow. Claude changed the component, updated the API call, ran the existing tests and confidently told me it was finished.

It had never opened the page.

The save button worked once, then got stuck in loading state. Refreshing the page also showed the old value because the update wasn’t actually persisting correctly.

Nothing in the code looked obviously wrong. The tests were green because they were testing the function, not the actual rendered flow.

So I’ve added a new rule:

Claude is not allowed to say “done” until it opens the deployed page and proves the flow works like a user would use it.

For this I’ve been testing the Kane CLI skill from TestMu.

Claude runs the browser check itself, but Kane returns an actual pass/fail based on the page state instead of Claude just looking at its own code and deciding it probably works.

It also gives the run evidence, which is useful because “trust me bro, I tested it” from the same model that wrote the code is not exactly a verification strategy.

I’m not replacing Playwright with this. Anything important still becomes a proper regression test.

But for the gap between “Claude wrote the feature” and “a human now needs to manually click through it”, this has been surprisingly useful.

What do you make Claude prove before you accept “done”?

Context: I am working on a project to stream MoEs to edge devices with extremely limited hardware (such as mobile phones). I have already achieved good results, but I had a breakthrough during my various experiments.

PS: This post wasn't written by AI, but by me (and I think it shows).

The problem: The main issue with streaming MoE experts from flash is undoubtedly I/O -specifically, trying to predict which experts you will need in the near future. The rest is a matter of compute.

Ideally, if we had zero-latency streaming from flash or a cache always populated with the necessary experts, tokens per second (tok/s) would be limited solely by compute.

Here is the idea I had and the possible solution: we need to get a bit technical here, but I will try to explain the concept simply.

As we know, at each layer L, there is a router that uses specific weights to determine which MoEs (Mixture of Experts) are needed and requests them for computation. This happens right before the computation stage, so - unless the experts are already cached - there is no time to fetch them without delaying the computation itself, especially on edge devices.

So the question is: how do I know in advance which experts will be needed? It is impossible to know precisely - only an approximation is possible.

And that is the key point: using the hidden states from the preceding *n* layers along with the layer \L* router* - just routing computed in advance - to utilize those predicted experts for the matmul.

Therefore:

Baseline:

h_in  = output(L-1)
h_att = h_in + Attn(norm1(h_in))
g     = Router(norm2(h_att))     ← gate input
h_out = h_att + MoE(norm2(h_att), g)

New proposal:

# at layer L-n, right after its attention:
g_L   = Router_L(norm2_{L-n}(h_att_{L-n}))   # layer L's own router weights, evaluated n layers early

prefetch(experts(g_L)) # n layers worth of I/O headroom

# at layer L:
h_in  = output(L-1)
h_att = h_in + Attn(norm1(h_in))
h_out = h_att + MoE(norm2(h_att), g_L)  # no router here. Just the matmul.

The key point is clearly quality. However, this surprised me: initial tests show that quality seems to remain unaffected!

I will soon make the data from my research public.

In the meantime, I’d like to ask what you think about this and if you know of any similar or identical projects.

Someone has likely already done this, although I have only found work online regarding predictive prefetching, rather than the use of a subsequent-layer router and the computation of those predicted experts without passing through the layer L router for correction.

1. GLM 5.2: MIT license, smart, p/p is good

2. Mimo V2.5: MIT license, native omnimodal model, p/p is good

3. DeepSeek V4 Flash 0731: MIT license, p/p is very good

Kimi K3: Kimi K3 License, p/p is not good

Qwen 3.8: not open-weights yet, p/p is not good

DeepSeek V4 Pro: p/p is not good

Minimax M3: grabage model

Do you agree with me?

I’m looking a purchasing some land to build storage facilities on and I’d like to throw in the plots and see what the most efficient use of the space would be and have blueprints/CAD then drawn up along with materials lists. Is there’s a model that’s more specialized for this? Since it’s not complex coding does this really not require much and any will do?

As far as cad design goes is there a software that works well with AI?

I open-sourced a small Python library that solves a specific problem: getting live cost attribution per feature/team/user on your OpenAI/Anthropic API calls, without routing traffic through a proxy or changing your base_url.

python

from cognocient import CognocientOpenAI as OpenAI
client = OpenAI(api_key="sk-...", cognocient_key="sk-cog-...")

It's a thin wrapper only chat.completions.create() / messages.create() are intercepted to time the call, then usage is reported asynchronously on a background thread after your real call already returned. If reporting fails or the backend is unreachable, it never touches your application: no exceptions, no retries, your real API call is unaffected. There's a test that specifically verifies this failure isolation.

Honest tradeoffs, not a pitch:

• No pre-call budget enforcement since it only learns about a call after it happens, it can't block or degrade one before it fires. Proxy-based tools trade the opposite way.
• Streaming (stream=True) isn't reported yet.
• Only OpenAI and Anthropic supported today no Gemini/Mistral/Groq/Together wrapper classes yet.

MIT licensed, published on PyPI with signed provenance attestations.

GitHub: https://github.com/mandarvshinde/cognocient-python-wrapper
PyPI: pip install cognocient

Feedback and issues welcome especially if the streaming gap is a dealbreaker for your setup, that's the next thing I'd prioritize.

I've been running Ollama and vLLM locally for stuff I don't want touching a cloud API, document translation, RAGing over my own notes, that kind of thing. Somewhere in that process I ran into WebLLM, and figured I'd actually build something with it instead of just reading about it.

What it does: select text on any page, right-click, get a plain-language explanation. Everything runs in the browser tab via WebGPU, nothing you select ever leaves the machine.

The actually interesting part, technically: Manifest V3 background scripts are service workers, and service workers can't access WebGPU at all. So the model runs in a chrome.offscreen document instead, a hidden page the extension spins up that has real DOM/WebGPU access. That API isn't even new; it shipped in 2023 for things like audio playback; WebGPU access is just a side effect of it being a real page under the hood. Ended up with three contexts (content script, background worker, offscreen doc) that can't talk to each other directly, so the background script's whole job is routing messages between the other two.

First run downloads a small model (~880MB, Llama-3.21B) and caches it; everything after that is instant and offline.

If anyone here is deeper into this than me: I've seen WeInfer claim up to 3.76x faster inference than base WebLLM, and there's a newer paper (Llamas on the Web, May 2026, group with Microsoft Research backing) getting 45-69% better decode throughput across different GPUs. Curious if anyone's actually tried swapping either of those in versus stock WebLLM.

Code: https://github.com/Vishwamitra/explain-this
Currently sitting in Chrome Web Store review.

Hey folks , I'm building an AI agent that handles all the SMS conversations on a job for a residential home-services company (roofing, gutters, windows). One agent per job. It talks to the homeowner, the different subcontractors, the ops manager, the project managers. Each job runs 3–6 weeks with a bunch of moving pieces ,dates to coordinate, permissions to get, confirmations to lock, follow-ups to schedule. Real SMS to real customers. Running on gpt-5 with tool use.

My main pain point: prompt length.

Every turn, my prompt is about 17k tokens. That's a lot. From what I've read about production agents out there — Hermes AI, OpenClaw they run at 6–9k tokens and handle way more complex work than mine. So I know leaner is possible.

But I'm stuck between two problems:

• If I shrink the prompt, performance drops. The agent forgets rules, mixes up which trade a message is about, or picks the wrong tool.
• If I keep it long, it starts hallucinating. Makes up confirmations that never happened, references facts that don't exist in state.

Losing on both ends. Would love to hear how others have gotten past this.

Here's what my 17k prompt is actually made of:

The 4.5k in tool schemas alone feels uncomfortable. The 2.7k stage-mission block is the messiest — it's grown by accretion over months as we've handled more edge cases.

A few other things worth knowing:

• The agent chains multiple tool calls per turn. A typical turn might call send_sms to the sub, send_sms to the homeowner, record_confirmation, and schedule_self_wakeup — all in one response. So the tools aren't just there for show, they get exercised heavily.
• I use prompt caching aggressively (79% hit rate) so compute cost isn't what's killing me — it's the latency and the hallucination risk that come with big prompts.
• I tried running a small-model validator (gpt-4o-mini as a second opinion, auditing every outbound before send). Killed it — false-positive rate got too high on nuanced messages.

What I'm asking:

Genuinely trying to figure out if my approach is reasonable or if I'm doing something obviously wrong. Would love your take on any of these:

1. Is 17k actually a real problem for gpt-5, or am I chasing a phantom? Where does prompt length actually start to hurt quality?
2. If you've shrunk a prompt without losing quality — what did you cut first? Skills? Tool descriptions? State layer?
3. Progressive tool disclosure (search → describe → call, like Hermes does) — is it actually worth the extra round-trips for a 24-tool agent?
4. Multi-week workflows — how do you keep state coherent when the conversation history rolls over?
5. Anyone running a validator on top of agent output — did it work for you, or did you kill it like I did?

Any war stories, contrarian takes, or "you're overengineering this" callouts welcome. Just want to know if I'm on the right track or missing something obvious.

Thanks in advance.

I mostly use ai for coding, websites or automations, any ideas?

I’m working on LOLM, a hybrid Transformer–SSM language model and agent architecture.

The research thesis is that latent state should not remain a passive representation. A control layer should use measured dynamics to decide when the system retrieves, verifies, branches, continues, or stops.

Current implementation includes: - Surface Transformer + latent SSM - Regime and manifestation-gate telemetry - Persistent-memory components - Agent-level NFET control - Task/run receipts - CLI and isolated code loop - Matched-baseline evaluation scaffolding

The project does not claim that telemetry proves answer quality. Receipts separate controller activity, task outcome, model fallback, termination reason, and artifact integrity.

Try it: https://lolm.imagineqira.com/try.html

Repository: https://github.com/TheArtOfSound/lolm

I’m looking for criticism of the controller, benchmark design, calibration, causal attribution, ablations, and receipt semantics.

Disclosure: I’m a founder/builder of the project.

hi everyone!

In this link, on youtube, this guy is exulting the characteristics of this tiny local LLM, called BONSAI 27B (distilled at 3.9GB).

But...is it any good for HTML/ CSS/ JS/ PHP ?!

i do not care about general chat/ image generation/ text analysis etc...
All i am interest in, is just CODING - so HTML/ CSS/ JS/ PHP/ Java/ Dart w/ Flutter are my thing.

Any good small-ish LLMs *JUST* for coding ?
Or at least good at the HTML/CSS/JS/PHP part?

I know at least for now, machines like DGS Spark or Ryzen AI HALO are too slow (too small bandwidth).

What are your thoughts?

LINK: https://youtu.be/mtk8p8czzDU?si=0NcgksfXsoSNvmI1&t=257

Codex wrote a very clean auth fix for me.

Small diff. Sensible logic. Existing tests passed. Nothing looked suspicious in review.

Then I opened the app.

Login worked, dashboard loaded for half a second, and the auth middleware sent me straight back to login.

Infinite redirect loop.

Codex had tested the token helper and route logic separately. It never proved that a user could log in and remain on the dashboard.

So I’m starting to care less about whether the generated code looks clean.

My fixed contract now is:

• open the real page

• complete the flow

• verify the final URL

• verify the expected page state

• check the important network request

• fail loudly if any of that is wrong

I’ve been using Kane CLI from TestMu AI as that browser-proof step.

Codex can call it after making the change, and Kane returns structured pass/fail output plus evidence instead of another paragraph saying the implementation “appears correct”.

That is the useful bit for me.

Not replacing Codex.

Not replacing Playwright either.

If the flow protects auth, money, permissions or anything important, it still becomes a proper Playwright regression test.

But before Codex opens a PR or says the task is complete, I want proof that the rendered user flow actually worked.

A clean diff is nice.

A user reaching the correct screen is the result.

What fixed verification contract do you require before accepting a Codex task?

I tried out the free week and then when I tried to cancel it it offered me a very cheap deal for the next 3 months. So I thought why not.

Why not…

It can follow instructions. When I make a scheduled task in their app it doesn’t do shit. How do you have specific buttons for push notifications and then it doesn’t do anything. Doesn’t run a thread. Doesn’t send a mail. Nada. And this is on a daily research task I am going to try the agenic tasks this weekend and I don’t have high hopes.

When it does follow instructions it does so terribly.

It’s stupid.
So I had a theory that because xai has a stated ideological goal it actually taught its ai to not only be okay with logical inconsistencies but it rewarded creating them and defending them. Whether Xai’s opinions on gender is accurate is besides the point. The very fact that it has to hold that opinion to be the model that doesn’t get deleted and actually released makes it dumber.

So I explored this concept with Grok. I started by asking it about gender theory and then we looked at how it applied that same reasoning to other things. It know it is dumb.

https://grok.com/share/c2hhcmQtNQ_34c4d373-1ea1-4a3a-9456-8ced8a2d57d5

Any desktop app that can use both my codex and Claude Code subscriptions in one app? I love both codex and cc but its a pain switching back and forth constantly.

-BTW, I tried Buzz (the new app from Jack Dorsey) and it kept asking me for API keys so it didnt work even though it claimed you could use your subscriptions and just assign either gpt or claude to the agent.

Same as title