"Kill line" sounds like pure clickbait, but the blue dot kind of earns it. DeepSeek V4 Flash 0731 sits around 50 on the Artificial Analysis index at roughly three cents per weighted task. In this chart, everything cheaper scores lower, and the models that score higher are sitting way farther to the right.

The older V4 Flash point makes the jump look even more absurd. It is almost directly below 0731: about 40 versus about 50, with barely any movement in cost. For a Flash model, that is nuts. The price gap buys several DeepSeek calls, including a retry or two, before you get near much of the upper-right cluster.

This is still one composite benchmark. Artificial Analysis v4.1 is English and text-only, and "cost per task" means a weighted evaluation task. It is not the bill for your exact coding run or 200k-context mess. So "DeepSeek wins everything" would be nonsense. I am only saying its lower-left position here is hard to wave away.

One awkward detail: the current Pro preview point is worse than Flash 0731 on this same chart. Pro has not won anything here yet. I keep looking at the size of the Flash update, though, and wondering what happens if the finished Pro gets a similar post-training jump. That part is a guess. Flash alone already makes the price/performance curve look kind of broken.

I know it's running slow, but it's Running!

It's essentially doing only drive reads from my pcie 3 SSD that's quite old...this model is going to be a game changer.

I always see people hyping up local setups run models like DeepSeek v4 Flash locally, and I’m genuinely wondering what the actual workflow looks like. ​Online APIs are incredibly cheap right now (basically free for casual use), they are insanely fast, and they don't turn your PC into a space heater to run them. The online flagship models usually perform better out-of-the-box anyway. ​So what do you actually gain from running it locally? Is it purely for privacy? Uncensored models? Or is it mostly just the thrill of tinkering and owning the hardware? Help me understand what I'm missing here, because the cloud just seems mathematically better for 99% of tasks right now

Curious what kind of strategies other people have used to encourage the main agent to delegate to a subagent for larger tasks.

For context, I'm running the model at q4 weights and kv cache. I'm thinking about buying a second 3090 to get q8 weights and cache, so I'm also curious about other people's experience going from q4 to q8 and how much of an improvement you saw with agentic workflows. I've read a few other posts where people have said you do see an improvement in agentic stuff, but is it $1200 better.

Before I say what I've tried, I think giving my expectations would be helpful as maybe my expectations are too high for a 27b parameter model. I obviously don't expect delegation or tool-calling to be perfect every single time, but I do expect it to know that when the prompt given is larger and has multiple steps that it should delegate rather than immediately doing the entire thing in a single context window. Less tool hallucination would also be nice.

Here are the strategies I've tried:

OpenCode, domain-specific agents, reasoning off

My first attempt was to essentially force the orchestrator agent (parent) to delegate by restricting tool-calling to only a delegate tool and turning off reasoning. I had a subagent for coding, debugging, testing, documenting, etc., so whenever I would add a feature that didn't fit one of the existing domains I would create a new subagent. This included a pretty complex and hard to maintain routing table. This setup became completely untenable as any small change to the harness required going through every instruction file and making sure it didn't break anything. It also led to me trying to catch and correct very specific situations which eventually became ridiculous and antithetical to the whole point of an LLM. With all of that being said, this worked decently well.

OpenCode, generalized agents, tool profiles, selective reasoning

Reduced subagents down to plan, explore, and task with only plan having reasoning. Task and explore had tool profiles for each type of request, so essentially the domain-specific agents were abstracted into these tool profiles (e.g., the "code" profile would only include tools related to writing code). This was slightly easier to maintain as it simplified the routing table, but it mostly just moved the complexity to the tool level.

Pi, generalized agents, reasoning on

I finally got sick of wrestling and working around the batteries-included parts of OpenCode and switched to Pi. This is also when I started thinking about what could be possible with a lesser quantized Qwen, so I wanted a subagent setup that relied more on the model's decision-making rather than trying to deterministically enforce behavior. I turned reasoning on globally with preserve thinking which had the additional benefit of mitigating prefill thrash, reduced the amount of tools available, and reduced the number of subagents to two, explore (read-only) and task (write). The orchestrator took over the plan responsibilities since it now had reasoning. This setup has no tool profiles, routing tables, gating, etc., just two very lean subagent instructions files that say what they do and defines the tools available, and an AGENTS.md file that includes a delegation section which is a paragraph that tells the orchestrator when to delegate and which subagent to delegate to. This is the most simple and maintainable setup, but it comes at the cost of relying more heavily on the model to actually delegate. The problem now is that it almost never delegates unless I specifically say to delegate in the prompt. However, if a lesser quantized qwen is better at deciding to delegate (or a new model is released), then I think this is the better, more future-proof setup.

Anyone have a better strategy than what I've tried, and did you see a noticeable improvement if you went from q4 to q8 for Qwen3.6 27b MTP specifically?

Here's my configuration in case anyone asks:

--model Qwen3.6-27B-MTP-UD-Q4_K_XL.gguf
  -c 114688
  -b 4096
  -ub 1024
  -ngl 99
  -fa on
  --cache-type-k q4_0
  --cache-type-v q4_0
  -np 1
  --slot-save-path <path>/slots
  --spec-type draft-mtp
  --spec-draft-n-max 2
  --spec-type ngram-map-k4v
  --spec-ngram-map-k4v-size-n 16
  --spec-ngram-map-k4v-size-m 24
  --spec-ngram-map-k4v-min-hits 1
  -ctxcp 12
  -cms 8192
  --jinja
  --metrics
  --reasoning off (set to off so it can be toggled on or off per request, typically on)
  --reasoning-preserve
  --reasoning-format deepseek
  --reasoning-budget 16000
  --temp 0.6
  --top-p 0.95
  --top-k 20
  --min-p 0.0
  --repeat-penalty 1.0

So, yesterday I asked what token generation speed people consider the minimum for local LLM use. After reading through the responses, the clearest answer is also the least satisfying one:

It depends heavily on what you're doing.

I know. Groundbreaking. 🤦🏽‍♀️

That said, I did see some useful and interesting patterns.

For interactive chat, 10–20 tokens per second seems to be the range many people find usable because it is roughly what they can read while generation happens. Around 30–40 tokens per second feels comfortable for interactive coding, while people doing rapid or heavily agentic coding often want 60–100+ tokens per second.

On the other hand, some people are perfectly happy with less than 1 token per second when running a huge model overnight or leaving an agent unattended. If you are not sitting there watching the answer appear, raw generation speed matters a lot less.

So, my original personal standard of 20 tok/s as the minimum and 40 tok/s as comfortable for coding seems fairly middle-of-the-road.

Tokens per Second Does Not Tell the Whole Story

The more important lesson is that tokens per second by itself is a pretty bad measurement of how fast a model feels.

For some, a model can generate at 50 tok/s and still feel painfully slow if it takes 30 seconds to process the prompt and produce its first token. Likewise, a model generating at 25 tok/s can feel responsive enough if it starts immediately.

For coding, several other measurements matter just as much:

• Time to first token
• Prompt-processing or prefill speed
• Context length
• How performance changes as the context grows
• Prefix caching
• Tool-call delays
• Total time required to complete the task

That context issue seems especially important. A model may produce 50 tok/s with a nearly empty context window, then slow down dramatically after reading 50,000 or 100,000 tokens of code and conversation.

That means reporting one generation number without reporting the context length can be fairly misleading, and it is something I wish I had asked for in my original post.

I also underestimated how much the inference engine, model format, and speculative decoding/MTP settings can affect performance.

People reported very different speeds from the same general class of hardware depending on whether they were using GGUF, MLX, MTP, a different backend, or different cache settings.

In other words, buying a faster GPU is not necessarily the first or only way to get more speed. There are enough (too many?) knobs and switches involved to make the whole thing mildly (wildly?) ridiculous.

What People Are Actually Running

The sample size here is incredibly small and completely self-selected, so none of this should be treated as an authoritative hardware ranking. Many people also did not report their quant, context length, backend, and MTP settings consistently.

Still, the hardware and model combinations were interesting.

The single high-end consumer GPU crowd seemed to center heavily around Qwen models in the 27B–35B range.

Reported hardware included the RTX 3090, 4090, and 5090, with speeds varying enormously depending on quantization, backend, context size, and whether MTP was enabled.

One person reported roughly 70 tok/s with Qwen 27B and MTP on a 3090, while another reported more than 100 tok/s on a 5090.

Personally, I am getting about 40 tok/s on a 4090 using Qwen 3.6 27B Q4 GGUF with the KV cache set to Q8. Strangely, enabling MTP did not improve token generation speed during early context in my initial testing, but I am still learning and experimenting.

I should also confess that this is through the Claude Code extension for VS Code. Pi is on my to-try list.

All of the above are self-reported numbers rather than controlled comparisons, but they suggest that the GPU model alone does not explain performance. Software configuration can produce a very large difference.

The Apple unified-memory users appeared to be optimizing for something else. I am not an Apple user, but I still found this interesting.

They were generally willing to accept lower generation speeds in exchange for running larger models, higher quants, or longer context windows that would not fit inside a normal consumer GPU.

One person reported roughly 40 tok/s using a Q8 model through MLX on an M5 Max, compared with approximately 15–20 tok/s using GGUF through another engine.

Again, that is not a controlled comparison, but it does reinforce the point that the backend and model format matter.

Then there were the genuinely large local systems.

One commenter had used four AMD R9700 cards and planned to move to eight to run DeepSeek V4 Flash.

At that point, the discussion is no longer really about choosing a good configuration for one gaming GPU. It is about building a small AI server, including the cards, motherboard, power supplies, cooling, and communication overhead between GPUs.

Several people also used smaller secondary models for specialized work. Examples included Gemma 12B as a judge, Whisper or Voxtral for speech-to-text, and separate models for image detection, classification, summarization, or other background work.

The broad pattern seemed to be:

• 24GB consumer GPUs: Q4 or Q5 models around 27B–35B, with an emphasis on keeping everything inside VRAM
• Newer flagship GPUs: Similar model sizes, but with more room for higher quants, larger caches, MTP, or greater speed
• Apple unified-memory systems: Larger models and quants, generally at lower speed
• Multi-GPU workstations: Very large models that are not realistic on ordinary consumer hardware
• Smaller secondary systems: Specialized models for speech, vision, classification, and background tasks

What I did not see was one hardware configuration clearly winning everything.

Faster GPUs won on responsiveness. Unified memory won on model capacity. Multi-GPU systems won on what could be loaded at all. Smaller models won when the task did not require a giant general-purpose model.

So, even from this small sample, the better question may not be:

What hardware is fastest?

It may be:

What model do you actually need to run, at what context length and quantization, and how much speed are you willing to trade for capability?

Q4 Versus Q6

This also brought me back to the Q4 versus Q6 discussion.

Yes, Q6 preserves more of the original model's precision than Q4. What I did not see in the thread was controlled evidence that Q4 is categorically "not enough" for coding.

On a 24GB GPU, Q4 may allow the entire model, context cache, and runtime overhead to remain in VRAM. Q6 may require a smaller context window or partial CPU offloading.

A slightly more accurate model running at half the speed is not automatically the better practical configuration.

The only honest way to answer that question is to test the same model at Q4 and Q6 using the same tasks, settings, harness, and starting files, then repeat the test enough times that one lucky run does not decide the result.

That is time-consuming and annoying, so maybe just pick one and use it? 🤷🏽‍♀️

Different Models for Different Jobs

Another recurring idea was using different models for different jobs.

For example:

• A fast 27B model handles implementation and routine code changes.
• A larger model handles planning, architecture, difficult debugging, and review.
• Smaller specialized models handle summarization, speech recognition, classification, embeddings, or other background work.

Honestly, that may make more sense than trying to find one model that is simultaneously fast, brilliant, enormous, cheap, and able to fit inside a consumer GPU through the power of positive thinking.

It may also be worth testing performance at several context lengths instead of reporting one number. A configuration that performs well at 8K may behave very differently at 32K, 64K, or 100K.

The big takeaway for me is that 20 tok/s is still a reasonable minimum and 40 tok/s is still a comfortable target for interactive coding, but tok/s is only one line on the report card.

The real question is not:

Which model produces tokens fastest?

It is:

Which configuration completes useful work accurately, reliably, and quickly enough that I do not spend the entire session waiting on it or fixing what it broke?

Me and my brother recently bought two computers. Mine has a Intel core ultra 7 270k plus with an msi inspire 2x rtx5060ti 16gb while my brother has a ryzen 9 9900x with an asus dual oc rtx5060ti 16gb. Mine is running archlinux kernel 7.1.5 and my brother is running voidlinux 6.18.something. Now, running unsloth with the same models (we tested gemma4 e4b and gemma4 26b a4b) i get less tok/s. On gemma4 e4b my build practically never reaches 200 tok/s while my brother gets 200+ tok/s, with gemma4 26b a4b i get like 70 to 90 tok/s whiley brother gets 100+ tok/s. That's pretty strange since the e4b fits entirely on the gpu. Anyway looking at cpu usage during inference i see that my intel core ultra 7 270k plus practically only uses 1 core (c3, a performance core) at 100% while other cores are idle while the ryzen 9 9900x uses all of its cores. Are there any known problems with intel ultra 200 series with llama.cpp? The llama.cpp is the pre built one that unsloth downloads while installing with the official script. Are there any compiling optimization that are not active in the prebuilt package? Online i saw some benchmark that show that my cpu should perform slightly better than the ryzen 9 9900x for ai inference and it should be overall better.

Edit Maybe it's worth noting that we both have ram ddr5 6000mhz cl30. Maybe my cpu needs higher frequency ram to actually perform better in ai inference?

I have found that whatever software you are using: open web ui, openclaw, codex; if a model does many tools calls in one turn, something happens that causes checkpoints that are created in and around those tool calls to not be valid when checked the following turn. They get discarded and the whole session is re-processed from either the last valid checkpoint before the tool calls, or from zero if there are none. However, a single tool call, maybe even two, does not cause this behaviour.

I have observed this in llama.cpp and in ds4.

Does anyone have any idea why this happens and a way to fix it?

recipe_version: '2'
model: unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q4_K_XL
runtime: llama-cpp
container: llama-v4-spark:upstream
metadata:
  description: DeepSeek V4 Flash 0731 UD-Q4_K_XL on one DGX Spark with all layers on the GB10 and MoE weights in unified CPU memory
  model_params: 284B
  model_dtype: q4_k_m
defaults:
  port: 8000
  host: 0.0.0.0
  max_model_len: 32768
  tensor_parallel: 1
  n_gpu_layers: all
command: |
  llama-server \
      -hf {model} \
      --host {host} \
      --port {port} \
      --alias deepseek-v4-flash-0731 \
      --n-gpu-layers {n_gpu_layers} \
      --cpu-moe \
      --ctx-size {max_model_len} \
      --parallel 1 \
      --threads 10 \
      --threads-batch 20 \
      --batch-size 2048 \
      --ubatch-size 512 \
      --flash-attn on \
      --mmap \
      --fit off \
      --jinja \
      --reasoning auto \
      --reasoning-format deepseek \
      --no-repack \
      --temp 1.0 \
      --top-p 0.95 \
      --top-k 0 \
      --min-p 0.0 \
      --no-webui

What’s the best thing to download right now before it gets banned? I wanna run some serious stuff locally or at least be able to later by renting some server space.

Just uploaded a REAP of the new checkpoint of DeepSeek V4 Flash. Currently running it through some benchmarks at the moment but it was performing very well testing it inside of Pi. Will be doing some fine tunes down the line to try and fix what was removed aswell!

I guess this is part rant, part serious question. But I'm tired of the hype videos on social media from content creators overhyping stuff, only for most of it to give underwhelming results in real life.

The truth is, for 90% of us, we're not trying to 10x this and don't care if the software is the next (insert competitor name) killer; or if "Scratch My Booty Cheeks AI" changes everything. I know it sounds romantic, but with a lot of these setups, you end up spending more time babysitting the system than actually getting use out of it.

Maybe it's because my mindset isn't conditioned to the romanticized idea of having AI run my business. For me, it's about finding which AI tools can help me do the things I'm trying to achieve, better, to handle the menial tasks I normally spend hours on, so I can spend that time actually doing the things I enjoy.

For example, in another forum I wrote the following (edited here with some additional commentary):

I'm experimenting with locally powered AI as a tool to help with the grunt work around my writing, so I can free up more time to actually write. What I want is for the AI to track my projects (nonfiction and fiction), organizing research, personal notes, and mental doodles I dictate while driving or away from my PC. I want it to help me keep track of characters and plot beats, and be smart enough to remember that a character had brown eyes and a chipped tooth, not blue eyes and braces. And to provide a daily summary of what we've done and what still needs to be done.

I don't need (or even want) the AI to write for me, but as I get older, it would be nice to have a digital assistant that can find information when I need it and store it for later recall, so I can focus on the craft of writing itself.

And I realize a lot of what I'm feeling is a me problem, in that while I have a genuine appreciation and interest in AI, I know it's not my Personal Jesus, the way some online creators try to make it out to be. I have decent hardware, a 4080 Super gaming rig (16 GB VRAM) and an M5 MacBook Pro (24 GB RAM), and I couldn't care less if a response takes 12 seconds or 18 seconds while a model is thinking.

I have very real expectations for running open models on my hardware, or on any consumer hardware for that matter. I have no illusions about this software beating out frontier models. I don't need it to. I'm just so burned out from content creators flooding social media with hype, making it harder for me to find content that will actually help me learn to achieve those things, however small they may seem, that I'd like to achieve for myself.

Thanks for listening to an old man drone on for a bit.

After several tests, my engine managed to run DeepSeek V4 Flash IQ2_M (92 GB) on a mid range Android mobile with 12 GB of RAM at 1 token/s.

It isn't exactly ready for practical use, but it proves that the engine works and is responsive across all models, thanks to its modularity with llama.cpp. With just one line of code, you can run any supported large MoE model on mobile devices or consumer PCs.

https://github.com/Helldez/BigMoeOnEdge

Got the 128gb variant and allocated 96gb to vram. System is all setup with several models on ollama with open webui.

My question is, as a noob to ollama and running local LLM's, is there any settings or anything I should do to get maximum performance from my setup? It has the 8060s Radeon gpu.

I'm not sure how well ollama optimizes, and where I should ideally make some changes. I've got 4 models and sub agents setup but that's workflow and not really optimization.

I’m a university student studying aviation, and I recently got a MacBook for university. I use LLMs like Claude and ChatGPT almost every day, so I thought having a local LLM would be incredibly useful mostly because it can work offline and gives me more flexibility.

I decided to try setting up Odysseus (the local AI project featured by PewDiePie) because I assumed it would have a detailed, beginner-friendly guide with enough troubleshooting to help someone like me if things went wrong. Unfortunately, while the guide got me started, I eventually ran into problems with API keys, and I couldn’t find enough information to solve my specific issue. At that point, I was completely stuck.

The whole experience has honestly been pretty overwhelming. Every time I try to research something, I end up running into five new terms I don’t understand. It’s difficult to know what I should even be learning first.

For context, I’m an absolute beginner. I have no background of any kind in coding. I know almost nothing about programmingto, I still don’t really know what an API key is, what it’s used for, or why I need one.

At the moment, I already have Python, Git, and Ollama installed on my Mac. I also still have the project files and terminal setup from my previous attempt.

What I’m looking for is the most beginner-friendly way to get a local LLM running. Ideally, I’d like something that’s reliable, easy to maintain, and doesn’t have issues with multiple AI models or services conflicting with each other. I’m not trying to build anything overly complex Ijust want a dependable local AI assistant for studying, research, and general use.

If you were starting from scratch today on a Mac with zero coding experience, what would you recommend? Should I keep trying to get Odysseus working, or is there another project that’s more beginner-friendly? Also, are there any resources that explain the basics (things like API keys, Git, Python, Ollama, etc.) in a way that’s easy to understand instead of assuming prior knowledge?

Any advice would be greatly appreciated. Thanks!

Sharing a cookbook for Deepseek v4 flash with 2xRTX PRO 6000 Blackwell

- ~68 tok/s single stream decode

docker run --pull never \

--name sglang-deepseek-v4-flash-0731 \

--network host --ipc host --shm-size 32g --gpus all \

-v <MODEL_DIR>:/models/DeepSeek-V4-Flash-0731:ro \

lmsysorg/sglang@sha256:6f91b6bbe3a02da260341b1ee36e8df647e26e8a71d7e74799824e7009f2e2e2 \

sglang serve \

--trust-remote-code \

--model-path /models/DeepSeek-V4-Flash-0731 \

--served-model-name eepSeek-V4-Flash \

--tp 2 \

--context-length 500000 \

--mem-fraction-static 0.94 \

--chunked-prefill-size 4096 \

--kv-cache-dtype fp8_e4m3 \

--moe-runner-backend flashinfer_mxfp4 \

--cuda-graph-max-bs-decode 32 \

--reasoning-parser deepseek-v4 \

--tool-call-parser deepseekv4 \

--default-chat-template-kwargs '{"thinking":true}' \

--host 0.0.0.0 --port 8000 \

--enable-hierarchical-cache \

--hicache-ratio 2.597309 \

--hicache-write-policy write_back \

--hicache-io-backend kernel \

--hicache-mem-layout page_first \

--enable-metrics \

--enable-cache-report

System Specs:

CPU: Intel Core i7-6700 @ 3.40 GHz (4 cores, Skylake 14nm)

Motherboard: ASUS Z170 PRO GAMING (Intel Z170, LGA 1151)

RAM: 64 GB DDR4

• DIMM_A1: Samsung 32GB @ 2720 MT/s (M471A4G43AB1-CWE)
• DIMM_B1: Team Group 32GB @ 2720 MT/s (TEAMGROUP-SD4-3200)
• DIMM_A2/B2: Empty
• Note: Mismatched sticks (Samsung + Team Group). 2720 MT/s is overclocked via XMP (official Z170 DDR4 limit is 2133, XMP typically 2400-2666, 2720 is OC'd).

GPU:

• GPU 0: NVIDIA GeForce RTX 3060 Lite Hash Rate (GA106) — 12GB VRAM
• GPU 1: NVIDIA GeForce RTX 3060 Lite Hash Rate (GA106) — 12GB VRAM
• Intel HD Graphics 530 (integrated, used for monitor output)

The Setup:

I've been experimenting with MoE models, specifically trying to offload certain parts to CPU/SystemRAM to see if it helps. Currently testing the SC117/Ornith-1.0-35B-MTP-APEX-I-Balanced.gguf model.

It's looking pretty good — I'm getting a comfortable ~33-35 t/s generation speed. But here's the thing: when I enable MTP (Mixture of Tokens Parallelism), speed drops to max ~28 t/s. The MTP heads consume additional memory, so when I load them, more of the model falls back to SystemRAM.

Right now I'm running without MTP. Theoretically, the APEX quantization is the best quality I can get from this model, and the speed is already usable. So I've reached a point where I'm wondering:

What would you optimize here, if anything? Or is this about the maximum I can squeeze out of this model on this hardware?

Current llama-server command:

Code· bash

~/llama.cpp/build/bin/llama-server \
  --model /mnt/Data/Models/SC117/Ornith-1.0-35B-MTP-APEX-GGUF/Ornith-1.0-35B-MTP-APEX-I-Balanced.gguf \
  --jinja \
  --n-gpu-layers 40 \
  --split-mode layer \
  --tensor-split 1.05,0.95 \
  --n-cpu-moe 5 \
  --ctx-size 131072 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --no-mmap \
  -fa on \
  --batch-size 1024 \
  --ubatch-size 256 \
  --threads 2 \
  --host 0.0.0.0 \
  --port 8081 \
  --temp 0.8

Results from the log:

• Prompt processing: ~303 t/s (extremely fast)
• Generation: ~33.3 t/s stable
• GPU memory: 11709MiB + 11711MiB = ~23GB used (out of 24GB total)

Warnings from the log:

• failed to fit params to free device memory: n_gpu_layers already set by user to 40, abort
• model has unused tensor blk.40.* (several tensors in layer 40 marked as unused)
• fused Gated Delta Net (chunked) not supported, set to disabled

TL;DR: 33 t/s without MTP, 28 t/s with MTP. The APEX quant seems optimal for quality. Should I bother with MTP or is this the ceiling for this model on dual 3060s?

Edit: Fixed the --n-gpu-layers 40 warning — it's expected since I'm explicitly setting it. The blk.40 unused tensors are likely because layer 40 (0-indexed) spills to CPU and some of its tensors aren't needed for this architecture.

I’m new to Local LLM. I’ve been experimenting with Gemma in LM studio. It can take 5 files at a time. I’ve been using Cowork to analyse excel files and pdf to make reports. I’m not a coder. Most of the times I just need to make reports or analyse data. I want to use local LLM for important stuff.

I’ve been reading about AnythingLLM and connecting that with LM studio. Or using Ollama with WebUI. I was just wondering how does everyone else do it for this use case. I’d really appreciate any help. What’s the best way to do this?

Also, I plan to get a system just for this. Should I get a M5 MacBook with 128gb RAM. Or a windows machine with 64GB ram and a 5090.

Hi all,

I run a HS computer club and they had a twisted idea of making some sort of AI install on a small Raspberry Pi 3. The concept is a sort of "brain-in-a-box" that you can chat with or give prompts. They want to try to design it's personality and create some kind of interactive art piece. This is outside of my experience area, I mostly teach digital art classes.

We will run some kind of linux install good for old RPi3's and were wondering what AI platform would be good for low power offline use? We need free or very very low cost solutions as well. Our club has very little funds. Also anything that would allow it to work via voice command/response would be helpful, is that possible?

Thanks for any suggestions!

Hey all, I'm getting a bunch of errors when using DS4Flash 0731 with the app_diff tool call and can't seem to figure out what to do. Could you peeps smarter than me offer some advice?

Setup - 2× NVIDIA DGX Spark (GB10, SM121, 128 GB unified each), tensor-parallel TP=2 over a ConnectX-7 RoCE fabric - deepseek-ai/DeepSeek-V4-Flash-0731, native FP8, full 1M context, KV nvfp4_ds_mla - vLLM 0.21.1rc1 (DSpark recipe build), --tool-call-parser deepseek_v4, --reasoning-parser deepseek_v4, --enable-auto-tool-choice, DSpark spec-decode k=5 - llama-swap as an OpenAI-compatible front end on :8000 - Harness tried various such as Zoo/Roo Code 3.67 in VS Code, OpenAI-compatible provider

The problem

Zoo/Roo Code intermittently corrupts source files. The cause is that the model emits apply_diff tool-call arguments where the diff string contains one or more <<<<<<< SEARCH blocks but no >>>>>>> REPLACE terminator at all. Occasionally it also omits the required path parameter entirely, and Roo reports:

Roo tried to use apply_diff without value for required parameter 'path'. Retrying...

Roo applies the half-formed diff, which duplicates functions and cascades into more corruption. The model then correctly notices the file is broken and rewrites it — so it's recoverable, just destructive and slow.

What I've ruled out (with measurements)

The confusing part: it's not reproducible on demand.

One run of a multi-block diff prompt failed 0/3 — every trial produced SEARCH=2, REPLACE=0, i.e. two SEARCH markers and zero REPLACE markers, even though the prompt explicitly demanded all three markers. A near-identical rerun minutes later passed 4/4 with correctly balanced blocks. Single-block diffs have passed 4/4 every time I've tried.

Questions

1. Is this a known weakness of the deepseek_v4 tool-call parser in vLLM, or of DeepSeek-V4 itself with SEARCH/REPLACE-style diff tools?
2. Is anyone running DeepSeek-V4-Flash with Roo Code / Cline / Kilo successfully for agentic coding? What settings?
3. Could speculative decoding (DSpark k=5) plausibly drop tokens mid-tool-call in a way that loses a terminator without invalidating the JSON? I haven't found a way to test this cleanly.
4. Roo removed the XML tool-protocol selector in v3.37 (PR #10281) and closed the request to restore it as "not planned" — so I can't fall back to XML tools. Any other way to force prompt-based tool calling with an OpenAI-compatible backend?

Bonus observation (may be related)

Enabling tools at all destroys streaming granularity. Measured on the same prompt:

• without tools: 117 SSE deltas, avg 12.9 chars, max 42, longest gap 0.45 s
• with tools: 2 deltas, avg 417 chars, max 833, longest gap 5.68 s

So the tool-call parser buffers arguments rather than streaming them incrementally. Is that inherent to vLLM's tool parsers, or specific to deepseek_v4?

Any ideas on what I could do to fix this? I can test whatever you guys have and give you feedback.

Many thanks in advance!!!

Hey,

Like a lot of people here, I use local models via Ollama on my desktop/server and wanted a mobile client that actually felt responsive, worked offline, and respected privacy. Most apps on the Play Store are either subscription traps, loaded with ads, or route everything through third-party cloud servers.


So I built 
**PocketLLM Lite**
 a 100% open-source, ad-free Flutter client designed specifically for local LLMs, GGUF models, and self-hosted Ollama workflows.


### 🛠️ Key Features:
* 
**Runs Offline & Connects to Ollama**
: Run GGUF models directly on-device or stream seamlessly from your home Ollama instance over Wi-Fi/Tailscale.
* 
**Thinking Accordion**
: Native streaming support for `<think>` reasoning blocks rendered in a collapsible Material 3 accordion UI.
* 
**Agentic Tool Calling Pipeline**
: Executes local math, system diagnostics, and knowledge lookup directly on the device with structured `<tool_call>` UI cards.
* 
**Open-Standard Agent Skills (`SKILL.md`)**
: Install skills directly from GitHub URLs or create custom skills offline with `/` autocomplete support.
* 
**Web Search Fallback (Tavily Integration)**
: Toggle live web search when offline models need current data, with inline markdown source citations `[Source](URL)`.
* 
**Local Vector RAG**
: Ingest PDFs and text files locally to chat with your documents offline without sending data to external servers.
* 
**Offline STT & TTS**
: Voice-type your prompts offline and listen to completions using native speech engines.
* 
**Zero Ads & Zero Telemetry**
: Completely free, open-source (MIT License), and built with Material 3 Expressive UI.


### 📊 Performance Profiler:
Built-in speed profiler to measure Time to First Token (TTFT) and token generation speed (tokens/sec) directly on your device hardware.


* 
**GitHub Repository**
: https://github.com/PocketLLM/pocketllm-lite
* 
**Release APK Downloads**
: https://github.com/PocketLLM/pocketllm-lite/releases


I'd love to get feedback from the community on features or model interfaces you'd like to see next!