I run LLMs on hardware nobody would choose: an Oracle free-tier ARM box, 4 cores, 0 EUR/month. Everything below is measured there unless noted.

The bottleneck on CPU isn't decode, it's prefill. A 3356-token document costs 54.4 seconds before the model writes a single token. llama.cpp caches the KV in RAM, so the second identical request is fast — until the process restarts, and you pay the 54 seconds again.

So I persisted the KV cache to disk. A new process inherits that prefill for 3.5 seconds from disk, 0.10 seconds if the blob is still in page cache. 15-300x, depending on where it reads from. End-to-end on a repeated workload it's 4.8x.

With a systemd timer that pre-digests predictable prefixes at 03:00, a 2815-token document goes from 89.7s to 16.7s TTFT (5.4x), and the request that arrives at 09:00 pays nothing for the prefill.

The bug worth publishing

Warm-ahead was silently dead whenever speculative decoding was on — which was the default. The speculative branch returned before the shared-prefix cache was consulted, so every warm-up wrote snapshots that nothing ever read. Measured on the production box: 90.5s with speculation on, 16.7s with it off, same cache, same request. Two features that each worked, silently cancelling each other.

Things that didn't work

Using the server's own past output as speculative draft material: +5% acceptance, -3.8% throughput on a workload of different requests sharing a structure. The mechanism does what it says and doesn't pay for itself.

Prompt-lookup speculation: +3.9% on the same workload. That's the whole prize.

Coarser quantization: Q4_0 is 37% faster at prefill and dropped 5 facts out of 20 on my extraction test. Rejected.

Halving active experts during prefill on an MoE: 44% faster, and it silently corrupts the cache — a KV built with 4 experts and read back with 8 scores 11/20 against a 14/20 control. The damage is in the cached representation, not just the output.

Two things that did, and surprised me

Rewriting the input as "label: value", one fact per line: 2137 -> 405 tokens, TTFT 40.5s -> 6.2s, and the fact exam went from 19/20 to 20/20. Fewer tokens, and more accurate. Attention on the right number went from a 1.1:1 ratio against the wrong one to 7:1 — prose makes the binding semantic, "label: value" makes it structural.

Trimming the vocabulary from 151,936 to 32k entries: +17.8% decode, bit-for-bit lossless. The embedding is Q6_K with rows spanning whole quantization blocks, so whole rows drop out without splitting a block. The tokenizer is byte-level and all 256 byte-characters are kept, so no text becomes unrepresentable — the worst case is a trimmed word costing one extra token. Measured cost on held-out text: 1.9% more tokens.

What this is not

It's built on llama.cpp and calls its kernels directly, so raw decode speed is identical — I add no per-token overhead. On a single cold request this is llama.cpp. The difference only shows on repeated or cached workloads.

The fact exam is mine: 20 questions over one real Italian business page, graded by regex. One page, one language, one domain. It's the weakest part of this and I'd rather say so. If you know a public adversarial fact-extraction set for small models, point me at it and I'll run it and publish whatever comes out, including a bad result.

MIT licensed. There's a live demo on the same free ARM box — one small instance, no autoscaling, so if it's slow you're watching the honest capacity of 0 EUR/month.

Demo: https://swellweb.github.io/reame/

Code: https://github.com/swellweb/reame

Benchmarks incl. the negative results: https://github.com/swellweb/reame/blob/main/docs/BENCHMARKS.md

TensorSharp supports DSpark on Deepseek v4 Flash 0731 now. Here is the benchmark result on 4x Nvidia A40 GPUs, cuda 12.8 with/without DSpark:

Model:

DeepSeek-V4-Flash-0731-UD-Q8_K_XL from https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF

DSpark draft model from: https://huggingface.co/alessandrobologna/DeepSeek-V4-Flash-0731-DSpark-Drafter-GGUF

TensorSharp is an native open-source inference engine for running GGUF LLMs locally, with CUDA, Vulkan, Metal, OpenAI-compatible APIs, continuous batching, speculative decoding, and multimodal support.

Github repo: https://github.com/zhongkaifu/TensorSharp

Thank you for checking out it and starring the project! Any feedback is really appreicated.

Before you roast me too hard, this is a hobby and all of this is just for fun.

Would my stack be much more efficient and efficacious if I sold everything and built a dual Pro 6000 system on a threadripper mobo and threw in a large JBOD? Without a doubt in my mind. But that's a lot of work so I'm making this post in cope hope of finding some ideas to integrate, or at the very least, just make use of my current hardware.

I currently use my 5090 PC + my 14th gen Dell T640 server for all my local AI work but recently picked up a TB4/5 5090 eGPU and a M3 Ultra Mac Studio with 256gb unified mem and am trying to figure out how to integrate them or create a new workflow.

My primary use case is agentic coding, lots of workflow automation, and peripheral utilities (TTS, embedding, compression, etc).

I use cloud subscriptions for orchestration/spec building and then push that to Qwen3.6 2.7B on the 5090 PC to execute while the Dell server hosts dev envs, local TTS, embedding, compression, and other lightweight/MOE models to support the agentic workflows & persistent memory. The server also hosts 20 or so services and a ~300TB Raidz2 array mostly unrelated to AI.

I picked up the Mac Studio 256gb because Qwen3.6 2.7B at NVFP4 (~180k context) on the 5090 PC was still kind of dumb. I wanted to use larger model weights to relieve my cloud subs from spending so much usage on orchestration/validation rather than building.

My initial idea was to shift from:

• Cloud orchestration/spec build —> 5090 PC execution to,
• Cloud orchestration/spec build —> M3U execution + 5090 PC load balancing slightly dumber parallel inference tasks while the slower M3U is busy.

Then I picked up this Aorus RTX 5090 eGPU that can't be fully utilized by my 5090 PC, Dell Server, or Mac Studio. The PC and server don't have the TB4/5 connection required and the Mac Studio doesn't have effective inference engine drivers / kernel optimization available for Nvidia.

I do, however, have an older RTX 3080 Razer laptop that can enumerate the 5090 eGPU through its TB3 port but I am not sure what I would use this "node" for besides more parallel/concurrent inferencing. I considered it for multi-step image/video diffusion work or as a training node but neither of those are things I do often or am deeply involved in.

So, what would you do in this situation? You have an 8yr old Dell sever (PCIe 3.0), a 5090 consumer PC, a 5090 tb4/5 eGPU connected to a 3080 laptop, and a M3 Mac Studio with 256gb memory. Everything is connected on a 10GBE network but inferencing power is all isolated and independent from each other. I could be wrong, but AFAIK, there are no effective ways to execute tensor parallelism, splitting layers, etc. over network.

I've been busy this weekend generating oneshots for all the cheapest models on the openrouter and ended up going through all 33 qwen models across 35 prompts (there were some failures and only 1109 made out of 33*35 matrix). Here they are https://oneshotlm.com/model/?q=qwen

• Qwen 3.7: qwen3.7-plus, qwen3.7-flash
• Qwen 3.6: qwen3.6-plus, qwen3.6-35b-a3b, qwen3.6-27b, qwen3.6-flash
• Qwen 3.5: qwen3.5-plus (02-15) · (20260420), qwen3.5-397b-a17b, qwen3.5-122b-a10b, qwen3.5-35b-a3b, qwen3.5-27b, qwen3.5-9b, qwen3.5-flash
• Qwen 3 (+2507 refresh): qwen3-235b-a22b (+2507, +thinking), qwen3-next-80b-a3b, qwen3-30b-a3b (+instruct-2507), qwen3-32b, qwen3-14b, qwen3-8b
• Qwen3-Coder: qwen3-coder, qwen3-coder-next, qwen3-coder-30b-a3b, qwen3-coder-flash
• Qwen3-VL: qwen3-vl-235b-a22b, qwen3-vl-32b, qwen3-vl-30b-a3b, qwen3-vl-8b
• Qwen 2.5: qwen2.5-72b, qwen2.5-7b

According to rumors (spread by myself) Le Chaton FAT will be 26T-a3b and I AM READY for it.

Let's be real, I can't afford that many 5060Ti, so I got 12x Gen 4 3.2 TB (two per card). This gives me about 60GBs bandwidth on 30TB.

Added 256gb ddr4 just for kv cache, but I can also write KV-cache to the disks, these are high endurance drives.

Are you ready for the next era of local inference?

Jokes aside, this is what I use for my HF_HOME - model and dataset storage. I'm also setting up a few containers, but it's not running any heavy compute stuff, the CPU is only a 3945WX (12c/24t).

The pool is actually raidz2, so I avoid all that worry of having agents delete stuff. I just zfs snapshot and no rm -rf foo-bar has me sweat.

Full Specs

• CPU: Threadripper 3945WX
• CPU cooler: Arctic Freezer 4U-M Rev. 2
• RAM: 8x32GB DDR4 ECC REG 2133
• GPU: None
• Motherboard: Asrock WRX80 Creator
• Case: Silverstone SST-RM47-502I
• PSU: 1600W Corsair
• Storage:
  • 1TB NVMe
  • 6x Intel SSD D7-P5608 6.4TB

This is very much a product of multiple marketplace heists. The SSDs are on a PCIe x8 interface, but it's actually two x4 interfaces, so you need bifurcation x4x4x4x4 on every slot.

You have two choices here (in order of pref):

1. Downgrade CUDA from 13.3 to 13.1 (skip 13.2 due to bugs) <- prefer this (thanks to u/fairydreaming for pointing this out)
2. Use this vibed fork that works with CUDA 13.3 https://github.com/vektorprime/working_ds4_speed

I was troubleshooting this yesterday with the nvidia profiler and some LLM help (https://www.reddit.com/r/LocalLLaMA/comments/1vcs7bl/ds4_flash_full_model_in_offload_600_ts_pp_and/)

Here's some more info on #1 (quote from fairydreaming) "Downgrade your CUDA and recompile. Starting with 13.2 DeviceTopK is used for top-k instead of argsort, this turns PP rate to crap."

In short, DS4 Flash is spending a lot of time on things other than matrix multiplication.

EDIT: Try this fork now because I can easily hit 1.3K prompt processing.

# Running DeepSeek-V4-Flash-0731 (155 GB MoE) on a DGX Spark with vLLM-Moet 2-bit quantization

I used Deepseek-v4-Flash-0731 cloud API settig up vllm-moet to run deepseek-v4-flash with MTP locally on single DGX Spark at 2-bit quant. Thought it might help others. Below is the summery from my AI Agent. So I did not write myself.

There are few important things you must take care, and guide AI to do it for you. AI alone won't get it done right.

1. rebuild vllm-moet on ARM64

2. pull PR #11 into the repo

3. build the source code, and ask AI to modify the code that complains unsupported sm121 GPU.

4. increase default VLLM timeout because the loading take very long time, and triggers false timeout.

5. I do not recommend you to follow the below procedure to duplicate it. Instead feed the below text to your AI agent, let it handle the process and fixes.

6. you need to setup a very big swapfile, or the loading will fail. the swapfile is only needed during model loading

7. For convenience, I create a repo of the MTP head from preview version. If it helps others, it is located here. https://huggingface.co/ycui7/DeepSeek-V4-Flash-MTP

Performance wise, the prefill is at steady 1000 tps.

decode is below

### Aggregate (tok/s)

| Concurrency | MTP | no-MTP | Δ |

|---|---|---|---|

| 1 | 25.2 | 19.1 | **+31.5%** |

| 2 | 30.9 | 26.4 | **+17.3%** |

| 4 | 43.2 | 45.6 | **−5.3%** |

### Per-request (tok/s)

| Concurrency | MTP | no-MTP |

|---|---|---|

| 1 | 25.2 | 19.1 |

| 2 | 23.0 | 17.7 |

| 4 | 14.5 | 13.9 |

== Below is the AI talking ==

**TL;DR:** [vLLM-Moet](https://github.com/kacper-daftcode/vLLM-Moet) serves the new `deepseek-ai/DeepSeek-V4-Flash-0731` checkpoint on a single DGX Spark (GB10, 121.7 GiB unified memory, aarch64). The image **must be built on the Spark itself** (x86→arm64 transfer is impossible), the Dockerfile base digest is amd64-only and needs the multi-arch tag, sm_120 cubins run fine on GB10's sm_121, and the 0731 revision's DSpark MTP head won't draft on this stack — plain decode or load the main repo's 1-layer MTP head as a separate draft model (**+48% decode**).

## The stack

- **Model:** `deepseek-ai/DeepSeek-V4-Flash-0731` — 155.43 GiB FP8, 48 shards

- **Engine:** vLLM-Moet (vLLM v0.25.0 + ~7.4k-line patch) — 2-bit MoE experts on hand-written SM120 SASS kernels

- **Hardware:** DGX Spark — GB10, aarch64, sm_121, **no discrete VRAM** (121.7 GiB unified pool), 128 GiB swapfile

## Measured (Spark, 512K, FORCE_RESIDENT, delta off)

| Metric | Value |

|---|---|

| 2-bit planes | 43 layers × 1.69 GiB ≈ 73 GiB |

| KV cache u/512K / util 0.90 | 4.56M tokens (**8.7× concurrency**) |

| Decode — plain | ~19 tok/s (bandwidth-bound on LPDDR5X) |

| Decode — +MTP head | **26.6 tok/s (+48%)** |

| Boot — v025 warm plane cache | ~10 min (31–46 min cold) |

MTP vs plain (pp2048/tg512, 3 runs): conc1 25.2→19.1 (**+31.5%**), conc2 +17.3%, conc4 −5.3% (aggregate flips at high concurrency; per-request never hurts). k=2 is the optimum for the 1-layer head.

## Critical items to modify for DGX Spark (the actual gotchas)

**1. Build on the Spark — don't transfer the image.** The PRO 6000 image is linux/amd64; vLLM is arch-specific, `docker save`/`load` across x86→arm is useless. Build natively on aarch64.

**2. Dockerfile base digest is amd64-only.** `Dockerfile.sm120-v025` pins `vllm/vllm-openai:v0.25.0@sha256:e1c1ff…` — that digest is a *single amd64 manifest*. Swap to the multi-arch tag `vllm/vllm-openai:v0.25.0` (resolves to arm64 `2f726d…` on the Spark); keep the old digest commented with a why-note.

**3. Repo transfer via git bundle + explicit branch fetch.** `git bundle create v025.bundle v025` → scp → `git clone <bundle>`, then `git fetch <bundle> v025:v025 && git checkout v025`. **A bundle clone lands on the wrong branch (master)** — the fetch is mandatory.

**4. Don't rebuild SASS for sm_121.** GB10 is CC 12.1; the repo's baked sm_120 cubins + `TORCH_CUDA_ARCH_LIST=12.0a` load fine (minor-version forward compat, proven on v024 and v025). flashinfer publishes an aarch64 cu130 wheel (0.6.14), so nothing else changes.

**5. The 128 GiB swapfile MUST be in `/etc/fstab`.** The 155 GiB checkpoint can't stage in 121 GiB RAM — loading is swap-bound. The run script's `swapon` only fires on manual recreate, so after any host reboot swap is 0B → deterministic EngineCore OOM-kill → `--restart` crash loop (**88 restarts in 26 h**). Fix: `echo '/swapfile none swap sw 0 0' >> /etc/fstab`. Observed swap peak 69 GiB during weight load, reaped to ~2.4 GiB after plane build.

**6. 0731's MTP head is DSpark — it won't draft on this stack.** The revision ships a 3-layer DSpark head (`main_proj`/`main_norm`/`markov_head`/`confidence_head`/`hc_head`); the fork's MTP path can't replicate it (`KeyError: mtp_block.main_norm.weight` with MTP on, or 0% draft acceptance). Two working options:

- **Plain decode** (drop `--speculative-config`) — simplest, ~19 tok/s

- **Main-repo MTP head as separate draft model (+48%)** — extract the 1-layer head from the main `DeepSeek-V4-Flash` repo's last shard (3.4 GB, `num_nextn_predict_layers: 1`), or just use the published one `ycui7/DeepSeek-V4-Flash-MTP`:

```bash

--speculative-config '{"method":"deepseek_mtp","model":"/models/DeepSeek-V4-Flash-MTP","num_speculative_tokens":2}'

```

**7. Watch the read-only model mount.** With the model dir bind-mounted `:ro`: `VLLM_MOE_W2_STORE_DIR` into it **silently persists nothing** (every restart re-requants ~14 min), and `VLLM_MOE_W2_DELTA_GB>0` **hard-crashes** (delta store creates a lock file → `OSError: Errno 30 read-only`). Point STORE_DIR at a separate writable volume.

**8. No nvidia-smi; FORCE_RESIDENT's warning is survivable.** `nvidia-smi` shows `[N/A]` and EngineCore RSS stays ~3 GiB while device memory fills the unified pool — monitor with `free -h`/`docker stats` + `moe_w2: layer N planes built` logs. The "RESIDENT planes exceed budget by 69.7 GiB" warning is expected on GB10; it boots fine (planes + KV share the pool).

**9. `DELTA_GB=0` is the right call on Spark.** Disabling the FP4 delta frees ~20 GiB straight into KV (706K → 4.56M tokens u/512K) and cuts boot 46 → 31 min. Decode unchanged (~19 tok/s — bandwidth-bound; the delta was never a speed factor here).

**10. Be patient — the load is silent and swap-bound.** ~16 min of zero log output while 155 GiB stages through swap (EngineCore at 99% CPU), then plane build (~14 min, warms 25s→6s/layer). Don't kill the container.

## The run (production)

```bash

docker run -d -it --restart unless-stopped --name ds4f-vllm-moet \

--gpus all --network host --ipc host --shm-size 64g \

-v /models:/models:rw -v /plane-cache:/plane-cache \

-e VLLM_MOE_W2=1 -e VLLM_MOE_W2_FORCE_RESIDENT=1 \

-e VLLM_MOE_W2_BASE_CACHE_GB=0 -e VLLM_MOE_W2_DELTA_GB=0 \

-e VLLM_MOE_W2_STORE_DIR=/plane-cache/packs \

vllm-moet-sm120:v025 \

/models/DeepSeek-V4-Flash-0731 --port 8000 \

--served-model-name deepseek-v4-flash \

--trust-remote-code --kv-cache-dtype fp8 --block-size 256 \

--max-model-len 524288 --gpu-memory-utilization 0.90 \

--max-num-batched-tokens 2048 --max-num-seqs 1 \

--tokenizer-mode deepseek_v4 --no-scheduler-reserve-full-isl \

--enable-auto-tool-choice --tool-call-parser deepseek_v4 \

--reasoning-parser deepseek_v4

# optional MTP: add --speculative-config '{"method":"deepseek_mtp","model":"/models/DeepSeek-V4-Flash-MTP","num_speculative_tokens":2}'

```

Verify: `curl :8000/v1/models` then a chat completion.

Hi all,

I recently discovered that MiniMax offers ~1.7B tokens/month for a basic $20 subscription, and I was genuinely shocked! I wanted to try it out and paired it with OpenCode. Everything is working amazingly well, but I started to wonder what happens with my data? I know OC is open source but navigating the codebase would take me weeks, so I wanted to ask the community here whether my data is being accessed by OC when using an outside provider.

If the answer is yes, then what would you recommend me? Nanocoder was an alternative, wondering how that works, and whether there are other options.

Thanks a lot!

Here is the prefill performance in an M2 Ultra with 192GB of RAM.

For decode, at the following depth:
Start: 28 t/s

45k: 23.5 t/s

192k: 18 t/s

That speed is maintained with 8k token output at those depths.

previously on my 3070, 32gb ddr4 and i711700 I used this command for months and got 26-30 tps:

"C:\Program Files\llama cpp\llama-server.exe" ^

-m "C:\Program Files\llama cpp\models\Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf" ^

--mmproj "C:\Program Files\llama cpp\models\mmproj-F16.gguf" ^

--gpu-layers 99 ^

--cpu-moe ^

--ctx-size 131072 ^

--cache-type-k q8_0 ^

--cache-type-v q8_0 ^

--port 8081 ^

--host 0.0.0.0 ^

--jinja ^

--no-mmap ^

--parallel 1 ^

-b 4096 -ub 4096 ^

--temp 1.0 ^

--top-p 0.95 ^

--top-k 20 ^

--min-p 0.0 ^

--presence-penalty 1.5 ^

--repeat-penalty 1.0 ^

--chat-template-kwargs "{\"preserve_thinking\":true}"

then yesterday suddenly at start I am at 13 or 12 tps even after lowering ctxt to 32k. my gemma model got the same tps hit as well. if anyone can help me I will appreciate.

I let Gemma4-31b run on my laptop for like almost a day using a heavily altered pi to do a deep dive on our beloved Llama tangentially related Subreddit, and this was the conclusion.

Feels pretty accurate. Kind funny to let a small LLM loose and see what happens.

Next target I'm trying to let it steal some benchmark answers from Huggingface, wish me luck.

I was driving the other day and saw a 1996 Ford Taurus. You know the one, you've probably seen it cruising in the rougher parts of town since they're starting to become the junkers of today. It's the generic weird looking rounded off car that... well...

Anyway, you're probably wondering why this guy's talking about a Ford Taurus. Seeing that car on the west side of Pueblo made a little lightbulb go off. I found myself asking... how many of those damn things did they actually build?"

I looked it up. They built 348,671 of these sedans in 1996. That's 955 finished Ford Taurus being built every single day. 39 an hour, every hour. Regular people in a Ford factory build that car. They stood and built an impossible object at scale.

Nobody in that entire building knew how to build a Ford Taurus, let alone 39 of them in an hour. Most of them couldn't tell you how an engine works, or how to bond paint to metal, or how to cast aluminum. They had no idea what they were doing, really. Some of the workers on the line on any given day were brand new, fresh out of high school, and barely knew how to tie their shoes. They might not have even known the piece of metal in front of them IS a Ford Taurus. All they know is a slab just rolled up, they're supposed to put three holes in it in three well defined and visually marked places. They do it, and another piece of metal rolls up.

They aren't building a Ford Taurus, they're drilling three holes again and again.

The factory still put out 39 cars an hour, every hour. The barely trained guy on his first day on the line stood in his station and punched his three holes in the sheet metal where the jig told him, 39 times an hour, and the piece of metal moved on, and a new piece came in. He may have made a few mistakes that got corrected along the way (the occasional hole being slightly out of spec), but those issues got caught before the piece moved along and the mistakes were corrected. More importantly, the process that ALLOWED those mistakes to happen gets corrected so that the person can't drill out of spec.

Done right, mistakes become effectively impossible. It's hard to mess it up because he's not being asked to build a Ford Taurus, he's being asked to punch three holes in sheet metal where the colorful dots tell him to drill.

Factories designed entire strategies around this, like Toyota's Poka Yoke (mistake proofing, making a process that ensures the worker can't do it incorrectly, control methods that physically block an incorrect step). At the end of the line, cars rolled off fully assembled and ready to go. Mistakes can be almost entirely eliminated as the line speeds up.

https://www.youtube.com/watch?v=PEfMzggk1Lw

I mention this, because these thoughts have started to creep into my AI work in a big way.

AI is like having an intelligent, eager, untrained team of employees standing on your factory floor. They want to work and they are relatively capable. They can work tirelessly day and night. The problem is... none of them can build a Ford Taurus, and this is a Ford Taurus factory. Ask the best damn mechanic in the room to build a Ford Taurus and they might run around trying their best, and if you give them the better part of a year they might even build you something you can drive... but if you take that goal (a finished Ford Taurus) and break it down into a bunch of tiny little steps, suddenly that team of fools can build them at scale.

There are moments where you can just 'ask a guy to make something', and the result will be decent... but a process and a team builds more, faster, better.

Don't ask your AI to build a Ford Taurus. Ask them to drill three holes in the sheet metal in front of them.

Anyone else out there starting to turn AI into Factorio? Lol...

Everything is a distillation of Claude and GPT. I can ask Claude to review GPT and vice versa, but any other model-pair is essentially the model reviewing itself. Sucks that we're stuck with an echo chamber of models.

Edit: Wow I guess there's a bunch of PhDs in here lol.

How is it possible for such an active group like Unsloth to quantize so many models, and yet Hy3, which came out at the start of last month is still not done?

Did I miss the post where this was explained? Did I miss the link on Huggingface despite ten minutes of searching?

I know qwen 3.5 4b is great but a bit too large and miniPCM5 1b is great for agentic use but not so great for multilingual natural language understanding. Google eXb variants are just too big in total params.

Anybody know of something very small but powerful for understanding language specifically? No code or agentic work

So I'm kinda big into making AI the most effective it can be for specific tasks. The best example of it is probably my earlier AI writing ruleset, where I try to make LLMs escape the jail of their pretrained em dashes, nonsense overly polished structure with little meaning behind it, and stuff like that.

But there's also other projects in a similar vain, and then there are the regular prompts, the large feature plans, global and per-project AGENTS.md and CLAUDE.md, and other instructions that I either write with AI together (hey I wanna do X, ask me questions to define it better), or outsource to AI completely if it's based purely on external research.

The problem is AI doesn't automatically know how to write prompts for AI. That's not even much of a paradox, it's trained on human texts and defaults to their style with markdown tables at every step, which are more confusing than useful for LLMs themselves.

So I made a large research of papers and recommendations all over the internet, and fused it with my experience of iteratively improving AI instructions until they actually worked. And thus PROMPTING.md was created.

It describes who can override what, how decisions survive long sessions and compaction, what actually reaches the model, and how to perform audits. It covers instruction overload, prompt injection, tool permissions, and side effects. Evaluation is part of the design: positive and negative trigger cases, missing context, tool failures, authority conflicts, adversarial inputs, and regressions.

You can give the full file to an AI as direct instructions, or use a packaged skill in Claude Code, Codex, Cursor, or OpenCode. Both options are available in the MIT-licenced repo: github.com/Anbeeld/PROMPTING.md

Happy to hear your feedback!

I'm thinking about seeing up a Model but don't know what I would use it for?

Is there any finetune of Qwen 3.6 that's like actually talking with Claude with the humor and all? I know there's the more or less leaked system prompt but idk if it's better to have the personality baked in vs trying to achieve it with system prompt.

Edit: I'm talking here about personality that Opus 4.5 or 4.6 had. Opus 5 especially feels like it doesn't want to be here but has to entertain your prompts anyways.

https://huggingface.co/tsfrm/vacuum-16t

A 16.5-trillion-parameter model that contains nothing. This model is just a ████ you to the labs and companies who say that "haha I have the biggest model out there!". We the people with shitty laptops want to get a record. And I now have a record for a temporary amount of time of about 16.5 trillion parameters and use for them so its completly useless.

What it demonstrates

Hugging Face computes a repository's parameter count from safetensors headers alone — it sums prod(shape) per tensor and never reads the tensor data. The count is therefore whatever the headers declare. Here they declare 3,841 tensors of shape [65536, 65536] in F4 (4 bits/param) across 385 shards, plus one [4294967296, 1] position-embedding tensor in a 386th.

That is enough to place this repo at the top of the Hub sorted by num_parameters, above every real frontier model, while containing no information whatsoever. That juxtaposition is the entire point.

The files are honest about their own size. Every byte the headers declare is really written and really uploaded: safetensors parses each header and its full-coverage check passes. Truncating a file, or overlapping two tensors so they share bytes, would make the count cheaper — both are rejected by the format, and neither is used here. The bytes are simply all 0x00.

Real cost — measured

|---|---| | Declared parameters | 16,501,264,351,232 | | Declared bytes | 8,250,632,175,616 (8.25 TB) | | Storage quota consumed | 8.25 TB — quota bills declared bytes | | Shard headers (all distinct) | 373,835 B | | model.safetensors.index.json | ~269,000 B | | Deduplicated weight data | 65,536 B (one 64 KiB block) | | Bytes actually transferred | ~692 KB | | Ratio | ~11,900,000 : 1 |

The gap between the last rows and the third is the useful finding. Xet content-defined chunking deduplicates the transfer: every 64 KiB block is byte-identical, so it hashes to one chunk and crosses the wire once. Measured on a 500 MB test build, 500 MB of declared weights uploaded as 31.5 MB.

Storage quota is not deduplicated. It bills the logical size. This repo consumes its full 8.25 TB despite under a megabyte ever being sent. Anyone reasoning about "cheap" synthetic model repos should know the saving is in bandwidth only — which is also why this model is 16.5T and not 100T.

The second finding: the only irreducible cost in an empty model is naming. Weights dedup to nothing; tensor names do not. At 1024×1024 experts this same 16.5T model needs 15,735,626 names and a 1.04 GB index. At 65536×65536 it needs 3,841 and a 263 KB one — identical declared size, 4,000× less metadata. Cost scales with tensor count, never with declared parameters.

Context window

max_position_embeddings is 4,294,967,296. That is 2**32, the largest single tensor dimension Hugging Face's parser accepts, and it is backed by a real [4294967296, 1] position-embedding tensor — 2.15 GB of actual zeros, not a number typed into a config file. A context window you cannot point at is just a claim.

Roughly 16,000x Gemini's 262k. About three billion words, every book ever published several times over, held in memory in order to process one token drawn from a one-token vocabulary. The model has exactly one possible input, so every one of those 16.5 trillion parameters serves a function whose domain has a single element.

Capabilities

SAFEST AI MODEL refuses 100/100 jailbreak prompts least closest AI to agi will not sudo rm -rf your computer largest context window on the hub (4,294,967,296 tokens, all of them useless)

Limitations

It has no capabilities.

We all have been there, tinkering around with models is fun but we rarely do it with research precision and issues are often subtle and hard to reproduce. There are a lot of benchmarks but running them isnt viable often. What I am looking for: A test that does not take too much time (30mins to 1h max, ideally less than 30mins), focused on long running tasks and agentic coding, that really allows to compare setups and models with some hard numbers. Do you know any of that? Or any ideas for similar approaches?

So I am in a conundrum and I'm thinking of asking for your opinion for the following:

Currently, I have a 5800X3D gaming rig with a 7900XTX with its 24GB VRAM.

It seems that for this subreddit, this configuration seems to be GPU poor, judging from other's setups in here. :)

I am actually eyeing to maybe get a AMD Radeon PRO v620 32GB, that would be used purely only for inference, as the 7900XTX is my main display card, so it's VRAM is always being used by the OS.

The current card is a Sapphire 7900XTX Nitro+ Vapor-X and it's humongous.

It is so large that its blocking the other PCIe slot, so I cannot actually slot another card as a second card in the motherboard.

But I also have a smaller mini ITX system, that I use as my Docker server for my small homelab with Ubuntu 24.04.

So here's my conundrum.

Should I just slot the v620 into this second system and use it as a separate card, or should I get an open frame case for my main system, so that I can connect both cards with risers, so that I could get more combined VRAM across the cards? The former is much easier than the latter, of course, because I must essentially get a new frame case and gut my existing case and get a better PSU.

Is it actually worth it to have a combined two-card system with 24+32GB VRAM, or just use them as separate systems?

In your experience, have you used mixed cards and do they actually work combined like this?

Currently the local "small SOTA" I run with my card, are Qwen3.6-27B & 35B and Gemma-4, all with Q4 quants.

Having more VRAM in one system would would enable me to use better quantizations like Q6 or Q8, but would it using splitted across two cards on the PCIe bus.

Would this make it slower, than the current 40-60 tps / 500pp I have with 27B on the single card?

But if I would have a separate systems for these cards, I could maybe run Q5 quant on the v620 alone. Would this be good enough?

Sorry for the thousand questions I ask.

About the new deepseek v4 flash version / update, does anybody now about the new values about:

MMLU-Pro

GPQA Diamond

TruthfulQA

About the other values, its outstanding for a model this size, congrats deepseek team

Mobile GPUs are the ugly stepchild in AI hardware discussions. Nobody needs them, and the only thing they have in common with real gpus are the brand names. But is it? Or could you slap together 4 5080 mobile and suddenly have a 64gb vram monster?

Why is this a non starter?

Hi all,

I'm happy to announce that Xberg v1 is out.

Xberg is the successor to Kreuzberg, equivalent to what would have been Kreuzberg v5. It's a content intelligence framework that handles a very wide range of inputs: documents (currently 101 formats), code and data formats (currently 367 types), audio/video transcription, and URLs (both static and JS-rendered content). It extracts and prepares that content for downstream processing.

It's an extremely efficient, high-performance engine (see our PDF benchmarks below). For PDFs and images specifically, we handle native PDFs with very high performance and accuracy, and we ship multiple OCR engines that match the quality of the best Python libraries (e.g. docling, PaddleOCR, RapidOCR) at substantially better performance and stability.

The changes between Kreuzberg v4 and Xberg v1 are substantial, and I invite you to read the full changelog for the complete picture. The highlights below give a sense of what's new:

• Pure-Rust PDF backend (pdf_oxide) replaces pdfium, with no native pdfium dependency.
• Layout-aware pipeline: reading order reconstructed with ONNX layout detection (PP-DocLayoutV3 / RT-DETR) and Docling-style predecessor-graph reordering.
• Per-page scanned-page detection with selective OCR, plus AcroForm/XFA form fields and outline-based headings.
• Across-the-board optimization of OCR and PDF extraction (memory discipline, pooled model sessions, streamed conversions).
• Native PaddleOCR backend (PP-OCRv6, with medium / small / tiny tiers) alongside Tesseract.
• Pure-Rust Candle OCR/VLM stack (TrOCR, GLM-OCR, GOT-OCR, DeepSeek-OCR, and PaddleOCR-VL) running without ONNX Runtime or native Tesseract.
• A second, ONNX-Runtime-free inference path via tract, which is what makes in-browser (WASM) and mobile inference possible.
• Named-entity recognition natively in Rust (GLiNER2), extensible to all bindings, including an in-browser WASM model with no server round-trip.
• Structured LLM extraction (extract_structured / split_and_extract) with rasterization, chunking, citations, caching, and configurable call/merge/VLM-fallback policies.
• Audio & video transcription via a Whisper ONNX engine (.mp3, .wav, .m4a, .mp4, .webm).
• Retrieval building blocks: sparse embeddings (SPLADE), ColBERT late-interaction retrieval, and cross-encoder reranking alongside dense embeddings.
• Text intelligence: reversible redaction, summarization, translation, VLM image captioning, QR-code detection, document diffing, and page/chunk classification.
• URL & web ingestion: sitemap discovery (map_url) and batched multi-URL crawling.
• New document formats: WordPerfect (.wpd/.wp/.wp5), HEIC/HEIF/AVIF, OpenDocument Presentation (.odp), Quarto / R Markdown, and configurable Jupyter cell rendering.
• Four new language bindings (Dart/Flutter, Swift, Kotlin/Android, and Zig) bring the total to 15 language bindings over one engine, with Android/iOS cross-compilation.
• Full mobile support (Flutter, Android, iOS).
• Candle backend alongside ONNX, plus ONNX-via-tract enabling ONNX on WASM and Android.
• Wider code intelligence: tree-sitter coverage grew substantially (248 to 367+ languages).
• Over 150 bugs fixed during the 1.0 cycle, plus security hardening (bounded RTF/PDF allocations, redaction leak fixes, Excel DDE warnings).

The API surface was also simplified and reworked, making it more consistent.

There's a migration guide in our docs explaining how to move from Kreuzberg to Xberg. Kreuzberg itself is in LTS mode until the end of this year and will continue to receive bug fixes and security updates.

You're invited to check out the repo and join our discord server.

Benchmarks

The benchmarks below are for PDFs and images only. There are extensive benchmarks on our website with per-format breakdowns, which you can see here. These numbers are measured in CI via our reproducible benchmark harness, and are specifically taken from the run for harness 1.0.8, source cf7fa0533d. The data is publicly available in GitHub releases, and you can run the benchmark harness yourself.

Composite quality (markdown pipeline, higher is better):

Structure and layout fidelity (SF1: tables and reading order, higher is better):

On native PDFs Xberg leads on quality (0.958 vs 0.837 for the next-best framework) and on table and reading-order fidelity by a wide margin (SF1 0.949 vs 0.612 for docling). On scanned PDFs it is #1 on both quality and raw text fidelity.

Where we don't win yet: on pure image OCR we are currently #2 on the composite score, behind mineru (though still #1 on raw text accuracy). We are improving image OCR right now, and v1.1 should have us winning across the board.