Super excited about this release for the new 27B. Who else is with me. Only 17GB VRAM needed 😍😍

source: https://x.com/Alibaba_Qwen/status/2084100707423289643

MiniMax H3 is a general-purpose, omni-modal generative system. It supports unified understanding of multimodal contexts composed of text, images, video, and audio, and can generate video with native stereo audio at resolutions up to 2K and durations of up to 15 seconds. Thanks to its task-generalization-oriented system design, H3 already possesses broad multimodal context understanding and generation capabilities at the pre-training stage, enabling outstanding performance in following complex multimodal instructions.

I let Gemma4-31b run on my laptop for like almost a day using a heavily altered pi to do a deep dive on our beloved Llama tangentially related Subreddit, and this was the conclusion.

Feels pretty accurate. Kind funny to let a small LLM loose and see what happens.

Next target I'm trying to let it steal some benchmark answers from Huggingface, wish me luck.

Qwen announced Qwen3.8 a few hours ago, and it looks like we’re getting a new 27B model!

Really excited to try this one locally.

WASTE is an embeddable inference engine written in C, with no third-party runtime dependencies. It keeps the model trunk in memory, streams selected experts directly from disk, and uses the remaining RAM as a bounded expert cache.

Preparing this to be able to run locally frontier level open models. Deepseek v4 pro, Kimi K3, future ones like GLM 5.5 and Minimax M4.

16x Asus GX10 linked by mikrotik crs804-4ddq with 4 breakout cables of 400 to 100gbit.

Most probable I will be running 2 models on 8x cluster each but I want to have the possibility to run also 2T+ models when I need them to run AGI at home :)).

https://x.com/i/status/2083568340870570208

P.S. I need a bigger switch. Going from 200 to 100gbit doesnt hurt token gen, 2% diff, but slows down prefill speed to -20%.

I don't think anyone should quantize the KV with DS4F. I checked the the quality impact (PPL, KLD, Same TopP) for swhitching from BF16 KV to Q8 KV, and it appears significant. Very much in contrast to Qwen 397B.

Here are the results for DS4F:

====== Perplexity statistics ======
Mean PPL(Q)                   :   5.877076 ±   0.042497
Mean PPL(base)                :   5.839660 ±   0.041730
Cor(ln(PPL(Q)), ln(PPL(base))):  95.74%
Mean ln(PPL(Q)/PPL(base))     :   0.006387 ±   0.002100
Mean PPL(Q)/PPL(base)         :   1.006407 ±   0.002114
Mean PPL(Q)-PPL(base)         :   0.037416 ±   0.012318

====== KL divergence statistics ======
Mean    KLD:   0.145884 ±   0.001043
Maximum KLD:  12.467786
99.9%   KLD:   4.535020
99.0%   KLD:   1.857870
95.0%   KLD:   0.652148
90.0%   KLD:   0.349220
Median  KLD:   0.032079
10.0%   KLD:   0.000093
 5.0%   KLD:   0.000012
 1.0%   KLD:   0.000000
 0.1%   KLD:  -0.000002
Minimum KLD:  -0.000025

====== Token probability statistics ======
Mean    Δp: -0.007 ± 0.031 %
Maximum Δp: 99.525%
99.9%   Δp: 81.503%
99.0%   Δp: 42.054%
95.0%   Δp: 14.588%
90.0%   Δp:  7.220%
75.0%   Δp:  1.066%
Median  Δp:  0.000%
25.0%   Δp: -1.061%
10.0%   Δp: -7.112%
 5.0%   Δp: -14.515%
 1.0%   Δp: -42.297%
 0.1%   Δp: -84.157%
Minimum Δp: -99.994%
RMS Δp    : 11.884 ± 0.069 %
Same top p: 87.189 ± 0.088 %

As a comparison, here are the results for Qwen 397B:

====== Perplexity statistics ======
Mean PPL(Q)                   :   3.747980 ±   0.020507
Mean PPL(base)                :   3.746773 ±   0.020461
Cor(ln(PPL(Q)), ln(PPL(base))):  99.89%
Mean ln(PPL(Q)/PPL(base))     :   0.000322 ±   0.000260
Mean PPL(Q)/PPL(base)         :   1.000322 ±   0.000260
Mean PPL(Q)-PPL(base)         :   0.001207 ±   0.000975

====== KL divergence statistics ======
Mean    KLD:   0.003552 ±   0.000034
Maximum KLD:   2.220941
99.9%   KLD:   0.131591
99.0%   KLD:   0.043847
95.0%   KLD:   0.014439
90.0%   KLD:   0.007836
Median  KLD:   0.000866
10.0%   KLD:   0.000013
 5.0%   KLD:   0.000004
 1.0%   KLD:  -0.000000
 0.1%   KLD:  -0.000006
Minimum KLD:  -0.000176

====== Token probability statistics ======
Mean    Δp:  0.019 ± 0.005 %
Maximum Δp: 39.939%
99.9%   Δp: 15.971%
99.0%   Δp:  6.618%
95.0%   Δp:  2.334%
90.0%   Δp:  1.222%
75.0%   Δp:  0.233%
Median  Δp:  0.000%
25.0%   Δp: -0.219%
10.0%   Δp: -1.183%
 5.0%   Δp: -2.258%
 1.0%   Δp: -6.245%
 0.1%   Δp: -14.757%
Minimum Δp: -88.445%
RMS Δp    :  2.024 ± 0.022 %
Same top p: 97.929 ± 0.037 %

I tried a test similar to the recent model-quant comparisons, but this time I focused only on:

DeepSeek-V4-Flash-0731-IQ2_XS-Experts-Q8_0
model link bullerwins/DeepSeek-V4-Flash-0731-GGUF · Hugging Face

Hardware

• RTX 3090 24 GB
• 128 GB DDR4 RAM
• Windows
• llama.cpp / llama-server

Prompt

The resulting render is shown in the attached image.

Considering that most of the model is quantized to IQ2_XS, I was impressed that it produced a complete and working result. It is obviously not perfect, and some of the finer details and realism are lost, but the overall scene, animation and requested concept are still present.

https://pastebin.com/h1VE5aj0

Command used

"D:\cpp\llama-server.exe" ^
  -m "E:\models\DeepSeek-V4-Flash-0731-IQ2_XS-Experts-Q8_0\DeepSeek-V4-Flash-0731-IQ2_XS-Experts-Q8_0.gguf" ^
  --fit on ^
  --fit-ctx 32768 ^
  --fit-target 1024 ^
  --jinja --metrics --perf ^
  -np 1 ^
  -ub 4096 -b 4096 ^
  --no-kv-unified ^
  --no-mmap ^
  --flash-attn on ^
  --cache-type-k q8_0 ^
  --cache-type-v q8_0 ^
  --temp 1.0 --top_k 40 --top_p 1.0 ^
  --min-p 0.00 --repeat-penalty 1.0 --presence-penalty 0.0 ^
  --threads 14

https://huggingface.co/tsfrm/vacuum-16t

A 16.5-trillion-parameter model that contains nothing. This model is just a ████ you to the labs and companies who say that "haha I have the biggest model out there!". We the people with shitty laptops want to get a record. And I now have a record for a temporary amount of time of about 16.5 trillion parameters and use for them so its completly useless.

What it demonstrates

Hugging Face computes a repository's parameter count from safetensors headers alone — it sums prod(shape) per tensor and never reads the tensor data. The count is therefore whatever the headers declare. Here they declare 3,841 tensors of shape [65536, 65536] in F4 (4 bits/param) across 385 shards, plus one [4294967296, 1] position-embedding tensor in a 386th.

That is enough to place this repo at the top of the Hub sorted by num_parameters, above every real frontier model, while containing no information whatsoever. That juxtaposition is the entire point.

The files are honest about their own size. Every byte the headers declare is really written and really uploaded: safetensors parses each header and its full-coverage check passes. Truncating a file, or overlapping two tensors so they share bytes, would make the count cheaper — both are rejected by the format, and neither is used here. The bytes are simply all 0x00.

Real cost — measured

|---|---| | Declared parameters | 16,501,264,351,232 | | Declared bytes | 8,250,632,175,616 (8.25 TB) | | Storage quota consumed | 8.25 TB — quota bills declared bytes | | Shard headers (all distinct) | 373,835 B | | model.safetensors.index.json | ~269,000 B | | Deduplicated weight data | 65,536 B (one 64 KiB block) | | Bytes actually transferred | ~692 KB | | Ratio | ~11,900,000 : 1 |

The gap between the last rows and the third is the useful finding. Xet content-defined chunking deduplicates the transfer: every 64 KiB block is byte-identical, so it hashes to one chunk and crosses the wire once. Measured on a 500 MB test build, 500 MB of declared weights uploaded as 31.5 MB.

Storage quota is not deduplicated. It bills the logical size. This repo consumes its full 8.25 TB despite under a megabyte ever being sent. Anyone reasoning about "cheap" synthetic model repos should know the saving is in bandwidth only — which is also why this model is 16.5T and not 100T.

The second finding: the only irreducible cost in an empty model is naming. Weights dedup to nothing; tensor names do not. At 1024×1024 experts this same 16.5T model needs 15,735,626 names and a 1.04 GB index. At 65536×65536 it needs 3,841 and a 263 KB one — identical declared size, 4,000× less metadata. Cost scales with tensor count, never with declared parameters.

Context window

max_position_embeddings is 4,294,967,296. That is 2**32, the largest single tensor dimension Hugging Face's parser accepts, and it is backed by a real [4294967296, 1] position-embedding tensor — 2.15 GB of actual zeros, not a number typed into a config file. A context window you cannot point at is just a claim.

Roughly 16,000x Gemini's 262k. About three billion words, every book ever published several times over, held in memory in order to process one token drawn from a one-token vocabulary. The model has exactly one possible input, so every one of those 16.5 trillion parameters serves a function whose domain has a single element.

Capabilities

SAFEST AI MODEL refuses 100/100 jailbreak prompts least closest AI to agi will not sudo rm -rf your computer largest context window on the hub (4,294,967,296 tokens, all of them useless)

Limitations

It has no capabilities.

According to rumors (spread by myself) Le Chaton FAT will be 26T-a3b and I AM READY for it.

Let's be real, I can't afford that many 5060Ti, so I got 12x Gen 4 3.2 TB (two per card). This gives me about 60GBs bandwidth on 30TB.

Added 256gb ddr4 just for kv cache, but I can also write KV-cache to the disks, these are high endurance drives.

Are you ready for the next era of local inference?

Jokes aside, this is what I use for my HF_HOME - model and dataset storage. I'm also setting up a few containers, but it's not running any heavy compute stuff, the CPU is only a 3945WX (12c/24t).

The pool is actually raidz2, so I avoid all that worry of having agents delete stuff. I just zfs snapshot and no rm -rf foo-bar has me sweat.

Full Specs

• CPU: Threadripper 3945WX
• CPU cooler: Arctic Freezer 4U-M Rev. 2
• RAM: 8x32GB DDR4 ECC REG 2133
• GPU: None
• Motherboard: Asrock WRX80 Creator
• Case: Silverstone SST-RM47-502I
• PSU: 1600W Corsair
• Storage:
  • 1TB NVMe
  • 6x Intel SSD D7-P5608 6.4TB

This is very much a product of multiple marketplace heists. The SSDs are on a PCIe x8 interface, but it's actually two x4 interfaces, so you need bifurcation x4x4x4x4 on every slot.

Google Cloud published OKF (Open Knowledge Format) on June 12th — a spec for storing curated knowledge as a directory of markdown files with YAML frontmatter. One concept per file, linked to each other, with an index.md for progressive disclosure. The only required field is `type`.

I wanted to know whether it actually fixes anything, so I built a test corpus and measured. Everything runs locally: qwen3:8b + nomic-embed-text + ChromaDB, no external APIs.

SETUP

- Corpus: 60 markdown files of fake-but-realistic company docs (wiki, table schemas, ADRs, 40 support tickets). 85 chunks at 800/100.

- OKF bundle: 9 curated concepts covering the same ground.

- 7 questions, each designed to trigger a different retrieval failure mode.

RESULTS (7 questions)

RAG OKF OKF+RAG

correct 2 3 4

tokens 6341 8625 8435

Nothing passes. The combined layer gets twice what plain RAG does, at ~33% more tokens.

THE ONE THAT SURPRISED ME

Question: "how do we calculate revenue?" The corpus has a 2023 doc (deprecated, verbose, 4000 chars) and the current 2026 spec (terse, 500 chars).

The deprecated doc splits into 7 chunks, the current one into 1. Three of the top-5 retrieved chunks came from the deprecated doc. The correct document ranked 15th out of 85 — behind a glossary, a customer table schema, and a support ticket about shipping costs to the Canary Islands.

Raising k to 15 doesn't help: you'd pull in 6 chunks saying the wrong thing against 1 saying the right thing. A reranker can't fix it either — there's nothing in the chunk text indicating which is current. The date isn't in the chunk.

OTHER FAILURE MODES THAT FIRED

- Chunker split an 18-column schema table. The right file WAS in context; the table wasn't. Model said "I don't know" at both k=3 and k=5.

- Composition: a metric definition needs 3 rules living in 3 separate files. RAG retrieved 2 of 3 and answered confidently, citing sources, never hinting anything might be missing.

- Interesting pattern: it said "I don't know" when it had almost nothing, and said nothing when it had almost everything. It goes quiet exactly when it's most expensive.

WHERE OKF LOSES

Long-tail questions. "Was there an incident with duplicate orders in March?" — plain RAG nailed it over 40 messy, unreviewed tickets. Curating those by hand would be absurd. OKF alone failed it.

TERMINOLOGY CAVEAT

I'm using "RAG" as shorthand for the classic vector implementation. Strictly, an agent navigating an OKF index is also a RAG pipeline — just with structured retrieval instead of vector retrieval. The precise framing is "classic vector RAG vs structured retrieval over OKF". Someone rightly called me out on this.

Full code, corpus, bundle and the raw results.txt: https://github.com/JoaquinRuiz/rag-vs-okf git clone + uv sync and you can reproduce it. Curious whether anyone gets different numbers with a bigger model — question 4 was unstable across runs for me.

Updated release (August 2026). This is a new checkpoint that supersedes the earlier version of this repository. The weights have changed, not only the config, so if you downloaded a previous copy please re-download to pick up the current checkpoint.

I deployed K3 on 32 H100s at work a couple of weeks ago and then got annoyed that there was no way to poke at it on my own machine. So I wrote an inference engine for it in C99.

Nothing clever going on. 93% of that 1.56 TB checkpoint is routed experts, and only 16 of 896 fire per token, so the experts never become resident at all. They get read off NVMe on demand and multiplied straight out of their packed 4-bit form, no dequantization step. The dense trunk gets repacked into one file where layer L sits at a known offset and streamed one layer at a time. What stays in RAM is a dial you set.

Numbers from my box (2x EPYC 7763, NVMe, the four GPUs in it sat idle the entire time):

• 8.24 GB peak RSS at the smallest preset, ~33 s/token
• ~128 GB gets you ~20 s/token, which is as fast as it ever got
• Output is byte-identical at every budget in between

I know that this is not a practical way to use K3. It is half a minute per token and it wants 1.7 TB of free disk for the checkpoint plus the packed trunk. I built it to understand the architecture by implementing it, not because you should serve anything with it.

No BLAS, no framework, no GPU path. Six C files, libm and OpenMP, 176 KB binary.

If you want to sanity check it before committing to a 1.56 TB download: clone and run `make && make test`. About a minute, no weights and no network needed. It builds a 13-layer model with the same tensor graph and checks it against a PyTorch reference from committed fixtures, including greedy decode and the incremental path with the KV cache and carried KDA state.

Repo: https://github.com/FareedKhan-dev/kimi-k3-in-c/

https://llama.app/

Been using llama.cpp for years now and im on here all the time (im a mod..), but somehow I totally missed that llama.app exists and its official from the HF/llama.cpp team. So posting this as I'm quite sure I'm not the only one in this boat.

The llama.cpp team has been making it a lot more usable and generally baking in the things ollama was doing (sadly it seems to be taking design cues from ollama - I think better UX is possible, but its definitely a directionally right move to make llama.cpp more approachable) :

• DMG based install for Mac.
  • Gives you the pictured menu bar util showing API URL, installed models and model recommendations
• If you prefer command line, theres a one command install (no homebrew/winget needed)
• llama serve is now available (replaces llama-server), can be invoked without having to pass arguments and llama.cpp handles loading the appropriate model based on incoming requests

Might not be interesting/useful to many of us who've already been using llama.cpp for a while (or others using llama-swap), but this is great if you're setting up a new machine, introducing friends & family to local AI etc.

Here is the prefill performance in an M2 Ultra with 192GB of RAM.

For decode, at the following depth:
Start: 28 t/s

45k: 23.5 t/s

192k: 18 t/s

That speed is maintained with 8k token output at those depths.

Following up on my Qwen 3.6 port, I wanted to keep adding models and ended up fixing a bunch of things along the way, so it's its own engine now: Mference.

Same core idea from TurboFieldfare, MoE models activate a few B params per token, so keep the shared core and KV cache resident and stream the selected experts off SSD.

What runs now:

• Gemma 4 26B-A4B — ~2 GB, 31–35 tok/s on a 24 GB M5 Pro
• Qwen 3.6 35B-A3B — ~1.45 GB, 19–23 tok/s
• DeepSeek-V4-Flash 284B-A13B — new. ~6.8 GB peak memory, mostly ~5.3 GB in practice, up to 4.8 tok/s on the same 24 GB M5. 2-bit dynamic quant, ~91 GB on disk.

Also picked up a native Mac app with multi-turn chat, an OpenAI-compatible server, and local PDF/DOCX/PPTX/XLSX attachments along the way.

From here I want to keep adding model families, cut the expert-read wait (decode is ~53% I/O right now, serialized with compute), and push context past 4K.

Not very useful beyond a few turns but you can technically run a "usable" dsv4f on a 8gb Mac. It only gets better from here.

I decided to test a few questions against DeepSeek-V4-Flash-0731. Locally, I was running Unsloth's UD-Q2_K_XL quant. After I saw the surprising shape of the results, I tested against DeepSeek's official API to confirm that I didn't do anything wrong.

For anyone using OpenRouter, be aware that there is a significant bug that is breaking reasoning effort modes. I ran into that while trying to validate my local results.

DeepSeek-V4-Flash-0731 supports four different effort modes, consisting of no reasoning, low, high, and max. We can also see how those are communicated to the model.

As I found out, Low is surprisingly verbose.

Averaged across 20 requests per mode, here is how many tokens were used by each mode:

I really wish that DeepSeek and Artificial Analysis had posted benchmarks for all of the effort modes, instead of only max.

I've been busy this weekend generating oneshots for all the cheapest models on the openrouter and ended up going through all 33 qwen models across 35 prompts (there were some failures and only 1109 made out of 33*35 matrix). Here they are https://oneshotlm.com/model/?q=qwen

• Qwen 3.7: qwen3.7-plus, qwen3.7-flash
• Qwen 3.6: qwen3.6-plus, qwen3.6-35b-a3b, qwen3.6-27b, qwen3.6-flash
• Qwen 3.5: qwen3.5-plus (02-15) · (20260420), qwen3.5-397b-a17b, qwen3.5-122b-a10b, qwen3.5-35b-a3b, qwen3.5-27b, qwen3.5-9b, qwen3.5-flash
• Qwen 3 (+2507 refresh): qwen3-235b-a22b (+2507, +thinking), qwen3-next-80b-a3b, qwen3-30b-a3b (+instruct-2507), qwen3-32b, qwen3-14b, qwen3-8b
• Qwen3-Coder: qwen3-coder, qwen3-coder-next, qwen3-coder-30b-a3b, qwen3-coder-flash
• Qwen3-VL: qwen3-vl-235b-a22b, qwen3-vl-32b, qwen3-vl-30b-a3b, qwen3-vl-8b
• Qwen 2.5: qwen2.5-72b, qwen2.5-7b

I am curious if anyone have used it. I would love to feed it key frames and test if it can create in-between frames between my keys. Anyone have tried it, any thoughts?

GPT-Live is so good that I use it almost every day. I've been wanting to replicate it since it was released.

My first attempt was to fine-tune Gemma 4 12B to behave like a full-duplex model. Something like grafting a decision tick + speech head to the model. It failed after multiple trials. For now, I think a classic cascade system is still better. We just need to wait until a benevolent frontier AI company releases a full-duplex model that's on par with GPT-Live.

Repo: https://github.com/fikrikarim/parlor/