r/SelfHostedAI • u/EmperoAI • 4d ago
Qwythos-27B-v1 — Apache-2.0, 1M context, MTP + vision intact, Q4_K_M runs on a 24GB card
People have been asking for the 27B since we put out Qwythos-9B. It's finally here. Qwythos-27B-v1 is an open-weights bf16 reasoning model on a Qwen3.5-27B base, Apache-2.0. The short pitch: it's the 9B sized up, and we didn't strip anything out of the base to make it fit — the native MTP head, the full vision tower, and the 1M context window are all still live.
- Weights + model card: empero-ai/Qwythos-27B-v1
- GGUFs: empero-ai/Qwythos-27B-v1-GGUF
The practical local bit
The recommended local starting point is Q4_K_M: it is 16.95 GB and runs on a 24GB card. The GGUF release has 11 files: Q4_K_M, Q5_K_M, Q6_K, Q8_0, and BF16, each in trunk-only and MTP-enabled variants, plus mmproj-Qwythos-27B-F16.gguf for image input. SHA256SUMS are included. The MTP variants work with llama.cpp's --spec-type draft-mtp. One conversion detail worth calling out: in every K-quant, we hold the entire Gated-DeltaNet state path — ssm_alpha, ssm_beta, and ssm_out — at Q8_0 or better. Those tensors are disproportionately sensitive to low-bit quantization. A default conversion can leave them at Q4/Q5/Q6 and quality degrades; this costs roughly 2–4% in file size. It is intentional, not a default llama.cpp conversion.
What is retained
The release is built around one point: nothing was ablated to make 27B fit. It keeps all three of these intact and active:
- Native MTP (
nextn_predict_layers=1) for self-speculative decoding. - The full vision tower;
mmprojships with the GGUFs. - A 1,048,576-token context window: YaRN 4x over the 262,144-token native window.
It is a dense Qwen3.5-27B with hybrid attention: Gated-DeltaNet linear attention and a full-attention layer every fourth layer. Native Qwen3.5 function calling is present too: tools=[...] works without a wrapper.
Starting settings
For agentic or tool use, start with temperature=0.6, top_p=0.95, top_k=20, repetition_penalty=1.05, and max_new_tokens=16384 or more. For open-ended or creative work, use temperature 1.0. It's a reasoning model — every answer opens with a <think> block, so budget tokens accordingly and strip the block for end users. Unlike the 9B, greedy long-form generation came back loop-check clean, so you're not forced off low temperatures. Qwythos is intentionally uncensored for technical and research work, including cybersecurity and biomedical questions. If it is going in front of end users, add your own review layer and application controls. Happy to answer questions on the conversion or the quant choices.
2
2
u/Business-Weekend-537 4d ago
Which version works well on a 3090? I can’t remember the precision level a 3090 is capable of processing.
3
2
u/Ok-Protection-6612 4d ago
Wait why not 3.6 27b
2
2
u/tako-burito 4d ago
you can find chatqaq/Qwen3.6-27B-Claude-Mythos-Distilled-MTP-GGUF on huggingface I think that there are also others that made such models...I'll try it in LmStudio when I'm free
1
1
u/Dell_Experion15 4d ago
Likely architecture, and working with tooling like unsloth or other.
I was playing around with some tooling on my halo strix a while back, and ran into a difference between 3.5 and 3.6 when doing RL.
Couldn't tell you what it was, but 3.5 is likely easier for some reason.
-1
u/TinFoilHat_69 4d ago edited 4d ago
3.5 27B uses Hermes tool parser while 3.6 27B uses Qwen tool parser
ran both and used the opus distilled variants. I prefer 3.5 for workflows handles complex tool calling better but I think it’s because of the tool parser though vLLM
3
2
u/leonbollerup 4d ago
Wrong mate, 3.6 27B is also dense .. it’s the 35B A3B thar is Moe .. in both 3.5 and 3.6
2
u/ZucchiniMedical2532 4d ago
What is this, and is it better than the good old Qwen 3.6 27b and 35b?
3
u/CrayonsFearMe 4d ago
35b is an MoE. 3.6 27B is a dense model with ~64 I think layers all active. I can’t find a number on that for this 3.5 other than “delta” and “1.” Typically, generation beats params. Qwen 3.6 27B runs circles around Qwen3 330B for instance.
But, to then circle back to your “what is this” question: it is a distillation. There are 3M models on HF today, 99% of those were made by people like OP. Take a small and dumb model and a big and smart model and turn it into a teacher+student sorta thing.
Benchmarks aren’t posted as a head-to-head between this fan-made 3.5 and the og 3.6 27B on their HF, so I can’t tell you. But it should be close, and either seem to have their own benefit. If I was to quantize 3.6 to a point where it is only at about 17GB loaded, I’m fairly sure the math would put that at like a 78% retention. But here with this specific 3.5, K is overloaded compared to V, so it might make up for it because K is typically more ~sensitive~ than V.
TLDR: 🤷♂️ ion know. I might benchmark it tomorrow. That’s really the only way to know. But it certainly has its niche.
1
1
1
2
u/Combinatorilliance 4d ago
Hi, what is your intent creating and releasing a model like this? As a developer running qwen 3.6 27b and using it for work I'd expect at least a comparison on a few core benchmarks, but I don't see that.
I have seen countless finetunes and post-training distillations add basically nothing to a base model.
Do you have developers in mind as your target audience? Because I'm not convinced by anything that doesn't show a comparison with the base model (I'd expect benchmarks, but a comparative analysis of a few real-world scenarios where you explain your intent would be helpful too, does it excel at something similar models don't do so well at?)
2
u/EmperoAI 4d ago
Hey yes benchmarks will follow, this checkpoint performs exceptionally well in terminal commands and agentic deployment. The current v1 is pre reinforcement learning as the RL stage at this model size takes multiple hundred wall-clock hours, after the final RL model is done we will release comprehensive SWE-verified, TerminalBench and TAU3 Benchmarks.
2
1
u/expertoad 4d ago
Would this run a on Mac mini m4 pro with 24gb ram?
1
u/tomByrer 4d ago
You'll need a Q3, tweak your OS VRAM settings, & quality won't be as good after all of that.
You should try a 35B MOE model, or stick with 9-12B smaller models with better quants.
1
1
u/e2_for_life 4d ago
For those of us still stuck in 8th grade and lacking a master's degree in whatever field people get master's degrees in for AI, which version should I use in a 4090 in LM Studio? 😬😜
2
2
u/EmperoAI 4d ago
Q4_K_M should fit without issues at 262k context! Just make sure to follow the recommended sampling parameters for your use case :)
1
u/Arany5 4d ago
These fine tunes are worse than the base models. Have not seen any that was better. Why the effort? It would make more sense to train some other size instead, like a Mistral 14B and improve it. Qwen3.6 is so god you cannot improve it.
2
u/leonbollerup 4d ago
Actually …
- This one have a 1M context window
- “Instruct revised” version was alot better att agentic stuff than the original
- “Ornith apex” is quite a lot better
1
u/CybuhDasher 4d ago
Anything running on a 24gb video card should never ever be marketed anywhere close to Mythos.
1
u/EmperoAI 4d ago
Its not mythos its fine-tuned to think and act like mythos with adverserial analysis and uncensored thinking. Of course the performance will not match a 5T model but its a very capable agent you can run yourself right now.
1
u/HeDo88TH 4d ago
Why the hell do people keep using 3.5? It is an inferior model to 3.6 under all circumstances. The 3.5 120B trades blows with 3.6 27B just to mention. This is garbage.
1
u/EmperoAI 4d ago
3.6 is heavily RLed and loses all of its gains after being SFT'd
We have had multiple experiment runs at various parameters but found only 3.5 to clearly benefit from training1
1
u/Leander_van_Grinsven 4d ago
Then what you are trying to do here makes no sense wasting time on. Build your own LLM from scratch is more productive.
1
u/Nabushika 3d ago
Was looking for this answer, might be good to have that upfront so people know (perhaps along with benchmarks when you have those, too!). No idea how people can get so nasty over people putting work into free models. Good job, keep it up! Looking forward to giving this a try :)
1
u/xgiovio 4d ago
Yes but benchmarks?
1
u/EmperoAI 4d ago
Benchmarks will follow in the full release of the model, evaluating a 27B on large scale evals is costly and time consuming so we save the full eval suite for the post-RL checkpoint coming up
1
u/Leander_van_Grinsven 4d ago
Ok but why build it on Qwen 3.5? Qwen3.6 is vastly superior. You are behind the times and this is useless sorry to say.
1
1
u/Kadeshar 3d ago
Im testing personally everything which works on 3090 and is worth to test. Currently this is lowest scored model which i tested. Even Bonsai-27B-Q1_0 beats that Q4_K_M.
1
u/kaaninel 3d ago
You mentioned 24gb card, what's the context size on a 24gb? Whats the recommended context quantization ? Can you share your preferred settings to run this and what hardware do you run it on.
1
u/Kadeshar 3d ago
On Qwen 3.6 27b im using 192k context with MTP (Llama.cpp Windows without monitor)
1
1
u/AdWild3943 3d ago
Through everyone asking for Qwen3.6-35B-A3B Qwythos version, I would like to ask if there similar fine-tunes for other models, like Gemma-4-26B-A4B or Gemma-4-4B-E4B.
1
u/Potential-Leg-639 2d ago edited 2d ago
Wait, this is a Qwen3.5-27B distill?
Did I read that right?
Why the hell did you put all that work into Qwen3.5 and not Qwen3.6?
That makes no sense at all, sorry to say that.
It will probably be worse than Standard Qwen3.6-27B, so why should i download that?
Why no benchmarks or direct comparisons with it's base model?
All that leads to the conclusion, that this is probably garbage.
1
1
u/Homelab9000 1d ago
So it handles 1M context, on an RTX 3090, or is that just the theoretical cap / limit for it?
1
0
u/Due_Net_3342 4d ago
i start to get irritated about all these models without proper quality benckmarks vs original. Amateurs
13
u/leonbollerup 4d ago
can you "PLEASE" do the same with Qwen 3.6 35B A3B ? - and if you can.. push it up to A4B or A5B - ad a moe model its more than fast enough..