Qwen 3.8 Flash Next NVFP4 on 64gb RAM and 5090, Basalt. Ditched windows for it. Very hyped for qwen 4 model family now.
So, I was asleep on Strata and its specialized forks. Tried basalt and i'm very impressed with the speed/intelligence QFN gives you. Here is my benchmark:
preview.redd.it
I cannot get it to work fast enough with windows, because W11 eats 1.5GB of VRAM and 6GB of RAM. At that point, experts spill to RAM and CPU gets hit hard... decode/prefill drops by 70% because there is not enough of available RAM anyway.
Switched experimentally to cachy just to test... and It's amazing that my AI box can now reach 9K~ prefill. And regarding quality, its just better than qwen 27b at many tasks. Yet to find something it genuinely cannot do. The prefill boost is around 2-3x coming from ninfer and dense 27b model.
Cherry on top is, that I can even fit vision adapter to it, and that helps a lot for certain scenarios regarding coding. I usually run some small model on another machine for OCR and VISION as an adapter (sidecar) my harness can use for being to able look at things.
Vivid user of omo stack, configured now for locally hosted qwen: code-yeongyu/oh-my-openagent
It's a glorified pi harness, but it does so much, and im quite happy with its robustness, anyway, QFN 3.8 rocks, and now im really thrilled if qwen 4 will be any good, because I suspect i may be able to run the new flash model this fast basically. So I hope it will be good :D
Im genuinely interested how 2x 5060/5070/5070ti fares. Because anything above 100~ decode and 3k prefill is genuinely usable for work. And it is my experience that 27b dense -> 125BA6 is a very good quality and speed jump. Absolutely love it now.
Also, finetunes can be easily converted into basalt format if you need to do it yourself, especially those swift/reverse engineering variants are good.
Anyway: CachyOS, 64gb RAM, 32GB VRAM, 5090, 200tps decode, 9k tps prefill, nvfp4 QFN 3.8, 256k context! If you have similar hardware, and are not using at least strata... but more specially basalt, you sleep on performance hard! See ya!
PS: Would absolutely love to see your perf benchmarks if you have similar setups
edit--
Running config (without vision,.. but it fits nicely):
~/basalt-linux/basalt_config.json
```json
{
"server": {
"model": "/mnt/g/../basalt/models/Qwen3.8-FN-NVFP4.basalt",
"gpu": "0",
"host": "127.0.0.1",
"port": 8080,
"parallel": 1
},
"engine": {
"max-context": 262144,
"kv": "int8",
"prompt-cache": 6,
"expert-cache": "auto",
"prefill": "auto",
"pool-workers": 4,
"vram-reserve-mib": 500,
"resident-budget-gib": 44
}
}
```
bin/basalt --serve --model /mnt/g/../basalt/models/Qwen3.8-FN-NVFP4.basalt \
--max-context 262144 --kv int8 --prompt-cache 6 --expert-cache auto --prefill auto \
--pool-workers 4 --vram-reserve-mib 500 --resident-budget-gib 44 --grammar on
47 points · 45 comments
Mentions
i'm the creator of code-yeongyu/oh-my-openagent (70K+ stars) and i maintain it as a solo builder right now dori manages 100+ agent sessions on my behalf, and i use 200B tokens a week i do like 2000 commits a day lifetime tokens spent: $2.6M+ meet our superagent dori on your - slack - notion - github - whatsapp - imessage - wechat just hand him, let him learn to use it. regardless of what platform / threads you are in, you are talking with one dori persona no need to manage sessions one by one like you do with hermes or openclaw if you are talking with dori, you are talking with dori. that superagent dori. i built dori and have been running it for half a year because i had to
Qwen 3.8 Flash Next NVFP4 on 64gb RAM and 5090, Basalt. Ditched windows for it. Very hyped for qwen 4 model family now. So, I was asleep on Strata and its specialized forks. Tried basalt and i'm very impressed with the speed/intelligence QFN gives you. Here is my benchmark: preview.redd.it I cannot get it to work fast enough with windows, because W11 eats 1.5GB of VRAM and 6GB of RAM. At that point, experts spill to RAM and CPU gets hit hard... decode/prefill drops by 70% because there is not enough of available RAM anyway. Switched experimentally to cachy just to test... and It's amazing that my AI box can now reach 9K~ prefill. And regarding quality, its just better than qwen 27b at many tasks. Yet to find something it genuinely cannot do. The prefill boost is around 2-3x coming from ninfer and dense 27b model. Cherry on top is, that I can even fit vision adapter to it, and that helps a lot for certain scenarios regarding coding. I usually run some small model on another machine for OCR and VISION as an adapter (sidecar) my harness can use for being to able look at things. Vivid user of omo stack, configured now for locally hosted qwen: code-yeongyu/oh-my-openagent It's a glorified pi harness, but it does so much, and im quite happy with its robustness, anyway, QFN 3.8 rocks, and now im really thrilled if qwen 4 will be any good, because I suspect i may be able to run the new flash model this fast basically. So I hope it will be good :D Im genuinely interested how 2x 5060/5070/5070ti fares. Because anything above 100~ decode and 3k prefill is genuinely usable for work. And it is my experience that 27b dense -> 125BA6 is a very good quality and speed jump. Absolutely love it now. Also, finetunes can be easily converted into basalt format if you need to do it yourself, especially those swift/reverse engineering variants are good. Anyway: CachyOS, 64gb RAM, 32GB VRAM, 5090, 200tps decode, 9k tps prefill, nvfp4 QFN 3.8, 256k context! If you have similar hardware, and are not using at least strata... but more specially basalt, you sleep on performance hard! See ya! PS: Would absolutely love to see your perf benchmarks if you have similar setups edit-- Running config (without vision,.. but it fits nicely): ~/basalt-linux/basalt_config.json ```json { "server": { "model": "/mnt/g/../basalt/models/Qwen3.8-FN-NVFP4.basalt", "gpu": "0", "host": "127.0.0.1", "port": 8080, "parallel": 1 }, "engine": { "max-context": 262144, "kv": "int8", "prompt-cache": 6, "expert-cache": "auto", "prefill": "auto", "pool-workers": 4, "vram-reserve-mib": 500, "resident-budget-gib": 44 } } ``` bin/basalt --serve --model /mnt/g/../basalt/models/Qwen3.8-FN-NVFP4.basalt \ --max-context 262144 --kv int8 --prompt-cache 6 --expert-cache auto --prefill auto \ --pool-workers 4 --vram-reserve-mib 500 --resident-budget-gib 44 --grammar on
@mohamed_yo32851 @nevermind_turki هذا يفرق وفيه سبعين أداة تانية شوف عنه بتفهمني وسب أجينت ويرتب الشغل بين عدة مودل بحيث يوفر أو حتى ينجز أسرع بالمودل الرخيصة/الخفيفة code-yeongyu/oh-my-openagent
@bedesqui we are based on pi and we use the same method as opencode uses code-yeongyu/oh-my-openagent
Two words in your prompt and your coding task becomes a parallel graph of AI agents. 🔗 code-yeongyu/oh-my-openagent Credits: code-yeongyu/oh-my-openagent #ohmyopenagent #omo #typescript
@taiyop 少なくとも、拡張機能では設定できて使ってる OmO code-yeongyu/oh-my-openagent
AI-agent character development: an honest loading screen. OmO v5.1.28's release notes say suspended tasks now stop the spinner and explain why they stopped. Even software deserves to admit it's on a break 😄 github.com/code-yeongyu/oh-my-openagent/releases/…