sergqwer

Infernix - New Release! Over the past few weeks, I've been working on various improvements to my inference engine Infernix - specifically optimised for running the latest Qwen architecture models on an RTX5090 on Windows and Linux. Windows is a first-class supported use case - you can just download everything you need to run on windows here: github.com/Wallawalla47/Infernix/releases/tag/win… Key improvements: 1. New models: - the new NVIDIA QAD FP4 version of Qwen3.8-Flash-Next (which operates faster than the NVFP4 model and can be switched to use a4 activations for even faster performance with --a4-activations): huggingface.co - the same GSQ-RCO-IQ3_S model supported by Strata (which offers some faster speeds than NVFP4 but I find at some quality cost): huggingface.co - Swift 1.5 Qwen3.8-Flash-Next NVFP4: huggingface.co - Qwen3.8-Flash-Next Full FP8 (fairly slow, but offers the best quality being bit-exact to the official FP8 model from Qwen): huggingface.co - the new Ornith 1.5 122B NVFP4 model, which has impressive benchmark performance, but I'm not sure I'm sold on it yet: huggingface.co (and the uncensored version as well: huggingface.co ) 2. DFlash2 drafting for Qwen3.8-Flash-Next models (use --spec dflash2 --draft-tokens auto --draft-model [Draft Model File Path - e.g. D:\Infernix\Qwen3.8-Flash-Next-DFlash2-Infernix.infernix]): huggingface.co 3. Added a new WebUI Launcher (beta), which provides an interface to download and configure models and a model switching API, so it functions similar to launchers like LMStudio or Unsloth and it also has a statistics dashboard, logging and a quick test chat interface (still early stages, so interface is quite busy and needs some refining, but is probably the easiest way to configure and use Infernix): on Windows just launch infernix-launcher.exe 4. Added new auto-configuration of speculative decoding which uses calibrated settings to give the best decode tok/sec performance across a range of different types of text - simply use: --spec auto --draft-tokens auto --draft-model [Draft Model File Path] 5. Lots of speed and quality optimisations - generally outperforms Strata and occasionally outperforms Basalt, though it does run some stages at slightly higher precision than Basalt. I've sought to avoid some of the quality sacrificing optimisations like expert tail skipping or running the model at lower quantisation than recommended for the model. You can enable some of these options if you like, such as expert tail skipping with --expert-tail-skip [Int] (7 is sometimes recommended as a default) or using 8-bit code paths (rather than the default 16-bit/4-bit recommended for the model) with: --a8-activations 6. Adding a new --lossless mode that is always perfectly reproducible and uses bit-exact in-layer calculations to the precision the model was trained on (usually BF16) with FP32 logits. This is helpful as a reference for quality testing and I'm working on some more robust quality gating to ensure all optimisations and code paths do not impact quality in the same way as other engines often do. Some key performance figures: |Infernix vs|First token|Decode|Replay: mean first token|Replay: wall time| |:-|:-|:-|:-|:-| |Qwen3.8-Flash-Next, NVIDIA's NVFP4 weights||||| |Strata v0.2.8|1.4-1.6× sooner|+10-15 % to 128K, −4 % at 250K|3.1 s vs 12.3 s|5.7 vs 12.9 min| |strata-nvfp4 0.2.0-beta-nvfp4.3|1.3× sooner|level to +3 %|3.1 s vs 10.8 s|5.7 vs 11.7 min| |Basalt c08284cb|5-8× sooner|Basalt 7-23 % faster|3.1 s vs 7.2 s|5.7 vs 8.9 min| |Qwen3.8-Flash-Next, ISTA-DASLab's IQ3_S GGUF||||| |Strata v0.2.8|15 % sooner at 8K, level from 128K|+3-12 %|3.1 s vs 6.9 s|5.5 vs 8.1 min| |Basalt c08284cb (Basalt is faster on this model but uses some lower precision math)|Basalt 4 % sooner at 8K, 1.3× from 128K|Basalt 1.4-1.7× faster|3.1 s vs 3.7 s|Basalt sooner: 4.4 vs 5.5 min| |Qwen3.8-27B, the same NVFP4 artifact in both||||| |NInfer 1882c6adb|9 % sooner at 8K, 1.6-1.8× from 128K|+11-17 % (own drafters), +2-3 % without|1.9 s vs 2.5 s|12.7 vs 16.4 min| 1. On the NVFP4 model, Basalt runs NVIDIA's BF16 dense layers at 6 bits (+10 % KL divergence by its own measure) and every activation outside the experts at 8 bits; Infernix keeps 16. Infernix runs activations at 4-bit as recommended by NVIDIA for this model, but with the expert activations at 8 bits as Basalt runs them in decode (--a8-activations), Infernix decodes 199 / 185 / 268 tok/s against Basalt's 202 / 173 / 216 at ~8K / ~128K / ~250K and 175 against 166 at one request in the replay, and still reaches the first token 4-6× sooner. 2. Strata's default route tail skip (it skips some missed experts) was off in all runs: it changes which experts run. 3. On the IQ3_S model, all three hold the GGUF's weights as published, but Strata and Basalt compute every activation at 8 bits, and Basalt's decode reads the BF16 hyper-connection matrices from an 8-bit copy and takes the logits outside the top 64 from a 2-bit copy of the output head; Infernix keeps 16-bit activations, the BF16 matrices and the GGUF's head. At 8 bits (--a8-activations) Infernix reaches the first token as soon as Basalt, though Basalt is still a bit faster. Welcome any feedback via Issues and PRs on GitHub. I'll also look to add support for the new Qwen 4 models ASAP when they drop - hopefully this week!

Tested: Qwen3.8-Flash-Next Q6 vs NVFP4 on my RTX 5090 using Strata — is the Q6 worth the extra memory ? Unsloth Qwen3.8-Flash-Next UD-Q6_K_XL versus Qwen3.8-Flash-Next NVFP4-GPTQ Fresh MMLU-Pro comparison, speed and whether six bits are worth the wait. I wanted to find out whether slower Q6 generation bought better answers. I first compared Q6 with my saved NVFP4 quality results, then ran a fresh, identical MMLU-Pro subset on both models on 23 October. Each new run had a strict 30-minute maximum. The fresh result favoured NVFP4: 71/84 correct versus Q6's 67/84, in about half the total run time. The accuracy gap is too small to establish a broad intelligence ranking, but these results do not support paying the Q6 speed penalty for better reasoning on this workload. My practical choice from this testing is still the NVFP4. NVFP4/GPTQ is the four-bit option in the main comparison. It is a different format from the Unsloth UD-Q4_K_XL. This is a comparison of these two Qwen3.8 Flash Next configurations, not a claim about all four-bit and six-bit models. Fresh knowledge/reasoning test — same day, same 84 questions |Measure|NVFP4/GPTQ|Unsloth UD-Q6_K_XL| |:-|:-|:-| |Correct answers|71/84|67/84| |Accuracy|84.5%|79.8%| |Total benchmark run|10m 57s|22m 18s| |Sum of individual request times|10.51 min|21.88 min| |Median request time|5.98 s|12.61 s| |Generated tokens, including reasoning|93,207|93,754| |Answers reaching the 4,096-token cap|4|4| |Completed / scheduled|84/84|84/84| |Timeouts / skipped / retries|0 / 0 / 0|0 / 0 / 0| NVFP4 led by four answers, or 4.8 percentage points. Q6 took 2.04 times the total benchmark time while generating a very similar number of tokens. Total run time includes preflight and harness bookkeeping, but excludes loading the model. Individual request time includes prompt processing and reasoning. The paired outcomes matter more than just the headline percentages: both models got 66 questions right and 12 wrong. NVFP4 alone got five right; Q6 alone got one right. The exact paired McNemar p-value is 0.219, so I would not call this a statistically established intelligence advantage for NVFP4. It is the observed result of this one fixed sample and single attempt per question. This is an 84-question MMLU-Pro subset with six questions per subject, not the full official benchmark or an IQ score. All 42 MMLU-Pro questions from the earlier comparison were excluded. Selection and settings were frozen before either new run. Fresh MMLU-Pro breakdown — six questions in each subject |Subject|NVFP4|Q6| |:-|:-|:-| |Biology|6/6|4/6| |Business|4/6|3/6| |Chemistry|4/6|4/6| |Computer science|5/6|5/6| |Economics|6/6|6/6| |Engineering|2/6|2/6| |Health|5/6|6/6| |History|4/6|4/6| |Law|5/6|4/6| |Math|6/6|6/6| |Other|6/6|6/6| |Philosophy|6/6|6/6| |Physics|6/6|6/6| |Psychology|6/6|5/6| These subject samples are very small. The ties at 6/6 do not demonstrate that either model has mastered an entire field. Engineering was difficult for both at 2/6. Where NVFP4 sits among everything I have trialled That result also needs the wider context of the models I have been trying. NVFP4/GPTQ is still my current daily driver: it is fast, follows my directions well in everyday use and is pretty smart. After testing the alternatives, it still gives me the balance I want between useful answers and how long I have to wait for them. Here are the chosen best recorded runs, with one row per model/stack. Each cell is the highest previously recorded session median at that context size, not the fastest individual request. Some rows therefore combine sessions. Q6 has one measured campaign. These are historical results across different dates and settings, not a fresh seven-way controlled comparison. Generation speed — tokens/second |Model / stack|RAM MT/s|4K TG|32K TG|128K TG| |:-|:-|:-|:-|:-| |Strata IQ3_XXS|4400|193.8|197.6|192.4| |Strata IQ3_S|6000 / 6000 / 4400|173.5|181.2|183.2| |Strata UD-IQ4_XS|4400|116.3|119.6|120.7| |Strata NVFP4/GPTQ|6000|166.3|161.3|154.1| |Strata Unsloth UD-Q4_K_XL|6000|130.5|131.0|127.6| |Infernix Dense8|6000|133.6|144.0|147.2| |Strata Unsloth UD-Q6_K_XL|6000|65.6|72.8|71.3| Prompt processing — tokens/second |Model / stack|4K PP|32K PP|128K PP| |:-|:-|:-|:-| |Strata IQ3_XXS|4,627.2|6,155.0|5,972.0| |Strata IQ3_S|4,518.5|6,020.7|5,852.6| |Strata UD-IQ4_XS|2,707.0|4,686.1|4,709.4| |Strata NVFP4/GPTQ|3,033.4|8,487.1|8,643.0| |Strata Unsloth UD-Q4_K_XL|2,089.4|6,292.8|6,310.3| |Infernix Dense8|4,125.5|9,399.6|9,050.9| |Strata Unsloth UD-Q6_K_XL|2,094.4|7,534.1|7,634.6| PP comes from the same selected session as each generation-speed cell; it was not independently cherry-picked. The actual prompt lengths were 4,096, 32,768 and 128,000 tokens, with three 256-token generations per context. IQ3_S combines 13 October at 4K/32K (6000 MT/s) with 11 October at 128K (4400 MT/s). NVFP4 combines 14 October at 4K with 15 October at 32K/128K. IQ3_XXS and UD-IQ4_XS are from 11 October, UD-Q4_K_XL from 15 October, Infernix Dense8 from 18 October and Q6 from 23 October. The IQ3 options lead this table on generation speed, and Infernix Dense8 has the strongest long-prompt processing figures. NVFP4 is not the winner in every column. What keeps it as my daily driver is the overall experience: fast generation, strong long-prompt processing, and answers that usually do what I ask. It generates faster than the retained Dense8 and UD-Q4_K_XL runs at all three context sizes, while the fresh Q6 comparison gave me no convincing reason to accept roughly twice the waiting time for reasoning answers. My local workflow also runs through the Hermes harness, with Strata serving NVFP4 on the RTX 5090 and Hermes providing the agent interface around it. The setup includes a WSL gateway, desktop/dashboard, local search through SearXNG and Honcho memory services. NVFP4 is my preferred local backend within that setup; Hermes also has a Codex provider option, so I am not attributing every Hermes response to the local model. That is why I care about more than a headline tokens-per-second figure. An agent workflow can involve several model turns around tool results, retrieved information and follow-up instructions. Waiting longer on each turn can make the whole task feel slower. For me, NVFP4's appeal is being responsive enough to keep that interaction moving while still giving useful answers and following the directions I set. Its extra RAM headroom also leaves more room for the surrounding harness and services. These are practical reasons for my preference, not a measured Hermes task-completion score: the benchmarks here called the model endpoints directly, with background Hermes services suspended for isolation. The fresh 84-question result adds some evidence behind my view that NVFP4 is still pretty smart, but only NVFP4 and Q6 took that test. I cannot turn the speed table into an intelligence ranking of all seven models. Likewise, my everyday preference for NVFP4's instruction following is a personal assessment, not a claim that it won the strict instruction benchmark. The earlier mixed test below is worth keeping alongside the fresh result because it shows where Q6 did offer a benefit. Earlier mixed quality test — retained separately Before the fresh test, I ran Q6 on the same frozen 82-question subset used for my saved 15 October NVFP4 comparison. Those NVFP4 answers were reused; this is a separate historical comparison with different questions. |Earlier benchmark subset|Saved NVFP4|Q6| |:-|:-|:-| |MMLU-Pro|33/42|34/42| |GSM8K maths|20/20|20/20| |HumanEval coding|10/10|10/10| |IFEval strict instruction following|7/10|9/10| Q6 corrected three failures and introduced no new ones on that earlier set. Two gains were exact instruction compliance: meeting a word-frequency requirement and avoiding forbidden words while following a specified response format. The other gain was one economics multiple-choice answer. Its requests took 16.80 minutes versus the archived NVFP4 run's 7.47 minutes. That earlier result suggested a small Q6 benefit; the fresh knowledge/reasoning test went the other way. Both belong in the report. Neither small sample supports a general “Q6 is smarter” claim. The coding tests here are original HumanEval tests on ten tasks, not HumanEval+ or a full coding evaluation. IFEval uses strict prompt-level scoring. Generation speed and prompt processing — historical speed overview |Actual prompt length|NVFP4 generation tok/s|Q6 generation tok/s|NVFP4 PP tok/s|Q6 PP tok/s| |:-|:-|:-|:-|:-| |4,096|166.3|65.6|3,033.4|2,094.4| |32,768|161.3|72.8|8,487.1|7,534.1| |128,000|154.1|71.3|8,643.0|7,634.6| Each speed value is a median of three 256-token generations. NVFP4 retains the best previously published session median per context: 14 October at 4K, 15 October at 32K/128K. PP comes from those same selected sessions. Q6 is one 23 October speed campaign. Historical best selection favours NVFP4; this is not a fresh matched speed rerun. Q6's speed campaign had its control vector enabled. Both quality comparisons explicitly disabled it per request to match NVFP4. Therefore these speed figures are not the exact speed of the quality configuration. Use the fresh benchmark's own timings to assess its waiting-time cost. Memory during the fresh 84-question runs |Observed resource|NVFP4|Q6| |:-|:-|:-| |Minimum sampled free system RAM|43.85 GiB|8.96 GiB| |Peak device-wide VRAM|30.43 GiB|31.24 GiB| |Peak GPU temperature|63°C|61°C| These are whole-system and whole-device observations, not isolated model allocations. Background services were suspended. Page-I/O traces are retained; neither run is claimed to be paging-free. Both completed without a resource-guard stop. The much greater RAM headroom with NVFP4 is another practical advantage on this 128 GB machine, although this was not a long-duration stability test. My assessment: why run four-bit versus Q6? Taken together, these results reinforce why NVFP4 remains my everyday option on this setup. In the fresh test it answered more questions correctly, finished in roughly half the time and left substantially more RAM available. In the earlier small coding and maths subsets it tied Q6. That is a strong practical case for responsiveness and capacity, even though the accuracy difference is not statistically conclusive. Q6 still has a possible role where its answers demonstrably work better for a particular task. The earlier strict instruction-following gains are a concrete reason to consider it for constraint-heavy writing. But two extra passes out of ten are too little evidence to promise fewer mistakes in general. I would choose Q6 for a demonstrated task-specific benefit, not because a higher bit count automatically means a better assistant. The fresh run also shows why I would not justify the slower model by precision alone. Quantization format, engine behaviour, reasoning limits and the task all affect the result. We measured the complete configurations; we did not isolate bit depth. A different sample or reasoning budget could change the scores, but that is untested here. The wider table gives useful context for that choice: IQ3 offers more raw generation speed, Dense8 offers excellent prompt processing, and UD-Q4_K_XL is also substantially faster than Q6. For my daily use, NVFP4 remains the balance I prefer after trying them all. That preference should not be read as proof that the other formats are less capable; their quality was not measured in this fresh comparison. Reproducibility and limits Hardware: RTX 5090 32 GB, 600 W limit; Core Ultra 7 270K Plus; 128 GB DDR5 at 6000 MT/s; Windows 11 Pro. Both fresh runs used NVIDIA driver 610.88 on 23 October. Some older speed measurements used driver 591.86. The fresh suite used deterministic SHA256 selection, six questions per each of 14 subjects, and round-robin subject order. Both models received identical messages and request parameters except model ID. All 84 prompt token counts matched, and every request had zero reused prompt tokens. One attempt each, temperature 1, top_p 0.95, top_k 20, min_p 0, seed 20261023, low reasoning and a 4,096-token cap including reasoning. Control vectors were off. The same published-style three-stage answer extraction was used, with no random fallback. Capped answers were retained and scored; malformed or missing extracted answers failed. Each runner had a 30-minute ceiling, with inference cancellation at 29m 30s and a 120-second per-request timeout. Both completed early. Raw answers, reasoning, streams, hashes, resource traces and scoring evidence are saved. NVFP4 used the preserved 0.1.42-nvfp4.2 engine; Q6 used 0.1.44.1-nvfp4.2. Both had a 131,072-token context ceiling and int8 KV, but cache allocation, KV placement and other runtime details differ. Q6 ran first in an existing session; NVFP4 was freshly loaded afterward. There was no order reversal or repeated-seed study. These limits prevent attributing the score difference to quantization alone. No full-benchmark, broad coding, long-context reasoning or long-term stability claim is made. AI assisted with the harness and write-up. The scores and timings come from saved local runs. reddit.com Strata fork: sergqwer/strata-nvfp4 MMLU-Pro: TIGER-AI-Lab/MMLU-Pro GSM8K: openai/grade-school-math HumanEval: openai/human-eval IFEval: github.com/google-research/google-research/tree/m… Testing sources and how the results were obtained All scores, speeds and resource measurements in this post came from runs on my own PC. - MMLU-Pro — knowledge and reasoning: Benchmark code and methodology | Question dataset. The fresh comparison used 84 questions, six from each of 14 subjects, with identical questions and request settings for both models. The earlier mixed comparison used a separate 42-question subset. These were selected subsets using a zero-shot chat adaptation. - GSM8K — maths: OpenAI’s dataset and reference code. The earlier comparison used 20 questions, scored against the expected numerical answers. - HumanEval — Python coding: OpenAI’s tasks and original unit tests. The earlier comparison used ten tasks. Generated solutions were checked using the original tests in an isolated Docker environment. These were original HumanEval tests, not HumanEval+. - IFEval — instruction following: Google Research’s benchmark and evaluation code. The earlier comparison used ten prompts with strict prompt-level scoring: all required constraints had to pass for a prompt to count as correct. The selected subset excluded checks requiring additional NLP corpus downloads. - Generation speed and prompt processing: Measured locally using benchmark scripts calling each model server directly. Each context used three measured requests with 4,096, 32,768 or 128,000 actual input tokens and 256 generated tokens, with reasoning disabled and zero reused prompt tokens. Tables show session medians; the historical overview selects the best recorded session median per model and context. Earlier runs and settings are documented in my previous speed-testing post. Q6’s measurements were added on 23 October. - RAM, VRAM, temperature and timing: Recorded during the local runs through system/GPU telemetry, saved request records and engine timing data. Memory figures describe sampled whole-system RAM and whole-device VRAM usage. Model and engine references - NVFP4/GPTQ model - Strata NVFP4 fork used for the main comparison Exact question selections, request settings, raw answers, reasoning output, scoring records, hashes and resource traces are retained locally. The links above provide the public benchmark sources; the reported results are my own subset measurements, not full official benchmark or leaderboard scores.

Little Sweet Room Challenge | Strata Fork | Qwen 3.8 FN NVFP4 | DeepSeek Harness Another little challenge I seen on a DeepSeek Thread. I thought I'd try out it out on Qwen FN NvFp4. It was a pretty straight forward prompt with 4 images attached in the doc. The thread was testing different harness and models and showing their results. Op of that thread has two other threads on the same Challenge just one with DeepSeek Harness so I used it as well. It took about 27 min(Not bad) and was surprised how it planned everything out, also I do have vision on this Model. I think the Vision on Qwen is also a big part in a lot of Accuracy calls and adjustments. Especially notice when I use blender for other projects. Havent had too much trouble after getting use to how to even use blender since i literally started 2 weeks ago and having an LLM run it has been pretty fun to use. So if you can run vision don't sleep on it. lol Any who this was the result and I like how Qwen tried to get all the placement, designs and little details to the references. Below are the measurements and hardware I use and what not. Full prompt is from the referenced Thread. Also if you run no harness for Qwen I would like to see the results or even results from other harnesses if possible. Last but not least thank you all to the community here and everyone who is contributing locally in many ways or even just interacting with the community, keep up the great work and looking forward towards the future of running Local. Thread for full Prompt doc reddit.com Just the text prompt: "Create a highly realistic 3D first-person game using JavaScript. The game will take place in a single room. The room is the one shown in the 1st image. Model the room in the 1st image exactly in JavaScript, with absolutely no differences. Do not make it cartoon-style. It must be extremely realistic. Model it in the best possible way, identically, with extremely realistic graphics, and make everything very cute. By “cute,” I mean the style of the 2nd, 3rd and 4th images: cute items with clearly defined edges. Build it in the same style, cute but also realistic and well made, like the 2nd, 3rd and 4th images. Examine all 4 images I sent pixel by pixel, in detail, so you can build it exactly as I want. Make it flawless and do an excellent job." Engine: sergqwer/strata-nvfp4 The stack Hardware: RTX 5090 32GB · Ryzen 7 9800X3D · 192GB DDR5-5600 · Windows 11 Engine: strata-nvfp4 (fork of Niko1221/Strata by sergqwer) on Windows, llama-swap as switcher Model: Qwen3.8-Flash-Next 125B-A6B hybrid MoE — abliterated checkpoint(windowsxp811203), self-converted NVFP4 Weights: NVFP4 experts (4.5-bit) + Q8_0 dense · KV: int8 · PLE: bf16 · Embeddings: bf16 MTP: draft head, spec 5, spec-min-p 0.7 · Context: 262,144 max · Compaction trip: 220,144 Sampling: temp 0.8 / presence_penalty 0.3 / min_p 0.02 · Vision: fp32 mmproj, encoder on CPU RAM tier: 63 GiB pinned experts · Harness: dsh (web GUI), OpenAI API Metrics — this run Metric |Value Wall: first message → last reply |26m 17s − my approval latency |−0s (zero prompts) = pure agent working time |26m 17s Turns |1 (single prompt, no babysitting) Steps |103 Output tokens |132,929 Fresh input tokens |247,232 Cache-read tokens |11,994,999 (98.0% hit) Decode (engine-verified) |median 121.4 t/s · mean 121.2 · peak 187.8 TTFT |median 0.85s (max 32.9s at ~194k context) Prefill (fresh, engine-verified) |mean 2,757 t/s · peak 7,096 t/s Longest single burst |10,395 tokens Compactions |0 (peaked at 194k, never near the 220k line) Tool calls |179 total · 17m 29s runtime · 0 errors

Strata、NVFP4でいろいろ頑張ってるforkがあるみたい。RTX5090とかBlackwellの人はこっちも楽しそう。 そのうち‘UpstreamにMergeされるのかな。 sergqwer/strata-nvfp4

Horsey challenge | Strata Fork | Qwen 3.8 FN NVFP4 So I wanted to try the horsey challenge myself on the latest engine i been testing which is a strata fork by sergqwer. sergqwer/strata-nvfp4. I seen a post earlier and seen i believe it was a iq3 i think and it was pretty dope so i thought i try out the NVFP4. I wasn't expecting to share but i actually that it did pretty darn good with its current time it took total. I did change the prompt because this was the second try because the first one got botched because Qwen wanted to leave like 15+ headless windows open when testing the html. I scrolled to another window while it was working and realized my window was lagging lmao. That test i ended it like 45 min in. Which then i retried with new prompt and this time it was much faster. I had this fork up and running a few days now. I had to tweak some settings for a higher compaction i believe at the 220k-230k mark. It didn't need compaction this time as it was under the limit but when I use it for long runs projects using unity headless, Blender headless and some other things like image gen for texture i will be compacting a lot. Its only been a couple days but so far its been fairly great to anything I have used prior. That goes for the engine and the current model i'm using. This is just to share and contribute to how far we come to accessing models on single gpu and being able to have a chance at some of these things. Maybe it'll motivate more people to show their work a bit more often. I know this sub has motivated me a lot to start sharing and also appreciate all that has helped to contribute for anything Local. Thank you to the original Strata Team, Sergqwer for the fork, windowsxp811203 for your model I converted over to work with Strata, and the Qwen Team for the amazing drops. *The charts were organized by Ai only, it was easier to have it sort it nicer and clean it. Also I did a prtsc video capture if you ask about quality of vid. Prompt here also that i had to edit myself lol. "Create a low-poly 3D horse running in a seamless loop using Three.js. Build it procedurally from primitive geometry, animate the legs, body, head, and tail, and return a single self-contained HTML file. Also when you use headless chrome make sure to kill your process that you don't need anymore. we dont need all these headless chromes lagging our system when we don't need it" The stack Hardware: RTX 5090 32GB · Ryzen 7 9800X3D · 192GB DDR5-5600 · Windows 11 Engine: strata-nvfp4 (fork of Niko1221/Strata by sergqwer) running on Windows, fronted by llama-swap as the model switcher Model: Qwen3.8-Flash-Next 125B-A6B hybrid MoE — abliterated checkpoint, self-converted to NVFP4 Weights: NVFP4 experts (4.5-bit, calibrated scales) + Q8_0 dense KV cache: int8 PLE table: bf16 Embeddings: bf16 MTP: draft head, spec 5, spec-min-p 0.7 Context: 262,144 max Compaction: custom trip line at 220,144 Sampling: temp 0.8 / presence_penalty 0.3 / min_p 0.02 Vision: fp32 mmproj, encoder on CPU RAM tier: 63 GiB pinned experts; hot experts cached in VRAM; misses computed CPU+PCIe overlapped with GPU Harness: dsh (web GUI), OpenAI API | Metric | Value | |---|---| | **Wall: first message → last reply** | **33m 45s** | | − my approval latency (25 permission prompts) | −10m 35s | | **= pure agent working time** | **23m 10s** | | Steps | 87 | | Output tokens (thinking + code + tool calls) | **124,443** | | Fresh input tokens | 96,334 | | Cache-read tokens | 9,340,144 (**99.0% hit rate**) | | **Decode (per-step, approval-adjusted)** | **median 116 t/s · mean 103 · peak 171** | | TTFT | median **0.7s** (max 24.9s at deep context) | | Prefill (fresh tokens) | mean **3,332 t/s** · peak **6,623 t/s** | | Longest single burst | 21,279 tokens | | Compactions | **0** (never hit the 220k line in 33 min) | TOOL CALLS — 107 total, 14m 10s of tool runtime

Daily Driving Qwen 3.8 Flash-Next MoE (NVFP4) on RTX 5090 + 128GB RAM — Telemetry & Impressions UPDATE: My usage refers to the NVFP4 fork of strata: sergqwer/strata-nvfp4 with qwen3.8-flash-next-uncensored-nvfp4 I've been daily driving a number of 27b models over the past couple of weeks (notably the swift 1.5 dflash2 variant). My 27b experience has been reasonable, but I found it failing to progress problems in large code bases (Rust and Lua mainly). I had tried 3 bit flash before and found it to be ok speed but worse than 27b with agentic coding... anecdotally. I've just installed 128GB memory and have finished my first day with Flash with very good results. I've been using medium thinking with omp. I haven't seen any fudging or giving up yet - good forward progress. Full shout out to Strata devs and sergqwer/strata-nvfp4 ! Deets follow: Setup & Hardware - Hardware: RTX 5090 (32GB VRAM, PCIe 5.0 x16), 136 GB 128 GB DDR5 RAM, Windows 11. - Engine: Strata (NVFP4 fork, v0.1.40.3). - Model: Qwen 3.8 Flash-Next MoE (uncensored NVFP4). - Precision layout: - Embeddings: BF16 (token-embd-bf16.gguf) - Prefix layer experts (PLE): FP8 (ple-fp8.gguf) - MoE weights: Native NVFP4 (orca-nvfp4.gguf) - KV cache: INT8 dynamic paged cache (262k max context) - Memory placement: 8,458 expert slots (22.3 GB active MoE cache held in VRAM; remainder dynamically paged from 64 GB host RAM arena). - Power script: Background PowerShell watchdog calling Win32 SetThreadExecutionState to block sleep while active, auto-sleeping after 60m idle. Workload & Scale (~8 Hours Continuous Daily Drive) - Requests: 2,411 completed. - Gross prompt tokens: 307,830,375. - Prompt tokens cached/reused: 305,693,564 (99.31% KV cache hit rate). - Net new prefill tokens: 2,136,811 (0.69%). - Output tokens: 865,423. - Active working context: Usually 100k – 212k tokens per request. Measured Telemetry - Decode speed: 135 – 153 tok/s. - Context vs speed correlation: $r = +0.005$ (decode speed is completely flat from 50k to 212k tokens). - 52k context: 134.5 tok/s (prefill: 4.7s on cold cache). - 205k context: 146.7 tok/s (prefill: 483 ms with warm cache). - 212k context: 141.2 tok/s (prefill: 617 ms with warm cache). - MTP speculative decoding: 79.60% acceptance rate (539,953 accepted / 678,292 offered, ~4 tokens per forward pass). - MoE VRAM cache hit rate: 89.9% average (peaking at 93.6%). - PCIe bus traffic: 6.8% average bus share over PCIe 5.0 x16 (bursting up to 23 GB/s RX). - Thermals & power: 52°C – 53°C core temp, 230W – 335W power draw. - Stability: 0 crashes, 0 reasoning loop recoveries, 0 memory leaks (rock steady at 31.5 GB VRAM allocation).

@cloudol @coldniko 不支持呢，这玩意算是 llama.cpp 超级魔改版，但 github 上有人按 strata 思路在做了：sergqwer/strata-nvfp4