huangserva

Make Volta Great Again! — Strata: 500 tok/s System Aggregate Decode, 6,000+ tok/s Prefill, Built on six $150 Volta GPUs A revisit on old datacenter GPU preview.redd.it Content Index 1. TLDR(result) 2. Hardware 3. Cooling & Power 4. Engine 5. Benchmark 6. Future Potential Speedup 1)TLDR: Qwen 3.8 Next Flash |engine|Decode,single user, 6x V100|Decode,concurrent=3, but in "hardware mode*"|128k ctx prefill| |:-|:-|:-|:-| |llama.cpp(baseline)|70 tk/s|-|730 tk/s| |strata official|175 tk/s|500 tk/s**|4995 tk/s| |strata v100 fork|141 tk/s|345 tk/s|6368 tk/s| Qwen 3.8 27B |Engine|Decode,single user, Dual V100, MTP|Decode,concurrent=3, but in "hardware mode"|64k/128k ctx prefill| |:-|:-|:-|:-| |Llama.cpp(baseline)|74 tk/s|222 tk/s|~600 tk/s| |Ninfer-v100-tpx|248 tk/s|744 tk/s|1032/893 tk/s| |llama-superfast-v100 fork|214 tk/s|642 tk/s|887/777 tk/s| ^(1)model used: Swift 1.5 IQ3_XXS GSQ-RCO, Swift 1.5 Q2_0 GSQ-RCO*) ^(2)**"hardware mode" concurrency refer to running multiple independant instance of inference server,using independant gpu, in the same pc) ^(3)Unless otherwise stated: Q8 (INT8) KV cache, greedy decoding, and draft length of 7.) preview.redd.it preview.redd.it Dark mode Screenshot: Official Strata — stronger decode performance. Light mode Screenshot: Strata fork — stronger prefill performance. 2)Hardware Sorry for the aesthetics and cable spaghetti. I'm a utilitarian at heart, and I've given up on cable management for this behemoth. 😂 cable management nightmare Hardware specs: A $130 Chinese X99 dual-socket server motherboard. |Component|Specification|Comment| |:-|:-|:-| |PCIe|6 slot,80 lane,Rev 3.0 (x16 x16 x16 x16 x8 x8)|All slots support further bifurcation, enabling additional GPUs via PCIe splitters.| |PCIe Clearance|70mm (3.5 slot)|Ample space for custom DIY cooling solutions.| |CPU|Dual E5-2680v4 28C/56T|| |RAM|112GB DRR4 Quad Channel,ECC 2133Mhz,8 slot total(512GB max)|Slowest-tier memory, but largely irrelevant when the model runs entirely in VRAM.| Motherboard and GPU configuration Cleaner photo before mayhem,source:vendor I'm using an exotic cryptocurrency-mining motherboard because most models provide only PCIe x1 connectivity per slot. From my experience, Strata's layer-splitting approach requires relatively little PCIe bandwidth, making x4 or x8 lanes sufficient. In contrast, ninfer and the llama.cpp fork use tensor parallelism(27B), which demands more inter-GPU communication bandwidth; utilization peaks at around 70% of a PCIe x8 link. Since the V100 uses PCIe 3.0, this motherboard is a practical match for my workloads. Interestingly, SXM2 variants of the V100 can be cheaper than their PCIe counterparts, even after accounting for the additional components required. Since LLM inference is often memory-bandwidth-bound, the V100's formidable 900 GB/s HBM2 bandwidth makes it a practical choice despite its lack of support for newer GPU instructions. For comparison, you'd need an RTX 5080 (960 GB/s) or higher to surpass the V100's raw memory bandwidth. This demonstrates just how formidable the V100's HBM bandwidth remains. The system houses six NVIDIA Tesla V100 16 GB SXM2 GPUs, each installed through an 50$ SXM2-to-PCIe x16 adapter. source:vendor preview.redd.it 3)Cooling & Power 3.1)Cooling - 3 GPUs: Cooled by repurposed AIO liquid coolers, secured with iron wire and zip ties. - 2 GPUs: Air-cooled using custom heatsinks supplied by the seller. - 1 GPU: The outermost card uses a repurposed CPU tower heatsink. Most fans are modified cryptocurrency-mining fans running at half speed, drawing approximately 1.5 A each. For comparison, the full-power 3 A fans sound like a jet taking off, while the 1.5 A configuration produces a much more manageable, low-frequency hum. Air-cooled GPUs rarely experience thermal throttling, thanks to a 150 W power limit imposed via nvidia-smi. This incurs an approximately 5–10% performance penalty in tensor-parallel workloads such as Qwen3.8-27B, but virtually no performance loss in pipeline-parallel workloads, such as Strata running Qwen3.8-Next-Flash. Forget mounting-hole compatibility—cable ties offer universal compatibility |Cooling|Strata (Layer Splitting / Pipeline Parallelism) 150W/300W PL|ninfer / llama.cpp fork (Tensor Parallelism) 150W PL|ninfer / llama.cpp fork (Tensor Parallelism) 300W PL| |:-|:-|:-|:-| |Air(stock heatsink + Medium Cryptofan)|No throttle|Occasionally Throttle|Frequent throttle| |Water (120mm AIO)|No throttle|No throttle|Occasionally throttle| |Water (240mm AIO)|No throttle|No throttle|No throttle| With layer splitting, GPUs process successive layers in turn, leaving other GPUs relatively idle while one GPU performs its computation. Consequently, power consumption per GPU can be substantially lower than its TDP, although the effective draw is workload-dependent and does not necessarily scale as (1/N). This helps explain why multi-GPU Strata configurations can maintain temperatures below 70°C despite sustained inference workloads. Although the V100's specified maximum operating temperature is 87°C, practical thermal throttling may begin around 82°C. This 5°C gap may reflect firmware thermal-control behavior, where throttling is triggered by an individual sensor reaching its threshold rather than by the reported average GPU temperature. This is especially relevant in tensor parallelism (TP) mode, where GPUs compute concurrently. Sustained workloads can create localized hotspots even when average temperatures appear safe, making hotspot temperature—not just reported core temperature—a critical factor in thermal management. 3.2)Power Management Power-limited to 150W via nvidia-smi -pl 150—trading just 8% performance for a 30% reduction in power consumption. A pretty good deal. |Power Limit(per card),W|True Gpu power(2 card combined),W|Decode speed,Tk/s|Base Speed|Energy efficiency,Tk/s per W| |:-|:-|:-|:-|:-| |150w|291|219|92%|0.75| |300w|427|237|100%|0.56| Power-limited to 150W via nvidia-smi -pl 150—trading just 8% performance for a 30% reduction in power consumption. A pretty good deal. Each GPU consumes approximately 25W at idle with no inference engine loaded, rising to around 50W when model weights are loaded but no inference computation is taking place. Each V100 has a stock TDP of 300W, but limiting each card to 150W keeps total GPU power consumption below 900W in the worst-case tensor-parallel (TP) configuration, or around 500W for the six-GPU Strata setup(for comparison ,RTX 5090 TDP is 575W) One caveat: This system requires two 8-pin PCIe power connectors per GPU, for a total of 12 connectors. Most mainstream 1,000W gaming PSUs lack that many connectors, even if they can supply sufficient power electrically. Inexpensive 1,000–2,000W cryptocurrency-mining PSUs are a more practical option due to their higher connector count. 4)Engine used llama.cpp build 572 0.6.0-dev llama-superfast-v100 build 2 0.4.1-dev* Strata official 0.1.44.1 Strata V100 0.1.42* Ninfer-v100-tpx* ^(*fork link provided at the end of post) I've also tested other inference engines, such as PXA and the VLLM 1Cat V100 fork, but their performance was only marginally better than stock llama.cpp. Since they're optimized for 32GB V100s and NVLink configurations, which I don't have, I decided not to pursue further testing. With six GPUs at my disposal, I can experiment with different topologies depending on the workload. However, benchmarks show that increasing engine concurrency beyond 2 produces mixed results, with performance fluctuating by roughly ±10%—sometimes improving, sometimes worsening. At that level of variability, the gains aren't compelling enough to justify the added complexity. Enter hardware-level concurrency acceleration: instead of pushing concurrency higher within a single inference engine, distribute independent workloads across separate GPUs within the same PC. This allows multiple inference instances to run concurrently without competing for the same GPU resources, potentially delivering more predictable throughput gains. |Config|Use case|GPU config|Remark| |:-|:-|:-|:-| |"MegaTitan"|Deep research, architectural planning, intricate debugging|1 Strata × 6 GPUs|Maximum model capacity and context depth; target 1M context with FP16 KV cache| |"Twin giant"|Two simultaneous, independent complex tasks|2 Strata × 3 GPUs|Parallel problem-solving without sacrificing too much per-instance capacity| |"Cerberus"|High-volume, complex coding and debugging|3 Strata × 2 GPUs|Three capable inference workers for concurrent workloads| |"big.LITTLE"|Multi-agent workflows with unequal reasoning demands|1 Strata × 4 GPUs + 1 ninfer × 2 GPUs|Next Flash handles orchestration; 27B handles demanding subtasks| |"Chimera"|Multiple independent projects and heterogeneous agent workflows|3 ninfer × 2 GPUs|High aggregate throughput with isolated workloads| |"Asura"|Mass classification, extraction, and short-context batch inference|6 llama.cpp-fork instances × 1 GPU|Maximum parallelism for short-context, fire-and-forget jobs| 5)Benchmark Before reading further, keep these benchmark configurations and methodology in mind: - Strata & its fork: Qwen 3.8 Next Flash, 256K context; benchmarked with a warm-up run. - ninfer fork: Qwen 3.8 27B Swift 1.5 NVFP4*, 200K context; benchmarked cold, without warm-up. - llama.cpp fork: Qwen 3.8 27B Swift 1.5 Q4_K_M, 160K context; benchmarked cold, without warm-up. Unless otherwise stated, all benchmarks use an INT8 (Q8) KV cache, greedy decoding, and a draft length of 7. Each benchmark is repeated three times(mean) using a fresh server session to minimize caching effects. Strata is benchmarked warm because its architecture is designed to benefit from warm-up, whereas the ninfer and llama.cpp forks do not require warm-up. * Yes, you read that correctly—NVFP4 on non-Blackwell GPUs (RTX 5000 series). It's not a typo! 5.1)Benchmark-strata preview.redd.it A single server with six V100s maximizes prefill throughput through highly parallel pipeline execution. As noted earlier, increasing Strata's concurrency to two yields only modest aggregate gains, suggesting the configuration is already highly optimized. Splitting the GPUs into three pairs, each operating as an independent instance, may therefore maximize aggregate decode throughput through three-way hardware-level concurrency. For single-user workloads, the Strata-V100 fork delivers roughly 25% faster prefill, while official Strata offers roughly 25% faster decoding. Future updates may narrow this gap. The break-even input/output token ratio is 6.31:1: above it, the fork wins; below it, official Strata wins. In my own week of usage—135M input tokens and 25M output tokens, a 5.4:1 ratio—official Strata was faster overall. For prefill-dominated workloads, however, the fork's advantage may justify the trade-off. Choose based on your actual workload profile. preview.redd.it For Strata, MTP performance plateaus at draft length 4. preview.redd.it Strata pair with Qwen 3.8 Next Flash handles context-weight management impressively well, losing only 14% of decode speed at 128K context compared to 0K. 5.2)Benchmark Ninfer + Llama fork preview.redd.it While the ninfer and llama.cpp forks paired with Qwen3.8-27B deliver impressive initial decode speeds, both suffer significant degradation at longer contexts, losing more than half their decode speed at 128K compared to 0K context. Comparing decode speeds at just 0K and 128K context doesn't tell the whole story. Averaging performance across the entire context range provides a more representative picture of real-world performance. Strata averages 138 tok/s across 0–128K context, compared to 165 tok/s for the ninfer fork and 147 tok/s for the llama.cpp fork. For short-context workloads, ninfer is the undisputed king. However, beyond 200K context, Strata's more gradual decode-speed degradation allows it to shine as the context grows. preview.redd.it In general, the ninfer fork is roughly 15% faster than the llama.cpp fork, but its higher VRAM footprint reduces context capacity by approximately 20%. Despite running models with more active parameters (27B vs. 6B), both forks achieve substantially higher decode speeds than Strata, thanks to efficient tensor parallelism that pushes GPU utilization close to 100%. Strata does not support tensor parallelism, while stock llama.cpp's tensor-parallel implementation is less efficient, reportedly achieving only around 60% GPU utilization at best. 6)Potential speedup (hypothesized,not proven) 1. CPU and RAM upgrade: A lower-core-count CPU with a higher base frequency, such as the E5-2643 v4, might offer some improvement. However, since I'm using the slowest-tier 2133 MHz RAM and my 27B model weights reside entirely in VRAM, faster RAM may provide little to negligible gains. A two-GPU Strata configuration might benefit more. 2. Motherboard upgrade: Switching to a platform with full x16 lanes for every GPU, such as AMD EPYC, might improve performance in engines that utilize tensor parallelism (TP). However, I'm skeptical about the actual gains: even with TP=2 over x8 PCIe lanes, GPU utilization only reaches around 70%, suggesting that PCIe bandwidth may not be the primary bottleneck. Unless the workload is demonstrably bandwidth- or latency-limited, upgrading to x16 lanes might deliver only marginal improvements. 3. NVLink upgrade: Populating a two- or four-GPU NVLink baseboard could deliver meaningful improvements for Qwen3.8-27B, but likely much less for Qwen3.8-Next-Flash, which currently only explicitly supports layer splitting. However, PCIe-based TP=2 already pushes GPU utilization close to 100%, so further gains may be limited, perhaps coming primarily from reduced communication latency. A four-way NVLink setup could offer greater gains for Qwen3.8-27B, but unfortunately, the fork doesn't support TP=4, and four-GPU NVLink baseboards are expensive. 4. Cooling upgrade: Replace air cooling or 120 mm AIOs with 240 mm AIOs across all GPUs to eliminate thermal throttling in models that support TP, helping sustain peak performance during prolonged workloads. 5. My decision: I'll skip these hardware optimization routes. The cost of each upgrade could buy several additional V100s, offering more direct and measurable performance gains. Unfortunately, my current motherboard is already at its maximum GPU capacity. 6. Software optimization: For Strata running Qwen3.8-Next-Flash (NF), developing my own fork with tensor parallelism (TP) support could deliver the biggest performance gains. Alternatively, I could wait for the official repository to implement it. For ninfer and the llama.cpp fork, implementing TP=4 support myself could also yield significant performance improvements, particularly for Qwen3.8-27B. Link: huangserva/ninfer-v100-tpx jmnargi/Strata-V100 codeberg.org had removed by author

MiniMax H3 这套导演台工作流，是有人专门攒出来的。 直接丢进 ComfyUI 就能开跑，文生视频、图生视频、首尾帧、视频换人这些活儿全都接得住。 🔗 huangserva/ComfyUI_MiniMaxH3_Director t.co

MiniMax H3这套导演台工作流，是有人专门攒出来的。 直接丢进ComfyUI就能开跑，文生视频、图生视频、首尾帧、视频换人这些活儿全都接得住。 🔗 huangserva/ComfyUI_MiniMaxH3_Director 想捣鼓的，进仓库翻翻就明白怎么用了。 t.co

有位哥们儿张罗出一套 MiniMax H3 导演台工作流， 扔进 ComfyUI 里就能直接开跑，文生视频、图生视频、首尾帧、视频换人这几样全给你兜住了。 🔗 huangserva/ComfyUI_MiniMaxH3_Director t.co

有位哥们儿张罗出一套 MiniMax H3 导演台工作流 扔进 ComfyUI 里就能直接开跑，文生视频、图生视频、首尾帧、视频换人这几样全给你兜住了。 🔗 huangserva/ComfyUI_MiniMaxH3_Director t.co

ComfyUI 玩家有口福了，这套 MiniMax H3 工作流直接照着抄就行。 有人把 MiniMax H3 的常用玩法攒成了一个「导演台」，导进 ComfyUI 就能直接用。 huangserva/ComfyUI_MiniMaxH3_Director 支持文生视频、图生视频、首尾帧续写、视频换人等常用工作流，不用自己一堆节点慢慢搭，打开就能开跑。MiniMax H3 最近已经能在 ComfyUI 里运行，社区也在快速补齐各种工作流和插件。