Same config.json, different serve command: what to check before serving AREX-2
When a fine-tune declares the same architecture class as a model you already serve (run behind an API on your own GPU), the tempting move is to swap --model-path and restart. BAAI's AREX-2 is a good test of that habit. It is a 27B model trained to work on a problem in rounds (propose, measure, reflect, revise), released on 29 September under Apache 2.0, not gated, and its config.json opens with Qwen3_5ForConditionalGeneration, the same class as Qwen3.8-27B from the Qwen team.
We had been serving base Qwen3.8-27B with sglang, an open-source inference server, so we diffed the two configs. One caveat first: we have not run AREX-2. Every speed below is from the base model. Tables, command and sources are in the full AREX-2 serving analysis on inferencekey.com.
The diff that says "drop-in"
BAAI shipped a clean release: layer layout, head counts, context length, vocab and vision tower all match Qwen3.8-27B. The config differs only in transformers_version and two redundant top-level keys, and chat_template.jinja, the file that formats the conversation and tool calls, is byte-identical.
Stop reading there and you ship the old command.
Check the weights, not the config
The base model carries a multi-token prediction (MTP) head, a small extra layer that guesses a few tokens ahead so the server can emit several per step. AREX-2's config still declares it. Its weight files do not contain it.
config.json "mtp_num_hidden_layers": 1
safetensors index (AREX-2) 0 mtp tensors
safetensors index (Qwen3.8) 15 mtp tensors
The safetensors index lists every weight tensor in the release, so it is the honest inventory. A config field is a promise; check that something actually backs it.
The second quiet change is in tokenizer_config.json, which controls how text is cut into tokens. A community repack reports that the pre-tokenizer and decoder settings differ from Qwen3.8. Likely harmless. Also what surfaces as a garbled tool call hours into an agent run.
Flags that were right for the base model
Our quickest base-model setup ran on an RTX PRO 5000 in a CUDA 13 pod, with RadixArk's NVFP4 quant (weights compressed to 4-bit floats) and the z-lab DFlash2 drafter. It decoded (wrote output) at 115 to 136 tok/s. Little of that survives the swap:
- No NVFP4 build of AREX-2 exists yet. Today you pick from BAAI's BF16 original (54.7 GB), prithivMLmods' FP8 (35.9 GB), cyankiwi's AWQ INT4 (20.2 GB) or numsu's INT4 W4A16 (18.6 GB).
- The drafter was trained against another model. Speculative decoding has a small drafter propose tokens that the big model then verifies in one pass; it only pays off when most guesses are accepted. z-lab's DFlash2 drafter targets base Qwen3.8-27B, and a fine-tune changes which tokens the model favors, so nobody knows its hit rate on AREX-2. numsu's INT4 card does publish speeds with a DFlash2 drafter on two RTX 3090s, but without an acceptance length.
- mamba_ssm_dtype. AREX-2 asks for float32. Our base command forced bfloat16, fine on Qwen3.8, untried on a model tuned for long reflective runs.
- Parsers are a guess. These sglang options turn raw model text into separate reasoning and tool-call fields. qwen3 and qwen3_coder ought to work since the template matches. No sglang run of AREX-2 has been published.
- Pick long context or speculation. The base model's 1M context is a YARN override that stretches its native 262,144 positions, and DFlash2 would not run with it. So: 1M without a drafter, or 256K with one.
FP8 on Ampere is the expensive mistake
The RTX A6000 is Ampere and has no FP8 hardware, so sglang keeps the weights in FP8 and unpacks them on the fly through Marlin kernels. Base Qwen3.8-27B decoded there at 24 tok/s. RedHatAI's INT4 W4A16 quant, same card, same 1M context: 41 tok/s. prithivMLmods' AREX-2 FP8 card agrees that full speed needs Ada, Hopper or later. An A100 is Ampere too, so reach for INT4 there.
Weights are not the whole bill. In our base-model INT4 run at 1M context, sglang took about 46 GB, of which only 17.8 GB was weights. Another 11.5 GB held state for the linear attention layers, and 12.8 GB went to the KV cache, the per-token memory the model keeps so it does not reread the whole context on every step. The per-GPU sizing for AREX-2 goes card by card from 80 GB to 24 GB, as estimates.
What to measure on the first run
Start dull: BF16 on an 80 GB card, the native 256K context, no speculative flags, the same parsers, one long agent task. Then change a single thing per run, like a soundcheck that starts with the kick drum. Watch:
- tool calls parsing cleanly late in a long session, given the tokenizer change
- decode tok/s against the base model on the same card, which the shared architecture says should match
- acceptance length if you add a drafter, the number nobody has published
- GPU hours per task, since the benchmark protocol allows up to 1,500 model calls and 24 hours per case
That last item drives the economics. AREX-2's self-reported 70.7 on Frontier-CS and 84.0 on BrowseComp come from that long protocol in BAAI's evaluation repo, with no independent reproduction we could find. Renting a GPU by the hour beats per-token pricing only while the card has work queued, for instance several cases side by side. For the rest of a single-GPU agent setup, see our earlier post on an 80B coding model on one rented A100 driven by Claude Code.
This is a plausible one-card project thanks to BAAI and the Qwen team for the models, prithivMLmods, cyankiwi, numsu, RedHatAI and RadixArk for the quants, z-lab for DFlash2, numsu again for an experimental MTP head trained on AREX-2 outputs, AI Search for flagging the release, and the sglang and vLLM projects. The config tables, the base-model command verbatim, the worker settings for slow warmups and the full limits list are in the complete AREX-2 on one GPU write-up.
InfoVis: Unified Slide Image Generation and Editing
Generating and editing presentation slides requires models to combine accurate text rendering, instruction following, visual grounding, aesthetic design, and precise editing within one canvas. Existing work covers only subsets of these capabilities and lacks large-scale slide-native supervision and diagnostic evaluation. We formulate four slide-centric tasks and build a scalable data engine with over 13M continual-pretraining records and 1.9M supervised fine-tuning records. We also introduce VisSlideBench, a benchmark of 1,527 manually screened instances that covers all four tasks with task-specific, multidimensional evaluation. Its automatic rankings agree with blinded, dimension-wise human rankings. Using these resources, we train InfoVis, a unified 9B slide generation and editing model. On VisSlideBench, InfoVis leads every evaluated open-source model on all eleven metrics and scores 7.94 overall against 7.99 for Nano-Banana-2. Starting from FLUX.2-klein-base-9B, slide-specific post-training adds 1.6 points overall, with the largest generation gains in content faithfulness; the remaining gap to proprietary models is smallest in content faithfulness and largest in aesthetic quality. Built on InfoVis, our document-to-slide workflow achieves the second-highest Aesthetics score on the external SlidesGen-Bench. We will release the model, code, data, and benchmark.
Comments: Code: VectorSpaceLab/InfoVis
AgentSWE: Can Coding Agents Build the Agent You Actually Want?
As the production of software passes into the hands of software itself, coding agents are increasingly commissioned to build agents fitted to their users' own workflows. This delegation carries a quiet assumption, that the delivered agent will faithfully serve the requirement it was built for, rather than merely appear to. No existing benchmark tests that assumption. We introduce AgentSWE, a benchmark for agent software engineering spanning three stages of the agent development lifecycle: Creation of a complete agent from a natural-language requirement, Editing of production agent codebases, and Optimization toward a measurable target. Each of its 25 tasks is framed as a commission rather than an exam: the builder iterates against a handful of illustrative cases, while acceptance exercises the frozen submission on held-out cases designed to span the full requirement, grading behavior against requirement-derived rubrics under hard pass/fail gates or, for Optimization, against the source benchmark's own objective. Evaluating a range of coding agents across models and harnesses, we find that none comes close to delivering what a requirement asks for: the gap between what a builder demonstrates and what it delivers is the norm, not the corner case. An accompanying empirical analysis of the failure trajectories identifies recurring failure modes, among them overfitting to the visible cases, hollow runtimes behind code that looks complete, and unsupported claims of completion, each a signal by which coding agents can learn to build the agents they are asked for.
Comments: Code: VectorSpaceLab/AgentSWE
Mentions
AgentSWE (BAAI / VectorSpaceLab) asks coding agents to create, edit, or optimize agents from a brief. builders get a few illustrative cases. acceptance freezes the delivery and runs held out cases covering clauses the demos left out arxiv.org VectorSpaceLab/AgentSWE
Same config.json, different serve command: what to check before serving AREX-2 When a fine-tune declares the same architecture class as a model you already serve (run behind an API on your own GPU), the tempting move is to swap --model-path and restart. BAAI's AREX-2 is a good test of that habit. It is a 27B model trained to work on a problem in rounds (propose, measure, reflect, revise), released on 29 September under Apache 2.0, not gated, and its config.json opens with Qwen3_5ForConditionalGeneration, the same class as Qwen3.8-27B from the Qwen team. We had been serving base Qwen3.8-27B with sglang, an open-source inference server, so we diffed the two configs. One caveat first: we have not run AREX-2. Every speed below is from the base model. Tables, command and sources are in the full AREX-2 serving analysis on inferencekey.com. The diff that says "drop-in" BAAI shipped a clean release: layer layout, head counts, context length, vocab and vision tower all match Qwen3.8-27B. The config differs only in transformers_version and two redundant top-level keys, and chat_template.jinja, the file that formats the conversation and tool calls, is byte-identical. Stop reading there and you ship the old command. Check the weights, not the config The base model carries a multi-token prediction (MTP) head, a small extra layer that guesses a few tokens ahead so the server can emit several per step. AREX-2's config still declares it. Its weight files do not contain it. config.json "mtp_num_hidden_layers": 1 safetensors index (AREX-2) 0 mtp tensors safetensors index (Qwen3.8) 15 mtp tensors The safetensors index lists every weight tensor in the release, so it is the honest inventory. A config field is a promise; check that something actually backs it. The second quiet change is in tokenizer_config.json, which controls how text is cut into tokens. A community repack reports that the pre-tokenizer and decoder settings differ from Qwen3.8. Likely harmless. Also what surfaces as a garbled tool call hours into an agent run. Flags that were right for the base model Our quickest base-model setup ran on an RTX PRO 5000 in a CUDA 13 pod, with RadixArk's NVFP4 quant (weights compressed to 4-bit floats) and the z-lab DFlash2 drafter. It decoded (wrote output) at 115 to 136 tok/s. Little of that survives the swap: - No NVFP4 build of AREX-2 exists yet. Today you pick from BAAI's BF16 original (54.7 GB), prithivMLmods' FP8 (35.9 GB), cyankiwi's AWQ INT4 (20.2 GB) or numsu's INT4 W4A16 (18.6 GB). - The drafter was trained against another model. Speculative decoding has a small drafter propose tokens that the big model then verifies in one pass; it only pays off when most guesses are accepted. z-lab's DFlash2 drafter targets base Qwen3.8-27B, and a fine-tune changes which tokens the model favors, so nobody knows its hit rate on AREX-2. numsu's INT4 card does publish speeds with a DFlash2 drafter on two RTX 3090s, but without an acceptance length. - mamba_ssm_dtype. AREX-2 asks for float32. Our base command forced bfloat16, fine on Qwen3.8, untried on a model tuned for long reflective runs. - Parsers are a guess. These sglang options turn raw model text into separate reasoning and tool-call fields. qwen3 and qwen3_coder ought to work since the template matches. No sglang run of AREX-2 has been published. - Pick long context or speculation. The base model's 1M context is a YARN override that stretches its native 262,144 positions, and DFlash2 would not run with it. So: 1M without a drafter, or 256K with one. FP8 on Ampere is the expensive mistake The RTX A6000 is Ampere and has no FP8 hardware, so sglang keeps the weights in FP8 and unpacks them on the fly through Marlin kernels. Base Qwen3.8-27B decoded there at 24 tok/s. RedHatAI's INT4 W4A16 quant, same card, same 1M context: 41 tok/s. prithivMLmods' AREX-2 FP8 card agrees that full speed needs Ada, Hopper or later. An A100 is Ampere too, so reach for INT4 there. Weights are not the whole bill. In our base-model INT4 run at 1M context, sglang took about 46 GB, of which only 17.8 GB was weights. Another 11.5 GB held state for the linear attention layers, and 12.8 GB went to the KV cache, the per-token memory the model keeps so it does not reread the whole context on every step. The per-GPU sizing for AREX-2 goes card by card from 80 GB to 24 GB, as estimates. What to measure on the first run Start dull: BF16 on an 80 GB card, the native 256K context, no speculative flags, the same parsers, one long agent task. Then change a single thing per run, like a soundcheck that starts with the kick drum. Watch: - tool calls parsing cleanly late in a long session, given the tokenizer change - decode tok/s against the base model on the same card, which the shared architecture says should match - acceptance length if you add a drafter, the number nobody has published - GPU hours per task, since the benchmark protocol allows up to 1,500 model calls and 24 hours per case That last item drives the economics. AREX-2's self-reported 70.7 on Frontier-CS and 84.0 on BrowseComp come from that long protocol in BAAI's evaluation repo, with no independent reproduction we could find. Renting a GPU by the hour beats per-token pricing only while the card has work queued, for instance several cases side by side. For the rest of a single-GPU agent setup, see our earlier post on an 80B coding model on one rented A100 driven by Claude Code. This is a plausible one-card project thanks to BAAI and the Qwen team for the models, prithivMLmods, cyankiwi, numsu, RedHatAI and RadixArk for the quants, z-lab for DFlash2, numsu again for an experimental MTP head trained on AREX-2 outputs, AI Search for flagging the release, and the sglang and vLLM projects. The config tables, the base-model command verbatim, the worker settings for slow warmups and the full limits list are in the complete AREX-2 on one GPU write-up.
InfoVis: Unified Slide Image Generation and Editing Generating and editing presentation slides requires models to combine accurate text rendering, instruction following, visual grounding, aesthetic design, and precise editing within one canvas. Existing work covers only subsets of these capabilities and lacks large-scale slide-native supervision and diagnostic evaluation. We formulate four slide-centric tasks and build a scalable data engine with over 13M continual-pretraining records and 1.9M supervised fine-tuning records. We also introduce VisSlideBench, a benchmark of 1,527 manually screened instances that covers all four tasks with task-specific, multidimensional evaluation. Its automatic rankings agree with blinded, dimension-wise human rankings. Using these resources, we train InfoVis, a unified 9B slide generation and editing model. On VisSlideBench, InfoVis leads every evaluated open-source model on all eleven metrics and scores 7.94 overall against 7.99 for Nano-Banana-2. Starting from FLUX.2-klein-base-9B, slide-specific post-training adds 1.6 points overall, with the largest generation gains in content faithfulness; the remaining gap to proprietary models is smallest in content faithfulness and largest in aesthetic quality. Built on InfoVis, our document-to-slide workflow achieves the second-highest Aesthetics score on the external SlidesGen-Bench. We will release the model, code, data, and benchmark. Comments: Code: VectorSpaceLab/InfoVis
Play it free: neuralloot.github.io Paper (AgentSWE, arXiv 2610.13361): arxiv.org Code: VectorSpaceLab/AgentSWE
Can coding agents actually create, edit, or optimize usable agents? Current frontier models score poorly. Let em cook. arxiv.org VectorSpaceLab/AgentSWE
Paper: paperswithcode.co Code: VectorSpaceLab/AgentSWE Finding: builders show working agents, then deliver hollow runtimes and unsupported completion claims.
AgentSWE: Can Coding Agents Build the Agent You Actually Want? Jiahao Wang, Hongjin Qian, Yuyang Hu, Jiajun Zhang, Zheng Liu arxiv.org [𝚌𝚜.𝚂𝙴 𝚌𝚜.𝙰𝙸 𝚌𝚜.𝙻𝙶] 💬Code: VectorSpaceLab/AgentSWE t.co
AgentSWE: Can Coding Agents Build the Agent You Actually Want? As the production of software passes into the hands of software itself, coding agents are increasingly commissioned to build agents fitted to their users' own workflows. This delegation carries a quiet assumption, that the delivered agent will faithfully serve the requirement it was built for, rather than merely appear to. No existing benchmark tests that assumption. We introduce AgentSWE, a benchmark for agent software engineering spanning three stages of the agent development lifecycle: Creation of a complete agent from a natural-language requirement, Editing of production agent codebases, and Optimization toward a measurable target. Each of its 25 tasks is framed as a commission rather than an exam: the builder iterates against a handful of illustrative cases, while acceptance exercises the frozen submission on held-out cases designed to span the full requirement, grading behavior against requirement-derived rubrics under hard pass/fail gates or, for Optimization, against the source benchmark's own objective. Evaluating a range of coding agents across models and harnesses, we find that none comes close to delivering what a requirement asks for: the gap between what a builder demonstrates and what it delivers is the norm, not the corner case. An accompanying empirical analysis of the failure trajectories identifies recurring failure modes, among them overfitting to the visible cases, hollow runtimes behind code that looks complete, and unsupported claims of completion, each a signal by which coding agents can learn to build the agents they are asked for. Comments: Code: VectorSpaceLab/AgentSWE