Paper: "System1-Arena: Which Decisions Can Be Left to System 1?" by Tianyu Chen, Xinyang Liu, Ruyi An, Zhendong Wang, Mingyuan Zhou (UT Austin)
arxiv.org
PDF: arxiv.org
Code: TianyuCodings/S1-Arena
Leaderboard: tianyucodings.github.io
Numbers from Table 1 (81.1% / 72.6% / 71.2%, $10.38 vs $0.052 per 1,000, 321 vs 2,293 ms, 78,362 questions over 87 tasks) and Table 2 (32.9% best loop success; 67.8% step accuracy for Jev, Decision-2.0-Vega-27B and Qwen3.5-9B with 17.4% / 26.5% / 5.7% loop success, 127 environments). Caveats in Sec. 7.
10 views
System1-Arena: Which Decisions Can Be Left to System 1?
Large language models (LLMs) often make discrete decisions, such as flagging phishing or escalating to a human. System-1 models, such as Jev and the OpenAI Decisions API, return probabilities over options in a single forward pass, while System-2 approaches generate reasoning steps. Which decisions can System-1 models handle, and which require generative reasoning? We introduce System1-Arena, a benchmark that gives all models the same decision requests and scores their probabilities. It contains 92k questions across 105 tasks from 268 public sources, in four groups: text classification and knowledge, applied decisions, evaluation and control, and agent and action. It also includes 138 closed-loop environments, from web shopping and SWE-bench coding control to robotics, games, and driving; in real-time mode, the world evolves while a model decides. We evaluate 30 models: 7 closed LLMs, 13 open LLMs read at the answer token, and 10 System-1 models. The largest System-1 models match or exceed fast closed LLMs in non-agentic accuracy, with better-calibrated probabilities at one-quarter of the latency. A reasoning LLM is the most accurate, at about 70x the cost of a fast LLM. In closed loops, large open LLMs succeed most often (up to 32.9%); small System-1 models rarely succeed.
Comments: 40 pages. Code: TianyuCodings/S1-Arena ; leaderboard: tianyucodings.github.io ; dataset: huggingface.co
Mentions
System1-Arena: Which Decisions Can Be Left to System 1? Tianyu Chen, Xinyang Liu, Ruyi An, Zhendong Wang, Mingyuan Zhou arxiv.org [𝚌𝚜.𝙲𝙻 𝚌𝚜.𝙰𝙸 𝚌𝚜.𝙻𝙶] 💬Code: TianyuCodings/S1-Arena t.co
Paper: "System1-Arena: Which Decisions Can Be Left to System 1?" by Tianyu Chen, Xinyang Liu, Ruyi An, Zhendong Wang, Mingyuan Zhou (UT Austin) arxiv.org PDF: arxiv.org Code: TianyuCodings/S1-Arena Leaderboard: tianyucodings.github.io Numbers from Table 1 (81.1% / 72.6% / 71.2%, $10.38 vs $0.052 per 1,000, 321 vs 2,293 ms, 78,362 questions over 87 tasks) and Table 2 (32.9% best loop success; 67.8% step accuracy for Jev, Decision-2.0-Vega-27B and Qwen3.5-9B with 17.4% / 26.5% / 5.7% loop success, 127 environments). Caveats in Sec. 7.
System1-Arena: Which Decisions Can Be Left to System 1? Large language models (LLMs) often make discrete decisions, such as flagging phishing or escalating to a human. System-1 models, such as Jev and the OpenAI Decisions API, return probabilities over options in a single forward pass, while System-2 approaches generate reasoning steps. Which decisions can System-1 models handle, and which require generative reasoning? We introduce System1-Arena, a benchmark that gives all models the same decision requests and scores their probabilities. It contains 92k questions across 105 tasks from 268 public sources, in four groups: text classification and knowledge, applied decisions, evaluation and control, and agent and action. It also includes 138 closed-loop environments, from web shopping and SWE-bench coding control to robotics, games, and driving; in real-time mode, the world evolves while a model decides. We evaluate 30 models: 7 closed LLMs, 13 open LLMs read at the answer token, and 10 System-1 models. The largest System-1 models match or exceed fast closed LLMs in non-agentic accuracy, with better-calibrated probabilities at one-quarter of the latency. A reasoning LLM is the most accurate, at about 70x the cost of a fast LLM. In closed loops, large open LLMs succeed most often (up to 32.9%); small System-1 models rarely succeed. Comments: 40 pages. Code: TianyuCodings/S1-Arena ; leaderboard: tianyucodings.github.io ; dataset: huggingface.co