rankui

Rethinking Opinion Measurement Interfaces for Human-in-the-Loop Optimization Changkun Ou, Sven Mayer, Daniel Buschek, Andreas Butz arxiv.org [𝚌𝚜.𝙷𝙲 𝚌𝚜.𝙻𝙶] 💬Code: changkun/rankui t.co

Rethinking Opinion Measurement Interfaces for Human-in-the-Loop Optimization Human-in-the-loop optimization refers to systems that present choices to a human decision-maker and query their judgment to optimize system parameters iteratively. The user interface typically provides either an absolute rating scale or asks for a preferred option. However, these approaches can potentially suffer from calibration problems, inconsistencies, and completeness violations of user preferences in such a sequential evaluation. This paper revisits the design space of opinion measurement interfaces for human-in-the-loop optimization. We conducted a study (N=360) using text summarization and image color enhancement tasks to evaluate human sequential feedback in six representative opinion measurement interfaces. Based on our analysis, we recommend using a listwise approach to support sequential opinion measurement, which counterweighs individual rating and preferential choice limitations and conveys more information for subsequent use. We further discuss the trade-offs in different user interface designs and provide guidelines to inform future research. Comments: 29 pages. Written in 2022, published as last revised. Dataset, analysis scripts, and system source code: changkun/rankui