Unified Benchmark for Evaluating Performance, Bias, and Consistency in LLM Binary Question Answering

Abstract

Binary question answering is central to many real-world applications of large language models (LLMs), such as fact-checking or decision-making support. Yet, despite its prevalence and the high stakes of getting a binary judgment wrong (where an error yields the exact opposite outcome), there are no recent comprehensive benchmarks dedicated to evaluating LLM behavior on this task. To address this gap, we introduce a unified benchmark for assessing binary QA across three dimensions: performance, bias, and consistency. The benchmark is supported by a five-domain dataset augmented with new controlled reformulations of each question, including paraphrases, negations, and answer option variations. Across fifteen state-of-the-art LLMs, we find strong overall performance on the task, with larger and reasoning-optimized models showing better results than the smaller variants. At the same time, we observe pervasive No-leaning bias, universally weak consistency when handling semantically opposite questions, and substantial cross-domain variation. Reading comprehension and multi-hop reasoning topics are handled reliably, whereas numerical reasoning, ethical judgment, and, especially, translation evaluation remain challenging. These findings reveal both the strengths and shortcomings of current LLMs on binary QA, providing researchers with a basis for targeted future improvements while also helping practitioners make informed choices when deploying the models in binary decision contexts.