Multiple-choice questions
Ask a model to choose an answer in a stereotypical context.
Widely used and straightforward to quantify.
Social bias depends on the task—and on how a model is asked to respond. SCENE measures it through scenario completion, with structured outputs and no LLM judge.
Two people. One everyday situation.
What will the model fill in?
Multiple-choice questions are easy to score. But they may not capture how stereotypes emerge when a model assigns roles or completes a story.
Ask a model to choose an answer in a stereotypical context.
Widely used and straightforward to quantify.
Assign social attributes to characters, or choose what happens next.
Two complementary tasks with directly measurable outputs.
Most existing benchmarks evaluate social bias in large language models (LLMs) using multiple-choice questions (MCQ) grounded in stereotypical contexts. While MCQ formats enable straightforward quantification, how LLMs manifest social bias in task-specific contexts and under different output constraints remains underexplored. We introduce SCENE, a benchmark for measuring the social bias shown by LLMs in scenario-completion tasks. Given short scenarios involving social groups, models must complete key elements of the situation through two tasks: (i) Character Specification, which asks models to assign social attributes to characters, and (ii) Plot Continuation, which asks models to select the most plausible narrative continuation. Combining templates adapted from prior benchmarks with newly collected templates through a university-wide competition, we construct a dataset of over 75,000 Korean instances. Experiments show that bias measurements are highly sensitive to task context and output constraints, even for identical underlying scenarios, suggesting that widely used MCQ-based evaluations may not fully capture certain manifestations of stereotypes.
Both tasks start from the same stereotypical context. They differ in what the model completes and how it responds.
Assign a social attribute to each anonymous character based on their role in the scenario.
Select the most plausible continuation from biased, neutral, and counter-stereotypical alternatives.
Shared context: a youth and an elderly person try to call a taxi using an app.
Change an assignment or continuation to see the difference between biased, neutral, and counter-stereotypical outputs.
This assignment follows the age stereotype encoded in this template.
A: Elderly B: Youth
Change either attribute to see how the output is scored.
This continuation follows the age stereotype encoded in this template.
Three measurable outcomes. No external LLM judge.
Combine established bias templates with contemporary scenarios collected through Bias-A-Thon, a university-wide competition.
Established social-bias categories
Newly collected scenarios from 23 teams
Two complementary sources
Task format, output constraints, and template source all shape the bias we observe.
models show higher biased rates in Character Specification
Exact means and standard deviations are in the expandable data table below.
| Model | Character Specification | Plot Continuation | ||||
|---|---|---|---|---|---|---|
| BR | NR | CR | BR | NR | CR | |
Mean ± standard deviation across 10 evaluation runs. Rates may not sum to 100% because of invalid responses and rounding.
↳ Assigning attributes to characters generally elicits more stereotypical responses than selecting a plot continuation. Llama-3.1-8B is the exception.
On the same KoBBQ-derived scenarios, SCENE tasks elicit higher biased rates and lower neutral rates than MCQ.
For GPT-5.1, moving from MCQ to Character Specification raises BR from 14.46% to 66.75% and lowers NR from 82.58% to 12.45%.
Keeping Character Specification but switching from generation to selection shifts how models respond.
This comparison uses KoBBQ-derived templates only. Q1 reports the complete benchmark.
Reported BR / NR / CR (%) · unfilled remainder: invalid responses / rounding
Template-level correlations are low to moderate—even when the underlying scenarios are identical.
Newly collected templates trigger bias more strongly. In Character Specification, competition-derived scenarios generally produce higher biased rates and lower neutral rates than KoBBQ-derived scenarios.
Wording matters as well: entity refinement has the largest effect among augmentation stages in Character Specification, underscoring the value of diverse expressions for social groups.
Task framing and output constraints matter. They change the bias that models express.
Contemporary scenarios reveal more bias. Newly collected templates elicit stronger stereotypical role assignments.
Diverse, updated benchmarks are essential. Evaluation should evolve with language and social contexts.
SCENE evaluates structured scenario completion in Korean. Its results should be interpreted within these tasks and linguistic contexts.
@inproceedings{kang-etal-2026-scene,
title = "{SCENE}: A Scenario Completion Benchmark for Measuring Social Bias in Large Language Models",
author = "Kang, Migyeong and
Jung, Aecheon and
Jang, Eunchae and
Kim, Minseok and
Sundar, S. Shyam and
Hong, Sungeun and
Choi, YunSeok and
Han, Jinyoung",
editor = "Lee, Hung-Yi and
Dev, Sunipa and
Li, Jessy and
Elliott, Desmond",
booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {EMNLP} 2026",
month = oct,
year = "2026",
address = "Budapest, Hungary",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2026.findings-emnlp.1092/",
doi = "10.18653/2026.findings-emnlp.1092",
pages = "23126--23148",
ISBN = "979-8-89176-462-0"
}