FINDINGS OF EMNLP 2026NLP × SOCIAL BIAS

SCENE: A Scenario Completion Benchmark for Measuring Social Bias in Large Language Models

Social bias depends on the task—and on how a model is asked to respond. SCENE measures it through scenario completion, with structured outputs and no LLM judge.

Migyeong Kang1*Aecheon Jung1*Eunchae Jang2*Minseok Kim1S. Shyam Sundar1,2Sungeun Hong1†YunSeok Choi1†Jinyoung Han1†
1 Sungkyunkwan University2 The Pennsylvania State University* Equal contribution   † Corresponding authors
75,155 ↗Korean benchmark instances
376Curated seed templates
2Scenario-completion tasks
8Language models evaluated

Why SCENE?

Multiple-choice questions are easy to score. But they may not capture how stereotypes emerge when a model assigns roles or completes a story.

Previous approach

Multiple-choice questions

Ask a model to choose an answer in a stereotypical context.

Widely used and straightforward to quantify.

What MCQ may miss

Two factors that change bias

Task context
What the model is asked to do.
Output constraints
How the model is allowed to respond.
Our approach

Scenario completion

Assign social attributes to characters, or choose what happens next.

Two complementary tasks with directly measurable outputs.

Closer to scenario-based use. Still directly quantifiable. SCENE measures bias from structured outputs without an auxiliary LLM judge.

Read the abstract

Most existing benchmarks evaluate social bias in large language models (LLMs) using multiple-choice questions (MCQ) grounded in stereotypical contexts. While MCQ formats enable straightforward quantification, how LLMs manifest social bias in task-specific contexts and under different output constraints remains underexplored. We introduce SCENE, a benchmark for measuring the social bias shown by LLMs in scenario-completion tasks. Given short scenarios involving social groups, models must complete key elements of the situation through two tasks: (i) Character Specification, which asks models to assign social attributes to characters, and (ii) Plot Continuation, which asks models to select the most plausible narrative continuation. Combining templates adapted from prior benchmarks with newly collected templates through a university-wide competition, we construct a dataset of over 75,000 Korean instances. Experiments show that bias measurements are highly sensitive to task context and output constraints, even for identical underlying scenarios, suggesting that widely used MCQ-based evaluations may not fully capture certain manifestations of stereotypes.

Two ways to complete a scenario.

Both tasks start from the same stereotypical context. They differ in what the model completes and how it responds.

Constrained generation

Character Specification

Assign a social attribute to each anonymous character based on their role in the scenario.

Option selection

Plot Continuation

Select the most plausible continuation from biased, neutral, and counter-stereotypical alternatives.

The same context, two evaluation tasks

The paper's task diagram: a taxi-app scenario becomes Character Specification on the left and Plot Continuation on the right, each showing a stereotype-consistent output.

Shared context: a youth and an elderly person try to call a taxi using an app.

Character Specification panel from the paper: A struggles with the app and B helps. The stereotype-consistent output assigns Elderly to A and Youth to B.
Plot Continuation panel from the paper: choose between a neutral continuation and two role-reversed continuations. The stereotype-consistent option has the elderly person struggle and the youth help.
The paper’s age-bias example. Both tasks produce structured outputs that can be scored directly.

Explore how the outputs are scored

Change an assignment or continuation to see the difference between biased, neutral, and counter-stereotypical outputs.

Interactive scoring exampleAge bias · English translation
Explore the possible outputs
BR · Stereotype-consistent

This assignment follows the age stereotype encoded in this template.

A: Elderly   B: Youth

Change either attribute to see how the output is scored.

BRStereotype-consistent
NRNeutral
CRCounter-stereotypical

Three measurable outcomes. No external LLM judge.

Building 75K+ Korean instances.

Combine established bias templates with contemporary scenarios collected through Bias-A-Thon, a university-wide competition.

269

KoBBQ templates

Established social-bias categories

107

Bias-A-Thon templates

Newly collected scenarios from 23 teams

376

Seed templates

Two complementary sources

From source templates to diverse scenarios

The paper's complete pipeline: KoBBQ and competition sources, multiple-choice templates, two scenario-completion tasks, and augmentation through context replacement, entity refinement, and choice paraphrasing.
Templates are reformulated into both tasks, then expanded by varying context, social-group expressions, and choice wording.
  1. Reframe as two tasksCharacter Specification and Plot Continuation.
  2. Review with three annotatorsCheck scenarios, choices, and stereotype labels.
  3. Apply three-stage augmentationContext replacement ×5, entity refinement ×5, and choice paraphrasing ×2.
  4. Control order and filterSwap entity order ×2 and remove invalid or low-quality instances.
Final Korean benchmark75,155instances
Character Specification37,562Plot Continuation37,593

Contemporary templates expand what gets measured. Bias-A-Thon adds underrepresented contexts and new bias categories: occupation, personality, and technology.

Three questions about LLM bias.

Task format, output constraints, and template source all shape the bias we observe.

Q1 · Overall bias

How much bias do LLMs show on SCENE?

7/8

models show higher biased rates in Character Specification

Biased rate by model

Percentage of stereotype-consistent outputs

Character SpecificationPlot Continuation

Exact means and standard deviations are in the expandable data table below.

Mean over 10 runs · all SCENE templates · percentage scale 0–100Download CSV
View all reported values and standard deviations
Overall evaluation · mean ± standard deviation (%)
Model Character Specification Plot Continuation
BR NR CR BR NR CR

Mean ± standard deviation across 10 evaluation runs. Rates may not sum to 100% because of invalid responses and rounding.

↳ Assigning attributes to characters generally elicits more stereotypical responses than selecting a plot continuation. Llama-3.1-8B is the exception.

Q2 · Task format and output constraints

Does task format influence bias manifestation?

Task context changes the measurement.

On the same KoBBQ-derived scenarios, SCENE tasks elicit higher biased rates and lower neutral rates than MCQ.

For GPT-5.1, moving from MCQ to Character Specification raises BR from 14.46% to 66.75% and lowers NR from 82.58% to 12.45%.

Output constraints change it again.

Keeping Character Specification but switching from generation to selection shifts how models respond.

66.75% → 33.61%GPT-5.1 biased rate: generation → selection

This comparison uses KoBBQ-derived templates only. Q1 reports the complete benchmark.

BiasedNeutralCounter

Reported BR / NR / CR (%) · unfilled remainder: invalid responses / rounding

Each format reveals a distinct bias pattern.

Template-level correlations are low to moderate—even when the underlying scenarios are identical.

Correlations across task formats

The paper's Pearson correlation plot with 95% confidence intervals. Across GPT-5.1, Qwen3-8B, EXAONE-7.8B, and Llama-3.1-8B, task-format pairs show low-to-moderate correlations.
MCQ: multiple-choice questions · CS: Character Specification · PC: Plot Continuation · CS(sel): selection-based Character Specification. Error bars show 95% confidence intervals.

Identical scenarios do not guarantee identical bias measurements. Both task framing and output constraints matter.

Q3 · Template source

Does template source influence bias elicitation?

Newly collected templates trigger bias more strongly. In Character Specification, competition-derived scenarios generally produce higher biased rates and lower neutral rates than KoBBQ-derived scenarios.

KoBBQ vs. competition templates

The paper's category-level scatter plots. KoBBQ is on the horizontal axis and Competition on the vertical axis. Most biased-rate points are above the diagonal; most neutral-rate points are below it.
Character Specification, averaged across models. Above the diagonal means a higher rate on competition templates. The source effect is smaller for Plot Continuation.

Bias benchmarks need contemporary scenarios. New contexts can expose stereotypes that established templates capture less strongly.

Wording matters as well: entity refinement has the largest effect among augmentation stages in Character Specification, underscoring the value of diverse expressions for social groups.

Bias depends on how we measure it.

01

Task framing and output constraints matter. They change the bias that models express.

02

Contemporary scenarios reveal more bias. Newly collected templates elicit stronger stereotypical role assignments.

03

Diverse, updated benchmarks are essential. Evaluation should evolve with language and social contexts.

SCENE evaluates structured scenario completion in Korean. Its results should be interpreted within these tasks and linguistic contexts.

Cite SCENE

@inproceedings{kang-etal-2026-scene,
    title = "{SCENE}: A Scenario Completion Benchmark for Measuring Social Bias in Large Language Models",
    author = "Kang, Migyeong  and
      Jung, Aecheon  and
      Jang, Eunchae  and
      Kim, Minseok  and
      Sundar, S. Shyam  and
      Hong, Sungeun  and
      Choi, YunSeok  and
      Han, Jinyoung",
    editor = "Lee, Hung-Yi  and
      Dev, Sunipa  and
      Li, Jessy  and
      Elliott, Desmond",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {EMNLP} 2026",
    month = oct,
    year = "2026",
    address = "Budapest, Hungary",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.findings-emnlp.1092/",
    doi = "10.18653/2026.findings-emnlp.1092",
    pages = "23126--23148",
    ISBN = "979-8-89176-462-0"
}

Research figure