Potential answer-distribution bias in several EventBench tasks

[HYLZ-2019]

Hi, thank you for releasing EventBench.

While analyzing the annotations, we noticed potential answer-distribution bias in several tasks:

• Detailed Understanding (DU): The correct answer is heavily concentrated on option B. A fixed-answer baseline that always selects B gets 257/483 = 53.21% accuracy. We also noticed that this task appears to mix questions with different numbers of options, including 3-choice and 5-choice questions.
• Causal Reasoning (CR): The correct answer is heavily concentrated on option A. A fixed-answer baseline that always selects A gets 119/180 = 66.11% accuracy.
• Spatial Relationship (SR): The label distribution is imbalanced. A fixed-answer baseline that always predicts left gets 109/301 = 36.21% accuracy.
• Object Counting (OC): The count distribution is strongly imbalanced. A fixed-answer baseline that always predicts 2 gets 253/414 = 61.11% accuracy.

These fixed-answer baselines suggest that models may obtain non-trivial scores without relying on event understanding, especially for CR, DU, and OC.

If these issues are confirmed after checking the annotations, would it be possible to release an updated version of the benchmark? For the multiple-choice tasks, one possible fix is to shuffle the answer options and regenerate the questions/annotations so that the correct option positions are more balanced.

Thanks again for your work on this benchmark.

[XduSyL]

Thank you for your detailed feedback and analysis of EventBench. We appreciate your contribution to making this benchmark more robust.

Evaluation Validity: It is important to clarify that EventBench is a test set, not a training set. For a normally pre-trained model, the evaluation remains valid even if the option distribution is not perfectly balanced, as such models do not inherently tend to bias toward a single repeating option.

Commitment to Fairness: We sincerely thank you for pointing this out. To ensure the highest level of fairness and prevent models from exploiting statistical shortcuts, we will shuffle the answer options and rebalance the distribution as you suggested.

Future Updates: While we emphasize that this does not undermine current evaluation results, we agree that a balanced distribution will improve the benchmark's rigor. We will release an updated version of EventBench after verifying the annotations and re-shuffling the options.

Thank you again for your support and for helping us improve EventBench!

[HYLZ-2019]

Thank you very much for the thoughtful reply and for the plan to verify the annotations and release a re-shuffled version. I agree that EventBench remains useful as a test set for event understanding. I would also like to share one additional controlled experiment, since it suggests that option placement can still affect the measured accuracy for some multiple-choice tasks.

We created a copy of the EventBench multiple-choice annotations and shuffled the answer options with a fixed random seed. The option contents were randomly permuted per question, and the gold answer letter was updated accordingly. We then evaluated EventGPT+ 2B with the same evaluation pipeline.

Task	Most frequent option	Max option share	Accuracy before shuffling	Accuracy after shuffling	Change
Gesture Recognition	C	27.07%	54.47%	55.48%	+1.01 pp
Action Recognition	C	30.41%	80.27%	79.45%	-0.82 pp
Detailed Understanding	B	53.21%	78.26%	67.08%	-11.18 pp
Causal Reasoning	A	66.11%	80.00%	71.67%	-8.33 pp

The two tasks with relatively balanced answer positions are almost unchanged after shuffling. In contrast, Detailed Understanding and Causal Reasoning, where the gold answers are concentrated on one option, show clear drops. This does not mean the benchmark is invalid, but it suggests that the original option ordering may have inflated the scores for tasks with strong answer-position imbalance.

This is also consistent with prior work on LLM multiple-choice and selection bias. Zheng et al. (ICLR 2024) show that LLMs are not robust MCQ selectors and can prefer particular option IDs such as A/B/C/D. Yang et al. (NAACL 2025 main) show that changing only the option symbols can cause large MCQA accuracy changes. More directly on MCQ generation, Tang et al. report that LLM-generated MCQs can exhibit systematic, model-specific correct-answer position biases.

So even for a test-only benchmark, balancing or randomly shuffling option positions is useful for making the reported results easier to interpret. Thanks again for taking this seriously and for improving EventBench.

References:

• Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2024. Large Language Models Are Not Robust Multiple Choice Selectors. ICLR 2024. https://openreview.net/forum?id=shr9PXz7T0
• Yang et al. 2025. Option Symbol Matters: Investigating and Mitigating Multiple-Choice Option Symbol Bias of Large Language Models. NAACL 2025. https://aclanthology.org/2025.naacl-long.95/
• Xuemei Tang, Xufeng Duan, and Zhenguang G. Cai. 2026. Do Large Language Models Plan Answer Positions? Position Bias in Multiple-Choice Question Generation. arXiv:2605.01846. https://arxiv.org/html/2605.01846v1

[XduSyL]

Thank you for your thoughtful feedback and the detailed experimental results. Our primary goal in open-sourcing this work is to collaborate with the community to continuously refine it.

We are very pleased to receive such constructive suggestions and references. We believe this contribution is highly beneficial for establishing EventBench as a robust infrastructure for the event understanding community. To address the potential variance mentioned, we will incorporate the random shuffling of choices as suggested to ensure a more balanced and rigorous evaluation.

We look forward to working with you and other researchers to further improve this field. Thanks again for your contribution.

[XduSyL]

We are pleased to inform the community that we have fully completed the option uniformization for the Detailed Understanding and Causal Reasoning tasks to guarantee absolutely fair and unbiased evaluation. All answer positions in these two tasks have been rebalanced to a strictly uniform distribution, completely eliminating the potential score inflation caused by the original answer-position imbalance identified in your experiments.

Corresponding updated evaluations for all tracks have been successfully deployed on the Codabench platform:

• Event Understanding Track: https://www.codabench.org/competitions/16100/
• Spatial Reasoning Track: https://www.codabench.org/competitions/16303/

The test servers will officially launch for public evaluation on May 20, 2026. We sincerely welcome researchers from the community to participate in the benchmarking and provide further feedback to help us continuously improve EventBench.