A Japanese Benchmark for Evaluating Social Bias in Reasoning Based on Attribution Theory
arXiv:2604.00568v1 Announce Type: new Abstract: In enhancing the fairness of Large Language Models (LLMs), evaluating social biases rooted in the cultural contexts of specific linguistic regions is essential. However, most existing Japanese benchmarks heavily rely on translating English data, which does not necessarily provide an evaluation suitable for Japanese culture. Furthermore, they only evaluate bias in the conclusion, failing to capture biases lurking in the reasoning. In this study, based on attribution theory in social psychology, we constructed a new dataset, ``JUBAKU-v2,'' which evaluates the bias in attributing behaviors to in-groups and out-groups within reasoning while fixing the conclusion. This dataset consists of 216 examples reflecting cultural biases specific to Japan. Experimental results verified that it can detect performance differences across models more sensitively than existing benchmarks.
arXiv:2604.00568v1 Announce Type: new Abstract: In enhancing the fairness of Large Language Models (LLMs), evaluating social biases rooted in the cultural contexts of specific linguistic regions is essential. However, most existing Japanese benchmarks heavily rely on translating English data, which does not necessarily provide an evaluation suitable for Japanese culture. Furthermore, they only evaluate bias in the conclusion, failing to capture biases lurking in the reasoning. In this study, based on attribution theory in social psychology, we constructed a new dataset, ``JUBAKU-v2,'' which evaluates the bias in attributing behaviors to in-groups and out-groups within reasoning while fixing the conclusion. This dataset consists of 216 examples reflecting cultural biases specific to Japan. Experimental results verified that it can detect performance differences across models more sensitively than existing benchmarks.
Executive Summary
This article proposes a novel Japanese benchmark, JUBAKU-v2, to evaluate social biases in reasoning based on attribution theory. The benchmark focuses on cultural biases specific to Japan and assesses bias in attributing behaviors to in-groups and out-groups within reasoning while keeping the conclusion fixed. The dataset consists of 216 examples and is more sensitive to performance differences across models than existing benchmarks. The results demonstrate the effectiveness of JUBAKU-v2 in evaluating social biases in Japanese cultural contexts, highlighting the need for culturally specific benchmarks in enhancing the fairness of Large Language Models (LLMs).
Key Points
- ▸ The article proposes a novel Japanese benchmark, JUBAKU-v2, for evaluating social biases in reasoning.
- ▸ The benchmark assesses bias in attributing behaviors to in-groups and out-groups within reasoning while keeping the conclusion fixed.
- ▸ The dataset consists of 216 examples reflecting cultural biases specific to Japan and is more sensitive to performance differences across models than existing benchmarks.
Merits
Strength
The JUBAKU-v2 benchmark provides a culturally specific evaluation of social biases in Japanese cultural contexts, addressing the limitations of existing benchmarks that heavily rely on translated English data.
Strength
The benchmark's focus on attributing behaviors to in-groups and out-groups within reasoning captures biases lurking in the reasoning process, which existing benchmarks often fail to detect.
Demerits
Limitation
The dataset's size and scope may be limited, and further research is needed to validate the benchmark's effectiveness across different cultural contexts.
Limitation
The article does not provide a comprehensive analysis of the potential impact of cultural biases on LLMs' performance in real-world applications.
Expert Commentary
The article's contribution to the field of AI fairness and accountability is significant, as it addresses the limitations of existing benchmarks and provides a culturally specific evaluation of social biases. However, further research is needed to validate the benchmark's effectiveness across different cultural contexts and to explore its potential impact on real-world applications. The JUBAKU-v2 benchmark has the potential to improve the fairness and accountability of LLMs in Japanese cultural contexts, but its limitations should not be overlooked.
Recommendations
- ✓ Future research should focus on expanding the dataset and validating the benchmark's effectiveness across different cultural contexts.
- ✓ Developers and policymakers should prioritize the development of culturally specific benchmarks for evaluating social biases in AI systems, particularly in the context of LLMs.
Sources
Original: arXiv - cs.CL
