Span Modeling for Idiomaticity and Figurative Language Detection with Span Contrastive Loss
arXiv:2603.22799v1 Announce Type: new Abstract: The category of figurative language contains many varieties, some of which are non-compositional in nature. This type of phrase or multi-word expression (MWE) includes idioms, which represent a single meaning that does not consist of the sum of its words. For language models, this presents a unique problem due to tokenization and adjacent contextual embeddings. Many large language models have overcome this issue with large phrase vocabulary, though immediate recognition frequently fails without one- or few-shot prompting or instruction finetuning. The best results have been achieved with BERT-based or LSTM finetuning approaches. The model in this paper contains one such variety. We propose BERT- and RoBERTa-based models finetuned with a combination of slot loss and span contrastive loss (SCL) with hard negative reweighting to improve idiomaticity detection, attaining state of the art sequence accuracy performance on existing datasets. Co
arXiv:2603.22799v1 Announce Type: new Abstract: The category of figurative language contains many varieties, some of which are non-compositional in nature. This type of phrase or multi-word expression (MWE) includes idioms, which represent a single meaning that does not consist of the sum of its words. For language models, this presents a unique problem due to tokenization and adjacent contextual embeddings. Many large language models have overcome this issue with large phrase vocabulary, though immediate recognition frequently fails without one- or few-shot prompting or instruction finetuning. The best results have been achieved with BERT-based or LSTM finetuning approaches. The model in this paper contains one such variety. We propose BERT- and RoBERTa-based models finetuned with a combination of slot loss and span contrastive loss (SCL) with hard negative reweighting to improve idiomaticity detection, attaining state of the art sequence accuracy performance on existing datasets. Comparative ablation studies show the effectiveness of SCL and its generalizability. The geometric mean of F1 and sequence accuracy (SA) is also proposed to assess a model's span awareness and general performance together.
Executive Summary
This article proposes a novel approach to detecting idiomaticity and figurative language using span contrastive loss (SCL) with hard negative reweighting. Building on existing BERT-based and RoBERTa-based models, the authors finetune these models with a combination of slot loss and SCL to achieve state-of-the-art sequence accuracy performance on existing datasets. The proposed method demonstrates improved idiomaticity detection and generalizability, as shown through comparative ablation studies. Additionally, the authors introduce a new evaluation metric, the geometric mean of F1 and sequence accuracy (SA), to assess a model's span awareness and overall performance. The method's effectiveness and generalizability make it a valuable contribution to the field of natural language processing, particularly in the area of figurative language detection.
Key Points
- ▸ Proposed a novel approach to detecting idiomaticity and figurative language using span contrastive loss (SCL) with hard negative reweighting.
- ▸ Finetuned BERT-based and RoBERTa-based models with a combination of slot loss and SCL to achieve state-of-the-art sequence accuracy performance.
- ▸ Introduced a new evaluation metric, the geometric mean of F1 and sequence accuracy (SA), to assess a model's span awareness and overall performance.
Merits
Improved Idiomaticity Detection
The proposed method demonstrates improved idiomaticity detection and generalizability, outperforming existing methods on existing datasets.
Generalizability
The method's effectiveness and generalizability make it a valuable contribution to the field of natural language processing, particularly in the area of figurative language detection.
Demerits
Limited Evaluation
The evaluation of the proposed method is limited to existing datasets and may not generalize to other domains or languages.
Computational Complexity
The proposed method may require significant computational resources due to the finetuning process and the use of SCL with hard negative reweighting.
Expert Commentary
The proposed method is a significant contribution to the field of natural language processing, particularly in the area of figurative language detection. The use of span contrastive loss with hard negative reweighting is a novel approach that demonstrates improved idiomaticity detection and generalizability. However, the evaluation of the proposed method is limited to existing datasets, and the computational complexity of the method may be a concern. Despite these limitations, the proposed method has the potential to be widely adopted in various natural language processing tasks and may have significant implications for policy-making and decision-making.
Recommendations
- ✓ Future research should focus on extending the proposed method to other domains and languages to evaluate its generalizability.
- ✓ The computational complexity of the proposed method should be addressed through the development of more efficient algorithms or the use of more powerful hardware.
Sources
Original: arXiv - cs.CL
