PMIScore: An Unsupervised Approach to Quantify Dialogue Engagement
arXiv:2603.13796v1 Announce Type: new Abstract: High dialogue engagement is a crucial indicator of an effective conversation. A reliable measure of engagement could help benchmark large language models, enhance the effectiveness of human-computer interactions, or improve personal communication skills. However, quantifying engagement is challenging, since it is subjective and lacks a "gold standard". This paper proposes PMIScore, an efficient unsupervised approach to quantify dialogue engagement. It uses pointwise mutual information (PMI), which is the probability of generating a response conditioning on the conversation history. Thus, PMIScore offers a clear interpretation of engagement. As directly computing PMI is intractable due to the complexity of dialogues, PMIScore learned it through a dual form of divergence. The algorithm includes generating positive and negative dialogue pairs, extracting embeddings by large language models (LLMs), and training a small neural network using a
arXiv:2603.13796v1 Announce Type: new Abstract: High dialogue engagement is a crucial indicator of an effective conversation. A reliable measure of engagement could help benchmark large language models, enhance the effectiveness of human-computer interactions, or improve personal communication skills. However, quantifying engagement is challenging, since it is subjective and lacks a "gold standard". This paper proposes PMIScore, an efficient unsupervised approach to quantify dialogue engagement. It uses pointwise mutual information (PMI), which is the probability of generating a response conditioning on the conversation history. Thus, PMIScore offers a clear interpretation of engagement. As directly computing PMI is intractable due to the complexity of dialogues, PMIScore learned it through a dual form of divergence. The algorithm includes generating positive and negative dialogue pairs, extracting embeddings by large language models (LLMs), and training a small neural network using a mutual information loss function. We validated PMIScore on both synthetic and real-world datasets. Our results demonstrate the effectiveness of PMIScore in PMI estimation and the reasonableness of the PMI metric itself.
Executive Summary
The article introduces PMIScore, an unsupervised framework for quantifying dialogue engagement using pointwise mutual information (PMI). By leveraging PMI as a proxy for engagement, the authors propose an innovative, scalable method to evaluate conversational effectiveness without relying on subjective annotations or a gold standard. The approach employs dual divergence to approximate PMI through generated positive and negative dialogue pairs, utilizing LLMs for embeddings and a neural network trained via a mutual information loss function. Validation on synthetic and real-world datasets supports the feasibility and interpretability of the PMI metric. This work addresses a critical gap in conversational AI evaluation by offering a quantitative, objective indicator of engagement.
Key Points
- ▸ PMIScore introduces an unsupervised PMI-based metric for dialogue engagement.
- ▸ Utilizes dual divergence to approximate PMI without direct computation.
- ▸ Validated on both synthetic and real-world datasets, demonstrating effectiveness.
Merits
Innovation
PMIScore fills a void in engagement measurement by offering an unsupervised, interpretable metric leveraging PMI.
Demerits
Computational Complexity
While efficient, the dual divergence approximation may introduce approximations that could affect precision in highly nuanced dialogues.
Expert Commentary
PMIScore represents a significant step forward in the evaluation of dialogue systems. The use of PMI as a proxy for engagement is theoretically sound and aligns with established information-theoretic principles. The authors' approach to circumventing the computational intractability of PMI via dual divergence is both clever and pragmatic. Importantly, the validation on real-world data adds credibility to the metric’s applicability beyond academic simulations. However, the authors should consider extending their evaluation to more diverse dialogue domains—particularly in high-stakes or culturally sensitive contexts—to further validate generalizability. Additionally, while the neural network’s role is clear, future iterations might explore alternative architectures or ensemble methods to improve robustness. Overall, PMIScore demonstrates the potential of information-theoretic metrics to transform the landscape of conversational evaluation.
Recommendations
- ✓ 1. Extend validation to cross-domain datasets to assess generalizability.
- ✓ 2. Explore hybrid approaches combining PMIScore with human-annotated metrics for complementary evaluation.