Design and evaluation of an agentic workflow for crisis-related synthetic tweet datasets
arXiv:2603.13625v1 Announce Type: new Abstract: Twitter (now X) has become an important source of social media data for situational awareness during crises. Crisis informatics research has widely used tweets from Twitter to develop and evaluate artificial intelligence (AI) systems for various crisis-relevant tasks, such as extracting locations and estimating damage levels from tweets to support damage assessment. However, recent changes in Twitter's data access policies have made it increasingly difficult to curate real-world tweet datasets related to crises. Moreover, existing curated tweet datasets are limited to past crisis events in specific contexts and are costly to annotate at scale. These limitations constrain the development and evaluation of AI systems used in crisis informatics. To address these limitations, we introduce an agentic workflow for generating crisis-related synthetic tweet datasets. The workflow iteratively generates synthetic tweets conditioned on prespecified
arXiv:2603.13625v1 Announce Type: new Abstract: Twitter (now X) has become an important source of social media data for situational awareness during crises. Crisis informatics research has widely used tweets from Twitter to develop and evaluate artificial intelligence (AI) systems for various crisis-relevant tasks, such as extracting locations and estimating damage levels from tweets to support damage assessment. However, recent changes in Twitter's data access policies have made it increasingly difficult to curate real-world tweet datasets related to crises. Moreover, existing curated tweet datasets are limited to past crisis events in specific contexts and are costly to annotate at scale. These limitations constrain the development and evaluation of AI systems used in crisis informatics. To address these limitations, we introduce an agentic workflow for generating crisis-related synthetic tweet datasets. The workflow iteratively generates synthetic tweets conditioned on prespecified target characteristics, evaluates them using predefined compliance checks, and incorporates structured feedback to refine them in subsequent iterations. As a case study, we apply the workflow to generate synthetic tweet datasets relevant to post-earthquake damage assessment. We show that the workflow can generate synthetic tweets that capture their target labels for location and damage level. We further demonstrate that the resulting synthetic tweet datasets can be used to evaluate AI systems on damage assessment tasks like geolocalization and damage level prediction. Our results indicate that the workflow offers a flexible and scalable alternative to real-world tweet data curation, enabling the systematic generation of synthetic social media data across diverse crisis events, societal contexts, and crisis informatics applications.
Executive Summary
This arXiv paper proposes an agentic workflow for generating crisis-related synthetic tweet datasets to address the limitations of existing curated tweet datasets in crisis informatics. The workflow iteratively generates synthetic tweets, evaluates them, and refines them based on feedback. The authors demonstrate the workflow's effectiveness in generating synthetic tweets relevant to post-earthquake damage assessment, and show that these datasets can be used to evaluate AI systems on damage assessment tasks. This approach offers a flexible and scalable alternative to real-world tweet data curation, enabling the systematic generation of synthetic social media data across diverse crisis events and applications. The findings have significant implications for crisis informatics, AI system development, and social media data analysis.
Key Points
- ▸ The proposed agentic workflow addresses the limitations of existing curated tweet datasets in crisis informatics.
- ▸ The workflow iteratively generates synthetic tweets, evaluates them, and refines them based on feedback.
- ▸ The generated synthetic tweet datasets can be used to evaluate AI systems on damage assessment tasks.
Merits
Strength in Addressing Limitations
The proposed workflow effectively addresses the limitations of existing curated tweet datasets, including the difficulty of data access, cost of annotation, and limited scope of existing datasets.
Demerits
Limitation in Handling Complex Real-World Scenarios
The workflow may struggle to capture the complexity and nuance of real-world scenarios, which can lead to inaccurate or incomplete synthetic data.
Expert Commentary
The proposed agentic workflow is a significant contribution to the field of crisis informatics, offering a flexible and scalable alternative to real-world tweet data curation. While the workflow shows promising results, it is essential to consider the potential limitations and challenges associated with generating synthetic data. As researchers and policymakers, we must carefully evaluate the implications of this approach and consider its potential applications and limitations in various crisis contexts. Furthermore, the findings of this study highlight the need for more research on the development and evaluation of AI systems for crisis informatics, particularly in the context of social media data analysis.
Recommendations
- ✓ Future research should focus on applying the workflow to diverse crisis events and applications to further evaluate its effectiveness and scalability.
- ✓ Developers and policymakers should consider the potential implications of using synthetic data for AI system development and evaluation, and ensure that these approaches do not compromise the accuracy or reliability of AI systems.
