Large Language Models for Biomedical Article Classification
arXiv:2603.11780v1 Announce Type: new Abstract: This work presents a systematic and in-depth investigation of the utility of large language models as text classifiers for biomedical article classification. The study uses several small and mid-size open source models, as well as selected closed source ones, and is more comprehensive than most prior work with respect to the scope of evaluated configurations: different types of prompts, output processing methods for generating both class and class probability predictions, as well as few-shot example counts and selection methods. The performance of the most successful configurations is compared to that of conventional classification algorithms. The obtained average PR AUC over 15 challenging datasets above 0.4 for zero-shot prompting and nearly 0.5 for few-shot prompting comes close to that of the na\"ive Bayes classifier (0.5), the random forest algorithm (0.5 with default settings or 0.55 with hyperparameter tuning) and fine-tuned trans
arXiv:2603.11780v1 Announce Type: new Abstract: This work presents a systematic and in-depth investigation of the utility of large language models as text classifiers for biomedical article classification. The study uses several small and mid-size open source models, as well as selected closed source ones, and is more comprehensive than most prior work with respect to the scope of evaluated configurations: different types of prompts, output processing methods for generating both class and class probability predictions, as well as few-shot example counts and selection methods. The performance of the most successful configurations is compared to that of conventional classification algorithms. The obtained average PR AUC over 15 challenging datasets above 0.4 for zero-shot prompting and nearly 0.5 for few-shot prompting comes close to that of the na\"ive Bayes classifier (0.5), the random forest algorithm (0.5 with default settings or 0.55 with hyperparameter tuning) and fine-tuned transformer models (0.5). These results confirm the utility of large language models as text classifiers for non-trivial domains and provide practical recommendations of the most promising setups, including in particular using output token probabilities for class probability prediction.
Executive Summary
This study investigates the utility of large language models for biomedical article classification, employing various open-source and closed-source models to analyze the effectiveness of different prompts, output processing methods, and few-shot example counts. The results demonstrate that large language models can achieve comparable performance to conventional classification algorithms, such as the naive Bayes classifier and fine-tuned transformer models, particularly when using output token probabilities for class probability prediction. The study provides practical recommendations for utilizing large language models in non-trivial domains, including the selection of suitable model configurations and prompt types.
Key Points
- ▸ Large language models can be effectively utilized for biomedical article classification.
- ▸ The study employs a comprehensive evaluation of various model configurations and prompt types.
- ▸ The results demonstrate comparable performance to conventional classification algorithms.
Merits
Strength in Comparative Analysis
The study provides a thorough comparison of large language models to conventional classification algorithms, allowing for a better understanding of their relative effectiveness in biomedical article classification.
Practical Recommendations
The study offers valuable insights and recommendations for selecting suitable model configurations and prompt types, enabling practical applications of large language models in non-trivial domains.
Demerits
Limited Generalizability
The study's findings may not be directly generalizable to other domains or applications due to the specific focus on biomedical article classification and the use of limited datasets.
Dependence on Model Selection
The study's results may be heavily dependent on the specific model configurations and prompt types employed, which may not be universally applicable or optimal for all use cases.
Expert Commentary
The study provides a comprehensive evaluation of the utility of large language models for biomedical article classification, demonstrating their effectiveness in achieving comparable performance to conventional classification algorithms. The results of this study are particularly significant given the growing importance of natural language processing in biomedical research and healthcare applications. However, the study's limitations, including the dependence on model selection and limited generalizability, highlight the need for continued research and development in this area.
Recommendations
- ✓ Further research is needed to explore the application of large language models in other biomedical text classification tasks and domains.
- ✓ Developers and practitioners should carefully consider the selection of model configurations and prompt types to ensure optimal performance in biomedical article classification tasks.