Automated Detection of Dosing Errors in Clinical Trial Narratives: A Multi-Modal Feature Engineering Approach with LightGBM
Summary: arXiv:2604.19759v1 Announce Type: new
Dosing errors in clinical trials pose significant concerns for patient safety and the integrity of trial results. Despite stringent medication protocols, these errors continue to challenge researchers and healthcare professionals alike. In response to this pressing issue, a new automated system has been developed to detect dosing errors in unstructured clinical trial narratives. This system employs gradient boosting techniques, specifically LightGBM, coupled with an innovative multi-modal feature engineering approach.
Key Features of the Study
The study leverages a comprehensive feature set, comprising 3,451 features sourced from various methodologies:
- Traditional NLP Techniques: Utilizing TF-IDF and character n-grams to analyze text patterns.
- Dense Semantic Embeddings: Incorporating embeddings from models such as all-MiniLM-L6v2 to capture contextual meaning.
- Domain-Specific Medical Patterns: Identifying unique patterns relevant to the medical field to enhance detection capabilities.
- Transformer-Based Scores: Implementing advanced models like BiomedBERT and DeBERTa-v3 for improved feature representation.
Features are meticulously extracted from nine complementary text fields, averaging 5,400 characters per sample, allowing for an extensive overview of 42,112 clinical trial narratives. This thorough approach aims to ensure that no critical information is overlooked during the analysis.
Performance and Results
The system was evaluated using the CT-DEB benchmark dataset, which is characterized by a severe class imbalance (only 4.9% of the instances are positive cases). The results are promising, with the model achieving a test ROC-AUC score of 0.8725 via 5-fold ensemble averaging. Cross-validation efforts yielded a score of 0.8833, with a standard deviation of 0.0091 AUC, indicating robust model performance.
Ablation Studies and Feature Efficiency
To further understand the impact of different features on model performance, systematic ablation studies were conducted. These studies revealed that removing sentence embeddings led to the most significant drop in performance, with a decrease of 2.39%. This finding underscores the critical importance of these embeddings in the overall feature set, even though they contribute only 37.07% to the total feature importance.
Additionally, an analysis of feature efficiency indicated that selecting the top 500-1000 features yielded optimal performance, achieving an AUC between 0.886 and 0.887. This method outperformed the complete feature set of 3,451 features, which recorded an AUC of 0.879, demonstrating the effectiveness of feature selection as a regularization technique.
Conclusion
This study highlights the critical role of feature selection and demonstrates that a combination of sparse lexical features and dense representations can enhance the classification of specialized clinical texts, even in the context of severe class imbalance. The automated detection system not only improves the accuracy of identifying dosing errors but also contributes to advancing patient safety and the integrity of clinical trials.