FMI@SU ToxHabits: Evaluating LLMs Performance on Toxic Habit Extraction in Spanish Clinical Texts
The recent paper titled “FMI@SU ToxHabits” published on arXiv under the identifier 2604.06403v1, introduces a novel approach for recognizing toxic habits in Spanish clinical texts. This research was developed for the ToxHabits Shared Task, focusing specifically on identifying and classifying mentions of substance use and abuse in clinical case reports.
In the field of natural language processing (NLP), the challenge of analyzing clinical texts in non-English languages has gained considerable attention. The ToxHabits Shared Task provided an excellent platform for researchers to assess the capabilities of large language models (LLMs) in handling such tasks. The team participating in this task aimed to detect mentions of various substances, classifying them into four distinct categories: Tobacco, Alcohol, Cannabis, and Drug.
Methodology and Approach
The researchers explored a variety of methods to leverage LLMs for the task at hand. These methods included:
- Zero-shot prompting
- Few-shot prompting
- Prompt optimization
Among the different approaches tested, the few-shot prompting technique using GPT-4.1 yielded the most promising results. This method allowed the model to draw from a limited number of examples to better understand the context and nuances of the clinical texts.
Results and Findings
The team’s method achieved an impressive F1 score of 0.65 on the test set, indicating a significant advancement in the recognition of named entities related to substance use in Spanish clinical texts. This finding highlights the potential of LLMs in processing languages beyond English, which has traditionally dominated the NLP landscape.
By successfully identifying and classifying toxic habit mentions, the research contributes valuable insights into the capabilities of AI in clinical settings. The results suggest that LLMs can be effectively adapted to support clinical professionals in identifying substance abuse issues, which is crucial for patient care and intervention strategies.
Implications for Future Research
The promising results from this study open several avenues for future research and development. Some potential directions include:
- Expanding the dataset to include more diverse clinical texts from different Spanish-speaking countries.
- Investigating additional LLM architectures and their performance in toxic habit recognition.
- Exploring the integration of this technology into clinical decision support systems.
As the field of NLP continues to evolve, the findings from the FMI@SU ToxHabits study underscore the importance of developing robust models capable of understanding and analyzing clinical data in various languages. The successful application of LLMs in this context not only enhances the understanding of substance use in clinical populations but also demonstrates the broader potential of AI in healthcare.
In conclusion, the FMI@SU ToxHabits research highlights the advancements in AI and its application in recognizing toxic habits through language processing, paving the way for improved clinical practices and patient outcomes.