MedMT-Bench: Can LLMs Memorize and Understand Long Multi-Turn Conversations in Medical Scenarios?
The rapid evolution of Large Language Models (LLMs) has led to their integration into various specialized fields, including medicine. While these models have showcased remarkable capabilities, there remains a significant gap in their ability to handle long-context memory and complex multi-turn conversations critical for medical scenarios. In response to this challenge, researchers have introduced a new benchmark called MedMT-Bench, designed to rigorously evaluate LLMs in medical multi-turn instruction following.
Introduction to MedMT-Bench
Existing medical benchmarks often fail to adequately test the long-context memory, interference robustness, and safety defense mechanisms required in high-stakes medical environments. MedMT-Bench aims to fill this critical gap by simulating the complete diagnosis and treatment process within a medical context. This benchmark was constructed through a meticulous process that involved scene-by-scene data synthesis, supplemented by manual editing from medical experts.
Benchmark Details
The MedMT-Bench consists of 400 test cases that reflect real-world medical scenarios. Each test case is designed to challenge LLMs with an average of 22 conversational rounds, and some cases extend up to 52 rounds. The benchmark addresses five distinct types of complex instruction-following issues, which are paramount in clinical settings.
Evaluation Methodology
To evaluate the performance of LLMs on this benchmark, a novel LLM-as-judge protocol was proposed. This evaluation framework includes instance-level rubrics and atomic test points, which have been validated against expert annotations. Remarkably, the agreement between human and LLM evaluations reached an impressive 91.94%. This high level of accuracy underscores the benchmark’s potential for reliable assessment.
Performance Insights
The performance of 17 leading LLMs was tested using the MedMT-Bench, and the results revealed a concerning trend. All models underperformed significantly, with overall accuracy rates falling below 60.00%. The highest-performing model achieved an accuracy of only 59.75%, indicating that current LLMs struggle to navigate the complexities of long multi-turn conversations in medical contexts.
Implications for Future Research
The introduction of MedMT-Bench marks a pivotal step in advancing research in medical AI. By providing a rigorous evaluation framework, it can serve as a critical tool for developers and researchers aimed at enhancing the safety and reliability of AI applications in healthcare. The insights gained from this benchmark can inform the design of future models that are better equipped to handle the intricacies of medical dialogues.
Accessing MedMT-Bench
Researchers and developers interested in exploring the MedMT-Bench can access the benchmark and supplementary materials at the following link:
MedMT-Bench Supplementary Material.
Conclusion
As the integration of AI into medicine continues to grow, tools like MedMT-Bench will be essential in ensuring that LLMs can effectively understand and engage in long multi-turn conversations. This benchmark not only highlights current limitations but also paves the way for future advancements in medical AI.