Development, Evaluation, and Deployment of a Multi-Agent System for Thoracic Tumor Board
Summary: arXiv:2604.12161v1 Announce Type: new
Abstract
Tumor boards are multidisciplinary conferences dedicated to producing actionable patient care recommendations with live review of primary radiology and pathology data. Succinct patient case summaries are needed to drive efficient and accurate case discussions. We developed a manual AI-based workflow to generate patient summaries to display live at the Stanford Thoracic Tumor board. To improve on this manually intensive process, we developed several automated AI chart summarization methods and evaluated them against physician gold standard summaries and fact-based scoring rubrics. We report these comparative evaluations as well as our deployment of the final state automated AI chart summarization tool along with post-deployment monitoring. We also validate the use of an LLM as a judge evaluation strategy for fact-based scoring. This work is an example of integrating AI-based workflows into routine clinical practice.
Introduction
The integration of artificial intelligence (AI) into healthcare has been a transformative force, particularly in the realm of oncology. The Stanford Thoracic Tumor Board represents a critical intersection where multidisciplinary expertise converges to assess and recommend patient care strategies for thoracic tumors. This article discusses the development and implementation of an AI-based workflow designed to enhance the efficiency and accuracy of case discussions during these tumor board meetings.
Methodology
The project began with a manual AI-based workflow for generating patient summaries. However, recognizing the limitations of a manually intensive process, we aimed to develop a more streamlined solution. The following steps were undertaken:
- Workflow Development: We created several automated AI chart summarization methods tailored to streamline the generation of patient case summaries.
- Evaluation Against Gold Standards: The automated methods were rigorously evaluated against summaries produced by physicians, which served as the gold standard.
- Fact-Based Scoring Rubrics: We utilized fact-based scoring rubrics to objectively assess the quality and utility of the generated summaries.
Results
Our comparative evaluations revealed significant advancements in generating succinct, accurate patient summaries using the automated AI methods. The deployment of the final automated AI chart summarization tool was executed, accompanied by comprehensive post-deployment monitoring to ensure ongoing quality and efficacy. A notable aspect of our evaluation strategy included validating the use of a Language Learning Model (LLM) as a judge evaluation strategy for fact-based scoring.
Discussion
The findings from this project underscore the potential of AI to enhance clinical workflows within tumor boards. By automating the summarization process, we aim to reduce the cognitive load on physicians, allowing them to focus on critical decision-making. Additionally, the positive outcomes from our evaluation of LLMs as an evaluative tool suggest promising avenues for future research and application.
Conclusion
In conclusion, the deployment of an automated AI chart summarization tool serves as a valuable example of how AI-based workflows can be integrated into routine clinical practice. The Stanford Thoracic Tumor Board’s experience highlights the efficacy of such systems in improving patient care discussions and reinforces the importance of interdisciplinary collaboration in leveraging technology for better health outcomes.