Adoption and Use of LLMs at an Academic Medical Center
In recent years, the integration of large language models (LLMs) in healthcare has gained significant attention. A recent study published as arXiv:2602.00074v2 highlights the development and implementation of ChatEHR, a system designed to enhance clinical documentation by leveraging LLMs effectively. Despite the potential of LLMs to streamline workflows, previous standalone tools faced challenges due to “workflow friction” arising from manual data entry. This article explores how ChatEHR addresses these challenges and its impact on clinical operations.
Overview of ChatEHR
ChatEHR is a pioneering system that allows healthcare professionals to utilize LLMs with comprehensive patient timelines that span several years. By integrating automations—static combinations of prompts and data designed to perform fixed tasks—ChatEHR transforms the electronic health record (EHR) experience. The system supports both automated and interactive use through a user-friendly interface (UI), enabling clinicians to access crucial patient medical records for a variety of applications.
Use Cases and Benefits
The versatility of ChatEHR allows for numerous practical applications, which include:
- Pre-visit chart reviews
- Screening for transfer eligibility
- Monitoring for surgical site infections
- Chart abstraction
These use cases redefine LLM use as an institutional capability rather than a mere add-on, significantly enhancing clinical workflows and patient care.
Implementation and User Adoption
Over a period of 1.5 years, the ChatEHR system was developed, resulting in the creation of seven automations. Remarkably, 1,075 users have been trained to utilize the UI, leading to an impressive 23,000 user sessions within the first three months post-launch. The model-agnostic nature of these automations, combined with access to various types of data, proved essential in aligning specific clinical or administrative tasks with the most suitable LLM.
Performance Monitoring and Evaluation
Traditional benchmark-based evaluations were found to be inadequate for monitoring the ChatEHR UI’s performance. The generation of summaries emerged as the most frequently executed task, with performance metrics indicating an estimated 0.73 hallucinations and 1.60 inaccuracies per generated summary. This highlighted the necessity for new methodologies to assess the system’s effectiveness and reliability.
Impact and Value Assessment
The initial findings of using ChatEHR have shown promising results, including:
- Estimated savings of $6 million in the first year
- Significant time savings
- Increased revenue growth
However, these figures do not account for the enhanced quality of care provided to patients. To fully understand the value of adopting LLMs in healthcare, a robust framework for value assessment is essential, enabling institutions to prioritize initiatives and quantify the overall impact of LLM utilization.
Conclusion
The development of ChatEHR represents a significant advancement in the application of LLMs within clinical settings. By adopting a “build-from-within” strategy, health systems can maintain control over their LLM platforms, fostering an environment that is vendor-agnostic and internally governed. As the healthcare industry continues to evolve, the potential for LLMs to streamline operations and improve patient care remains vast.
Related AI Insights
- Learned Feedback Codes for Enhanced Secure Communications
- ExCyTIn-Bench: Benchmarking LLMs for Cyber Threat Detection
- Advanced Weakly-Supervised Camouflaged Object Detection
- AI-Powered Expansion of Alexandria Materials Database
- ATLAS: Adaptive AI Trading with Dynamic Prompt Optimization
- Vanishing Contributions: Smooth Iterative Model Compression
- Sentra-Guard: Real-Time Multilingual Defense for LLMs
- LinkAnchor: AI Agent for Accurate Issue-to-Commit Linking
- Game-Time Benchmark: Testing Temporal Skills in Spoken AI
- Reasoning-Intensive Regression in AI: Breakthrough with MENTAT
