In the early days of the Covid-19 pandemic, hospitals were in urgent need of solutions to handle the influx of severely ill patients. Many hospitals, including those in Yale’s health system, turned to an artificial intelligence algorithm created by Epic Systems, a company that specializes in electronic health records. The purpose of this AI algorithm was to predict which patients were at a high risk of deteriorating rapidly so that they could receive the necessary critical care in time.
However, the effectiveness of such proprietary AI algorithms has always been a topic of uncertainty. In a recent study conducted by Yale’s health system, the statistical performance of six early warning scores, including Epic’s AI tool, was evaluated using clinical data from seven hospitals. The results of this analysis, published in JAMA Network Open, shed light on the actual performance of these clinical AI models, revealing that some may not be as reliable as previously thought.
Co-author Deborah Rhodes, who serves as the chief quality officer for Yale New Haven Health System and associate dean of quality for Yale School of Medicine, explained that the primary goal of the study was to identify the most effective tool for predicting patient deterioration. Despite the widespread use of Epic’s early warning score in health systems across the country due to its integration with the company’s electronic health record, the study found that the tool may not live up to its reputation.
Rhodes mentioned that the decision to implement Epic’s tool was partly driven by cost considerations, as it was a built-in feature of the electronic health record system. However, the study’s findings suggest that the effectiveness of a tool should be prioritized over its cost, especially when it comes to patient care and outcomes.
The study conducted by Yale’s health system serves as a reminder that the performance of AI algorithms in a clinical setting should be rigorously evaluated to ensure that they deliver accurate and reliable results. As the use of AI in healthcare continues to grow, it is essential for health systems to critically assess the performance of these tools to provide the best possible care for patients.
In conclusion, while AI algorithms like Epic’s early warning score may offer promising solutions for predicting patient outcomes, it is crucial to conduct thorough evaluations to determine their true effectiveness in a clinical setting. By prioritizing the performance and reliability of these tools, health systems can ensure that patients receive the highest standard of care based on accurate and data-driven predictions.
