The rapid advancement of artificial intelligence (AI) in healthcare has ushered in a new era of possibilities, promising not just improved patient outcomes, but also enhanced operational efficiency, and groundbreaking research. This wave of innovation brings with it a sense of optimism and excitement for the future of healthcare. However, as we stand at this technological crossroads, it is crucial to establish robust frameworks for developing, testing, and maintaining AI systems in healthcare.
November 2022 marked a significant milestone in the public’s experience with AI, as ChatGPT demonstrated the capabilities of large language models (LLMs) to millions of users worldwide. This event catalyzed discussions about the potential applications of generative AI in various sectors, including healthcare. Unlike traditional predictive AI models, generative AI, or GenAI, produces natural language outputs, making it a form of relational AI that can interact with users in more human-like ways.
The potential applications of GenAI in healthcare are vast and promising. From drafting patient-portal messages to creating conversational interfaces for patient education and even facilitating preliminary self-diagnosis, GenAI could revolutionize how healthcare information is communicated and accessed. However, this new frontier of AI also brings a host of ethical considerations that must be carefully addressed.
Ethical Considerations
Sim and Cassel note in their recent New England Journal of Medicine article that the introduction of AI-generated text, speech, images, and video between clinicians and patients fundamentally alters the ethical landscape of healthcare delivery. Physicians’ traditional fiduciary responsibility to uphold principles of beneficence, respect for persons, and justice now extends to the AI systems they employ.
The ethical implementation of AI in healthcare demands that these systems be Fair, Appropriate, Valid, Effective, and Safe (FAVES). However, ensuring adherence to these principles is challenging due to the complex nature of AI, particularly generative AI. The probabilistic nature of LLMs introduces inherent risks of errors or “hallucinations,” which could have severe consequences in a healthcare setting.
Moreover, the potential for AI systems to pursue goal-oriented behavior misaligned with medical ethics – such as optimizing for insurer profits rather than patient outcomes – underscores the need for rigorous ethical oversight. As healthcare leaders, we must ensure that AI systems prioritize patient benefit above all else, followed by considerations for providers and broader societal impacts.
The Call for AI Assurance Laboratories
There is a growing consensus on the need for a nationwide network of health AI assurance laboratories to address the potential problems of healthcare AI. As proposed by Shah et al. in a recent JAMA article, these labs would serve as shared resources for validating AI models, accelerating responsible innovation, and ensuring safe deployment in healthcare settings.
These assurance labs would provide comprehensive evaluations of AI models, ranging from technical performance assessments to analyses of potential biases and simulations of real-world impacts. By offering different levels of evaluation, from basic technical validations to in-depth assessments of usability and adoption via human-machine teaming, these labs could provide crucial insights into the potential benefits and risks of AI systems before their deployment in clinical settings.
Creating such a network would promote transparency and accountability in AI development. By publishing evaluation results openly in a nationwide registry, these labs would enable healthcare providers, policymakers, and the public to make informed decisions about the use of AI in healthcare. In addition, the labs form a repository of knowledge on developing, testing, and deploying AI in various healthcare settings with different overall goals.
The Need for Ongoing Monitoring and Revalidation
Robust AI assurance requires continuous monitoring and revalidation of AI models. Unlike traditional medical interventions, AI systems can “drift” over time, potentially leading to degraded performance or unexpected behaviors and, in turn, undesirable outcomes. This issue is particularly relevant for generative AI models, which may be updated or fine-tuned regularly.
As a result, we must establish processes for managing AI models’ lifecycles to ensure they maintain their performance over time across diverse populations and in various clinical settings. This ongoing monitoring, which includes regular revalidation, is not just a precaution, but a reassurance that we are committed to identifying and mitigating potential harms before they impact patient care.
Transparency and Ethical Imperatives
While generative AI offers tremendous benefits to patients and providers, we must establish stringent standards for its development, testing, and maintenance. This includes a strong emphasis on transparency. Black box AI systems, where there is no transparency into the algorithms used, training data, or steps taken to prevent harm, are unacceptable in healthcare. By ensuring transparency, we keep all stakeholders informed and involved in the evolution of healthcare AI.
We must prioritize the development of interpretable and explainable AI models that allow for meaningful oversight and accountability. This transparency is crucial for regulatory compliance and building and maintaining trust among patients, providers, and the public.
Furthermore, we must be wary of perverse incentives that may arise when testing is left solely to AI developers. As Sim and Cassel point out, there is a risk of AI systems optimized for outcomes that benefit insurers or other stakeholders at the expense of patient care or provider autonomy. Independent assurance laboratories and ongoing third-party evaluations are essential to mitigate this risk.
As we integrate AI, particularly generative AI, into healthcare, we must do so with a steadfast commitment to ethical principles and patient safety. Establishing a nationwide network of health AI assurance laboratories, coupled with robust standards for ongoing monitoring and revalidation, is crucial in realizing AI’s potential while mitigating its risks.
Healthcare leaders must actively participate in shaping the future of AI in medicine, advocating for transparency, ethical development, and rigorous evaluation. By doing so, we can ensure that AI is a powerful tool to enhance patient care, support healthcare providers, and ultimately improve health outcomes for all.
Sources:
A Nationwide Network of Health AI Assurance Laboratories, JAMA< December 20, 2023
The Ethics of Relational AI — Expanding and Implementing the Belmont Principles, NEJM, July 13, 2024
I look forward to your thoughts so please put them in this post and subscribe to my bi-weekly newsletter Future-Primed Healthcare on LinkedIn and my Dr Barry Speaks channel on YouTube.
