Tea and Sympathy: Do AI-Based Chat Systems Have What It Takes to Offer Accurate and Empathetic Patient Care?
AI-based chat systems are getting “smarter” – but are they smart enough to offer reliable patient care?
One of the first available scientific studies indicates that AI-based Chat systems can perform admirably when answering patient questions.
A research team headed by Dr. John W. Ayers at the University of California San Diego recently set out to evaluate which is better, ChatGPT or a human physician.
The team compiled a set of approximately 200 typical patient questions (extracted from the AskDocs forum on Redditt.)
A three-panel jury of licensed healthcare providers evaluated the responses – without knowing if the answer was provided by a human physician or by ChatGPT.
Sorry if you are a physician reading this article, but the results came back strong in favor of ChatGPT – the three-panel jury preferred the ChatGPT response nearly 80% of the time.
The jurors rated ChatGPT’s responses to common medical questions as being of much higher quality 78.5% of the time (versus only 22.1% for the physician-sourced answers.)
ChatGPT also demonstrated a much higher level of empathy toward patients (in 45% of responses), a statistically higher achievement than the 4.6% of times the physician’s answer was more empathetic.
Should physicians be concerned about AI taking over their jobs?
In the short term, the answer is no.
Instead, AI might be a godsend for helping healthcare providers keep up with their ever-growing workload. The widespread adoption of online healthcare portals (part of the push toward electronic records) has led to an overwhelming increase in the number of electronic messages sent by patients. Reading, prioritizing, and responding to these messages has become very burdensome for healthcare providers, so AI-based tools could help a lot – either by reading the inquiries and providing draft responses offline (for the provider to review and send to the patient) or by “chatting” with the patient directly.
AI Chat Trustworthiness is Marred by Potential Hallucinations. How Will Patients React?
Unfortunately, the Large Language Model (LLM) systems that power AI-based chat systems can sometimes get it wrong.
This can happen either because the language models were trained on faulty data or they happen upon a gap or glitch in their “knowledge,” which they sometimes “fill in” with related though potentially incorrect information.
A recent (now infamous) non-medical example comes to us from Google’s AI Overviews feature, which suggested (incorrectly, we assure you!) that the best way to keep the cheese topping from sliding off pizza is to glue it in place!
AI researchers are trying to figure out ways to reduce or eliminate these so-called “hallucination” occurrences, in which the AI systems provide incorrect information.
Faulty recipes could be lethal if followed by unsuspecting cooks. The same applies to medical advice. Imagine a case where a patient is asking an AI-based chat system about some concerning symptoms they are having, such as pain in their chest.
Given today’s current technology, it’s not prudent to allow the AI-based chat system to diagnose serious health conditions – such as the onset of a heart attack or a bout with Gastroesophageal reflux disease (GERD) – without consulting a human health care provider.
On the other hand, AI-based systems might be quicker at scanning all the incoming requests than humans, helping to prioritize potentially critical incoming patient concerns to the highest priority for attending care providers to address immediately.
The bottom line is that state-of-the-art AI-based chat tools seem ready to collect patient information (which they can ask the patient to review and confirm before submitting). They can also create draft responses, but from a duty of care (and malpractice perspective), the provider should review everything before issuing orders, updating the medical records, prescribing medications, etc.
Are AI Chat-Based Patient Care Systems Susceptible to Fraud?
Fraudulent actors have had a field day with AI-based tools.
In 2019, criminals were able to “clone” the voice of a German CEO to place a fraudulent voice call to one of his direct reports (the head of a company subsidiary based in the UK.) The synthesized voice convinced the victim to expedite a $243,000 payment to a Hungarian supplier, which the criminals subsequently transferred to a bank in Mexico and other locations.
Unfortunately, telehealth has already become a conduit for criminal activity.
There have already been cases of company insiders taking advantage of the more relaxed Post-Covid rules for prescribing drugs to patients via telehealth calls – the CEO of Done Global, Ruthia He, and clinical president David Brody, were recently arrested for running a “pill mill” operation to distribute Adderall and other stimulants to patients who didn’t meet the prescription requirements. The criminal charges accuse Re, Brody, and the company of collecting $100 million in fraudulent prescription reimbursements.
While we are not aware of any such cases, organized drug trafficking gangs could also theoretically leverage AI-based chat systems to automate prescription requests for controlled substances (such as opioids) – tricking unwitting providers into prescribing medications for multiple fake AI patients.
The widespread availability of tools to create cloned voices or real-time deep fake videos could exacerbate the fraud problem as well.
A fake voice or video call could trick patients into having conversations with criminals, impersonating their healthcare provider, tricking them into providing private information (for identity fraud purposes), or requesting they make fraudulent payments.
Unfortunately, these and other video and telephone-based frauds may cause patients (and providers) to lose confidence in conducting business over the phone (or on video), which would be a terrible loss for those who need telehealth medical services.
How Will Compliance with HIPAA and Other Emerging AI Regulatory Regimes Affect the Adoption of AI Chat-Based Tools in Patient Care?
Medical providers who want to experiment with using AI-based chat systems in a medical setting must not forget that these systems still need to comply with HIPAA regulations (via a legally binding Business Associate Agreement or BAA) if they are handling personally identifiable healthcare information (PHI).
Legal analysts in the medical field argue that ordinary AI-based chat systems don’t yet comply with HIPAA, leaving medical providers potentially exposed to very hefty fines.
Progress is being made.
Some startup companies, such as BastionGPT, CompliantChatGPT, and DearDoc, are offering modified AI-based chat systems that they claim are HIPAA compliant.
Other new regulatory/compliance environments are on the horizon.
For example, there is a major debate on the future of AI in California, as legislators debate the merits of the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act. In a nutshell, this proposed law (which only applies to the largest AI providers, based on energy and computing power usage) would enable the state’s attorney general to sue companies that cause critical harm (such as a mass casualty event), require a “kill switch” to turn off AI systems and establish a new Frontier Model Division to develop and enforce new safety standards.
Developers and deployers (!) of AI technology operating in the European Union need to be aware of the new Artificial Intelligence Act, which was enacted in March 2024.
The new rules attempt to define what AI is and classify its use according to increasing levels of risk: none/minimal, limited, high, and unacceptable risk levels. Each risk level has a corresponding set of regulations.
Depending on interpretation, the medical uses of AI outlined in this article would appear to fall under the high-risk level category, which would trigger a series of requirements for quality, conformity, and human rights assessments, data governance, registration with government officials, and incident reporting.
How Will CMS and Insurance Companies Compensate Doctors Using AI-Based Chat Systems to Provide Care for Patients?
Reimbursement rates are naturally a big topic of conversation for US-based healthcare providers considering implementing an AI-based Chat system to interact with their patients.
Some providers are concerned that AI-based reimbursement rates will follow the precedents set by telehealth compensation rates, which are frustratingly low in the eyes of many providers.
According to a recent paper published by George Wu of Baylor College of Medicine, the Centers for Medicare & Medicaid Services (CMS) is only providing limited reimbursement to providers for the use of AI-based services through its three primary systems – the Physician Fee Schedule (PFS), Hospital Outpatient Prospective Payment System (HOPPS), and Inpatient Prospective Payment System (IPPS).
To date, direct reimbursements for AI-based services have been limited to instances where it can be demonstrated that AI algorithms provide unique value to the physician, either by improving patient outcomes and/or decreasing overall costs. Wu cites one example that has met this standard: AI-based noninvasive assessment of coronary fractional flow reserve to assess patients with coronary heart disease.
Medical insurance companies tend to hew very closely to CMS reimbursement policies, so it’s unlikely we will see them open their wallets unless CMS reimbursement policies become more liberal vis-a-vis AI-based systems.
Formaspace is Your Laboratory Research Partner
Evolving Workspaces. It’s in our DNA.
Talk to your Formaspace Sales Representative or Strategic Dealer Partner today to learn more about how we can work together to make your next construction project or remodel a success.
