ChatGPT Gives ‘Largely Appropriate’ Answers to Cardiovascular Disease Prevention Qs: Study

The era of chatbots seems to have arrived in full swing. With tech giants like Google, Microsoft introducing new chatbots, features and integrations every day, it is becoming increasingly attractive for the healthcare industry to explore more use cases for this emerging technology.

In a latest study to assess ChatGPT’s responses to medical questions, researchers put together 25 questions about fundamental concepts for preventing heart disease, including risk factor counseling, test results and medication information and found that it provided "largely appropriate" responses to the questions.

They posed the questions to ChatGPT three times and each set of responses was graded by a clinician. The clinician rated the responses as appropriate, inappropriate or unreliable if the chatbot's responses varied in each set. The reviewers also evaluated the responses in two contexts: first as a patient-facing platform and then as draft responses to patient messages sent to a clinician for review. 

The study found 21 of the 25 questions were considered “appropriate” in both contexts, while four were graded “inappropriate” in both scenarios. In three of the four sets of inappropriate responses, all three answers were incorrect, while one set had only one inappropriate response. 

While the researchers noted several limitations in their analysis, they said researchers could use more reviewers to evaluate responses or set up a formal system for grading responses that didn't rely as heavily on a clinician's subjective opinion. 

At this time, the chatbot isn't designed for medical use, and cardiovascular disease prevention can't be covered in only 25 questions. 

The results of the study were published in a research letter in the journal JAMA. 

More and more companies are touting chatbots for healthcare purposes. Wysa uses a chatbot to guide users through cognitive behavioral therapy for concerns like low mood, anxiety and stress.  Symptom checking chatbot Ada Health recently launched a digital COVID-19 care journey that was developed with Pfizer. Woebot Health launched a mental health chatbot and received FDA Breakthrough Device Designation for its postpartum depression digital therapeutic in 2021.

While this seems to be just the beginning - exciting times lay ahead for the healthcare industry and its newest companion - AI. 

Join the HealthXL Meeting on ‘Digital Biomarkers for Mental Health Evaluation and Management’ on 16th February. Click here to Request to Join.