A brand new examine from Mount Sinai means that utilizing generative synthetic intelligence to assist with coding automation has some important limitations.
WHY IT MATTERS
For the analysis, Mount Sinai’s Icahn Faculty of Medication evaluated the potential utility for giant language fashions in healthcare to automate medical code assignments – primarily based on scientific textual content – for reimbursement and analysis functions.
The examine in contrast LLMs from OpenAI, Google and Meta to evaluate whether or not they might successfully match the fitting medical codes to their corresponding official textual content descriptions.
To evaluate and benchmark the efficiency of GPT-3.5, GPT-4, Gemini Professional and Llama2-70b, researchers extracted greater than 27,000 distinctive prognosis and process codes from 12 months of routine care within the Mount Sinai Well being System, excluding affected person knowledge.
“Earlier research point out that newer massive language fashions battle with numerical duties,” Dr. Eyal Klang, director of Icahn Mount Sinai’s Knowledge-Pushed and Digital Medication Generative AI Analysis Program and senior coauthor of the examine, defined in an announcement final week.
“Nonetheless, the extent of their accuracy in assigning medical codes from scientific textual content had not been completely investigated throughout totally different fashions.”
In assessing whether or not the 4 accessible fashions might successfully match medical codes via qualitative and quantitative strategies, the researchers decided all LLMs scored beneath 50% accuracy in producing distinctive prognosis and process codes.
Whereas GPT-4 carried out the most effective within the examine with the very best precise match charges for ICD-9-CM at 45.9%, ICD-10-CM at 33.9% and CPT codes at 49.8%, “unacceptably massive” errors remained.
The researchers mentioned GPT-4 produced probably the most incorrectly generated codes, whereas GPT-3.5 had the best tendency to be imprecise, figuring out extra basic moderately than exact codes.
The examine outcomes, which the New England Journal of Medication AI printed final week, led the researchers to warning that the efficiency of LLMs in real-world medical coding might have worse outcomes.
“LLMs should not acceptable to be used on medical coding duties with out extra analysis,” the researchers mentioned within the report.
“Whereas AI holds nice potential, it have to be approached with warning and ongoing improvement to make sure its reliability and efficacy in healthcare,” Dr. Ali Soroush, assistant professor of D3M and drugs, cautioned in a press release.
Mount Sinai famous that the researchers will look to develop tailor-made LLM instruments for correct medical knowledge extraction and billing code project.
THE LARGER TREND
Regardless of the findings of the Mount Sinai examine, others see worth in AI-enabled coding, and say AI methods might help doctor teams keep away from lacking income alternatives and elevate their documentation compliance.
Dr. Bruce Cohen, a surgeon and former CEO at OrthoCarolina in Charlotte, North Carolina.
“As annual coding necessities are instituted, an AI-based system will combine and implement these modifications in real-time,” Dr. Bruce Cohen, a surgeon and former CEO at OrthoCarolina in Charlotte, North Carolina, informed Healthcare IT Information.
AI-based methods don’t get rid of coders’ jobs, he added: “It expands the oversight and accuracy of each cost going out primarily based on analysis and administration coding.”
ON THE RECORD
“Our findings underscore the essential want for rigorous analysis and refinement earlier than deploying AI applied sciences in delicate operational areas like medical coding,” Soroush asserted in a press release concerning the Mount Sinai analysis.
“This examine sheds gentle on the present capabilities and challenges of AI in healthcare, emphasizing the necessity for cautious consideration and extra refinement previous to widespread adoption,” added Dr. Girish Nadkarni, director of the Charles Bronfman Institute of Customized Medication and system chief of D3M.
Andrea Fox is senior editor of Healthcare IT Information.
E mail: [email protected]
Healthcare IT Information is a HIMSS Media publication.