NYU Langone Health LLM can predict hospital readmissions

A team of researchers at New York University's Langone Health academic medical centre has developed a large language model (LLM) deployed at three of its hospitals. This model predicts a patient's risk of 30-day readmission and other clinical outcomes. In conjunction with the publication of their study in Nature, the researchers have released the code base of the NYUTron model on GitHub. This allows other healthcare organisations to train their own LLMs and provide doctors with insights to identify patients who may require intervention to reduce readmissions.

The NYUTron model has been utilised to evaluate 50,000 patients who were discharged from NYU's healthcare system. It shares predictions of readmission risk with physicians via email. NYU collaborated with NVIDIA to develop and run the LLM on the company's artificial intelligence platforms, leveraging their stack, library, and software.

Dr. Eric Oermann, assistant professor of neurosurgery, radiology, and data science at NYU Langone Health, highlighted the accessibility of pretrained models like NYUTron for hospitals that may not have the resources to train their own large language models from scratch. He stated that adopting pretrained models and fine-tuning them with local data using GPUs in the cloud is feasible for almost everyone in healthcare.

The researchers found that after pretraining the LLM, fine-tuning it with specific hospital data on-site significantly enhanced accuracy. NYUTron was initially pre trained on ten years of health records from NYU Langone Health, comprising over four billion words of clinical notes representing nearly 400,000 patients. The team also developed four additional algorithms that predict hospital stay length, the likelihood of in-hospital mortality, and the chances of insurance claims being denied.

The approach taken by the researchers involved treating readmission prediction as a natural language processing task. This involved creating an LLM that was pretrained on a health-system scale corpus of clinical text using high-end multi-node GPU servers. They addressed challenges related to long sequence length, label imbalance, and the impact of noisy labels on model evaluation.

Comparing NYUTron with a group of six physicians at different levels of seniority, the researchers found that the model outperformed the median physician performance in predicting 30-day readmission. Physicians had a median F1 score (a machine learning evaluation metric that measures a model's accuracy) of 62.8%, compared with NYUTron, which had a median F1 score of 77.8%

NYU Langone Health is also exploring the possibility of licensing its models to organisations that lack the resources to build their own models from scratch. The next phase for the research team involves conducting a clinical trial to determine whether interventions based on NYUTron's analyses can effectively reduce readmission rates.

Click here to read the original news story.