RESEARCH
RESEARCH
Evaluating large language models for clinical note processing: local fine-tuning and internal-external validation using electronic health records from South Asia
Abstract
Objective Large Language Models hold the potential for clinical task-shifting by processing unstructured clinical text, enabling tasks such as clinical concept extraction and medical question answering from electronic health records. If implemented reliably, such approaches may benefit over-burdened healthcare systems, particularly in resource-limited settings and for traditionally overlooked populations, provided that local fine-tuning is supported by appropriate clinical and technical expertise. However, this powerful technology remains largely understudied in real-world contexts, particularly in the Global South. This study aims to assess whether openly available Large Language Models can be used reliably for processing medical notes in real-world settings in South Asia.
Methods We used publicly available Large Language Models to parse de-identified clinical notes from a large electronic health records database in Pakistan, containing hospital records for eight point two million patients. ChatGPT as a general-purpose Large Language Model, and GatorTron, BioMegatron, BioBERT, and ClinicalBERT as medical Large Language Models were evaluated when applied to these data, after fine-tuning them with (a) publicly available clinical datasets namely Informatics for Integrating Biology and the Bedside and National NLP Clinical Challenges for medical concept extraction and emrQA for medical question answering, and (b) the local Pakistani de-identified electronic health records dataset, which includes inpatient Discharge Summaries and Subjective, Objective, Assessment, and Plan notes, as detailed in this paper. Medical concept extraction models were applied to these clinical notes using both three-label and nine-label formats, while medical question answering models were applied to medical questions. Internal and external validation performance was measured for (a) and (b) using F1 score, precision, recall, and accuracy for medical concept extraction and BLEU and ROUGE-L, which measure lexical and sequence similarity, for medical question answering.
Results When clinical Large Language Models were not fine-tuned on the local electronic health records dataset, their performance during external validation on local data was notably poorer compared to internal validation on the dataset used for fine-tuning, with reductions of at least fifteen percent in F1 scores for medical concept extraction and thirty-five percent in ROUGE-L and BLEU scores for medical question answering tasks. This suggests potential bias and highlights the inability of the medical Large Language Models to reliably handle the data distribution of the local population without further fine-tuning and adaptation. This trend persisted across two distinct natural language processing tasks: concept extraction and question answering, spanning a spectrum of task complexities. However, fine-tuning the Large Language Models with local electronic health records data significantly improved model performance across both tasks, yielding a seven point five percent to fifteen percent increase in the F1 score for medical concept extraction and a twenty-seven percent to fifty-three percent increase in ROUGE-L and BLEU scores for medical question answering. Notably, ChatGPT, as a general-purpose Large Language Model, stood out as an exception, demonstrating superior performance across all measured metrics on the local dataset compared to the publicly available dataset, with improvements ranging from three percent to seventeen percent on the local electronic health records dataset, even without fine-tuning on the local data.
Conclusions Publicly available Large Language Models, predominantly trained on data from high-income regions, were found to be unreliable when applied in a real-world clinical setting in Pakistan. Fine-tuning them with local electronic health records data and regional clinical contexts improved their reliability, demonstrating a feasible adaptation strategy that is substantially less resource-intensive than training large language models from scratch. Close collaboration between local clinical and technical experts to curate and leverage more representative, inclusive, and unbiased medical datasets, can play a crucial role in further ensuring reliability of Large Language Models for resource-limited, overburdened settings, to be used in ways that are safe, fair, and beneficial for all.
Introduction
Introduction
Background: large language models in clinical natural language processing
Large Language Models have emerged as powerful tools for processing unstructured clinical text, offering new opportunities to support clinical decision-making, task-shifting, and knowledge extraction from electronic health records. By leveraging the power of transformer architectures trained on vast corpora of medical text, Large Language Models can extract, summarise, and contextualise clinical information at scale, enabling downstream applications such as clinical concept extraction, medical question answering, and summarisation of clinical notes. These applications may inform decision support for frontline healthcare workers, clinical trial selection, medical training, and medical data discovery. If done reliably and rapidly, such use of Large Language Models can benefit over-burdened healthcare systems everywhere, but particularly in resource-limited settings such as South Asia, home to a quarter of the world's population, and where rural and urban public health facilities are largely over-subscribed and under pressure.
Several medical-domain Large Language Models have been developed to address the linguistic and semantic characteristics of clinical text. Models such as ClinicalBERT, BioBERT, BioMegatron, and GatorTron are typically pretrained on biomedical literature and clinical corpora, including PubMed abstracts and intensive care records, to learn domain-specific representations. Additional fine-tuning enables Large Language Models to perform specific tasks such as medical concept extraction, medical relation extraction, natural language inference, semantic textual similarity, medical event prediction, and medical question answering, while substantially reducing computational and data requirements compared to full pretraining.
Electronic health records, particularly free-text clinical notes, represent rich but underutilised repositories of information to aid patient care. Key subjective information including family history, adverse drug events, and social, behavioural, and environmental determinants of health, all of which are commonly required in time-critical decision-making, is well-documented only within the full-text patient notes of electronic health records rather than structured fields. The lack of syntactic, structural, and semantic standardisation across electronic health record systems limits the reuse of this information for clinical care and research. Large Language Models offer a potential mechanism to overcome these challenges by enabling scalable interpretation and extraction of information from unstructured clinical text. Used in combination with structured data, this information can provide key contextual, socio-demographic and cultural nuances to improve health care, especially for traditionally marginalised communities with limited health access and representation in health data.