Checking Medical Process Conformance by Exploiting LLMs

Clinical guidelines, which represent the normative process models for healthcare organizations, are typically available in a textual, unstructured form. This issue hampers the application of classical conformance-checking algorithms to the medical domain, which take in input of a formalized and computer-interpretable description of the process. In this paper, (i) we propose overcoming this problem by taking advantage of a Large Language Model (LLM), in order to extract normative rules from textual guidelines; (ii) we then check and quantify the conformance of the patient event log with respect to such rules. Additionally, (iii) we adopt the approach as a means for evaluating the quality of the models mined by different process discovery algorithms from the event log, by comparing their conformance to the rules. We have tested our work in the domain of stroke. As regards conformance checking, we have proved the compliance of four Northern Italy hospitals to a general rule for diagnosis timing and to two rules that refer to thrombolysis treatment, and have identified some issues related to other rules, which involve the availability of magnetic resonance instruments. As regards process model discovery evaluation, we have assessed the superiority of Heuristic Miner with respect to other mining algorithms on our dataset. It is worth noting that the easy extraction of rules in our LLM-assisted approach would make it quickly applicable to other fields as well.