Context-Dependent FHIR Serialization Strategies for Clinical LLM Deployment: A Multi-Layer Benchmark on UK Core Data
Context-Dependent FHIR Serialisation Strategies for Clinical LLM Deployment - medRxiv
Summary
This study demonstrated that the choice of serialization format when feeding FHIR data to Large Language Models (LLMs) significantly impacts clinical quality. The optimal format is context-dependent, requiring task-specific selection such as 'raw_json,' 'hybrid_adaptive,' or 'structured_markdown.'
Details
Using a benchmark called "FHIRBench-UK," this study evaluated five large language models across six serialization formats using 100 UK Core FHIR patient bundles. The findings indicate that no single optimal format exists; instead, the appropriate strategy must be selected based on the specific task. Specifically, raw_json was found to dominate for clinical Question Answering (QA), hybrid_adaptive was best for clinical reasoning, and structured_markdown was optimal for summarization. Furthermore, model capability matters: adaptive serialization is highlighted as particularly valuable for budget-constrained deployments using mid-tier models. Crucially, the study confirmed a strong inverse correlation (rho=-0.90) between token-level F1 scores and actual clinical quality, suggesting that technical metrics alone are insufficient for LLM evaluation. The authors recommend implementing "task-aware serialization routing" as a zero-cost quality intervention for FHIR-based LLM deployments in the NHS.
Original content copyright by respective publishers