Clinical Data Platform for Integrating FHIR and CSV Clinical Research Data
Clinical Data Platform | Jinyan Shao
Summary
Jinyan Shao developed a 'Clinical Data Platform' to process clinical research data in both FHIR and CSV formats. The system normalizes, validates, and manages provenance, enabling traceable data ingestion across multiple sources and studies.
Details
This platform addresses the challenge of handling inconsistent source exports from clinical research teams using different formats (FHIR and CSV). Its core strength lies in a shared pipeline that centralizes normalization, validation, domain services, persistence, study binding, and provenance management, going beyond simple data transformation. Technically, it is built on a FastAPI backend with FHIR-aware data modeling and PostgreSQL persistence. To ensure data reliability, Celery workers and Redis are used for queueing, retries, and timeouts, recording the status (completed, partially failed) and versioned error history of every attempt. Furthermore, external identity is defined as a pair (source_namespace, external_id), and import idempotency is guaranteed by hashing the file checksum along with the target study and namespace. This prevents duplicate uploads or collisions while enforcing 'study-scoped authorization,' ensuring researchers only access patients bound to studies they are authorized for. The design provides robust ingestion processes and auditability for handling semi-structured data like CSV, all built upon a FHIR foundation, thereby contributing to solving complex challenges in clinical data lifecycle management.
Original content copyright by respective publishers