[MedMod] Add MedFuse model for multimodal EHR and CXR fusion

[Rose-Wei10]

Contributor

Contributor: Rong Wei
Net ID: rongwei5
Email: rongwei5@illinois.edu

Contribution Type

Model

Original Paper Reference

• Title: MedMod: Multimodal Benchmark for Medical Prediction Tasks with Electronic Health Records and Chest X-Ray Scans
• Link: https://proceedings.mlr.press/v287/elsharief25a.html
• Venue: Proceedings of the 6th Conference on Health, Inference, and Learning (CHIL 2025)

Description

This PR implements the MedFuse model, which enables PyHealth to perform multimodal fusion between Electronic Health Records (EHR) and Chest X-Ray (CXR) embeddings. It reproduces the fusion logic proposed in the MedMod paper.

Technical Implementation Details:

• Dual-Channel Encoding: Implements an LSTM-based encoder for sequential clinical data (EHR) and a Linear projection layer to align pre-extracted image features (CXR) into a shared latent space.
• Modality Fusion: Features are combined using a late-fusion strategy (concatenation) before being passed to a final classification head for mortality prediction.
• Robustness: Includes logic to handle missing CXR data by substituting zero-tensors of the appropriate embedding dimension, ensuring the pipeline is robust to incomplete data scenarios.

Files to Review

• pyhealth/models/medfuse.py: New MedFuse model implementation, including EHR/Image encoders and fusion logic.
• tests/test_medfuse.py: Comprehensive unit tests verifying instantiation, forward pass validation, and gradient computation.
• examples/mimic3_mortality_medfuse.py: Example usage and ablation study comparing hidden dimensions (64 vs 256) on synthetic MIMIC-III data.
• docs/api/models/pyhealth.models.medfuse.rst: API documentation for the new model.
• pyhealth/models/__init__.py: Updated to export the MedFuse model.