Séminaire TAD

Cross-modal Swin Transformer for computer-assisted diagnosis of age-related macular degeneration

Sébastien Richard

Age-related macular degeneration (AMD) is one of the leading causes of legal blindness worldwide. However, manual screening of the retina for diagnosis is impractical for large populations,and computer-assisted diagnosis models provide a scalable alternative. Existing approaches often rely on a single imaging modality, overlooking modality-specific information, and typically depend on single or few curated datasets that do not reflect real-world clinical variability, resulting in poor generalization. To address these challenges, we introduce a cross-modal Swin-Transformer model that integrates optical coherence tomography (OCT) and color fundus photography (CFP) for AMD diagnosis. This model focuses on one key biomarker and use multi-label classification. Our approach leverages a meta-dataset compiled from international sources, including both labeled and unlabeled data. The model is trained through a two-stage process: first using a self-supervised multimodal pretraining method based on Barlow's Twins, followed by fine-tuning with loose pairing and spatially-limited cross attention with 2.5D integration. Experimental results demonstrate that this approach improve classification metrics over baseline.