Multimodal representation learning of brain pathology

Image-text retrieval model inspired by CLIP.

Abstract: We propose a method for learning features of pairs of magnetic resonance images of human brains and their respective medical report. This representation can be used to train a classifier, or to match other samples with similar features by computing a similarity score. We train a deep learning model with a contrastive loss, which learns to match image-text pairs. We depend on well-established state-of-the-art methods to construct our architecture. Finally, we examine the obtained results, demonstrating that although the suggested method is not yet optimal, it still exhibits promising behavior that could be investigated further in future work.

One case from the dataset and its available MRI sequences. The proposed model was trained only on DWI B1000, T2S and T2 FLAIR.

Example of a similarity matrix computed on one batch during training. The ideal matrix is the identity matrix, with values equal to one (bright) on the diagonal. However, when training on similar sample, we would also expect images with the same class to have higher similarity values.

For each report in the test set, we compute its similarity between each other sample. We plot the similarity score for each pair-pair match only if they have the same ground truth class annotation..