Arshia Hemmat

Arshia Hemmat

Researcher at University of Oxford

Posts by Collection

portfolio

publications

CLIP Exhibits Improved Compositional Generalization Through Representation Disentanglement

Published in conference, 2023

Vision-language models (VLMs), such as CLIP, have shown promising Out-of-Distribution (OoD) generalization under various flavors of distribution shifts. Recent studies attempted to investigate the leading cause of this property. In this work, we target the same goal, but focus on a certain type of distribution shift, in which test images contain unseen compositions of attribute-object pairs, but with the objects and attributes being individually seen during training. The models are expected to classify those images into the composition classes, i.e. attribute-object pairs, and also into object classes by ignoring attributes. We carefully designed an authentic image test dataset consisting of attributes for objects that are unlikely encountered in the CLIP training data. We found that the compositions diversity in the training data, as measured by normalized mutual information between objects and attributes, has a significant effect on the improvement of compositional generalization in the CLIP models. We found that image/text representation disentanglement with respect to the composition constituents also plays a key role in the improved generalization of these models. We notice that larger training datasets could potentially trigger emergence of such a disentanglement, as the compositions are typically more diverse in such datasets. We validate this hypothesis through different representation disentanglement metrics, including Z-Diff, and explicitness scores for various CLIPs.

Download here

talks

Zero-shot Learning in Medical Domain

Published:

Focusing on developing models capable of recognizing unseen medical conditions from imaging data, this research tackles the challenge of improving diagnostic tools without extensive labeled datasets. Dr. Peyman Adibi guides the project, aiming to revolutionize diagnostic methodologies through zero-shot learning.

Text Summarization Using Graph Neural Network

Published:

Under Dr. Hamidreza Baradaran’s guidance, this project employs graph neural networks for efficient text summarization. The objective is to enhance the coherence and relevance of automated summaries, facilitating improved information retrieval and understanding across large text corpora.

Vision-Linguistic Models

Published:

This project aims to enhance the synergy between visual perception and language processing in AI systems, exploring the boundaries of machine understanding and interpretation of complex visual-textual information. The work under Prof. Mohammad Hossein Rohban investigates innovative methodologies to advance the capabilities of vision-linguistic models.

3D Vessel Segmentation Using Geometric Approaches

Published:

This research focuses on leveraging geometric deep learning techniques for the accurate segmentation of blood vessels in 3D medical images. The aim is to enhance diagnostic and therapeutic procedures in medicine, showcasing the potential of geometric approaches in complex image segmentation tasks.

Counterfactual Modelling Using Vision-Linguistic Models

Published:

Investigates generating counterfactual scenarios through integration of visual and linguistic data to enhance AI interpretability and reliability, focusing on applications such as autonomous driving and medical diagnostics. This research highlights the potential of vision-linguistic models in creating diverse, hypothetical scenarios for advanced problem-solving and decision-making processes.

teaching

Deep Learning

Graduate course, Sharif University of Technology, Department of EE, 2023

Machine Learning

undergraduate course, University of Isfahan, Department of CE, 2023