malper/hierarcaps-clip-l

HierarCaps CLIP-L

Fine-tuned CLIP-L (ViT-L/14) checkpoint from the paper Emergent Visual-Semantic Hierarchies in Image-Text Representations (ECCV 2024).

• Paper: https://arxiv.org/abs/2407.08521
• Project page: https://hierarcaps.github.io/
• Code: https://github.com/TAU-VAILab/hierarcaps
• Base model: openai/clip-vit-large-patch14

This model was fine-tuned on the HierarCaps dataset to enhance hierarchical organization in CLIP's embedding space, improving hierarchical reasoning about specificity between images and texts.

Usage

from transformers import CLIPModel, AutoProcessor

model = CLIPModel.from_pretrained("malper/hierarcaps-clip-l")
processor = AutoProcessor.from_pretrained("malper/hierarcaps-clip-l")

See the official code repository for training and evaluation scripts used with this checkpoint.

Citation

@InProceedings{alper2024hierarcaps,
  author    = {Morris Alper and Hadar Averbuch-Elor},
  title     = {Emergent Visual-Semantic Hierarchies in Image-Text Representations},
  booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)},
  year      = {2024}
}