train-CLIP 📎

A PyTorch Lightning solution to training CLIP from scratch.

Goal ⚽

Our aim is to create an easy to use Lightning implementation of OpenAI's clip training script. We want our end product to be as inline with the orignal paper as possible. We will live by:

TODO ✅

• Get OpenAI's model creation script
• Create model inits
  • ResNet50
  • ResNet50x4
  • ResNet101
  • ViT-B/32
  • all models
• Create model wrapper
• Create lightning trainer
• Create dataset files
• Performance boosts
  • Mixed-precision
  • Gradient checkpointing
  • Half-precision Adam statistics
  • Half-precision stochastically rounded text encoder weights