[Tech Blog] CLIP (Contrastive Language-Image Pre-training) Explained
kotai2003TLDR
Paper title, authors and year
- Title: Learning Transferable Visual Models From Natural Language Supervision
- Authors: Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever
- Year: 2021
- Link: https://arxiv.org/abs/2103.00020
What is it?
This paper proposes a new training method called CLIP (Contrastive Language-Image Pre-training). It trains a vision model under natural language supervision, using 400 million image-text pairs collected from the internet. CLIP learns through the task of predicting which text goes with which image, and can then use text to transfer to new visual concepts zero-shot.
What makes it better than prior work?
Conventional computer vision models are trained on a fixed set of object categories, which makes zero-shot learning difficult. CLIP's strength is that, by learning from natural language, it acquires visual representations that do not depend on fixed categories and can adapt to a wide range of tasks. CLIP can also transfer zero-shot to multiple tasks, with accuracy equal to or better than conventional fully supervised models.
What is the key to the technique?
The key to CLIP is contrastive pre-training on image-text pairs. A text encoder and an image encoder are trained jointly to predict whether an image and a text are a correct pair. This lets CLIP learn a broad range of visual concepts and refer to or describe them using text.
How was it validated?
CLIP was evaluated on more than 30 different computer vision tasks, including OCR, action recognition, geo-localization and fine-grained object classification, and performed well zero-shot on many of them. Notably, on the ImageNet dataset it matched the accuracy of ResNet-50 without using any of the 1.28 million labeled training examples.
Is there any discussion?
By using data diversity and natural language, CLIP's approach reduces a problem of conventional models, in particular their tendency to overfit to a specific dataset. On the other hand, it has been pointed out that zero-shot learning is not optimal for every task, and performance can be weaker, especially on complex or abstract tasks.
What should I read next?
Good next reads are other studies that use natural language supervision similar to CLIP, such as the papers on “VirTex,” “ICMLM” and “ConVIRT.” It is also worth following recent research trends in visual representation learning and zero-shot learning.
Explanation
CLIP: The Birth of a New Vision Model That Learns from Image-Text Pairs
The paper “Learning Transferable Visual Models From Natural Language Supervision,” published in 2021, is an important study that points to a new direction for visual recognition. It describes in detail a model called CLIP (Contrastive Language-Image Pre-training), proposed by researchers at OpenAI. The model takes a landmark approach: it trains a vision model through natural language, using 400 million image-text pairs collected from the internet.
1. What Is CLIP?
Unlike conventional vision models, CLIP does not depend on a fixed set of object categories. Conventional models were trained on specific categories (for example, “dog” and “cat”) and had difficulty adapting to other tasks. By using natural language, CLIP is able to transfer a broad range of visual concepts zero-shot.
Zero-shot refers to a model's ability to perform new tasks it was not trained on. For example, given a specific text such as “space station,” CLIP can correctly identify the corresponding images. This ability achieves accuracy equal to or better than conventional fully supervised models.
2. CLIP's Technical Background and Significance
The core technique of CLIP is contrastive pre-training. An image encoder and a text encoder are trained jointly to predict whether an image and a text are a correct pair. Through this contrastive learning, CLIP learns a wide variety of visual concepts and can refer to them with text.
CLIP is also trained on a large-scale dataset, and has been shown to perform well zero-shot on general vision tasks. Notably, in tests on the ImageNet dataset, CLIP matched the accuracy of ResNet-50 without using the 1.28 million labeled examples.
Figure 1
Description: Summary of our approach. While standard image models jointly train an image feature extractor and a linear classifier to predict some label, CLIP jointly trains an image encoder and a text encoder to predict the correct pairings of a batch of (image, text) training examples. At test time, the learned text encoder synthesizes a zero-shot linear classifier by embedding the names or descriptions of the target dataset's classes.
3. CLIP's Strengths and Limitations
By using data diversity and natural language, CLIP reduces the problem conventional vision models have of overfitting to a specific dataset. However, zero-shot learning is not optimal for every task, and performance can be weaker, especially on complex or abstract tasks.
CLIP's approach also compares favorably with other training methods that use natural language supervision; its effectiveness stands out especially against similar studies such as “VirTex,” “ICMLM” and “ConVIRT.”
4. Outlook for CLIP
CLIP's success opens new possibilities for visual recognition and is expected to have a major influence on future research. As next reads, studies using methods similar to CLIP are recommended. It is also important to follow the latest research on visual representation learning and zero-shot learning.
As this study shows, the future of vision models that learn from natural language is very promising, and further progress is expected.
5. CLIP's Technical Details and Mechanisms
CLIP incorporates several technical ideas that support its novelty and versatility. Here we explain the specific mechanisms: how CLIP builds its vision model and achieves zero-shot learning using natural language.
5.1 Contrastive Pre-training
CLIP's central technique is contrastive pre-training. It trains the model using image-text pairs, and specifically involves the following process:
-
Creating data pairs: First, 400 million image-text pairs collected from the internet are used. Each pair is presented to the model through the task “which text is the correct description of this image?”
-
Training the encoders: CLIP has an image encoder that processes images and a text encoder that processes text. The two encoders are trained jointly and produce feature vectors used to predict whether an image and a text match.
-
Computing cosine similarity: The cosine similarity between the image and text vectors is computed and adjusted so that correct pairs have higher similarity. Training proceeds so that incorrect pairs have lower similarity.
-
Optimizing the loss function: Finally, a contrastive loss function is used to train the model to identify pairs correctly. Specifically, the model is adjusted to maximize the similarity gap between real pairs and incorrect pairs.
5.2 Zero-Shot Learning and Model Transfer
Another important feature of CLIP is its ability to transfer what it has learned to other tasks zero-shot. This is because CLIP understands visual concepts through natural language and can adapt to new tasks using text.
-
Zero-shot classification: For example, for a new dataset, CLIP takes the dataset's class labels as text input and predicts which class each image belongs to. CLIP computes the cosine similarity between the image and the text of each class label, and predicts the class with the highest similarity.
-
Prompt engineering: To improve zero-shot classification accuracy, it is effective to turn class labels into more contextual sentences rather than single words. For example, changing the label “dog” to “a photo of a dog” helps CLIP understand the intent of the text more accurately.
5.3 CLIP's Scaling and Efficiency
CLIP's training is very large in scale, so efficiency is also an important factor. CLIP consists of several different models (ResNet and Vision Transformer), each trained with different computational resources.
-
ResNet models: CLIP uses models based on the conventional ResNet architecture, with several modifications to improve efficiency. For example, replacing ResNet's pooling layer with an attention mechanism lets it capture image features more effectively.
-
Vision Transformer (ViT): CLIP also adopts the ViT architecture, which has drawn attention recently. This architecture is computationally efficient and performs especially well on large datasets.
Training CLIP requires very expensive computational resources, yet CLIP delivers competitive performance in zero-shot classification and substantially reduces computational cost compared with conventional fully supervised models.
6. Conclusion
CLIP is a powerful model that shows new possibilities for visual recognition. Understanding the details of its internal mechanisms and training process makes clear how it handles a wide range of tasks and achieves zero-shot learning.
However, CLIP still has technical challenges, and overcoming them will require more diverse data and stronger models. We look forward to CLIP's further development.
