Duplicate from apple/MobileCLIP-B

Co-authored-by: Pedro Cuenca <pcuenq@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

LICENSE

@@ -0,0 +1,88 @@
+Disclaimer: IMPORTANT: This Apple Machine Learning Research Model is
+specifically developed and released by Apple Inc. ("Apple") for the sole purpose
+of scientific research of artificial intelligence and machine-learning
+technology. “Apple Machine Learning Research Model” means the model, including
+but not limited to algorithms, formulas, trained model weights, parameters,
+configurations, checkpoints, and any related materials (including
+documentation).
+
+This Apple Machine Learning Research Model is provided to You by
+Apple in consideration of your agreement to the following terms, and your use,
+modification, creation of Model Derivatives, and or redistribution of the Apple
+Machine Learning Research Model constitutes acceptance of this Agreement. If You
+do not agree with these terms, please do not use, modify, create Model
+Derivatives of, or distribute this Apple Machine Learning Research Model or
+Model Derivatives.
+
+* License Scope: In consideration of your agreement to abide by the following
+  terms, and subject to these terms, Apple hereby grants you a personal,
+  non-exclusive, worldwide, non-transferable, royalty-free, revocable, and
+  limited license, to use, copy, modify, distribute, and create Model
+  Derivatives (defined below) of the Apple Machine Learning Research Model
+  exclusively for Research Purposes. You agree that any Model Derivatives You
+  may create or that may be created for You will be limited to Research Purposes
+  as well. “Research Purposes” means non-commercial scientific research and
+  academic development activities, such as experimentation, analysis, testing
+  conducted by You with the sole intent to advance scientific knowledge and
+  research. “Research Purposes” does not include any commercial exploitation,
+  product development or use in any commercial product or service.
+
+* Distribution of Apple Machine Learning Research Model and Model Derivatives:
+  If you choose to redistribute Apple Machine Learning Research Model or its
+  Model Derivatives, you must provide a copy of this Agreement to such third
+  party, and ensure that the following attribution notice be provided: “Apple
+  Machine Learning Research Model is licensed under the Apple Machine Learning
+  Research Model License Agreement.” Additionally, all Model Derivatives must
+  clearly be identified as such, including disclosure of modifications and
+  changes made to the Apple Machine Learning Research Model. The name,
+  trademarks, service marks or logos of Apple may not be used to endorse or
+  promote Model Derivatives or the relationship between You and Apple. “Model
+  Derivatives” means any models or any other artifacts created by modifications,
+  improvements, adaptations, alterations to the architecture, algorithm or
+  training processes of the Apple Machine Learning Research Model, or by any
+  retraining, fine-tuning of the Apple Machine Learning Research Model.
+
+* No Other License: Except as expressly stated in this notice, no other rights
+  or licenses, express or implied, are granted by Apple herein, including but
+  not limited to any patent, trademark, and similar intellectual property rights
+  worldwide that may be infringed by the Apple Machine Learning Research Model,
+  the Model Derivatives or by other works in which the Apple Machine Learning
+  Research Model may be incorporated.
+
+* Compliance with Laws: Your use of Apple Machine Learning Research Model must
+  be in compliance with all applicable laws and regulations.
+
+* Term and Termination: The term of this Agreement will begin upon your
+  acceptance of this Agreement or use of the Apple Machine Learning Research
+  Model and will continue until terminated in accordance with the following
+  terms. Apple may terminate this Agreement at any time if You are in breach of
+  any term or condition of this Agreement. Upon termination of this Agreement,
+  You must cease to use all Apple Machine Learning Research Models and Model
+  Derivatives and permanently delete any copy thereof. Sections 3, 6 and 7 will
+  survive termination.
+
+* Disclaimer and Limitation of Liability: This Apple Machine Learning Research
+  Model and any outputs generated by the Apple Machine Learning Research Model
+  are provided on an “AS IS” basis. APPLE MAKES NO WARRANTIES, EXPRESS OR
+  IMPLIED, INCLUDING WITHOUT LIMITATION THE IMPLIED WARRANTIES OF
+  NON-INFRINGEMENT, MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE,
+  REGARDING THE APPLE MACHINE LEARNING RESEARCH MODEL OR OUTPUTS GENERATED BY
+  THE APPLE MACHINE LEARNING RESEARCH MODEL. You are solely responsible for
+  determining the appropriateness of using or redistributing the Apple Machine
+  Learning Research Model and any outputs of the Apple Machine Learning Research
+  Model and assume any risks associated with Your use of the Apple Machine
+  Learning Research Model and any output and results. IN NO EVENT SHALL APPLE BE
+  LIABLE FOR ANY SPECIAL, INDIRECT, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING
+  IN ANY WAY OUT OF THE USE, REPRODUCTION, MODIFICATION AND/OR DISTRIBUTION OF
+  THE APPLE MACHINE LEARNING RESEARCH MODEL AND ANY OUTPUTS OF THE APPLE MACHINE
+  LEARNING RESEARCH MODEL, HOWEVER CAUSED AND WHETHER UNDER THEORY OF CONTRACT,
+  TORT (INCLUDING NEGLIGENCE), STRICT LIABILITY OR OTHERWISE, EVEN IF APPLE HAS
+  BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+* Governing Law: This Agreement will be governed by and construed under the laws
+  of the State of California without regard to its choice of law principles. The
+  Convention on Contracts for the International Sale of Goods shall not apply to
+  the Agreement except that the arbitration clause and any arbitration hereunder
+  shall be governed by the Federal Arbitration Act, Chapters 1 and 2. 
+
+Copyright (C) 2025 Apple Inc. All Rights Reserved.

README.md

@@ -0,0 +1,65 @@
+---
+license: apple-amlr
+license_name: apple-ascl
+license_link: https://github.com/apple/ml-mobileclip/blob/main/LICENSE_weights_data
+library_name: mobileclip
+---
+
+# MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training
+
+MobileCLIP was introduced in [MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training
+](https://arxiv.org/pdf/2311.17049.pdf) (CVPR 2024), by Pavan Kumar Anasosalu Vasu, Hadi Pouransari, Fartash Faghri, Raviteja Vemulapalli, Oncel Tuzel.
+
+This repository contains the **MobileCLIP-B** checkpoint.
+
+![MobileCLIP Performance Figure](fig_accuracy_latency.png)
+
+### Highlights
+
+* Our smallest variant `MobileCLIP-S0` obtains similar zero-shot performance as [OpenAI](https://arxiv.org/abs/2103.00020)'s ViT-B/16 model while being 4.8x faster and 2.8x smaller.
+* `MobileCLIP-S2` obtains better avg zero-shot performance than [SigLIP](https://arxiv.org/abs/2303.15343)'s ViT-B/16 model while being 2.3x faster and 2.1x smaller, and trained with 3x less seen samples.
+* `MobileCLIP-B`(LT) attains zero-shot ImageNet performance of **77.2%** which is significantly better than recent works like [DFN](https://arxiv.org/abs/2309.17425) and [SigLIP](https://arxiv.org/abs/2303.15343) with similar architectures or even [OpenAI's ViT-L/14@336](https://arxiv.org/abs/2103.00020).
+
+## Checkpoints
+
+| Model                                                     | # Seen <BR>Samples (B) | # Params (M) <BR> (img + txt) | Latency (ms) <BR> (img + txt) | IN-1k Zero-Shot <BR> Top-1 Acc. (%) | Avg. Perf. (%) <BR> on 38 datasets |
+|:----------------------------------------------------------|:----------------------:|:-----------------------------:|:-----------------------------:|:-----------------------------------:|:----------------------------------:|
+| [MobileCLIP-S0](https://hf.co/pcuenq/MobileCLIP-S0)       |           13           |          11.4 + 42.4          |           1.5 + 1.6           |                67.8                 |                58.1                |
+| [MobileCLIP-S1](https://hf.co/pcuenq/MobileCLIP-S1)       |           13           |          21.5 + 63.4          |           2.5 + 3.3           |                72.6                 |                61.3                |
+| [MobileCLIP-S2](https://hf.co/pcuenq/MobileCLIP-S2)       |           13           |          35.7 + 63.4          |           3.6 + 3.3           |                74.4                 |                63.7                |
+| [MobileCLIP-B](https://hf.co/pcuenq/MobileCLIP-B)         |           13           |          86.3 + 63.4          |          10.4 + 3.3           |                76.8                 |                65.2                |
+| [MobileCLIP-B (LT)](https://hf.co/pcuenq/MobileCLIP-B-LT) |           36           |          86.3 + 63.4          |          10.4 + 3.3           |                77.2                 |                65.8                |
+
+## How to Use
+
+First, download the desired checkpoint visiting one of the links in the table above, then click the `Files and versions` tab, and download the PyTorch checkpoint.
+For programmatic downloading, if you have `huggingface_hub` installed, you can also run:
+
+```
+huggingface-cli download pcuenq/MobileCLIP-B
+``` 
+
+Then, install [`ml-mobileclip`](https://github.com/apple/ml-mobileclip) by following the instructions in the repo. It uses an API similar to [`open_clip`'s](https://github.com/mlfoundations/open_clip).
+You can run inference with a code snippet like the following:
+
+```py
+import torch
+from PIL import Image
+import mobileclip
+
+model, _, preprocess = mobileclip.create_model_and_transforms('mobileclip_b', pretrained='/path/to/mobileclip_b.pt')
+tokenizer = mobileclip.get_tokenizer('mobileclip_b')
+
+image = preprocess(Image.open("docs/fig_accuracy_latency.png").convert('RGB')).unsqueeze(0)
+text = tokenizer(["a diagram", "a dog", "a cat"])
+
+with torch.no_grad(), torch.cuda.amp.autocast():
+    image_features = model.encode_image(image)
+    text_features = model.encode_text(text)
+    image_features /= image_features.norm(dim=-1, keepdim=True)
+    text_features /= text_features.norm(dim=-1, keepdim=True)
+
+    text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
+
+print("Label probs:", text_probs)
+```

config.json

@@ -0,0 +1,18 @@
+{
+    "embed_dim": 512,
+    "image_cfg": {
+        "image_size": 224,
+        "model_name": "vit_b16"
+    },
+    "text_cfg": {
+        "context_length": 77,
+        "vocab_size": 49408,
+        "dim": 512,
+        "ffn_multiplier_per_layer": 4.0,
+        "n_heads_per_layer": 8,
+        "n_transformer_layers": 12,
+        "norm_layer": "layer_norm_fp32",
+        "causal_masking": true,
+        "model_name": "base"
+    }
+}

mobileclip_b.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0dd1d92c911841121a0a82af2db74941ffa78bc05d215334b95db030242ec7f9
+size 599218731