Update README.md

README.md

@@ -1,65 +1,151 @@
+````markdown
+# 📸 MobileCLIP-B Zero-Shot Image Classifier — HF Inference Endpoint
+
+This repository packages Apple’s **MobileCLIP-B** model as a production-ready
+Hugging Face Inference Endpoint.
+
+* **One-shot image → class probabilities**  
+  ⚡ < 30 ms on an A10G / T4 once the image arrives.
+* **Branch-fused / FP16** MobileCLIP for fast GPU inference.
+* **Pre-computed text embeddings** for your custom label set
+  (`items.json`) — every request encodes **only** the image.
+* Built with vanilla **`open-clip-torch`** (no forks) and a
+  60-line local helper (`reparam.py`) to fuse MobileOne blocks.
+
 ---
-license: apple-amlr
-license_name: apple-ascl
-license_link: https://github.com/apple/ml-mobileclip/blob/main/LICENSE_weights_data
-library_name: mobileclip
----
 
-# MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training
+## ✨ What’s inside
+
+| File | Purpose |
+|------|---------|
+| `handler.py` | Hugging Face entry-point — loads weights, caches text features, serves requests |
+| `reparam.py` | Stand-alone copy of `reparameterize_model` from Apple’s repo (removes heavy upstream dependency) |
+| `requirements.txt` | Minimal, conflict-free dependency set (`torch`, `torchvision`, `open-clip-torch`) |
+| `items.json` | Your label spec — each element must have `id`, `name`, and `prompt` fields |
+| `README.md` | You are here |
 
-MobileCLIP was introduced in [MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training
-](https://arxiv.org/pdf/2311.17049.pdf) (CVPR 2024), by Pavan Kumar Anasosalu Vasu, Hadi Pouransari, Fartash Faghri, Raviteja Vemulapalli, Oncel Tuzel.
+---
 
-This repository contains the **MobileCLIP-B** checkpoint.
+## 🔧 Quick start (local smoke-test)
 
-![MobileCLIP Performance Figure](fig_accuracy_latency.png)
+```bash
+python -m venv venv && source venv/bin/activate
+pip install -r requirements.txt
+python - <<'PY'
+from pathlib import Path, PurePosixPath
+import base64, json, requests
 
-### Highlights
+# Load a demo image and encode it
+img_path = Path("tests/cat.jpg")
+payload = {
+    "image": base64.b64encode(img_path.read_bytes()).decode()
+}
 
-* Our smallest variant `MobileCLIP-S0` obtains similar zero-shot performance as [OpenAI](https://arxiv.org/abs/2103.00020)'s ViT-B/16 model while being 4.8x faster and 2.8x smaller.
-* `MobileCLIP-S2` obtains better avg zero-shot performance than [SigLIP](https://arxiv.org/abs/2303.15343)'s ViT-B/16 model while being 2.3x faster and 2.1x smaller, and trained with 3x less seen samples.
-* `MobileCLIP-B`(LT) attains zero-shot ImageNet performance of **77.2%** which is significantly better than recent works like [DFN](https://arxiv.org/abs/2309.17425) and [SigLIP](https://arxiv.org/abs/2303.15343) with similar architectures or even [OpenAI's ViT-L/14@336](https://arxiv.org/abs/2103.00020).
+# Local simulation — spin up uvicorn the same way the HF container does
+import handler, uvicorn
+app = handler.EndpointHandler()
 
-## Checkpoints
+print(app({"inputs": payload})[:5])   # top-5 classes
+PY
+````
 
-| Model                                                     | # Seen <BR>Samples (B) | # Params (M) <BR> (img + txt) | Latency (ms) <BR> (img + txt) | IN-1k Zero-Shot <BR> Top-1 Acc. (%) | Avg. Perf. (%) <BR> on 38 datasets |
-|:----------------------------------------------------------|:----------------------:|:-----------------------------:|:-----------------------------:|:-----------------------------------:|:----------------------------------:|
-| [MobileCLIP-S0](https://hf.co/pcuenq/MobileCLIP-S0)       |           13           |          11.4 + 42.4          |           1.5 + 1.6           |                67.8                 |                58.1                |
-| [MobileCLIP-S1](https://hf.co/pcuenq/MobileCLIP-S1)       |           13           |          21.5 + 63.4          |           2.5 + 3.3           |                72.6                 |                61.3                |
-| [MobileCLIP-S2](https://hf.co/pcuenq/MobileCLIP-S2)       |           13           |          35.7 + 63.4          |           3.6 + 3.3           |                74.4                 |                63.7                |
-| [MobileCLIP-B](https://hf.co/pcuenq/MobileCLIP-B)         |           13           |          86.3 + 63.4          |          10.4 + 3.3           |                76.8                 |                65.2                |
-| [MobileCLIP-B (LT)](https://hf.co/pcuenq/MobileCLIP-B-LT) |           36           |          86.3 + 63.4          |          10.4 + 3.3           |                77.2                 |                65.8                |
+---
 
-## How to Use
+## 🚀 Calling the deployed endpoint
+
+```bash
+ENDPOINT_URL="https://<your-endpoint>.aws.endpoints.huggingface.cloud"
+HF_TOKEN="hf_xxxxxxxxxxxxxxxxx"
+IMG="cat.jpg"
+
+python - <<'PY'
+import base64, json, requests, sys, os
+url   = os.environ["ENDPOINT_URL"]
+token = os.environ["HF_TOKEN"]
+img   = sys.argv[1]
+
+payload = {
+    "inputs": {
+        "image": base64.b64encode(open(img, "rb").read()).decode()
+    }
+}
+resp = requests.post(
+    url,
+    headers={
+        "Authorization": f"Bearer {token}",
+        "Content-Type":  "application/json",
+        "Accept":        "application/json",
+    },
+    json=payload,
+    timeout=60,
+)
+print(json.dumps(resp.json()[:5], indent=2))   # top-5
+PY
+$IMG
+```
 
-First, download the desired checkpoint visiting one of the links in the table above, then click the `Files and versions` tab, and download the PyTorch checkpoint.
-For programmatic downloading, if you have `huggingface_hub` installed, you can also run:
+Sample response:
 
+```json
+[
+  { "id": 23, "label": "cat",          "score": 0.92 },
+  { "id": 11, "label": "tiger cat",    "score": 0.05 },
+  { "id": 48, "label": "siamese cat",  "score": 0.02 },
+  …
+]
 ```
-huggingface-cli download pcuenq/MobileCLIP-B
-``` 
 
-Then, install [`ml-mobileclip`](https://github.com/apple/ml-mobileclip) by following the instructions in the repo. It uses an API similar to [`open_clip`'s](https://github.com/mlfoundations/open_clip).
-You can run inference with a code snippet like the following:
+---
+
+## 🏗️ How the handler works (high-level)
+
+1. **Startup**
+
+   * Downloads / loads the `datacompdr` MobileCLIP-B checkpoint.
+   * Runs `reparameterize_model` to fuse MobileOne branches.
+   * Reads `items.json`, tokenises all prompts, and caches the resulting
+     text embeddings (`[n_classes, 512]`).
 
-```py
-import torch
-from PIL import Image
-import mobileclip
+2. **Per request**
 
-model, _, preprocess = mobileclip.create_model_and_transforms('mobileclip_b', pretrained='/path/to/mobileclip_b.pt')
-tokenizer = mobileclip.get_tokenizer('mobileclip_b')
+   * Decodes the incoming base-64 JPEG/PNG.
+   * Applies the exact OpenCLIP preprocessing (224 × 224 center-crop,
+     mean/std normalisation).
+   * Encodes the image, L2-normalises, and performs one `softmax(cosine)`
+     against the cached text matrix.
+   * Returns a sorted JSON list `[{"id", "label", "score"}, …]`.
 
-image = preprocess(Image.open("docs/fig_accuracy_latency.png").convert('RGB')).unsqueeze(0)
-text = tokenizer(["a diagram", "a dog", "a cat"])
+This design keeps bandwidth low (compressed image over the wire) and
+latency low (no per-request text encoding).
 
-with torch.no_grad(), torch.cuda.amp.autocast():
-    image_features = model.encode_image(image)
-    text_features = model.encode_text(text)
-    image_features /= image_features.norm(dim=-1, keepdim=True)
-    text_features /= text_features.norm(dim=-1, keepdim=True)
+---
+
+## 📝 Updating the label set
+
+Edit `items.json`, **rebuild the endpoint**, done.
+
+```json
+[
+  { "id": 0, "name": "cat",  "prompt": "a photo of a cat"  },
+  { "id": 1, "name": "dog",  "prompt": "a photo of a dog"  },
+  …
+]
+```
+
+* `id` is your internal numeric key (stays stable).
+* `name` is the human-readable label returned to clients.
+* `prompt` is what the model actually “sees” — tweak wording to improve accuracy.
 
-    text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
+---
+
+## ⚖️ Licence
+
+* **Weights**: Apple AMLR (see [`LICENSE_weights_data`](./LICENSE_weights_data)).
+* **Code in this repo**: MIT.
 
-print("Label probs:", text_probs)
+---
+
+<div align="center"><sub>Maintained with ❤️ by Your Team — August 2025</sub></div>
 ```
+::contentReference[oaicite:0]{index=0}
+