initial commit

.gitattributes

@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+model.onnx_data filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,3 +1,127 @@
----
-license: gemma
----
+---
+library_name: transformers
+tags:
+- colpali
+license: gemma
+datasets:
+- vidore/colpali_train_set
+language:
+- en
+base_model:
+- vidore/colpaligemma-3b-pt-448-base
+pipeline_tag: visual-document-retrieval
+---
+
+> [!IMPORTANT]
+> This version of ColPali should be loaded with the `transformers 🤗` release, not with `colpali-engine`.
+> It was converted using the [`convert_colpali_weights_to_hf.py` script](https://github.com/tonywu71/transformers/blob/21c1309637aee97ca4fb8eb3b31830913a0f99a5/src/transformers/models/colpali/convert_colpali_weights_to_hf.py)
+> from the [`vidore/colpali-v1.3-merged`](https://huggingface.co/vidore/colpali-v1.3-merged) checkpoint.
+
+# ColPali: Visual Retriever based on PaliGemma-3B with ColBERT strategy
+
+ColPali is a model based on a novel model architecture and training strategy based on Vision Language Models (VLMs) to efficiently index documents from their visual features.
+It is a [PaliGemma-3B](https://huggingface.co/google/paligemma-3b-mix-448) extension that generates [ColBERT](https://arxiv.org/abs/2004.12832)- style multi-vector representations of text and images. 
+It was introduced in the paper [ColPali: Efficient Document Retrieval with Vision Language Models](https://arxiv.org/abs/2407.01449) and first released in [this repository](https://github.com/ManuelFay/colpali)
+
+The HuggingFace `transformers` 🤗 implementation was contributed by Tony Wu ([@tonywu71](https://huggingface.co/tonywu71)) and Yoni Gozlan ([@yonigozlan](https://huggingface.co/yonigozlan)).
+
+<p align="center"><img width=800 src="https://github.com/illuin-tech/colpali/blob/main/assets/colpali_architecture.webp?raw=true"/></p>
+
+## Model Description
+
+Read the `transformers` 🤗 model card: https://huggingface.co/docs/transformers/en/model_doc/colpali.
+
+## Model Training
+
+### Dataset
+Our training dataset of 127,460 query-page pairs is comprised of train sets of openly available academic datasets (63%) and a synthetic dataset made up of pages from web-crawled PDF documents and augmented with VLM-generated (Claude-3 Sonnet) pseudo-questions (37%). 
+Our training set is fully English by design, enabling us to study zero-shot generalization to non-English languages. We explicitly verify no multi-page PDF document is used both [*ViDoRe*](https://huggingface.co/collections/vidore/vidore-benchmark-667173f98e70a1c0fa4db00d) and in the train set to prevent evaluation contamination. 
+A validation set is created with 2% of the samples to tune hyperparameters.
+
+*Note: Multilingual data is present in the pretraining corpus of the language model (Gemma-2B) and potentially occurs during PaliGemma-3B's multimodal training.*
+
+### Parameters
+
+All models are trained for 1 epoch on the train set. Unless specified otherwise, we train models in `bfloat16` format, use low-rank adapters ([LoRA](https://arxiv.org/abs/2106.09685)) 
+with `alpha=32`  and `r=32` on the transformer layers from the language model, 
+as well as the final randomly initialized projection layer, and use a `paged_adamw_8bit` optimizer. 
+We train on an 8 GPU setup with data parallelism, a learning rate of 5e-5 with linear decay with 2.5% warmup steps, and a batch size of 32.
+
+## Usage
+
+```python
+import torch
+from PIL import Image
+
+from transformers import ColPaliForRetrieval, ColPaliProcessor
+
+model_name = "vidore/colpali-v1.3-hf"
+
+model = ColPaliForRetrieval.from_pretrained(
+    model_name,
+    torch_dtype=torch.bfloat16,
+    device_map="cuda:0",  # or "mps" if on Apple Silicon
+).eval()
+
+processor = ColPaliProcessor.from_pretrained(model_name)
+
+# Your inputs
+images = [
+    Image.new("RGB", (32, 32), color="white"),
+    Image.new("RGB", (16, 16), color="black"),
+]
+queries = [
+    "What is the organizational structure for our R&D department?",
+    "Can you provide a breakdown of last year’s financial performance?",
+]
+
+# Process the inputs
+batch_images = processor(images=images).to(model.device)
+batch_queries = processor(text=queries).to(model.device)
+
+# Forward pass
+with torch.no_grad():
+    image_embeddings = model(**batch_images)
+    query_embeddings = model(**batch_queries)
+
+# Score the queries against the images
+scores = processor.score_retrieval(query_embeddings.embeddings, image_embeddings.embeddings)
+```
+
+## Resources
+
+- The *ColPali* arXiv paper can be found [here](https://doi.org/10.48550/arXiv.2407.01449). 📄
+- The official blog post detailing ColPali can be found [here](https://huggingface.co/blog/manu/colpali). 📝
+- The original model implementation code for the ColPali model and for the `colpali-engine` package can be found [here](https://github.com/illuin-tech/colpali). 🌎
+- Cookbooks for learning to use the transformers-native version of *ColPali*, fine-tuning, and similarity maps generation can be found [here](https://github.com/tonywu71/colpali-cookbooks). 📚
+
+## Limitations
+
+ - **Focus**: The model primarily focuses on PDF-type documents and high-ressources languages, potentially limiting its generalization to other document types or less represented languages.
+ - **Support**: The model relies on multi-vector retreiving derived from the ColBERT late interaction mechanism, which may require engineering efforts to adapt to widely used vector retrieval frameworks that lack native multi-vector support.
+
+## License
+
+ColPali's vision language backbone model (PaliGemma) is under `gemma` license as specified in its [model card](https://huggingface.co/google/paligemma-3b-mix-448). ColPali inherits from this `gemma` license.
+
+## Contact
+
+- Manuel Faysse: manuel.faysse@illuin.tech
+- Hugues Sibille: hugues.sibille@illuin.tech
+- Tony Wu: tony.wu@illuin.tech
+
+## Citation
+
+If you use any datasets or models from this organization in your research, please cite the original dataset as follows:
+
+```bibtex
+@misc{faysse2024colpaliefficientdocumentretrieval,
+  title={ColPali: Efficient Document Retrieval with Vision Language Models}, 
+  author={Manuel Faysse and Hugues Sibille and Tony Wu and Bilel Omrani and Gautier Viaud and Céline Hudelot and Pierre Colombo},
+  year={2024},
+  eprint={2407.01449},
+  archivePrefix={arXiv},
+  primaryClass={cs.IR},
+  url={https://arxiv.org/abs/2407.01449}, 
+}
+```

config.json

@@ -0,0 +1,55 @@
+{
+  "_attn_implementation_autoset": true,
+  "_name_or_path": "vidore/colpali-v1.3-hf",
+  "architectures": [
+    "ColPaliForRetrieval"
+  ],
+  "embedding_dim": 128,
+  "is_composition": false,
+  "model_type": "colpali",
+  "text_config": {
+    "hidden_size": 2048,
+    "intermediate_size": 16384,
+    "model_type": "gemma",
+    "num_attention_heads": 8,
+    "num_hidden_layers": 18,
+    "num_image_tokens": 1024,
+    "num_key_value_heads": 1,
+    "torch_dtype": "float32",
+    "vocab_size": 257216
+  },
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.48.3",
+  "vlm_config": {
+    "_name_or_path": "google/paligemma-3b-mix-448",
+    "_vocab_size": 257216,
+    "bos_token_id": 2,
+    "eos_token_id": 1,
+    "image_token_index": 257152,
+    "model_type": "paligemma",
+    "pad_token_id": 0,
+    "text_config": {
+      "hidden_size": 2048,
+      "intermediate_size": 16384,
+      "num_attention_heads": 8,
+      "num_hidden_layers": 18,
+      "num_image_tokens": 1024,
+      "num_key_value_heads": 1,
+      "torch_dtype": "float32",
+      "vocab_size": 257216
+    },
+    "torch_dtype": "float32",
+    "vision_config": {
+      "hidden_size": 1152,
+      "image_size": 448,
+      "intermediate_size": 4304,
+      "num_attention_heads": 16,
+      "num_hidden_layers": 27,
+      "num_image_tokens": 1024,
+      "patch_size": 14,
+      "projection_dim": 2048,
+      "projector_hidden_act": "gelu_fast",
+      "vision_use_head": false
+    }
+  }
+}

model.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a1d446daff68d97785f0baba276fa2faa00b5d97f8cfc498fc89897bd75f9c18
+size 2657565

model.onnx_data

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a1021ebbec07ae305969cefe0dc273998d90df0aa32689db31cfc6ae92299123
+size 11698461632

preprocessor_config.json

@@ -0,0 +1,25 @@
+{
+  "do_convert_rgb": null,
+  "do_normalize": true,
+  "do_rescale": true,
+  "do_resize": true,
+  "image_mean": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "image_processor_type": "SiglipImageProcessor",
+  "image_seq_length": 1024,
+  "image_std": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "processor_class": "ColPaliProcessor",
+  "resample": 3,
+  "rescale_factor": 0.00392156862745098,
+  "size": {
+    "height": 448,
+    "width": 448
+  }
+}

special_tokens_map.json

@@ -0,0 +1,39 @@
+{
+  "additional_special_tokens": [
+    {
+      "content": "<image>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    }
+  ],
+  "bos_token": {
+    "content": "<bos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<eos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:da2a07b519fc15ff0bbebb8c671c7802fad5012fef72f3d2f1b6d32057be694e
+size 16955946