Update model card for Mono-InternVL-2B with Mono-InternVL-1.5 paper and comprehensive details

This PR updates the model card for `Mono-InternVL-2B` to reflect information from the more recent paper [Mono-InternVL-1.5: Towards Cheaper and Faster Monolithic Multimodal Large Language Models](https://huggingface.co/papers/2507.12566).

Specifically, it:
- Updates the primary paper link and enriches the introduction with details from the Mono-InternVL-1.5 abstract and the project's introduction.
- Adds visual charts (radar and architecture) from the GitHub repository for better illustration.
- Includes the comprehensive performance benchmark table.
- Expands the inference section to include both `transformers` and `LMDeploy` examples.
- Adds a new section for supervised finetuning, including installation, dataset preparation, and training instructions.
- Updates the citation section to include both Mono-InternVL V1 and V1.5 papers for complete attribution.
- Removes the "File information" section, as it is internal context and not part of the public model card.

Please review and merge this PR.

README.md

@@ -1,38 +1,417 @@
 ---
-license: mit
-pipeline_tag: image-text-to-text
-library_name: transformers
 base_model:
-  - internlm/internlm2-chat-1_8b
-base_model_relation: merge
+- internlm/internlm2-chat-1_8b
 language:
-  - multilingual
+- multilingual
+library_name: transformers
+license: mit
+pipeline_tag: image-text-to-text
 tags:
-  - internvl
-  - vision
-  - ocr
-  - custom_code
-  - moe
+- internvl
+- vision
+- ocr
+- custom_code
+- moe
+base_model_relation: merge
 ---
 
 # Mono-InternVL-2B
 
 This repository contains the instruction-tuned Mono-InternVL-2B model, which has 1.8B activated parameters (3B in total). It is built upon [internlm2-chat-1_8b](https://huggingface.co/internlm/internlm2-chat-1_8b).
 
-Please refer to our [**paper**](https://huggingface.co/papers/2410.08202), [**project page**](https://internvl.github.io/blog/2024-10-10-Mono-InternVL/) and [**GitHub repository**](https://github.com/OpenGVLab/mono-internvl) for introduction and usage.
+**Mono-InternVL-2B** is part of the **Mono-InternVL-1.5** family, presented in the paper [Mono-InternVL-1.5: Towards Cheaper and Faster Monolithic Multimodal Large Language Models](https://huggingface.co/papers/2507.12566). Mono-InternVL-1.5 integrates visual encoding and language decoding into a single model, addressing optimization challenges and catastrophic forgetting common in monolithic MLLMs. It does this by embedding a new visual parameter space into a pre-trained LLM, enabling stable learning of visual knowledge from noisy data via delta tuning. This version features improved Endogenous Visual Pre-training (EViP++) with additional visual attention experts and re-organized pre-training for efficiency. During inference, it includes a fused CUDA kernel to speed up MoE operations, significantly reducing training and inference costs while maintaining competitive performance.
+
+Please refer to our [**project page**](https://internvl.github.io/blog/2024-10-10-Mono-InternVL/) and [**GitHub repository**](https://github.com/OpenGVLab/mono-internvl) for further introduction and usage.
+
+<p align="center">
+<img src="https://github.com/OpenGVLab/mono-internvl/raw/main/images/fig1.jpg" alt="radar chart" style="width: 100%; height: auto;" />
+<br>
+<br>
+<img src="https://github.com/OpenGVLab/mono-internvl/raw/main/images/fig2.jpg" alt="architecture" style="width: 100%; height: auto;" />
+</p>
+
+## Introduction
+
+We release Mono-InternVL, a **monolithic** multimodal large language model (MLLM) that integrates visual encoding and textual decoding into a single LLM. In Mono-InternVL, a set of visual experts is embedded into the pre-trained LLM via a **mixture-of-experts (MoE) mechanism**. By freezing the LLM, Mono-InternVL ensures that visual capabilities are optimized without compromising the pre-trained language knowledge. Based on this structure, an innovative **Endogenous Visual Pretraining (EViP)** is introduced to realize coarse-to-fine visual learning.
+
+Mono-InternVL achieves superior performance compared to state-of-the-art MLLM Mini-InternVL-2B-1.5 and significantly outperforms other monolithic MLLMs, as shown in the radar chart above. Meanwhile, it achieves better deployment efficiency, with first token latency reduced by up to 67%.
+
+## Performance
+
+|          Benchmark           | Chameleon-7B | EVE-7B (HD) |    Emu3    | Mini-InternVL-2B-1-5 | Mono-InternVL-2B |
+| :--------------------------: | :----------: | :---------: | :--------: | :------------------: | :--------------: |
+|             Type             |  Monolithic  | Monolithic  | Monolithic |       Modular        |    Monolithic    |
+|      #Activated Params       |      7B      |     7B      |     8B     |         2.2B         |       1.8B       |
+|                              |              |             |            |                      |                  |
+|            MMVet             |     8.3      |    25.7     |    37.2    |         39.3         |       40.1       |
+|      MMMU<sub>val</sub>      |     25.4     |    32.6     |    31.6    |         34.6         |       33.7       |
+|      MME<sub>sum</sub>       |     170      |    1628     |     —      |         1902         |       1875       |
+|  MMBench-EN<sub>test</sub>   |     31.1     |    52.3     |    58.5    |         70.9         |       65.5       |
+| MathVista<sub>testmini</sub> |     22.3     |    34.2     |     —      |         41.1         |       45.7       |
+|          SEED-Image          |     30.6     |    64.6     |    68.2    |         69.8         |       67.4       |
+|           OCRBench           |      7       |     398     |    687     |         654          |       767        |
+|       Hallusion-Bench        |     17.1     |    26.4     |     —      |         37.5         |       34.8       |
+|    CCBench<sub>dev</sub>     |     3.5      |    16.3     |     —      |         63.5         |       66.3       |
+|   Avg<sub>multimodal</sub>   |     16.1     |    38.9     |     —      |         54.4         |       55.2       |
+|                              |              |             |            |                      |                  |
+|    TextVQA<sub>val</sub>     |     4.8      |    56.8     |    64.7    |         70.5         |       72.6       |
+|     SQA-I<sub>test</sub>     |     47.2     |    64.9     |    89.2    |         84.9         |       93.6       |
+|      GQA<sub>test</sub>      |      —       |    62.6     |    60.3    |         61.6         |       59.5       |
+|    DocVQA<sub>test</sub>     |     1.5      |    53.0     |    76.3    |         85.0         |       80.0       |
+|     AI2D<sub>test</sub>      |     46.0     |    61.0     |    70.0    |         69.8         |       68.6       |
+|    ChartQA<sub>test</sub>    |     2.9      |    59.1     |    68.6     |         74.8         |       73.7       |
+|    InfoVQA<sub>test</sub>    |     5.0      |    25.0     |    43.8    |         55.4         |       43.0       |
+|      Avg<sub>VQA</sub>       |     17.9     |    54.6     |    67.6    |         71.7         |       70.1       |
+
+> * Sources of the results include the original papers, our evaluation with [VLMEvalKit](https://github.com/open-compass/VLMEvalKit), and [OpenCompass](https://rank.opencompass.org.cn/leaderboard-multimodal/?m=REALTIME).
+> * Average scores are computed by normalizing each metric to a range between 0 and 100.
+> * Please note that evaluating the same model using different testing toolkits can result in slight differences, which is normal. Updates to code versions and variations in environment and hardware can also cause minor discrepancies in results.
+
+## Inference
+
+We provide an example code to run Mono-InternVL-2B inference using `transformers`.
+
+> Please use transformers==4.37.2 to ensure the model works normally.
+
+<details>
+<summary>Inference with Transformers (click to expand)</summary>
+
+```python
+import numpy as np
+import torch
+import torchvision.transforms as T
+from decord import VideoReader, cpu
+from PIL import Image
+from torchvision.transforms.functional import InterpolationMode
+from transformers import AutoModel, AutoTokenizer
+
+IMAGENET_MEAN = (0.485, 0.456, 0.406)
+IMAGENET_STD = (0.229, 0.224, 0.225)
+
+def build_transform(input_size):
+    MEAN, STD = IMAGENET_MEAN, IMAGENET_STD
+    transform = T.Compose([
+        T.Lambda(lambda img: img.convert('RGB') if img.mode != 'RGB' else img),
+        T.Resize((input_size, input_size), interpolation=InterpolationMode.BICUBIC),
+        T.ToTensor(),
+        T.Normalize(mean=MEAN, std=STD)
+    ])
+    return transform
+
+def find_closest_aspect_ratio(aspect_ratio, target_ratios, width, height, image_size):
+    best_ratio_diff = float('inf')
+    best_ratio = (1, 1)
+    area = width * height
+    for ratio in target_ratios:
+        target_aspect_ratio = ratio[0] / ratio[1]
+        ratio_diff = abs(aspect_ratio - target_aspect_ratio)
+        if ratio_diff < best_ratio_diff:
+            best_ratio_diff = ratio_diff
+            best_ratio = ratio
+        elif ratio_diff == best_ratio_diff:
+            if area > 0.5 * image_size * image_size * ratio[0] * ratio[1]:
+                best_ratio = ratio
+    return best_ratio
+
+def dynamic_preprocess(image, min_num=1, max_num=12, image_size=448, use_thumbnail=False):
+    orig_width, orig_height = image.size
+    aspect_ratio = orig_width / orig_height
+
+    # calculate the existing image aspect ratio
+    target_ratios = set(
+        (i, j) for n in range(min_num, max_num + 1) for i in range(1, n + 1) for j in range(1, n + 1) if
+        i * j <= max_num and i * j >= min_num)
+    target_ratios = sorted(target_ratios, key=lambda x: x[0] * x[1])
+
+    # find the closest aspect ratio to the target
+    target_aspect_ratio = find_closest_aspect_ratio(
+        aspect_ratio, target_ratios, orig_width, orig_height, image_size)
+
+    # calculate the target width and height
+    target_width = image_size * target_aspect_ratio[0]
+    target_height = image_size * target_aspect_ratio[1]
+    blocks = target_aspect_ratio[0] * target_aspect_ratio[1]
+
+    # resize the image
+    resized_img = image.resize((target_width, target_height))
+    processed_images = []
+    for i in range(blocks):
+        box = (
+            (i % (target_width // image_size)) * image_size,
+            (i // (target_width // image_size)) * image_size,
+            ((i % (target_width // image_size)) + 1) * image_size,
+            ((i // (target_width // image_size)) + 1) * image_size
+        )
+        # split the image
+        split_img = resized_img.crop(box)
+        processed_images.append(split_img)
+    assert len(processed_images) == blocks
+    if use_thumbnail and len(processed_images) != 1:
+        thumbnail_img = image.resize((image_size, image_size))
+        processed_images.append(thumbnail_img)
+    return processed_images
+
+def load_image(image_file, input_size=448, max_num=12):
+    image = Image.open(image_file).convert('RGB')
+    transform = build_transform(input_size=input_size)
+    images = dynamic_preprocess(image, image_size=input_size, use_thumbnail=True, max_num=max_num)
+    pixel_values = [transform(image) for image in images]
+    pixel_values = torch.stack(pixel_values)
+    return pixel_values
+
+
+path = 'OpenGVLab/Mono-InternVL-2B'
+model = AutoModel.from_pretrained(
+    path,
+    torch_dtype=torch.bfloat16,
+    low_cpu_mem_usage=True,
+    trust_remote_code=True).eval().cuda()
+tokenizer = AutoTokenizer.from_pretrained(path, trust_remote_code=True, use_fast=False)
+
+# set the max number of tiles in `max_num`
+pixel_values = load_image('./examples/image1.jpg', max_num=12).to(torch.bfloat16).cuda()
+generation_config = dict(max_new_tokens=1024, do_sample=True)
+
+# pure-text conversation
+question = 'Hello, who are you?'
+response, history = model.chat(tokenizer, None, question, generation_config, history=None, return_history=True)
+print(f'User: {question}
+Assistant: {response}')
+
+question = 'Can you tell me a story?'
+response, history = model.chat(tokenizer, None, question, generation_config, history=history, return_history=True)
+print(f'User: {question}
+Assistant: {response}')
+
+# single-image single-round conversation
+question = '<image>
+Please describe the image shortly.'
+response = model.chat(tokenizer, pixel_values, question, generation_config)
+print(f'User: {question}
+Assistant: {response}')
+
+# single-image multi-round conversation
+question = '<image>
+Please describe the image in detail.'
+response, history = model.chat(tokenizer, pixel_values, question, generation_config, history=None, return_history=True)
+print(f'User: {question}
+Assistant: {response}')
+
+question = 'Please write a poem according to the image.'
+response, history = model.chat(tokenizer, pixel_values, question, generation_config, history=history, return_history=True)
+print(f'User: {question}
+Assistant: {response}')
+```
+
+</details>
+
+<details>
+<summary>Inference with LMDeploy (click to expand)</summary>
+
+Please install lmdeploy>=0.6.3 for Mono-InternVL support.
+
+```python
+from lmdeploy import pipeline
+from lmdeploy.vl import load_image
+
+image = load_image('./examples/image1.jpg')
+pipe = pipeline('OpenGVLab/Mono-InternVL-2B')
+response = pipe(('Please describe the image shortly.', image))
+print(response.text)
+```
+</details>
+
+## Supervised Finetuning
+
+Currently we provide the supervised finetuning (S2 instruction tuning) code on the LLaVA-v1.5-mix665k dataset. For details on the dataset, please refer to [LLaVA-v1.5](https://github.com/haotian-liu/LLaVA).
 
+<details>
+<summary>Installation (click to expand)</summary>
 
+- Clone this repository:
+
+  ```bash
+  git clone https://github.com/OpenGVLab/Mono-InternVL.git
+  ```
+  
+- Create a conda virtual environment and activate it:
+
+  ```bash
+  conda create -n monointernvl python=3.9 -y
+  conda activate monointernvl
+  ```
+
+- Install dependencies using `requirements.txt`:
+
+  ```bash
+  pip install -r requirements.txt
+  ```
+
+- Additional: Install `flash-attn==2.5.6`:
+
+  ```bash
+  pip install flash-attn==2.5.6 --no-build-isolation
+  ```
+
+  Alternatively you can compile from source:
+
+  ```bash
+  git clone https://github.com/Dao-AILab/flash-attention.git
+  cd flash-attention
+  git checkout v2.5.6
+  python setup.py install
+  ```
+</details>
+
+<details>
+<summary>Dataset Preparation (click to expand)</summary>
+
+#### LLaVA-v1.5-mix665k Dataset
+
+1. Download the instruction tuning data:
+```sh
+mkdir playground
+wget https://huggingface.co/datasets/liuhaotian/LLaVA-Instruct-150K/resolve/main/llava_v1_5_mix665k.json -P playground/
+```
+
+2. Download image datasets:
+
+- COCO: [train2017](http://images.cocodataset.org/zips/train2017.zip)
+- GQA: [images](https://downloads.cs.stanford.edu/nlp/data/gqa/images.zip)
+- OCR-VQA: [download script](https://drive.google.com/drive/folders/1_GYPY5UkUy7HIcR0zq3ZCFgeZN7BAfm_?usp=sharing)
+- TextVQA: [train_val_images](https://dl.fbaipublicfiles.com/textvqa/images/train_val_images.zip)
+- VisualGenome: [part1](https://cs.stanford.edu/people/rak248/VG_100K_2/images.zip), [part2](https://cs.stanford.edu/people/rak248/VG_100K_2/images2.zip)
+  
+3. Organize data as follows:
+
+```none
+playground/
+├── data/
+│   ├── coco/train2017/
+│   ├── gqa/images/
+│   ├── ocr_vqa/images/
+│   ├── textvqa/train_images/
+│   └── vg/
+│       ├── VG_100K/
+│       └── VG_100K_2/
+└── llava_v1_5_mix665k.json
+```
+
+#### Custom Dataset
+
+For custom dataset, format your data in to a JSONL file, where each entry is a dictionary organized in the following format (similar to `llava_v1_5_mix665k.json`):
+
+```python
+{
+    "id": "000000120375",
+    "image": "coco/train2017/000000120375.jpg",
+    "conversations": [
+        {
+            "from": "human",
+            "value": "<image>
+What type of vehicle is driving down the street in the image?"
+        },
+        {
+            "from": "gpt",
+            "value": "A red sports utility vehicle (SUV) is driving down the street in the image."
+        },
+        {
+            "from": "human",
+            "value": "Is the street crowded with people?"
+        },
+        {
+            "from": "gpt",
+            "value": "Yes, the street is filled with a considerable number of people, which indicates that the area is busy."
+        }
+        # (more turns ...)
+    ]
+}
+```
+
+Then modify the metadata file `shell/data_llava_finetune.json`:
+
+```python
+{
+  "name of your dataset": {
+    "root": "playground/data/", # combination of "root" and "image" in the JSONL gives the complete image path
+    "annotation": "path to your JSONL",
+    "data_augment": false,
+    "repeat_time": 1,
+    "length": 12345 # change to the actual number of samples in your dataset
+  }
+}
+```
+
+</details>
+
+<details>
+<summary>Model Preparation (click to expand)</summary>
+
+We provide pretrained models of different stages (S1.1 concept learning, S1.2 semantic learning, S1.3 alignment learning).
+Choose from the following models and download the weights to `workdirs/` folder.
+
+
+| model name              | download                                                               |  size  |
+| ----------------------- | ---------------------------------------------------------------------- |:------:|
+| Mono-InternVL-2B-S1-1   | 🤗 [HF link](https://huggingface.co/OpenGVLab/Mono-InternVL-2B-S1-1) | 6.2 GB |
+| Mono-InternVL-2B-S1-2   | 🤗 [HF link](https://huggingface.co/OpenGVLab/Mono-InternVL-2B-S1-2) | 6.2 GB |
+| Mono-InternVL-2B-S1-3   | 🤗 [HF link](https://huggingface.co/OpenGVLab/Mono-InternVL-2B-S1-3) | 6.2 GB |
+
+
+```sh
+mkdir workdirs
+cd workdirs/
+# pip install -U huggingface_hub
+huggingface-cli download --resume-download --local-dir-use-symlinks False OpenGVLab/Mono-InternVL-2B-S1-1 --local-dir Mono-InternVL-2B-S1-1
+```
+
+The directory structure is:
+
+```sh
+workdirs/
+├── Mono-InternVL-2B-S1-1/
+├── Mono-InternVL-2B-S1-2/
+└── Mono-InternVL-2B-S1-3/
+```
+</details>
+
+<details>
+<summary>Training (click to expand)</summary>
+
+Finetuning takes around 12 hours on 8x A100 (80G) GPUs.
+
+#### Single Node Multi-GPU
+```sh
+MODEL="./workdirs/Mono-InternVL-2B-S1-3" OUTPUT_DIR="./workdirs/mono_internvl_llava_sft" sh shell/mono_internvl_finetune_llava_torchrun.sh
+```
+
+#### Slurm Cluster
+```sh
+PARTITION="your partition" MODEL="./workdirs/Mono-InternVL-2B-S1-3" OUTPUT_DIR="./workdirs/mono_internvl_llava_sft" sh shell/mono_internvl_finetune_llava_slurm.sh
+```
+
+</details>
+
+## License
+
+This project is released under the [MIT License](LICENSE).
 
 ## Citation
 
-If you find this project useful in your research, please consider citing:
+If you find this work helpful in your research, please consider giving this repo a star ⭐ and citing our paper:
 
-```BibTeX
-@article{luo2024mono,
+```bibtex
+@article{mono_internvl_v1,
   title={Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-training},
   author={Luo, Gen and Yang, Xue and Dou, Wenhan and Wang, Zhaokai and Liu, Jiawen and Dai, Jifeng and Qiao, Yu and Zhu, Xizhou},
   journal={arXiv preprint arXiv:2410.08202},
   year={2024}
 }
-```
 
+@article{mono_internvl_v1.5,
+  title={Mono-InternVL-1.5: Towards Cheaper and Faster Monolithic Multimodal Large Language Models},
+  author={Luo, Gen and Dou, Wenhan and Li, Wenhao and Wang, Zhaokai and Yang, Xue and Tian, Changyao and Li, Hao and Wang, Weiyun and Wang, Wenhai and Zhu, Xizhou and Qiao, Yu and Dai, Jifeng},
+  journal={arXiv preprint arXiv:2507.12566},
+  year={2025}
+}
+```