File size: 8,604 Bytes
9570a60 4025465 9570a60 ad0b3fc 9570a60 ec07908 9570a60 ec07908 9570a60 ec07908 9570a60 ec07908 9570a60 ec07908 b2e3f0c ec07908 9570a60 193cd55 9570a60 9e89ffa 9570a60 660054d 9570a60 9718063 9570a60 4025465 9570a60 4025465 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 |
---
license: apache-2.0
datasets:
- lmms-lab/LLaVA-One-Vision-1.5-Mid-Training-85M
- lmms-lab/LLaVA-OneVision-1.5-Insturct-Data
base_model:
- Qwen/Qwen3-8B-Base
- DeepGlint-AI/rice-vit-large-patch14-560
pipeline_tag: image-text-to-text
library_name: transformers
---
# LLaVA-OneVision-1.5: Fully Open-Source State-of-the-Art VLM Model
**LLaVA-OneVision1.5** introduces a novel family of **fully open-source** Large Multimodal Models (LMMs) that achieves **state-of-the-art performance** with substantially **lower cost** through training on **native resolution** images.
- **Superior Performance**
A family of fully open-source large multimodal models demonstrating
- Superior performance across multiple multimodal benchmarks
- outperforming **Qwen2.5-VL** in most evaluation tasks.
- **High-Quality Data at Scale**
Meticulously curated **pre-training and SFT data** with rigorous filtering and quality control, achieving **superior data efficiency** with only **64B tokens**.
- Concept-balanced, highly diverse, high-quality caption data
- Comprehensive instruction fine-tuning data covering a wide range of tasks
- **Ultra-Efficient Training Framework** Complete end-to-end training framework designed for maximum efficiency:
- $16000 total budget for full model training on A100 GPUs ($0.6 per GPU/Hour)
- 45% HFU efficiency in 8k context length
- Built on **MegatronLM** with support for **MoE**, **FP8**, and **long sequence parallelization**
- Optimized codebase for cost-effective scaling
- **Fully Open Framework** for community access and reproducibility:
- High-quality pre-training & SFT data
- Complete training framework & code
- Training recipes & configurations
- Comprehensive training logs & metrics
## Models
| Model | HF Link | Training Log |
|--------------------------|--------------------------------------------------------------------------------------------------------|-------------|
| LLaVA-OV-1.5-4B-Instruct | [🤗 HF / 4B-Instruct](https://huggingface.co/lmms-lab/LLaVA-OneVision-1.5-4B-Instruct) | Uploading… |
| LLaVA-OV-1.5-8B-Instruct | [🤗 HF / 8B-Instruct](https://huggingface.co/lmms-lab/LLaVA-OneVision-1.5-8B-Instruct) | [📈 Tensorboard](https://huggingface.co/lmms-lab/LLaVA-OneVision-1.5-8B-Instruct/tensorboard) |
## Datasets
<p align="left">
<strong>(a)</strong> The vocabulary coverage proportion in the LLaVA-OneVision-1.5 Mid-Training dataset before and after concept balancing.
<strong>(b)</strong> Distribution of data sources within the LLaVA-OneVision-1.5 Mid-Training dataset.
<strong>(c)</strong> Distribution of data sources within the LLaVA-OneVision-1.5 Insturct dataset.
</p>
| Description | Link | Status |
|--------------------|--------------------------------------------------------------------------------------------------------|-------------|
| OV-1.5-Mid-Training-85M | [🤗HF/85M](https://huggingface.co/datasets/lmms-lab/LLaVA-One-Vision-1.5-Mid-Training-85M) | Uploading… |
| OV-1.5-Instruct | [🤗HF/Inst](https://huggingface.co/datasets/lmms-lab/LLaVA-OneVision-1.5-Insturct-Data) | Uploading… |
## Code
This model is trained using a fully open-source, end-to-end training framework, with all code available at [EvolvingLMMs-Lab/LLaVA-OneVision-1.5](https://github.com/EvolvingLMMs-Lab/LLaVA-OneVision-1.5).
## Evaluation Results
All evaluations were conducted using [lmms_eval](https://github.com/EvolvingLMMs-Lab/lmms-eval).
| | **LLaVA-OV-1.5-8B** | **Qwen2.5 VL 7B** |
|:----------------------------------|:---------------:|:-------------:|
| MMMU (Validation) | **55.44** | 51.33 |
| MMMU-Pro (Standard) | **37.40** | 36.30 |
| MMMU-Pro (Vision) | 25.15 | **32.83** |
| MMBench (English; Test) | **84.14** | 83.40 |
| MMBench (Chinese; Test) | 81.00 | **81.61** |
| MME-RealWorld (English) | **62.31** | 57.33 |
| MME-RealWorld (Chinese) | **56.11** | 51.50 |
| AI2D (With Mask) | **84.16** | 82.58 |
| AI2D (Without Mask) | **94.11** | 93.36 |
| CV-Bench | **80.82** | 79.95 |
| VL-RewardBench | 45.90 | **49.65** |
| V* | **78.01** | 76.96 |
| PixmoCount | 62.19 | **63.33** |
| CountBench | **88.19** | 86.35 |
| ChartQA | **86.48** | 84.08 |
| CharXiv (Direct Questions) | **74.10** | 69.80 |
| DocVQA (Test) | **95.00** | 94.93 |
| InfoVQA (Test) | 78.42 | **81.67** |
| WeMath | **33.62** | 33.33 |
| MathVista (Mini) | **69.57** | 68.60 |
| MathVision | **25.56** | 22.37 |
| MMStar | **67.72** | 62.54 |
| SEED-Bench (Image) | 77.32 | **77.53** |
| ScienceQA | **94.98** | 88.75 |
| SEED-Bench 2-Plus | 69.21 | **70.93** |
| OCRBench | 82.90 | **84.20** |
| RealWorldQA | 68.10 | **68.50** |
### Using 🤗 Transformers to Chat
Here we show a code snippet to show you how to use the chat model with `transformers` and `qwen_vl_utils`:
```python
from transformers import AutoTokenizer, AutoProcessor, AutoModelForCausalLM
from qwen_vl_utils import process_vision_info
model_path = "lmms-lab/LLaVA-One-Vision-1.5-8B-Instruct"
# default: Load the model on the available device(s)
model = AutoModelForCausalLM.from_pretrained(
model_path, torch_dtype="auto", device_map="auto", trust_remote_code=True
)
# default processer
processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)
messages = [
{
"role": "user",
"content": [
{
"type": "image",
"image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg",
},
{"type": "text", "text": "Describe this image."},
],
}
]
# Preparation for inference
text = processor.apply_chat_template(
messages, tokenize=False, add_generation_prompt=True
)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
text=[text],
images=image_inputs,
videos=video_inputs,
padding=True,
return_tensors="pt",
)
inputs = inputs.to("cuda")
# Inference: Generation of the output
generated_ids = model.generate(**inputs, max_new_tokens=1024)
generated_ids_trimmed = [
out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text)
```
## Citation
If you find *LLaVA-OneVision-1.5* useful in your research, please consider to cite the following related papers:
```
@inproceedings{LLaVA-OneVision-1.5,
title={LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training},
author={LLaVA Community Contributors},
booktitle={arxiv},
year={2025}
}
@inproceedings{xie2025region,
title={Region-based Cluster Discrimination for Visual Representation Learning},
author={Xie, Yin and Yang, Kaicheng and An, Xiang and Wu, Kun and Zhao, Yongle and Deng, Weimo and Ran, Zimin and Wang, Yumeng and Feng, Ziyong and Miles, Roy and Elezi, Ismail and Deng, Jiankang},
booktitle={ICCV},
year={2025}
}
@article{lillava,
title={LLaVA-OneVision: Easy Visual Task Transfer},
author={Li, Bo and Zhang, Yuanhan and Guo, Dong and Zhang, Renrui and Li, Feng and Zhang, Hao and Zhang, Kaichen and Zhang, Peiyuan and Li, Yanwei and Liu, Ziwei and Li, Chunyuan},
journal={Transactions on Machine Learning Research}
year={2024}
}
``` |