Update README.md

README.md

@@ -7,31 +7,56 @@ tags:
 - lora
 - transformers
 model-index:
-- name: SmolVLM2-500M-Video-Instruct-vqav2
-  results: []
+- name: Susant-Achary/SmolVLM2-500M-Video-Instruct-VQA2
+  results:
+  - task:
+      type: visual-question-answering
+    dataset:
+        type: jinaai/table-vqa
+        name: jinaai/table-vqa
+    metrics:
+      - type: training_loss
+        value: 0.7473664236068726
+datasets:
+- jinaai/table-vqa
+language:
+- en
+pipeline_tag: visual-question-answering
 ---
 
-<!-- This model card has been generated automatically according to the information the Trainer had access to. You
-should probably proofread and complete it, then remove this comment. -->
 
 # SmolVLM2-500M-Video-Instruct-vqav2
 
-This model is a fine-tuned version of [HuggingFaceTB/SmolVLM2-500M-Video-Instruct](https://huggingface.co/HuggingFaceTB/SmolVLM2-500M-Video-Instruct) on an unknown dataset.
+This model is a fine-tuned version of [HuggingFaceTB/SmolVLM2-500M-Video-Instruct](https://huggingface.co/HuggingFaceTB/SmolVLM2-500M-Video-Instruct) on the [jinaai/table-vqa](https://huggingface.co/datasets/jinaai/table-vqa) dataset.
 
 ## Model description
 
-More information needed
+This model is a SmolVLM2-500M-Video-Instruct model fine-tuned for Visual Question Answering on table images using the jinaai/table-vqa dataset. It was fine-tuned using QLoRA for efficient training on consumer GPUs.
 
 ## Intended uses & limitations
 
-More information needed
+This model is intended for Visual Question Answering tasks specifically on images containing tables. It can be used to answer questions about the content of tables within images.
+
+Limitations:
+- Performance may vary on different types of images or questions outside of the table VQA domain.
+- The model was fine-tuned on a small subset of the dataset for demonstration purposes.
+- The model's performance is dependent on the quality and nature of the jinaai/table-vqa dataset.
 
 ## Training and evaluation data
 
-More information needed
+The model was trained on a subset of the [jinaai/table-vqa](https://huggingface.co/datasets/jinaai/table-vqa) dataset. The training dataset size is 800 examples, and the test dataset size is 200 examples.
 
 ## Training procedure
 
+The model was fine-tuned using the QLoRA method with the following configuration:
+- `r=8`
+- `lora_alpha=8`
+- `lora_dropout=0.1`
+- `target_modules=['down_proj','o_proj','k_proj','q_proj','gate_proj','up_proj','v_proj']`
+- `use_dora=False`
+- `init_lora_weights="gaussian"`
+- 4-bit quantization (`bnb_4bit_use_double_quant=True`, `bnb_4bit_quant_type="nf4"`, `bnb_4bit_compute_dtype=torch.bfloat16`)
+
 ### Training hyperparameters
 
 The following hyperparameters were used during training:
@@ -44,8 +69,73 @@ The following hyperparameters were used during training:
 - lr_scheduler_warmup_steps: 50
 - num_epochs: 1
 
-### Training results
-
+### Direct Use
+```python
+import torch
+from peft import PeftModel, PeftConfig
+from transformers import AutoProcessor, Idefics3ForConditionalGeneration, BitsAndBytesConfig
+from PIL import Image
+import requests
+
+# Define the base model and the fine-tuned adapter repository
+base_model_id = "HuggingFaceTB/SmolVLM2-500M-Video-Instruct"
+adapter_model_id = "Susant-Achary/SmolVLM2-500M-Video-Instruct-vqav2" 
+
+# Load the processor from the base model
+processor = AutoProcessor.from_pretrained(base_model_id)
+
+# Load the base model with quantization 
+bnb_config = BitsAndBytesConfig(
+    load_in_4bit=True,
+    bnb_4bit_use_double_quant=True,
+    bnb_4bit_quant_type="nf4",
+    bnb_4bit_compute_dtype=torch.bfloat16
+)
+
+model = Idefics3ForConditionalGeneration.from_pretrained(
+    base_model_id,
+    quantization_config=bnb_config,
+    _attn_implementation="flash_attention_2",
+    device_map="auto"
+)
+
+# Load the adapter and add it to the base model
+model = PeftModel.from_pretrained(model, adapter_model_id)
+
+# Prepare an example image and question
+# You can replace this with your own image and question
+url = "https://www.gstatic.com/webp/gallery/2.jpg"
+image = Image.open(requests.get(url, stream=True).raw)
+question = "What is in the image?"
+
+# Prepare the input for the model
+messages = [
+    {
+        "role": "user",
+        "content": [
+            {"type": "text", "text": "Answer briefly."},
+            {"type": "image"},
+            {"type": "text", "text": question}
+        ]
+    },
+    {
+        "role": "assistant",
+        "content": [
+            {"type": "text", "text": None}
+        ]
+    }
+]
+
+prompt = processor.apply_chat_template(messages, add_generation_prompt=False)
+inputs = processor(text=[prompt], images=[image], return_tensors="pt").to(model.device) # Move inputs to model device
+
+# Generate a response
+generated_ids = model.generate(**inputs, max_new_tokens=100)
+generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
+
+# Print the generated response
+print(generated_text)
+```
 
 
 ### Framework versions