vine_hf/OVERVIEW.md

# VINE HuggingFace Interface - Complete Overview

This directory contains a complete HuggingFace-compatible interface for the VINE (Video Understanding with Natural Language) model. The interface allows you to easily use, share, and deploy your VINE model through the HuggingFace ecosystem.

## 📁 Directory Structure

```
vine_hf/
├── __init__.py                 # Package initialization and exports
├── vine_config.py              # VineConfig class (PretrainedConfig)
├── vine_model.py               # VineModel class (PreTrainedModel)  
├── vine_pipeline.py            # VinePipeline class (Pipeline)
├── example_usage.py            # Comprehensive usage examples
├── convert_inference.py        # Migration guide from inference.py
├── push_to_hub.py             # Script to push model to HF Hub
├── setup.py                   # Package setup configuration
├── README.md                  # Detailed documentation
└── OVERVIEW.md                # This file
```

## 🏗️ Architecture Components

### 1. VineConfig (`vine_config.py`)
- Inherits from `PretrainedConfig`
- Configures model parameters, segmentation methods, and processing options
- Compatible with HuggingFace configuration system

### 2. VineModel (`vine_model.py`)  
- Inherits from `PreTrainedModel`
- Implements the core VINE model with three CLIP backbones
- Supports categorical, unary, and binary predictions
- Provides both `forward()` and `predict()` methods

### 3. VinePipeline (`vine_pipeline.py`)
- Inherits from `Pipeline`
- Handles end-to-end video processing workflow
- Integrates segmentation (SAM2, Grounding DINO + SAM2)
- Provides user-friendly interface for video understanding

## 🚀 Key Features

✅ **Full HuggingFace Compatibility**
- Compatible with `transformers` library
- Supports `AutoModel` and `pipeline` interfaces
- Can be pushed to and loaded from HuggingFace Hub

✅ **Flexible Segmentation**
- Support for SAM2 automatic segmentation
- Support for Grounding DINO + SAM2 text-guided segmentation
- Configurable thresholds and parameters

✅ **Multi-Modal Understanding**
- Categorical classification (object types)
- Unary predicates (single object actions)
- Binary relations (object-object relationships)

✅ **Easy Integration**
- Simple pipeline interface for end users
- Direct model access for researchers
- Comprehensive configuration options

## 📖 Usage Examples

### Quick Start with Pipeline
```python
from transformers import pipeline
from vine_hf import VineModel, VinePipeline

# Create pipeline
vine_pipeline = pipeline(
    "vine-video-understanding",
    model="your-username/vine-model",
    trust_remote_code=True
)

# Process video
results = vine_pipeline(
    "video.mp4",
    categorical_keywords=['human', 'dog', 'frisbee'],
    unary_keywords=['running', 'jumping'],
    binary_keywords=['chasing', 'behind']
)
```

### Direct Model Usage
```python
from vine_hf import VineConfig, VineModel

config = VineConfig(segmentation_method="grounding_dino_sam2")
model = VineModel(config)

results = model.predict(
    video_frames=video_tensor,
    masks=masks_dict,
    bboxes=bboxes_dict,
    categorical_keywords=['human', 'dog'],
    unary_keywords=['running', 'sitting'],
    binary_keywords=['chasing', 'near']
)
```

## 🔧 Migration from Original Code

The `convert_inference.py` script shows how to migrate from the original `inference.py` workflow:

**Original Approach:**
- Manual model loading and configuration
- Direct handling of segmentation pipeline  
- Custom result processing
- Complex setup requirements

**New HuggingFace Interface:**
- Standardized model configuration
- Automatic preprocessing/postprocessing
- Simple pipeline interface
- Easy sharing via HuggingFace Hub

## 📤 Sharing Your Model

Use the `push_to_hub.py` script to share your trained model:

```bash
python vine_hf/push_to_hub.py \
    --weights path/to/your/model.pth \
    --repo your-username/vine-model \
    --login
```

## 🛠️ Installation & Setup

1. **Install Dependencies:**
```bash
pip install transformers torch torchvision opencv-python pillow numpy
```

2. **Install Segmentation Models (Optional):**
   - SAM2: https://github.com/facebookresearch/sam2
   - Grounding DINO: https://github.com/IDEA-Research/GroundingDINO

3. **Install VINE HF Interface:**
```bash
cd vine_hf
pip install -e .
```

## 🎯 Configuration Options

The `VineConfig` class supports extensive configuration:

- **Model Settings:** CLIP backbone, hidden dimensions
- **Segmentation:** Method, thresholds, target FPS  
- **Processing:** Alpha values, top-k results, video length limits
- **Performance:** Multi-class mode, output format options

## 📊 Output Format

The interface returns structured predictions:

```python
{
    "categorical_predictions": {obj_id: [(prob, category), ...]},
    "unary_predictions": {(frame, obj): [(prob, action), ...]},
    "binary_predictions": {(frame, pair): [(prob, relation), ...]},
    "confidence_scores": {"categorical": float, "unary": float, "binary": float},
    "summary": {
        "num_objects_detected": int,
        "top_categories": [(category, prob), ...],
        "top_actions": [(action, prob), ...],
        "top_relations": [(relation, prob), ...]
    }
}
```

## 🔍 Testing & Validation

Run the example scripts to test your setup:

```bash
# Test basic functionality
python vine_hf/example_usage.py

# Test migration from original code  
python vine_hf/convert_inference.py
```

## 🤝 Contributing

To contribute or customize:

1. **Modify Configuration:** Edit `vine_config.py` for new parameters
2. **Extend Model:** Add functionality to `vine_model.py`
3. **Enhance Pipeline:** Improve preprocessing/postprocessing in `vine_pipeline.py`
4. **Add Features:** Create additional utility scripts

## 📝 Next Steps

1. **Load Your Weights:** Use your trained VINE model weights
2. **Test Segmentation:** Set up Grounding DINO and SAM2 models
3. **Validate Results:** Compare with original inference.py output
4. **Share Model:** Push to HuggingFace Hub for community use
5. **Deploy:** Use in applications, demos, or research projects

## 🐛 Troubleshooting

**Common Issues:**
- **Import Errors:** Check PYTHONPATH and package installation
- **Segmentation Failures:** Verify Grounding DINO/SAM2 setup
- **Weight Loading:** Adjust weight loading logic in `convert_inference.py`
- **CUDA Issues:** Check GPU availability and PyTorch installation

**Support:**
- Check the README.md for detailed documentation
- Review example_usage.py for working code examples
- Examine convert_inference.py for migration guidance

---

This HuggingFace interface makes VINE accessible to the broader ML community while maintaining all the powerful video understanding capabilities of the original model. The standardized interface enables easy sharing, deployment, and integration with existing HuggingFace workflows.