File size: 2,024 Bytes
5d75a67 7a6ae87 5d75a67 7a6ae87 5d75a67 7a6ae87 5d75a67 7a6ae87 5d75a67 7a6ae87 5d75a67 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 |
---
pipeline_tag: image-to-video
library_name: diffusers
license: mit
---
# VIRES model card
**Model Page**: [VIRES](https://hjzheng.net/projects/VIRES/)
## Model Information
Summary description and brief definition of inputs and outputs.
### Description
VIRES is a video instance repainting method with sketch and text guidance, enabling video instance repainting, replacement, generation, and removal. It leverages the generative priors of text-to-video models to maintain temporal consistency and produce visually pleasing results. Key features include a Sequential ControlNet for structure layout extraction and detail capture, sketch attention for injecting fine-grained semantics, and a sketch-aware encoder for alignment.
### Inputs and outputs
- **Input:**
- Text string describing the desired changes.
- Mask Sequence (51 x 512 x 512 resolution).
- Sketch Sequence (51 x 512 x 512 resolution).
- **Output:**
- A repainted video.
### Usage
A basic example using the `diffusers` library (requires appropriate model weights and dependencies):
```python
from diffusers import DiffusionPipeline #Import necessary libraries
# Load the model (replace with your actual paths)
pipe = DiffusionPipeline.from_pretrained("suimu/VIRES", torch_dtype=torch.float16).to("cuda")
# Prepare inputs: text prompt, mask, and sketch
prompt = "A cat replaces the dog in this video"
mask = ... #Load your mask sequence
sketch = ... #Load your sketch sequence
# Generate the video
video = pipe(prompt, mask, sketch).videos[0]
# Save or display the video
...
```
For complete usage instructions and advanced options, refer to our GitHub page: https://github.com/suimuc/VIRES/
## Citation
```BibTeX
@article{vires,
title={VIRES: Video Instance Repainting via Sketch and Text Guided Generation},
author={Weng, Shuchen and Zheng, Haojie and Zhang, Peixuan and Hong, Yuchen and Jiang, Han and Li, Si and Shi, Boxin},
journal={arXiv preprint arXiv:2411.16199},
year={2024}
}
``` |