|
|
--- |
|
|
license: apache-2.0 |
|
|
pipeline_tag: any-to-any |
|
|
library_name: transformers |
|
|
base_model: |
|
|
- Qwen/Qwen2.5-Omni-7B |
|
|
base_model_relation: quantized |
|
|
language: |
|
|
- en |
|
|
- zh |
|
|
--- |
|
|
For more information (including how to compress models yourself), check out https://huggingface.co/DFloat11 and https://github.com/LeanModels/DFloat11 |
|
|
|
|
|
Feel free to request for other models for compression as well (for either the `diffusers` library, ComfyUI, or any other model), although compressing models that are of architectures that are unfamiliar to me might be more difficult. |
|
|
|
|
|
### How to Use |
|
|
|
|
|
#### `transformers` |
|
|
|
|
|
```python |
|
|
import soundfile as sf |
|
|
from transformers import Qwen2_5OmniForConditionalGeneration, Qwen2_5OmniProcessor |
|
|
from qwen_omni_utils import process_mm_info |
|
|
|
|
|
# Highly recommended to enable flash_attention_2 for better acceleration and memory saving. |
|
|
model = Qwen2_5OmniForConditionalGeneration.from_pretrained( |
|
|
"Qwen/Qwen2.5-Omni-7B", |
|
|
attn_implementation="flash_attention_2", |
|
|
dtype="auto", |
|
|
device_map="cpu" |
|
|
) |
|
|
|
|
|
DFloat11Model.from_pretrained("mingyi456/Qwen2.5-Omni-7B-DF11", device = "cpu", bfloat16_model = model) |
|
|
model.to("cuda") |
|
|
|
|
|
# IMPORTANT: If you want to disable the talker module, do it here, only after calling `model.to("cuda")` |
|
|
# model.disable_talker() |
|
|
|
|
|
processor = Qwen2_5OmniProcessor.from_pretrained("Qwen/Qwen2.5-Omni-7B") |
|
|
conversation = [ |
|
|
{ |
|
|
"role": "system", |
|
|
"content": [ |
|
|
{"type": "text", "text": "You are Qwen, a virtual human developed by the Qwen Team, Alibaba Group, capable of perceiving auditory and visual inputs, as well as generating text and speech."} |
|
|
], |
|
|
}, |
|
|
{ |
|
|
"role": "user", |
|
|
"content": [ |
|
|
{"type": "video", "video": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen2.5-Omni/draw.mp4"}, |
|
|
], |
|
|
}, |
|
|
] |
|
|
# set use audio in video |
|
|
USE_AUDIO_IN_VIDEO = True |
|
|
# Preparation for inference |
|
|
text = processor.apply_chat_template(conversation, add_generation_prompt=True, tokenize=False) |
|
|
audios, images, videos = process_mm_info(conversation, use_audio_in_video=USE_AUDIO_IN_VIDEO) |
|
|
inputs = processor(text=text, audio=audios, images=images, videos=videos, return_tensors="pt", padding=True, use_audio_in_video=USE_AUDIO_IN_VIDEO) |
|
|
inputs = inputs.to(model.device).to(model.dtype) |
|
|
# Inference: Generation of the output text and audio |
|
|
text_ids, audio = model.generate(**inputs, use_audio_in_video=USE_AUDIO_IN_VIDEO) |
|
|
text = processor.batch_decode(text_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False) |
|
|
print(text) |
|
|
sf.write( |
|
|
"output.wav", |
|
|
audio.reshape(-1).detach().cpu().numpy(), |
|
|
samplerate=24000, |
|
|
) |
|
|
``` |
