Spaces:
Running
Running
A Le Thanh Son
commited on
Commit
·
e02c9de
1
Parent(s):
a6403d5
fix
Browse files- README.md +39 -39
- app.py +195 -139
- generator.py +10 -10
- test_model.py +22 -22
README.md
CHANGED
|
@@ -11,35 +11,35 @@ pinned: false
|
|
| 11 |
|
| 12 |
# CSM-1B Text-to-Speech Demo
|
| 13 |
|
| 14 |
-
|
| 15 |
|
| 16 |
-
##
|
| 17 |
|
| 18 |
-
- **
|
| 19 |
-
- **
|
| 20 |
-
- **
|
| 21 |
|
| 22 |
-
##
|
| 23 |
|
| 24 |
-
###
|
| 25 |
|
| 26 |
-
|
| 27 |
|
| 28 |
- [meta-llama/Llama-3.2-1B](https://huggingface.co/meta-llama/Llama-3.2-1B)
|
| 29 |
- [sesame/csm-1b](https://huggingface.co/sesame/csm-1b)
|
| 30 |
|
| 31 |
-
###
|
| 32 |
|
| 33 |
-
1.
|
| 34 |
-
2.
|
| 35 |
-
3.
|
| 36 |
-
4.
|
| 37 |
```bash
|
| 38 |
export HF_TOKEN=your_token_here
|
| 39 |
```
|
| 40 |
-
5.
|
| 41 |
|
| 42 |
-
###
|
| 43 |
|
| 44 |
```bash
|
| 45 |
git clone https://github.com/yourusername/csm-1b-gradio.git
|
|
@@ -47,54 +47,54 @@ cd csm-1b-gradio
|
|
| 47 |
pip install -r requirements.txt
|
| 48 |
```
|
| 49 |
|
| 50 |
-
##
|
| 51 |
|
| 52 |
-
1.
|
| 53 |
```bash
|
| 54 |
python app.py
|
| 55 |
```
|
| 56 |
-
2.
|
| 57 |
-
3.
|
| 58 |
-
4.
|
| 59 |
-
5.
|
| 60 |
-
6.
|
| 61 |
|
| 62 |
-
##
|
| 63 |
|
| 64 |
-
CSM-1B
|
| 65 |
|
| 66 |
## ZeroGPU
|
| 67 |
|
| 68 |
-
|
| 69 |
|
| 70 |
```python
|
| 71 |
import spaces
|
| 72 |
|
| 73 |
@spaces.GPU
|
| 74 |
def my_gpu_function():
|
| 75 |
-
#
|
| 76 |
-
#
|
| 77 |
pass
|
| 78 |
```
|
| 79 |
|
| 80 |
-
|
| 81 |
|
| 82 |
-
##
|
| 83 |
|
| 84 |
-
-
|
| 85 |
-
-
|
| 86 |
-
-
|
| 87 |
|
| 88 |
-
##
|
| 89 |
|
| 90 |
-
|
| 91 |
|
| 92 |
-
1.
|
| 93 |
-
2.
|
| 94 |
-
3.
|
| 95 |
-
4.
|
| 96 |
|
| 97 |
-
##
|
| 98 |
|
| 99 |
- [GitHub Repository](https://github.com/SesameAILabs/csm-1b)
|
| 100 |
- [Hugging Face Model](https://huggingface.co/sesame/csm-1b)
|
|
|
|
| 11 |
|
| 12 |
# CSM-1B Text-to-Speech Demo
|
| 13 |
|
| 14 |
+
This application uses the CSM-1B (Collaborative Speech Model) to convert text to high-quality speech.
|
| 15 |
|
| 16 |
+
## Features
|
| 17 |
|
| 18 |
+
- **Simple Audio Generation**: Convert text to speech with options for speaker ID, duration, temperature, and top-k.
|
| 19 |
+
- **Audio Generation with Context**: Provide audio clips and text as context to help the model generate more appropriate speech.
|
| 20 |
+
- **GPU Optimization**: Uses Hugging Face Spaces' ZeroGPU to optimize GPU usage.
|
| 21 |
|
| 22 |
+
## Installation and Configuration
|
| 23 |
|
| 24 |
+
### Access Requirements
|
| 25 |
|
| 26 |
+
To use the CSM-1B model, you need access to the following models on Hugging Face:
|
| 27 |
|
| 28 |
- [meta-llama/Llama-3.2-1B](https://huggingface.co/meta-llama/Llama-3.2-1B)
|
| 29 |
- [sesame/csm-1b](https://huggingface.co/sesame/csm-1b)
|
| 30 |
|
| 31 |
+
### Hugging Face Token Configuration
|
| 32 |
|
| 33 |
+
1. Create a Hugging Face account if you don't have one.
|
| 34 |
+
2. Go to [Hugging Face Settings](https://huggingface.co/settings/tokens) to create a token.
|
| 35 |
+
3. Request access to the models if needed.
|
| 36 |
+
4. Set the `HF_TOKEN` environment variable with your token:
|
| 37 |
```bash
|
| 38 |
export HF_TOKEN=your_token_here
|
| 39 |
```
|
| 40 |
+
5. Or you can enter your token directly in the "Configuration" tab of the application.
|
| 41 |
|
| 42 |
+
### Installation
|
| 43 |
|
| 44 |
```bash
|
| 45 |
git clone https://github.com/yourusername/csm-1b-gradio.git
|
|
|
|
| 47 |
pip install -r requirements.txt
|
| 48 |
```
|
| 49 |
|
| 50 |
+
## How to Use
|
| 51 |
|
| 52 |
+
1. Start the application:
|
| 53 |
```bash
|
| 54 |
python app.py
|
| 55 |
```
|
| 56 |
+
2. Open a web browser and go to the displayed address (usually http://127.0.0.1:7860).
|
| 57 |
+
3. Enter the text you want to convert to speech.
|
| 58 |
+
4. Choose a speaker ID (from 0-10).
|
| 59 |
+
5. Adjust parameters like maximum duration, temperature, and top-k.
|
| 60 |
+
6. Click the "Generate Audio" button to create speech.
|
| 61 |
|
| 62 |
+
## About the Model
|
| 63 |
|
| 64 |
+
CSM-1B is an advanced text-to-speech model developed by Sesame AI Labs. This model can generate natural speech from text with various voices.
|
| 65 |
|
| 66 |
## ZeroGPU
|
| 67 |
|
| 68 |
+
This application uses Hugging Face Spaces' ZeroGPU to optimize GPU usage. ZeroGPU helps free up GPU memory when not in use, saving resources and improving performance.
|
| 69 |
|
| 70 |
```python
|
| 71 |
import spaces
|
| 72 |
|
| 73 |
@spaces.GPU
|
| 74 |
def my_gpu_function():
|
| 75 |
+
# This function will only use GPU when called
|
| 76 |
+
# and release GPU after completion
|
| 77 |
pass
|
| 78 |
```
|
| 79 |
|
| 80 |
+
When deployed on Hugging Face Spaces, ZeroGPU will automatically manage GPU usage, making the application more efficient.
|
| 81 |
|
| 82 |
+
## Notes
|
| 83 |
|
| 84 |
+
- This model uses watermarking to mark audio generated by AI.
|
| 85 |
+
- Audio generation time depends on text length and hardware configuration.
|
| 86 |
+
- You need access to the CSM-1B model on Hugging Face to use this application.
|
| 87 |
|
| 88 |
+
## Deployment on Hugging Face Spaces
|
| 89 |
|
| 90 |
+
To deploy this application on Hugging Face Spaces:
|
| 91 |
|
| 92 |
+
1. Create a new Space on Hugging Face with Gradio SDK.
|
| 93 |
+
2. Upload all project files.
|
| 94 |
+
3. In the Space settings, add the `HF_TOKEN` environment variable with your token.
|
| 95 |
+
4. Choose appropriate hardware configuration (GPU recommended).
|
| 96 |
|
| 97 |
+
## Resources
|
| 98 |
|
| 99 |
- [GitHub Repository](https://github.com/SesameAILabs/csm-1b)
|
| 100 |
- [Hugging Face Model](https://huggingface.co/sesame/csm-1b)
|
app.py
CHANGED
|
@@ -11,64 +11,82 @@ from dataclasses import dataclass
|
|
| 11 |
from generator import Segment, load_csm_1b
|
| 12 |
from huggingface_hub import login
|
| 13 |
|
| 14 |
-
#
|
| 15 |
torch._dynamo.config.suppress_errors = True
|
| 16 |
|
| 17 |
-
#
|
| 18 |
device = "cuda" if torch.cuda.is_available() else "cpu"
|
| 19 |
-
print(f"
|
| 20 |
|
| 21 |
-
#
|
| 22 |
def login_huggingface():
|
| 23 |
hf_token = os.environ.get("HF_TOKEN")
|
| 24 |
if hf_token:
|
| 25 |
-
print("
|
| 26 |
login(token=hf_token)
|
| 27 |
-
print("
|
| 28 |
else:
|
| 29 |
-
print("
|
| 30 |
|
| 31 |
-
#
|
| 32 |
login_huggingface()
|
| 33 |
|
| 34 |
-
#
|
| 35 |
generator = None
|
| 36 |
model_loaded = False
|
| 37 |
|
| 38 |
-
#
|
| 39 |
-
@spaces.GPU
|
| 40 |
def initialize_model():
|
| 41 |
global generator, model_loaded
|
| 42 |
if not model_loaded:
|
| 43 |
-
print("
|
| 44 |
generator = load_csm_1b(device="cuda")
|
| 45 |
model_loaded = True
|
| 46 |
-
print("
|
| 47 |
return generator
|
| 48 |
|
| 49 |
-
#
|
| 50 |
-
@spaces.GPU
|
| 51 |
def get_model():
|
| 52 |
global generator, model_loaded
|
| 53 |
if not model_loaded:
|
| 54 |
return initialize_model()
|
| 55 |
return generator
|
| 56 |
|
| 57 |
-
#
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 58 |
def audio_to_tensor(audio_path: str) -> Tuple[torch.Tensor, int]:
|
| 59 |
waveform, sample_rate = torchaudio.load(audio_path)
|
| 60 |
-
waveform = waveform.mean(dim=0) #
|
| 61 |
return waveform, sample_rate
|
| 62 |
|
| 63 |
-
#
|
| 64 |
def save_audio(audio_tensor: torch.Tensor, sample_rate: int) -> str:
|
| 65 |
temp_dir = tempfile.gettempdir()
|
| 66 |
output_path = os.path.join(temp_dir, f"csm1b_output_{int(time.time())}.wav")
|
| 67 |
torchaudio.save(output_path, audio_tensor.unsqueeze(0), sample_rate)
|
| 68 |
return output_path
|
| 69 |
|
| 70 |
-
#
|
| 71 |
-
@spaces.GPU
|
| 72 |
def generate_speech(
|
| 73 |
text: str,
|
| 74 |
speaker_id: int,
|
|
@@ -83,49 +101,62 @@ def generate_speech(
|
|
| 83 |
top_k: int = 50,
|
| 84 |
progress=gr.Progress()
|
| 85 |
) -> str:
|
| 86 |
-
|
| 87 |
-
|
| 88 |
-
|
| 89 |
-
|
| 90 |
-
|
| 91 |
-
|
| 92 |
-
|
| 93 |
-
|
| 94 |
-
|
| 95 |
-
|
| 96 |
-
|
| 97 |
-
|
| 98 |
-
|
| 99 |
-
|
| 100 |
-
|
| 101 |
-
|
| 102 |
-
|
| 103 |
-
|
| 104 |
-
|
| 105 |
-
|
| 106 |
-
|
| 107 |
-
|
| 108 |
-
|
| 109 |
-
|
| 110 |
-
|
| 111 |
-
|
| 112 |
-
|
| 113 |
-
|
| 114 |
-
|
| 115 |
-
|
| 116 |
-
|
| 117 |
-
|
| 118 |
-
|
| 119 |
-
|
| 120 |
-
|
| 121 |
-
|
| 122 |
-
|
| 123 |
-
|
| 124 |
-
|
| 125 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 126 |
|
| 127 |
-
#
|
| 128 |
-
@spaces.GPU
|
| 129 |
def generate_speech_simple(
|
| 130 |
text: str,
|
| 131 |
speaker_id: int,
|
|
@@ -134,43 +165,56 @@ def generate_speech_simple(
|
|
| 134 |
top_k: int = 50,
|
| 135 |
progress=gr.Progress()
|
| 136 |
) -> str:
|
| 137 |
-
|
| 138 |
-
|
| 139 |
-
|
| 140 |
-
|
| 141 |
-
|
| 142 |
-
|
| 143 |
-
|
| 144 |
-
|
| 145 |
-
|
| 146 |
-
|
| 147 |
-
|
| 148 |
-
|
| 149 |
-
|
| 150 |
-
|
| 151 |
-
|
| 152 |
-
|
| 153 |
-
|
| 154 |
-
|
| 155 |
-
|
| 156 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 157 |
|
| 158 |
-
#
|
| 159 |
def create_demo():
|
| 160 |
with gr.Blocks(title="CSM-1B Text-to-Speech") as demo:
|
| 161 |
gr.Markdown("# CSM-1B Text-to-Speech Demo")
|
| 162 |
-
gr.Markdown("
|
| 163 |
|
| 164 |
-
with gr.Tab("
|
| 165 |
with gr.Row():
|
| 166 |
with gr.Column():
|
| 167 |
text_input = gr.Textbox(
|
| 168 |
-
label="
|
| 169 |
-
placeholder="
|
| 170 |
lines=5
|
| 171 |
)
|
| 172 |
speaker_id = gr.Number(
|
| 173 |
-
label="ID
|
| 174 |
value=0,
|
| 175 |
precision=0,
|
| 176 |
minimum=0,
|
|
@@ -179,7 +223,7 @@ def create_demo():
|
|
| 179 |
|
| 180 |
with gr.Row():
|
| 181 |
max_duration = gr.Slider(
|
| 182 |
-
label="
|
| 183 |
minimum=1000,
|
| 184 |
maximum=90000,
|
| 185 |
value=30000,
|
|
@@ -200,38 +244,38 @@ def create_demo():
|
|
| 200 |
step=1
|
| 201 |
)
|
| 202 |
|
| 203 |
-
generate_btn = gr.Button("
|
| 204 |
|
| 205 |
with gr.Column():
|
| 206 |
-
output_audio = gr.Audio(label="
|
| 207 |
|
| 208 |
-
with gr.Tab("
|
| 209 |
-
gr.Markdown("
|
| 210 |
|
| 211 |
with gr.Row():
|
| 212 |
with gr.Column():
|
| 213 |
-
context_text1 = gr.Textbox(label="
|
| 214 |
-
context_audio1 = gr.Audio(label="
|
| 215 |
-
context_speaker1 = gr.Number(label="ID
|
| 216 |
|
| 217 |
-
context_text2 = gr.Textbox(label="
|
| 218 |
-
context_audio2 = gr.Audio(label="
|
| 219 |
-
context_speaker2 = gr.Number(label="ID
|
| 220 |
|
| 221 |
text_input_context = gr.Textbox(
|
| 222 |
-
label="
|
| 223 |
-
placeholder="
|
| 224 |
lines=3
|
| 225 |
)
|
| 226 |
speaker_id_context = gr.Number(
|
| 227 |
-
label="ID
|
| 228 |
value=0,
|
| 229 |
precision=0
|
| 230 |
)
|
| 231 |
|
| 232 |
with gr.Row():
|
| 233 |
max_duration_context = gr.Slider(
|
| 234 |
-
label="
|
| 235 |
minimum=1000,
|
| 236 |
maximum=90000,
|
| 237 |
value=30000,
|
|
@@ -252,27 +296,27 @@ def create_demo():
|
|
| 252 |
step=1
|
| 253 |
)
|
| 254 |
|
| 255 |
-
generate_context_btn = gr.Button("
|
| 256 |
|
| 257 |
with gr.Column():
|
| 258 |
-
output_audio_context = gr.Audio(label="
|
| 259 |
|
| 260 |
-
#
|
| 261 |
-
with gr.Tab("
|
| 262 |
-
gr.Markdown("###
|
| 263 |
gr.Markdown("""
|
| 264 |
-
|
| 265 |
|
| 266 |
-
|
| 267 |
-
1.
|
| 268 |
-
2.
|
| 269 |
|
| 270 |
-
|
| 271 |
""")
|
| 272 |
|
| 273 |
hf_token_input = gr.Textbox(
|
| 274 |
-
label="Hugging Face Token (
|
| 275 |
-
placeholder="
|
| 276 |
type="password"
|
| 277 |
)
|
| 278 |
|
|
@@ -280,57 +324,69 @@ def create_demo():
|
|
| 280 |
if token:
|
| 281 |
os.environ["HF_TOKEN"] = token
|
| 282 |
login(token=token)
|
| 283 |
-
return "
|
| 284 |
-
return "
|
| 285 |
|
| 286 |
-
set_token_btn = gr.Button("
|
| 287 |
-
token_status = gr.Textbox(label="
|
| 288 |
|
| 289 |
set_token_btn.click(fn=set_token, inputs=hf_token_input, outputs=token_status)
|
| 290 |
|
| 291 |
-
#
|
| 292 |
-
with gr.Tab("
|
| 293 |
-
gr.Markdown("###
|
| 294 |
gr.Markdown("""
|
| 295 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 296 |
|
| 297 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 298 |
|
| 299 |
-
|
|
|
|
|
|
|
|
|
|
| 300 |
|
| 301 |
-
|
| 302 |
""")
|
| 303 |
|
| 304 |
-
@spaces.GPU
|
| 305 |
def check_gpu():
|
| 306 |
if torch.cuda.is_available():
|
| 307 |
gpu_name = torch.cuda.get_device_name(0)
|
| 308 |
gpu_memory = torch.cuda.get_device_properties(0).total_memory / (1024**3)
|
| 309 |
-
return f"GPU: {gpu_name}\
|
| 310 |
else:
|
| 311 |
-
return "
|
| 312 |
|
| 313 |
-
check_gpu_btn = gr.Button("
|
| 314 |
-
gpu_info = gr.Textbox(label="
|
| 315 |
|
| 316 |
check_gpu_btn.click(fn=check_gpu, inputs=None, outputs=gpu_info)
|
| 317 |
|
| 318 |
-
#
|
| 319 |
-
load_model_btn = gr.Button("
|
| 320 |
-
model_status = gr.Textbox(label="
|
| 321 |
|
| 322 |
-
@spaces.GPU
|
| 323 |
def load_model_and_report():
|
| 324 |
global model_loaded
|
| 325 |
if model_loaded:
|
| 326 |
-
return "
|
| 327 |
else:
|
| 328 |
initialize_model()
|
| 329 |
-
return "
|
| 330 |
|
| 331 |
load_model_btn.click(fn=load_model_and_report, inputs=None, outputs=model_status)
|
| 332 |
|
| 333 |
-
#
|
| 334 |
generate_btn.click(
|
| 335 |
fn=generate_speech_simple,
|
| 336 |
inputs=[
|
|
@@ -363,7 +419,7 @@ def create_demo():
|
|
| 363 |
|
| 364 |
return demo
|
| 365 |
|
| 366 |
-
#
|
| 367 |
if __name__ == "__main__":
|
| 368 |
demo = create_demo()
|
| 369 |
demo.queue().launch()
|
|
|
|
| 11 |
from generator import Segment, load_csm_1b
|
| 12 |
from huggingface_hub import login
|
| 13 |
|
| 14 |
+
# Disable torch compile feature to avoid triton error
|
| 15 |
torch._dynamo.config.suppress_errors = True
|
| 16 |
|
| 17 |
+
# Check if GPU is available and configure the device
|
| 18 |
device = "cuda" if torch.cuda.is_available() else "cpu"
|
| 19 |
+
print(f"Using device: {device}")
|
| 20 |
|
| 21 |
+
# Login to Hugging Face Hub if token is available
|
| 22 |
def login_huggingface():
|
| 23 |
hf_token = os.environ.get("HF_TOKEN")
|
| 24 |
if hf_token:
|
| 25 |
+
print("Logging in to Hugging Face Hub...")
|
| 26 |
login(token=hf_token)
|
| 27 |
+
print("Login successful!")
|
| 28 |
else:
|
| 29 |
+
print("HF_TOKEN not found in environment variables. Some models may not be accessible.")
|
| 30 |
|
| 31 |
+
# Login at startup
|
| 32 |
login_huggingface()
|
| 33 |
|
| 34 |
+
# Global variables to track model state
|
| 35 |
generator = None
|
| 36 |
model_loaded = False
|
| 37 |
|
| 38 |
+
# Function to load model in ZeroGPU
|
| 39 |
+
@spaces.GPU(duration=30)
|
| 40 |
def initialize_model():
|
| 41 |
global generator, model_loaded
|
| 42 |
if not model_loaded:
|
| 43 |
+
print("Loading CSM-1B model in GPU...")
|
| 44 |
generator = load_csm_1b(device="cuda")
|
| 45 |
model_loaded = True
|
| 46 |
+
print("Model loaded successfully!")
|
| 47 |
return generator
|
| 48 |
|
| 49 |
+
# Function to get the loaded model
|
| 50 |
+
@spaces.GPU(duration=30)
|
| 51 |
def get_model():
|
| 52 |
global generator, model_loaded
|
| 53 |
if not model_loaded:
|
| 54 |
return initialize_model()
|
| 55 |
return generator
|
| 56 |
|
| 57 |
+
# Preload model if environment variable is set
|
| 58 |
+
def preload_model_if_needed():
|
| 59 |
+
if os.environ.get("PRELOAD_MODEL", "").lower() in ("true", "1", "yes"):
|
| 60 |
+
print("PRELOAD_MODEL is set. Attempting to preload model...")
|
| 61 |
+
try:
|
| 62 |
+
# We can't directly call initialize_model() here because it's decorated with @spaces.GPU
|
| 63 |
+
# Instead, we'll set a flag that will be checked when the first request comes in
|
| 64 |
+
global model_loaded
|
| 65 |
+
model_loaded = False
|
| 66 |
+
print("Model will be loaded on first request.")
|
| 67 |
+
except Exception as e:
|
| 68 |
+
print(f"Error during model preloading setup: {e}")
|
| 69 |
+
else:
|
| 70 |
+
print("PRELOAD_MODEL is not set. Model will be loaded on demand.")
|
| 71 |
+
|
| 72 |
+
# Call preload function at startup
|
| 73 |
+
preload_model_if_needed()
|
| 74 |
+
|
| 75 |
+
# Function to convert audio to tensor
|
| 76 |
def audio_to_tensor(audio_path: str) -> Tuple[torch.Tensor, int]:
|
| 77 |
waveform, sample_rate = torchaudio.load(audio_path)
|
| 78 |
+
waveform = waveform.mean(dim=0) # Convert stereo to mono if needed
|
| 79 |
return waveform, sample_rate
|
| 80 |
|
| 81 |
+
# Function to save audio tensor to file
|
| 82 |
def save_audio(audio_tensor: torch.Tensor, sample_rate: int) -> str:
|
| 83 |
temp_dir = tempfile.gettempdir()
|
| 84 |
output_path = os.path.join(temp_dir, f"csm1b_output_{int(time.time())}.wav")
|
| 85 |
torchaudio.save(output_path, audio_tensor.unsqueeze(0), sample_rate)
|
| 86 |
return output_path
|
| 87 |
|
| 88 |
+
# Function to generate speech from text using ZeroGPU
|
| 89 |
+
@spaces.GPU(duration=30)
|
| 90 |
def generate_speech(
|
| 91 |
text: str,
|
| 92 |
speaker_id: int,
|
|
|
|
| 101 |
top_k: int = 50,
|
| 102 |
progress=gr.Progress()
|
| 103 |
) -> str:
|
| 104 |
+
try:
|
| 105 |
+
# Get the loaded model
|
| 106 |
+
generator = get_model()
|
| 107 |
+
|
| 108 |
+
# Prepare context
|
| 109 |
+
context = []
|
| 110 |
+
progress(0.1, "Processing context...")
|
| 111 |
+
|
| 112 |
+
# Process context 1
|
| 113 |
+
if context_audio_path1 and context_text1:
|
| 114 |
+
waveform, sample_rate = audio_to_tensor(context_audio_path1)
|
| 115 |
+
# Resample if needed
|
| 116 |
+
if sample_rate != generator.sample_rate:
|
| 117 |
+
waveform = torchaudio.functional.resample(waveform, orig_freq=sample_rate, new_freq=generator.sample_rate)
|
| 118 |
+
context.append(Segment(speaker=context_speaker1, text=context_text1, audio=waveform))
|
| 119 |
+
|
| 120 |
+
# Process context 2
|
| 121 |
+
if context_audio_path2 and context_text2:
|
| 122 |
+
waveform, sample_rate = audio_to_tensor(context_audio_path2)
|
| 123 |
+
# Resample if needed
|
| 124 |
+
if sample_rate != generator.sample_rate:
|
| 125 |
+
waveform = torchaudio.functional.resample(waveform, orig_freq=sample_rate, new_freq=generator.sample_rate)
|
| 126 |
+
context.append(Segment(speaker=context_speaker2, text=context_text2, audio=waveform))
|
| 127 |
+
|
| 128 |
+
progress(0.3, "Generating audio...")
|
| 129 |
+
# Generate audio from text
|
| 130 |
+
audio = generator.generate(
|
| 131 |
+
text=text,
|
| 132 |
+
speaker=speaker_id,
|
| 133 |
+
context=context,
|
| 134 |
+
max_audio_length_ms=max_duration_ms,
|
| 135 |
+
temperature=temperature,
|
| 136 |
+
topk=top_k
|
| 137 |
+
)
|
| 138 |
+
|
| 139 |
+
progress(0.8, "Saving audio...")
|
| 140 |
+
# Save audio to file
|
| 141 |
+
output_path = save_audio(audio, generator.sample_rate)
|
| 142 |
+
|
| 143 |
+
progress(1.0, "Completed!")
|
| 144 |
+
return output_path
|
| 145 |
+
except spaces.zero.gradio.HTMLError as e:
|
| 146 |
+
# Handle ZeroGPU quota exceeded error
|
| 147 |
+
error_message = str(e)
|
| 148 |
+
if "GPU quota exceeded" in error_message:
|
| 149 |
+
# Extract wait time from error message
|
| 150 |
+
import re
|
| 151 |
+
wait_time_match = re.search(r"Try again in (\d+:\d+:\d+)", error_message)
|
| 152 |
+
wait_time = wait_time_match.group(1) if wait_time_match else "some time"
|
| 153 |
+
return f"GPU quota exceeded. Please try again in {wait_time}."
|
| 154 |
+
return f"GPU error: {error_message}"
|
| 155 |
+
except Exception as e:
|
| 156 |
+
return f"Error generating speech: {str(e)}"
|
| 157 |
|
| 158 |
+
# Function to generate simple speech without context
|
| 159 |
+
@spaces.GPU(duration=30)
|
| 160 |
def generate_speech_simple(
|
| 161 |
text: str,
|
| 162 |
speaker_id: int,
|
|
|
|
| 165 |
top_k: int = 50,
|
| 166 |
progress=gr.Progress()
|
| 167 |
) -> str:
|
| 168 |
+
try:
|
| 169 |
+
# Get the loaded model
|
| 170 |
+
generator = get_model()
|
| 171 |
+
|
| 172 |
+
progress(0.3, "Generating audio...")
|
| 173 |
+
# Generate audio from text
|
| 174 |
+
audio = generator.generate(
|
| 175 |
+
text=text,
|
| 176 |
+
speaker=speaker_id,
|
| 177 |
+
context=[], # No context
|
| 178 |
+
max_audio_length_ms=max_duration_ms,
|
| 179 |
+
temperature=temperature,
|
| 180 |
+
topk=top_k
|
| 181 |
+
)
|
| 182 |
+
|
| 183 |
+
progress(0.8, "Saving audio...")
|
| 184 |
+
# Save audio to file
|
| 185 |
+
output_path = save_audio(audio, generator.sample_rate)
|
| 186 |
+
|
| 187 |
+
progress(1.0, "Completed!")
|
| 188 |
+
return output_path
|
| 189 |
+
except spaces.zero.gradio.HTMLError as e:
|
| 190 |
+
# Handle ZeroGPU quota exceeded error
|
| 191 |
+
error_message = str(e)
|
| 192 |
+
if "GPU quota exceeded" in error_message:
|
| 193 |
+
# Extract wait time from error message
|
| 194 |
+
import re
|
| 195 |
+
wait_time_match = re.search(r"Try again in (\d+:\d+:\d+)", error_message)
|
| 196 |
+
wait_time = wait_time_match.group(1) if wait_time_match else "some time"
|
| 197 |
+
return f"GPU quota exceeded. Please try again in {wait_time}."
|
| 198 |
+
return f"GPU error: {error_message}"
|
| 199 |
+
except Exception as e:
|
| 200 |
+
return f"Error generating speech: {str(e)}"
|
| 201 |
|
| 202 |
+
# Create Gradio interface
|
| 203 |
def create_demo():
|
| 204 |
with gr.Blocks(title="CSM-1B Text-to-Speech") as demo:
|
| 205 |
gr.Markdown("# CSM-1B Text-to-Speech Demo")
|
| 206 |
+
gr.Markdown("CSM-1B (Collaborative Speech Model) is an advanced text-to-speech model capable of generating natural-sounding speech from text.")
|
| 207 |
|
| 208 |
+
with gr.Tab("Simple Audio Generation"):
|
| 209 |
with gr.Row():
|
| 210 |
with gr.Column():
|
| 211 |
text_input = gr.Textbox(
|
| 212 |
+
label="Text to convert to speech",
|
| 213 |
+
placeholder="Enter the text you want to convert to speech...",
|
| 214 |
lines=5
|
| 215 |
)
|
| 216 |
speaker_id = gr.Number(
|
| 217 |
+
label="Speaker ID",
|
| 218 |
value=0,
|
| 219 |
precision=0,
|
| 220 |
minimum=0,
|
|
|
|
| 223 |
|
| 224 |
with gr.Row():
|
| 225 |
max_duration = gr.Slider(
|
| 226 |
+
label="Maximum Duration (ms)",
|
| 227 |
minimum=1000,
|
| 228 |
maximum=90000,
|
| 229 |
value=30000,
|
|
|
|
| 244 |
step=1
|
| 245 |
)
|
| 246 |
|
| 247 |
+
generate_btn = gr.Button("Generate Audio")
|
| 248 |
|
| 249 |
with gr.Column():
|
| 250 |
+
output_audio = gr.Audio(label="Output Audio", type="filepath")
|
| 251 |
|
| 252 |
+
with gr.Tab("Audio Generation with Context"):
|
| 253 |
+
gr.Markdown("This feature allows you to provide audio clips and text as context to help the model generate more appropriate speech.")
|
| 254 |
|
| 255 |
with gr.Row():
|
| 256 |
with gr.Column():
|
| 257 |
+
context_text1 = gr.Textbox(label="Context Text 1", lines=2)
|
| 258 |
+
context_audio1 = gr.Audio(label="Context Audio 1", type="filepath")
|
| 259 |
+
context_speaker1 = gr.Number(label="Speaker ID 1", value=0, precision=0)
|
| 260 |
|
| 261 |
+
context_text2 = gr.Textbox(label="Context Text 2", lines=2)
|
| 262 |
+
context_audio2 = gr.Audio(label="Context Audio 2", type="filepath")
|
| 263 |
+
context_speaker2 = gr.Number(label="Speaker ID 2", value=1, precision=0)
|
| 264 |
|
| 265 |
text_input_context = gr.Textbox(
|
| 266 |
+
label="Text to convert to speech",
|
| 267 |
+
placeholder="Enter the text you want to convert to speech...",
|
| 268 |
lines=3
|
| 269 |
)
|
| 270 |
speaker_id_context = gr.Number(
|
| 271 |
+
label="Speaker ID",
|
| 272 |
value=0,
|
| 273 |
precision=0
|
| 274 |
)
|
| 275 |
|
| 276 |
with gr.Row():
|
| 277 |
max_duration_context = gr.Slider(
|
| 278 |
+
label="Maximum Duration (ms)",
|
| 279 |
minimum=1000,
|
| 280 |
maximum=90000,
|
| 281 |
value=30000,
|
|
|
|
| 296 |
step=1
|
| 297 |
)
|
| 298 |
|
| 299 |
+
generate_context_btn = gr.Button("Generate Audio with Context")
|
| 300 |
|
| 301 |
with gr.Column():
|
| 302 |
+
output_audio_context = gr.Audio(label="Output Audio", type="filepath")
|
| 303 |
|
| 304 |
+
# Add Hugging Face configuration tab
|
| 305 |
+
with gr.Tab("Configuration"):
|
| 306 |
+
gr.Markdown("### Hugging Face Token Configuration")
|
| 307 |
gr.Markdown("""
|
| 308 |
+
To use the CSM-1B model, you need access to the model on Hugging Face.
|
| 309 |
|
| 310 |
+
You can configure your token by:
|
| 311 |
+
1. Create a token at [Hugging Face Settings](https://huggingface.co/settings/tokens)
|
| 312 |
+
2. Set the `HF_TOKEN` environment variable with your token value
|
| 313 |
|
| 314 |
+
Note: In Hugging Face Spaces, you can set environment variables in the Space Settings.
|
| 315 |
""")
|
| 316 |
|
| 317 |
hf_token_input = gr.Textbox(
|
| 318 |
+
label="Hugging Face Token (Only for this session)",
|
| 319 |
+
placeholder="Enter your token...",
|
| 320 |
type="password"
|
| 321 |
)
|
| 322 |
|
|
|
|
| 324 |
if token:
|
| 325 |
os.environ["HF_TOKEN"] = token
|
| 326 |
login(token=token)
|
| 327 |
+
return "Token set successfully! You can now load the model."
|
| 328 |
+
return "Invalid token. Please enter a valid token."
|
| 329 |
|
| 330 |
+
set_token_btn = gr.Button("Set Token")
|
| 331 |
+
token_status = gr.Textbox(label="Status", interactive=False)
|
| 332 |
|
| 333 |
set_token_btn.click(fn=set_token, inputs=hf_token_input, outputs=token_status)
|
| 334 |
|
| 335 |
+
# Add GPU information tab
|
| 336 |
+
with gr.Tab("GPU Information"):
|
| 337 |
+
gr.Markdown("### About ZeroGPU")
|
| 338 |
gr.Markdown("""
|
| 339 |
+
This application uses Hugging Face Spaces' ZeroGPU to optimize GPU usage.
|
| 340 |
+
|
| 341 |
+
ZeroGPU helps free up GPU memory when not in use, saving resources and improving performance.
|
| 342 |
+
|
| 343 |
+
When you generate audio, the GPU will be used automatically and released after completion.
|
| 344 |
|
| 345 |
+
Note: In the ZeroGPU environment, CUDA is not initialized in the main process, but only in functions with the @spaces.GPU decorator.
|
| 346 |
+
""")
|
| 347 |
+
|
| 348 |
+
gr.Markdown("### GPU Quota Information")
|
| 349 |
+
gr.Markdown("""
|
| 350 |
+
Hugging Face Spaces has GPU quota limitations:
|
| 351 |
|
| 352 |
+
- Each GPU operation has a default duration of 60 seconds
|
| 353 |
+
- We've reduced this to 30 seconds for audio generation and 10 seconds for GPU checks
|
| 354 |
+
- If you exceed your quota, you'll need to wait for it to reset (usually a few hours)
|
| 355 |
+
- For better performance, try generating shorter audio clips
|
| 356 |
|
| 357 |
+
If you encounter a "GPU quota exceeded" error, please wait for the specified time and try again.
|
| 358 |
""")
|
| 359 |
|
| 360 |
+
@spaces.GPU(duration=10)
|
| 361 |
def check_gpu():
|
| 362 |
if torch.cuda.is_available():
|
| 363 |
gpu_name = torch.cuda.get_device_name(0)
|
| 364 |
gpu_memory = torch.cuda.get_device_properties(0).total_memory / (1024**3)
|
| 365 |
+
return f"GPU: {gpu_name}\nMemory: {gpu_memory:.2f} GB"
|
| 366 |
else:
|
| 367 |
+
return "No GPU found. The application will run on CPU."
|
| 368 |
|
| 369 |
+
check_gpu_btn = gr.Button("Check GPU")
|
| 370 |
+
gpu_info = gr.Textbox(label="GPU Information", interactive=False)
|
| 371 |
|
| 372 |
check_gpu_btn.click(fn=check_gpu, inputs=None, outputs=gpu_info)
|
| 373 |
|
| 374 |
+
# Add model loading button
|
| 375 |
+
load_model_btn = gr.Button("Load Model")
|
| 376 |
+
model_status = gr.Textbox(label="Model Status", interactive=False)
|
| 377 |
|
| 378 |
+
@spaces.GPU(duration=10)
|
| 379 |
def load_model_and_report():
|
| 380 |
global model_loaded
|
| 381 |
if model_loaded:
|
| 382 |
+
return "Model has already been loaded!"
|
| 383 |
else:
|
| 384 |
initialize_model()
|
| 385 |
+
return "Model loaded successfully!"
|
| 386 |
|
| 387 |
load_model_btn.click(fn=load_model_and_report, inputs=None, outputs=model_status)
|
| 388 |
|
| 389 |
+
# Connect components
|
| 390 |
generate_btn.click(
|
| 391 |
fn=generate_speech_simple,
|
| 392 |
inputs=[
|
|
|
|
| 419 |
|
| 420 |
return demo
|
| 421 |
|
| 422 |
+
# Launch the application
|
| 423 |
if __name__ == "__main__":
|
| 424 |
demo = create_demo()
|
| 425 |
demo.queue().launch()
|
generator.py
CHANGED
|
@@ -10,7 +10,7 @@ from tokenizers.processors import TemplateProcessing
|
|
| 10 |
from transformers import AutoTokenizer
|
| 11 |
from watermarking import CSM_1B_GH_WATERMARK, load_watermarker, watermark
|
| 12 |
|
| 13 |
-
#
|
| 14 |
torch._dynamo.config.suppress_errors = True
|
| 15 |
|
| 16 |
@dataclass
|
|
@@ -167,19 +167,19 @@ class Generator:
|
|
| 167 |
|
| 168 |
def load_csm_1b(device: str = "cuda") -> Generator:
|
| 169 |
"""
|
| 170 |
-
|
| 171 |
|
| 172 |
Args:
|
| 173 |
-
device:
|
| 174 |
|
| 175 |
Returns:
|
| 176 |
-
Generator:
|
| 177 |
"""
|
| 178 |
try:
|
| 179 |
-
#
|
| 180 |
-
#
|
| 181 |
if 'cuda' in device and not torch.cuda.is_initialized():
|
| 182 |
-
#
|
| 183 |
model = Model.from_pretrained("sesame/csm-1b")
|
| 184 |
else:
|
| 185 |
model = Model.from_pretrained("sesame/csm-1b")
|
|
@@ -188,7 +188,7 @@ def load_csm_1b(device: str = "cuda") -> Generator:
|
|
| 188 |
generator = Generator(model)
|
| 189 |
return generator
|
| 190 |
except Exception as e:
|
| 191 |
-
print(f"
|
| 192 |
-
print("
|
| 193 |
-
print("
|
| 194 |
raise e
|
|
|
|
| 10 |
from transformers import AutoTokenizer
|
| 11 |
from watermarking import CSM_1B_GH_WATERMARK, load_watermarker, watermark
|
| 12 |
|
| 13 |
+
# Disable torch compile feature to avoid triton error
|
| 14 |
torch._dynamo.config.suppress_errors = True
|
| 15 |
|
| 16 |
@dataclass
|
|
|
|
| 167 |
|
| 168 |
def load_csm_1b(device: str = "cuda") -> Generator:
|
| 169 |
"""
|
| 170 |
+
Load the CSM-1B model from Hugging Face Hub.
|
| 171 |
|
| 172 |
Args:
|
| 173 |
+
device: Device to run the model on (cuda or cpu)
|
| 174 |
|
| 175 |
Returns:
|
| 176 |
+
Generator: Generator object to create audio from text
|
| 177 |
"""
|
| 178 |
try:
|
| 179 |
+
# In ZeroGPU, CUDA should not be initialized in the main process
|
| 180 |
+
# Only move the model to GPU when called in a function with the @spaces.GPU decorator
|
| 181 |
if 'cuda' in device and not torch.cuda.is_initialized():
|
| 182 |
+
# Use CPU for the main process
|
| 183 |
model = Model.from_pretrained("sesame/csm-1b")
|
| 184 |
else:
|
| 185 |
model = Model.from_pretrained("sesame/csm-1b")
|
|
|
|
| 188 |
generator = Generator(model)
|
| 189 |
return generator
|
| 190 |
except Exception as e:
|
| 191 |
+
print(f"Error loading model: {e}")
|
| 192 |
+
print("Please check if you are logged in to Hugging Face Hub.")
|
| 193 |
+
print("You may need to request access to the model at: https://huggingface.co/sesame/csm-1b")
|
| 194 |
raise e
|
test_model.py
CHANGED
|
@@ -6,29 +6,29 @@ from generator import Segment, load_csm_1b
|
|
| 6 |
from huggingface_hub import login
|
| 7 |
|
| 8 |
def login_huggingface():
|
| 9 |
-
"""
|
| 10 |
hf_token = os.environ.get("HF_TOKEN")
|
| 11 |
|
| 12 |
if not hf_token:
|
| 13 |
-
print("
|
| 14 |
-
hf_token = input("
|
| 15 |
|
| 16 |
if hf_token:
|
| 17 |
-
print("
|
| 18 |
login(token=hf_token)
|
| 19 |
-
print("
|
| 20 |
return True
|
| 21 |
else:
|
| 22 |
-
print("
|
| 23 |
return False
|
| 24 |
|
| 25 |
@spaces.GPU
|
| 26 |
def generate_test_audio(text, speaker_id, device):
|
| 27 |
-
"""
|
| 28 |
generator = load_csm_1b(device=device)
|
| 29 |
-
print("
|
| 30 |
|
| 31 |
-
print(f"
|
| 32 |
audio = generator.generate(
|
| 33 |
text=text,
|
| 34 |
speaker=speaker_id,
|
|
@@ -41,33 +41,33 @@ def generate_test_audio(text, speaker_id, device):
|
|
| 41 |
return audio, generator.sample_rate
|
| 42 |
|
| 43 |
def test_model():
|
| 44 |
-
print("
|
| 45 |
|
| 46 |
-
#
|
| 47 |
login_huggingface()
|
| 48 |
|
| 49 |
-
#
|
| 50 |
device = "cuda" if torch.cuda.is_available() else "cpu"
|
| 51 |
-
print(f"
|
| 52 |
|
| 53 |
-
#
|
| 54 |
-
print("
|
| 55 |
try:
|
| 56 |
-
#
|
| 57 |
-
text = "
|
| 58 |
speaker_id = 0
|
| 59 |
|
| 60 |
audio, sample_rate = generate_test_audio(text, speaker_id, device)
|
| 61 |
|
| 62 |
-
#
|
| 63 |
output_path = "test_output.wav"
|
| 64 |
torchaudio.save(output_path, audio.unsqueeze(0), sample_rate)
|
| 65 |
-
print(f"
|
| 66 |
|
| 67 |
-
print("
|
| 68 |
except Exception as e:
|
| 69 |
-
print(f"
|
| 70 |
-
print("
|
| 71 |
|
| 72 |
if __name__ == "__main__":
|
| 73 |
test_model()
|
|
|
|
| 6 |
from huggingface_hub import login
|
| 7 |
|
| 8 |
def login_huggingface():
|
| 9 |
+
"""Login to Hugging Face Hub using token from environment variable or user input"""
|
| 10 |
hf_token = os.environ.get("HF_TOKEN")
|
| 11 |
|
| 12 |
if not hf_token:
|
| 13 |
+
print("HF_TOKEN not found in environment variables.")
|
| 14 |
+
hf_token = input("Please enter your Hugging Face token: ")
|
| 15 |
|
| 16 |
if hf_token:
|
| 17 |
+
print("Logging in to Hugging Face Hub...")
|
| 18 |
login(token=hf_token)
|
| 19 |
+
print("Login successful!")
|
| 20 |
return True
|
| 21 |
else:
|
| 22 |
+
print("No token provided. Some models may not be accessible.")
|
| 23 |
return False
|
| 24 |
|
| 25 |
@spaces.GPU
|
| 26 |
def generate_test_audio(text, speaker_id, device):
|
| 27 |
+
"""Generate test audio using ZeroGPU"""
|
| 28 |
generator = load_csm_1b(device=device)
|
| 29 |
+
print("Model loaded successfully!")
|
| 30 |
|
| 31 |
+
print(f"Generating audio for text: '{text}'")
|
| 32 |
audio = generator.generate(
|
| 33 |
text=text,
|
| 34 |
speaker=speaker_id,
|
|
|
|
| 41 |
return audio, generator.sample_rate
|
| 42 |
|
| 43 |
def test_model():
|
| 44 |
+
print("Testing CSM-1B model...")
|
| 45 |
|
| 46 |
+
# Login to Hugging Face Hub
|
| 47 |
login_huggingface()
|
| 48 |
|
| 49 |
+
# Check if GPU is available and configure the device
|
| 50 |
device = "cuda" if torch.cuda.is_available() else "cpu"
|
| 51 |
+
print(f"Using device: {device}")
|
| 52 |
|
| 53 |
+
# Load CSM-1B model and generate audio
|
| 54 |
+
print("Loading CSM-1B model...")
|
| 55 |
try:
|
| 56 |
+
# Use ZeroGPU to generate audio
|
| 57 |
+
text = "Hello, this is a test of the CSM-1B model."
|
| 58 |
speaker_id = 0
|
| 59 |
|
| 60 |
audio, sample_rate = generate_test_audio(text, speaker_id, device)
|
| 61 |
|
| 62 |
+
# Save audio to file
|
| 63 |
output_path = "test_output.wav"
|
| 64 |
torchaudio.save(output_path, audio.unsqueeze(0), sample_rate)
|
| 65 |
+
print(f"Audio saved to file: {output_path}")
|
| 66 |
|
| 67 |
+
print("Test completed!")
|
| 68 |
except Exception as e:
|
| 69 |
+
print(f"Error testing model: {e}")
|
| 70 |
+
print("Please check your token and access permissions.")
|
| 71 |
|
| 72 |
if __name__ == "__main__":
|
| 73 |
test_model()
|