Spaces:
Running
Running
Update README.md
Browse files
README.md
CHANGED
|
@@ -50,278 +50,3 @@ pip install fastrtc[vad, tts]
|
|
| 50 |
## Docs
|
| 51 |
|
| 52 |
[https://fastrtc.org](https://fastrtc.org)
|
| 53 |
-
|
| 54 |
-
## Examples
|
| 55 |
-
See the [Cookbook](https://fastrtc.org/pr-preview/pr-60/cookbook/) for examples of how to use the library.
|
| 56 |
-
|
| 57 |
-
<table>
|
| 58 |
-
<tr>
|
| 59 |
-
<td width="50%">
|
| 60 |
-
<h3>π£οΈπ Gemini Audio Video Chat</h3>
|
| 61 |
-
<p>Stream BOTH your webcam video and audio feeds to Google Gemini. You can also upload images to augment your conversation!</p>
|
| 62 |
-
<video width="100%" src="https://github.com/user-attachments/assets/9636dc97-4fee-46bb-abb8-b92e69c08c71" controls></video>
|
| 63 |
-
<p>
|
| 64 |
-
<a href="https://huggingface.co/spaces/freddyaboulton/gemini-audio-video-chat">Demo</a> |
|
| 65 |
-
<a href="https://huggingface.co/spaces/freddyaboulton/gemini-audio-video-chat/blob/main/app.py">Code</a>
|
| 66 |
-
</p>
|
| 67 |
-
</td>
|
| 68 |
-
<td width="50%">
|
| 69 |
-
<h3>π£οΈ Google Gemini Real Time Voice API</h3>
|
| 70 |
-
<p>Talk to Gemini in real time using Google's voice API.</p>
|
| 71 |
-
<video width="100%" src="https://github.com/user-attachments/assets/ea6d18cb-8589-422b-9bba-56332d9f61de" controls></video>
|
| 72 |
-
<p>
|
| 73 |
-
<a href="https://huggingface.co/spaces/fastrtc/talk-to-gemini">Demo</a> |
|
| 74 |
-
<a href="https://huggingface.co/spaces/fastrtc/talk-to-gemini/blob/main/app.py">Code</a>
|
| 75 |
-
</p>
|
| 76 |
-
</td>
|
| 77 |
-
</tr>
|
| 78 |
-
|
| 79 |
-
<tr>
|
| 80 |
-
<td width="50%">
|
| 81 |
-
<h3>π£οΈ OpenAI Real Time Voice API</h3>
|
| 82 |
-
<p>Talk to ChatGPT in real time using OpenAI's voice API.</p>
|
| 83 |
-
<video width="100%" src="https://github.com/user-attachments/assets/178bdadc-f17b-461a-8d26-e915c632ff80" controls></video>
|
| 84 |
-
<p>
|
| 85 |
-
<a href="https://huggingface.co/spaces/fastrtc/talk-to-openai">Demo</a> |
|
| 86 |
-
<a href="https://huggingface.co/spaces/fastrtc/talk-to-openai/blob/main/app.py">Code</a>
|
| 87 |
-
</p>
|
| 88 |
-
</td>
|
| 89 |
-
<td width="50%">
|
| 90 |
-
<h3>π€ Hello Computer</h3>
|
| 91 |
-
<p>Say computer before asking your question!</p>
|
| 92 |
-
<video width="100%" src="https://github.com/user-attachments/assets/afb2a3ef-c1ab-4cfb-872d-578f895a10d5" controls></video>
|
| 93 |
-
<p>
|
| 94 |
-
<a href="https://huggingface.co/spaces/fastrtc/hello-computer">Demo</a> |
|
| 95 |
-
<a href="https://huggingface.co/spaces/fastrtc/hello-computer/blob/main/app.py">Code</a>
|
| 96 |
-
</p>
|
| 97 |
-
</td>
|
| 98 |
-
</tr>
|
| 99 |
-
|
| 100 |
-
<tr>
|
| 101 |
-
<td width="50%">
|
| 102 |
-
<h3>π€ Llama Code Editor</h3>
|
| 103 |
-
<p>Create and edit HTML pages with just your voice! Powered by SambaNova systems.</p>
|
| 104 |
-
<video width="100%" src="https://github.com/user-attachments/assets/98523cf3-dac8-4127-9649-d91a997e3ef5" controls></video>
|
| 105 |
-
<p>
|
| 106 |
-
<a href="https://huggingface.co/spaces/fastrtc/llama-code-editor">Demo</a> |
|
| 107 |
-
<a href="https://huggingface.co/spaces/fastrtc/llama-code-editor/blob/main/app.py">Code</a>
|
| 108 |
-
</p>
|
| 109 |
-
</td>
|
| 110 |
-
<td width="50%">
|
| 111 |
-
<h3>π£οΈ Talk to Claude</h3>
|
| 112 |
-
<p>Use the Anthropic and Play.Ht APIs to have an audio conversation with Claude.</p>
|
| 113 |
-
<video width="100%" src="https://github.com/user-attachments/assets/fb6ef07f-3ccd-444a-997b-9bc9bdc035d3" controls></video>
|
| 114 |
-
<p>
|
| 115 |
-
<a href="https://huggingface.co/spaces/fastrtc/talk-to-claude">Demo</a> |
|
| 116 |
-
<a href="https://huggingface.co/spaces/fastrtc/talk-to-claude/blob/main/app.py">Code</a>
|
| 117 |
-
</p>
|
| 118 |
-
</td>
|
| 119 |
-
</tr>
|
| 120 |
-
|
| 121 |
-
<tr>
|
| 122 |
-
<td width="50%">
|
| 123 |
-
<h3>π΅ Whisper Transcription</h3>
|
| 124 |
-
<p>Have whisper transcribe your speech in real time!</p>
|
| 125 |
-
<video width="100%" src="https://github.com/user-attachments/assets/87603053-acdc-4c8a-810f-f618c49caafb" controls></video>
|
| 126 |
-
<p>
|
| 127 |
-
<a href="https://huggingface.co/spaces/fastrtc/whisper-realtime">Demo</a> |
|
| 128 |
-
<a href="https://huggingface.co/spaces/fastrtc/whisper-realtime/blob/main/app.py">Code</a>
|
| 129 |
-
</p>
|
| 130 |
-
</td>
|
| 131 |
-
<td width="50%">
|
| 132 |
-
<h3>π· Yolov10 Object Detection</h3>
|
| 133 |
-
<p>Run the Yolov10 model on a user webcam stream in real time!</p>
|
| 134 |
-
<video width="100%" src="https://github.com/user-attachments/assets/f82feb74-a071-4e81-9110-a01989447ceb" controls></video>
|
| 135 |
-
<p>
|
| 136 |
-
<a href="https://huggingface.co/spaces/fastrtc/object-detection">Demo</a> |
|
| 137 |
-
<a href="https://huggingface.co/spaces/fastrtc/object-detection/blob/main/app.py">Code</a>
|
| 138 |
-
</p>
|
| 139 |
-
</td>
|
| 140 |
-
</tr>
|
| 141 |
-
|
| 142 |
-
<tr>
|
| 143 |
-
<td width="50%">
|
| 144 |
-
<h3>π£οΈ Kyutai Moshi</h3>
|
| 145 |
-
<p>Kyutai's moshi is a novel speech-to-speech model for modeling human conversations.</p>
|
| 146 |
-
<video width="100%" src="https://github.com/user-attachments/assets/becc7a13-9e89-4a19-9df2-5fb1467a0137" controls></video>
|
| 147 |
-
<p>
|
| 148 |
-
<a href="https://huggingface.co/spaces/freddyaboulton/talk-to-moshi">Demo</a> |
|
| 149 |
-
<a href="https://huggingface.co/spaces/freddyaboulton/talk-to-moshi/blob/main/app.py">Code</a>
|
| 150 |
-
</p>
|
| 151 |
-
</td>
|
| 152 |
-
<td width="50%">
|
| 153 |
-
<h3>π£οΈ Hello Llama: Stop Word Detection</h3>
|
| 154 |
-
<p>A code editor built with Llama 3.3 70b that is triggered by the phrase "Hello Llama". Build a Siri-like coding assistant in 100 lines of code!</p>
|
| 155 |
-
<video width="100%" src="https://github.com/user-attachments/assets/3e10cb15-ff1b-4b17-b141-ff0ad852e613" controls></video>
|
| 156 |
-
<p>
|
| 157 |
-
<a href="https://huggingface.co/spaces/freddyaboulton/hey-llama-code-editor">Demo</a> |
|
| 158 |
-
<a href="https://huggingface.co/spaces/freddyaboulton/hey-llama-code-editor/blob/main/app.py">Code</a>
|
| 159 |
-
</p>
|
| 160 |
-
</td>
|
| 161 |
-
</tr>
|
| 162 |
-
</table>
|
| 163 |
-
|
| 164 |
-
## Usage
|
| 165 |
-
|
| 166 |
-
This is an shortened version of the official [usage guide](https://freddyaboulton.github.io/gradio-webrtc/user-guide/).
|
| 167 |
-
|
| 168 |
-
- `.ui.launch()`: Launch a built-in UI for easily testing and sharing your stream. Built with [Gradio](https://www.gradio.app/).
|
| 169 |
-
- `.fastphone()`: Get a free temporary phone number to call into your stream. Hugging Face token required.
|
| 170 |
-
- `.mount(app)`: Mount the stream on a [FastAPI](https://fastapi.tiangolo.com/) app. Perfect for integrating with your already existing production system.
|
| 171 |
-
|
| 172 |
-
|
| 173 |
-
## Quickstart
|
| 174 |
-
|
| 175 |
-
### Echo Audio
|
| 176 |
-
|
| 177 |
-
```python
|
| 178 |
-
from fastrtc import Stream, ReplyOnPause
|
| 179 |
-
import numpy as np
|
| 180 |
-
|
| 181 |
-
def echo(audio: tuple[int, np.ndarray]):
|
| 182 |
-
# The function will be passed the audio until the user pauses
|
| 183 |
-
# Implement any iterator that yields audio
|
| 184 |
-
# See "LLM Voice Chat" for a more complete example
|
| 185 |
-
yield audio
|
| 186 |
-
|
| 187 |
-
stream = Stream(
|
| 188 |
-
handler=ReplyOnPause(detection),
|
| 189 |
-
modality="audio",
|
| 190 |
-
mode="send-receive",
|
| 191 |
-
)
|
| 192 |
-
```
|
| 193 |
-
|
| 194 |
-
### LLM Voice Chat
|
| 195 |
-
|
| 196 |
-
```py
|
| 197 |
-
from fastrtc import (
|
| 198 |
-
ReplyOnPause, AdditionalOutputs, Stream,
|
| 199 |
-
audio_to_bytes, aggregate_bytes_to_16bit
|
| 200 |
-
)
|
| 201 |
-
import gradio as gr
|
| 202 |
-
from groq import Groq
|
| 203 |
-
import anthropic
|
| 204 |
-
from elevenlabs import ElevenLabs
|
| 205 |
-
|
| 206 |
-
groq_client = Groq()
|
| 207 |
-
claude_client = anthropic.Anthropic()
|
| 208 |
-
tts_client = ElevenLabs()
|
| 209 |
-
|
| 210 |
-
|
| 211 |
-
# See "Talk to Claude" in Cookbook for an example of how to keep
|
| 212 |
-
# track of the chat history.
|
| 213 |
-
def response(
|
| 214 |
-
audio: tuple[int, np.ndarray],
|
| 215 |
-
):
|
| 216 |
-
prompt = groq_client.audio.transcriptions.create(
|
| 217 |
-
file=("audio-file.mp3", audio_to_bytes(audio)),
|
| 218 |
-
model="whisper-large-v3-turbo",
|
| 219 |
-
response_format="verbose_json",
|
| 220 |
-
).text
|
| 221 |
-
response = claude_client.messages.create(
|
| 222 |
-
model="claude-3-5-haiku-20241022",
|
| 223 |
-
max_tokens=512,
|
| 224 |
-
messages=[{"role": "user", "content": prompt}],
|
| 225 |
-
)
|
| 226 |
-
response_text = " ".join(
|
| 227 |
-
block.text
|
| 228 |
-
for block in response.content
|
| 229 |
-
if getattr(block, "type", None) == "text"
|
| 230 |
-
)
|
| 231 |
-
iterator = tts_client.text_to_speech.convert_as_stream(
|
| 232 |
-
text=response_text,
|
| 233 |
-
voice_id="JBFqnCBsd6RMkjVDRZzb",
|
| 234 |
-
model_id="eleven_multilingual_v2",
|
| 235 |
-
output_format="pcm_24000"
|
| 236 |
-
|
| 237 |
-
)
|
| 238 |
-
for chunk in aggregate_bytes_to_16bit(iterator):
|
| 239 |
-
audio_array = np.frombuffer(chunk, dtype=np.int16).reshape(1, -1)
|
| 240 |
-
yield (24000, audio_array)
|
| 241 |
-
|
| 242 |
-
stream = Stream(
|
| 243 |
-
modality="audio",
|
| 244 |
-
mode="send-receive",
|
| 245 |
-
handler=ReplyOnPause(response),
|
| 246 |
-
)
|
| 247 |
-
```
|
| 248 |
-
|
| 249 |
-
### Webcam Stream
|
| 250 |
-
|
| 251 |
-
```python
|
| 252 |
-
from fastrtc import Stream
|
| 253 |
-
import numpy as np
|
| 254 |
-
|
| 255 |
-
|
| 256 |
-
def flip_vertically(image):
|
| 257 |
-
return np.flip(image, axis=0)
|
| 258 |
-
|
| 259 |
-
|
| 260 |
-
stream = Stream(
|
| 261 |
-
handler=flip_vertically,
|
| 262 |
-
modality="video",
|
| 263 |
-
mode="send-receive",
|
| 264 |
-
)
|
| 265 |
-
```
|
| 266 |
-
|
| 267 |
-
### Object Detection
|
| 268 |
-
|
| 269 |
-
```python
|
| 270 |
-
from fastrtc import Stream
|
| 271 |
-
import gradio as gr
|
| 272 |
-
import cv2
|
| 273 |
-
from huggingface_hub import hf_hub_download
|
| 274 |
-
from .inference import YOLOv10
|
| 275 |
-
|
| 276 |
-
model_file = hf_hub_download(
|
| 277 |
-
repo_id="onnx-community/yolov10n", filename="onnx/model.onnx"
|
| 278 |
-
)
|
| 279 |
-
|
| 280 |
-
# git clone https://huggingface.co/spaces/fastrtc/object-detection
|
| 281 |
-
# for YOLOv10 implementation
|
| 282 |
-
model = YOLOv10(model_file)
|
| 283 |
-
|
| 284 |
-
def detection(image, conf_threshold=0.3):
|
| 285 |
-
image = cv2.resize(image, (model.input_width, model.input_height))
|
| 286 |
-
new_image = model.detect_objects(image, conf_threshold)
|
| 287 |
-
return cv2.resize(new_image, (500, 500))
|
| 288 |
-
|
| 289 |
-
stream = Stream(
|
| 290 |
-
handler=detection,
|
| 291 |
-
modality="video",
|
| 292 |
-
mode="send-receive",
|
| 293 |
-
additional_inputs=[
|
| 294 |
-
gr.Slider(minimum=0, maximum=1, step=0.01, value=0.3)
|
| 295 |
-
]
|
| 296 |
-
)
|
| 297 |
-
```
|
| 298 |
-
|
| 299 |
-
## Running the Stream
|
| 300 |
-
|
| 301 |
-
Run:
|
| 302 |
-
|
| 303 |
-
### Gradio
|
| 304 |
-
|
| 305 |
-
```py
|
| 306 |
-
stream.ui.launch()
|
| 307 |
-
```
|
| 308 |
-
|
| 309 |
-
### Telephone (Audio Only)
|
| 310 |
-
|
| 311 |
-
```py
|
| 312 |
-
stream.fastphone()
|
| 313 |
-
```
|
| 314 |
-
|
| 315 |
-
### FastAPI
|
| 316 |
-
|
| 317 |
-
```py
|
| 318 |
-
app = FastAPI()
|
| 319 |
-
stream.mount(app)
|
| 320 |
-
|
| 321 |
-
# Optional: Add routes
|
| 322 |
-
@app.get("/")
|
| 323 |
-
async def _():
|
| 324 |
-
return HTMLResponse(content=open("index.html").read())
|
| 325 |
-
|
| 326 |
-
# uvicorn app:app --host 0.0.0.0 --port 8000
|
| 327 |
-
```
|
|
|
|
| 50 |
## Docs
|
| 51 |
|
| 52 |
[https://fastrtc.org](https://fastrtc.org)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|