|
|
--- |
|
|
license: apache-2.0 |
|
|
pipeline_tag: other |
|
|
datasets: |
|
|
- RS2002/RPMC-L2 |
|
|
tags: |
|
|
- stage-lighting |
|
|
- generative-task |
|
|
- music-to-light |
|
|
--- |
|
|
|
|
|
# Skip-BART |
|
|
|
|
|
Skip-BART is an end-to-end generative model designed for **Automatic Stage Lighting Control (ASLC)**. Unlike traditional rule-based methods, Skip-BART conceptualizes lighting control as a generative task, learning directly from professional lighting engineers to predict vivid, human-like lighting sequences synchronized with music. |
|
|
|
|
|
This model was presented in the paper [Automatic Stage Lighting Control: Is it a Rule-Driven Process or Generative Task?](https://huggingface.co/papers/2506.01482). |
|
|
|
|
|
- **Repository**: [https://github.com/RS2002/Skip-BART](https://github.com/RS2002/Skip-BART) |
|
|
- **Dataset**: [RS2002/RPMC-L2](https://huggingface.co/datasets/RS2002/RPMC-L2) |
|
|
|
|
|
## Model Details |
|
|
|
|
|
- **Model Type**: Transformer-based model (BART architecture) with skip connections. |
|
|
- **Task**: Stage lighting sequence generation (predicting light hue and intensity). |
|
|
- **Architecture**: BART-based structure enhanced with a novel skip-connection mechanism to strengthen the relationship between musical frames and lighting states. |
|
|
- **Input Format**: Encoder input (batch_size, length, 512) for audio features; Decoder input (batch_size, length, 2) for lighting parameters. |
|
|
- **Output Format**: Hidden states representing lighting control parameters (dimension 1024). |
|
|
|
|
|
## Training Data |
|
|
|
|
|
The model was trained on the **RPMC-L2** dataset, a self-collected dataset containing music and corresponding stage lighting data synchronized within a frame grid. |
|
|
|
|
|
## Usage |
|
|
|
|
|
### Installation |
|
|
|
|
|
```shell |
|
|
git clone https://huggingface.co/RS2002/Skip-BART |
|
|
``` |
|
|
|
|
|
### Example Code |
|
|
|
|
|
The following snippet demonstrates how to load the model and perform a forward pass (requires `model.py` from the official repository). |
|
|
|
|
|
```python |
|
|
import torch |
|
|
from model import Skip_BART |
|
|
|
|
|
# Load the model |
|
|
model = Skip_BART.from_pretrained("RS2002/Skip-BART") |
|
|
|
|
|
# Example input |
|
|
x_encoder = torch.rand((2, 1024, 512)) |
|
|
x_decoder = torch.randint(0, 10, (2, 1024, 2)) |
|
|
encoder_attention_mask = torch.zeros((2, 1024)) |
|
|
decoder_attention_mask = torch.zeros((2, 1024)) |
|
|
|
|
|
# Forward pass |
|
|
output = model(x_encoder, x_decoder, encoder_attention_mask, decoder_attention_mask) |
|
|
print(output.size()) # Output: [2, 1024, 1024] |
|
|
``` |
|
|
|
|
|
## Citation |
|
|
|
|
|
```bibtex |
|
|
@article{zhao2025automatic, |
|
|
title={Automatic Stage Lighting Control: Is it a Rule-Driven Process or Generative Task?}, |
|
|
author={Zhao, Zijian and Jin, Dian and Zhou, Zijing and Zhang, Xiaoyu}, |
|
|
journal={arXiv preprint arXiv:2506.01482}, |
|
|
year={2025} |
|
|
} |
|
|
``` |
|
|
|
|
|
## Contact |
|
|
|
|
|
Zijian Zhao: zzhaock@connect.ust.hk |
