Upload MOJO
Browse files
mojo.py
CHANGED
|
@@ -16,7 +16,7 @@ class RotaryEmbeddingConfig:
|
|
| 16 |
Parameters to initialize the RotaryEmbedding layer. The rescaling factor allows
|
| 17 |
to adapt the rotary embeddings to larger lengths than what was used for training.
|
| 18 |
One of this strategy is presented in the Yarn paper: https://arxiv.org/pdf/2309.00071.pdf. # noqa
|
| 19 |
-
Args:
|
| 20 |
"""
|
| 21 |
|
| 22 |
rescaling_factor: Optional[float]
|
|
|
|
| 16 |
Parameters to initialize the RotaryEmbedding layer. The rescaling factor allows
|
| 17 |
to adapt the rotary embeddings to larger lengths than what was used for training.
|
| 18 |
One of this strategy is presented in the Yarn paper: https://arxiv.org/pdf/2309.00071.pdf. # noqa
|
| 19 |
+
Args:b
|
| 20 |
"""
|
| 21 |
|
| 22 |
rescaling_factor: Optional[float]
|