Model Summary
T5Gemma is a family of lightweight, open-weight language models built from the same research and technology used to create the Gemma models. Based on the T5 text-to-text architecture, T5Gemma is a decoder-only model designed for a wide range of natural language processing (NLP) tasks. It's trained on a massive corpus of text and code, enabling it to excel at tasks like summarization, question answering, and text generation. T5Gemma offers a balance of high performance and computational efficiency, making it suitable for both research and deployment in various applications.
Key Features:
- Decoder-Only Architecture: A modern, efficient design based on the T5 framework.
- Open Weights: Freely available for research, and commercial use, promoting accessibility and innovation.
- Strong Performance: Achieves state-of-the-art results on NLP benchmarks for its size.
- Multiple Sizes: Comes in various sizes (e.g., 2B and 7B parameters) to fit different hardware capabilities.
- Text-to-Text Framework: Trained to transform an input text sequence into a target text sequence, making it versatile for many tasks.
Training Strategies:
T5Gemma is pre-trained on a large and diverse dataset of up to 6 trillion tokens of text and code from web documents, mathematics, and code. The training process uses a causal masking scheme tailored for decoder-only models, enabling it to learn complex patterns and generate coherent, contextually relevant text.
Weights are released under the Gemma Terms of Use. Keras model code is released under the Apache 2 License.
Links
- T5Gemma Quickstart Notebook
- [T5Gemma API Documentation](Coming Soon!!)
- T5Gemma Model Card
- KerasHub Beginner Guide
- KerasHub Model Publishing Guide
Installation
Keras and KerasHub can be installed with:
pip install -U -q keras-hub
pip install -U -q keras
Jax, TensorFlow, and Torch come preinstalled in Kaggle Notebooks. For instructions on installing them in another environment see the Keras Getting Started page.
Available T5Gemma Presets
The following model checkpoints are provided by the Keras team. Full code examples for each are available below.
| Preset | Parameters | Description |
|---|---|---|
| t5gemma_s_s_ul2 | 312.5M | T5Gemma S/S model with a small encoder and small decoder, adapted as a UL2 model. |
| t5gemma_s_s_prefixlm | 312.5M | T5Gemma S/S model with a small encoder and small decoder, adapted as a prefix language model. |
| t5gemma_s_s_ul2_it | 312.5M | T5Gemma S/S model with a small encoder and small decoder, adapted as a UL2 model and fine-tuned for instruction following. |
| t5gemma_s_s_prefixlm_it | 312.5M | T5Gemma S/S model with a small encoder and small decoder, adapted as a prefix language model and fine-tuned for instruction following. |
| t5gemma_b_b_ul2 | 591.5M | T5Gemma B/B model with a base encoder and base decoder, adapted as a UL2 model. |
| t5gemma_b_b_prefixlm | 591.5M | T5Gemma B/B model with a base encoder and base decoder, adapted as a prefix language model. |
| t5gemma_b_b_ul2_it | 591.5M | T5Gemma B/B model with a base encoder and base decoder, adapted as a UL2 model and fine-tuned for instruction following. |
| t5gemma_b_b_prefixlm_it | 591.5M | T5Gemma B/B model with a base encoder and base decoder, adapted as a prefix language model and fine-tuned for instruction following. |
| t5gemma_l_l_ul2 | 1.24B | T5Gemma L/L model with a large encoder and large decoder, adapted as a UL2 model. |
| t5gemma_l_l_prefixlm | 1.24B | T5Gemma L/L model with a large encoder and large decoder, adapted as a prefix language model. |
| t5gemma_l_l_ul2_it | 1.24B | T5Gemma L/L model with a large encoder and large decoder, adapted as a UL2 model and fine-tuned for instruction following. |
| t5gemma_l_l_prefixlm_it | 1.24B | T5Gemma L/L model with a large encoder and large decoder, adapted as a prefix language model and fine-tuned for instruction following. |
| t5gemma_ml_ml_ul2 | 2.20B | T5Gemma ML/ML model with a medium-large encoder and medium-large decoder, adapted as a UL2 model. |
| t5gemma_ml_ml_prefixlm | 2.20B | T5Gemma ML/ML model with a medium-large encoder and medium-large decoder, adapted as a prefix language model. |
| t5gemma_ml_ml_ul2_it | 2.20B | T5Gemma ML/ML model with a medium-large encoder and medium-large decoder, adapted as a UL2 model and fine-tuned for instruction following. |
| t5gemma_ml_ml_prefixlm_it | 2.20B | T5Gemma ML/ML model with a medium-large encoder and medium-large decoder, adapted as a prefix language model and fine-tuned for instruction following. |
| t5gemma_xl_xl_ul2 | 3.77B | T5Gemma XL/XL model with an extra-large encoder and extra-large decoder, adapted as a UL2 model. |
| t5gemma_xl_xl_prefixlm | 3.77B | T5Gemma XL/XL model with an extra-large encoder and extra-large decoder, adapted as a prefix language model. |
| t5gemma_xl_xl_ul2_it | 3.77B | T5Gemma XL/XL model with an extra-large encoder and extra-large decoder, adapted as a UL2 model and fine-tuned for instruction following. |
| t5gemma_xl_xl_prefixlm_it | 3.77B | T5Gemma XL/XL model with an extra-large encoder and extra-large decoder, adapted as a prefix language model and fine-tuned for instruction following. |
| t5gemma_2b_2b_ul2 | 5.60B | T5Gemma 2B/2B model with a 2-billion-parameter encoder and 2-billion-parameter decoder, adapted as a UL2 model. |
| t5gemma_2b_2b_prefixlm | 5.60B | T5Gemma 2B/2B model with a 2-billion-parameter encoder and 2-billion-parameter decoder, adapted as a prefix language model. |
| t5gemma_2b_2b_ul2_it | 5.60B | T5Gemma 2B/2B model with a 2-billion-parameter encoder and 2-billion-parameter decoder, adapted as a UL2 model and fine-tuned for instruction following. |
| t5gemma_2b_2b_prefixlm_it | 5.60B | T5Gemma 2B/2B model with a 2-billion-parameter encoder and 2-billion-parameter decoder, adapted as a prefix language model and fine-tuned for instruction following. |
| t5gemma_9b_2b_ul2 | 12.29B | T5Gemma 9B/2B model with a 9-billion-parameter encoder and 2-billion-parameter decoder, adapted as a UL2 model. |
| t5gemma_9b_2b_prefixlm | 12.29B | T5Gemma 9B/2B model with a 9-billion-parameter encoder and 2-billion-parameter decoder, adapted as a prefix language model. |
| t5gemma_9b_2b_ul2_it | 12.29B | T5Gemma 9B/2B model with a 9-billion-parameter encoder and 2-billion-parameter decoder, adapted as a UL2 model and fine-tuned for instruction following. |
| t5gemma_9b_2b_prefixlm_it | 12.29B | T5Gemma 9B/2B model with a 9-billion-parameter encoder and 2-billion-parameter decoder, adapted as a prefix language model and fine-tuned for instruction following. |
| t5gemma_9b_9b_ul2 | 20.33B | T5Gemma 9B/9B model with a 9-billion-parameter encoder and 9-billion-parameter decoder, adapted as a UL2 model. |
| t5gemma_9b_9b_prefixlm | 20.33B | T5Gemma 9B/9B model with a 9-billion-parameter encoder and 9-billion-parameter decoder, adapted as a prefix language model. |
| t5gemma_9b_9b_ul2_it | 20.33B | T5Gemma 9B/9B model with a 9-billion-parameter encoder and 9-billion-parameter decoder, adapted as a UL2 model and fine-tuned for instruction following. |
| t5gemma_9b_9b_prefixlm_it | 20.33B | T5Gemma 9B/9B model with a 9-billion-parameter encoder and 9-billion-parameter decoder, adapted as a prefix language model and fine-tuned for instruction following. |
- Downloads last month
- 454