|
|
--- |
|
|
language: en |
|
|
license: mit |
|
|
tags: |
|
|
- mistral |
|
|
- lora |
|
|
- peft |
|
|
- transcript-chunking |
|
|
- text-segmentation |
|
|
- topic-detection |
|
|
- transformers |
|
|
model_type: mistral |
|
|
base_model: mistralai/Mistral-7B-v0.2 |
|
|
datasets: |
|
|
- custom-transcript-chunking |
|
|
metrics: |
|
|
- loss |
|
|
- accuracy |
|
|
--- |
|
|
|
|
|
# 🧠 Mistral LoRA Transcript Chunking Model |
|
|
|
|
|
## Model Overview |
|
|
This LoRA adapter was trained on a custom dataset of **1,000 English transcript examples** to teach a **Mistral-7B-v0.2** model how to segment long transcripts into topic-based chunks using 'section #:' as delimiters. |
|
|
It enables automated **topic boundary detection** in conversation, meeting, and podcast transcripts — ideal for preprocessing before summarization, classification, or retrieval. |
|
|
|
|
|
--- |
|
|
|
|
|
## 🧩 Training Objective |
|
|
The model learns to: |
|
|
|
|
|
- Detect topic changes in unstructured transcripts |
|
|
- Insert `--` where those shifts occur |
|
|
- Preserve the original flow of speech |
|
|
|
|
|
**Example:** |
|
|
|
|
|
--- |
|
|
|
|
|
## ⚙️ Training Configuration |
|
|
- **Base Model:** `mistralai/Mistral-7B-v0.2` |
|
|
- **Adapter Type:** LoRA |
|
|
- **PEFT Library:** `peft==0.10.0` |
|
|
- **Training Framework:** Hugging Face Transformers |
|
|
- **Epochs:** 2 |
|
|
- **Optimizer:** AdamW |
|
|
- **Learning Rate:** 2e-4 |
|
|
- **Batch Size:** 8 |
|
|
- **Sequence Length:** 512 |
|
|
|
|
|
--- |
|
|
|
|
|
## 📊 Training Metrics |
|
|
|
|
|
| Step | Training Loss | Validation Loss | Entropy | Num Tokens | Mean Token Accuracy | |
|
|
|------|----------------|----------------|----------|-------------|---------------------| |
|
|
| 100 | 0.2961 | 0.1603 | 0.1644 | 204,800 | 0.9594 | |
|
|
| 200 | 0.1362 | 0.1502 | 0.1609 | 409,600 | 0.9603 | |
|
|
| 300 | 0.1360 | 0.1451 | 0.1391 | 612,864 | 0.9572 | |
|
|
| 400 | 0.0951 | 0.1351 | 0.1279 | 817,664 | 0.9635 | |
|
|
| 500 | 0.0947 | 0.1297 | 0.0892 | 1,022,464 | 0.9657 | |
|
|
|
|
|
**Summary:** |
|
|
Loss steadily decreased during training, and accuracy remained consistently above **95%**, indicating the model effectively learned transcript reconstruction and accurate delimiter placement. |
|
|
|
|
|
--- |
|
|
|
|
|
## 🧰 Usage Example |
|
|
```python |
|
|
from transformers import AutoTokenizer, AutoModelForCausalLM |
|
|
from peft import PeftModel |
|
|
|
|
|
base = "mistralai/Mistral-7B-Instruct-v0.2" |
|
|
adapter = "Dc-4nderson/transcript_summarizer_model" |
|
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained(base) |
|
|
model = AutoModelForCausalLM.from_pretrained(base) |
|
|
model = PeftModel.from_pretrained(model, adapter) |
|
|
|
|
|
text = ( |
|
|
"Break this transcript wherever a new topic begins. Use 'section #:' as a delimiter.\n" |
|
|
"Transcript: Let's start with last week's performance metrics. " |
|
|
"Next, we’ll review upcoming campaign deadlines." |
|
|
) |
|
|
|
|
|
inputs = tokenizer(text, return_tensors="pt") |
|
|
outputs = model.generate(**inputs, max_new_tokens=30000) |
|
|
print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
|
|
|
|
|
``` |
|
|
🧾 License |
|
|
|
|
|
Released under the MIT License — free for research and commercial use with attribution. |
|
|
|
|
|
🙌 Credits |
|
|
|
|
|
Developed by Dequan Anderson for automated transcript segmentation and chunked text preprocessing tasks. |
|
|
Built using Hugging Face Transformers, PEFT, and Mistral 7B for efficient LoRA fine-tuning. |
