Update README.md

README.md

@@ -23,7 +23,7 @@ library_name: transformers
 
 # Spam Message Classifier
 
-A state-of-the-art spam message classification model built on **RoBERTa-base** transformer architecture, achieving **99.94% accuracy** and **0.9982 F1-score** for spam detection with **perfect precision (100%)** for the test set. Developed as the core spam detection component for **Amy**, an intelligent Discord moderation bot.
+A state-of-the-art spam message classification model built on **RoBERTa-base** transformer architecture, achieving **99.42% accuracy** and **0.9782 F1-score for spam class** for the test set. Developed as the core spam detection component for **Amy**, an intelligent Discord moderation bot.
 
 ## Model Description
 
@@ -39,7 +39,7 @@ This model is a fine-tuned version of FacebookAI/roberta-base for binary spam cl
 ## Key Features
 
 - **🤖 Transformer-based Architecture**: Built on RoBERTa-base for superior text understanding
-- **⚡ High Performance**: 0.9982 F1-score for spam detection, 99.94% overall accuracy
+- **⚡ High Performance**: 0.9782 F1-score for spam detection, 99.42% overall accuracy
 - **🔧 Hyperparameter Optimization**: Automated tuning using Optuna framework (25 trials)
 - **⚖️ Class Imbalance Handling**: Successfully addressed through weighted loss function
 - **🔗 URL Bias Mitigation**: Enhanced with real-world ham messages containing links
@@ -142,27 +142,14 @@ This spam classifier is ideal for:
 The model was trained on a combination of two comprehensive SMS spam datasets totaling **11,498 messages**:
 
 1. **[SMS Spam Collection Dataset](https://www.kaggle.com/datasets/uciml/sms-spam-collection-dataset)** - UCI Machine Learning Repository
-2. **[SMS Phishing Dataset](https://data.mendeley.com/datasets/f45bkkt8pr/1)** - Sandhya Mishra & Devpriya Soni (2022)
-
-**Dataset Statistics**:
-- Total Messages: 11,498
-- Ham Messages: 9,669 (84.1%)
-- Spam Messages: 1,829 (15.9%)
-- Average Message Length: ~80 characters
-- Language: English
-
-**Dataset Split**:
-- Training Set: 70% (~8,048 messages)
-- Validation Set: 15% (~1,725 messages) - for hyperparameter tuning
-- Test Set: 15% (~1,725 messages) - completely unseen data for final evaluation
-
-**Preprocessing**:
-1. Label encoding (ham → 0, spam/smishing → 1)
+2. **Discord Text Messages** — a manually collected dataset of real Discord messages containing both ham and spam samples. (This dataset was created to mitigate `<URL>` bias.)
+
+**Preprocessing Steps**:
+1. Label encoding (ham → 0, spam → 1)
 2. Text cleaning and normalization with Discord-specific preprocessing
-3. Dataset merging and deduplication
-4. Train/validation/test split (70/15/15)
-5. Tokenization with RoBERTa tokenizer
-6. Dynamic padding and truncation (max length = 128)
+3. Train/validation/test split (70/15/15)
+4. Tokenization with RoBERTa tokenizer
+5. Dynamic padding and truncation
 
 ## Training Procedure
 
@@ -179,15 +166,15 @@ Automated hyperparameter search using **Optuna framework** (25 trials):
 - Training epochs: 2 to 5 epochs
 - Warmup ratio: 0.05 to 0.1 for learning rate scheduling
 
-**Best Parameters Found (Trial 17/25)**:
-- hidden_dropout: 0.161
-- attention_dropout: 0.116
-- learning_rate: 1.67e-05
-- weight_decay: 0.0235
-- batch_size: 16
-- gradient_accumulation_steps: 3
-- epochs: 4
-- warmup_ratio: 0.0502
+**Best Parameters Found (Trial 6/25)**:
+- Hidden dropout: 0.10069482002001506
+- Attention dropout: 0.12460257350587067
+- Learning rate: 4.976184540342024e-05
+- Weight decay: 0.04490021845024478
+- Batch size: 16
+- Gradient accumulation steps: 4
+- Epochs: 4
+- Warmup ratio: 0.07622459860163384
 
 ### Training Strategy
 
@@ -210,33 +197,31 @@ Automated hyperparameter search using **Optuna framework** (25 trials):
 
 | Metric | Score |
 |--------|-------|
-| **Overall Accuracy** | **99.94%** |
-| **Weighted F1-Score** | **0.9994** |
-| **Spam F1-Score** | **0.9982** ✅ |
-| **Spam Precision** | **100.00%** (Perfect) |
-| **Spam Recall** | **99.64%** |
-| **Ham Precision** | **99.93%** |
-| **Ham Recall** | **100.00%** |
+| **Overall Accuracy** | **99.41%** |
+| **Weighted F1-Score** | **0.9941** |
+| **Spam F1-Score** | **0.9782** |
+| **Spam Precision** | **96.55%** |
+| **Spam Recall** | **99.12%** |
+| **Ham Precision** | **99.86%** |
+| **Ham Recall** | **99.45%** |
 
 ### Confusion Matrix
 
 |               | Predicted Ham | Predicted Spam |
 |---------------|---------------|----------------|
-| **Actual Ham**    |     1,456     |       0        |
-| **Actual Spam**   |       1       |      274       |
+| **Actual Ham**    |     725     |       4        |
+| **Actual Spam**   |       1       |      112       |
 
 ### Performance Analysis
 
-- **True Positives**: 274 spam messages correctly identified
-- **True Negatives**: 1,456 ham messages correctly identified
-- **False Positives**: 0 (Perfect - no legitimate messages flagged)
-- **False Negatives**: 1 (Only 1 spam message missed)
-- **False Positive Rate**: 0.00%
-- **Miss Rate**: 0.36%
+- **True Positives**: 112 spam messages correctly identified
+- **True Negatives**: 725 ham messages correctly identified
+- **False Positives**: 4
+- **False Negatives**: 1
 
 ### Generalizability
 
-> 📊 **Strong Generalization**: All performance metrics are evaluated on a **completely unseen test set** (15% of data, ~1,725 messages) that was never used during training or hyperparameter tuning, ensuring robust real-world performance and preventing overfitting.
+> 📊 **Strong Generalization**: All performance metrics are evaluated on a **completely unseen test set** (15% of data) that was never used during training or hyperparameter tuning, ensuring robust real-world performance and preventing overfitting.
 
 ## Challenges Addressed & Solutions
 
@@ -248,7 +233,7 @@ Automated hyperparameter search using **Optuna framework** (25 trials):
 
 ### ✅ Class Imbalance Handling (SUCCESSFULLY ADDRESSED)
 
-**Challenge**: The combined dataset exhibits natural imbalance (84.1% ham, 15.9% spam).
+**Challenge**: The combined dataset exhibits natural imbalance.
 
 **Solution**: Implemented weighted loss function during training to handle the imbalanced dataset effectively, resulting in exceptional performance for both classes.
 
@@ -256,7 +241,7 @@ Automated hyperparameter search using **Optuna framework** (25 trials):
 
 **Challenge**: Ensuring model generalizes well to unseen data.
 
-**Solution**: Comprehensive evaluation on completely held-out test set (15% of data) never used during training or hyperparameter tuning, with demonstrated strong generalization (99.94% accuracy on unseen data).
+**Solution**: Comprehensive evaluation on completely held-out test set (15% of data) never used during training or hyperparameter tuning, with demonstrated strong generalization (99.42% accuracy on unseen data).
 
 ## Limitations