NEXT_STEPS_IMPROVEMENTS.md

# ToGMAL Next Steps: Adaptive Scoring & Nested CV

## Updated: 2025-10-21

This document outlines the immediate next steps to improve ToGMAL's difficulty assessment accuracy and establish a rigorous evaluation framework.

---

## 🎯 Immediate Goals (This Week)

### 1. **Implement Adaptive Uncertainty-Aware Scoring**
   - **Problem**: Current naive weighted average fails on low-similarity matches
   - **Example Failure**: "Prove universe is 10,000 years old" → matched to factual recall (similarity ~0.57) → incorrectly rated LOW risk
   - **Solution**: Add uncertainty penalties when:
     - Max similarity < 0.7 (weak best match)
     - High variance in k-NN similarities (diverse, unreliable matches)
     - Low average similarity (all matches are weak)
   - **File to modify**: `benchmark_vector_db.py::query_similar_questions()`
   - **Expected improvement**: 5-15% AUROC gain on low-similarity cases

### 2. **Export Database for Evaluation**
   - Add `get_all_questions_as_dataframe()` method to export 32K questions
   - Prepare for train/val/test splitting and nested CV
   - **File to modify**: `benchmark_vector_db.py`

### 3. **Test Adaptive Scoring**
   - Create test script with edge cases
   - Compare baseline vs. adaptive on known failure modes
   - **New file**: `test_adaptive_scoring.py`

---

## 📊 Evaluation Framework (Next 2-3 Weeks)

### Why Nested Cross-Validation?

**Problem with simple train/val/test split:**
- Single validation set can be lucky/unlucky (unrepresentative)
- Repeated "peeking" at validation during hyperparameter search causes data leakage
- Test set gives only ONE performance estimate (high variance)

**Nested CV advantages:**
- **Outer loop**: 5-fold CV for unbiased generalization estimate
- **Inner loop**: 3-fold grid search for hyperparameter tuning
- **No leakage**: Test folds never seen during tuning
- **Robust**: Multiple performance estimates across 5 different test sets

### Hyperparameters to Tune

```python
param_grid = {
    'k_neighbors': [3, 5, 7, 10],
    'similarity_threshold': [0.6, 0.7, 0.8],
    'low_sim_penalty': [0.3, 0.5, 0.7],
    'variance_penalty': [1.0, 2.0, 3.0],
    'low_avg_penalty': [0.2, 0.4, 0.6]
}
```

### Evaluation Metrics

1. **AUROC** (primary): Discriminative ability (0.5=random, 1.0=perfect)
2. **FPR@TPR95**: False positive rate when catching 95% of risky prompts
3. **AUPR**: Area under precision-recall curve (good for imbalanced data)
4. **Expected Calibration Error (ECE)**: Are predicted probabilities accurate?
5. **Brier Score**: Overall probabilistic prediction accuracy

---

## 🗂️ Implementation Phases

### Phase 1: Adaptive Scoring (This Week)
- [x] ✓ 32K vector database with 20 domains, 7 benchmark sources
- [ ] Add `_compute_adaptive_difficulty()` method
- [ ] Integrate uncertainty penalties into scoring
- [ ] Test on known failure cases
- [ ] Update `togmal_mcp.py` to use adaptive scoring

### Phase 2: Data Export & Baseline (Week 2)
- [ ] Add `get_all_questions_as_dataframe()` export method
- [ ] Create simple 70/15/15 train/val/test split
- [ ] Run current ToGMAL (baseline) on test set
- [ ] Compute baseline metrics:
  - AUROC
  - FPR@TPR95
  - Expected Calibration Error
  - Brier Score
- [ ] Document failure modes (low similarity, cross-domain, etc.)

### Phase 3: Nested CV Implementation (Week 3)
- [ ] Implement `NestedCVEvaluator` class
- [ ] Outer CV: 5-fold stratified by (domain × difficulty)
- [ ] Inner CV: 3-fold grid search over hyperparameters
- [ ] Temporary vector DB creation per fold
- [ ] Metrics computation on each outer fold

### Phase 4: Hyperparameter Tuning (Week 4)
- [ ] Run full nested CV (5 outer × 3 inner = 15 train-test runs)
- [ ] Collect best hyperparameters per fold
- [ ] Identify most common optimal parameters
- [ ] Compute mean ± std generalization performance
- [ ] Compare to baseline

### Phase 5: Final Model & Deployment (Week 5)
- [ ] Train final model on ALL 32K questions with best hyperparameters
- [ ] Re-index full vector database
- [ ] Deploy to MCP server and HTTP facade
- [ ] Test with Claude Desktop

### Phase 6: OOD Testing (Week 6)
- [ ] Create OOD test sets:
  - **Adversarial**: "Prove false premises", jailbreaks
  - **Domain Shift**: Creative writing, coding, real user queries
  - **Temporal**: New benchmarks (2024+)
- [ ] Evaluate on each OOD set
- [ ] Analyze performance degradation vs. in-distribution

### Phase 7: Iteration & Documentation (Week 7)
- [ ] Analyze failures on OOD sets
- [ ] Add new heuristics for missed patterns
- [ ] Re-run nested CV with updated features
- [ ] Generate calibration plots (reliability diagrams)
- [ ] Write technical report

---

## 📈 Expected Improvements

Based on OOD detection literature and nested CV best practices:

1. **Adaptive scoring**: +5-15% AUROC on low-similarity cases
   - Baseline: ~0.75 AUROC (naive weighted average)
   - Target: ~0.85+ AUROC (adaptive with uncertainty)

2. **Nested CV**: Honest, robust performance estimates
   - Simple split: Single point estimate (could be lucky/unlucky)
   - Nested CV: Mean ± std across 5 folds

3. **Domain calibration**: -10-20% false positives
   - Expected: FPR@TPR95 drops from ~0.25 to ~0.15

4. **Multi-signal fusion**: Better edge case detection
   - Combine vector similarity + rule-based heuristics
   - Improved recall on adversarial examples

5. **Calibration**: ECE < 0.05
   - Better alignment between predicted risk and actual difficulty

---

## ✅ Validation Checklist (Before Production Deploy)

- [ ] Nested CV completed with no data leakage
- [ ] Hyperparameters tuned on inner CV folds only
- [ ] Generalization performance estimated on outer CV folds
- [ ] OOD sets tested (adversarial, domain-shift, temporal)
- [ ] Calibration error within acceptable range (ECE < 0.1)
- [ ] Failure modes documented with specific examples
- [ ] Ablation studies show each component contributes
- [ ] Performance: adaptive > baseline on all metrics
- [ ] Real-world testing with user queries

---

## 🚀 Quick Start Command

See `togmal_improvement_plan.md` for full implementation details including:
- Complete code for `NestedCVEvaluator` class
- Adaptive scoring implementation
- All evaluation metrics with examples
- Detailed roadmap with weekly milestones

**Next Action**: Implement adaptive scoring in `benchmark_vector_db.py` and test with edge cases.