Before Run the 1st Evalation: Add Precision & MRR Chart Generator and a sample test query

- Implemented a new Python script `precision_mrr_chart_generator.py` for generating comprehensive charts for Precision@K and Mean Reciprocal Rank (MRR) analysis from JSON results.
- The script includes functionalities for creating various visualizations such as comparison charts, heatmaps, and detailed statistics tables.
- Added a sample test query in `single_test_query.txt` for evaluation purposes.

evaluation/latency_evaluator.py

@@ -1,21 +1,48 @@
 #!/usr/bin/env python3
 """
-OnCall.ai System - Comprehensive Evaluator (Metrics 1-6)
+OnCall.ai System - Comprehensive Evaluator (Metrics 1-8)
 ========================================================
 
-Single execution to collect all metrics 1-6 data from app.py pipeline:
+Single execution to collect all metrics 1-4 data from app.py pipeline.
+Generates foundation data for metrics 5-8 evaluation in downstream processors.
 
-RETRIEVAL METRICS (Only available for RAG systems):
-1. Total Latency (總處理時長) - Complete pipeline timing
-2. Condition Extraction Success Rate (條件抽取成功率) - user_prompt.py success  
-3. Retrieval Relevance (檢索相關性) - cosine similarity from retrieval.py
-4. Retrieval Coverage (檢索覆蓋率) - advice utilization of retrieved content
+COMPLETE METRICS OVERVIEW:
 
-LLM EVALUATION METRICS (Available for all systems):
-5. Clinical Actionability (臨床可操作性) - Third-party LLM evaluation  
-6. Clinical Evidence Quality (臨床證據品質) - Third-party LLM evaluation
+PIPELINE PERFORMANCE METRICS (Collected by this evaluator):
+1. Total Latency (總處理時長) - Complete pipeline processing time from query to response
+2. Condition Extraction Success Rate (條件抽取成功率) - Success rate of user_prompt.py condition extraction
+3. Retrieval Relevance (檢索相關性) - Average cosine similarity scores from retrieval.py results
+4. Retrieval Coverage (檢索覆蓋率) - Medical keyword utilization rate between retrieved content and generated advice
 
-Note: This evaluator focuses on metrics 1-4. Metrics 5-6 require separate LLM evaluation.
+LLM JUDGE METRICS (Processed by metric5_6_llm_judge_evaluator.py):
+5. Clinical Actionability (臨床可操作性) - Third-party LLM evaluation of medical advice actionability (1-10 scale)
+   * Uses batch evaluation strategy with Llama3-70B as judge
+   * Measures: Can healthcare providers immediately act on this advice?
+   * Target threshold: ≥7.0/10 for acceptable actionability
+   
+6. Clinical Evidence Quality (臨床證據品質) - Third-party LLM evaluation of evidence-based quality (1-10 scale)
+   * Uses same batch evaluation call as metric 5 for efficiency
+   * Measures: Is the advice evidence-based and follows medical standards?
+   * Target threshold: ≥7.5/10 for acceptable evidence quality
+
+RETRIEVAL PRECISION METRICS (Processed by metric7_8_precision_MRR.py):
+7. Precision@K (檢索精確率) - Proportion of relevant results in top-K retrieval results
+   * Uses adaptive threshold based on query complexity (0.15 for complex, 0.25 for simple queries)
+   * Query complexity determined by unique emergency keywords count (≥4 = complex)
+   * Measures: relevant_results / total_retrieved_results
+   
+8. Mean Reciprocal Rank (平均倒數排名) - Average reciprocal rank of first relevant result
+   * Uses same adaptive threshold as Precision@K
+   * Measures: 1 / rank_of_first_relevant_result (0 if no relevant results)
+   * Higher MRR indicates relevant results appear earlier in ranking
+
+DATA FLOW ARCHITECTURE:
+1. latency_evaluator.py → comprehensive_details_*.json (metrics 1-4 + pipeline data)
+2. latency_evaluator.py → medical_outputs_*.json (medical advice for judge evaluation)
+3. metric5_6_llm_judge_evaluator.py → judge_evaluation_*.json (metrics 5-6)
+4. metric7_8_precision_MRR.py → precision_mrr_analysis_*.json (metrics 7-8)
+
+Note: This evaluator focuses on metrics 1-4 collection. Metrics 5-8 require separate downstream evaluation.
 
 Author: YanBo Chen  
 Date: 2025-08-04
@@ -320,6 +347,31 @@ class ComprehensiveEvaluator:
                 "timestamp": datetime.now().isoformat()
             }
             
+            # Validate data completeness for metrics 7-8 analysis
+            ready = True
+            data = comprehensive_result.get('pipeline_data', {})
+            
+            # 1. Check retrieval results completeness for precision/MRR calculation
+            retr = data.get('retrieval_results', {}).get('processed_results', [])
+            if not retr or 'distance' not in retr[0]:
+                ready = False
+            
+            # 2. Check condition extraction completeness for complexity analysis
+            cond = data.get('condition_result', {}).get('condition')
+            if not cond:
+                ready = False
+            
+            # 3. Check overall execution status
+            if not comprehensive_result.get('overall_success', False):
+                ready = False
+            
+            # 4. Check retrieval timing data completeness
+            if 'retrieval_time' not in comprehensive_result.get('relevance_metrics', {}):
+                ready = False
+            
+            # Set metrics 7-8 readiness flag for downstream precision/MRR analysis
+            comprehensive_result['precision_mrr_ready'] = ready
+            
             # Store result
             self.comprehensive_results.append(comprehensive_result)
             
@@ -386,8 +438,9 @@ class ComprehensiveEvaluator:
             },
             
             # Note: Metrics 5-6 (Clinical Actionability & Evidence Quality) 
-            # are not collected here - they require separate LLM evaluation
-            # using the medical_outputs saved by this evaluator
+            # are collected by metric5_6_llm_judge_evaluator.py using medical_outputs
+            # Metrics 7-8 (Precision@K & MRR) are collected by metric7_8_precision_MRR.py 
+            # using comprehensive_details pipeline data
             
             "overall_success": False,
             "status": status,
@@ -395,6 +448,9 @@ class ComprehensiveEvaluator:
             "timestamp": datetime.now().isoformat()
         }
         
+        # For failed results, precision/MRR analysis data is not ready
+        failed_result['precision_mrr_ready'] = False
+        
         self.comprehensive_results.append(failed_result)
         return failed_result
     
@@ -741,8 +797,8 @@ if __name__ == "__main__":
     if len(sys.argv) > 1:
         query_file = sys.argv[1]
     else:
-        # Default to evaluation/pre_user_query_evaluate.txt
-        query_file = Path(__file__).parent / "pre_user_query_evaluate.txt"
+        # Default to evaluation/single_test_query.txt for initial testing
+        query_file = Path(__file__).parent / "single_test_query.txt"
     
     if not os.path.exists(query_file):
         print(f"❌ Query file not found: {query_file}")
@@ -829,7 +885,9 @@ if __name__ == "__main__":
         print(f"   📊 {metric_name.capitalize()}: {filepath}")
     print(f"   📝 Medical Outputs: {outputs_path}")
     print(f"   📋 Comprehensive Details: {details_path}")
-    print(f"\n💡 Next step: Run chart generators for individual metrics")
+    print(f"\n💡 Next step: Run downstream evaluators for metrics 5-8")
+    print(f"   python metric5_6_llm_judge_evaluator.py rag")
+    print(f"   python metric7_8_precision_MRR.py {details_path}")
     print(f"   python latency_chart_generator.py")
     print(f"   python extraction_chart_generator.py  # (create separately)")
     print(f"   python relevance_chart_generator.py   # (create separately)")

evaluation/metric7_8_precision_MRR.py

@@ -0,0 +1,391 @@
+#!/usr/bin/env python3
+"""
+OnCall.ai System - Precision & MRR Analyzer (Metrics 7-8)
+========================================================
+
+Specialized analyzer for calculating Precision@K and Mean Reciprocal Rank (MRR)
+using data collected from latency_evaluator.py comprehensive evaluation.
+
+METRICS CALCULATED:
+7. Precision@K (檢索精確率) - Proportion of relevant results in top-K retrieval
+8. Mean Reciprocal Rank (平均倒數排名) - Average reciprocal rank of first relevant result
+
+DESIGN PRINCIPLE:
+- Reuses comprehensive_details_*.json from latency_evaluator.py
+- Implements adaptive threshold based on query complexity
+- Query complexity determined by actual matched emergency keywords count
+- No additional LLM calls required
+
+Author: YanBo Chen  
+Date: 2025-08-04
+"""
+
+import json
+import os
+import sys
+from typing import Dict, List, Any, Set
+from datetime import datetime
+from pathlib import Path
+import re
+import statistics
+
+class PrecisionMRRAnalyzer:
+    """Specialized analyzer for metrics 7-8 using existing comprehensive evaluation data"""
+    
+    def __init__(self):
+        """Initialize analyzer"""
+        print("🔧 Initializing Precision & MRR Analyzer...")
+        self.analysis_results = []
+        print("✅ Analyzer initialization complete")
+    
+    def load_comprehensive_data(self, filepath: str) -> List[Dict]:
+        """
+        Load comprehensive evaluation data from latency_evaluator.py output
+        
+        Args:
+            filepath: Path to comprehensive_details_*.json file
+            
+        Returns:
+            List of comprehensive evaluation results
+        """
+        try:
+            with open(filepath, 'r', encoding='utf-8') as f:
+                data = json.load(f)
+            
+            comprehensive_results = data.get('comprehensive_results', [])
+            
+            print(f"📁 Loaded {len(comprehensive_results)} comprehensive evaluation results")
+            print(f"📊 Ready for precision/MRR analysis: {sum(1 for r in comprehensive_results if r.get('precision_mrr_ready'))}")
+            
+            return comprehensive_results
+            
+        except Exception as e:
+            print(f"❌ Failed to load comprehensive data: {e}")
+            return []
+    
+    def _is_complex_query(self, query: str, processed_results: List[Dict]) -> bool:
+        """
+        Determine query complexity based on actual matched emergency keywords
+        
+        Args:
+            query: Original query text
+            processed_results: Retrieval results with matched keywords
+            
+        Returns:
+            True if query is complex (should use lenient threshold)
+        """
+        # Collect unique emergency keywords actually found in retrieval results
+        unique_emergency_keywords = set()
+        
+        for result in processed_results:
+            if result.get('type') == 'emergency':
+                matched_keywords = result.get('matched', '')
+                if matched_keywords:
+                    keywords = [kw.strip() for kw in matched_keywords.split('|') if kw.strip()]
+                    unique_emergency_keywords.update(keywords)
+        
+        keyword_count = len(unique_emergency_keywords)
+        
+        # Business logic: 4+ different emergency keywords indicate complex case
+        is_complex = keyword_count >= 4
+        
+        print(f"   🧠 Query complexity: {'Complex' if is_complex else 'Simple'} ({keyword_count} emergency keywords)")
+        print(f"   🔑 Found keywords: {', '.join(list(unique_emergency_keywords)[:5])}")
+        
+        return is_complex
+    
+    def calculate_precision_mrr_single(self, query_data: Dict) -> Dict[str, Any]:
+        """
+        Calculate precision@K and MRR for single query
+        
+        Args:
+            query_data: Single query's comprehensive evaluation result
+            
+        Returns:
+            Precision and MRR metrics for this query
+        """
+        query = query_data['query']
+        category = query_data['category']
+        
+        # Extract processed results from pipeline data
+        pipeline_data = query_data.get('pipeline_data', {})
+        retrieval_results = pipeline_data.get('retrieval_results', {})
+        processed_results = retrieval_results.get('processed_results', [])
+        
+        print(f"🔍 Analyzing precision/MRR for: {query[:50]}...")
+        print(f"📋 Category: {category}, Results: {len(processed_results)}")
+        
+        if not processed_results:
+            return self._create_empty_precision_mrr_result(query, category)
+        
+        # Step 1: Determine query complexity
+        is_complex = self._is_complex_query(query, processed_results)
+        
+        # Step 2: Choose adaptive threshold
+        threshold = 0.15 if is_complex else 0.25
+        
+        print(f"   🎯 Using relevance threshold: {threshold} ({'lenient' if is_complex else 'strict'})")
+        
+        # Step 3: Calculate relevance scores (1 - distance)
+        relevance_scores = []
+        for result in processed_results:
+            distance = result.get('distance', 1.0)
+            relevance = 1.0 - distance
+            relevance_scores.append(relevance)
+        
+        # Step 4: Calculate Precision@K
+        relevant_count = sum(1 for score in relevance_scores if score >= threshold)
+        precision_at_k = relevant_count / len(processed_results)
+        
+        # Step 5: Calculate MRR
+        first_relevant_rank = None
+        for i, score in enumerate(relevance_scores, 1):
+            if score >= threshold:
+                first_relevant_rank = i
+                break
+        
+        mrr_score = (1.0 / first_relevant_rank) if first_relevant_rank else 0.0
+        
+        # Detailed analysis
+        result = {
+            "query": query,
+            "category": category,
+            "query_complexity": "complex" if is_complex else "simple",
+            "threshold_used": threshold,
+            
+            # Metric 7: Precision@K
+            "precision_at_k": precision_at_k,
+            "relevant_count": relevant_count,
+            "total_results": len(processed_results),
+            
+            # Metric 8: MRR
+            "mrr_score": mrr_score,
+            "first_relevant_rank": first_relevant_rank,
+            
+            # Supporting data
+            "relevance_scores": relevance_scores,
+            "avg_relevance": sum(relevance_scores) / len(relevance_scores),
+            "max_relevance": max(relevance_scores),
+            "min_relevance": min(relevance_scores),
+            
+            "timestamp": datetime.now().isoformat()
+        }
+        
+        print(f"   📊 Precision@{len(processed_results)}: {precision_at_k:.3f} ({relevant_count}/{len(processed_results)} relevant)")
+        print(f"   📊 MRR: {mrr_score:.3f} (first relevant at rank {first_relevant_rank})")
+        
+        return result
+    
+    def _create_empty_precision_mrr_result(self, query: str, category: str) -> Dict[str, Any]:
+        """Create empty result for failed queries"""
+        return {
+            "query": query,
+            "category": category,
+            "query_complexity": "unknown",
+            "threshold_used": 0.0,
+            "precision_at_k": 0.0,
+            "relevant_count": 0,
+            "total_results": 0,
+            "mrr_score": 0.0,
+            "first_relevant_rank": None,
+            "relevance_scores": [],
+            "timestamp": datetime.now().isoformat()
+        }
+    
+    def analyze_all_queries(self, comprehensive_results: List[Dict]) -> List[Dict]:
+        """
+        Analyze precision/MRR for all queries in comprehensive evaluation
+        
+        Args:
+            comprehensive_results: Results from latency_evaluator.py
+            
+        Returns:
+            List of precision/MRR analysis results
+        """
+        print(f"\n📊 Analyzing Precision@K and MRR for {len(comprehensive_results)} queries...")
+        
+        analysis_results = []
+        
+        for i, query_data in enumerate(comprehensive_results):
+            if not query_data.get('precision_mrr_ready'):
+                print(f"⏭️  Skipping query {i+1}: Not ready for precision/MRR analysis")
+                continue
+            
+            if not query_data.get('overall_success'):
+                print(f"⏭️  Skipping query {i+1}: Pipeline failed")
+                analysis_results.append(self._create_empty_precision_mrr_result(
+                    query_data['query'], 
+                    query_data['category']
+                ))
+                continue
+            
+            # Analyze this query
+            result = self.calculate_precision_mrr_single(query_data)
+            analysis_results.append(result)
+            
+            print("")  # Spacing between queries
+        
+        self.analysis_results = analysis_results
+        return analysis_results
+    
+    def calculate_statistics(self) -> Dict[str, Any]:
+        """Calculate comprehensive statistics for metrics 7-8"""
+        
+        if not self.analysis_results:
+            return {"error": "No analysis results available"}
+        
+        # Separate by complexity and category
+        stats = {
+            "overall_statistics": {},
+            "by_complexity": {"simple": {}, "complex": {}},
+            "by_category": {"diagnosis": {}, "treatment": {}, "mixed": {}},
+            "timestamp": datetime.now().isoformat()
+        }
+        
+        # Overall statistics
+        all_precision = [r['precision_at_k'] for r in self.analysis_results]
+        all_mrr = [r['mrr_score'] for r in self.analysis_results]
+        
+        stats["overall_statistics"] = {
+            "total_queries": len(self.analysis_results),
+            "avg_precision": statistics.mean(all_precision),
+            "avg_mrr": statistics.mean(all_mrr),
+            "precision_std": statistics.stdev(all_precision) if len(all_precision) > 1 else 0.0,
+            "mrr_std": statistics.stdev(all_mrr) if len(all_mrr) > 1 else 0.0
+        }
+        
+        # By complexity
+        for complexity in ["simple", "complex"]:
+            complexity_results = [r for r in self.analysis_results if r['query_complexity'] == complexity]
+            if complexity_results:
+                precision_scores = [r['precision_at_k'] for r in complexity_results]
+                mrr_scores = [r['mrr_score'] for r in complexity_results]
+                
+                stats["by_complexity"][complexity] = {
+                    "query_count": len(complexity_results),
+                    "avg_precision": statistics.mean(precision_scores),
+                    "avg_mrr": statistics.mean(mrr_scores),
+                    "avg_threshold": statistics.mean([r['threshold_used'] for r in complexity_results])
+                }
+        
+        # By category
+        for category in ["diagnosis", "treatment", "mixed"]:
+            category_results = [r for r in self.analysis_results if r['category'] == category]
+            if category_results:
+                precision_scores = [r['precision_at_k'] for r in category_results]
+                mrr_scores = [r['mrr_score'] for r in category_results]
+                
+                stats["by_category"][category] = {
+                    "query_count": len(category_results),
+                    "avg_precision": statistics.mean(precision_scores),
+                    "avg_mrr": statistics.mean(mrr_scores)
+                }
+        
+        return stats
+    
+    def save_results(self, filename: str = None) -> str:
+        """Save precision/MRR analysis results"""
+        if filename is None:
+            timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
+            filename = f"precision_mrr_analysis_{timestamp}.json"
+        
+        # Ensure results directory exists
+        results_dir = Path(__file__).parent / "results"
+        results_dir.mkdir(exist_ok=True)
+        
+        filepath = results_dir / filename
+        
+        # Create output data
+        output_data = {
+            "analysis_metadata": {
+                "total_queries": len(self.analysis_results),
+                "analysis_type": "precision_mrr_metrics_7_8",
+                "timestamp": datetime.now().isoformat(),
+                "adaptive_threshold": True
+            },
+            "detailed_results": self.analysis_results,
+            "statistics": self.calculate_statistics()
+        }
+        
+        with open(filepath, 'w', encoding='utf-8') as f:
+            json.dump(output_data, f, indent=2, ensure_ascii=False)
+        
+        print(f"📊 Precision/MRR analysis saved to: {filepath}")
+        return str(filepath)
+
+
+# Independent execution interface
+if __name__ == "__main__":
+    """Independent precision/MRR analysis interface"""
+    
+    print("📊 OnCall.ai Precision & MRR Analyzer - Metrics 7-8")
+    
+    if len(sys.argv) > 1:
+        comprehensive_file = sys.argv[1]
+    else:
+        # Look for latest comprehensive_details file
+        results_dir = Path(__file__).parent / "results"
+        if results_dir.exists():
+            comprehensive_files = list(results_dir.glob("comprehensive_details_*.json"))
+            if comprehensive_files:
+                comprehensive_file = str(sorted(comprehensive_files)[-1])  # Latest file
+                print(f"📁 Using latest comprehensive file: {comprehensive_file}")
+            else:
+                print("❌ No comprehensive_details_*.json files found")
+                print("Please run latency_evaluator.py first to generate comprehensive data")
+                sys.exit(1)
+        else:
+            print("❌ Results directory not found")
+            sys.exit(1)
+    
+    if not os.path.exists(comprehensive_file):
+        print(f"❌ Comprehensive file not found: {comprehensive_file}")
+        print("Usage: python precision_MRR.py [comprehensive_details_file.json]")
+        sys.exit(1)
+    
+    # Initialize analyzer
+    analyzer = PrecisionMRRAnalyzer()
+    
+    # Load comprehensive data from latency_evaluator.py
+    comprehensive_results = analyzer.load_comprehensive_data(comprehensive_file)
+    
+    if not comprehensive_results:
+        print("❌ No comprehensive data loaded")
+        sys.exit(1)
+    
+    # Analyze precision/MRR for all queries
+    analysis_results = analyzer.analyze_all_queries(comprehensive_results)
+    
+    # Calculate and display statistics
+    statistics_result = analyzer.calculate_statistics()
+    
+    print(f"\n📊 === PRECISION & MRR ANALYSIS SUMMARY ===")
+    
+    overall_stats = statistics_result['overall_statistics']
+    print(f"\nOVERALL METRICS:")
+    print(f"   Precision@K: {overall_stats['avg_precision']:.3f} (±{overall_stats['precision_std']:.3f})")
+    print(f"   MRR: {overall_stats['avg_mrr']:.3f} (±{overall_stats['mrr_std']:.3f})")
+    print(f"   Total Queries: {overall_stats['total_queries']}")
+    
+    # Complexity-based statistics
+    complexity_stats = statistics_result['by_complexity']
+    print(f"\nBY COMPLEXITY:")
+    for complexity, stats in complexity_stats.items():
+        if stats:
+            print(f"   {complexity.title()}: Precision={stats['avg_precision']:.3f}, MRR={stats['avg_mrr']:.3f} "
+                  f"(threshold={stats['avg_threshold']:.2f}, n={stats['query_count']})")
+    
+    # Category-based statistics  
+    category_stats = statistics_result['by_category']
+    print(f"\nBY CATEGORY:")
+    for category, stats in category_stats.items():
+        if stats:
+            print(f"   {category.title()}: Precision={stats['avg_precision']:.3f}, MRR={stats['avg_mrr']:.3f} "
+                  f"(n={stats['query_count']})")
+    
+    # Save results
+    saved_path = analyzer.save_results()
+    
+    print(f"\n✅ Precision & MRR analysis complete!")
+    print(f"📁 Results saved to: {saved_path}")
+    print(f"\n💡 Next step: Create precision_mrr_chart_generator.py for visualization")

evaluation/precision_mrr_chart_generator.py

@@ -0,0 +1,586 @@
+#!/usr/bin/env python3
+"""
+OnCall.ai System - Precision & MRR Chart Generator (Metrics 7-8)
+===============================================================
+
+Generates comprehensive Precision@K and MRR analysis charts from saved analysis results.
+Reads JSON files produced by metric7_8_precision_MRR.py and creates visualizations.
+
+Charts generated:
+1. Precision@K comparison by category and complexity
+2. MRR comparison by category and complexity  
+3. Combined metrics heatmap
+4. Threshold impact analysis
+5. Detailed statistics tables
+
+No LLM calls - pure data visualization.
+
+Author: YanBo Chen  
+Date: 2025-08-04
+"""
+
+import json
+import os
+import sys
+from typing import Dict, List, Any
+from datetime import datetime
+from pathlib import Path
+import glob
+
+# Visualization imports
+import matplotlib.pyplot as plt
+import seaborn as sns
+import pandas as pd
+import numpy as np
+
+
+class PrecisionMRRChartGenerator:
+    """Generate charts from precision/MRR analysis results - no LLM dependency"""
+    
+    def __init__(self):
+        """Initialize chart generator"""
+        print("📈 Initializing Precision & MRR Chart Generator...")
+        
+        # Set up professional chart style
+        plt.style.use('default')
+        sns.set_palette("husl")
+        
+        print("✅ Chart Generator ready")
+    
+    def load_latest_analysis(self, results_dir: str = None) -> Dict[str, Any]:
+        """
+        Load the most recent precision/MRR analysis file
+        
+        Args:
+            results_dir: Directory containing analysis files
+        """
+        if results_dir is None:
+            results_dir = Path(__file__).parent / "results"
+        
+        analysis_files = glob.glob(str(results_dir / "precision_mrr_analysis_*.json"))
+        
+        if not analysis_files:
+            raise FileNotFoundError("No precision_mrr_analysis_*.json files found. Run metric7_8_precision_MRR.py first.")
+        
+        latest_file = max(analysis_files, key=os.path.getctime)
+        print(f"📁 Loading latest analysis: {latest_file}")
+        
+        with open(latest_file, 'r', encoding='utf-8') as f:
+            return json.load(f)
+    
+    def create_precision_comparison_chart(self, analysis_data: Dict, save_path: str = None) -> str:
+        """Create Precision@K comparison chart"""
+        
+        fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15, 6))
+        
+        # Chart 1: Precision by Category
+        category_stats = analysis_data['statistics']['by_category']
+        categories = []
+        precisions = []
+        
+        for category, stats in category_stats.items():
+            if stats:
+                categories.append(category.title())
+                precisions.append(stats['avg_precision'])
+        
+        if categories:
+            bars1 = ax1.bar(categories, precisions, alpha=0.8, color=['#1f77b4', '#ff7f0e', '#d62728'])
+            ax1.set_title('Precision@K by Query Category', fontweight='bold')
+            ax1.set_ylabel('Precision@K')
+            ax1.set_xlabel('Query Category')
+            ax1.set_ylim(0, 1.0)
+            ax1.grid(True, alpha=0.3)
+            
+            # Add value labels
+            for bar, precision in zip(bars1, precisions):
+                height = bar.get_height()
+                ax1.text(bar.get_x() + bar.get_width()/2., height + 0.01,
+                        f'{precision:.3f}', ha='center', va='bottom', fontweight='bold')
+        
+        # Chart 2: Precision by Complexity
+        complexity_stats = analysis_data['statistics']['by_complexity']
+        complexities = []
+        comp_precisions = []
+        
+        for complexity, stats in complexity_stats.items():
+            if stats:
+                complexities.append(complexity.title())
+                comp_precisions.append(stats['avg_precision'])
+        
+        if complexities:
+            bars2 = ax2.bar(complexities, comp_precisions, alpha=0.8, color=['#2ca02c', '#d62728'])
+            ax2.set_title('Precision@K by Query Complexity', fontweight='bold')
+            ax2.set_ylabel('Precision@K')
+            ax2.set_xlabel('Query Complexity')
+            ax2.set_ylim(0, 1.0)
+            ax2.grid(True, alpha=0.3)
+            
+            # Add value labels and threshold info
+            for bar, precision, complexity in zip(bars2, comp_precisions, complexities):
+                height = bar.get_height()
+                threshold = 0.15 if complexity.lower() == 'complex' else 0.25
+                ax2.text(bar.get_x() + bar.get_width()/2., height + 0.01,
+                        f'{precision:.3f}\n(T={threshold})', ha='center', va='bottom', 
+                        fontweight='bold', fontsize=9)
+        
+        plt.tight_layout()
+        
+        # Save chart
+        if save_path is None:
+            save_path = Path(__file__).parent / "charts" / f"precision_comparison_{datetime.now().strftime('%Y%m%d_%H%M%S')}.png"
+        
+        save_path = Path(save_path)
+        save_path.parent.mkdir(parents=True, exist_ok=True)
+        plt.savefig(save_path, dpi=300, bbox_inches='tight')
+        plt.close()
+        
+        print(f"📊 Precision comparison chart saved: {save_path}")
+        return str(save_path)
+    
+    def create_mrr_comparison_chart(self, analysis_data: Dict, save_path: str = None) -> str:
+        """Create MRR comparison chart"""
+        
+        fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15, 6))
+        
+        # Chart 1: MRR by Category
+        category_stats = analysis_data['statistics']['by_category']
+        categories = []
+        mrr_scores = []
+        
+        for category, stats in category_stats.items():
+            if stats:
+                categories.append(category.title())
+                mrr_scores.append(stats['avg_mrr'])
+        
+        if categories:
+            bars1 = ax1.bar(categories, mrr_scores, alpha=0.8, color=['#9467bd', '#8c564b', '#e377c2'])
+            ax1.set_title('Mean Reciprocal Rank by Query Category', fontweight='bold')
+            ax1.set_ylabel('MRR Score')
+            ax1.set_xlabel('Query Category')
+            ax1.set_ylim(0, 1.0)
+            ax1.grid(True, alpha=0.3)
+            
+            # Add value labels
+            for bar, mrr in zip(bars1, mrr_scores):
+                height = bar.get_height()
+                ax1.text(bar.get_x() + bar.get_width()/2., height + 0.01,
+                        f'{mrr:.3f}', ha='center', va='bottom', fontweight='bold')
+        
+        # Chart 2: MRR by Complexity
+        complexity_stats = analysis_data['statistics']['by_complexity']
+        complexities = []
+        comp_mrr = []
+        
+        for complexity, stats in complexity_stats.items():
+            if stats:
+                complexities.append(complexity.title())
+                comp_mrr.append(stats['avg_mrr'])
+        
+        if complexities:
+            bars2 = ax2.bar(complexities, comp_mrr, alpha=0.8, color=['#17becf', '#bcbd22'])
+            ax2.set_title('MRR by Query Complexity', fontweight='bold')
+            ax2.set_ylabel('MRR Score')
+            ax2.set_xlabel('Query Complexity')
+            ax2.set_ylim(0, 1.0)
+            ax2.grid(True, alpha=0.3)
+            
+            # Add value labels
+            for bar, mrr in zip(bars2, comp_mrr):
+                height = bar.get_height()
+                ax2.text(bar.get_x() + bar.get_width()/2., height + 0.01,
+                        f'{mrr:.3f}', ha='center', va='bottom', fontweight='bold')
+        
+        plt.tight_layout()
+        
+        # Save chart
+        if save_path is None:
+            save_path = Path(__file__).parent / "charts" / f"mrr_comparison_{datetime.now().strftime('%Y%m%d_%H%M%S')}.png"
+        
+        save_path = Path(save_path)
+        save_path.parent.mkdir(parents=True, exist_ok=True)
+        plt.savefig(save_path, dpi=300, bbox_inches='tight')
+        plt.close()
+        
+        print(f"📊 MRR comparison chart saved: {save_path}")
+        return str(save_path)
+    
+    def create_combined_metrics_heatmap(self, analysis_data: Dict, save_path: str = None) -> str:
+        """Create combined precision/MRR heatmap"""
+        
+        # Prepare data for heatmap
+        detailed_results = analysis_data.get('detailed_results', [])
+        
+        if not detailed_results:
+            print("⚠️ No detailed results for heatmap")
+            return ""
+        
+        # Create DataFrame for heatmap
+        heatmap_data = []
+        for result in detailed_results:
+            heatmap_data.append({
+                'Category': result['category'].title(),
+                'Complexity': result['query_complexity'].title(),
+                'Precision@K': result['precision_at_k'],
+                'MRR': result['mrr_score'],
+                'Threshold': result['threshold_used']
+            })
+        
+        df = pd.DataFrame(heatmap_data)
+        
+        # Create pivot table for heatmap
+        fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(16, 6))
+        
+        # Precision heatmap
+        precision_pivot = df.pivot_table(values='Precision@K', index='Category', columns='Complexity', aggfunc='mean')
+        sns.heatmap(precision_pivot, annot=True, fmt='.3f', cmap='YlOrRd', ax=ax1, 
+                   cbar_kws={'label': 'Precision@K'}, vmin=0, vmax=1)
+        ax1.set_title('Precision@K Heatmap\n(Category vs Complexity)', fontweight='bold')
+        
+        # MRR heatmap  
+        mrr_pivot = df.pivot_table(values='MRR', index='Category', columns='Complexity', aggfunc='mean')
+        sns.heatmap(mrr_pivot, annot=True, fmt='.3f', cmap='YlGnBu', ax=ax2,
+                   cbar_kws={'label': 'MRR Score'}, vmin=0, vmax=1)
+        ax2.set_title('MRR Heatmap\n(Category vs Complexity)', fontweight='bold')
+        
+        plt.tight_layout()
+        
+        # Save chart
+        if save_path is None:
+            save_path = Path(__file__).parent / "charts" / f"precision_mrr_heatmap_{datetime.now().strftime('%Y%m%d_%H%M%S')}.png"
+        
+        save_path = Path(save_path)
+        save_path.parent.mkdir(parents=True, exist_ok=True)
+        plt.savefig(save_path, dpi=300, bbox_inches='tight')
+        plt.close()
+        
+        print(f"📊 Combined metrics heatmap saved: {save_path}")
+        return str(save_path)
+    
+    def create_threshold_impact_chart(self, analysis_data: Dict, save_path: str = None) -> str:
+        """Create threshold impact analysis chart"""
+        
+        detailed_results = analysis_data.get('detailed_results', [])
+        
+        if not detailed_results:
+            print("⚠️ No detailed results for threshold analysis")
+            return ""
+        
+        # Group by complexity and calculate average relevance
+        fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15, 6))
+        
+        # Prepare data
+        simple_queries = [r for r in detailed_results if r['query_complexity'] == 'simple']
+        complex_queries = [r for r in detailed_results if r['query_complexity'] == 'complex']
+        
+        # Chart 1: Relevance distribution for different complexities
+        if simple_queries:
+            simple_relevances = []
+            for query in simple_queries:
+                simple_relevances.extend(query.get('relevance_scores', []))
+            
+            ax1.hist(simple_relevances, bins=10, alpha=0.7, label=f'Simple (T=0.25)', color='#2ca02c', density=True)
+            ax1.axvline(x=0.25, color='#2ca02c', linestyle='--', linewidth=2, label='Simple Threshold')
+        
+        if complex_queries:
+            complex_relevances = []
+            for query in complex_queries:
+                complex_relevances.extend(query.get('relevance_scores', []))
+            
+            ax1.hist(complex_relevances, bins=10, alpha=0.7, label=f'Complex (T=0.15)', color='#d62728', density=True)
+            ax1.axvline(x=0.15, color='#d62728', linestyle='--', linewidth=2, label='Complex Threshold')
+        
+        ax1.set_title('Relevance Score Distribution\nby Query Complexity', fontweight='bold')
+        ax1.set_xlabel('Relevance Score')
+        ax1.set_ylabel('Density')
+        ax1.legend()
+        ax1.grid(True, alpha=0.3)
+        
+        # Chart 2: Metrics comparison
+        complexity_stats = analysis_data['statistics']['by_complexity']
+        
+        complexities = []
+        precisions = []
+        mrrs = []
+        thresholds = []
+        
+        for complexity, stats in complexity_stats.items():
+            if stats:
+                complexities.append(complexity.title())
+                precisions.append(stats['avg_precision'])
+                mrrs.append(stats['avg_mrr'])
+                thresholds.append(stats['avg_threshold'])
+        
+        x = np.arange(len(complexities))
+        width = 0.35
+        
+        bars1 = ax2.bar(x - width/2, precisions, width, label='Precision@K', alpha=0.8, color='#ff7f0e')
+        bars2 = ax2.bar(x + width/2, mrrs, width, label='MRR', alpha=0.8, color='#1f77b4')
+        
+        ax2.set_title('Metrics Comparison by Complexity\n(with Adaptive Thresholds)', fontweight='bold')
+        ax2.set_ylabel('Score')
+        ax2.set_xlabel('Query Complexity')
+        ax2.set_xticks(x)
+        ax2.set_xticklabels(complexities)
+        ax2.legend()
+        ax2.grid(True, alpha=0.3)
+        ax2.set_ylim(0, 1.0)
+        
+        # Add value labels
+        for bars, values, thresholds_vals in [(bars1, precisions, thresholds), (bars2, mrrs, thresholds)]:
+            for bar, value, threshold in zip(bars, values, thresholds_vals):
+                height = bar.get_height()
+                ax2.text(bar.get_x() + bar.get_width()/2., height + 0.01,
+                        f'{value:.3f}', ha='center', va='bottom', fontweight='bold', fontsize=9)
+        
+        plt.tight_layout()
+        
+        # Save chart
+        if save_path is None:
+            save_path = Path(__file__).parent / "charts" / f"threshold_impact_{datetime.now().strftime('%Y%m%d_%H%M%S')}.png"
+        
+        save_path = Path(save_path)
+        save_path.parent.mkdir(parents=True, exist_ok=True)
+        plt.savefig(save_path, dpi=300, bbox_inches='tight')
+        plt.close()
+        
+        print(f"📊 Threshold impact chart saved: {save_path}")
+        return str(save_path)
+    
+    def create_detailed_analysis_table(self, analysis_data: Dict, save_path: str = None) -> str:
+        """Create detailed statistics table"""
+        
+        fig, ax = plt.subplots(figsize=(12, 8))
+        ax.axis('tight')
+        ax.axis('off')
+        
+        # Prepare table data
+        table_data = []
+        
+        # Overall statistics
+        overall_stats = analysis_data['statistics']['overall_statistics']
+        table_data.append(['OVERALL METRICS', '', '', '', ''])
+        table_data.append(['Total Queries', str(overall_stats['total_queries']), '', '', ''])
+        table_data.append(['Avg Precision@K', f"{overall_stats['avg_precision']:.3f}", 
+                          f"±{overall_stats['precision_std']:.3f}", '', ''])
+        table_data.append(['Avg MRR', f"{overall_stats['avg_mrr']:.3f}", 
+                          f"±{overall_stats['mrr_std']:.3f}", '', ''])
+        table_data.append(['', '', '', '', ''])
+        
+        # By category
+        table_data.append(['BY CATEGORY', 'Queries', 'Precision@K', 'MRR', 'Notes'])
+        category_stats = analysis_data['statistics']['by_category']
+        for category, stats in category_stats.items():
+            if stats:
+                table_data.append([
+                    category.title(),
+                    str(stats['query_count']),
+                    f"{stats['avg_precision']:.3f}",
+                    f"{stats['avg_mrr']:.3f}",
+                    ''
+                ])
+        
+        table_data.append(['', '', '', '', ''])
+        
+        # By complexity
+        table_data.append(['BY COMPLEXITY', 'Queries', 'Precision@K', 'MRR', 'Threshold'])
+        complexity_stats = analysis_data['statistics']['by_complexity']
+        for complexity, stats in complexity_stats.items():
+            if stats:
+                table_data.append([
+                    complexity.title(),
+                    str(stats['query_count']),
+                    f"{stats['avg_precision']:.3f}",
+                    f"{stats['avg_mrr']:.3f}",
+                    f"{stats['avg_threshold']:.2f}"
+                ])
+        
+        # Create table
+        table = ax.table(cellText=table_data,
+                        colLabels=['Metric', 'Value 1', 'Value 2', 'Value 3', 'Value 4'],
+                        cellLoc='center',
+                        loc='center',
+                        bbox=[0, 0, 1, 1])
+        
+        # Style the table
+        table.auto_set_font_size(False)
+        table.set_fontsize(10)
+        table.scale(1, 2)
+        
+        # Header styling
+        for i in range(5):
+            table[(0, i)].set_facecolor('#40466e')
+            table[(0, i)].set_text_props(weight='bold', color='white')
+        
+        # Section headers styling
+        for i, row in enumerate(table_data):
+            if row[0] in ['OVERALL METRICS', 'BY CATEGORY', 'BY COMPLEXITY']:
+                table[(i+1, 0)].set_facecolor('#1f77b4')
+                table[(i+1, 0)].set_text_props(weight='bold', color='white')
+        
+        plt.title('Precision@K & MRR Detailed Analysis\nMetrics 7-8 Statistics', 
+                 fontweight='bold', fontsize=14, pad=20)
+        
+        # Save chart
+        if save_path is None:
+            save_path = Path(__file__).parent / "charts" / f"precision_mrr_table_{datetime.now().strftime('%Y%m%d_%H%M%S')}.png"
+        
+        save_path = Path(save_path)
+        save_path.parent.mkdir(parents=True, exist_ok=True)
+        plt.savefig(save_path, dpi=300, bbox_inches='tight')
+        plt.close()
+        
+        print(f"📊 Detailed analysis table saved: {save_path}")
+        return str(save_path)
+    
+    def create_individual_query_analysis(self, analysis_data: Dict, save_path: str = None) -> str:
+        """Create individual query analysis chart"""
+        
+        detailed_results = analysis_data.get('detailed_results', [])
+        
+        if not detailed_results:
+            print("⚠️ No detailed results for individual analysis")
+            return ""
+        
+        fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(14, 10))
+        
+        # Prepare data
+        query_indices = []
+        precisions = []
+        mrrs = []
+        colors = []
+        labels = []
+        
+        for i, result in enumerate(detailed_results):
+            query_indices.append(i + 1)
+            precisions.append(result['precision_at_k'])
+            mrrs.append(result['mrr_score'])
+            
+            # Color by complexity
+            if result['query_complexity'] == 'complex':
+                colors.append('#d62728')  # Red for complex
+            else:
+                colors.append('#2ca02c')  # Green for simple
+            
+            # Create short label
+            query_short = result['query'][:30] + "..." if len(result['query']) > 30 else result['query']
+            category = result['category'][:4].upper()
+            labels.append(f"{category}\n{query_short}")
+        
+        # Chart 1: Precision@K for each query
+        bars1 = ax1.bar(query_indices, precisions, color=colors, alpha=0.8)
+        ax1.set_title('Precision@K by Individual Query', fontweight='bold')
+        ax1.set_ylabel('Precision@K')
+        ax1.set_xlabel('Query Index')
+        ax1.set_ylim(0, 1.0)
+        ax1.grid(True, alpha=0.3)
+        
+        # Add value labels
+        for bar, precision in zip(bars1, precisions):
+            height = bar.get_height()
+            ax1.text(bar.get_x() + bar.get_width()/2., height + 0.01,
+                    f'{precision:.2f}', ha='center', va='bottom', fontsize=8)
+        
+        # Chart 2: MRR for each query
+        bars2 = ax2.bar(query_indices, mrrs, color=colors, alpha=0.8)
+        ax2.set_title('MRR by Individual Query', fontweight='bold')
+        ax2.set_ylabel('MRR Score')
+        ax2.set_xlabel('Query Index')
+        ax2.set_ylim(0, 1.0)
+        ax2.grid(True, alpha=0.3)
+        
+        # Add value labels
+        for bar, mrr in zip(bars2, mrrs):
+            height = bar.get_height()
+            ax2.text(bar.get_x() + bar.get_width()/2., height + 0.01,
+                    f'{mrr:.2f}', ha='center', va='bottom', fontsize=8)
+        
+        # Add legend
+        from matplotlib.patches import Patch
+        legend_elements = [
+            Patch(facecolor='#2ca02c', alpha=0.8, label='Simple Query (T=0.25)'),
+            Patch(facecolor='#d62728', alpha=0.8, label='Complex Query (T=0.15)')
+        ]
+        ax1.legend(handles=legend_elements, loc='upper right')
+        
+        plt.tight_layout()
+        
+        # Save chart
+        if save_path is None:
+            save_path = Path(__file__).parent / "charts" / f"individual_query_analysis_{datetime.now().strftime('%Y%m%d_%H%M%S')}.png"
+        
+        save_path = Path(save_path)
+        save_path.parent.mkdir(parents=True, exist_ok=True)
+        plt.savefig(save_path, dpi=300, bbox_inches='tight')
+        plt.close()
+        
+        print(f"📊 Individual query analysis saved: {save_path}")
+        return str(save_path)
+    
+    def generate_all_charts(self, analysis_data: Dict = None) -> Dict[str, str]:
+        """Generate all precision/MRR charts"""
+        
+        if analysis_data is None:
+            analysis_data = self.load_latest_analysis()
+        
+        print(f"\n📈 Generating all Precision & MRR charts...")
+        
+        saved_charts = {}
+        
+        # Generate all chart types
+        try:
+            saved_charts['precision_comparison'] = self.create_precision_comparison_chart(analysis_data)
+            saved_charts['mrr_comparison'] = self.create_mrr_comparison_chart(analysis_data)
+            saved_charts['combined_heatmap'] = self.create_combined_metrics_heatmap(analysis_data)
+            saved_charts['threshold_impact'] = self.create_threshold_impact_chart(analysis_data)
+            saved_charts['individual_analysis'] = self.create_individual_query_analysis(analysis_data)
+            
+        except Exception as e:
+            print(f"❌ Error generating charts: {e}")
+            return {"error": str(e)}
+        
+        print(f"\n✅ All precision/MRR charts generated successfully!")
+        print(f"📁 Charts saved to: evaluation/charts/")
+        
+        return saved_charts
+
+
+# Independent execution interface
+if __name__ == "__main__":
+    """Generate precision/MRR charts from analysis results"""
+    
+    print("📈 OnCall.ai Precision & MRR Chart Generator - Metrics 7-8")
+    
+    if len(sys.argv) > 1:
+        analysis_file = sys.argv[1]
+        
+        if not os.path.exists(analysis_file):
+            print(f"❌ Analysis file not found: {analysis_file}")
+            sys.exit(1)
+    else:
+        analysis_file = None  # Will use latest file
+    
+    # Initialize generator
+    generator = PrecisionMRRChartGenerator()
+    
+    try:
+        # Load analysis data
+        if analysis_file:
+            with open(analysis_file, 'r', encoding='utf-8') as f:
+                analysis_data = json.load(f)
+            print(f"📁 Using specified analysis file: {analysis_file}")
+        else:
+            analysis_data = generator.load_latest_analysis()
+        
+        # Generate all charts
+        saved_charts = generator.generate_all_charts(analysis_data)
+        
+        if 'error' not in saved_charts:
+            print(f"\n📊 === PRECISION & MRR CHART GENERATION SUMMARY ===")
+            for chart_type, filepath in saved_charts.items():
+                print(f"   📈 {chart_type.replace('_', ' ').title()}: {filepath}")
+            
+            print(f"\n💡 Charts ready for analysis and presentation!")
+        
+    except Exception as e:
+        print(f"❌ Chart generation failed: {e}")
+        sys.exit(1)

evaluation/single_test_query.txt

@@ -0,0 +1 @@
+1.diagnosis: 60-year-old patient with hypertension history, sudden chest pain. What are possible causes and how to assess?