Spaces:

MCP-1st-Birthday
/

TraceMind-mcp-server

Running

App Files Files Community

TraceMind-mcp-server / README.md

Mandark-droid

TraceMind MCP Server V1 Working and tested.

a3116de 23 days ago

preview code

raw

history blame

20.3 kB

metadata

title: TraceMind MCP Server
emoji: 🤖
colorFrom: blue
colorTo: purple
sdk: gradio
app_port: 7860
pinned: false
license: agpl-3.0
short_description: AI-powered MCP server for agent evaluation analysis with Gemini 2.5 Pro
tags:
  - building-mcp-track-enterprise
  - mcp
  - gradio
  - gemini
  - agent-evaluation
  - leaderboard

TraceMind MCP Server

AI-Powered Analysis Tools for Agent Evaluation Data

🎯 Track 1 Submission: Building MCP (Enterprise) 📅 MCP's 1st Birthday Hackathon: November 14-30, 2025

Overview

TraceMind MCP Server is a Gradio-based MCP (Model Context Protocol) server that provides a complete MCP implementation with:

🛠️ 5 AI-Powered Tools

📊 analyze_leaderboard: Generate insights from evaluation leaderboard data
🐛 debug_trace: Debug specific agent execution traces using OpenTelemetry data
💰 estimate_cost: Predict evaluation costs before running
⚖️ compare_runs: Compare two evaluation runs with AI-powered analysis
📦 get_dataset: Load SMOLTRACE datasets (smoltrace-* prefix only) as JSON for flexible analysis

📦 3 Data Resources

leaderboard data: Direct JSON access to evaluation results
trace data: Raw OpenTelemetry trace data with spans
cost data: Model pricing and hardware cost information

📝 3 Prompt Templates

analysis prompts: Standardized templates for different analysis types
debug prompts: Templates for debugging scenarios
optimization prompts: Templates for optimization goals

All analysis is powered by Google Gemini 2.5 Pro for intelligent, context-aware insights.

📱 Social Media & Demo

📢 Announcement Post: [Coming Soon - X/LinkedIn post]

🎥 Demo Video: [Coming Soon - YouTube/Loom link showing MCP server integration with Claude Desktop]

Why This MCP Server?

Problem: Agent evaluation generates massive amounts of data (leaderboards, traces, metrics), but developers struggle to:

Understand which models perform best for their use case
Debug why specific agent executions failed
Estimate costs before running expensive evaluations

Solution: This MCP server provides AI-powered analysis tools that connect to HuggingFace datasets and deliver actionable insights in natural language.

Impact: Developers can make informed decisions about agent configurations, debug issues faster, and optimize costs—all through a simple MCP interface.

Features

🎯 Track 1 Compliance: Building MCP (Enterprise)

✅ Complete MCP Implementation: Tools, Resources, AND Prompts
✅ MCP Standard Compliant: Built with Gradio's native MCP support (@gr.mcp.* decorators)
✅ Production-Ready: Deployable to HuggingFace Spaces with SSE transport
✅ Testing Interface: Beautiful Gradio UI for testing all components
✅ Enterprise Focus: Cost optimization, debugging, and decision support
✅ Google Gemini Powered: Leverages Gemini 2.5 Pro for intelligent analysis
✅ 11 Total Components: 5 Tools + 3 Resources + 3 Prompts

🛠️ Five Production-Ready Tools

1. analyze_leaderboard

Analyzes evaluation leaderboard data from HuggingFace datasets and provides:

Top performers by selected metric (accuracy, cost, latency, CO2)
Trade-off analysis (e.g., "GPT-4 is most accurate but Llama-3.1 is 25x cheaper")
Trend identification
Actionable recommendations

Example Use Case: Before choosing a model for production, get AI-powered insights on which configuration offers the best cost/performance for your requirements.

2. debug_trace

Analyzes OpenTelemetry trace data and answers specific questions like:

"Why was tool X called twice?"
"Which step took the most time?"
"Why did this test fail?"

Example Use Case: When an agent test fails, understand exactly what happened without manually parsing trace spans.

3. estimate_cost

Predicts costs before running evaluations:

LLM API costs (token-based)
HuggingFace Jobs compute costs
CO2 emissions estimate
Hardware recommendations

Example Use Case: Compare the cost of evaluating GPT-4 vs Llama-3.1 across 1000 tests before committing resources.

4. compare_runs

Compares two evaluation runs with AI-powered analysis across multiple dimensions:

Success rate comparison with statistical significance
Cost efficiency analysis (total cost, cost per test, cost per successful test)
Speed comparison (average duration, throughput)
Environmental impact (CO2 emissions per test)
GPU efficiency (for GPU jobs)

Focus Options:

comprehensive: Complete comparison across all dimensions
cost: Detailed cost efficiency and ROI analysis
performance: Speed and accuracy trade-off analysis
eco_friendly: Environmental impact and carbon footprint comparison

Example Use Case: After running evaluations with two different models, compare them head-to-head to determine which is better for production deployment based on your priorities (accuracy, cost, speed, or environmental impact).

5. get_dataset

Loads SMOLTRACE datasets from HuggingFace and returns raw data as JSON:

Simple, flexible tool that returns complete dataset with metadata
Works with any dataset containing "smoltrace-" prefix
Returns total rows, columns list, and data array
Automatically sorts by timestamp if available
Configurable row limit (1-200) to manage token usage

Security Restriction: Only datasets with "smoltrace-" in the repository name are allowed.

Primary Use Cases:

Load smoltrace-leaderboard to find run IDs and model names
Discover supporting datasets via results_dataset, traces_dataset, metrics_dataset fields
Load smoltrace-results-* datasets to see individual test case details
Load smoltrace-traces-* datasets to access OpenTelemetry trace data
Load smoltrace-metrics-* datasets to get GPU performance data
Answer specific questions requiring raw data access

Example Workflow:

LLM calls get_dataset("kshitijthakkar/smoltrace-leaderboard") to see all runs
Examines the JSON response to find run IDs, models, and supporting dataset names
Calls get_dataset("username/smoltrace-results-gpt4") to load detailed results
Can now answer questions like "What are the last 10 run IDs?" or "Which models were tested?"

Example Use Case: When the user asks "Can you provide me with the list of last 10 runIds and model names?", the LLM loads the leaderboard dataset and extracts the requested information from the JSON response.

MCP Resources Usage

Resources provide direct data access without AI analysis:

# Access leaderboard data
GET leaderboard://kshitijthakkar/smoltrace-leaderboard
# Returns: JSON with all evaluation runs

# Access specific trace
GET trace://trace_abc123/username/agent-traces-gpt4
# Returns: JSON with trace spans and attributes

# Get model cost information
GET cost://model/openai/gpt-4
# Returns: JSON with pricing and hardware costs

MCP Prompts Usage

Prompts provide reusable templates for standardized interactions:

# Get analysis prompt template
analysis_prompt(analysis_type="leaderboard", focus_area="cost", detail_level="detailed")
# Returns: "Provide a detailed analysis. Analyze cost efficiency in the leaderboard..."

# Get debug prompt template
debug_prompt(debug_type="performance", context="tool_calling")
# Returns: "Analyze tool calling performance. Identify which tools are slow..."

# Get optimization prompt template
optimization_prompt(optimization_goal="cost", constraints="maintain_quality")
# Returns: "Analyze this evaluation setup and recommend cost optimizations..."

Use these prompts when interacting with the tools to get consistent, high-quality analysis.

Quick Start

1. Installation

git clone https://github.com/Mandark-droid/TraceMind-mcp-server.git
cd TraceMind-mcp-server

# Create virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies (note: gradio[mcp] includes MCP support)
pip install -r requirements.txt

2. Environment Setup

Create .env file:

cp .env.example .env
# Edit .env and add your API keys

Get your keys:

Gemini API Key: https://ai.google.dev/
HuggingFace Token: https://huggingface.co/settings/tokens

3. Run Locally

python app.py

Open http://localhost:7860 to test the tools via Gradio interface.

4. Test with Live Data

Try the live example with real HuggingFace dataset:

In the Gradio UI, Tab "📊 Analyze Leaderboard":

Leaderboard Repository: kshitijthakkar/smoltrace-leaderboard
Metric Focus: overall
Time Range: last_week
Top N Models: 5

Click "🔍 Analyze" and get AI-powered insights from live data!

MCP Integration

How It Works

This Gradio app uses mcp_server=True in the launch configuration, which automatically:

Exposes all async functions with proper docstrings as MCP tools
Handles MCP protocol communication
Provides a standard MCP interface via SSE (Server-Sent Events)

Connecting from MCP Clients

Once deployed to HuggingFace Spaces, your MCP server will be available at:

MCP Endpoint (SSE):

https://huggingface.co/spaces/kshitijthakkar/TraceMind-mcp-server/gradio_api/mcp/sse

Schema Endpoint:

https://huggingface.co/spaces/kshitijthakkar/TraceMind-mcp-server/gradio_api/mcp/schema

Configure your MCP client (Claude Desktop, Cursor, Cline, etc.) with the SSE endpoint.

Available MCP Components

Tools (5):

analyze_leaderboard: AI-powered leaderboard analysis with Gemini 2.5 Pro
debug_trace: Trace debugging with AI insights
estimate_cost: Cost estimation with optimization recommendations
compare_runs: Compare two evaluation runs with AI-powered analysis
get_dataset: Load SMOLTRACE datasets (smoltrace-* only) as JSON

Resources (3):

leaderboard://{repo}: Direct access to raw leaderboard data in JSON
trace://{trace_id}/{repo}: Direct access to trace data with spans
cost://model/{model_name}: Model pricing and hardware cost information

Prompts (3):

analysis_prompt: Reusable templates for different analysis types
debug_prompt: Reusable templates for debugging scenarios
optimization_prompt: Reusable templates for optimization goals

See full API documentation in the Gradio interface under "📖 API Documentation" tab.

Architecture

TraceMind-mcp-server/
├── app.py                      # Gradio UI + MCP server (mcp_server=True)
├── gemini_client.py            # Google Gemini 2.5 Pro integration
├── mcp_tools.py                # 3 tool implementations
├── requirements.txt            # Python dependencies
├── .env.example                # Environment variable template
├── .gitignore
└── README.md

Key Technologies:

Gradio 6 with MCP support: gradio[mcp] provides native MCP server capabilities
Google Gemini 2.5 Pro: Latest AI model for intelligent analysis
HuggingFace Datasets: Data source for evaluations
SSE Transport: Server-Sent Events for real-time MCP communication

Deploy to HuggingFace Spaces

1. Create Space

Go to https://huggingface.co/new-space

Space name: TraceMind-mcp-server
License: MIT
SDK: Gradio
Hardware: CPU Basic (free tier works fine)

2. Add Files

Upload all files from this repository to your Space:

app.py
gemini_client.py
mcp_tools.py
requirements.txt
README.md

3. Add Secrets

In Space settings → Variables and secrets, add:

GEMINI_API_KEY: Your Gemini API key
HF_TOKEN: Your HuggingFace token

4. Add Hackathon Tag

In Space settings → Tags, add:

building-mcp-track-enterprise

5. Access Your MCP Server

Your MCP server will be publicly available at:

https://huggingface.co/spaces/kshitijthakkar/TraceMind-mcp-server

Testing

Test 1: Analyze Leaderboard (Live Data)

# In Gradio UI - Tab "📊 Analyze Leaderboard":
Repository: kshitijthakkar/smoltrace-leaderboard
Metric: overall
Time Range: last_week
Top N: 5
Click "🔍 Analyze"

Expected: AI-generated analysis of top performing models from live HuggingFace dataset

Test 2: Estimate Cost

# In Gradio UI - Tab "💰 Estimate Cost":
Model: openai/gpt-4
Agent Type: both
Number of Tests: 100
Hardware: auto
Click "💰 Estimate"

Expected: Cost breakdown with LLM costs, HF Jobs costs, duration, and CO2 estimate

Test 3: Debug Trace

Note: This requires actual trace data from an evaluation run. For testing purposes, this will show an error about missing data, which is expected behavior.

Hackathon Submission

Track 1: Building MCP (Enterprise)

Tag: building-mcp-track-enterprise

Why Enterprise Track?

Solves real business problems (cost optimization, debugging, decision support)
Production-ready tools with clear ROI
Integrates with enterprise data infrastructure (HuggingFace datasets)

Technology Stack

AI Analysis: Google Gemini 2.5 Pro for all intelligent insights
MCP Framework: Gradio 6 with native MCP support
Data Source: HuggingFace Datasets
Transport: SSE (Server-Sent Events)

Related Project: TraceMind UI (Track 2)

This MCP server is designed to be consumed by TraceMind UI (separate submission for Track 2: MCP in Action).

TraceMind UI is a Gradio-based agent evaluation platform that uses these MCP tools to provide:

AI-powered leaderboard insights
Interactive trace debugging
Pre-evaluation cost estimates

File Descriptions

app.py

Main Gradio application with:

Testing UI for all 5 tools
MCP server enabled via mcp_server=True
API documentation

gemini_client.py

Google Gemini 2.5 Pro client that:

Handles API authentication
Provides specialized analysis methods for different data types
Formats prompts for optimal results
Uses gemini-2.5-pro-latest model (can switch to gemini-2.5-flash-latest)

mcp_tools.py

Complete MCP implementation with 11 components:

Tools (5 async functions):

analyze_leaderboard(): AI-powered leaderboard analysis
debug_trace(): AI-powered trace debugging
estimate_cost(): AI-powered cost estimation
compare_runs(): AI-powered run comparison
get_dataset(): Load SMOLTRACE datasets as JSON

Resources (3 decorated functions with @gr.mcp.resource()):

get_leaderboard_data(): Raw leaderboard JSON data
get_trace_data(): Raw trace JSON data with spans
get_cost_data(): Model pricing and hardware cost JSON

Prompts (3 decorated functions with @gr.mcp.prompt()):

analysis_prompt(): Templates for different analysis types
debug_prompt(): Templates for debugging scenarios
optimization_prompt(): Templates for optimization goals

Each function includes:

Appropriate decorator (@gr.mcp.tool(), @gr.mcp.resource(), or @gr.mcp.prompt())
Detailed docstring with "Args:" section
Type hints for all parameters and return values
Descriptive function name (becomes the MCP component name)

Environment Variables

Required environment variables:

GEMINI_API_KEY=your_gemini_api_key_here
HF_TOKEN=your_huggingface_token_here

Development

Running Tests

# Test Gemini client
python -c "from gemini_client import GeminiClient; client = GeminiClient(); print('✅ Gemini client initialized')"

# Test with live leaderboard data
python app.py
# Open browser, test "Analyze Leaderboard" tab

Adding New Tools

To add a new MCP tool (with Gradio's native MCP support):

Add function to mcp_tools.py with proper docstring:

async def your_new_tool(
    gemini_client: GeminiClient,
    param1: str,
    param2: int = 10
) -> str:
    """
    Brief description of what the tool does.

    Longer description explaining the tool's purpose and behavior.

    Args:
        gemini_client (GeminiClient): Initialized Gemini client for AI analysis
        param1 (str): Description of param1 with examples if helpful
        param2 (int): Description of param2. Default: 10

    Returns:
        str: Description of what the function returns
    """
    # Your implementation
    return result

Add UI tab in app.py (optional, for testing):

with gr.Tab("Your Tool"):
    # Add UI components
    # Wire up to your_new_tool()

That's it! Gradio automatically exposes it as an MCP tool based on:
- Function name (becomes tool name)
- Docstring (becomes tool description)
- Args section (becomes parameter descriptions)
- Type hints (become parameter types)

Switching to Gemini 2.5 Flash

For faster (but slightly less capable) responses, switch to Gemini 2.5 Flash:

# In app.py, change:
gemini_client = GeminiClient(model_name="gemini-2.5-flash-latest")

🙏 Credits & Acknowledgments

Hackathon Sponsors

Special thanks to the sponsors of MCP's 1st Birthday Hackathon (November 14-30, 2025):

🤗 HuggingFace - Hosting platform and dataset infrastructure
🧠 Google Gemini - AI analysis powered by Gemini 2.5 Pro API
⚡ Modal - Serverless infrastructure partner
🏢 Anthropic - MCP protocol creators
🎨 Gradio - Native MCP framework support
🎙️ ElevenLabs - Audio AI capabilities
🦙 SambaNova - High-performance AI infrastructure
🎯 Blaxel - Additional compute credits

Related Open Source Projects

This MCP server builds upon our open source agent evaluation ecosystem:

📊 SMOLTRACE - Agent Evaluation Engine

Description: Lightweight, production-ready evaluation framework for AI agents with OpenTelemetry instrumentation
GitHub: https://github.com/Mandark-droid/SMOLTRACE
PyPI: https://pypi.org/project/smoltrace/
Social: @smoltrace on X

🔭 TraceVerde - GenAI OpenTelemetry Instrumentation

Description: Automatic OpenTelemetry instrumentation for LLM frameworks (LiteLLM, Transformers, LangChain, etc.)
GitHub: https://github.com/Mandark-droid/genai_otel_instrument
PyPI: https://pypi.org/project/genai-otel-instrument
Social: @genai_otel on X

Built By

Track: Building MCP (Enterprise) Author: Kshitij Thakkar Powered by: Google Gemini 2.5 Pro Built with: Gradio 6 (native MCP support)

📄 License

AGPL-3.0 License

This project is licensed under the GNU Affero General Public License v3.0. See the LICENSE file for details.

💬 Support

For issues or questions:

📧 Open an issue on GitHub
💬 Join the HuggingFace Discord - Channel: #agents-mcp-hackathon-winter25
🏷️ Tag building-mcp-track-enterprise for hackathon-related questions
🐦 Follow us on X: @TraceMindAI (placeholder)

Changelog

v1.0.0 (2025-11-14)

Initial release for MCP Hackathon
Complete MCP Implementation: 11 components total
- 5 AI-powered tools (analyze_leaderboard, debug_trace, estimate_cost, compare_runs, get_dataset)
- 3 data resources (leaderboard, trace, cost data)
- 3 prompt templates (analysis, debug, optimization)
Gradio native MCP support with decorators (@gr.mcp.*)
Google Gemini 2.5 Pro integration for all AI analysis
Live HuggingFace dataset integration
SSE transport for MCP communication
Production-ready for HuggingFace Spaces deployment