msugimura/gatekeeper_agent_responding

Gatekeeper Agent Responding Model

This is a fine-tuned sentence-transformers model based on all-MiniLM-L12-v2 that has been specifically trained for agent routing and conversation matching. The model determines whether agents should respond to conversations based on semantic similarity.

Model Details

Base Model

• Base Model: sentence-transformers/all-MiniLM-L12-v2
• Model Architecture: MiniLM-L12 (Microsoft)
• Embedding Dimensions: 384
• Max Sequence Length: 256 tokens

Training Data

The model was fine-tuned on two custom datasets using triplet training:

• semantic_triplet_training_data_round1.pkl: 469 samples
• inverse_semantic_triplet_training_data.pkl: 475 samples

Each sample contains:

• anchor: Conversation text or agent description
• positive: Similar/relevant text to the anchor
• negative: Dissimilar/irrelevant text to the anchor

Training Configuration

• Loss Function: MultipleNegativesRankingLoss
• Batch Size: 16
• Learning Rate: 2e-5
• Epochs: 1
• Warmup Ratio: 0.1
• Training Framework: sentence-transformers v2.7.0+

Performance

Evaluation results on held-out test sets:

• Semantic Triplets Accuracy: 97.87%
• Inverse Semantic Triplets Accuracy: 100.00%

Usage

Direct Usage (Sentence Transformers)

from sentence_transformers import SentenceTransformer

# Load the model
model = SentenceTransformer('msugimura/gatekeeper_agent_responding')

# Example: Agent routing for conversation
conversation = "I've been feeling anxious and need help with stress management"
agent_descriptions = [
    "Licensed therapist specializing in anxiety and stress management",
    "Fitness trainer who creates workout routines for stress relief",
    "Financial advisor who helps with investment planning"
]

# Get embeddings
conversation_embedding = model.encode(conversation)
agent_embeddings = model.encode(agent_descriptions)

# Calculate similarities
from sentence_transformers.util import cos_sim
similarities = cos_sim(conversation_embedding, agent_embeddings)

print("Similarity scores:", similarities)
# Expected: Highest similarity with the therapist

API Usage (Portcullis Service)

import requests

# Example API call to Portcullis service
response = requests.post("http://localhost:8000/should_agents_respond", json={
    "conversation": "I've been feeling anxious and need help",
    "agent_descriptions": [
        "Licensed therapist specializing in anxiety treatment",
        "Fitness trainer for workout routines",
        "Financial advisor for investments"
    ],
    "threshold": 0.4
})

result = response.json()
print("Qualified agents:", result["qualified_agents"])

Intended Use Cases

1. Agent Routing: Automatically route conversations to appropriate specialist agents
2. Conversation Matching: Match user queries with relevant service providers
3. Semantic Search: Find similar conversations or agent descriptions
4. Content Recommendation: Recommend agents based on conversation context

Limitations

• Domain Specific: Optimized for agent-conversation matching scenarios
• English Only: Trained primarily on English text
• Context Length: Limited to 256 tokens per input
• Training Data: Performance depends on similarity to training domain

Technical Details

Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False})
  (2): Normalize()
)

Training Process

1. Data Preprocessing: Cleaned triplet datasets, removed extraneous columns
2. Multi-Dataset Training: Combined training on both semantic and inverse semantic data
3. Loss Function: MultipleNegativesRankingLoss with in-batch negatives
4. Evaluation: TripletEvaluator on held-out validation sets

Citation

If you use this model, please cite:

@misc{gatekeeper_agent_responding_2024,
  title={Gatekeeper Agent Responding Model},
  author={Michael Sugimura},
  year={2024},
  publisher={Hugging Face},
  url={https://huggingface.co/msugimura/gatekeeper_agent_responding}
}

Contact

For questions or issues, please contact [your-email] or open an issue in the model repository.