backend/docs/INTENT_CLASSIFICATION_IMPROVEMENTS.md

Intent Classification Improvements

Overview

This document describes the improvements made to intent classification in Plan 5.

Problem Identified

Query "Cảnh báo lừa đảo giả danh công an" was being classified as search_office instead of search_advisory.

Root Cause

1. Keyword Conflict: The keyword "công an" appears in both search_office and queries about search_advisory
2. Order of Checks: The code checked has_office_keywords before has_advisory_keywords, causing office keywords to match first
3. Limited Training Data: The search_advisory intent had only 7 examples, compared to more examples in other intents

Solutions Implemented

1. Improved Keyword Matching Logic

File: backend/hue_portal/chatbot/chatbot.py

• Changed order: Check has_advisory_keywords before has_office_keywords
• Added more keywords for advisory: "mạo danh", "thủ đoạn", "cảnh giác"
• This ensures advisory queries are matched first when they contain both advisory and office keywords

2. Enhanced Training Data

File: backend/hue_portal/chatbot/training/intent_dataset.json

• Expanded search_advisory examples from 7 to 23 examples
• Added specific examples:
  • "cảnh báo lừa đảo giả danh công an"
  • "mạo danh cán bộ công an"
  • "lừa đảo mạo danh"
  • And 15 more variations

3. Retrained Model

• Retrained intent classification model with improved training data
• Model accuracy improved
• Better handling of edge cases

Results

Before Improvements

• Query "Cảnh báo lừa đảo giả danh công an" → search_office (incorrect)
• Limited training examples for search_advisory

After Improvements

• Query "Cảnh báo lừa đảo giả danh công an" → search_advisory (correct)
• More balanced training data across all intents
• Better keyword matching logic

Testing

Test queries that now work correctly:

• "Cảnh báo lừa đảo giả danh công an" → search_advisory
• "Lừa đảo mạo danh cán bộ" → search_advisory
• "Mạo danh cán bộ công an" → search_advisory

2025-11-14 Update — Serialization & API Regression

• Added _serialize_document in backend/hue_portal/chatbot/chatbot.py so RAG responses return JSON-safe payloads (no more TypeError: Object of type type is not JSON serializable when embeddings include model instances).
• Re-tested intents end-to-end via scripts/test_api_endpoint.py (6 queries spanning all intents):
  • Result: 6/6 passed, 100 % intent accuracy.
  • Latency: avg ~3.7 s (note: first call warms up keepitreal/vietnamese-sbert-v2, subsequent calls ≤1.8 s).
• Health checklist before testing:
  1. POSTGRES_HOST=localhost POSTGRES_PORT=5433 ../../.venv/bin/python manage.py runserver 0.0.0.0:8090
  2. API_BASE_URL=http://localhost:8090 python scripts/test_api_endpoint.py
  3. Watch server logs for any serialization warnings (none observed after fix).

Files Modified

1. backend/hue_portal/chatbot/training/intent_dataset.json - Enhanced training data
2. backend/hue_portal/chatbot/chatbot.py - Improved keyword matching logic
3. backend/hue_portal/chatbot/training/artifacts/intent_model.joblib - Retrained model

Future Improvements

• Continue to add more training examples as edge cases are discovered
• Consider using more sophisticated ML models (e.g., transformer-based)
• Implement active learning to automatically improve from user feedback