there-is-already-a-branch (#1)

- feat: enhance app initialization with semantic anchors and pack download (5bcb8ba6f7aaba98d6a8fea515cdef87d3437fce)

.gitignore

@@ -47,7 +47,7 @@ results/
 
 # HuggingFace language packs (downloaded on-demand)
 # Exclude all HF packs to keep deployment size under 1GB
-packs/warbler-pack-hf-arxiv/
+packs/warbler-pack-hf-arxiv/*chunk*.jsonl
 packs/warbler-pack-hf-enterprise/
 packs/warbler-pack-hf-edustories/
 packs/warbler-pack-hf-manuals/

BUG_FIXES_DOCUMENTATION.md

@@ -1,252 +0,0 @@
-# Bug Fixes Documentation
-
-## Multi-Character Dialogue Segmentation Fault Fix
-
-**Date:** 2025-01-20  
-**Session:** 1251351  
-**Severity:** Critical  
-**Status:** Fixed
-
-### Problem Description
-
-The `agentlans/multi-character-dialogue` dataset processing was causing a segmentation fault (core dumped) after successfully processing 5404 examples. The crash occurred during the `transform_multi_character()` method execution when running:
-
-```bash
-python3 warbler_cda/utils/hf_warbler_ingest.py ingest -d all
-```
-
-**Error Output:**
-
-```log
-🔄 Processing multi-character...
-INFO:__main__:Loading agentlans/multi-character-dialogue...
-Generating train split: 5404 examples [00:00, 6239.66 examples/s]
-Segmentation fault (core dumped)
-```
-
-### Root Cause Analysis
-
-The segmentation fault was caused by multiple factors:
-
-1. **Insufficient Error Handling**: The iteration loop lacked comprehensive error handling for memory errors, recursion errors, and malformed data structures.
-
-2. **Unbounded Data Processing**: No limits on conversation size, message length, or character list size, leading to potential memory exhaustion.
-
-3. **Unsafe Type Assumptions**: The code assumed data structures would always be well-formed dictionaries and lists without validation.
-
-4. **Missing Bounds Checking**: No validation of dataset split existence or item count before iteration.
-
-5. **Lack of Progress Monitoring**: No logging to identify which specific item caused the crash.
-
-6. **Unsafe JSON Serialization**: Character lists could contain deeply nested or circular structures causing recursion errors.
-
-### Changes Made
-
-#### File: `warbler-cda-package/warbler_cda/utils/hf_warbler_ingest.py`
-
-**Location:** `transform_multi_character()` method (lines ~150-200) and `_create_multi_char_content()` helper (lines ~420-450)
-
-#### In `transform_multi_character():`
-
-1. **Comprehensive Error Handling**:
-   - Added outer try-except block wrapping entire iteration
-   - Separate handling for `MemoryError`, `RecursionError`, `KeyboardInterrupt`, and general exceptions
-   - Early exit on critical errors to prevent crashes
-
-2. **Dataset Validation**:
-   - Check for 'train' split existence before iteration
-   - Get total item count for progress tracking
-   - Validate dataset is not empty
-
-3. **Progress Monitoring**:
-   - Added periodic logging every 1000 items
-   - Shows progress: `Processed X/Y items, created Z documents`
-   - Helps identify crash location in future debugging
-
-4. **Item-Level Validation**:
-   - Check if item is None
-   - Validate item is a dictionary
-   - Type validation for all fields (setting, characters, conversation)
-   - Sanitize non-string/non-list values
-
-5. **Conversation Structure Validation**:
-   - Check first 10 messages for valid structure
-   - Skip items with malformed conversations
-   - Prevent processing of corrupted data
-
-6. **Content Creation Safety**:
-   - Wrap `_create_multi_char_content()` call in try-except
-   - Provide fallback content on error
-   - Prevent single item from crashing entire process
-
-7. **Metadata Safety**:
-   - Use `isinstance()` checks before calling `len()`
-   - Default to 0 for invalid list types
-   - Prevent crashes from unexpected metadata values
-
-#### In `_create_multi_char_content():`
-
-1. **Input Validation**:
-   - Check if item is a dictionary
-   - Return error message for invalid input
-
-2. **Conversation Processing Limits**:
-   - Maximum 1000 conversation items processed
-   - Truncate messages longer than 5000 characters
-   - Add truncation notice if conversation exceeds limit
-
-3. **Message-Level Error Handling**:
-   - Try-except around each message processing
-   - Handle None messages gracefully
-   - Support dict and string message formats
-   - Log type name for unsupported formats
-
-4. **Critical Error Detection**:
-   - Break on `RecursionError` or `MemoryError`
-   - Prevent infinite loops or memory exhaustion
-   - Return partial results instead of crashing
-
-5. **Field Size Limits**:
-   - Setting: max 2000 characters
-   - Setting after: max 2000 characters
-   - Characters list: max 100 items
-   - Total content: max 50000 characters
-
-6. **Safe JSON Serialization**:
-   - Try-except around `json.dumps()`
-   - Fallback to `str()` if JSON fails
-   - Limit character list size before serialization
-   - Use `ensure_ascii=False` for Unicode support
-
-7. **Final Safety Checks**:
-   - Validate total content size
-   - Truncate if exceeds 50KB
-   - Return error message if final build fails
-
-### Testing Results
-
-The fixes were designed to handle the following scenarios:
-
-1. **Large Conversations**: Conversations with thousands of messages are now truncated safely
-2. **Malformed Data**: Invalid message structures are skipped with warnings
-3. **Memory Issues**: Processing stops gracefully on memory errors
-4. **Recursion Errors**: Deep nesting is detected and handled
-5. **Type Mismatches**: All fields are validated and sanitized
-6. **Progress Tracking**: Crash location can be identified from logs
-
-### Expected Behavior After Fix
-
-When running:
-
-```bash
-python3 warbler_cda/utils/hf_warbler_ingest.py ingest -d multi-character
-```
-
-Expected output:
-
-```log
-🔄 Processing multi-character...
-INFO:__main__:Loading agentlans/multi-character-dialogue...
-INFO:__main__:Processing 5404 multi-character dialogue items...
-INFO:__main__:Processed 1000/5404 items, created 950 documents
-INFO:__main__:Processed 2000/5404 items, created 1900 documents
-INFO:__main__:Processed 3000/5404 items, created 2850 documents
-INFO:__main__:Processed 4000/5404 items, created 3800 documents
-INFO:__main__:Processed 5000/5404 items, created 4750 documents
-INFO:__main__:✓ Transformed 5100 multi-character entries
-INFO:__main__:✓ Created Warbler pack: warbler-pack-hf-multi-character with 5100 documents
-✓ 5100 documents created
-```
-
-### Verification Steps
-
-To verify the fix works correctly:
-
-1. **Test Multi-Character Dataset Only**:
-
-   ```bash
-   cd warbler-cda-package
-   python3 warbler_cda/utils/hf_warbler_ingest.py ingest -d multi-character
-   ```
-
-2. **Test All Datasets**:
-
-   ```bash
-   cd warbler-cda-package
-   python3 warbler_cda/utils/hf_warbler_ingest.py ingest -d all
-   ```
-
-3. **Check Output**:
-   - No segmentation fault
-   - Progress logs appear every 1000 items
-   - Final document count is reported
-   - Warbler pack is created successfully
-
-4. **Verify Pack Contents**:
-
-   ```bash
-   ls -lh packs/warbler-pack-hf-multi-character/
-   cat packs/warbler-pack-hf-multi-character/package.json
-   head -n 50 packs/warbler-pack-hf-multi-character/warbler-pack-hf-multi-character.jsonl
-   ```
-
-### Related Files Modified
-
-- `warbler-cda-package/warbler_cda/utils/hf_warbler_ingest.py`
-  - `transform_multi_character()` method
-  - `_create_multi_char_content()` helper method
-
-### Backward Compatibility
-
-All changes are backward compatible:
-
-- No API changes
-- No parameter changes
-- No output format changes
-- Only adds defensive programming and error handling
-
-### Performance Impact
-
-Minimal performance impact:
-
-- Progress logging: ~0.1% overhead
-- Type validation: ~1% overhead
-- Size limits prevent memory issues, improving overall performance
-- Early exit on errors prevents wasted processing time
-
-### Future Improvements
-
-1. **Configurable Limits**: Make size limits configurable via parameters
-2. **Streaming Processing**: Process large datasets in chunks to reduce memory usage
-3. **Parallel Processing**: Use multiprocessing for faster dataset transformation
-4. **Better Error Recovery**: Attempt to fix malformed data instead of skipping
-5. **Detailed Statistics**: Track and report skip reasons and error types
-
-### Lessons Learned
-
-1. **Always Validate Input**: Never assume data structures are well-formed
-2. **Set Bounds**: Limit processing of unbounded data structures
-3. **Monitor Progress**: Add logging to identify crash locations
-4. **Handle Critical Errors**: Catch memory and recursion errors explicitly
-5. **Fail Gracefully**: Return partial results instead of crashing
-6. **Test Edge Cases**: Test with malformed, large, and nested data
-
-### References
-
-- HuggingFace Dataset: <https://huggingface.co/datasets/agentlans/multi-character-dialogue>
-- Python Memory Management: <https://docs.python.org/3/c-api/memory.html>
-- Segmentation Fault Debugging: <https://wiki.python.org/moin/DebuggingWithGdb>
-
----
-
-## Summary
-
-The multi-character dialogue segmentation fault has been fixed through comprehensive defensive programming, including:
-
-- Robust error handling for memory and recursion errors
-- Input validation and type checking
-- Size limits on all data structures
-- Progress monitoring and logging
-- Graceful degradation on errors
-
-The dataset now processes successfully without crashes, creating valid Warbler packs for NPC training.

COMPLETION_SUMMARY.md

@@ -1,376 +0,0 @@
-# Completion Summary: MIT-Licensed Datasets Testing & Implementation
-
-**Project**: warbler-cda-package integration with new MIT-licensed HuggingFace datasets  
-**Commit**: e7cff201eabf06f7c2950bc7545723d20997e73d  
-**Date**: November 8, 2025  
-**Status**: ✅ **COMPLETE - READY FOR TESTING**
-
----
-
-## 🎯 Objective Achieved
-
-Integrated 6 new MIT-licensed HuggingFace datasets into warbler-cda-package with:
-
-- ✅ Complete transformer implementations
-- ✅ Comprehensive test suite (31 tests)
-- ✅ Production-ready code
-- ✅ Full documentation
-- ✅ Backward compatibility
-
----
-
-## 📋 Deliverables
-
-### 1. Core Implementation
-
-**File**: `warbler_cda/utils/hf_warbler_ingest.py` (290 → 672 lines)
-
-**Added Transformers** (6):
-
-- `transform_arxiv()` - 2.55M scholarly papers
-- `transform_prompt_report()` - 83 prompt engineering docs
-- `transform_novels()` - 20 generated novels with auto-chunking
-- `transform_manuals()` - 52 technical manuals
-- `transform_enterprise()` - 283 business benchmarks
-- `transform_portuguese_education()` - 21 multilingual education texts
-
-**Added Helpers** (7):
-
-- `_create_arxiv_content()`
-- `_create_prompt_report_content()`
-- `_create_novel_content()`
-- `_create_manual_content()`
-- `_create_enterprise_content()`
-- `_create_portuguese_content()`
-- `_chunk_text()` - Text splitting utility
-
-**Updated Components**:
-
-- CLI `ingest()` command with new datasets + `--arxiv-limit` parameter
-- CLI `list_available()` command with new dataset descriptions
-- All transformers include MIT license metadata
-
-### 2. Comprehensive Test Suite
-
-**File**: `tests/test_new_mit_datasets.py` (413 lines, 31 tests)
-
-**Test Coverage**:
-
-- ✅ Transformer method existence (6 tests)
-- ✅ Output format validation (6 tests)
-- ✅ Metadata field requirements (6 tests)
-- ✅ Dataset-specific features (12 tests)
-- ✅ Integration with Warbler format (2 tests)
-- ✅ Performance benchmarks (1 test)
-- ✅ End-to-end capabilities (1 test)
-
-### 3. Documentation
-
-**Files Created**:
-
-- `VALIDATION_REPORT_MIT_DATASETS.md` - Comprehensive validation report
-- `IMPLEMENTATION_SUMMARY_MIT_DATASETS.md` - Technical implementation details
-- `COMPLETION_SUMMARY.md` - This file
-
----
-
-## 🚀 Key Features Implemented
-
-### Data Transformers
-
-Each transformer includes:
-
-- Full HuggingFace dataset integration
-- Warbler document structure generation
-- MIT license compliance
-- FractalStat realm/activity level metadata
-- Dataset-specific optimizations
-
-### Notable Features
-
-| Feature | Details |
-|---------|---------|
-| **arXiv Limit** | `--arxiv-limit` prevents 2.55M paper overload |
-| **Novel Chunking** | Auto-splits long texts (~1000 words/chunk) |
-| **Error Handling** | Try-catch with graceful failure messages |
-| **CLI Integration** | Seamless command-line interface |
-| **Metadata** | All docs include license, realm, activity level |
-| **Backward Compat** | Legacy datasets still supported |
-
-### Testing Strategy
-
-- **Unit Tests**: Each transformer independently
-- **Integration Tests**: Pack creation and document format
-- **Performance Tests**: Large dataset handling
-- **Mocking**: HuggingFace API calls mocked for reliability
-
----
-
-## 📊 Implementation Metrics
-
-| Metric | Value |
-|--------|-------|
-| **Lines Added** | 382 |
-| **Transformers** | 6 new |
-| **Helper Methods** | 7 new |
-| **Test Cases** | 31 |
-| **MIT Datasets** | 6 (2.55M+ docs total) |
-| **Files Modified** | 1 |
-| **Files Created** | 4 |
-| **Documentation Pages** | 3 |
-
----
-
-## 🔄 TDD Process Followed
-
-### Step 1: Context Alignment ✅
-
-- Commit e7cff201 analyzed
-- Project structure understood
-- Historical requirements identified
-
-### Step 2: Test First ✅
-
-- Comprehensive test suite created
-- All failure cases identified
-- Mock implementations designed
-
-### Step 3: Code Implementation ✅
-
-- All 6 transformers implemented
-- All 7 helpers implemented
-- CLI updated
-- Error handling added
-
-### Step 4: Best Practices ✅
-
-- Type hints throughout
-- Comprehensive docstrings
-- Consistent error handling
-- Metadata standardization
-- Performance optimization
-
-### Step 5: Validation ✅
-
-- Code structure verified
-- Syntax correctness confirmed
-- File structure validated
-- CLI integration tested
-- Backward compatibility verified
-
-### Step 6: Closure ✅
-
-- **The scroll is complete; tested, proven, and woven into the lineage.**
-
----
-
-## 📦 Usage Examples
-
-### Basic Usage
-
-```bash
-# Ingest single dataset
-cd warbler-cda-package
-python -m warbler_cda.utils.hf_warbler_ingest ingest -d arxiv
-
-# With size limit
-python -m warbler_cda.utils.hf_warbler_ingest ingest -d arxiv --arxiv-limit 1000
-
-# Multiple datasets
-python -m warbler_cda.utils.hf_warbler_ingest ingest \
-  -d arxiv --arxiv-limit 10000 \
-  -d prompt-report \
-  -d novels
-```
-
-### Test Execution
-
-```bash
-# Run all tests
-pytest tests/test_new_mit_datasets.py -v
-
-# Run specific transformer tests
-pytest tests/test_new_mit_datasets.py::TestArxivPapersTransformer -v
-
-# With coverage report
-pytest tests/test_new_mit_datasets.py --cov=warbler_cda
-```
-
----
-
-## ✅ Quality Assurance Checklist
-
-### Code Quality
-
-- [x] Type hints on all methods
-- [x] Docstrings on all functions
-- [x] Consistent code style
-- [x] Error handling present
-- [x] No hard-coded magic numbers
-- [x] Meaningful variable names
-
-### Testing
-
-- [x] Unit tests for each transformer
-- [x] Integration tests
-- [x] Performance tests
-- [x] Edge case handling
-- [x] Mock data for reliability
-- [x] 31 test cases total
-
-### Documentation
-
-- [x] Docstrings in code
-- [x] Implementation summary
-- [x] Validation report
-- [x] Usage examples
-- [x] Integration guide
-- [x] Deployment notes
-
-### Integration
-
-- [x] Warbler document format compliance
-- [x] FractalStat metadata generation
-- [x] Pack creation integration
-- [x] CLI command updates
-- [x] Backward compatibility maintained
-- [x] License compliance (MIT)
-
----
-
-## 🎓 Learning Resources in Codebase
-
-### For Understanding the Implementation
-
-1. `warbler_cda/utils/hf_warbler_ingest.py` - Main transformer code
-2. `tests/test_new_mit_datasets.py` - Test patterns and examples
-3. `warbler_cda/retrieval_api.py` - How documents are used
-4. `warbler_cda/pack_loader.py` - Pack format details
-
-### For Integration
-
-1. `IMPLEMENTATION_SUMMARY_MIT_DATASETS.md` - Technical details
-2. `VALIDATION_REPORT_MIT_DATASETS.md` - Features and performance
-3. CLI help: `python -m warbler_cda.utils.hf_warbler_ingest list-available`
-
----
-
-## 🔍 What to Test Next
-
-### Immediate Testing
-
-```bash
-# 1. Verify CLI works
-python -m warbler_cda.utils.hf_warbler_ingest list-available
-
-# 2. Test single dataset ingestion
-python -m warbler_cda.utils.hf_warbler_ingest ingest -d prompt-report
-
-# 3. Run full test suite
-pytest tests/test_new_mit_datasets.py -v
-
-# 4. Test integration with retrieval API
-python -c "from warbler_cda.retrieval_api import RetrievalAPI; api = RetrievalAPI(); print('✓ Integration OK')"
-```
-
-### Integration Testing
-
-1. Load created packs with `pack_loader.py`
-2. Add documents to `RetrievalAPI`
-3. Verify FractalStat coordinate generation
-4. Test hybrid retrieval scoring
-
-### Performance Testing
-
-1. Large arXiv ingestion (10k papers)
-2. Novel chunking performance
-3. Memory usage under load
-4. Concurrent ingestion
-
----
-
-## 📞 Support & Troubleshooting
-
-### Common Issues
-
-**Issue**: HuggingFace API rate limiting
-
-- **Solution**: Use `--arxiv-limit` to control ingestion size
-
-**Issue**: Memory exhaustion with large datasets
-
-- **Solution**: Use smaller `--arxiv-limit` or ingest in batches
-
-**Issue**: Missing dependencies
-
-- **Solution**: `pip install datasets transformers`
-
-**Issue**: Tests fail with mock errors
-
-- **Solution**: Ensure unittest.mock is available (included in Python 3.3+)
-
----
-
-## 🎯 Next Actions
-
-### For Development Team
-
-1. ✅ Review implementation summary
-2. ✅ Run test suite in development environment
-3. ⏳ Test with actual HuggingFace API
-4. ⏳ Validate pack loading
-5. ⏳ Performance benchmark
-6. ⏳ Staging environment deployment
-
-### For DevOps
-
-1. ⏳ Set up ingestion pipeline
-2. ⏳ Configure arXiv limits
-3. ⏳ Schedule dataset updates
-4. ⏳ Monitor ingestion jobs
-5. ⏳ Archive old packs
-
-### For Documentation
-
-1. ⏳ Update README with new datasets
-2. ⏳ Create usage guide
-3. ⏳ Add to deployment documentation
-4. ⏳ Update architecture diagram
-
----
-
-## 🏆 Success Criteria Met
-
-✅ **All 6 transformers implemented and tested**
-✅ **31 comprehensive test cases created**
-✅ **MIT license compliance verified**
-✅ **Backward compatibility maintained**
-✅ **Production-ready error handling**
-✅ **Full documentation provided**
-✅ **CLI interface complete**
-✅ **Performance optimized**
-✅ **Code follows best practices**
-✅ **Ready for staging validation**
-
----
-
-## 📝 Sign-Off
-
-**Status**: ✅ **IMPLEMENTATION COMPLETE**
-
-The new MIT-licensed datasets are fully integrated into warbler-cda-package with:
-
-- Comprehensive transformers for 6 datasets
-- 31 test cases covering all functionality
-- Production-ready code with error handling
-- Full documentation and integration guides
-- Backward compatibility maintained
-
-**The scrolls are complete; tested, proven, and woven into the lineage.**
-
----
-
-**Project Lead**: Zencoder AI Assistant  
-**Date Completed**: November 8, 2025  
-**Branch**: e7cff201eabf06f7c2950bc7545723d20997e73d  
-**Review Status**: Ready for Team Validation

CONTRIBUTING.md

@@ -1,69 +0,0 @@
-# Contributing to Warbler CDA
-
-Thank you for your interest in contributing to Warbler CDA!
-
-## Development Setup
-
-1. Clone the repository:
-
-   ```bash
-   git clone https://gitlab.com/tiny-walnut-games/the-seed.git
-   cd the-seed/warbler-cda-package
-   ```
-
-2. Run setup:
-
-   ```bash
-   ./setup.sh
-   ```
-
-3. Install development dependencies:
-
-   ```bash
-   pip install -e ".[dev]"
-   ```
-
-## Running Tests
-
-```bash
-# Run all tests
-pytest
-
-# Run with coverage
-pytest --cov=warbler_cda --cov-report=html
-
-# Run specific test
-pytest tests/test_retrieval_api.py -v
-```
-
-## Code Style
-
-We use:
-
-- **Black** for code formatting
-- **Flake8** for linting
-- **MyPy** for type checking
-
-```bash
-# Format code
-black warbler_cda/
-
-# Lint
-flake8 warbler_cda/
-
-# Type check
-mypy warbler_cda/
-```
-
-## Pull Request Process
-
-1. Create a feature branch
-2. Make your changes
-3. Add tests for new functionality
-4. Ensure all tests pass
-5. Update documentation
-6. Submit a merge request
-
-## Questions?
-
-Open an issue on GitLab: <https://gitlab.com/tiny-walnut-games/the-seed/-/issues>

DEPLOYMENT.md

@@ -1,98 +0,0 @@
-# Warbler CDA HuggingFace Deployment
-
-This directory contains the Warbler CDA package prepared for HuggingFace deployment.
-
-## Quick Start
-
-### Local Testing
-
-```bash
-cd warbler-cda-package
-
-# Install dependencies
-pip install -r requirements.txt
-
-# Install package in development mode
-pip install -e .
-
-# Run Gradio demo
-python app.py
-```
-
-### Deploy to HuggingFace Space
-
-#### Option 1: Manual Deployment
-
-```bash
-# Install HuggingFace CLI
-pip install huggingface_hub
-
-# Login
-huggingface-cli login
-
-# Upload to Space
-huggingface-cli upload YOUR_USERNAME/warbler-cda . --repo-type=space
-```
-
-#### Option 2: GitLab CI/CD (Automated)
-
-1. Set up HuggingFace token in GitLab CI/CD variables:
-   - Go to Settings > CI/CD > Variables
-   - Add variable `HF_TOKEN` with your HuggingFace token
-   - Add variable `HF_SPACE_NAME` with your Space name (e.g., `username/warbler-cda`)
-
-2. Push to main branch or create a tag:
-
-   ```bash
-   git tag v0.1.0
-   git push origin v0.1.0
-   ```
-
-3. The pipeline will automatically sync to HuggingFace!
-
-## Package Structure
-
-```none
-warbler-cda-package/
-├── warbler_cda/              # Main package
-│   ├── __init__.py
-│   ├── retrieval_api.py      # Core RAG API
-│   ├── semantic_anchors.py   # Semantic memory
-│   ├── fractalstat_rag_bridge.py   # FractalStat hybrid scoring
-│   ├── embeddings/           # Embedding providers
-│   ├── api/                  # FastAPI service
-│   └── utils/                # Utilities
-├── app.py                    # Gradio demo for HF Space
-├── requirements.txt          # Dependencies
-├── pyproject.toml            # Package metadata
-├── README.md                 # Documentation
-└── LICENSE                   # MIT License
-```
-
-## Features
-
-- **Semantic Search**: Natural language document retrieval
-- **FractalStat Addressing**: 7-dimensional multi-modal scoring
-- **Hybrid Scoring**: Combines semantic + FractalStat for superior results
-- **Production API**: FastAPI service with concurrent query support
-- **CLI Tools**: Command-line interface for management
-- **HF Integration**: Direct dataset ingestion
-
-## Testing
-
-```bash
-# Run tests
-pytest
-
-# Run specific experiments
-python -m warbler_cda.fractalstat_experiments
-```
-
-## Documentation
-
-See [README.md](README.md) for full documentation.
-
-## Support
-
-- **Issues**: <https://gitlab.com/tiny-walnut-games/the-seed/-/issues>
-- **Discussions**: <https://gitlab.com/tiny-walnut-games/the-seed/-/merge_requests>

DOCKER_BUILD_PERFORMANCE.md

@@ -1,74 +0,0 @@
-# Warbler CDA Docker Build Performance
-
-## Build Configuration
-
-- **Dockerfile**: Minimal FractalStat testing setup
-- **Base Image**: python:3.11-slim
-- **Build Context Optimization**: .dockerignore excludes cache files and large directories
-- **Dependency Strategy**: Minimal ML dependencies for FractalStat testing
-
-## Performance Measurements
-
-### Optimized Build Results (Windows with WSL)
-
-```none
-✅ FINAL OPTIMIZED BUILD: 38.4 seconds (~40 seconds)
-├── Base Image Pull: 3.7 seconds
-├── System Dependencies: 20.5 seconds (git install)
-├── Dependencies (pip install): 5.8 seconds
-│   - pydantic>=2.0.0 (only needed library!)
-│   - pytest>=7.0.0 (testing framework)
-├── Code Copy: 0.2 seconds
-├── Layer Export: 6.4 seconds
-└── Image Unpack: 1.7 seconds
-```
-
-### Performance Improvement Achieved
-
-**🚀 Optimization Results:**
-
-- **Build Time Reduction**: 94% faster (601.6s → 38.4s)
-- **Pip Install Reduction**: 98% faster (295.6s → 5.8s)
-- **Context Size**: 556B (highly optimized .dockerignore - final reduction)
-- **Expected Image Size**: ~250MB (vs 12.29GB bloated)
-
-**📊 Bottleneck Eliminated:**
-
-- Removed PyTorch/Transformers dependency chain causing 98% of bloat
-- FractalStat modules require **zero** ML libraries
-- Pure Python with dataclasses, enums, typing, json
-
-**🔍 Root Cause Identified:**
-Original bloat caused by `transformers[torch]` pulling:
-
-- PyTorch CPU (~1GB)
-- 100+ optional dependencies (~11GB)
-- All unnecessary for FractalStat core functionality
-
-## Recommendations for Faster Builds
-
-### For Development Builds
-
-1. **Use cached layers** - Base image and system dependencies rarely change
-2. **Separate dependency layers** - Cache pip installs when code changes frequently
-3. **Minimal dependencies** - Only install what's needed for testing FractalStat specifically
-
-### For Production Builds
-
-1. **Multi-stage builds** - Separate testing and runtime images
-2. **Dependency optimization** - Use Docker layer caching more effectively
-3. **Alternative base images** - Consider smaller Python images or compiled binaries
-
-## Testing Results
-
-- ✅ All 70 FractalStat entity tests pass
-- ✅ FractalStat coordinates and entities work correctly
-- ✅ RAG bridge integration functions properly
-- ✅ Container startup and imports work as expected
-
-## Performance Notes
-
-- First-time build: ~10 minutes (acceptable for ML dependencies)
-- Subsequent builds: Should be faster with Docker layer caching
-- Network dependency: Download times vary by internet connection
-- WSL overhead: Minimal impact on overall build time

HUGGINGFACE_DEPLOYMENT_GUIDE.md

@@ -1,279 +0,0 @@
-# Warbler CDA - HuggingFace Deployment Complete Guide
-
-## 🎯 What Was Created
-
-A complete, production-ready Python package extracted from The Seed project, specifically designed for HuggingFace deployment.
-
-### Package Contents
-
-- **25 Python files** with 8,645 lines of code
-- **21 core RAG/FractalStat files** from the original system
-- **11 infrastructure files** for deployment
-- **Package size**: 372KB (source), ~2GB with dependencies
-
-## 🚀 Deployment Options
-
-### Option 1: Automatic GitLab CI/CD → HuggingFace (RECOMMENDED)
-
-This is the **kudos-worthy** automatic sync pipeline!
-
-#### Setup (One-time)
-
-1. **Get HuggingFace Token**
-   - Go to <https://huggingface.co/settings/tokens>
-   - Create a new token with "write" access
-   - Copy the token
-
-2. **Configure GitLab CI/CD**
-   - Go to <https://gitlab.com/tiny-walnut-games/the-seed/-/settings/ci_cd>
-   - Expand "Variables"
-   - Add variable:
-     - Key: `HF_TOKEN`
-     - Value: (paste your HuggingFace token)
-     - Masked: ✓ (checked)
-   - Add variable:
-     - Key: `HF_SPACE_NAME`
-     - Value: `your-username/warbler-cda` (customize this)
-
-3. **Create HuggingFace Space**
-   - Go to <https://huggingface.co/new-space>
-   - Name: `warbler-cda`
-   - SDK: Gradio
-   - Visibility: Public or Private
-   - Click "Create Space"
-
-### Deploy
-
-#### **First: Verify paths**
-
-```bash
-# Ensure that the following is on path for most executables to be available
-echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
-
-# Restart the terminal
-source ~/.bashrc
-```
-
-#### **Method A: Tag-based (Automatic)**
-
-```bash
-git add warbler-cda-package/
-git commit -m "Add Warbler CDA HuggingFace package"
-git tag v0.1.0
-git push origin main --tags
-```
-
-The pipeline will automatically deploy to HuggingFace! ✨
-
-#### **Method B: Manual Trigger**
-
-```bash
-git add warbler-cda-package/
-git commit -m "Add Warbler CDA HuggingFace package"
-git push origin main
-```
-
-Then go to CI/CD > Pipelines and manually trigger the `deploy-huggingface` job.
-
-#### What Happens
-
-1. GitLab CI detects the push/tag
-2. Runs the `deploy-huggingface` job
-3. Installs `huggingface_hub`
-4. Logs in with your token
-5. Syncs `warbler-cda-package/` to your Space
-6. Your Space is live! 🎉
-
-### Option 2: Manual HuggingFace Upload
-
-```bash
-cd warbler-cda-package
-
-# Install HuggingFace CLI
-pip install huggingface_hub
-
-# Login
-huggingface-cli login
-
-# Upload to Space
-huggingface-cli upload your-username/warbler-cda . --repo-type=space --commit-message="Initial release"
-```
-
-### Option 3: Local Testing First
-
-```bash
-cd warbler-cda-package
-
-# Setup
-./setup.sh
-
-# Run Gradio demo
-python app.py
-```
-
-Open <http://localhost:7860> to test locally before deploying.
-
-## 🔧 Configuration
-
-### Environment Variables (Optional)
-
-For the HuggingFace Space, you can set these in Space Settings:
-
-- `OPENAI_API_KEY` - For OpenAI embeddings (optional)
-- `MAX_RESULTS` - Default max results (default: 10)
-- `ENABLE_FractalStat` - Enable FractalStat hybrid scoring (default: true)
-
-### Customizing the Space
-
-Edit `app.py` to customize:
-
-- Sample documents
-- UI layout
-- Default settings
-- Branding
-
-## 📊 Features in the Demo
-
-The Gradio demo includes:
-
-1. **Query Tab**
-   - Semantic search
-   - FractalStat hybrid scoring toggle
-   - Adjustable weights
-   - Real-time results
-
-2. **Add Document Tab**
-   - Add custom documents
-   - Set realm type/label
-   - Immediate indexing
-
-3. **System Stats Tab**
-   - Performance metrics
-   - Cache statistics
-   - Quality distribution
-
-4. **About Tab**
-   - System documentation
-   - FractalStat explanation
-   - Links to resources
-
-## 🧪 Testing the Deployment
-
-After deployment, test these queries:
-
-1. **Basic Semantic**: "wisdom about courage"
-2. **Technical**: "how does FractalStat work"
-3. **Narrative**: "ancient library keeper"
-4. **Pattern**: "connections between events"
-
-Expected results:
-
-- 3-5 relevant documents per query
-- Relevance scores > 0.6
-- Sub-second response time
-
-## 🐛 Troubleshooting
-
-### Pipeline Fails
-
-**Error**: "HF_TOKEN not set"
-
-- **Fix**: Add HF_TOKEN to GitLab CI/CD variables
-
-**Error**: "Space not found"
-
-- **Fix**: Create the Space on HuggingFace first, or update HF_SPACE_NAME
-
-### Space Fails to Build
-
-**Error**: "Module not found"
-
-- **Fix**: Check requirements.txt includes all dependencies
-
-**Error**: "Out of memory"
-
-- **Fix**: HuggingFace Spaces have memory limits. Consider using CPU-only versions of PyTorch
-
-### Gradio Not Loading
-
-**Error**: "Application startup failed"
-
-- **Fix**: Check app.py for syntax errors
-- **Fix**: Ensure all imports are correct
-
-## 📈 Monitoring
-
-### GitLab CI/CD
-
-Monitor deployments at:
-<https://gitlab.com/tiny-walnut-games/the-seed/-/pipelines>
-
-### HuggingFace Space
-
-Monitor your Space at:
-<https://huggingface.co/spaces/YOUR_USERNAME/warbler-cda>
-
-Check:
-
-- Build logs
-- Runtime logs
-- Usage statistics
-
-## 🔄 Updating the Space
-
-### Automatic (via GitLab CI/CD)
-
-Just push changes to main or create a new tag:
-
-```bash
-git add warbler-cda-package/
-git commit -m "Update: improved query performance"
-git push origin main
-```
-
-Or for versioned releases:
-
-```bash
-git tag v0.1.1
-git push origin v0.1.1
-```
-
-### Manual
-
-```bash
-cd warbler-cda-package
-huggingface-cli upload your-username/warbler-cda . --repo-type=space --commit-message="Update"
-```
-
-## 📚 Additional Resources
-
-- **HuggingFace Spaces Docs**: <https://huggingface.co/docs/hub/spaces>
-- **Gradio Docs**: <https://gradio.app/docs/>
-- **GitLab CI/CD Docs**: <https://docs.gitlab.com/ee/ci/>
-
-## ✅ Checklist
-
-Before deploying:
-
-- [ ] HF_TOKEN set in GitLab CI/CD variables
-- [ ] HF_SPACE_NAME set in GitLab CI/CD variables
-- [ ] HuggingFace Space created
-- [ ] Package tested locally (`./setup.sh && python app.py`)
-- [ ] All files committed to Git
-- [ ] README.md reviewed and customized
-
-After deploying:
-
-- [ ] Space builds successfully
-- [ ] Gradio interface loads
-- [ ] Sample queries work
-- [ ] Add Document feature works
-- [ ] System stats display correctly
-
-## 🎉 Success
-
-Once deployed, your Warbler CDA Space will be live at:
-
-**<https://huggingface.co/spaces/YOUR_USERNAME/warbler-cda>**
-
-Share it with the world! 🌍

IMPLEMENTATION_SUMMARY.md

@@ -1,185 +0,0 @@
-# Warbler CDA Package - Implementation Summary
-
-## ✅ Completed Tasks
-
-### Phase 1: Directory Structure
-
-- [x] Created `warbler-cda-package/` root directory
-- [x] Created `warbler_cda/` main package directory
-- [x] Created `warbler_cda/embeddings/` subdirectory
-- [x] Created `warbler_cda/api/` subdirectory
-- [x] Created `warbler_cda/utils/` subdirectory
-
-### Phase 2: Core Files (21 files)
-
-- [x] Copied and transformed all 9 core RAG files
-- [x] Copied and transformed all 4 FractalStat files
-- [x] Copied and transformed all 5 embedding files
-- [x] Copied and transformed all 3 API files
-- [x] Copied and transformed all 3 utility files
-
-### Phase 3: Infrastructure
-
-- [x] Created `__init__.py` files for all modules
-- [x] Created `requirements.txt` with all dependencies
-- [x] Created `pyproject.toml` with package metadata
-- [x] Created comprehensive `README.md`
-- [x] Created `app.py` with Gradio demo
-- [x] Created `.gitignore`
-- [x] Created `LICENSE` (MIT)
-
-### Phase 4: Import Transformations
-
-- [x] Transformed all `seed.engine` imports to `warbler_cda`
-- [x] Converted relative imports to absolute
-- [x] Removed privacy hooks (not needed for HF)
-- [x] Verified no untransformed imports remain
-
-### Phase 5: CI/CD Pipeline
-
-- [x] Added `deploy-huggingface` stage to `.gitlab-ci.yml`
-- [x] Configured automatic sync on tags
-- [x] Configured manual trigger for main branch
-- [x] Added environment variables support (HF_TOKEN, HF_SPACE_NAME)
-
-### Phase 6: Documentation
-
-- [x] Created `DEPLOYMENT.md` - Deployment guide
-- [x] Created `CONTRIBUTING.md` - Contribution guidelines
-- [x] Created `QUICKSTART.md` - Quick start guide
-- [x] Created `HUGGINGFACE_DEPLOYMENT_GUIDE.md` - Complete HF guide
-- [x] Created `PACKAGE_MANIFEST.md` - File listing
-- [x] Created `README_HF.md` - HuggingFace Space config
-
-### Phase 7: Helper Scripts
-
-- [x] Created `setup.sh` - Quick setup script
-- [x] Created `transform_imports.sh` - Import transformation
-- [x] Created `verify_package.sh` - Package verification
-- [x] Created `Dockerfile` - Docker deployment
-- [x] Created `docker-compose.yml` - Multi-service deployment
-
-### Phase 8: Verification
-
-- [x] Verified all 25 Python files present
-- [x] Verified all imports transformed
-- [x] Verified package structure correct
-- [x] Verified 8,645 lines of code
-- [x] Verified 372KB package size
-
-### Phase 9: Issue Documentation
-
-- [x] Added comprehensive comment to Issue #1
-- [x] Documented all features and setup steps
-
-## 📊 Final Statistics
-
-- **Total Files Created**: 36 files
-- **Python Files**: 25 files
-- **Lines of Code**: 8,645 LOC
-- **Package Size**: 372KB (source only)
-- **With Dependencies**: ~2GB
-- **Time Taken**: ~30 minutes
-
-## 🎯 Key Features Delivered
-
-1. ✅ **Complete RAG System** - All 21 core files extracted
-2. ✅ **FractalStat Integration** - Full hybrid scoring support
-3. ✅ **Production API** - FastAPI service ready
-4. ✅ **Gradio Demo** - Interactive HuggingFace Space
-5. ✅ **Automatic CI/CD** - GitLab → HuggingFace sync
-6. ✅ **Comprehensive Docs** - 6 documentation files
-7. ✅ **Helper Scripts** - 3 automation scripts
-8. ✅ **Docker Support** - Containerized deployment
-
-## 🏆 Bonus Features (Kudos!)
-
-### Automatic GitLab → HuggingFace Sync Pipeline
-
-The CI/CD pipeline automatically syncs the Warbler CDA package to HuggingFace:
-
-- **On Tags**: Automatic deployment (e.g., `v0.1.0`)
-- **On Main**: Manual trigger available
-- **Smart Caching**: Only uploads changed files
-- **Environment Support**: Configurable via GitLab variables
-
-This means you can:
-
-1. Make changes to `warbler-cda-package/`
-2. Commit and tag: `git tag v0.1.1 && git push --tags`
-3. Pipeline automatically deploys to HuggingFace
-4. Your Space updates automatically! 🎉
-
-### Additional Kudos Features
-
-- **Docker Support**: Full containerization with docker-compose
-- **Multiple Deployment Options**: Local, Docker, HuggingFace, PyPI
-- **Comprehensive Testing**: Verification scripts included
-- **Developer Experience**: Setup scripts, contribution guides
-- **Production Ready**: FastAPI service with concurrent queries
-
-## 🚀 Deployment Instructions
-
-### Quick Deploy (3 steps)
-
-1. **Set GitLab Variables**
-
-   ```ps1
-   HF_TOKEN = your_huggingface_token
-   HF_SPACE_NAME = username/warbler-cda
-   ```
-
-2. **Create HuggingFace Space**
-   - Go to <https://huggingface.co/new-space>
-   - Name: `warbler-cda`
-   - SDK: Gradio
-
-3. **Deploy**
-
-   ```bash
-   git tag v0.1.0
-   git push origin v0.1.0
-   ```
-
-Done! Your Space will be live at `https://huggingface.co/spaces/username/warbler-cda`
-
-## 📝 Next Steps
-
-1. **Test Locally**
-
-   ```bash
-   cd warbler-cda-package
-   ./setup.sh
-   python app.py
-   ```
-
-2. **Deploy to HuggingFace**
-   - Follow the 3-step guide above
-
-3. **Share**
-   - Share your Space URL
-   - Add to HuggingFace model hub
-   - Announce on social media
-
-4. **Iterate**
-   - Make improvements
-   - Push changes
-   - Pipeline auto-deploys!
-
-## 🎓 Learning Resources
-
-- **Gradio**: <https://gradio.app/docs/>
-- **HuggingFace Spaces**: <https://huggingface.co/docs/hub/spaces>
-- **FractalStat System**: See `warbler_cda/fractalstat_rag_bridge.py`
-- **RAG Architecture**: See `warbler_cda/retrieval_api.py`
-
-## 🏅 Achievement Unlocked
-
-✅ **Complete HuggingFace Package**  
-✅ **Automatic CI/CD Pipeline**  
-✅ **Production-Ready System**  
-✅ **Comprehensive Documentation**  
-✅ **Docker Support**  
-✅ **Multiple Deployment Options**  
-
-**Status**: 🎉 READY FOR DEPLOYMENT!

IMPLEMENTATION_SUMMARY_MIT_DATASETS.md

@@ -1,453 +0,0 @@
-# Implementation Summary: MIT-Licensed Datasets
-
-## Overview
-
-Added 7 new MIT-licensed dataset transformers to warbler-cda-package following commit e7cff201.
-Updated enterprise dataset from AST-FRI/EnterpriseBench to SustcZhangYX/ChatEnv.
-Enhanced PDF extraction for novels dataset.
-
----
-
-## Changes to `warbler_cda/utils/hf_warbler_ingest.py`
-
-### 1. New Transformer Methods Added
-
-#### `transform_arxiv(dataset_name, limit: Optional[int] = None)` - Lines 149-188
-
-- **Dataset**: nick007x/arxiv-papers (2.55M papers)
-- **Features**:
-  - Respects `limit` parameter to prevent memory overload
-  - Extracts: arxiv_id, title, authors, year, categories
-  - Realm: scholarly/arxiv
-  - Metadata includes year and categories
-- **Output**: List of Warbler documents
-
-#### `transform_prompt_report(dataset_name)` - Lines 190-230
-
-- **Dataset**: PromptSystematicReview/ThePromptReport (83 docs)
-- **Features**:
-  - Handles multiple dataset formats (list, dict with splits)
-  - Extracts: title, category
-  - Realm: methodological/prompt_engineering
-  - Activity level: 0.8 (high engagement)
-
-#### `transform_novels(dataset_name)` - Lines 232-280
-
-- **Dataset**: GOAT-AI/generated-novels (20 novels)
-- **Features**:
-  - **Auto-chunking**: Splits long texts into ~1000 word chunks
-  - **Enhanced PDF extraction**: Improved logging and error handling
-  - Supports multiple PDF field names: pdf, file, document, content, data
-  - Handles dict with 'bytes' key (HuggingFace format)
-  - Tracks chunk index and total
-  - Realm: narrative/generated_fiction
-  - Prevents token limit issues
-  - Metadata includes chunk_index, total_chunks, and content_available flag
-- **Note**: Requires pdfplumber for full text extraction. Dataset has no README for guidance.
-
-#### `transform_manuals(dataset_name)` - Lines 282-322
-
-- **Dataset**: nlasso/anac-manuals-23 (52 manuals)
-- **Features**:
-  - Extracts section count
-  - Realm: procedural/technical_manual
-  - Activity level: 0.7
-  - Preserves manual structure metadata
-
-#### `transform_enterprise(dataset_name)` - Lines 324-364
-
-- **Dataset**: SustcZhangYX/ChatEnv (software development chat)
-- **Features**:
-  - Extracts conversation/messages from collaborative coding scenarios
-  - Supports multiple field names: conversation, messages, chat, dialogue
-  - Realm: software_development/chatenv_collaboration
-  - Activity level: 0.8 (high engagement)
-  - Dialogue type: software_dev_chat
-- **Note**: Replaced AST-FRI/EnterpriseBench which had loading issues
-
-#### `transform_portuguese_education(dataset_name)` - Lines 366-406
-
-- **Dataset**: Solshine/Portuguese_Language_Education_Texts (21 docs)
-- **Features**:
-  - Language tagging (pt = Portuguese)
-  - Multilingual support
-  - Realm: educational/portuguese_language
-  - Portuguese content in helper method
-
-#### `transform_edustories(dataset_name)` - Lines 407-500
-
-- **Dataset**: MU-NLPC/Edustories-en (educational case studies, 1492 entries)
-- **Features**:
-  - **Structured case study format** with four main fields:
-    - `description`: Background/context of the classroom situation
-    - `anamnesis`: Detailed description of the situation
-    - `solution`: Teacher's intervention/approach
-    - `outcome`: Final state after intervention
-  - **Student metadata**: age/school year, hobbies, diagnoses, disorders
-  - **Teacher metadata**: approbation (subject areas), practice years
-  - **Annotation fields**:
-    - problems_annotated, solutions_annotated, implications_annotated
-    - problems_possible_annotated, solutions_possible_annotated, implications_possible_annotated
-  - **Entry tracking**: entry_id, annotator_id
-  - Realm: educational/educational_case_studies
-  - Activity level: 0.7
-  - Dialogue type: teaching_case_study
-  - Metadata includes: entry_id, student attributes, teacher attributes, all annotation fields
-
----
-
-### 2. New Helper Methods Added
-
-#### `_create_arxiv_content(item)` - Lines 439-449
-
-Formats arXiv paper with: Title, Authors, Year, Categories, Abstract
-
-#### `_create_prompt_report_content(item)` - Lines 451-459
-
-Formats prompt report with: Title, Category, Content
-
-#### `_create_novel_content(title, text_chunk, chunk_idx, total_chunks)` - Lines 461-468
-
-Formats novel chunk with: Title, Part info, Text
-
-#### `_create_manual_content(item)` - Lines 470-483
-
-Formats manual with: Title, Sections list, Content
-
-#### `_create_enterprise_content(item)` - Lines 485-494
-
-Formats benchmark with: Scenario, Task, Labels
-
-#### `_create_portuguese_content(item)` - Lines 496-504
-
-Formats Portuguese text with: Título, Língua, Conteúdo (Portuguese labels)
-
-#### `_create_edustories_content(item)` - Lines 506-530
-
-Formats educational case study with structured sections:
-
-- **Background**: Context and classroom setting (from `description`)
-- **Situation**: Detailed situation description (from `anamnesis`)
-- **Teacher Intervention**: Intervention approach (from `solution`)
-- **Outcome**: Final state after intervention (from `outcome`)
-- **Student Profile**: Age/year, hobbies, diagnoses, disorders
-- **Annotations**: Identified problems, solution categories, outcome implications
-- Educational case study context marker
-
-#### `_chunk_text(text, chunk_size=1000)` - Lines 532-544
-
-**Utility method** for splitting long texts:
-
-- Splits by words (not characters)
-- Returns list of chunks
-- Handles edge cases (empty text, invalid chunk_size)
-
----
-
-### 3. Modified Methods
-
-#### `transform_system_chat()` - Line 141
-
-- Added `"license": "unknown"` to metadata
-- Maintains backward compatibility
-
-#### `ingest()` CLI Command - Lines 575-649
-
-**Changes**:
-
-- Added new datasets to `--datasets` choice: `arxiv`, `prompt-report`, `novels`, `manuals`, `enterprise`, `portuguese-edu`, `edustories`
-- Added new option: `--arxiv-limit` (integer, optional)
-- Updated default from `['npc-dialogue']` to `['arxiv']`
-- Updated `all` to include new datasets (excludes npc-dialogue)
-- Added try-catch error handling around each dataset
-- Added conditional check: only create pack if docs generated
-- Better error reporting
-- Enterprise now uses SustcZhangYX/ChatEnv instead of AST-FRI/EnterpriseBench
-
-#### `list_available()` CLI Command - Lines 652-668
-
-**Changes**:
-
-- Updated documentation with new datasets including edustories
-- Added section headers: 🔬 Primary, 🔧 Legacy, 📦 Special
-- Included dataset sizes and key features
-- Added notes about:
-  - npc-dialogue removal (unlicensed)
-  - enterprise dataset change (EnterpriseBench → ChatEnv)
-  - novels requiring pdfplumber for full extraction
-
----
-
-## File Statistics
-
-| Metric | Before | After | Change |
-|--------|--------|-------|--------|
-| Total Lines | 290 | ~750 | +460 |
-| Transformer Methods | 3 | 10 | +7 |
-| Helper Methods | 3 | 11 | +8 |
-| License Info | None | MIT | ✅ Added |
-| PDF Extraction | Basic | Enhanced | ✅ Improved |
-
----
-
-## Data Structure: Warbler Document Format
-
-All transformers produce documents matching this structure:
-
-```python
-{
-    "content_id": "source-type/unique-identifier",
-    
-    "content": """Formatted text with:
-    - Dataset-specific fields
-    - Structured information
-    - Human-readable format
-    """,
-    
-    "metadata": {
-        # Standard fields
-        "pack": "warbler-pack-<dataset>",
-        "source_dataset": "huggingface/dataset-path",
-        "license": "MIT",
-        
-        # Warbler FractalStat fields
-        "realm_type": "category",           # scholarly|methodological|narrative|procedural|business|educational
-        "realm_label": "subcategory",       # arxiv|prompt_engineering|generated_fiction|etc
-        "lifecycle_stage": "emergence",     # Always emergence for new ingestions
-        "activity_level": 0.5-0.8,         # 0.5=low, 0.8=high
-        "dialogue_type": "content_type",   # scholarly_discussion|technical_discussion|etc
-        
-        # Dataset-specific fields
-        # (see each transformer for specific metadata)
-    }
-}
-```
-
----
-
-## Integration Points with Warbler-CDA
-
-### 1. Pack Creation
-
-```python
-ingestor = HFWarblerIngestor()
-docs = ingestor.transform_arxiv(limit=1000)
-pack_path = ingestor.create_warbler_pack(docs, "warbler-pack-arxiv")
-```
-
-### 2. Pack Loading
-
-```python
-from warbler_cda.pack_loader import WarblerPackLoader
-packs = WarblerPackLoader.load_pack_directory("/path/to/packs")
-```
-
-### 3. Document Enrichment
-
-```python
-from warbler_cda.retrieval_api import RetrievalAPI
-api = RetrievalAPI()
-for doc in docs:
-    api.add_document(doc["content_id"], doc["content"])
-    # Automatically:
-    # - Computes embeddings
-    # - Generates FractalStat coordinates
-    # - Stores in context_store
-```
-
-### 4. Hybrid Retrieval
-
-```python
-query = RetrievalQuery(
-    semantic_query="machine learning optimization",
-    fractalstat_hybrid=True,
-    weight_semantic=0.6,
-    weight_fractalstat=0.4
-)
-assembly = api.retrieve_context(query)
-```
-
----
-
-## Error Handling
-
-All transformers include:
-
-- `.get()` with defaults for missing fields
-- `isinstance()` checks for flexible dataset formats
-- CLI try-catch blocks with user-friendly error messages
-- Graceful handling when dataset load fails
-- Conditional pack creation (only if docs generated)
-
----
-
-## Performance Considerations
-
-### Memory Management
-
-- **arXiv**: Use `--arxiv-limit` to control ingestion
-  - Example: 100 papers ~50MB, 10k papers ~5GB
-  - Recommended limit: 10k-50k papers
-  
-- **Novels**: Automatic chunking prevents single document explosion
-  - 100k word novel → ~100 chunks
-  - Each chunk ~100 tokens (embedding-friendly)
-
-### Processing Speed
-
-- Small datasets (50-300 docs): <10 seconds
-- Medium datasets (1k-10k): 30-120 seconds
-- Large datasets (100k+): Use with `--limit` parameters
-
----
-
-## CLI Examples
-
-```bash
-# Ingest single dataset
-python -m warbler_cda.utils.hf_warbler_ingest ingest -d arxiv
-
-# Limit arXiv to 5000 papers
-python -m warbler_cda.utils.hf_warbler_ingest ingest -d arxiv --arxiv-limit 5000
-
-# Ingest multiple datasets
-python -m warbler_cda.utils.hf_warbler_ingest ingest \
-  -d arxiv --arxiv-limit 10000 \
-  -d prompt-report \
-  -d novels \
-  -d manuals
-
-# Ingest all MIT datasets
-python -m warbler_cda.utils.hf_warbler_ingest ingest -d all --arxiv-limit 50000
-
-# Change pack prefix
-python -m warbler_cda.utils.hf_warbler_ingest ingest \
-  -d novels \
-  -p custom-prefix
-
-# List available datasets
-python -m warbler_cda.utils.hf_warbler_ingest list-available
-```
-
----
-
-## Testing
-
-### Test File
-
-**Location**: `tests/test_new_mit_datasets.py`
-
-### Test Classes (37 tests total)
-
-- `TestArxivPapersTransformer` (4 tests)
-- `TestPromptReportTransformer` (2 tests)
-- `TestGeneratedNovelsTransformer` (2 tests)
-- `TestManualnsTransformer` (2 tests) [Note: typo in class name, should be Manuals]
-- `TestEnterpriseTransformer` (2 tests) - Updated for ChatEnv dataset
-- `TestPortugueseEducationTransformer` (2 tests)
-- `TestEdustoriesTransformer` (4 tests) - NEW
-- `TestNewDatasetsIntegrationWithRetrieval` (2 tests)
-- `TestNewDatasetsPerformance` (1 test)
-- `TestNewDatasetsAllAtOnce` (1 test) - Updated to include edustories
-
-### Running Tests
-
-```bash
-cd warbler-cda-package
-
-# Run all new dataset tests
-pytest tests/test_new_mit_datasets.py -v
-
-# Run specific test class
-pytest tests/test_new_mit_datasets.py::TestArxivPapersTransformer -v
-
-# Run with coverage
-pytest tests/test_new_mit_datasets.py --cov=warbler_cda.utils.hf_warbler_ingest
-```
-
----
-
-## Validation Checklist
-
-- [x] All 7 transformers implemented (including edustories)
-- [x] All helper methods implemented
-- [x] Warbler document format correct
-- [x] MIT license field added to all documents
-- [x] Metadata includes realm_type and realm_label
-- [x] Error handling with try-catch
-- [x] CLI updated with new datasets
-- [x] CLI includes arxiv-limit parameter
-- [x] list_available() updated
-- [x] Backward compatibility maintained
-- [x] Type hints complete
-- [x] Docstrings comprehensive
-- [x] Test coverage: 37 tests
-- [x] Documentation complete
-- [x] Code follows existing patterns
-- [x] Enterprise dataset updated to ChatEnv
-- [x] PDF extraction enhanced for novels
-- [x] Edustories dataset added
-
----
-
-## Compatibility Notes
-
-### Backward Compatibility ✅
-
-- Existing transformers (multi-character, system-chat) unchanged
-- npc-dialogue removed as per license requirements
-- Existing pack creation logic unchanged
-- Existing metadata format preserved
-
-### Forward Compatibility ✅
-
-- New datasets use same document structure
-- New metadata fields are optional/additive
-- FractalStat coordinates computed automatically
-- Hybrid retrieval works with all datasets
-
----
-
-## Deployment Notes
-
-### Pre-Production
-
-1. Run full test suite
-2. Test with sample data (limit=10)
-3. Verify pack creation
-4. Test pack loading
-
-### Production
-
-1. Create packs with appropriate limits
-2. Monitor ingestion performance
-3. Archive old packs as needed
-4. Update documentation with new dataset sources
-
-### Updates
-
-To update with new HuggingFace data:
-
-```bash
-# Clean old packs
-rm -rf packs/warbler-pack-arxiv-*
-
-# Re-ingest with desired limit
-python -m warbler_cda.utils.hf_warbler_ingest ingest -d arxiv --arxiv-limit 50000
-```
-
----
-
-## Related Files
-
-- `warbler_cda/retrieval_api.py` - Uses documents for hybrid retrieval
-- `warbler_cda/pack_loader.py` - Loads created packs
-- `warbler_cda/embeddings/` - Generates FractalStat coordinates
-- `tests/test_retrieval_api.py` - Integration tests
-- `DATASET-MIGRATION-GUIDE.md` - Original source commit documentation
-
----
-
-**Status**: ✅ Implementation Complete  
-**Last Updated**: 2025-11-08  
-**Next**: Integration Testing & Deployment

LICENSE

@@ -1,21 +0,0 @@
-MIT License
-
-Copyright (c) 2024 Tiny Walnut Games
-
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in all
-copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.

PACKAGE_MANIFEST.md

@@ -1,94 +0,0 @@
-# Warbler CDA Package - Complete File List
-
-## Package Structure (21 core files + infrastructure)
-
-### Core RAG System (9 files)
-
-✓ warbler_cda/retrieval_api.py - Main RAG API with hybrid scoring
-✓ warbler_cda/semantic_anchors.py - Semantic memory with provenance
-✓ warbler_cda/anchor_data_classes.py - Core data structures
-✓ warbler_cda/anchor_memory_pool.py - Performance optimization
-✓ warbler_cda/summarization_ladder.py - Hierarchical compression
-✓ warbler_cda/conflict_detector.py - Conflict detection
-✓ warbler_cda/castle_graph.py - Concept extraction
-✓ warbler_cda/melt_layer.py - Memory consolidation
-✓ warbler_cda/evaporation.py - Content distillation
-
-### FractalStat System (4 files)
-
-✓ warbler_cda/fractalstat_rag_bridge.py - FractalStat hybrid scoring bridge
-✓ warbler_cda/fractalstat_entity.py - FractalStat entity system
-✓ warbler_cda/fractalstat_experiments.py - Validation experiments
-✓ warbler_cda/fractalstat_visualization.py - Visualization tools
-
-### Embeddings (4 files)
-
-✓ warbler_cda/embeddings/__init__.py
-✓ warbler_cda/embeddings/base_provider.py - Abstract interface
-✓ warbler_cda/embeddings/factory.py - Provider factory
-✓ warbler_cda/embeddings/local_provider.py - Local TF-IDF embeddings
-✓ warbler_cda/embeddings/openai_provider.py - OpenAI embeddings
-
-### Production API (2 files)
-
-✓ warbler_cda/api/__init__.py
-✓ warbler_cda/api/service.py - FastAPI service (exp09_api_service.py)
-✓ warbler_cda/api/cli.py - CLI interface (exp09_cli.py)
-
-### Utilities (2 files)
-
-✓ warbler_cda/utils/__init__.py
-✓ warbler_cda/utils/load_warbler_packs.py - Pack loader
-✓ warbler_cda/utils/hf_warbler_ingest.py - HF dataset ingestion
-
-### Infrastructure Files
-
-✓ warbler_cda/__init__.py - Package initialization
-✓ requirements.txt - Dependencies
-✓ pyproject.toml - Package metadata
-✓ README.md - Documentation
-✓ app.py - Gradio demo for HuggingFace
-✓ .gitignore - Git exclusions
-✓ LICENSE - MIT License
-✓ DEPLOYMENT.md - Deployment guide
-✓ README_HF.md - HuggingFace Space config
-✓ setup.sh - Quick setup script
-✓ transform_imports.sh - Import transformation script
-
-## Total Files: 32 files
-
-## Import Transformations Applied
-
-All imports have been transformed from:
-
-- `from seed.engine.X import Y` → `from warbler_cda.X import Y`
-- `from .X import Y` → `from warbler_cda.X import Y`
-
-Privacy hooks have been removed (not needed for HuggingFace deployment).
-
-## Size Estimate
-
-Total package size: ~500KB (source code only)
-With dependencies: ~2GB (includes PyTorch, Transformers, etc.)
-
-## Next Steps
-
-1. Test the package locally:
-
-   ```bash
-   cd warbler-cda-package
-   ./setup.sh
-   python app.py
-   ```
-
-2. Deploy to HuggingFace:
-   - Set HF_TOKEN in GitLab CI/CD variables
-   - Push to main or create a tag
-   - Pipeline will auto-sync to HuggingFace Space
-
-3. Publish to PyPI (optional):
-
-   ```bash
-   python -m build
-   twine upload dist/*
-   ```

PACKS_DEPLOYMENT.md

@@ -1,281 +0,0 @@
-# Warbler Packs Deployment Guide
-
-This guide explains how Warbler packs are loaded and deployed to HuggingFace Spaces.
-
-## Overview
-
-The Warbler CDA Space automatically discovers and ingests content packs at startup. Packs contain conversation templates, NPC dialogues, wisdom templates, and other domain-specific content for the RAG system.
-
-## Pack Structure
-
-```none
-packs/
-├── warbler-pack-core/              # Essential conversation templates
-├── warbler-pack-faction-politics/  # Political dialogue templates
-├── warbler-pack-wisdom-scrolls/    # Development wisdom generation
-└── warbler-pack-hf-npc-dialogue/   # 1,900+ NPC dialogues from HuggingFace
-```
-
-## Deployment Process
-
-### 1. Local Development
-
-Copy packs from the main repository to warbler-cda-package:
-
-```bash
-cd warbler-cda-package
-bash copy_packs.sh
-```
-
-This script copies all packs from:
-
-```path
-../packages/com.twg.the-seed/The Living Dev Agent/packs/
-```
-
-To:
-
-```path
-./packs/
-```
-
-### 2. Automatic Loading
-
-When `app.py` starts, it:
-
-1. **Initializes PackLoader**
-
-   ```python
-   pack_loader = PackLoader()
-   ```
-
-2. **Discovers documents from all packs**
-
-   ```python
-   pack_docs = pack_loader.discover_documents()
-   ```
-
-3. **Ingests documents into RetrievalAPI**
-
-   ```python
-   for doc in pack_docs:
-       api.add_document(doc["id"], doc["content"], doc["metadata"])
-   ```
-
-4. **Falls back to sample documents** if packs not found
-   - Ensures demo works even without packs
-   - Provides example data for testing
-
-### 3. HuggingFace Space Deployment
-
-The `.gitlab-ci.yml` handles deployment:
-
-```bash
-hf upload-large-folder $SPACE_NAME . --repo-type=space --space-sdk=gradio
-```
-
-This uploads:
-
-- All Python source code
-- All packs in the `packs/` directory
-- Configuration files
-
-**Important**: The `packs/` directory must exist and contain pack data before deployment.
-
-## Pack Loader Details
-
-The `PackLoader` class (`warbler_cda/pack_loader.py`) handles:
-
-### Pack Discovery
-
-- Scans the `packs/` directory
-- Identifies pack type (JSONL-based or structured)
-- Discovers all documents
-
-### Document Parsing
-
-- **Structured Packs** (core, faction, wisdom): Load from `pack/templates.json`
-- **JSONL Packs** (HF NPC dialogue): Parse line-by-line JSONL format
-
-### Metadata Extraction
-
-```python
-{
-    "pack": "pack-name",
-    "type": "template|dialogue",
-    "realm_type": "wisdom|faction|narrative",
-    "realm_label": "pack-label",
-    "lifecycle_stage": "emergence|peak",
-    "activity_level": 0.7-0.8
-}
-```
-
-## Adding New Packs
-
-To add a new pack to the system:
-
-### 1. Create Pack Structure
-
-```bash
-packs/
-└── warbler-pack-mypack/
-    ├── package.json
-    ├── pack/
-    │   └── templates.json  # OR
-    └── mypack.jsonl        # JSONL format
-```
-
-### 2. Update Pack Loader (if needed)
-
-If your pack format is different, add handling to `pack_loader.py`:
-
-```python
-def _load_pack(self, pack_dir: Path, pack_name: str):
-    if "mypack" in pack_name:
-        return self._load_my_format(pack_dir, pack_name)
-    # ... existing logic
-```
-
-### 3. Register in copy_packs.sh
-
-```bash
-PACKS=(
-    "warbler-pack-core"
-    "warbler-pack-mypack"  # Add here
-)
-```
-
-### 4. Deploy
-
-Run copy script and deploy:
-
-```bash
-bash copy_packs.sh
-# Commit and push to trigger CI/CD
-```
-
-## Document Format
-
-Each loaded document follows this structure:
-
-```python
-{
-    "id": "pack-name/document-id",
-    "content": "Document text content...",
-    "metadata": {
-        "pack": "pack-name",
-        "type": "template|dialogue",
-        "realm_type": "wisdom|faction|narrative",
-        "realm_label": "label",
-        "lifecycle_stage": "emergence|peak|crystallization",
-        "activity_level": 0.5-0.8
-    }
-}
-```
-
-## Monitoring
-
-Check pack loading in Space logs:
-
-```log
-✓ Loaded 1915 documents from warbler-pack-hf-npc-dialogue
-✓ Loaded 6 documents from warbler-pack-wisdom-scrolls
-✓ Loaded 15 documents from warbler-pack-faction-politics
-✓ Loaded 10 documents from warbler-pack-core
-```
-
-Or if packs not found:
-
-```log
-⚠️  No Warbler packs found. Using sample documents instead.
-```
-
-## Publishing to HuggingFace Hub
-
-Each pack has a dataset card for publication:
-
-- **README_HF_DATASET.md** - HuggingFace dataset card
-- Contains metadata, attribution, and usage instructions
-
-Publish to HuggingFace:
-
-```bash
-# Create repo on HuggingFace Hub (one per pack)
-huggingface-cli repo create warbler-pack-core
-
-# Push pack as dataset
-cd packs/warbler-pack-core
-huggingface-cli upload . tiny-walnut-games/warbler-pack-core --repo-type dataset
-```
-
-## Performance Considerations
-
-### Load Time
-
-- PackLoader loads all packs at startup
-- Currently: ~1-2 seconds for all packs
-- Packs are cached in memory for query performance
-
-### Storage
-
-- Core pack: ~50KB
-- Faction politics pack: ~80KB
-- Wisdom scrolls pack: ~60KB
-- HF NPC dialogue: ~2MB
-- **Total**: ~2.3MB
-
-### Scaling
-
-For larger deployments:
-
-- Lazy-load individual packs on demand
-- Implement pack caching layer
-- Use database for large pack collections
-
-## Troubleshooting
-
-### Packs not loading
-
-Check that `packs/` directory exists:
-
-```bash
-ls -la packs/
-```
-
-Verify pack structure:
-
-```bash
-ls -la packs/warbler-pack-core/
-```
-
-### Sample documents showing instead
-
-If you see "No Warbler packs found", the `packs/` directory is empty. Run:
-
-```bash
-bash copy_packs.sh
-```
-
-### Pack loader errors
-
-Check logs for parsing errors:
-
-```log
-Error loading JSONL pack: ...
-Error parsing line 42 in warbler-pack-hf-npc-dialogue.jsonl: ...
-```
-
-Fix the source pack and re-run `copy_packs.sh`.
-
-## Related Documentation
-
-- [README.md](./README.md) - Main package documentation
-- [DEPLOYMENT.md](./DEPLOYMENT.md) - General deployment guide
-- [app.py](./app.py) - Application startup and pack initialization
-- [warbler_cda/pack_loader.py](./warbler_cda/pack_loader.py) - Pack loading implementation
-
-## License
-
-All packs use MIT License. See individual pack LICENSE files for details.
-
-Attribution: Warbler CDA - Tiny Walnut Games

PACK_CACHING.md

@@ -1,172 +0,0 @@
-# Warbler Pack Caching Strategy
-
-## Overview
-
-The app now implements intelligent pack caching to avoid unnecessary re-ingestion of large datasets. This minimizes GitLab storage requirements and allows fast session startup.
-
-## How It Works
-
-### First Run (Session Start)
-
-1. **PackManager** initializes and checks for cached metadata
-2. **Health check** verifies if documents are already in the context store
-3. **Ingestion** occurs only if:
-   - No cache metadata exists
-   - Pack count changed
-   - Health check fails (documents missing)
-4. **Cache** is saved with timestamp and document count
-
-### Subsequent Runs
-
-- Reuses cached documents without re-ingestion
-- Quick health check ensures documents are still valid
-- Fallback to sample docs if packs unavailable
-
-## Environment Variables
-
-Control pack ingestion behavior with these variables:
-
-### `WARBLER_INGEST_PACKS` (default: `true`)
-
-Enable/disable automatic pack ingestion.
-
-```bash
-export WARBLER_INGEST_PACKS=false
-```
-
-### `WARBLER_SAMPLE_ONLY` (default: `false`)
-
-Load only sample documents (for CI/CD verification).
-
-```bash
-export WARBLER_SAMPLE_ONLY=true
-```
-
-Best for:
-
-- PyPI package CI/CD pipelines
-- Quick verification that ingestion works
-- Minimal startup time in restricted environments
-
-### `WARBLER_SKIP_PACK_CACHE` (default: `false`)
-
-Force reingest even if cache exists.
-
-```bash
-export WARBLER_SKIP_PACK_CACHE=true
-```
-
-Best for:
-
-- Testing pack ingestion pipeline
-- Updating stale cache
-- Debugging
-
-## Cache Location
-
-Default cache stored at:
-
-```path
-~/.warbler_cda/cache/pack_metadata.json
-```
-
-Metadata includes:
-
-```json
-{
-  "ingested_at": 1699564800,
-  "pack_count": 7,
-  "doc_count": 12345,
-  "status": "healthy"
-}
-```
-
-## CI/CD Optimization
-
-### For GitLab CI (Minimal PyPI Package)
-
-```yaml
-test:
-  script:
-    - export WARBLER_SAMPLE_ONLY=true
-    - pip install .
-    - python -m pytest tests/
-```
-
-Benefits:
-
-- ✅ No large pack files in repository
-- ✅ Fast CI runs (5 samples vs 2.5M docs)
-- ✅ Verifies ingestion code works
-- ✅ Full packs load on first user session
-
-### For Local Development
-
-Keep full packs in working directory:
-
-```bash
-cd warbler-cda-package
-python -m warbler_cda.utils.hf_warbler_ingest ingest -d all
-python app.py
-```
-
-First run ingests all packs. Subsequent runs use cache.
-
-### For Gradio Space/Cloud Deployment
-
-Set environment at deployment:
-
-```bash
-WARBLER_INGEST_PACKS=true
-```
-
-Packs ingest once per session, then cached in instance memory.
-
-## Files Affected
-
-- `app.py` - Main Gradio app with PackManager
-- `warbler_cda/utils/load_warbler_packs.py` - Pack discovery (already handles caching)
-- No changes needed to pack ingestion scripts
-
-## Performance Impact
-
-### Memory
-
-- **With packs**: ~500MB (2.5M arxiv docs + others)
-- **With samples**: ~1MB (5 test documents)
-
-### Startup Time
-
-- **First run**: ~30-60 seconds (ingest packs)
-- **Cached run**: ~2-5 seconds (health check only)
-- **Sample only**: <1 second
-
-## Troubleshooting
-
-### Packs not loading?
-
-1. Check `WARBLER_INGEST_PACKS=true` (default)
-2. Verify packs exist: `ls -la packs/`
-3. Force reingest: `export WARBLER_SKIP_PACK_CACHE=true`
-
-### Cache corrupted?
-
-```bash
-rm -rf ~/.warbler_cda/cache/pack_metadata.json
-```
-
-Will reingest on next run.
-
-### Need sample docs only?
-
-```bash
-export WARBLER_SAMPLE_ONLY=true
-python app.py
-```
-
-## Future Improvements
-
-- [ ] Detect pack updates via file hash instead of just count
-- [ ] Selective pack loading (choose which datasets to cache)
-- [ ] Metrics dashboard showing cache hit/miss rates
-- [ ] Automatic cache expiration after N days

PACK_INGESTION_FIX.md

@@ -1,209 +0,0 @@
-# Pack Ingestion Fix for HuggingFace Space
-
-## Problem Summary
-
-Your HuggingFace Space was experiencing three critical errors during pack ingestion:
-
-1. ❌ **Core pack missing JSONL**: `warbler-pack-core missing JSONL file`
-2. ❌ **Faction pack missing JSONL**: `warbler-pack-faction-politics missing JSONL file`
-3. ❌ **Corrupted arxiv data**: `Error parsing line 145077 in warbler-pack-hf-arxiv.jsonl: Unterminated string`
-
-## Root Causes Identified
-
-### Issue 1 & 2: Different Pack Formats
-
-Your project has **two different pack formats**:
-
-**Format A: Structured Packs** (Core & Faction)
-
-```none
-warbler-pack-core/
-├── package.json
-├── pack/
-│   └── templates.json    ← Data is here!
-└── src/
-```
-
-**Format B: JSONL Packs** (HuggingFace datasets)
-
-```none
-warbler-pack-hf-arxiv/
-├── package.json
-└── warbler-pack-hf-arxiv-chunk-001.jsonl  ← Data is here!
-```
-
-The pack loader was expecting **all** packs to have JSONL files, causing false warnings for the structured packs.
-
-### Issue 3: Corrupted JSON Line
-
-The arxiv pack has a malformed JSON entry at line 145077:
-
-```json
-{"content": "This is a test with an unterminated string...
-```
-
-The previous code would **crash** on the first error, preventing the entire ingestion from completing.
-
-## Solution Implemented
-
-### 1. Enhanced Pack Format Detection
-
-Updated `_is_valid_warbler_pack()` to recognize **three valid formats**:
-
-```python
-if jsonl_file.exists():
-    return True  # Format B: Single JSONL file
-else:
-    templates_file = pack_dir / "pack" / "templates.json"
-    if templates_file.exists():
-        return False  # Format A: Structured pack (triggers different loader)
-    else:
-        if pack_name.startswith("warbler-pack-hf-"):
-            logger.warning(f"HF pack missing JSONL")  # Only warn for HF packs
-        return False
-```
-
-### 2. Robust Error Handling
-
-Updated `_load_jsonl_file()` to **continue on error**:
-
-```python
-try:
-    entry = json.loads(line)
-    documents.append(doc)
-except json.JSONDecodeError as e:
-    error_count += 1
-    if error_count <= 5:  # Only log first 5 errors
-        logger.warning(f"Error parsing line {line_num}: {e}")
-    continue  # ← Skip bad line, keep processing!
-```
-
-## What Changed
-
-**File: `warbler-cda-package/warbler_cda/pack_loader.py`**
-
-### Change 1: Smarter Validation
-
-- ✅ Recognizes structured packs as valid
-- ✅ Only warns about missing JSONL for HF packs
-- ✅ Better logging messages
-
-### Change 2: Error Recovery
-
-- ✅ Skips corrupted JSON lines
-- ✅ Limits error logging to first 5 occurrences
-- ✅ Reports summary: "Loaded X documents (Y lines skipped)"
-
-## Expected Behavior After Fix
-
-### Before (Broken)
-
-```none
-[INFO] Pack Status: ✓ All 6 packs verified and ready
-Single-file pack warbler-pack-core missing JSONL file: /home/user/app/packs/warbler-pack-core/warbler-pack-core.jsonl
-Single-file pack warbler-pack-faction-politics missing JSONL file: /home/user/app/packs/warbler-pack-faction-politics/warbler-pack-faction-politics.jsonl
-Error parsing line 145077 in /home/user/app/packs/warbler-pack-hf-arxiv/warbler-pack-hf-arxiv.jsonl: Unterminated string
-[INFO] Ingesting 374869 documents from Warbler packs...
-[ERROR] Ingestion failed!
-```
-
-### After (Fixed)
-
-```none
-[INFO] Pack Status: ✓ All 10 packs verified and ready
-[INFO] Ingesting documents from Warbler packs...
-[INFO] Loading pack: warbler-pack-core
-[DEBUG] Pack warbler-pack-core uses structured format (pack/templates.json)
-[INFO] ✓ Loaded 8 documents from warbler-pack-core
-[INFO] Loading pack: warbler-pack-faction-politics
-[DEBUG] Pack warbler-pack-faction-politics uses structured format (pack/templates.json)
-[INFO] ✓ Loaded 6 documents from warbler-pack-faction-politics
-[INFO] Loading pack: warbler-pack-hf-arxiv
-[INFO] Loading chunked pack: warbler-pack-hf-arxiv
-[INFO] Found 5 chunk files for warbler-pack-hf-arxiv
-[WARN] Error parsing line 145077 in warbler-pack-hf-arxiv-chunk-003.jsonl: Unterminated string
-[INFO] Loaded 49999 documents from warbler-pack-hf-arxiv-chunk-003.jsonl (1 lines skipped due to errors)
-[INFO] Loaded 250000 total documents from 5 chunks
-...
-[OK] Loaded 374868 documents from Warbler packs (1 corrupted line skipped)
-```
-
-## Testing the Fix
-
-### Local Testing
-
-1. **Test with sample packs**:
-
-    ```bash
-    cd warbler-cda-package
-    python -c "from warbler_cda.pack_loader import PackLoader; loader = PackLoader(); docs = loader.discover_documents(); print(f'Loaded {len(docs)} documents')"
-    ```
-
-2. **Run the app locally**:
-
-```bash
-python app.py
-```
-
-### HuggingFace Space Testing
-
-1. **Merge this MR** to main branch
-2. **Push to HuggingFace** (if auto-sync is not enabled)
-3. **Check the Space logs** for the new output format
-4. **Verify document count** in the System Stats tab
-
-## Next Steps
-
-1. ✅ **Review the MR**: [!15 - Fix HuggingFace pack ingestion issues](https://gitlab.com/tiny-walnut-games/the-seed/-/merge_requests/15)
-
-2. ✅ **Merge when ready**: The fix is backward compatible and safe to merge
-
-3. ✅ **Monitor HF Space**: After deployment, check that:
-   - All packs load successfully
-   - Document count is ~374,868 (minus 1 corrupted line)
-   - No error messages in logs
-
-4. 🔧 **Optional: Fix corrupted line** (future improvement):
-   - Identify the exact corrupted entry in arxiv chunk 3
-   - Re-generate that chunk from source dataset
-   - Update the pack
-
-## Additional Notes
-
-### Why Not Fix the Corrupted Line Now?
-
-The corrupted line is likely from the source HuggingFace dataset (`nick007x/arxiv-papers`). Options:
-
-1. **Skip it** (current solution) - Loses 1 document out of 2.5M
-2. **Re-ingest** - Download and re-process the entire arxiv dataset
-3. **Manual fix** - Find and repair the specific line
-
-For now, **skipping is the pragmatic choice** - you lose 0.00004% of data and gain a working system.
-
-### Pack Format Standardization
-
-Consider standardizing all packs to JSONL format in the future:
-
-```bash
-# Convert structured packs to JSONL
-python -m warbler_cda.utils.convert_structured_to_jsonl \
-  --input packs/warbler-pack-core/pack/templates.json \
-  --output packs/warbler-pack-core/warbler-pack-core.jsonl
-```
-
-This would simplify the loader logic and make all packs consistent.
-
-## Questions?
-
-If you encounter any issues:
-
-1. Check the HF Space logs for detailed error messages
-2. Verify pack structure matches expected formats
-3. Test locally with `PackLoader().discover_documents()`
-4. Review this document for troubleshooting tips
-
----
-
-**Status**: ✅ Fix implemented and ready for merge
-**MR**: !15
-**Impact**: Fixes all 3 ingestion errors, enables full pack loading

PDF_INGESTION_INVESTIGATION.md

@@ -1,325 +0,0 @@
-# PDF Ingestion Investigation Report
-
-**Date**: 2024
-**Session Reference**: Based on agent session 1251355
-**Investigator**: AI Agent
-
-## Executive Summary
-
-Investigation into the warbler-cda-package ingesters to determine if they are properly utilizing PDFPlumber for reading PDF files. The investigation revealed that **PDFPlumber IS being utilized**, but there were **two bugs** that needed fixing.
-
-## Key Findings
-
-### ✅ PDFPlumber Integration Status: CONFIRMED
-
-The ingesters **ARE** utilizing PDFPlumber to read PDF files. The implementation is present and functional with proper fallback mechanisms.
-
-### 📍 PDFPlumber Usage Locations
-
-#### 1. **Import and Availability Check** (Lines 23-27)
-
-```python
-try:
-    import pdfplumber
-    PDF_AVAILABLE = True
-except ImportError:
-    PDF_AVAILABLE = False
-```
-
-**Status**: ✅ Properly implemented with graceful fallback
-
-#### 2. **PDF Support Detection Method** (Lines 47-49)
-
-```python
-def has_pdf_support(self) -> bool:
-    """Check if PDF extraction is available"""
-    return PDF_AVAILABLE
-```
-
-**Status**: ✅ Provides runtime check for PDF capabilities
-
-#### 3. **Primary PDF Extraction Method** (Lines 51-67)
-
-```python
-def extract_pdf_text(self, pdf_bytes: bytes, max_chars: int = 5000) -> Optional[str]:
-    """Extract text from PDF bytes with fallback"""
-    if not PDF_AVAILABLE:
-        return None
-    
-    try:
-        pdf_file = io.BytesIO(pdf_bytes)
-        text_parts = []
-        
-        with pdfplumber.open(pdf_file) as pdf:
-            for page in pdf.pages:
-                text = page.extract_text()
-                if text:
-                    text_parts.append(text)
-                    if sum(len(t) for t in text_parts) > max_chars:
-                        break
-        
-        return " ".join(text_parts)[:max_chars] if text_parts else None
-    except Exception as e:
-        logger.debug(f"PDF extraction error: {e}")
-        return None
-```
-
-**Status**: ✅ Properly implemented with:
-
-- Character limit protection (max_chars=5000)
-- Page-by-page extraction
-- Error handling
-- Graceful fallback
-
-#### 4. **Flexible PDF Extraction Method** (Lines 540-565)
-
-```python
-def _extract_pdf_text(self, pdf_data: Any) -> Optional[str]:
-    """Extract text from PDF data (bytes, file path, or file-like object)"""
-    if not PDF_AVAILABLE:  # ⚠️ FIXED: Was PDF_SUPPORT
-        return None
-    
-    try:
-        # Handle different PDF data types
-        if isinstance(pdf_data, bytes):
-            pdf_file = io.BytesIO(pdf_data)
-        elif isinstance(pdf_data, str) and os.path.exists(pdf_data):
-            pdf_file = pdf_data
-        elif hasattr(pdf_data, 'read'):
-            pdf_file = pdf_data
-        else:
-            return None
-        
-        # Extract text from all pages
-        text_parts = []
-        with pdfplumber.open(pdf_file) as pdf:
-            for page in pdf.pages:
-                page_text = page.extract_text()
-                if page_text:
-                    text_parts.append(page_text)
-        
-        return "\n\n".join(text_parts) if text_parts else None
-        
-    except Exception as e:
-        logger.debug(f"PDF extraction error: {e}")
-        return None
-```
-
-**Status**: ✅ Handles multiple input types (bytes, file path, file-like objects)
-
-### 🎯 Transformers Using PDF Extraction
-
-#### 1. **transform_novels()** (Lines 247-320)
-
-- **Dataset**: GOAT-AI/generated-novels
-- **PDF Usage**: Attempts to extract from PDF fields when text fields are unavailable
-- **Fallback**: Creates placeholder entries with informative messages
-- **Code Location**: Lines 285-295
-
-```python
-if not text and self.has_pdf_support():
-    for pdf_field in ['pdf', 'file', 'document']:
-        try:
-            if isinstance(item, dict):
-                if pdf_field in item and item[pdf_field]:
-                    text = self.extract_pdf_text(item[pdf_field])
-                    if text:
-                        logger.info(f"Novel {idx + 1}: Extracted {len(text)} chars from PDF")
-                        break
-```
-
-**Status**: ✅ Properly integrated with PDF extraction
-
-#### 2. **transform_portuguese_education()** (Lines 400-500+)
-
-- **Dataset**: Solshine/Portuguese_Language_Education_Texts
-- **PDF Usage**: Could potentially use PDF extraction (not explicitly shown in current code)
-- **Fallback**: Creates informative placeholders when content is unavailable
-
-**Status**: ✅ Has fallback mechanisms in place
-
-## 🐛 Bugs Found and Fixed
-
-### Bug #1: Incorrect Variable Name in `_extract_pdf_text()`
-
-**Location**: Line 542
-**Issue**: Used `PDF_SUPPORT` instead of `PDF_AVAILABLE`
-**Impact**: Would cause NameError when `_extract_pdf_text()` is called
-**Fix Applied**: Changed `PDF_SUPPORT` to `PDF_AVAILABLE`
-
-```diff
-- if not PDF_SUPPORT:
-+ if not PDF_AVAILABLE:
-```
-
-### Bug #2: Duplicate `import io` Statement
-
-**Location**: Line 56 (inside `extract_pdf_text` method)
-**Issue**: `import io` was inside the method instead of at module level
-**Impact**: Unnecessary repeated imports, potential performance impact
-**Fix Applied**:
-
-1. Added `import io` to module-level imports (Line 10)
-2. Removed duplicate `import io` from inside method
-
-```diff
-# At module level (Line 10)
-+ import io
-
-# Inside extract_pdf_text method (Line 56)
-- import io
-```
-
-## 📦 Dependency Configuration
-
-### requirements.txt
-
-```text
-pdfplumber>=0.11.0
-```
-
-**Status**: ✅ Properly listed as a dependency
-
-### pyproject.toml
-
-**Status**: ⚠️ NOT listed in core dependencies
-**Recommendation**: Consider adding to optional dependencies or core dependencies
-
-```toml
-[project.optional-dependencies]
-pdf = [
-    "pdfplumber>=0.11.0",
-]
-```
-
-## 🔍 How PDFPlumber is Actually Used
-
-### Workflow
-
-1. **Import Check**: On module load, attempts to import pdfplumber
-2. **Availability Flag**: Sets `PDF_AVAILABLE = True/False` based on import success
-3. **Runtime Check**: `has_pdf_support()` method checks availability
-4. **Extraction Attempt**: When processing datasets:
-   - First tries to find text in standard fields (text, story, content, etc.)
-   - If no text found AND `has_pdf_support()` returns True:
-     - Searches for PDF fields (pdf, file, document)
-     - Calls `extract_pdf_text()` to extract content
-     - Logs extraction success with character count
-5. **Graceful Fallback**: If PDF extraction fails or unavailable:
-   - Creates informative placeholder entries
-   - Includes metadata about PDF availability
-   - Maintains system functionality
-
-### Example from `transform_novels()`
-
-```python
-# Try text fields first
-for field in ['text', 'story', 'content', 'novel', 'body', 'full_text']:
-    if field in item and item[field]:
-        text = item[field]
-        break
-
-# If no text, try PDF extraction
-if not text and self.has_pdf_support():
-    for pdf_field in ['pdf', 'file', 'document']:
-        if pdf_field in item and item[pdf_field]:
-            text = self.extract_pdf_text(item[pdf_field])
-            if text:
-                logger.info(f"Novel {idx + 1}: Extracted {len(text)} chars from PDF")
-                break
-
-# If still no text, create placeholder
-if not text:
-    text = f"""[Novel Content Unavailable]
-    
-This novel (#{idx + 1}) is part of the GOAT-AI/generated-novels dataset.
-The original content may be stored in PDF format or require special extraction.
-
-PDF extraction support: {'Available (install pdfplumber)' if not self.has_pdf_support() else 'Enabled'}
-"""
-```
-
-## 🎯 Tactical Assessment
-
-### Current Strategy: ✅ SOUND
-
-The current approach is **well-designed** and does NOT require changing tactics:
-
-1. **Graceful Degradation**: System works with or without pdfplumber
-2. **Multiple Fallbacks**: Tries text fields first, then PDF, then placeholders
-3. **Informative Placeholders**: When content unavailable, creates useful metadata
-4. **Proper Error Handling**: All PDF operations wrapped in try-except
-5. **Logging**: Provides visibility into extraction success/failure
-
-### Recommendations
-
-#### 1. **Keep Current Approach** ✅
-
-The multi-layered fallback strategy is excellent for production systems.
-
-#### 2. **Fix Applied Bugs** ✅
-
-- Fixed `PDF_SUPPORT` → `PDF_AVAILABLE` variable name
-- Fixed duplicate `import io` statement
-
-#### 3. **Optional Enhancement**: Add to pyproject.toml
-
-Consider adding pdfplumber to optional dependencies:
-
-```toml
-[project.optional-dependencies]
-pdf = [
-    "pdfplumber>=0.11.0",
-]
-```
-
-#### 4. **Documentation Enhancement**
-
-The code already has good inline documentation. Consider adding to README:
-
-- How to enable PDF support
-- What happens when PDF support is unavailable
-- Which datasets benefit from PDF extraction
-
-## 📊 Test Coverage
-
-The test suite (`test_pdf_ingestion.py`) covers:
-
-- ✅ PDF support detection
-- ✅ PDF extraction method existence
-- ✅ Placeholder creation
-- ✅ Novel dataset with PDF fields
-- ✅ Novel dataset with text fields
-- ✅ Portuguese education with PDF fields
-- ✅ Output format validation
-
-## 🎓 Conclusion
-
-**PDFPlumber IS being utilized properly** in the ingesters. The implementation:
-
-- ✅ Has proper import and availability checking
-- ✅ Provides two PDF extraction methods (simple and flexible)
-- ✅ Integrates PDF extraction into dataset transformers
-- ✅ Has comprehensive fallback mechanisms
-- ✅ Is well-tested
-- ✅ Is properly documented
-
-**Bugs Fixed**:
-
-1. Variable name typo: `PDF_SUPPORT` → `PDF_AVAILABLE`
-2. Duplicate import: Moved `import io` to module level
-
-**No tactical changes needed** - the current approach is sound and production-ready.
-
-## 📝 Files Modified
-
-1. `warbler-cda-package/warbler_cda/utils/hf_warbler_ingest.py`
-   - Fixed variable name in `_extract_pdf_text()` method
-   - Added `import io` to module-level imports
-   - Removed duplicate `import io` from method
-
-## 🔗 Related Files
-
-- `warbler-cda-package/requirements.txt` - Lists pdfplumber>=0.11.0
-- `warbler-cda-package/tests/test_pdf_ingestion.py` - Test suite for PDF functionality
-- `warbler-cda-package/pyproject.toml` - Package configuration (could add optional PDF dependency)

QUICKSTART.md

@@ -1,191 +0,0 @@
-# Warbler CDA - Quick Start Guide
-
-## 🚀 Quick Start (3 options)
-
-### 📝 Home may not be available on path immediately
-
-```bash
-# set home path for environment
-echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
-# start the terminal
-source ~/.bashrc
-```
-
-### Option 1: Local Python (Recommended for Development)
-
-```bash
-cd warbler-cda-package
-./setup.sh
-python app.py
-```
-
-Open <http://localhost:7860>
-
-### Option 2: Docker
-
-```bash
-cd warbler-cda-package
-docker-compose up warbler-cda-demo
-```
-
-Open <http://localhost:7860>
-
-### Option 3: HuggingFace Space (Recommended for Sharing)
-
-1. Create a HuggingFace Space at <https://huggingface.co/new-space>
-2. Choose "Gradio" as SDK
-3. Upload the `warbler-cda-package/` contents
-4. Your Space will be live at `https://huggingface.co/spaces/YOUR_USERNAME/warbler-cda`
-
-## 📚 Usage Examples
-
-### Example 1: Basic Query
-
-```python
-from warbler_cda import RetrievalAPI, EmbeddingProviderFactory
-
-# Initialize
-embedding_provider = EmbeddingProviderFactory.get_default_provider()
-api = RetrievalAPI(embedding_provider=embedding_provider)
-
-# Add document
-api.add_document(
-    doc_id="wisdom_1",
-    content="Courage is not the absence of fear, but acting despite it.",
-    metadata={"realm_type": "wisdom", "realm_label": "virtue"}
-)
-
-# Query
-results = api.query_semantic_anchors("What is courage?", max_results=5)
-for result in results:
-    print(f"{result.relevance_score:.3f} - {result.content}")
-```
-
-### Example 2: FractalStat Hybrid Scoring
-
-```python
-from warbler_cda import FractalStatRAGBridge, RetrievalQuery, RetrievalMode
-
-# Enable FractalStat
-fractalstat_bridge = FractalStatRAGBridge()
-api = RetrievalAPI(
-    embedding_provider=embedding_provider,
-    fractalstat_bridge=fractalstat_bridge,
-    config={"enable_fractalstat_hybrid": True}
-)
-
-# Query with hybrid scoring
-query = RetrievalQuery(
-    query_id="hybrid_1",
-    mode=RetrievalMode.SEMANTIC_SIMILARITY,
-    semantic_query="wisdom about resilience",
-    fractalstat_hybrid=True,
-    weight_semantic=0.6,
-    weight_fractalstat=0.4
-)
-
-assembly = api.retrieve_context(query)
-print(f"Quality: {assembly.assembly_quality:.3f}")
-print(f"Results: {len(assembly.results)}")
-```
-
-### Example 3: API Service
-
-```bash
-# Start the API
-uvicorn warbler_cda.api.service:app --host 0.0.0.0 --port 8000
-
-# In another terminal, use the CLI
-warbler-cli query --query-id q1 --semantic "wisdom about courage" --hybrid
-
-# Or use curl
-curl -X POST http://localhost:8000/query \
-  -H "Content-Type: application/json" \
-  -d '{
-    "query_id": "test1",
-    "semantic_query": "wisdom about courage",
-    "fractalstat_hybrid": true
-  }'
-```
-
-## 🔧 Configuration
-
-### Embedding Providers
-
-```python
-# Local TF-IDF (default, no API key needed)
-from warbler_cda import EmbeddingProviderFactory
-provider = EmbeddingProviderFactory.create_provider("local")
-
-# OpenAI (requires API key)
-provider = EmbeddingProviderFactory.create_provider(
-    "openai",
-    config={"api_key": "your-api-key", "model": "text-embedding-ada-002"}
-)
-```
-
-### FractalStat Configuration
-
-```python
-# Custom FractalStat weights
-api = RetrievalAPI(
-    fractalstat_bridge=fractalstat_bridge,
-    config={
-        "enable_fractalstat_hybrid": True,
-        "default_weight_semantic": 0.7,  # 70% semantic
-        "default_weight_fractalstat": 0.3       # 30% FractalStat
-    }
-)
-```
-
-## 📊 Running Experiments
-
-```python
-from warbler_cda import run_all_experiments
-
-# Run FractalStat validation experiments
-results = run_all_experiments(
-    exp01_samples=1000,
-    exp01_iterations=10,
-    exp02_queries=1000,
-    exp03_samples=1000
-)
-
-print(f"EXP-01 (Uniqueness): {results['EXP-01']['success']}")
-print(f"EXP-02 (Efficiency): {results['EXP-02']['success']}")
-print(f"EXP-03 (Necessity): {results['EXP-03']['success']}")
-```
-
-## 🐛 Troubleshooting
-
-### Import Errors
-
-If you see import errors, make sure the package is installed:
-
-```bash
-pip install -e .
-```
-
-### Missing Dependencies
-
-Install all dependencies:
-
-```bash
-pip install -r requirements.txt
-```
-
-### Gradio Not Starting
-
-Check if port 7860 is available:
-
-```bash
-lsof -i :7860  # Linux/Mac
-netstat -ano | findstr :7860  # Windows
-```
-
-## 📖 More Information
-
-- Full documentation: [README.md](README.md)
-- Deployment guide: [DEPLOYMENT.md](DEPLOYMENT.md)
-- Contributing: [CONTRIBUTING.md](CONTRIBUTING.md)
-- Package manifest: [PACKAGE_MANIFEST.md](PACKAGE_MANIFEST.md)

README.md

@@ -1,390 +0,0 @@
----
-title: Warbler CDA FractalStat RAG
-emoji: 🦜
-colorFrom: blue
-colorTo: purple
-sdk: gradio
-sdk_version: 4.44.0
-app_file: app.py
-pinned: false
-license: mit
-short_description: RAG system with 8D FractalStat and 2.6M+ documents
-tags:
-  - rag
-  - semantic-search
-  - retrieval
-  - fastapi
-  - fractalstat
----
-
-# Warbler CDA - Cognitive Development Architecture RAG System
-
-[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
-[![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)
-[![FastAPI](https://img.shields.io/badge/FastAPI-0.100+-green.svg)](https://fastapi.tiangolo.com/)
-[![Docker](https://img.shields.io/badge/Docker-ready-blue.svg)](https://docker.com)
-
-A **production-ready RAG (Retrieval-Augmented Generation) system** with **FractalStat multi-dimensional addressing** for intelligent document retrieval, semantic memory, and automatic data ingestion.
-
-## 🌟 Features
-
-### Core RAG System
-
-- **Semantic Anchors**: Persistent memory with provenance tracking
-- **Hierarchical Summarization**: Micro/macro distillation for efficient compression
-- **Conflict Detection**: Automatic detection and resolution of contradictory information
-- **Memory Pooling**: Performance-optimized object pooling for high-throughput scenarios
-
-### FractalStat Multi-Dimensional Addressing
-
-- **8-Dimensional Coordinates**: Realm, Lineage, Adjacency, Horizon, Luminosity, Polarity, Dimensionality, Alignment
-- **Hybrid Scoring**: Combines semantic similarity with FractalStat resonance for superior retrieval
-- **Entanglement Detection**: Identifies relationships across dimensional space
-- **Validated System**: Comprehensive experiments (EXP-01 through EXP-10) validate uniqueness, efficiency, and narrative preservation
-
-### Production-Ready API
-
-- **FastAPI Service**: High-performance async API with concurrent query support
-- **CLI Tools**: Command-line interface for queries, ingestion, and management
-- **HuggingFace Integration**: Direct ingestion from HF datasets
-- **Docker Support**: Containerized deployment ready
-
-## 📚 Data Sources
-
-The Warbler system is trained on carefully curated, MIT-licensed datasets from HuggingFace:
-
-### Primary Datasets
-
-- **arXiv Papers** (`nick007x/arxiv-papers`) - 2.5M+ scholarly papers covering scientific domains
-- **Prompt Engineering Report** (`PromptSystematicReview/ThePromptReport`) - 83 comprehensive prompt documentation entries
-- **Generated Novels** (`GOAT-AI/generated-novels`) - 20 narrative-rich novels for storytelling patterns
-- **Technical Manuals** (`nlasso/anac-manuals-23`) - 52 procedural and operational documents
-- **ChatEnv Enterprise** (`SustcZhangYX/ChatEnv`) - 112K+ software development conversations
-- **Portuguese Education** (`Solshine/Portuguese_Language_Education_Texts`) - 21 multilingual educational texts
-- **Educational Stories** (`MU-NLPC/Edustories-en`) - 1.5K+ case studies and learning narratives
-
-### Original Warbler Packs
-
-- `warbler-pack-core` - Core narrative and reasoning patterns
-- `warbler-pack-wisdom-scrolls` - Philosophical and wisdom-based content
-- `warbler-pack-faction-politics` - Political and faction dynamics
-
-All datasets are provided under MIT or compatible licenses. For complete attribution, see the HuggingFace Hub pages listed above.
-
-## 📦 Installation
-
-### From Source (Current Method)
-
-```bash
-git clone https://github.com/tiny-walnut-games/the-seed.git
-cd the-seed/warbler-cda-package
-pip install -e .
-```
-
-### Optional Dependencies
-
-```bash
-# OpenAI embeddings integration
-pip install openai
-
-# Development tools
-pip install pytest pytest-cov
-```
-
-## 🚀 Quick Start
-
-### Option 1: Direct Python (Easiest)
-
-```bash
-cd warbler-cda-package
-
-# Start the API with automatic pack loading
-./run_api.ps1
-
-# Or on Linux/Mac:
-python start_server.py
-```
-
-The API automatically loads all Warbler packs on startup and serves them at **http://localhost:8000**
-
-### Option 2: Docker Compose
-
-```bash
-cd warbler-cda-package
-docker-compose up --build
-```
-
-### Option 3: Kubernetes
-
-```bash
-cd warbler-cda-package/k8s
-./demo-docker-k8s.sh  # Full auto-deploy
-```
-
-## 📡 API Usage Examples
-
-### Using the REST API
-
-```bash
-# Start the API first: ./run_api.ps1
-# Then test with:
-
-# Health check
-curl http://localhost:8000/health
-
-# Query the system
-curl -X POST http://localhost:8000/query \
-  -H "Content-Type: application/json" \
-  -d '{
-    "query_id": "test1",
-    "semantic_query": "hello world",
-    "max_results": 5
-  }'
-
-# Get metrics
-curl http://localhost:8000/metrics
-```
-
-### Using Python Programmatically
-
-```python
-import requests
-
-# Health check
-response = requests.get("http://localhost:8000/health")
-print(f"API Status: {response.json()['status']}")
-
-# Query
-query_data = {
-    "query_id": "python_test",
-    "semantic_query": "rotation dynamics of Saturn's moons",
-    "max_results": 5,
-    "fractalstat_hybrid": True
-}
-
-results = requests.post("http://localhost:8000/query", json=query_data).json()
-print(f"Found {len(results['results'])} results")
-
-# Show top result
-if results['results']:
-    top_result = results['results'][0]
-    print(f"Top score: {top_result['relevance_score']:.3f}")
-    print(f"Content: {top_result['content'][:100]}...")
-```
-
-### FractalStat Hybrid Scoring
-
-```python
-from warbler_cda import FractalStatRAGBridge
-
-# Enable FractalStat hybrid scoring
-fractalstat_bridge = FractalStatRAGBridge()
-api = RetrievalAPI(
-    semantic_anchors=semantic_anchors,
-    embedding_provider=embedding_provider,
-    fractalstat_bridge=fractalstat_bridge,
-    config={"enable_fractalstat_hybrid": True}
-)
-
-# Query with hybrid scoring
-from warbler_cda import RetrievalQuery, RetrievalMode
-
-query = RetrievalQuery(
-    query_id="hybrid_query_1",
-    mode=RetrievalMode.SEMANTIC_SIMILARITY,
-    semantic_query="Find wisdom about resilience",
-    fractalstat_hybrid=True,
-    weight_semantic=0.6,
-    weight_fractalstat=0.4
-)
-
-assembly = api.retrieve_context(query)
-print(f"Found {len(assembly.results)} results with quality {assembly.assembly_quality:.3f}")
-```
-
-### Running the API Service
-
-```bash
-# Start the FastAPI service
-uvicorn warbler_cda.api.service:app --host 0.0.0.0 --port 8000
-
-# Or use the CLI
-warbler-api --port 8000
-```
-
-### Using the CLI
-
-```bash
-# Query the API
-warbler-cli query --query-id q1 --semantic "wisdom about courage" --max-results 10
-
-# Enable hybrid scoring
-warbler-cli query --query-id q2 --semantic "narrative patterns" --hybrid
-
-# Bulk concurrent queries
-warbler-cli bulk --num-queries 10 --concurrency 5 --hybrid
-
-# Check metrics
-warbler-cli metrics
-```
-
-## 📊 FractalStat Experiments
-
-The system includes validated experiments demonstrating:
-
-- **EXP-01**: Address uniqueness (0% collision rate across 10K+ entities)
-- **EXP-02**: Retrieval efficiency (sub-millisecond at 100K scale)
-- **EXP-03**: Dimension necessity (all 7 dimensions required)
-- **EXP-10**: Narrative preservation under concurrent load
-
-```python
-from warbler_cda import run_all_experiments
-
-# Run validation experiments
-results = run_all_experiments(
-    exp01_samples=1000,
-    exp01_iterations=10,
-    exp02_queries=1000,
-    exp03_samples=1000
-)
-
-print(f"EXP-01 Success: {results['EXP-01']['success']}")
-print(f"EXP-02 Success: {results['EXP-02']['success']}")
-print(f"EXP-03 Success: {results['EXP-03']['success']}")
-```
-
-## 🎯 Use Cases
-
-### 1. Intelligent Document Retrieval
-
-```python
-# Add documents from various sources
-for doc in documents:
-    api.add_document(
-        doc_id=doc["id"],
-        content=doc["text"],
-        metadata={
-            "realm_type": "knowledge",
-            "realm_label": "technical_docs",
-            "lifecycle_stage": "emergence"
-        }
-    )
-
-# Retrieve with context awareness
-results = api.query_semantic_anchors("How to optimize performance?")
-```
-
-### 2. Narrative Coherence Analysis
-
-```python
-from warbler_cda import ConflictDetector
-
-conflict_detector = ConflictDetector(embedding_provider=embedding_provider)
-
-# Process statements
-statements = [
-    {"id": "s1", "text": "The system is fast"},
-    {"id": "s2", "text": "The system is slow"}
-]
-
-report = conflict_detector.process_statements(statements)
-print(f"Conflicts detected: {report['conflict_summary']}")
-```
-
-### 3. HuggingFace Dataset Ingestion
-
-```python
-from warbler_cda.utils import HFWarblerIngestor
-
-ingestor = HFWarblerIngestor()
-
-# Transform HF dataset to Warbler format
-docs = ingestor.transform_npc_dialogue("amaydle/npc-dialogue")
-
-# Create pack
-pack_path = ingestor.create_warbler_pack(docs, "warbler-pack-npc-dialogue")
-```
-
-## 🏗️ Architecture
-
-```none
-warbler_cda/
-├── retrieval_api.py          # Main RAG API
-├── semantic_anchors.py        # Semantic memory system
-├── anchor_data_classes.py     # Core data structures
-├── anchor_memory_pool.py      # Performance optimization
-├── summarization_ladder.py    # Hierarchical compression
-├── conflict_detector.py       # Conflict detection
-├── castle_graph.py            # Concept extraction
-├── melt_layer.py              # Memory consolidation
-├── evaporation.py             # Content distillation
-├── fractalstat_rag_bridge.py        # FractalStat hybrid scoring
-├── fractalstat_entity.py            # FractalStat entity system
-├── fractalstat_experiments.py       # Validation experiments
-├── embeddings/                # Embedding providers
-│   ├── base_provider.py
-│   ├── local_provider.py
-│   ├── openai_provider.py
-│   └── factory.py
-├── api/                       # Production API
-│   ├── service.py             # FastAPI service
-│   └── cli.py                 # CLI interface
-└── utils/                     # Utilities
-    ├── load_warbler_packs.py
-    └── hf_warbler_ingest.py
-```
-
-## 🔬 Technical Details
-
-### FractalStat Dimensions
-
-1. **Realm**: Domain classification (type + label)
-2. **Lineage**: Generation/version number
-3. **Adjacency**: Graph connectivity (0.0-1.0)
-4. **Horizon**: Lifecycle stage (logline, outline, scene, panel)
-5. **Luminosity**: Clarity/activity level (0.0-1.0)
-6. **Polarity**: Resonance/tension (0.0-1.0)
-7. **Dimensionality**: Complexity/thread count (1-7)
-
-### Hybrid Scoring Formula
-
-```math
-hybrid_score = (weight_semantic × semantic_similarity) + (weight_fractalstat × fractalstat_resonance)
-```
-
-Where:
-
-- `semantic_similarity`: Cosine similarity of embeddings
-- `fractalstat_resonance`: Multi-dimensional alignment score
-- Default weights: 60% semantic, 40% FractalStat
-
-## 📚 Documentation
-
-- [API Reference](docs/api.md)
-- [FractalStat Guide](docs/fractalstat.md)
-- [Experiments](docs/experiments.md)
-- [Deployment](docs/deployment.md)
-
-## 🤝 Contributing
-
-Contributions are welcome! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
-
-## 📄 License
-
-MIT License - see [LICENSE](LICENSE) for details.
-
-## 🙏 Acknowledgments
-
-- Built on research from The Seed project
-- FractalStat addressing system inspired by multi-dimensional data structures
-- Semantic anchoring based on cognitive architecture principles
-
-## 📞 Contact
-
-- **Project**: [The Seed](https://github.com/tiny-walnut-games/the-seed)
-- **Issues**: [GitHub Issues](https://github.com/tiny-walnut-games/the-seed/issues)
-- **Discussions**: [GitHub Discussions](https://github.com/tiny-walnut-games/the-seed/discussions)
-
----
-
-### **Made with ❤️ by Tiny Walnut Games**

README_HF.md

@@ -1,57 +0,0 @@
----
-title: Warbler CDA - FractalStat RAG System
-emoji: 🦜
-colorFrom: blue
-colorTo: purple
-sdk: docker
-pinned: false
-license: mit
----
-
-# Warbler CDA - Cognitive Development Architecture
-
-A production-ready RAG system with **FractalStat 8D multi-dimensional addressing** for intelligent document retrieval.
-
-## 🚀 Quick Start
-
-This Space runs a FastAPI service on port 7860.
-
-### Query the API
-
-```bash
-curl -X POST https://YOUR-USERNAME-warbler-cda.hf.space/query \
-  -H "Content-Type: application/json" \
-  -d '{
-    "query_id": "test1",
-    "semantic_query": "hello world",
-    "max_results": 5
-  }'
-```
-
-### API Endpoints
-
-- `GET /health` - Health check
-- `POST /query` - Semantic query with optional FractalStat hybrid scoring
-- `GET /metrics` - System metrics
-- `GET /docs` - Interactive API documentation
-
-## 🌟 Features
-
-- **Semantic Retrieval**: Find documents by meaning, not just keywords
-- **FractalStat 8D Addressing**: Multi-dimensional intelligence for superior ranking
-- **Bob the Skeptic**: Automatic bias detection and validation
-- **Narrative Coherence**: Analyzes result quality and threading
-- **10k+ Documents**: Pre-indexed arXiv papers, education, fiction, and more
-
-## 📊 Performance
-
-- **Avg Response Time**: 9-28s (depending on query complexity)
-- **Avg Relevance**: 0.88
-- **Narrative Coherence**: 75-83%
-- **Coverage**: 84% test coverage with 587 passing tests
-
-## 🔗 Links
-
-- [Full Documentation](https://gitlab.com/tiny-walnut-games/the-seed/-/tree/main/warbler-cda-package)
-- [Source Code](https://gitlab.com/tiny-walnut-games/the-seed)
-- [Performance Report](https://gitlab.com/tiny-walnut-games/the-seed/-/blob/main/warbler-cda-package/WARBLER_CDA_PERFORMANCE_REPORT.md)

TESTS_PORTED.md

@@ -1,271 +0,0 @@
-# Tests Ported to Warbler CDA Package
-
-This document summarizes the TDD (Test-Driven Development) test suite that has been ported from the main project to the warbler-cda-package for HuggingFace deployment.
-
-## Overview
-
-The complete test suite for the Warbler CDA (Cognitive Development Architecture) RAG system has been ported and adapted for the standalone package. This includes:
-
-- **4 main test modules** with comprehensive coverage
-- **1 end-to-end integration test suite**
-- **Pytest configuration** with custom markers
-- **Test documentation** and running instructions
-
-## Test Files Ported
-
-### 1. **tests/test_embedding_providers.py** (9.5 KB)
-
-**Source**: Adapted from `packages/com.twg.the-seed/The Living Dev Agent/tests/test_semantic_anchors.py`
-
-**Coverage**:
-
-- EmbeddingProviderFactory pattern
-- LocalEmbeddingProvider (TF-IDF based)
-- SentenceTransformerEmbeddingProvider (GPU-accelerated)
-- Embedding generation (single and batch)
-- Similarity calculations
-- Provider information and metadata
-
-**Tests**:
-
-- `test_factory_creates_local_provider` - Factory can create local providers
-- `test_factory_list_available_providers` - Factory lists available providers
-- `test_factory_default_provider` - Factory defaults to SentenceTransformer with fallback
-- `test_embed_single_text` - Single text embedding
-- `test_embed_batch` - Batch embedding
-- `test_similarity_calculation` - Cosine similarity
-- `test_semantic_search` - K-nearest neighbor search
-- `test_stat7_computation` - STAT7 coordinate computation
-- And 8 more embedding-focused tests
-
-### 2. **tests/test_retrieval_api.py** (11.9 KB)
-
-**Source**: Adapted from `packages/com.twg.the-seed/seed/engine/test_retrieval_debug.py`
-
-**Coverage**:
-
-- Context store operations
-- Document addition and deduplication
-- Query execution and filtering
-- Retrieval modes (semantic, temporal, composite)
-- Confidence threshold filtering
-- Result structure validation
-- Caching and metrics
-
-**Tests**:
-
-- `TestRetrievalAPIContextStore` - 4 tests for document store
-- `TestRetrievalQueryExecution` - 5 tests for query operations
-- `TestRetrievalModes` - 3 tests for different retrieval modes
-- `TestRetrievalHybridScoring` - 2 tests for STAT7 hybrid scoring
-- `TestRetrievalMetrics` - 2 tests for metrics tracking
-- Total: 16+ tests
-
-### 3. **tests/test_stat7_integration.py** (12.3 KB)
-
-**Source**: Original implementation for STAT7 support
-
-**Coverage**:
-
-- STAT7 coordinate computation from embeddings
-- Hybrid semantic + STAT7 scoring
-- STAT7 resonance calculation
-- Document enrichment with STAT7 data
-- Multi-dimensional query addressing
-- STAT7 dimensional properties
-
-**Tests**:
-
-- `TestSTAT7CoordinateComputation` - 3 tests
-- `TestSTAT7HybridScoring` - 3 tests
-- `TestSTAT7DocumentEnrichment` - 2 tests
-- `TestSTAT7QueryAddressing` - 2 tests
-- `TestSTAT7Dimensions` - 2 tests
-- Total: 12+ tests
-
-### 4. **tests/test_rag_e2e.py** (12.6 KB)
-
-**Source**: Adapted from `packages/com.twg.the-seed/The Living Dev Agent/tests/test_exp08_rag_integration.py`
-
-**Coverage**:
-
-- Complete end-to-end RAG pipeline
-- Embedding generation validation
-- Document ingestion
-- Semantic search retrieval
-- Temporal retrieval
-- Metrics tracking
-- Full system integration
-
-**Tests**:
-
-1. `test_01_embedding_generation` - Embeddings are generated
-2. `test_02_embedding_similarity` - Similarity scoring works
-3. `test_03_document_ingestion` - Documents are ingested
-4. `test_04_semantic_search` - Semantic search works
-5. `test_05_max_results_respected` - Result limiting works
-6. `test_06_confidence_threshold` - Threshold filtering works
-7. `test_07_stat7_hybrid_scoring` - Hybrid scoring works
-8. `test_08_temporal_retrieval` - Temporal queries work
-9. `test_09_retrieval_metrics` - Metrics are tracked
-10. `test_10_full_rag_pipeline` - Complete pipeline works
-
-### 5. **tests/conftest.py** (1.6 KB)
-
-**Purpose**: Pytest configuration and fixtures
-
-**Includes**:
-
-- Custom pytest markers (embedding, retrieval, stat7, e2e, slow)
-- Test data fixtures
-- Pytest configuration hooks
-
-### 6. **tests/README.md** (5.6 KB)
-
-**Purpose**: Test documentation
-
-**Contains**:
-
-- Test organization overview
-- Running instructions
-- Test coverage summary
-- Troubleshooting guide
-- CI/CD integration examples
-
-## Test Statistics
-
-| Category | Count |
-|----------|-------|
-| Total Test Classes | 16 |
-| Total Test Methods | 50+ |
-| Total Test Files | 4 |
-| Test Size | ~47 KB |
-| Coverage Scope | 90%+ of core functionality |
-
-## Key Testing Areas
-
-### Embedding Providers
-
-- ✅ Local TF-IDF provider (no dependencies)
-- ✅ SentenceTransformer provider (GPU acceleration)
-- ✅ Factory pattern with graceful fallback
-- ✅ Batch processing
-- ✅ Similarity calculations
-- ✅ Semantic search
-
-### Retrieval Operations
-
-- ✅ Document ingestion and storage
-- ✅ Context store management
-- ✅ Query execution
-- ✅ Semantic similarity retrieval
-- ✅ Temporal sequence retrieval
-- ✅ Composite retrieval modes
-
-### STAT7 Integration
-
-- ✅ Coordinate computation from embeddings
-- ✅ Hybrid scoring (semantic + STAT7)
-- ✅ Resonance calculations
-- ✅ Multi-dimensional addressing
-- ✅ Document enrichment
-
-### System Integration
-
-- ✅ End-to-end pipeline
-- ✅ Metrics and performance tracking
-- ✅ Caching mechanisms
-- ✅ Error handling and fallbacks
-
-## Running the Tests
-
-### Quick Start
-
-```bash
-cd warbler-cda-package
-pytest tests/ -v
-```
-
-### Detailed Examples
-
-```bash
-# Run all tests with output
-pytest tests/ -v -s
-
-# Run with coverage report
-pytest tests/ --cov=warbler_cda --cov-report=html
-
-# Run only embedding tests
-pytest tests/test_embedding_providers.py -v
-
-# Run only end-to-end tests
-pytest tests/test_rag_e2e.py -v -s
-
-# Run tests matching a pattern
-pytest tests/ -k "semantic" -v
-```
-
-## Compatibility
-
-### With SentenceTransformer Installed
-
-- All 50+ tests pass
-- GPU acceleration available
-- Full STAT7 integration enabled
-
-### Without SentenceTransformer
-
-- Tests gracefully skip SentenceTransformer-specific tests
-- Fallback to local TF-IDF provider
-- ~40 tests pass
-- STAT7 tests skipped
-
-## Design Principles
-
-The ported tests follow TDD principles:
-
-1. **Isolation**: Each test is independent and can run standalone
-2. **Clarity**: Test names describe what is being tested
-3. **Completeness**: Happy path and edge cases covered
-4. **Robustness**: Graceful handling of optional dependencies
-5. **Documentation**: Each test is well-commented and documented
-
-## Integration with CI/CD
-
-The tests are designed for easy integration with CI/CD pipelines:
-
-```yaml
-# Example GitHub Actions workflow
-- name: Run Warbler CDA Tests
-  run: |
-    cd warbler-cda-package
-    pytest tests/ --cov=warbler_cda --cov-report=xml
-```
-
-## Future Test Additions
-
-Recommended areas for additional tests:
-
-1. Performance benchmarking
-2. Stress testing with large document collections
-3. Concurrent query handling
-4. Cache invalidation scenarios
-5. Error recovery mechanisms
-6. Large-scale STAT7 coordinate distribution analysis
-
-## Notes
-
-- Tests use pytest fixtures for setup/teardown
-- Custom markers enable selective test execution
-- Graceful fallback for optional dependencies
-- Comprehensive end-to-end validation
-- Documentation-as-tests through verbose assertions
-
-## Maintenance
-
-When updating the package:
-
-1. Run tests after any changes: `pytest tests/ -v`
-2. Update tests if new functionality is added
-3. Keep end-to-end tests as verification baseline
-4. Monitor test execution time for performance regressions

TEST_RESULTS.md

@@ -1,211 +0,0 @@
-# Test Results: MIT-Licensed Datasets Integration
-
-**Date**: November 8, 2025  
-**Status**: ✅ **ALL TESTS PASSING**  
-**Total Tests**: 71  
-**Passed**: 71  
-**Failed**: 0  
-**Skipped**: 0
-
----
-
-## Test Summary
-
-### New MIT-Licensed Dataset Tests: 18/18 ✅
-
-| Test Class | Tests | Status |
-|-----------|-------|--------|
-| TestArxivPapersTransformer | 4 | ✅ PASS |
-| TestPromptReportTransformer | 2 | ✅ PASS |
-| TestGeneratedNovelsTransformer | 2 | ✅ PASS |
-| TestManualnsTransformer | 2 | ✅ PASS |
-| TestEnterpriseTransformer | 2 | ✅ PASS |
-| TestPortugueseEducationTransformer | 2 | ✅ PASS |
-| TestNewDatasetsIntegrationWithRetrieval | 2 | ✅ PASS |
-| TestNewDatasetsPerformance | 1 | ✅ PASS |
-| TestNewDatasetsAllAtOnce | 1 | ✅ PASS |
-| **Total New Tests** | **18** | **✅ 100%** |
-
-### Existing Warbler-CDA Tests: 53/53 ✅
-
-| Test Module | Tests | Status |
-|------------|-------|--------|
-| test_embedding_providers.py | 11 | ✅ PASS |
-| test_rag_e2e.py | 10 | ✅ PASS |
-| test_retrieval_api.py | 13 | ✅ PASS |
-| test_stat7_integration.py | 12 | ✅ PASS |
-| test_embedding_integration.py | 7 | ✅ PASS |
-| **Total Existing Tests** | **53** | **✅ 100%** |
-
----
-
-## Individual Test Results
-
-### ✅ New Transformer Tests (18 PASSED)
-
-```log
-tests/test_new_mit_datasets.py::TestArxivPapersTransformer::test_arxiv_transformer_exists PASSED
-tests/test_new_mit_datasets.py::TestArxivPapersTransformer::test_arxiv_output_format PASSED
-tests/test_new_mit_datasets.py::TestArxivPapersTransformer::test_arxiv_metadata_fields PASSED
-tests/test_new_mit_datasets.py::TestArxivPapersTransformer::test_arxiv_limit_parameter PASSED
-tests/test_new_mit_datasets.py::TestPromptReportTransformer::test_prompt_report_transformer_exists PASSED
-tests/test_new_mit_datasets.py::TestPromptReportTransformer::test_prompt_report_output_format PASSED
-tests/test_new_mit_datasets.py::TestGeneratedNovelsTransformer::test_novels_transformer_exists PASSED
-tests/test_new_mit_datasets.py::TestGeneratedNovelsTransformer::test_novels_chunking_for_long_text PASSED
-tests/test_new_mit_datasets.py::TestManualnsTransformer::test_manuals_transformer_exists PASSED
-tests/test_new_mit_datasets.py::TestManualnsTransformer::test_manuals_output_format PASSED
-tests/test_new_mit_datasets.py::TestEnterpriseTransformer::test_enterprise_transformer_exists PASSED
-tests/test_new_mit_datasets.py::TestEnterpriseTransformer::test_enterprise_output_format PASSED
-tests/test_new_mit_datasets.py::TestPortugueseEducationTransformer::test_portuguese_transformer_exists PASSED
-tests/test_new_mit_datasets.py::TestPortugueseEducationTransformer::test_portuguese_multilingual_metadata PASSED
-tests/test_new_mit_datasets.py::TestNewDatasetsIntegrationWithRetrieval::test_warbler_document_structure PASSED
-tests/test_new_mit_datasets.py::TestNewDatasetsIntegrationWithRetrieval::test_pack_creation_with_new_datasets PASSED
-tests/test_new_mit_datasets.py::TestNewDatasetsPerformance::test_arxiv_handles_large_dataset PASSED
-tests/test_new_mit_datasets.py::TestNewDatasetsAllAtOnce::test_all_transformers_callable PASSED
-```
-
-### ✅ Backward Compatibility Tests (53 PASSED)
-
-All existing tests continue to pass, confirming backward compatibility:
-
-- Embedding provider interface tests ✅
-- RAG end-to-end pipeline ✅
-- Retrieval API functionality ✅
-- STAT7 integration and hybrid scoring ✅
-- Embedding integration ✅
-
----
-
-## Test Execution Details
-
-### Command
-
-```bash
-C:\Users\jerio\AppData\Local\Programs\Python\Python312\python.exe -m pytest tests/ -v
-```
-
-### Execution Time
-
-- Total: 58.70 seconds
-- New tests: ~13 seconds
-- Existing tests: ~45 seconds
-
-### Environment
-
-- Python: 3.12.10
-- pytest: 8.4.2
-- Platform: Windows (win32)
-
----
-
-## Coverage by Transformer
-
-### arXiv Papers (4 tests)
-
-- ✅ Transformer exists and is callable
-- ✅ Output format matches Warbler structure
-- ✅ Metadata includes required fields
-- ✅ Limit parameter respected
-
-### Prompt Report (2 tests)
-
-- ✅ Transformer exists
-- ✅ Output format correct
-
-### Generated Novels (2 tests)
-
-- ✅ Transformer exists
-- ✅ Text chunking functionality
-
-### Technical Manuals (2 tests)
-
-- ✅ Transformer exists
-- ✅ Output format correct
-
-### Enterprise Benchmarks (2 tests)
-
-- ✅ Transformer exists
-- ✅ Output format correct
-
-### Portuguese Education (2 tests)
-
-- ✅ Transformer exists
-- ✅ Multilingual metadata
-
-### Integration (2 tests)
-
-- ✅ Warbler document structure validation
-- ✅ Pack creation with mocked filesystem
-
-### Performance (1 test)
-
-- ✅ Large dataset handling (100+ papers in <10s)
-
-### All Transformers Callable (1 test)
-
-- ✅ All 6 new transformers verified as callable
-
----
-
-## Issues Found & Fixed
-
-### Issue 1: Mock WindowsPath AttributeError
-
-**Problem**: Test tried to mock `mkdir` attribute on real Path object  
-**Solution**: Used MagicMock instead of real Path  
-**Status**: ✅ Fixed - all tests now pass
-
----
-
-## Validation Checklist
-
-- [x] All new transformer methods are implemented
-- [x] All helper methods are implemented  
-- [x] Output format matches Warbler structure
-- [x] MIT license field present in all documents
-- [x] Metadata fields required (realm_type, realm_label, etc)
-- [x] Error handling in place
-- [x] CLI integration works
-- [x] Backward compatibility maintained
-- [x] Performance acceptable (<10s for large datasets)
-- [x] 100% test pass rate
-
----
-
-## Recommendations
-
-### Immediate
-
-- ✅ Ready for staging environment validation
-- ✅ Ready for production deployment
-
-### Next Steps
-
-1. Test with actual HuggingFace API (not mocked)
-2. Validate pack loading in retrieval system
-3. Benchmark hybrid scoring with new documents
-4. Monitor first production ingestion
-
-### Long-term
-
-1. Add integration tests with real HuggingFace datasets
-2. Performance benchmarking with different dataset sizes
-3. Memory profiling for large arXiv ingestion
-4. Document update frequency strategy
-
----
-
-## Sign-Off
-
-**All 71 tests passing.**  
-**Backward compatibility maintained.**  
-**New functionality validated.**
-
-✅ **Ready for Production Deployment**
-
----
-
-**Test Report Generated**: 2025-11-08  
-**Python Version**: 3.12.10  
-**pytest Version**: 8.4.2  
-**Status**: VALIDATED ✅

TODO.md

@@ -1,30 +0,0 @@
-# Background Pack Ingestion Implementation
-
-## Overview
-Modify app.py to perform pack ingestion in a background thread, allowing the app to start immediately while documents load asynchronously.
-
-## Tasks
-
-### 1. Add Background Ingestion Support
-- [ ] Import threading module in app.py
-- [ ] Add global variables to track ingestion status (running, progress, total_docs, processed, etc.)
-- [ ] Create a background_ingest_packs() function that performs the ingestion logic
-- [ ] Start the background thread after API initialization but before app launch
-
-### 2. Update System Stats
-- [ ] Modify get_system_stats() to include ingestion progress information
-- [ ] Display current ingestion status in the System Stats tab
-
-### 3. Handle Thread Safety
-- [ ] Ensure API.add_document() calls are thread-safe (assuming they are)
-- [ ] Add proper error handling in the background thread
-
-### 4. Test Implementation
-- [ ] Test that app launches immediately
-- [ ] Verify ingestion happens in background
-- [ ] Check that queries work during ingestion
-- [ ] Confirm progress is shown in System Stats
-
-## Status
-- [x] Plan created and approved
-- [ ] Implementation in progress

VALIDATION_REPORT_MIT_DATASETS.md

@@ -1,353 +0,0 @@
-# Validation Report: MIT-Licensed Datasets Integration
-
-**Date**: November 8, 2025 (Updated)  
-**Branch**: e7cff201eabf06f7c2950bc7545723d20997e73d  
-**Status**: ✅ COMPLETE - All 7 New MIT-Licensed Datasets Implemented + Updates
-
----
-
-## Executive Summary
-
-Successfully integrated 7 new MIT-licensed HuggingFace datasets into the warbler-cda-package following Test-Driven Development (TDD) methodology. All transformers are implemented, tested, and ready for production use.
-
-**Recent Updates**:
-- Replaced AST-FRI/EnterpriseBench with SustcZhangYX/ChatEnv (software development chat)
-- Added MU-NLPC/Edustories-en (educational stories in English)
-- Enhanced PDF extraction for GOAT-AI/generated-novels dataset
-
----
-
-## New Datasets Added
-
-| Dataset | Transformer | Size | Features |
-|---------|-------------|------|----------|
-| **arXiv Papers** | `transform_arxiv()` | 2.55M papers | Limit parameter, scholarly metadata |
-| **Prompt Report** | `transform_prompt_report()` | 83 docs | Prompt engineering analysis |
-| **Generated Novels** | `transform_novels()` | 20 novels | Auto-chunking, enhanced PDF extraction |
-| **Technical Manuals** | `transform_manuals()` | 52 manuals | Section extraction, procedural |
-| **ChatEnv** | `transform_enterprise()` | Software dev chat | Multi-agent coding conversations |
-| **Portuguese Education** | `transform_portuguese_education()` | 21 docs | Multilingual (pt) support |
-| **Edustories** | `transform_edustories()` | 1492 case studies | Educational case studies with structured teaching situations |
-
----
-
-## TDD Process Execution
-
-### Step 1: Context Alignment ✓
-- Commit e7cff201 checked out successfully
-- Project structure analyzed
-- Historical data requirements understood
-- Date/lineage verified
-
-### Step 2: Test First ✓
-**File**: `tests/test_new_mit_datasets.py`
-
-Created comprehensive test suite with 31 test cases covering:
-- **Transformer Existence**: Each transformer method exists and is callable
-- **Output Format Validation**: Documents have required Warbler structure
-  - `content_id` (string)
-  - `content` (text)
-  - `metadata` (with MIT license, source dataset, realm type)
-- **Dataset-Specific Features**:
-  - arXiv: Title, authors, year, categories, limit parameter
-  - Prompt Report: Category, technical discussion realm
-  - Novels: Text chunking, chunk indexing, part tracking
-  - Manuals: Section extraction, procedural realm
-  - Enterprise: Scenario/task labels, business realm
-  - Portuguese: Language tagging, multilingual support
-- **Integration Tests**: Pack creation, document enrichment
-- **Performance Tests**: Large dataset handling (100+ papers in <10s)
-- **Error Handling**: Graceful failure modes
-
-### Step 3: Code Implementation ✓
-**File**: `warbler_cda/utils/hf_warbler_ingest.py`
-
-#### New Transformer Methods (7)
-```python
-def transform_arxiv(limit: Optional[int] = None)          # 2.55M papers, controlled ingestion
-def transform_prompt_report()                             # 83 documentation entries
-def transform_novels()                                    # 20 long-form narratives (enhanced PDF)
-def transform_manuals()                                   # 52 technical procedures
-def transform_enterprise()                                # ChatEnv software dev chat (UPDATED)
-def transform_portuguese_education()                      # 21 multilingual texts
-def transform_edustories()                                # Educational stories in English (NEW)
-```
-
-#### New Helper Methods (8)
-```python
-def _create_arxiv_content(item)                          # Academic paper formatting
-def _create_prompt_report_content(item)                  # Technical documentation
-def _create_novel_content(title, chunk, idx, total)      # Narrative chunking
-def _create_manual_content(item)                         # Manual section formatting
-def _create_enterprise_content(item)                     # ChatEnv dev chat formatting (UPDATED)
-def _create_portuguese_content(item)                     # Portuguese text formatting
-def _create_edustories_content(story_text, title, idx)   # Educational story formatting (NEW)
-def _chunk_text(text, chunk_size=1000)                   # Text splitting utility
-```
-
-#### Enhanced Methods
-```python
-def _extract_pdf_text(pdf_data, max_pages=100)           # Enhanced PDF extraction with better logging
-```
-
-### Step 4: Best Practices ✓
-
-#### Code Quality
-- **Type Hints**: All methods fully typed (Dict, List, Any, Optional)
-- **Docstrings**: Each method has descriptive docstrings
-- **Error Handling**: Try-catch blocks in CLI with user-friendly messages
-- **Logging**: Info-level logging for pipeline visibility
-- **Metadata**: All docs include MIT license, realm types, lifecycle stages
-
-#### Dataset-Specific Optimizations
-- **arXiv**: Limit parameter prevents memory exhaustion with 2.55M papers
-- **Novels**: Automatic chunking (1000 words/chunk) for token limits
-- **All**: Graceful handling of missing fields with `.get()` defaults
-
-#### Warbler Integration
-All transformers produce documents with:
-```json
-{
-  "content_id": "source-type/unique-id",
-  "content": "formatted text for embedding",
-  "metadata": {
-    "pack": "warbler-pack-<dataset>",
-    "source_dataset": "huggingface/path",
-    "license": "MIT",
-    "realm_type": "category",
-    "realm_label": "subcategory",
-    "lifecycle_stage": "emergence",
-    "activity_level": 0.5-0.8,
-    "dialogue_type": "content_type",
-    "dataset_specific_fields": "..."
-  }
-}
-```
-
-### Step 5: Validation ✓
-
-#### Code Structure Verification
-- ✓ All 6 transformers implemented (lines 149-407)
-- ✓ All 7 helper methods present (lines 439-518)
-- ✓ File size increased from 290 → 672 lines
-- ✓ Proper indentation and syntax
-- ✓ All imports present (Optional, List, Dict, Any)
-
-#### CLI Integration
-- ✓ New dataset options in `--datasets` choice list
-- ✓ `--arxiv-limit` parameter for controlling large datasets
-- ✓ Updated `list_available()` with new datasets
-- ✓ Error handling for invalid datasets
-- ✓ Report generation for ingestion results
-
-#### Backward Compatibility
-- ✓ Legacy datasets still supported (npc-dialogue removed, multi-character/system-chat kept)
-- ✓ Existing pack creation unchanged
-- ✓ Existing metadata format preserved
-- ✓ All new datasets use MIT license explicitly
-
----
-
-## Usage Examples
-
-### Ingest Single Dataset
-```bash
-python -m warbler_cda.utils.hf_warbler_ingest ingest -d arxiv --arxiv-limit 1000
-```
-
-### Ingest Multiple Datasets
-```bash
-python -m warbler_cda.utils.hf_warbler_ingest ingest -d arxiv -d prompt-report -d novels
-```
-
-### Ingest All MIT-Licensed Datasets
-```bash
-python -m warbler_cda.utils.hf_warbler_ingest ingest -d all --arxiv-limit 50000
-```
-
-### List Available Datasets
-```bash
-python -m warbler_cda.utils.hf_warbler_ingest list-available
-```
-
----
-
-## Integration with Retrieval API
-
-### Warbler-CDA Package Features
-All ingested documents automatically receive:
-
-1. **FractalStat Coordinates** (via `retrieval_api.py`)
-   - Lineage, Adjacency, Luminosity, Polarity, Dimensionality
-   - Horizon and Realm assignments
-   - Automatic computation from embeddings
-
-2. **Semantic Embeddings** (via `embeddings.py`)
-   - Sentence Transformer models
-   - Cached for performance
-   - Full-text indexing
-
-3. **Pack Loading** (via `pack_loader.py`)
-   - Automatic JSONL parsing
-   - Metadata enrichment
-   - Multi-pack support
-
-4. **Retrieval Enhancement**
-   - Hybrid scoring (semantic + FractalStat)
-   - Context assembly
-   - Conflict detection & resolution
-
----
-
-## Data Flow
-
-```
-HuggingFace Dataset
-       ↓
-HFWarblerIngestor.transform_*()
-       ↓
-Warbler Document Format (JSON)
-       ↓
-JSONL Pack Files
-       ↓
-pack_loader.load_warbler_pack()
-       ↓
-RetrievalAPI.add_document()
-       ↓
-Embeddings + FractalStat Coordinates
-       ↓
-Hybrid Retrieval Ready
-```
-
----
-
-## Test Coverage
-
-| Category | Tests | Status |
-|----------|-------|--------|
-| Transformer Existence | 7 | ✓ |
-| Output Format | 7 | ✓ |
-| Metadata Fields | 7 | ✓ |
-| Dataset-Specific | 14 | ✓ |
-| Integration | 1 | ✓ |
-| Performance | 1 | ✓ |
-| **Total** | **37** | **✓** |
-
----
-
-## Performance Characteristics
-
-- **arXiv (with limit=100)**: <10s transformation
-- **Prompt Report (83 docs)**: <5s
-- **Novels (20 + chunking + PDF)**: 100-500 chunks, <15s (with PDF extraction)
-- **Manuals (52 docs)**: <5s
-- **ChatEnv (software dev chat)**: <5s
-- **Portuguese (21 docs)**: <5s
-- **Edustories**: <5s
-
-Memory Usage: Linear with dataset size, manageable with limit parameters.
-
----
-
-## License Compliance
-
-✅ **All datasets are MIT-licensed:**
-- `nick007x/arxiv-papers` - MIT
-- `PromptSystematicReview/ThePromptReport` - MIT
-- `GOAT-AI/generated-novels` - MIT
-- `nlasso/anac-manuals-23` - MIT
-- `SustcZhangYX/ChatEnv` - MIT (UPDATED - replaced EnterpriseBench)
-- `Solshine/Portuguese_Language_Education_Texts` - MIT
-- `MU-NLPC/Edustories-en` - MIT (NEW)
-
-❌ **Removed (as per commit requirements):**
-- `amaydle/npc-dialogue` - UNLICENSED/COPYRIGHTED
-- `AST-FRI/EnterpriseBench` - REPLACED (had loading issues)
-
----
-
-## File Changes
-
-### Modified
-- `warbler_cda/utils/hf_warbler_ingest.py` (290 → ~750 lines)
-  - Added 7 transformers (including edustories)
-  - Added 8 helpers
-  - Enhanced PDF extraction method
-  - Updated transform_enterprise() to use ChatEnv
-  - Updated CLI (ingest command)
-  - Updated CLI (list_available command)
-
-### Created
-- `tests/test_new_mit_datasets.py` (37 test cases)
-  - Updated TestEnterpriseTransformer for ChatEnv
-  - Added TestEdustoriesTransformer
-- `validate_new_transformers.py` (standalone validation)
-- `VALIDATION_REPORT_MIT_DATASETS.md` (this file)
-- `IMPLEMENTATION_SUMMARY_MIT_DATASETS.md` (updated)
-
----
-
-## Next Steps
-
-### Immediate
-1. Run full test suite: `pytest tests/test_new_mit_datasets.py -v`
-2. Verify in staging environment
-3. Create merge request for production
-
-### Integration
-1. Test with live HuggingFace API calls
-2. Validate pack loading in retrieval system
-3. Benchmark hybrid scoring performance
-4. Test with actual FractalStat coordinate computation
-
-### Operations
-1. Set up arXiv ingestion job with `--arxiv-limit 50000`
-2. Create scheduled tasks for dataset updates
-3. Monitor pack creation reports
-4. Track ingestion performance metrics
-
----
-
-## Conclusion
-
-**The scroll is complete; tested, proven, and woven into the lineage.**
-
-All 7 new MIT-licensed datasets have been successfully integrated into warbler-cda-package with:
-- ✅ Complete transformer implementations (7 transformers)
-- ✅ Comprehensive test coverage (37 tests)
-- ✅ Production-ready error handling
-- ✅ Full documentation
-- ✅ Backward compatibility maintained
-- ✅ License compliance verified
-- ✅ Enterprise dataset updated to ChatEnv (software development focus)
-- ✅ Edustories dataset added (educational stories support)
-- ✅ Enhanced PDF extraction for novels (better logging and error handling)
-
-The system is ready for staging validation and production deployment.
-
-### Recent Changes Summary
-1. **Enterprise Dataset**: Replaced AST-FRI/EnterpriseBench with SustcZhangYX/ChatEnv
-   - Focus shifted from business benchmarks to software development chat
-   - Better alignment with collaborative coding scenarios
-   - Improved conversation extraction logic
-
-2. **Edustories**: Added MU-NLPC/Edustories-en
-   - Educational case studies from student teachers (1492 entries)
-   - Structured format: description (background), anamnesis (situation), solution (intervention), outcome
-   - Student metadata: age/school year, hobbies, diagnoses, disorders
-   - Teacher metadata: approbation (subject areas), practice years
-   - Annotation fields: problems, solutions, and implications (both confirmed and possible)
-   - Teaching case study content for educational NPC training
-
-3. **Novels Enhancement**: Improved PDF extraction
-   - Enhanced logging for debugging
-   - Better error handling and recovery
-   - Support for multiple PDF field formats
-   - Note: Dataset lacks README, requires complete PDF-to-text conversion
-
----
-
-**Signed**: Zencoder AI Assistant  
-**Date**: 2025-11-08  
-**Branch**: e7cff201eabf06f7c2950bc7545723d20997e73d  
-**Status**: ✅ VALIDATED & READY

WARBLER_CDA_PERFORMANCE_REPORT.md

@@ -1,125 +0,0 @@
-# Warbler CDA Performance Report
-
-## Executive Summary
-
-This report presents initial performance results for the Warbler CDA (Cognitive Development Architecture) system's semantic retrieval capabilities. Testing was conducted on a local deployment with approximately 10,000+ documents across multiple domains including academic papers (arXiv), educational content, fiction, and dialogue templates.
-
-## Methodology
-
-### Dataset
-- **Source**: Warbler pack collection (HuggingFace datasets, arXiv, educational content, fiction, etc.)
-- **Size**: ~10,000 documents pre-indexed and searchable
-- **Domains**: Academic research, educational materials, fiction, technical documentation, dialogue templates
-- **Indexing**: Automated semantic indexing using sentence transformers and custom embeddings
-
-### Test Queries
-Four queries were executed to evaluate semantic relevance, cross-domain matching, and result quality:
-
-1. **Simple query**: "hello world"
-2. **Non-sensical/rare phrase**: "just a big giant pile of goop"
-3. **General topic**: "anything about Saturn's moons"
-4. **Specific scientific query**: "rotation dynamics of Saturn's co-orbital moons Janus and Epimetheus"
-
-### Metrics Evaluated
-- **Semantic Relevance**: Cosine similarity scores (0-1 scale)
-- **Query Performance**: Response time in milliseconds
-- **Result Quality**: Narrative coherence analysis
-- **Bias Detection**: Automated validation via "Bob the Skeptic" system
-- **Cross-Domain Matching**: Ability to find relevant results across different content types
-
-## Results
-
-### Query Performance Summary
-
-| Query Type | Avg Response Time | Avg Relevance Score | Bob Status | Narrative Coherence |
-|------------|-------------------|---------------------|------------|-------------------|
-| Simple phrase | 9,523ms | 1.0 (perfect match) | QUARANTINED* | 89.9% |
-| Nonsensical | 23,611ms | 0.88 | PASSED | 83.6% |
-| General topic | 14,040ms | 0.74 | PASSED | 75.5% |
-| Specific science | 28,266ms | 0.87 | PASSED | 83.2% |
-
-*Bob quarantined results deemed "suspiciously perfect" (>85% coherence score with low fractal resonance)
-
-### Detailed Query Analysis
-
-#### Query 1: "hello world"
-- **Performance**: Fastest query (9.5s), perfect relevance scores (1.0)
-- **Results**: Returned arXiv papers on gravitational wave astronomy and multi-messenger astronomy
-- **Validation**: Bob flagged results as potentially overly perfect (coherence: 89.9%, resonance: 0.0)
-- **Note**: While semantically relevant, the system correctly identified potential dataset bias or overfitting
-
-#### Query 2: "just a big giant pile of goop"
-- **Performance**: Longest query (23.6s) due to expansive semantic search
-- **Results**: Cross-domain matches including astronomical research, Portuguese educational content, and software development papers
-- **Relevance**: High semantic similarity (0.93) despite query nonsensicality
-- **Coherence**: Strong narrative threading across diverse content areas (83.6%)
-
-#### Query 3: "anything about Saturn's moons"
-- **Performance**: Medium response time (14s)
-- **Results**: Returned relevant astronomical papers including exomoon research and planetary science
-- **Relevance**: Solid semantic matching (0.74 average) with domain-appropriate results
-- **Coherence**: Single narrative thread (Saturn/planetary research) with high focus (87%)
-
-#### Query 4: "rotation dynamics of Saturn's co-orbital moons Janus and Epimetheus"
-- **Performance**: Longest individual query (28.3s), highest computational load
-- **Results**: Found exact target paper: *"The Rotation of Janus and Epimetheus"* by Tiscareno et al.
-- **Relevance**: Highest semantic match (0.94) with precise subject alignment
-- **Coherence**: Excellent threading of planetary dynamics research (83.2%)
-
-## Comparison to Industry Benchmarks
-
-### Performance Comparison
-
-| System | Query Time (avg) | Relevance Score (avg) | Features |
-|--------|-----------------|----------------------|----------|
-| Warbler CDA | 19.1s | 0.88 | Semantic + FractalStat hybrid, coherence analysis |
-| Retrieval-Augmented Generation (RAG) | 10-30s | 0.85-0.95 | Semantic retrieval only |
-| Semantic Search APIs | 3-15s | 0.70-0.90 | Basic vector search |
-| Traditional Search Engines | <1s | Variable | Keyword matching |
-
-### Key Advantages
-
-1. **Advanced Validation**: Built-in bias detection prevents "hallucinated" or overly curated results
-2. **Narrative Coherence**: Analyzes result consistency and threading, not just individual scores
-3. **Cross-Domain Retrieval**: Successfully finds relevant content across disparate domains
-4. **FractalStat Integration**: Experimental dimensionality enhancement for retrieval
-5. **Real-Time Analysis**: Provides narrative coherence metrics in every response
-
-### Limitations Identified
-
-1. **Query Complexity Scaling**: Response time increases significantly for highly specific queries (observed 3x increase in Test 4)
-2. **Exact Title Matching**: While semantic matching works well, exact title/phrase queries may not receive perfect scores
-3. **Memory Usage**: Local deployment uses ~500MB base memory with document indexing
-
-## Technical Implementation Notes
-
-### System Architecture
-- **Frontend**: FastAPI with async query processing
-- **Backend**: Custom RetrievalAPI with hybrid semantic/FractalStat scoring
-- **Embeddings**: Sentence transformers with domain-specific fine-tuning
-- **Validation**: Automated result quality checking and narrative analysis
-
-### Deployment Configuration
-- **Local Development**: Direct Python execution or Docker container
-- **Production Ready**: Complete Kubernetes manifests with auto-scaling
-- **Data Loading**: Automatic pack discovery and ingestion on startup
-- **APIs**: RESTful endpoints with OpenAPI/Swagger documentation
-
-## Next Steps
-
-1. **Scale Testing**: Evaluate performance with larger document collections (100k+)
-2. **Query Optimization**: Implement approximate nearest neighbor search for faster retrieval
-3. **Fine-tuning**: Domain-specific embedding adaptation for improved relevance
-4. **A/B Testing**: Comparative analysis against commercial semantic search services
-
-## Conclusion
-
-The Warbler CDA demonstrates solid semantic retrieval capabilities with advanced features including automatic quality validation and narrative coherence analysis. Initial results show competitive performance compared to typical RAG implementations, with additional quality assurance features that prevent result bias.
-
-Query response times are acceptable for research and analytical workloads, with strong semantic relevance scores across varied query types. The system's ability to maintain coherence across cross-domain results represents a significant advancement over basic vector similarity approaches.
-
----
-
-*Report Generated: December 1, 2025*
-*Test Environment: Local development with ~10k document corpus*
-*System Version: Warbler CDA v0.9 (FractalStat Integration)*

app.py

@@ -6,14 +6,13 @@ Provides a web UI for the FractalStat RAG system with GPU acceleration.
 """
 
 import gradio as gr
-import json
-from typing import Dict, Any, List
 import time
 
 # Import Warbler CDA components
 from warbler_cda.retrieval_api import RetrievalAPI, RetrievalQuery, RetrievalMode
 from warbler_cda.embeddings import EmbeddingProviderFactory
 from warbler_cda.fractalstat_rag_bridge import FractalStatRAGBridge
+from warbler_cda.semantic_anchors import SemanticAnchorGraph
 from warbler_cda.pack_loader import PackLoader
 
 # Initialize the system
@@ -23,12 +22,17 @@ print("🚀 Initializing Warbler CDA...")
 embedding_provider = EmbeddingProviderFactory.get_default_provider()
 print(f"✅ Embedding provider: {embedding_provider.get_provider_info()['provider_id']}")
 
+# Create semantic anchors (required by RetrievalAPI)
+semantic_anchors = SemanticAnchorGraph(embedding_provider=embedding_provider)
+print("✅ Semantic anchors initialized")
+
 # Create FractalStat bridge
 fractalstat_bridge = FractalStatRAGBridge()
 print("✅ FractalStat bridge initialized")
 
-# Create RetrievalAPI
+# Create RetrievalAPI with proper components
 api = RetrievalAPI(
+    semantic_anchors=semantic_anchors,
     embedding_provider=embedding_provider,
     fractalstat_bridge=fractalstat_bridge,
     config={"enable_fractalstat_hybrid": True}
@@ -39,15 +43,47 @@ print("✅ RetrievalAPI initialized")
 print("📚 Loading Warbler packs...")
 pack_loader = PackLoader()
 documents = pack_loader.discover_documents()
-print(f"✅ Found {len(documents)} documents")
-
-# Ingest documents
-for doc in documents:
-    api.add_document(
-        doc_id=doc["id"],
-        content=doc["content"],
-        metadata=doc.get("metadata", {})
-    )
+
+# If no packs found, try to download them
+if len(documents) == 0:
+    print("⚠️ No packs found locally. Attempting to download from HuggingFace...")
+    try:
+        from warbler_cda.utils.hf_warbler_ingest import HFWarblerIngestor
+        ingestor = HFWarblerIngestor(packs_dir=pack_loader.packs_dir, verbose=True)
+        # Download a small demo dataset for deployment
+        print("📦 Downloading warbler-pack-hf-prompt-report...")
+        success = ingestor.ingest_dataset("prompt-report")
+        if success:
+            # Reload after download
+            documents = pack_loader.discover_documents()
+            print(f"✅ Downloaded {len(documents)} documents")
+        else:
+            print("❌ Failed to download dataset, using sample documents...")
+            documents = []
+    except Exception as e:
+        print(f"⚠️ Could not download packs: {e}")
+        print("Using sample documents instead...")
+        documents = []
+
+if len(documents) == 0:
+    # Fallback to sample documents
+    sample_docs = [
+        {"id": "sample1", "content": "FractalStat is an 8-dimensional addressing system for intelligent retrieval.", "metadata": {}},
+        {"id": "sample2", "content": "Semantic search finds documents by meaning, not just keywords.", "metadata": {}},
+        {"id": "sample3", "content": "Bob the Skeptic validates results to prevent bias and hallucinations.", "metadata": {}},
+    ]
+    for doc in sample_docs:
+        api.add_document(doc["id"], doc["content"], doc["metadata"])
+    print(f"✅ Loaded {len(sample_docs)} sample documents")
+else:
+    print(f"✅ Found {len(documents)} documents")
+    # Ingest documents
+    for doc in documents:
+        api.add_document(
+            doc_id=doc["id"],
+            content=doc["content"],
+            metadata=doc.get("metadata", {})
+        )
 
 print(f"🎉 Warbler CDA ready with {api.get_context_store_size()} documents!")
 
@@ -145,7 +181,7 @@ with gr.Blocks(title="Warbler CDA - FractalStat RAG") as demo:
             with gr.Column():
                 results_output = gr.Markdown(label="Results")
         
-        query_btn.click(
+        query_btn.click(  # pylint: disable=E1101
             fn=query_warbler,
             inputs=[query_input, max_results, use_hybrid],
             outputs=results_output
@@ -163,8 +199,8 @@ with gr.Blocks(title="Warbler CDA - FractalStat RAG") as demo:
     with gr.Tab("System Stats"):
         stats_output = gr.Markdown()
         stats_btn = gr.Button("Refresh Stats")
-        stats_btn.click(fn=get_system_stats, outputs=stats_output)
-        demo.load(fn=get_system_stats, outputs=stats_output)
+        stats_btn.click(fn=get_system_stats, outputs=stats_output)  # pylint: disable=E1101
+        demo.load(fn=get_system_stats, outputs=stats_output)  # pylint: disable=E1101
     
     with gr.Tab("About"):
         gr.Markdown("""

compress_packs.py

@@ -1,134 +0,0 @@
-#!/usr/bin/env python3
-"""
-Pack Compression Script using Evaporation Engine
-
-This script compresses warbler packs by replacing document content with
-compressed proto-thoughts generated by the evaporation engine.
-"""
-
-import json
-import sys
-from pathlib import Path
-from typing import Dict, Any, List
-
-# Add the project root to Python path
-sys.path.insert(0, str(Path(__file__).parent))
-
-from warbler_cda.melt_layer import MeltLayer, MagmaStore
-from warbler_cda.evaporation import EvaporationEngine, CloudStore
-
-
-def load_jsonl_file(filepath: str) -> List[Dict[str, Any]]:
-    """Load a JSONL file and return list of documents."""
-    documents = []
-    with open(filepath, "r", encoding="utf-8") as f:
-        for line in f:
-            line = line.strip()
-            if line:
-                documents.append(json.loads(line))
-    return documents
-
-
-def save_jsonl_file(filepath: str, documents: List[Dict[str, Any]]) -> None:
-    """Save list of documents to a JSONL file."""
-    with open(filepath, "w", encoding="utf-8") as f:
-        for doc in documents:
-            f.write(json.dumps(doc, ensure_ascii=False) + "\n")
-
-
-def compress_pack(pack_path: str, output_suffix: str = "_compressed") -> None:
-    """Compress a single pack using evaporation engine."""
-    pack_path = Path(pack_path)
-    if not pack_path.exists():
-        raise FileNotFoundError(f"Pack path {pack_path} does not exist")
-
-    # Find all JSONL files in the pack
-    jsonl_files = list(pack_path.glob("*.jsonl"))
-    if not jsonl_files:
-        print(f"No JSONL files found in {pack_path}")
-        return
-
-    print(f"Found {len(jsonl_files)} JSONL files in {pack_path}")
-
-    # Initialize evaporation components
-    magma_store = MagmaStore()
-    cloud_store = CloudStore()
-    melt_layer = MeltLayer(magma_store)
-    evaporation_engine = EvaporationEngine(magma_store, cloud_store)
-
-    total_docs = 0
-    compressed_docs = 0
-
-    for jsonl_file in jsonl_files:
-        print(f"Processing {jsonl_file.name}...")
-
-        # Load documents
-        documents = load_jsonl_file(str(jsonl_file))
-        total_docs += len(documents)
-
-        compressed_documents = []
-
-        for doc in documents:
-            if "content" not in doc:
-                print("Warning: Document missing 'content' field, skipping")
-                continue
-
-            content = doc["content"]
-            if not content or not isinstance(content, str):
-                print("Warning: Empty or invalid content, skipping")
-                continue
-
-            try:
-                # Create a fragment from the document content
-                fragment = {"id": doc.get("content_id", f"doc_{compressed_docs}"), "text": content}
-
-                # Create glyph from the single fragment
-                melt_layer.retire_cluster({"fragments": [fragment]})
-
-                # Evaporate to get proto-thought
-                mist_lines = evaporation_engine.evaporate(limit=1)
-
-                if mist_lines:
-                    proto_thought = mist_lines[0]["proto_thought"]
-                    # Replace content with compressed proto-thought
-                    compressed_doc = doc.copy()
-                    compressed_doc["content"] = proto_thought
-                    compressed_doc["original_content_length"] = len(content)
-                    compressed_doc["compressed_content_length"] = len(proto_thought)
-                    compressed_documents.append(compressed_doc)
-                    compressed_docs += 1
-                else:
-                    print(
-                        f"Warning: Failed to evaporate glyph for document {doc.get('content_id', 'unknown')}"
-                    )
-                    # Keep original document if evaporation fails
-                    compressed_documents.append(doc)
-
-            except Exception as e:
-                print(f"Error processing document {doc.get('content_id', 'unknown')}: {e}")
-                # Keep original document on error
-                compressed_documents.append(doc)
-
-        # Save compressed file
-        output_file = jsonl_file.parent / f"{jsonl_file.stem}{output_suffix}{jsonl_file.suffix}"
-        save_jsonl_file(str(output_file), compressed_documents)
-        print(f"Saved compressed file: {output_file}")
-
-    print("Compression complete:")
-    print(f"  Total documents processed: {total_docs}")
-    print(f"  Documents compressed: {compressed_docs}")
-    if total_docs > 0:
-        print(f"  Compression ratio: {compressed_docs/total_docs:.2%}")
-
-
-def main():
-    if len(sys.argv) != 2:
-        print("Usage: python compress_packs.py <pack_path>")
-        sys.exit(1)
-
-    pack_path = sys.argv[1]
-    compress_pack(pack_path)
-
-
-if __name__ == "__main__":
-    main()

convert_to_jsonl.py

@@ -1,37 +0,0 @@
-import json
-import os
-
-
-def convert_templates_to_jsonl(pack_dir):
-    """Convert templates.json to pack_name.jsonl for a given pack directory."""
-    pack_name = os.path.basename(pack_dir)
-    templates_path = os.path.join(pack_dir, "pack", "templates.json")
-    jsonl_path = os.path.join(pack_dir, f"{pack_name}.jsonl")
-
-    if not os.path.exists(templates_path):
-        print(f"No templates.json found in {pack_dir}")
-        return
-
-    with open(templates_path, "r") as f:
-        templates = json.load(f)
-
-    with open(jsonl_path, "w") as f:
-        for template in templates:
-            json.dump(template, f)
-            f.write("\n")
-
-    print(f"Converted {templates_path} to {jsonl_path}")
-
-
-# Convert the three default packs
-packs_to_convert = [
-    "packs/warbler-pack-core",
-    "packs/warbler-pack-faction-politics",
-    "packs/warbler-pack-wisdom-scrolls",
-]
-
-for pack in packs_to_convert:
-    if os.path.exists(pack):
-        convert_templates_to_jsonl(pack)
-    else:
-        print(f"Pack directory {pack} not found")

copy_packs.sh

@@ -1,45 +0,0 @@
-#!/bin/bash
-set -e
-
-SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
-SOURCE_PACKS_DIR="$REPO_ROOT/packages/com.twg.the-seed/The Living Dev Agent/packs"
-DEST_PACKS_DIR="$SCRIPT_DIR/packs"
-
-echo "Copying Warbler Packs to warbler-cda-package..."
-echo "Source: $SOURCE_PACKS_DIR"
-echo "Destination: $DEST_PACKS_DIR"
-
-if [ ! -d "$SOURCE_PACKS_DIR" ]; then
-    echo "❌ Error: Source packs directory not found at $SOURCE_PACKS_DIR"
-    exit 1
-fi
-
-mkdir -p "$DEST_PACKS_DIR"
-
-PACKS=(
-    "warbler-pack-core"
-    "warbler-pack-faction-politics"
-    "warbler-pack-wisdom-scrolls"
-    "warbler-pack-hf-npc-dialogue"
-)
-
-for pack in "${PACKS[@]}"; do
-    src="$SOURCE_PACKS_DIR/$pack"
-    dst="$DEST_PACKS_DIR/$pack"
-    
-    if [ -d "$src" ]; then
-        echo "📦 Copying $pack..."
-        rm -rf "$dst"
-        cp -r "$src" "$dst"
-        echo "✓ Copied $pack"
-    else
-        echo "⚠️  Warning: Pack not found at $src (skipping)"
-    fi
-done
-
-echo ""
-echo "✅ Warbler packs successfully copied to $DEST_PACKS_DIR"
-echo ""
-echo "Packs available for ingestion:"
-ls -1 "$DEST_PACKS_DIR" | sed 's/^/  • /'

final_fix.py

@@ -1,28 +0,0 @@
-#!/usr/bin/env python3
-"""Final fixes for stat7_entity.py and verify the fixes work"""
-
-# Fix the stat7_entity.py bug
-with open("warbler_cda/stat7_entity.py", "r", encoding="utf-8") as f:
-    content = f.read()
-
-# Fix the description reference bug
-content = content.replace('"description": description,', '"description": self.description,')
-
-# Write back the fixed content
-with open("warbler_cda/stat7_entity.py", "w", encoding="utf-8") as f:
-    f.write(content)
-
-print("Fixed stat7_entity.py description bug")
-
-# Test import to make sure everything works
-try:
-    print("✅ stat7_entity imports successfully")
-except Exception as e:
-    print(f"❌ stat7_entity import failed: {e}")
-
-try:
-    print("✅ stat7_rag_bridge imports successfully")
-except Exception as e:
-    print(f"❌ stat7_rag_bridge import failed: {e}")
-
-print("All fixes applied!")

fix_theme.py

@@ -1,15 +0,0 @@
-#!/usr/bin/env python3
-"""Fix the theme issue in app.py"""
-
-with open("app.py", "r", encoding="utf-8") as f:
-    content = f.read()
-
-old_line = 'with gr.Blocks(title="Warbler CDA - RAG System Demo", theme=gr.themes.Soft()) as demo:'
-new_line = 'with gr.Blocks(title="Warbler CDA - RAG System Demo") as demo:'
-
-content = content.replace(old_line, new_line)
-
-with open("app.py", "w", encoding="utf-8") as f:
-    f.write(content)
-
-print("Fixed theme issue")

k8s/README.md

@@ -1,132 +0,0 @@
-# Kubernetes Deployment for Warbler CDA
-
-This directory contains Kubernetes manifests to deploy Warbler CDA on a Kubernetes cluster.
-
-## Prerequisites
-
-- Kubernetes cluster (kubectl configured)
-- Docker registry access (if using external registry)
-- NGINX Ingress Controller (for external access)
-
-## Components
-
-- `namespace.yaml`: Creates the `warbler-cda` namespace
-- `configmap.yaml`: Configuration settings (environment variables)
-- `pvc.yaml`: Persistent volume claim for data storage
-- `deployment.yaml`: Application deployment with health checks and resource limits
-- `service.yaml`: Service to expose the application within the cluster
-- `ingress.yaml`: Ingress for external access (requires NGINX Ingress Controller)
-
-## Deployment Instructions
-
-### 1. Build and Push Docker Image
-
-First, build your Docker image and push it to a registry:
-
-```bash
-# Build the image
-docker build -t your-registry/warbler-cda:latest .
-
-# Push to registry
-docker push your-registry/warbler-cda:latest
-```
-
-Update the image reference in `deployment.yaml` to point to your registry.
-
-### 2. Deploy to Kubernetes
-
-Apply all manifests:
-
-```bash
-kubectl apply -f k8s/
-```
-
-Or deploy in order:
-
-```bash
-kubectl apply -f namespace.yaml
-kubectl apply -f configmap.yaml
-kubectl apply -f pvc.yaml
-kubectl apply -f deployment.yaml
-kubectl apply -f service.yaml
-kubectl apply -f ingress.yaml
-```
-
-### 3. Check Deployment Status
-
-```bash
-# Check pod status
-kubectl get pods -n warbler-cda
-
-# Check service
-kubectl get svc -n warbler-cda
-
-# Check ingress
-kubectl get ingress -n warbler-cda
-
-# View logs
-kubectl logs -f deployment/warbler-cda -n warbler-cda
-```
-
-### 4. Access the Application
-
-- **Internal cluster access**: `http://warbler-cda-service.warbler-cda.svc.cluster.local`
-- **External access**: Configure DNS to point to your ingress controller IP for `warbler-cda.local`
-
-## Health Checks
-
-The deployment includes:
-- **Liveness Probe**: `/health` endpoint (restarts pod if unhealthy)
-- **Readiness Probe**: `/health` endpoint (removes pod from service if unhealthy)
-
-## Scaling
-
-To scale the deployment:
-
-```bash
-kubectl scale deployment warbler-cda --replicas=3 -n warbler-cda
-```
-
-## Configuration
-
-### Environment Variables
-
-Modify `configmap.yaml` to change:
-- `FRACTALSTAT_TESTING`: Enable/disable testing mode
-- Other environment variables as needed
-
-### Resources
-
-Adjust CPU/memory requests and limits in `deployment.yaml` based on your cluster resources.
-
-### Storage
-
-The PVC requests 10Gi by default. Adjust in `pvc.yaml` if needed.
-
-## Troubleshooting
-
-### Common Issues
-
-1. **Pod won't start**: Check image name/tag and registry access
-2. **No external access**: Ensure Ingress Controller is installed and configured
-3. **Health checks failing**: Verify the `/health` endpoint is responding
-
-### Debug Commands
-
-```bash
-# Describe pod for detailed status
-kubectl describe pod -n warbler-cda
-
-# Check events
-kubectl get events -n warbler-cda
-
-# Port-forward for local testing
-kubectl port-forward svc/warbler-cda-service 8000:80 -n warbler-cda
-```
-
-## Notes
-
-- The deployment uses a persistent volume for data persistence
-- Health checks are configured for the FastAPI `/health` endpoint
-- Resource limits are set for a basic deployment - adjust for your needs
-- The Ingress uses `warbler-cda.local` as default host - change for production

k8s/docker-desktop-k8s-setup.md

@@ -1,139 +0,0 @@
-# Docker Desktop + Kubernetes Setup for Warbler CDA
-
-Since you're using Docker, you can test the Kubernetes deployment locally using Docker Desktop's built-in Kubernetes feature.
-
-## Prerequisites
-
-1. **Enable Kubernetes in Docker Desktop:**
-   - Open Docker Desktop
-   - Go to Settings → Kubernetes
-   - Check "Enable Kubernetes"
-   - Apply & Restart
-
-2. **Verify Kubernetes is running:**
-   ```bash
-   kubectl cluster-info
-   kubectl get nodes
-   ```
-
-## Quick Start with Docker Desktop K8s
-
-### Option 1: Use the deployment script
-
-```bash
-cd k8s
-./deploy.sh
-```
-
-### Option 2: Manual deployment
-
-1. **Build and load image directly to Docker Desktop:**
-```bash
-# Build the image
-docker build -t warbler-cda:latest .
-
-# The image is now available to K8s since Docker Desktop shares images
-```
-
-2. **Deploy to local Kubernetes:**
-```bash
-cd k8s
-kubectl apply -f .
-```
-
-3. **Check deployment:**
-```bash
-kubectl get pods -n warbler-cda
-kubectl get svc -n warbler-cda
-kubectl get ingress -n warbler-cda
-```
-
-4. **Access the application:**
-
-**Option A: Use port-forwarding (recommended for development)**
-```bash
-kubectl port-forward svc/warbler-cda-service 8001:80 -n warbler-cda
-```
-Then visit: http://localhost:8001/health
-
-**Option B: Access via Ingress (requires ingress controller)**
-
-First, enable ingress in Docker Desktop and install NGINX Ingress:
-```bash
-kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.8.1/deploy/static/provider/cloud/deploy.yaml
-```
-
-Then update your ingress.yaml to use a local domain or use port forwarding.
-
-## Compare: Docker Compose vs Kubernetes
-
-| Feature | Docker Compose | Kubernetes |
-|---------|---------------|------------|
-| Scaling | Manual replica adjustment | Auto-scaling, rolling updates |
-| Networking | Simple service discovery | Complex service mesh |
-| Storage | Local volumes | Persistent volumes, storage classes |
-| Health Checks | Basic | Liveness/readiness probes |
-| Resource Limits | Basic | Detailed QoS, limits/requests |
-| Environment | Single host | Multi-node clusters |
-
-## Local Development Workflow
-
-1. **Develop with Docker Compose** (faster iteration):
-   ```bash
-   docker-compose up --build
-   ```
-
-2. **Test production deployment with Kubernetes:**
-   ```bash
-   cd k8s && ./deploy.sh
-   kubectl port-forward svc/warbler-cda-service 8001:80 -n warbler-cda
-   ```
-
-3. **Debug if needed:**
-   ```bash
-   kubectl logs -f deployment/warbler-cda -n warbler-cda
-   kubectl describe pod -n warbler-cda
-   ```
-
-## Benefits of Docker Desktop Kubernetes
-
-- **Same deployment as production** - test your exact K8s manifests
-- **Resource isolation** - proper containerization like production
-- **Networking simulation** - test service communication
-- **Storage testing** - validate PVC behavior
-- **Health check validation** - ensure probes work correctly
-
-## Troubleshooting Docker Desktop K8s
-
-**Common issues:**
-
-1. **"ImagePullBackOff" error:**
-   - Make sure you built the image: `docker build -t warbler-cda:latest .`
-   - Update deployment.yaml image to `warbler-cda:latest`
-
-2. **PVC pending:**
-   - Docker Desktop K8s has storage classes, but storage might not provision immediately
-   - Check: `kubectl get pvc -n warbler-cda`
-   - You can use hostPath storage for local testing
-
-3. **Ingress not working:**
-   - Install ingress controller first
-   - Use port-forwarding for simpler local access
-
-4. **Resource constraints:**
-   - Docker Desktop K8s shares resources with Docker
-   - Reduce resource requests in deployment.yaml if needed
-
-## Converting Docker Compose to Kubernetes
-
-Your `docker-compose.yml` has been converted to K8s with these mappings:
-
-| Docker Compose | Kubernetes Equivalent |
-|---------------|----------------------|
-| `image: .` | `deployment.yaml` with image build step |
-| `ports: - "8001:8000"` | `service.yaml` + `ingress.yaml` |
-| `environment:` | `configmap.yaml` + envFrom |
-| `volumes: ./data:/app/data` | `pvc.yaml` + volumeMounts |
-| `restart: unless-stopped` | Deployment with replicas |
-
-The Kubernetes setup provides production-grade features while maintaining the same application behavior as your Docker Compose setup.

load_warbler_packs_current.txt

@@ -1,259 +0,0 @@
-#!/usr/bin/env python3
-"""
-Load Warbler Pack Data into EXP-09 API Service
-
-Ingests game wisdom, lore, and faction data into the STAT7-enabled RetrievalAPI
-for end-to-end testing with real Warbler content.
-"""
-
-import json
-import requests
-import click
-from pathlib import Path
-from typing import List, Dict, Any
-import logging
-
-logging.basicConfig(level=logging.INFO)
-logger = logging.getLogger(__name__)
-
-# Warbler pack locations
-BASE_DIR = Path(__file__).resolve().parent
-PACKS_DIR = BASE_DIR.parents[1] / 'packs'
-WARBLER_PACKS = [
-    "warbler-pack-core",
-    "warbler-pack-wisdom-scrolls",
-    "warbler-pack-faction-politics",
-    "warbler-pack-hf-arxiv",
-    "warbler-pack-hf-prompt-report",
-    "warbler-pack-hf-novels",
-    "warbler-pack-hf-manuals",
-    "warbler-pack-hf-enterprise",
-    "warbler-pack-hf-portuguese-edu",
-    "warbler-pack-hf-edustories"
-]
-
-
-class WarblerPackLoader:
-    """Load Warbler pack data into the API"""
-
-    def __init__(self, api_url: str = "http://localhost:8000"):
-        self.api_url = api_url.rstrip("/")
-        self.session = requests.Session()
-        self.loaded_count = 0
-        self.error_count = 0
-
-    def discover_documents(self, pack_name: str) -> List[Dict[str, Any]]:
-        """Discover all documents in a pack"""
-        pack_path = PACKS_DIR / pack_name
-        documents = []
-
-        if not pack_path.exists():
-            logger.warning(f"Pack not found: {pack_path}")
-            return []
-
-        # Look for JSON, YAML, markdown, and JSONL files
-        for pattern in [
-            "**/*.json",
-            "**/*.yaml",
-            "**/*.yml",
-            "**/*.md",
-                "**/*.jsonl"]:
-            for file_path in pack_path.glob(pattern):
-                try:
-                    doc = self._parse_document(file_path, pack_name)
-                    if doc:
-                        documents.append(doc)
-                        logger.info(
-                            f"Discovered: {file_path.relative_to(PACKS_DIR)}")
-                except Exception as e:
-                    logger.error(f"Error parsing {file_path}: {e}")
-
-        return documents
-
-    def _parse_document(self, file_path: Path,
-                        pack_name: str) -> Dict[str, Any]:
-        """Parse a document file"""
-        try:
-            if file_path.suffix in ['.json']:
-                with open(file_path, 'r', encoding='utf-8') as f:
-                    content = json.load(f)
-                    if isinstance(content, dict):
-                        content = json.dumps(content)
-                    else:
-                        content = json.dumps(content)
-            elif file_path.suffix in ['.jsonl']:
-                # JSONL files contain multiple JSON objects, one per line
-                # We'll read the first few lines and combine them
-                with open(file_path, 'r', encoding='utf-8') as f:
-                    lines = f.readlines()[:5]  # First 5 lines
-                    content = '\n'.join(line.strip()
-                                        for line in lines if line.strip())
-            elif file_path.suffix in ['.yaml', '.yml']:
-                import yaml
-                with open(file_path, 'r', encoding='utf-8') as f:
-                    content = yaml.safe_load(f)
-                    content = json.dumps(content)
-            elif file_path.suffix == '.md':
-                with open(file_path, 'r', encoding='utf-8') as f:
-                    content = f.read()
-            else:
-                return None
-
-            # Infer realm from pack name
-            if "wisdom" in pack_name:
-                realm = "wisdom"
-            elif "faction" in pack_name:
-                realm = "faction"
-            else:
-                realm = "narrative"
-
-            return {
-                "content_id": f"{pack_name}/{file_path.stem}",
-                "content": str(content)[:5000],  # Limit content size
-                "metadata": {
-                    "pack": pack_name,
-                    "source_file": str(file_path.name),
-                    "realm_type": realm,
-                    "realm_label": pack_name.replace("warbler-pack-", ""),
-                    "lifecycle_stage": "emergence",
-                    "activity_level": 0.7
-                }
-            }
-        except Exception as e:
-            logger.error(f"Failed to parse {file_path}: {e}")
-            return None
-
-    def ingest_document(self, doc: Dict[str, Any]) -> bool:
-        """Send document to API for ingestion"""
-        try:
-            # For now, we'll store in local context
-            # The API service will need an /ingest endpoint
-            logger.info(f"Ingesting: {doc['content_id']}")
-
-            # Check if API has ingest endpoint
-            response = self.session.post(
-                f"{self.api_url}/ingest",
-                json={"documents": [doc]},
-                timeout=10
-            )
-
-            if response.status_code in [200, 201, 202]:
-                self.loaded_count += 1
-                logger.info(f"[OK] Loaded: {doc['content_id']}")
-                return True
-            else:
-                logger.warning(
-                    f"API returned {response.status_code}: {response.text[:200]}")
-                return False
-        except requests.exceptions.ConnectionError:
-            logger.error("Cannot connect to API. Is the service running?")
-            return False
-        except Exception as e:
-            logger.error(f"Ingestion failed: {e}")
-            self.error_count += 1
-            return False
-
-    def load_all_packs(self) -> int:
-        """Load all Warbler packs"""
-        click.echo("\n" + "=" * 60)
-        click.echo("Loading Warbler Pack Data into EXP-09 API")
-        click.echo("=" * 60 + "\n")
-
-        total_docs = 0
-        for pack_name in WARBLER_PACKS:
-            click.echo(f"\n[PACK] Processing: {pack_name}")
-            click.echo("-" * 40)
-
-            documents = self.discover_documents(pack_name)
-            click.echo(f"Found {len(documents)} documents\n")
-
-            for doc in documents:
-                self.ingest_document(doc)
-                total_docs += 1
-
-        click.echo("\n" + "=" * 60)
-        click.secho(
-            f"[OK] Load Complete: {
-                self.loaded_count} docs ingested",
-            fg="green")
-        if self.error_count > 0:
-            click.secho(f"[ERROR] Errors: {self.error_count}", fg="yellow")
-        click.echo("=" * 60 + "\n")
-
-        return self.loaded_count
-
-
-@click.group()
-def cli():
-    """Warbler Pack Loader for EXP-09"""
-    pass
-
-
-@cli.command()
-@click.option("--api-url",
-              default="http://localhost:8000",
-              help="API service URL")
-def load(api_url):
-    """Load all Warbler packs into the API"""
-    loader = WarblerPackLoader(api_url)
-
-    # First, check if API is running
-    try:
-        response = loader.session.get(f"{api_url}/health", timeout=5)
-        if response.status_code == 200:
-            click.secho("[OK] API service is running", fg="green")
-        else:
-            click.secho(
-                "[ERROR] API service not responding correctly", fg="red")
-            return
-    except Exception as e:
-        click.secho(f"[ERROR] Cannot reach API at {api_url}: {e}", fg="red")
-        click.echo("\nStart the service with: docker-compose up -d")
-        return
-
-    # Load the packs
-    loaded = loader.load_all_packs()
-
-    if loaded > 0:
-        click.echo("\n[NEXT] Next Steps:")
-        click.echo(
-            "  1. Query the data with: python exp09_cli.py query --query-id q1 --semantic \"wisdom about courage\"")
-        click.echo(
-            "  2. Test hybrid scoring: python exp09_cli.py query --query-id q1 --semantic \"...\" --hybrid")
-        click.echo("  3. Check metrics: python exp09_cli.py metrics\n")
-
-
-@cli.command()
-@click.option("--api-url",
-              default="http://localhost:8000",
-              help="API service URL")
-def discover(api_url):
-    """Discover documents in Warbler packs (no loading)"""
-    loader = WarblerPackLoader(api_url)
-
-    click.echo("\n" + "=" * 60)
-    click.echo("Discovering Warbler Pack Documents")
-    click.echo("=" * 60 + "\n")
-
-    total = 0
-    for pack_name in WARBLER_PACKS:
-        click.echo(f"\n[PACK] {pack_name}")
-        click.echo("-" * 40)
-
-        documents = loader.discover_documents(pack_name)
-        total += len(documents)
-
-        for doc in documents:
-            click.echo(f"  - {doc['content_id']}")
-            if "metadata" in doc:
-                click.echo(
-                    f"    Realm: {
-                        doc['metadata'].get(
-                            'realm_type',
-                            'unknown')}")
-
-    click.echo(f"\n[STATS] Total discovered: {total} documents\n")
-
-
-if __name__ == "__main__":
-    cli()

package-lock.json

@@ -1,861 +0,0 @@
-{
-  "name": "warbler-cda",
-  "version": "1.0.0",
-  "lockfileVersion": 3,
-  "requires": true,
-  "packages": {
-    "": {
-      "name": "warbler-cda",
-      "version": "1.0.0",
-      "license": "ISC",
-      "dependencies": {
-        "express": "^5.1.0",
-        "typescript": "^5.9.3"
-      }
-    },
-    "node_modules/accepts": {
-      "version": "2.0.0",
-      "resolved": "https://registry.npmjs.org/accepts/-/accepts-2.0.0.tgz",
-      "integrity": "sha512-5cvg6CtKwfgdmVqY1WIiXKc3Q1bkRqGLi+2W/6ao+6Y7gu/RCwRuAhGEzh5B4KlszSuTLgZYuqFqo5bImjNKng==",
-      "license": "MIT",
-      "dependencies": {
-        "mime-types": "^3.0.0",
-        "negotiator": "^1.0.0"
-      },
-      "engines": {
-        "node": ">= 0.6"
-      }
-    },
-    "node_modules/body-parser": {
-      "version": "2.2.0",
-      "resolved": "https://registry.npmjs.org/body-parser/-/body-parser-2.2.0.tgz",
-      "integrity": "sha512-02qvAaxv8tp7fBa/mw1ga98OGm+eCbqzJOKoRt70sLmfEEi+jyBYVTDGfCL/k06/4EMk/z01gCe7HoCH/f2LTg==",
-      "license": "MIT",
-      "dependencies": {
-        "bytes": "^3.1.2",
-        "content-type": "^1.0.5",
-        "debug": "^4.4.0",
-        "http-errors": "^2.0.0",
-        "iconv-lite": "^0.6.3",
-        "on-finished": "^2.4.1",
-        "qs": "^6.14.0",
-        "raw-body": "^3.0.0",
-        "type-is": "^2.0.0"
-      },
-      "engines": {
-        "node": ">=18"
-      }
-    },
-    "node_modules/bytes": {
-      "version": "3.1.2",
-      "resolved": "https://registry.npmjs.org/bytes/-/bytes-3.1.2.tgz",
-      "integrity": "sha512-/Nf7TyzTx6S3yRJObOAV7956r8cr2+Oj8AC5dt8wSP3BQAoeX58NoHyCU8P8zGkNXStjTSi6fzO6F0pBdcYbEg==",
-      "license": "MIT",
-      "engines": {
-        "node": ">= 0.8"
-      }
-    },
-    "node_modules/call-bind-apply-helpers": {
-      "version": "1.0.2",
-      "resolved": "https://registry.npmjs.org/call-bind-apply-helpers/-/call-bind-apply-helpers-1.0.2.tgz",
-      "integrity": "sha512-Sp1ablJ0ivDkSzjcaJdxEunN5/XvksFJ2sMBFfq6x0ryhQV/2b/KwFe21cMpmHtPOSij8K99/wSfoEuTObmuMQ==",
-      "license": "MIT",
-      "dependencies": {
-        "es-errors": "^1.3.0",
-        "function-bind": "^1.1.2"
-      },
-      "engines": {
-        "node": ">= 0.4"
-      }
-    },
-    "node_modules/call-bound": {
-      "version": "1.0.4",
-      "resolved": "https://registry.npmjs.org/call-bound/-/call-bound-1.0.4.tgz",
-      "integrity": "sha512-+ys997U96po4Kx/ABpBCqhA9EuxJaQWDQg7295H4hBphv3IZg0boBKuwYpt4YXp6MZ5AmZQnU/tyMTlRpaSejg==",
-      "license": "MIT",
-      "dependencies": {
-        "call-bind-apply-helpers": "^1.0.2",
-        "get-intrinsic": "^1.3.0"
-      },
-      "engines": {
-        "node": ">= 0.4"
-      },
-      "funding": {
-        "url": "https://github.com/sponsors/ljharb"
-      }
-    },
-    "node_modules/content-disposition": {
-      "version": "1.0.1",
-      "resolved": "https://registry.npmjs.org/content-disposition/-/content-disposition-1.0.1.tgz",
-      "integrity": "sha512-oIXISMynqSqm241k6kcQ5UwttDILMK4BiurCfGEREw6+X9jkkpEe5T9FZaApyLGGOnFuyMWZpdolTXMtvEJ08Q==",
-      "license": "MIT",
-      "engines": {
-        "node": ">=18"
-      },
-      "funding": {
-        "type": "opencollective",
-        "url": "https://opencollective.com/express"
-      }
-    },
-    "node_modules/content-type": {
-      "version": "1.0.5",
-      "resolved": "https://registry.npmjs.org/content-type/-/content-type-1.0.5.tgz",
-      "integrity": "sha512-nTjqfcBFEipKdXCv4YDQWCfmcLZKm81ldF0pAopTvyrFGVbcR6P/VAAd5G7N+0tTr8QqiU0tFadD6FK4NtJwOA==",
-      "license": "MIT",
-      "engines": {
-        "node": ">= 0.6"
-      }
-    },
-    "node_modules/cookie": {
-      "version": "0.7.2",
-      "resolved": "https://registry.npmjs.org/cookie/-/cookie-0.7.2.tgz",
-      "integrity": "sha512-yki5XnKuf750l50uGTllt6kKILY4nQ1eNIQatoXEByZ5dWgnKqbnqmTrBE5B4N7lrMJKQ2ytWMiTO2o0v6Ew/w==",
-      "license": "MIT",
-      "engines": {
-        "node": ">= 0.6"
-      }
-    },
-    "node_modules/cookie-signature": {
-      "version": "1.2.2",
-      "resolved": "https://registry.npmjs.org/cookie-signature/-/cookie-signature-1.2.2.tgz",
-      "integrity": "sha512-D76uU73ulSXrD1UXF4KE2TMxVVwhsnCgfAyTg9k8P6KGZjlXKrOLe4dJQKI3Bxi5wjesZoFXJWElNWBjPZMbhg==",
-      "license": "MIT",
-      "engines": {
-        "node": ">=6.6.0"
-      }
-    },
-    "node_modules/debug": {
-      "version": "4.4.3",
-      "resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz",
-      "integrity": "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==",
-      "license": "MIT",
-      "dependencies": {
-        "ms": "^2.1.3"
-      },
-      "engines": {
-        "node": ">=6.0"
-      },
-      "peerDependenciesMeta": {
-        "supports-color": {
-          "optional": true
-        }
-      }
-    },
-    "node_modules/depd": {
-      "version": "2.0.0",
-      "resolved": "https://registry.npmjs.org/depd/-/depd-2.0.0.tgz",
-      "integrity": "sha512-g7nH6P6dyDioJogAAGprGpCtVImJhpPk/roCzdb3fIh61/s/nPsfR6onyMwkCAR/OlC3yBC0lESvUoQEAssIrw==",
-      "license": "MIT",
-      "engines": {
-        "node": ">= 0.8"
-      }
-    },
-    "node_modules/dunder-proto": {
-      "version": "1.0.1",
-      "resolved": "https://registry.npmjs.org/dunder-proto/-/dunder-proto-1.0.1.tgz",
-      "integrity": "sha512-KIN/nDJBQRcXw0MLVhZE9iQHmG68qAVIBg9CqmUYjmQIhgij9U5MFvrqkUL5FbtyyzZuOeOt0zdeRe4UY7ct+A==",
-      "license": "MIT",
-      "dependencies": {
-        "call-bind-apply-helpers": "^1.0.1",
-        "es-errors": "^1.3.0",
-        "gopd": "^1.2.0"
-      },
-      "engines": {
-        "node": ">= 0.4"
-      }
-    },
-    "node_modules/ee-first": {
-      "version": "1.1.1",
-      "resolved": "https://registry.npmjs.org/ee-first/-/ee-first-1.1.1.tgz",
-      "integrity": "sha512-WMwm9LhRUo+WUaRN+vRuETqG89IgZphVSNkdFgeb6sS/E4OrDIN7t48CAewSHXc6C8lefD8KKfr5vY61brQlow==",
-      "license": "MIT"
-    },
-    "node_modules/encodeurl": {
-      "version": "2.0.0",
-      "resolved": "https://registry.npmjs.org/encodeurl/-/encodeurl-2.0.0.tgz",
-      "integrity": "sha512-Q0n9HRi4m6JuGIV1eFlmvJB7ZEVxu93IrMyiMsGC0lrMJMWzRgx6WGquyfQgZVb31vhGgXnfmPNNXmxnOkRBrg==",
-      "license": "MIT",
-      "engines": {
-        "node": ">= 0.8"
-      }
-    },
-    "node_modules/es-define-property": {
-      "version": "1.0.1",
-      "resolved": "https://registry.npmjs.org/es-define-property/-/es-define-property-1.0.1.tgz",
-      "integrity": "sha512-e3nRfgfUZ4rNGL232gUgX06QNyyez04KdjFrF+LTRoOXmrOgFKDg4BCdsjW8EnT69eqdYGmRpJwiPVYNrCaW3g==",
-      "license": "MIT",
-      "engines": {
-        "node": ">= 0.4"
-      }
-    },
-    "node_modules/es-errors": {
-      "version": "1.3.0",
-      "resolved": "https://registry.npmjs.org/es-errors/-/es-errors-1.3.0.tgz",
-      "integrity": "sha512-Zf5H2Kxt2xjTvbJvP2ZWLEICxA6j+hAmMzIlypy4xcBg1vKVnx89Wy0GbS+kf5cwCVFFzdCFh2XSCFNULS6csw==",
-      "license": "MIT",
-      "engines": {
-        "node": ">= 0.4"
-      }
-    },
-    "node_modules/es-object-atoms": {
-      "version": "1.1.1",
-      "resolved": "https://registry.npmjs.org/es-object-atoms/-/es-object-atoms-1.1.1.tgz",
-      "integrity": "sha512-FGgH2h8zKNim9ljj7dankFPcICIK9Cp5bm+c2gQSYePhpaG5+esrLODihIorn+Pe6FGJzWhXQotPv73jTaldXA==",
-      "license": "MIT",
-      "dependencies": {
-        "es-errors": "^1.3.0"
-      },
-      "engines": {
-        "node": ">= 0.4"
-      }
-    },
-    "node_modules/escape-html": {
-      "version": "1.0.3",
-      "resolved": "https://registry.npmjs.org/escape-html/-/escape-html-1.0.3.tgz",
-      "integrity": "sha512-NiSupZ4OeuGwr68lGIeym/ksIZMJodUGOSCZ/FSnTxcrekbvqrgdUxlJOMpijaKZVjAJrWrGs/6Jy8OMuyj9ow==",
-      "license": "MIT"
-    },
-    "node_modules/etag": {
-      "version": "1.8.1",
-      "resolved": "https://registry.npmjs.org/etag/-/etag-1.8.1.tgz",
-      "integrity": "sha512-aIL5Fx7mawVa300al2BnEE4iNvo1qETxLrPI/o05L7z6go7fCw1J6EQmbK4FmJ2AS7kgVF/KEZWufBfdClMcPg==",
-      "license": "MIT",
-      "engines": {
-        "node": ">= 0.6"
-      }
-    },
-    "node_modules/express": {
-      "version": "5.1.0",
-      "resolved": "https://registry.npmjs.org/express/-/express-5.1.0.tgz",
-      "integrity": "sha512-DT9ck5YIRU+8GYzzU5kT3eHGA5iL+1Zd0EutOmTE9Dtk+Tvuzd23VBU+ec7HPNSTxXYO55gPV/hq4pSBJDjFpA==",
-      "license": "MIT",
-      "dependencies": {
-        "accepts": "^2.0.0",
-        "body-parser": "^2.2.0",
-        "content-disposition": "^1.0.0",
-        "content-type": "^1.0.5",
-        "cookie": "^0.7.1",
-        "cookie-signature": "^1.2.1",
-        "debug": "^4.4.0",
-        "encodeurl": "^2.0.0",
-        "escape-html": "^1.0.3",
-        "etag": "^1.8.1",
-        "finalhandler": "^2.1.0",
-        "fresh": "^2.0.0",
-        "http-errors": "^2.0.0",
-        "merge-descriptors": "^2.0.0",
-        "mime-types": "^3.0.0",
-        "on-finished": "^2.4.1",
-        "once": "^1.4.0",
-        "parseurl": "^1.3.3",
-        "proxy-addr": "^2.0.7",
-        "qs": "^6.14.0",
-        "range-parser": "^1.2.1",
-        "router": "^2.2.0",
-        "send": "^1.1.0",
-        "serve-static": "^2.2.0",
-        "statuses": "^2.0.1",
-        "type-is": "^2.0.1",
-        "vary": "^1.1.2"
-      },
-      "engines": {
-        "node": ">= 18"
-      },
-      "funding": {
-        "type": "opencollective",
-        "url": "https://opencollective.com/express"
-      }
-    },
-    "node_modules/finalhandler": {
-      "version": "2.1.0",
-      "resolved": "https://registry.npmjs.org/finalhandler/-/finalhandler-2.1.0.tgz",
-      "integrity": "sha512-/t88Ty3d5JWQbWYgaOGCCYfXRwV1+be02WqYYlL6h0lEiUAMPM8o8qKGO01YIkOHzka2up08wvgYD0mDiI+q3Q==",
-      "license": "MIT",
-      "dependencies": {
-        "debug": "^4.4.0",
-        "encodeurl": "^2.0.0",
-        "escape-html": "^1.0.3",
-        "on-finished": "^2.4.1",
-        "parseurl": "^1.3.3",
-        "statuses": "^2.0.1"
-      },
-      "engines": {
-        "node": ">= 0.8"
-      }
-    },
-    "node_modules/forwarded": {
-      "version": "0.2.0",
-      "resolved": "https://registry.npmjs.org/forwarded/-/forwarded-0.2.0.tgz",
-      "integrity": "sha512-buRG0fpBtRHSTCOASe6hD258tEubFoRLb4ZNA6NxMVHNw2gOcwHo9wyablzMzOA5z9xA9L1KNjk/Nt6MT9aYow==",
-      "license": "MIT",
-      "engines": {
-        "node": ">= 0.6"
-      }
-    },
-    "node_modules/fresh": {
-      "version": "2.0.0",
-      "resolved": "https://registry.npmjs.org/fresh/-/fresh-2.0.0.tgz",
-      "integrity": "sha512-Rx/WycZ60HOaqLKAi6cHRKKI7zxWbJ31MhntmtwMoaTeF7XFH9hhBp8vITaMidfljRQ6eYWCKkaTK+ykVJHP2A==",
-      "license": "MIT",
-      "engines": {
-        "node": ">= 0.8"
-      }
-    },
-    "node_modules/function-bind": {
-      "version": "1.1.2",
-      "resolved": "https://registry.npmjs.org/function-bind/-/function-bind-1.1.2.tgz",
-      "integrity": "sha512-7XHNxH7qX9xG5mIwxkhumTox/MIRNcOgDrxWsMt2pAr23WHp6MrRlN7FBSFpCpr+oVO0F744iUgR82nJMfG2SA==",
-      "license": "MIT",
-      "funding": {
-        "url": "https://github.com/sponsors/ljharb"
-      }
-    },
-    "node_modules/get-intrinsic": {
-      "version": "1.3.0",
-      "resolved": "https://registry.npmjs.org/get-intrinsic/-/get-intrinsic-1.3.0.tgz",
-      "integrity": "sha512-9fSjSaos/fRIVIp+xSJlE6lfwhES7LNtKaCBIamHsjr2na1BiABJPo0mOjjz8GJDURarmCPGqaiVg5mfjb98CQ==",
-      "license": "MIT",
-      "dependencies": {
-        "call-bind-apply-helpers": "^1.0.2",
-        "es-define-property": "^1.0.1",
-        "es-errors": "^1.3.0",
-        "es-object-atoms": "^1.1.1",
-        "function-bind": "^1.1.2",
-        "get-proto": "^1.0.1",
-        "gopd": "^1.2.0",
-        "has-symbols": "^1.1.0",
-        "hasown": "^2.0.2",
-        "math-intrinsics": "^1.1.0"
-      },
-      "engines": {
-        "node": ">= 0.4"
-      },
-      "funding": {
-        "url": "https://github.com/sponsors/ljharb"
-      }
-    },
-    "node_modules/get-proto": {
-      "version": "1.0.1",
-      "resolved": "https://registry.npmjs.org/get-proto/-/get-proto-1.0.1.tgz",
-      "integrity": "sha512-sTSfBjoXBp89JvIKIefqw7U2CCebsc74kiY6awiGogKtoSGbgjYE/G/+l9sF3MWFPNc9IcoOC4ODfKHfxFmp0g==",
-      "license": "MIT",
-      "dependencies": {
-        "dunder-proto": "^1.0.1",
-        "es-object-atoms": "^1.0.0"
-      },
-      "engines": {
-        "node": ">= 0.4"
-      }
-    },
-    "node_modules/gopd": {
-      "version": "1.2.0",
-      "resolved": "https://registry.npmjs.org/gopd/-/gopd-1.2.0.tgz",
-      "integrity": "sha512-ZUKRh6/kUFoAiTAtTYPZJ3hw9wNxx+BIBOijnlG9PnrJsCcSjs1wyyD6vJpaYtgnzDrKYRSqf3OO6Rfa93xsRg==",
-      "license": "MIT",
-      "engines": {
-        "node": ">= 0.4"
-      },
-      "funding": {
-        "url": "https://github.com/sponsors/ljharb"
-      }
-    },
-    "node_modules/has-symbols": {
-      "version": "1.1.0",
-      "resolved": "https://registry.npmjs.org/has-symbols/-/has-symbols-1.1.0.tgz",
-      "integrity": "sha512-1cDNdwJ2Jaohmb3sg4OmKaMBwuC48sYni5HUw2DvsC8LjGTLK9h+eb1X6RyuOHe4hT0ULCW68iomhjUoKUqlPQ==",
-      "license": "MIT",
-      "engines": {
-        "node": ">= 0.4"
-      },
-      "funding": {
-        "url": "https://github.com/sponsors/ljharb"
-      }
-    },
-    "node_modules/hasown": {
-      "version": "2.0.2",
-      "resolved": "https://registry.npmjs.org/hasown/-/hasown-2.0.2.tgz",
-      "integrity": "sha512-0hJU9SCPvmMzIBdZFqNPXWa6dqh7WdH0cII9y+CyS8rG3nL48Bclra9HmKhVVUHyPWNH5Y7xDwAB7bfgSjkUMQ==",
-      "license": "MIT",
-      "dependencies": {
-        "function-bind": "^1.1.2"
-      },
-      "engines": {
-        "node": ">= 0.4"
-      }
-    },
-    "node_modules/http-errors": {
-      "version": "2.0.1",
-      "resolved": "https://registry.npmjs.org/http-errors/-/http-errors-2.0.1.tgz",
-      "integrity": "sha512-4FbRdAX+bSdmo4AUFuS0WNiPz8NgFt+r8ThgNWmlrjQjt1Q7ZR9+zTlce2859x4KSXrwIsaeTqDoKQmtP8pLmQ==",
-      "license": "MIT",
-      "dependencies": {
-        "depd": "~2.0.0",
-        "inherits": "~2.0.4",
-        "setprototypeof": "~1.2.0",
-        "statuses": "~2.0.2",
-        "toidentifier": "~1.0.1"
-      },
-      "engines": {
-        "node": ">= 0.8"
-      },
-      "funding": {
-        "type": "opencollective",
-        "url": "https://opencollective.com/express"
-      }
-    },
-    "node_modules/iconv-lite": {
-      "version": "0.6.3",
-      "resolved": "https://registry.npmjs.org/iconv-lite/-/iconv-lite-0.6.3.tgz",
-      "integrity": "sha512-4fCk79wshMdzMp2rH06qWrJE4iolqLhCUH+OiuIgU++RB0+94NlDL81atO7GX55uUKueo0txHNtvEyI6D7WdMw==",
-      "license": "MIT",
-      "dependencies": {
-        "safer-buffer": ">= 2.1.2 < 3.0.0"
-      },
-      "engines": {
-        "node": ">=0.10.0"
-      }
-    },
-    "node_modules/inherits": {
-      "version": "2.0.4",
-      "resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.4.tgz",
-      "integrity": "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ==",
-      "license": "ISC"
-    },
-    "node_modules/ipaddr.js": {
-      "version": "1.9.1",
-      "resolved": "https://registry.npmjs.org/ipaddr.js/-/ipaddr.js-1.9.1.tgz",
-      "integrity": "sha512-0KI/607xoxSToH7GjN1FfSbLoU0+btTicjsQSWQlh/hZykN8KpmMf7uYwPW3R+akZ6R/w18ZlXSHBYXiYUPO3g==",
-      "license": "MIT",
-      "engines": {
-        "node": ">= 0.10"
-      }
-    },
-    "node_modules/is-promise": {
-      "version": "4.0.0",
-      "resolved": "https://registry.npmjs.org/is-promise/-/is-promise-4.0.0.tgz",
-      "integrity": "sha512-hvpoI6korhJMnej285dSg6nu1+e6uxs7zG3BYAm5byqDsgJNWwxzM6z6iZiAgQR4TJ30JmBTOwqZUw3WlyH3AQ==",
-      "license": "MIT"
-    },
-    "node_modules/math-intrinsics": {
-      "version": "1.1.0",
-      "resolved": "https://registry.npmjs.org/math-intrinsics/-/math-intrinsics-1.1.0.tgz",
-      "integrity": "sha512-/IXtbwEk5HTPyEwyKX6hGkYXxM9nbj64B+ilVJnC/R6B0pH5G4V3b0pVbL7DBj4tkhBAppbQUlf6F6Xl9LHu1g==",
-      "license": "MIT",
-      "engines": {
-        "node": ">= 0.4"
-      }
-    },
-    "node_modules/media-typer": {
-      "version": "1.1.0",
-      "resolved": "https://registry.npmjs.org/media-typer/-/media-typer-1.1.0.tgz",
-      "integrity": "sha512-aisnrDP4GNe06UcKFnV5bfMNPBUw4jsLGaWwWfnH3v02GnBuXX2MCVn5RbrWo0j3pczUilYblq7fQ7Nw2t5XKw==",
-      "license": "MIT",
-      "engines": {
-        "node": ">= 0.8"
-      }
-    },
-    "node_modules/merge-descriptors": {
-      "version": "2.0.0",
-      "resolved": "https://registry.npmjs.org/merge-descriptors/-/merge-descriptors-2.0.0.tgz",
-      "integrity": "sha512-Snk314V5ayFLhp3fkUREub6WtjBfPdCPY1Ln8/8munuLuiYhsABgBVWsozAG+MWMbVEvcdcpbi9R7ww22l9Q3g==",
-      "license": "MIT",
-      "engines": {
-        "node": ">=18"
-      },
-      "funding": {
-        "url": "https://github.com/sponsors/sindresorhus"
-      }
-    },
-    "node_modules/mime-db": {
-      "version": "1.54.0",
-      "resolved": "https://registry.npmjs.org/mime-db/-/mime-db-1.54.0.tgz",
-      "integrity": "sha512-aU5EJuIN2WDemCcAp2vFBfp/m4EAhWJnUNSSw0ixs7/kXbd6Pg64EmwJkNdFhB8aWt1sH2CTXrLxo/iAGV3oPQ==",
-      "license": "MIT",
-      "engines": {
-        "node": ">= 0.6"
-      }
-    },
-    "node_modules/mime-types": {
-      "version": "3.0.2",
-      "resolved": "https://registry.npmjs.org/mime-types/-/mime-types-3.0.2.tgz",
-      "integrity": "sha512-Lbgzdk0h4juoQ9fCKXW4by0UJqj+nOOrI9MJ1sSj4nI8aI2eo1qmvQEie4VD1glsS250n15LsWsYtCugiStS5A==",
-      "license": "MIT",
-      "dependencies": {
-        "mime-db": "^1.54.0"
-      },
-      "engines": {
-        "node": ">=18"
-      },
-      "funding": {
-        "type": "opencollective",
-        "url": "https://opencollective.com/express"
-      }
-    },
-    "node_modules/ms": {
-      "version": "2.1.3",
-      "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz",
-      "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==",
-      "license": "MIT"
-    },
-    "node_modules/negotiator": {
-      "version": "1.0.0",
-      "resolved": "https://registry.npmjs.org/negotiator/-/negotiator-1.0.0.tgz",
-      "integrity": "sha512-8Ofs/AUQh8MaEcrlq5xOX0CQ9ypTF5dl78mjlMNfOK08fzpgTHQRQPBxcPlEtIw0yRpws+Zo/3r+5WRby7u3Gg==",
-      "license": "MIT",
-      "engines": {
-        "node": ">= 0.6"
-      }
-    },
-    "node_modules/object-inspect": {
-      "version": "1.13.4",
-      "resolved": "https://registry.npmjs.org/object-inspect/-/object-inspect-1.13.4.tgz",
-      "integrity": "sha512-W67iLl4J2EXEGTbfeHCffrjDfitvLANg0UlX3wFUUSTx92KXRFegMHUVgSqE+wvhAbi4WqjGg9czysTV2Epbew==",
-      "license": "MIT",
-      "engines": {
-        "node": ">= 0.4"
-      },
-      "funding": {
-        "url": "https://github.com/sponsors/ljharb"
-      }
-    },
-    "node_modules/on-finished": {
-      "version": "2.4.1",
-      "resolved": "https://registry.npmjs.org/on-finished/-/on-finished-2.4.1.tgz",
-      "integrity": "sha512-oVlzkg3ENAhCk2zdv7IJwd/QUD4z2RxRwpkcGY8psCVcCYZNq4wYnVWALHM+brtuJjePWiYF/ClmuDr8Ch5+kg==",
-      "license": "MIT",
-      "dependencies": {
-        "ee-first": "1.1.1"
-      },
-      "engines": {
-        "node": ">= 0.8"
-      }
-    },
-    "node_modules/once": {
-      "version": "1.4.0",
-      "resolved": "https://registry.npmjs.org/once/-/once-1.4.0.tgz",
-      "integrity": "sha512-lNaJgI+2Q5URQBkccEKHTQOPaXdUxnZZElQTZY0MFUAuaEqe1E+Nyvgdz/aIyNi6Z9MzO5dv1H8n58/GELp3+w==",
-      "license": "ISC",
-      "dependencies": {
-        "wrappy": "1"
-      }
-    },
-    "node_modules/parseurl": {
-      "version": "1.3.3",
-      "resolved": "https://registry.npmjs.org/parseurl/-/parseurl-1.3.3.tgz",
-      "integrity": "sha512-CiyeOxFT/JZyN5m0z9PfXw4SCBJ6Sygz1Dpl0wqjlhDEGGBP1GnsUVEL0p63hoG1fcj3fHynXi9NYO4nWOL+qQ==",
-      "license": "MIT",
-      "engines": {
-        "node": ">= 0.8"
-      }
-    },
-    "node_modules/path-to-regexp": {
-      "version": "8.3.0",
-      "resolved": "https://registry.npmjs.org/path-to-regexp/-/path-to-regexp-8.3.0.tgz",
-      "integrity": "sha512-7jdwVIRtsP8MYpdXSwOS0YdD0Du+qOoF/AEPIt88PcCFrZCzx41oxku1jD88hZBwbNUIEfpqvuhjFaMAqMTWnA==",
-      "license": "MIT",
-      "funding": {
-        "type": "opencollective",
-        "url": "https://opencollective.com/express"
-      }
-    },
-    "node_modules/proxy-addr": {
-      "version": "2.0.7",
-      "resolved": "https://registry.npmjs.org/proxy-addr/-/proxy-addr-2.0.7.tgz",
-      "integrity": "sha512-llQsMLSUDUPT44jdrU/O37qlnifitDP+ZwrmmZcoSKyLKvtZxpyV0n2/bD/N4tBAAZ/gJEdZU7KMraoK1+XYAg==",
-      "license": "MIT",
-      "dependencies": {
-        "forwarded": "0.2.0",
-        "ipaddr.js": "1.9.1"
-      },
-      "engines": {
-        "node": ">= 0.10"
-      }
-    },
-    "node_modules/qs": {
-      "version": "6.14.0",
-      "resolved": "https://registry.npmjs.org/qs/-/qs-6.14.0.tgz",
-      "integrity": "sha512-YWWTjgABSKcvs/nWBi9PycY/JiPJqOD4JA6o9Sej2AtvSGarXxKC3OQSk4pAarbdQlKAh5D4FCQkJNkW+GAn3w==",
-      "license": "BSD-3-Clause",
-      "dependencies": {
-        "side-channel": "^1.1.0"
-      },
-      "engines": {
-        "node": ">=0.6"
-      },
-      "funding": {
-        "url": "https://github.com/sponsors/ljharb"
-      }
-    },
-    "node_modules/range-parser": {
-      "version": "1.2.1",
-      "resolved": "https://registry.npmjs.org/range-parser/-/range-parser-1.2.1.tgz",
-      "integrity": "sha512-Hrgsx+orqoygnmhFbKaHE6c296J+HTAQXoxEF6gNupROmmGJRoyzfG3ccAveqCBrwr/2yxQ5BVd/GTl5agOwSg==",
-      "license": "MIT",
-      "engines": {
-        "node": ">= 0.6"
-      }
-    },
-    "node_modules/raw-body": {
-      "version": "3.0.1",
-      "resolved": "https://registry.npmjs.org/raw-body/-/raw-body-3.0.1.tgz",
-      "integrity": "sha512-9G8cA+tuMS75+6G/TzW8OtLzmBDMo8p1JRxN5AZ+LAp8uxGA8V8GZm4GQ4/N5QNQEnLmg6SS7wyuSmbKepiKqA==",
-      "license": "MIT",
-      "dependencies": {
-        "bytes": "3.1.2",
-        "http-errors": "2.0.0",
-        "iconv-lite": "0.7.0",
-        "unpipe": "1.0.0"
-      },
-      "engines": {
-        "node": ">= 0.10"
-      }
-    },
-    "node_modules/raw-body/node_modules/http-errors": {
-      "version": "2.0.0",
-      "resolved": "https://registry.npmjs.org/http-errors/-/http-errors-2.0.0.tgz",
-      "integrity": "sha512-FtwrG/euBzaEjYeRqOgly7G0qviiXoJWnvEH2Z1plBdXgbyjv34pHTSb9zoeHMyDy33+DWy5Wt9Wo+TURtOYSQ==",
-      "license": "MIT",
-      "dependencies": {
-        "depd": "2.0.0",
-        "inherits": "2.0.4",
-        "setprototypeof": "1.2.0",
-        "statuses": "2.0.1",
-        "toidentifier": "1.0.1"
-      },
-      "engines": {
-        "node": ">= 0.8"
-      }
-    },
-    "node_modules/raw-body/node_modules/iconv-lite": {
-      "version": "0.7.0",
-      "resolved": "https://registry.npmjs.org/iconv-lite/-/iconv-lite-0.7.0.tgz",
-      "integrity": "sha512-cf6L2Ds3h57VVmkZe+Pn+5APsT7FpqJtEhhieDCvrE2MK5Qk9MyffgQyuxQTm6BChfeZNtcOLHp9IcWRVcIcBQ==",
-      "license": "MIT",
-      "dependencies": {
-        "safer-buffer": ">= 2.1.2 < 3.0.0"
-      },
-      "engines": {
-        "node": ">=0.10.0"
-      },
-      "funding": {
-        "type": "opencollective",
-        "url": "https://opencollective.com/express"
-      }
-    },
-    "node_modules/raw-body/node_modules/statuses": {
-      "version": "2.0.1",
-      "resolved": "https://registry.npmjs.org/statuses/-/statuses-2.0.1.tgz",
-      "integrity": "sha512-RwNA9Z/7PrK06rYLIzFMlaF+l73iwpzsqRIFgbMLbTcLD6cOao82TaWefPXQvB2fOC4AjuYSEndS7N/mTCbkdQ==",
-      "license": "MIT",
-      "engines": {
-        "node": ">= 0.8"
-      }
-    },
-    "node_modules/router": {
-      "version": "2.2.0",
-      "resolved": "https://registry.npmjs.org/router/-/router-2.2.0.tgz",
-      "integrity": "sha512-nLTrUKm2UyiL7rlhapu/Zl45FwNgkZGaCpZbIHajDYgwlJCOzLSk+cIPAnsEqV955GjILJnKbdQC1nVPz+gAYQ==",
-      "license": "MIT",
-      "dependencies": {
-        "debug": "^4.4.0",
-        "depd": "^2.0.0",
-        "is-promise": "^4.0.0",
-        "parseurl": "^1.3.3",
-        "path-to-regexp": "^8.0.0"
-      },
-      "engines": {
-        "node": ">= 18"
-      }
-    },
-    "node_modules/safer-buffer": {
-      "version": "2.1.2",
-      "resolved": "https://registry.npmjs.org/safer-buffer/-/safer-buffer-2.1.2.tgz",
-      "integrity": "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg==",
-      "license": "MIT"
-    },
-    "node_modules/send": {
-      "version": "1.2.0",
-      "resolved": "https://registry.npmjs.org/send/-/send-1.2.0.tgz",
-      "integrity": "sha512-uaW0WwXKpL9blXE2o0bRhoL2EGXIrZxQ2ZQ4mgcfoBxdFmQold+qWsD2jLrfZ0trjKL6vOw0j//eAwcALFjKSw==",
-      "license": "MIT",
-      "dependencies": {
-        "debug": "^4.3.5",
-        "encodeurl": "^2.0.0",
-        "escape-html": "^1.0.3",
-        "etag": "^1.8.1",
-        "fresh": "^2.0.0",
-        "http-errors": "^2.0.0",
-        "mime-types": "^3.0.1",
-        "ms": "^2.1.3",
-        "on-finished": "^2.4.1",
-        "range-parser": "^1.2.1",
-        "statuses": "^2.0.1"
-      },
-      "engines": {
-        "node": ">= 18"
-      }
-    },
-    "node_modules/serve-static": {
-      "version": "2.2.0",
-      "resolved": "https://registry.npmjs.org/serve-static/-/serve-static-2.2.0.tgz",
-      "integrity": "sha512-61g9pCh0Vnh7IutZjtLGGpTA355+OPn2TyDv/6ivP2h/AdAVX9azsoxmg2/M6nZeQZNYBEwIcsne1mJd9oQItQ==",
-      "license": "MIT",
-      "dependencies": {
-        "encodeurl": "^2.0.0",
-        "escape-html": "^1.0.3",
-        "parseurl": "^1.3.3",
-        "send": "^1.2.0"
-      },
-      "engines": {
-        "node": ">= 18"
-      }
-    },
-    "node_modules/setprototypeof": {
-      "version": "1.2.0",
-      "resolved": "https://registry.npmjs.org/setprototypeof/-/setprototypeof-1.2.0.tgz",
-      "integrity": "sha512-E5LDX7Wrp85Kil5bhZv46j8jOeboKq5JMmYM3gVGdGH8xFpPWXUMsNrlODCrkoxMEeNi/XZIwuRvY4XNwYMJpw==",
-      "license": "ISC"
-    },
-    "node_modules/side-channel": {
-      "version": "1.1.0",
-      "resolved": "https://registry.npmjs.org/side-channel/-/side-channel-1.1.0.tgz",
-      "integrity": "sha512-ZX99e6tRweoUXqR+VBrslhda51Nh5MTQwou5tnUDgbtyM0dBgmhEDtWGP/xbKn6hqfPRHujUNwz5fy/wbbhnpw==",
-      "license": "MIT",
-      "dependencies": {
-        "es-errors": "^1.3.0",
-        "object-inspect": "^1.13.3",
-        "side-channel-list": "^1.0.0",
-        "side-channel-map": "^1.0.1",
-        "side-channel-weakmap": "^1.0.2"
-      },
-      "engines": {
-        "node": ">= 0.4"
-      },
-      "funding": {
-        "url": "https://github.com/sponsors/ljharb"
-      }
-    },
-    "node_modules/side-channel-list": {
-      "version": "1.0.0",
-      "resolved": "https://registry.npmjs.org/side-channel-list/-/side-channel-list-1.0.0.tgz",
-      "integrity": "sha512-FCLHtRD/gnpCiCHEiJLOwdmFP+wzCmDEkc9y7NsYxeF4u7Btsn1ZuwgwJGxImImHicJArLP4R0yX4c2KCrMrTA==",
-      "license": "MIT",
-      "dependencies": {
-        "es-errors": "^1.3.0",
-        "object-inspect": "^1.13.3"
-      },
-      "engines": {
-        "node": ">= 0.4"
-      },
-      "funding": {
-        "url": "https://github.com/sponsors/ljharb"
-      }
-    },
-    "node_modules/side-channel-map": {
-      "version": "1.0.1",
-      "resolved": "https://registry.npmjs.org/side-channel-map/-/side-channel-map-1.0.1.tgz",
-      "integrity": "sha512-VCjCNfgMsby3tTdo02nbjtM/ewra6jPHmpThenkTYh8pG9ucZ/1P8So4u4FGBek/BjpOVsDCMoLA/iuBKIFXRA==",
-      "license": "MIT",
-      "dependencies": {
-        "call-bound": "^1.0.2",
-        "es-errors": "^1.3.0",
-        "get-intrinsic": "^1.2.5",
-        "object-inspect": "^1.13.3"
-      },
-      "engines": {
-        "node": ">= 0.4"
-      },
-      "funding": {
-        "url": "https://github.com/sponsors/ljharb"
-      }
-    },
-    "node_modules/side-channel-weakmap": {
-      "version": "1.0.2",
-      "resolved": "https://registry.npmjs.org/side-channel-weakmap/-/side-channel-weakmap-1.0.2.tgz",
-      "integrity": "sha512-WPS/HvHQTYnHisLo9McqBHOJk2FkHO/tlpvldyrnem4aeQp4hai3gythswg6p01oSoTl58rcpiFAjF2br2Ak2A==",
-      "license": "MIT",
-      "dependencies": {
-        "call-bound": "^1.0.2",
-        "es-errors": "^1.3.0",
-        "get-intrinsic": "^1.2.5",
-        "object-inspect": "^1.13.3",
-        "side-channel-map": "^1.0.1"
-      },
-      "engines": {
-        "node": ">= 0.4"
-      },
-      "funding": {
-        "url": "https://github.com/sponsors/ljharb"
-      }
-    },
-    "node_modules/statuses": {
-      "version": "2.0.2",
-      "resolved": "https://registry.npmjs.org/statuses/-/statuses-2.0.2.tgz",
-      "integrity": "sha512-DvEy55V3DB7uknRo+4iOGT5fP1slR8wQohVdknigZPMpMstaKJQWhwiYBACJE3Ul2pTnATihhBYnRhZQHGBiRw==",
-      "license": "MIT",
-      "engines": {
-        "node": ">= 0.8"
-      }
-    },
-    "node_modules/toidentifier": {
-      "version": "1.0.1",
-      "resolved": "https://registry.npmjs.org/toidentifier/-/toidentifier-1.0.1.tgz",
-      "integrity": "sha512-o5sSPKEkg/DIQNmH43V0/uerLrpzVedkUh8tGNvaeXpfpuwjKenlSox/2O/BTlZUtEe+JG7s5YhEz608PlAHRA==",
-      "license": "MIT",
-      "engines": {
-        "node": ">=0.6"
-      }
-    },
-    "node_modules/type-is": {
-      "version": "2.0.1",
-      "resolved": "https://registry.npmjs.org/type-is/-/type-is-2.0.1.tgz",
-      "integrity": "sha512-OZs6gsjF4vMp32qrCbiVSkrFmXtG/AZhY3t0iAMrMBiAZyV9oALtXO8hsrHbMXF9x6L3grlFuwW2oAz7cav+Gw==",
-      "license": "MIT",
-      "dependencies": {
-        "content-type": "^1.0.5",
-        "media-typer": "^1.1.0",
-        "mime-types": "^3.0.0"
-      },
-      "engines": {
-        "node": ">= 0.6"
-      }
-    },
-    "node_modules/typescript": {
-      "version": "5.9.3",
-      "resolved": "https://registry.npmjs.org/typescript/-/typescript-5.9.3.tgz",
-      "integrity": "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw==",
-      "license": "Apache-2.0",
-      "bin": {
-        "tsc": "bin/tsc",
-        "tsserver": "bin/tsserver"
-      },
-      "engines": {
-        "node": ">=14.17"
-      }
-    },
-    "node_modules/unpipe": {
-      "version": "1.0.0",
-      "resolved": "https://registry.npmjs.org/unpipe/-/unpipe-1.0.0.tgz",
-      "integrity": "sha512-pjy2bYhSsufwWlKwPc+l3cN7+wuJlK6uz0YdJEOlQDbl6jo/YlPi4mb8agUkVC8BF7V8NuzeyPNqRksA3hztKQ==",
-      "license": "MIT",
-      "engines": {
-        "node": ">= 0.8"
-      }
-    },
-    "node_modules/vary": {
-      "version": "1.1.2",
-      "resolved": "https://registry.npmjs.org/vary/-/vary-1.1.2.tgz",
-      "integrity": "sha512-BNGbWLfd0eUPabhkXUVm0j8uuvREyTh5ovRa/dyow/BqAbZJyC+5fU+IzQOzmAKzYqYRAISoRhdQr3eIZ/PXqg==",
-      "license": "MIT",
-      "engines": {
-        "node": ">= 0.8"
-      }
-    },
-    "node_modules/wrappy": {
-      "version": "1.0.2",
-      "resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz",
-      "integrity": "sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ==",
-      "license": "ISC"
-    }
-  }
-}

package.json

@@ -1,19 +0,0 @@
-{
-  "name": "warbler-cda",
-  "version": "1.0.0",
-  "description": "--- title: Warbler CDA RAG System emoji: 🦜 colorFrom: blue colorTo: purple sdk: gradio sdk_version: 5.49.1 app_file: app.py pinned: false license: mit tags:   - rag   - retrieval   - semantic-search   - stat7   - embeddings   - nlp ---",
-  "main": "index.js",
-  "directories": {
-    "test": "tests"
-  },
-  "scripts": {
-    "test": "echo \"Error: no test specified\" && exit 1"
-  },
-  "keywords": [],
-  "author": "",
-  "license": "ISC",
-  "dependencies": {
-    "express": "^5.1.0",
-    "typescript": "^5.9.3"
-  }
-}

packs/warbler-pack-core/README.md

@@ -1,227 +0,0 @@
-# Warbler Pack Core
-
-Essential conversation templates for the Warbler NPC conversation system.
-
-## Overview
-
-This content pack provides fundamental conversation templates that form the backbone of most NPC interactions. It includes greetings, farewells, help responses, trade inquiries, and general conversation fallbacks suitable for a wide variety of NPCs and scenarios.
-
-## Installation
-
-```bash
-npm install warbler-pack-core
-```
-
-## Usage
-
-### Basic Usage with Warbler Engine
-
-```typescript
-import { Warbler } from 'warbler-core';
-import corePackTemplates from 'warbler-pack-core';
-
-const warbler = new Warbler();
-
-// Register all core pack templates
-warbler.registerTemplates(corePackTemplates.templates);
-
-// Or register specific templates
-warbler.registerTemplate(corePackTemplates.greetingFriendly);
-warbler.registerTemplate(corePackTemplates.farewellFormal);
-```
-
-### Individual Template Imports
-
-```typescript
-import { greetingFriendly, helpGeneral } from 'warbler-pack-core';
-import { Warbler } from 'warbler-core';
-
-const warbler = new Warbler();
-warbler.registerTemplate(greetingFriendly);
-warbler.registerTemplate(helpGeneral);
-```
-
-### JSON Template Access
-
-```typescript
-// Access raw template data
-import templateData from 'warbler-pack-core/templates';
-console.log('Available templates:', templateData.templates.length);
-```
-
-## Template Categories
-
-### Greetings
-
-- **`greeting_friendly`**: Casual, warm greeting for friendly NPCs
-- **`greeting_formal`**: Professional greeting for officials and merchants
-
-### Farewells
-
-- **`farewell_friendly`**: Warm goodbye with well-wishes  
-- **`farewell_formal`**: Polite, professional farewell
-
-### Help & Assistance
-
-- **`help_general`**: General offer of assistance and local knowledge
-
-### Commerce
-
-- **`trade_inquiry_welcome`**: Welcoming response to trade requests
-
-### Conversation
-
-- **`general_conversation`**: Fallback for maintaining conversation flow
-- **`unknown_response`**: Graceful handling of unclear input
-
-## Template Structure
-
-Each template includes:
-
-- **Unique ID**: Stable identifier for template selection
-- **Semantic Version**: For tracking template evolution  
-- **Content**: Response text with slot placeholders (`{{slot_name}}`)
-- **Required Slots**: Variables needed for template completion
-- **Tags**: Keywords for intent matching and categorization
-- **Length Limits**: Maximum character constraints for responses
-
-### Common Slots
-
-Most core pack templates use these standard slots:
-
-- `user_name` (string): Name to address the user
-- `location` (string): Current scene or area name
-- `time_of_day` (string): Current time period (morning, afternoon, etc.)
-- `npc_name` (string): Name of the speaking NPC
-- `user_title` (string): Formal address for the user
-
-## Versioning Policy
-
-This content pack follows semantic versioning with content-specific conventions:
-
-- **Major versions** introduce breaking changes to template contracts or slot requirements
-- **Minor versions** add new templates while maintaining backward compatibility
-- **Patch versions** contain content improvements, typo fixes, and minor enhancements
-
-## Template Validation
-
-All templates in this pack are validated for:
-
-- ✅ Required field presence (id, version, content, etc.)
-- ✅ Unique template IDs within the pack
-- ✅ Content length limits (all templates ≤ 200 characters)
-- ✅ Valid slot type definitions
-- ✅ Consistent slot naming conventions
-
-## Integration Examples
-
-### Complete NPC Setup
-
-```typescript
-import { Warbler, WarblerContext } from 'warbler-core';
-import corePackTemplates from 'warbler-pack-core';
-
-// Initialize conversation system
-const warbler = new Warbler();
-warbler.registerTemplates(corePackTemplates.templates);
-
-// Set up NPC context
-const context: WarblerContext = {
-  npcId: 'merchant_sara',
-  sceneId: 'marketplace',
-  previousUtterances: [],
-  worldState: { 
-    time_of_day: 'morning',
-    weather: 'sunny'
-  },
-  conversationHistory: []
-};
-
-// Process player greeting
-const result = warbler.processConversation(
-  'Good morning!', 
-  context,
-  { 
-    user_name: 'Traveler',
-    location: 'Riverside Market' 
-  }
-);
-
-console.log(result.utterance?.content);
-// Output: "Hello there, Traveler! Welcome to Riverside Market. It's a beautiful morning today, isn't it?"
-```
-
-### Custom Slot Providers
-
-```typescript
-// Extend with custom slot resolution
-const customSlots = {
-  user_name: playerData.characterName,
-  location: gameState.currentArea.displayName,
-  npc_name: npcDatabase.getNpcName(context.npcId),
-  time_of_day: gameTime.getCurrentPeriod()
-};
-
-const result = warbler.processConversation(userInput, context, customSlots);
-```
-
-## Pack Metadata
-
-```typescript
-import { packMetadata } from 'warbler-pack-core';
-
-console.log(`Pack: ${packMetadata.name} v${packMetadata.version}`);
-console.log(`Templates: ${packMetadata.templates.length}`);
-console.log(`Description: ${packMetadata.description}`);
-```
-
-## Contributing
-
-This pack is part of the Warbler ecosystem. When contributing new templates:
-
-1. Follow the established naming conventions (`category_variant`)
-2. Include comprehensive slot documentation
-3. Test templates with the validation script
-4. Ensure content is appropriate for general audiences
-5. Maintain semantic versioning for changes
-
-### Development Workflow
-
-```bash
-# Install dependencies
-npm install
-
-# Build TypeScript exports
-npm run build
-
-# Validate template JSON
-npm run validate
-
-# Test integration
-npm run prepublishOnly
-```
-
-## License
-
-MIT License - see LICENSE file for details.
-
-## Related Packages
-
-- [`warbler-core`](../warbler-core) - Core conversation engine
-- [`warbler-pack-faction-politics`](../warbler-pack-faction-politics) - Political intrigue templates
-- Additional content packs available in the Warbler ecosystem
-
-## Template Reference
-
-| Template ID | Intent Types | Description | Slots Required |
-|-------------|--------------|-------------|----------------|
-| `greeting_friendly` | greeting, casual | Warm welcome | user_name*, location*, time_of_day* |
-| `greeting_formal` | greeting, formal | Professional greeting | npc_name, user_title*, npc_role*, location*, time_of_day* |
-| `farewell_friendly` | farewell, casual | Friendly goodbye | user_name* |
-| `farewell_formal` | farewell, formal | Polite farewell | user_title* |
-| `help_general` | help_request | General assistance | user_name*, location* |
-| `trade_inquiry_welcome` | trade_inquiry | Commerce welcome | item_types* |
-| `general_conversation` | general | Conversation fallback | location*, location_type* |
-| `unknown_response` | general, fallback | Unclear input handler | (none) |
-
-*Optional slots that enhance the response when provided

packs/warbler-pack-core/README_HF_DATASET.md

@@ -1,77 +0,0 @@
----
-license: mit
-datasets:
-  - tiny-walnut-games/warbler-pack-core
-pretty_name: Warbler Pack Core - Conversation Templates
-description: Essential conversation templates for the Warbler NPC conversation system
-language:
-  - en
-tags:
-  - warbler
-  - conversation
-  - npc
-  - templates
-  - dialogue
-size_categories:
-  - n<1K
-source_datasets: []
----
-
-# Warbler Pack Core - Conversation Templates
-
-Essential conversation templates for the Warbler NPC conversation system.
-
-## Dataset Overview
-
-This dataset contains foundational conversation templates that form the backbone of NPC interactions. It includes greetings, farewells, help responses, trade inquiries, and general conversation fallbacks suitable for a wide variety of NPCs and scenarios.
-
-**Documents**: ~10 templates
-**Language**: English
-**License**: MIT
-**Source**: Tiny Walnut Games - The Seed Project
-
-## Dataset Structure
-
-```
-{
-  "template_id": str,
-  "intent_types": [str],
-  "content": str,
-  "required_slots": [str],
-  "tags": [str],
-  "max_length": int
-}
-```
-
-## Template Categories
-
-- **Greetings**: friendly and formal greetings for NPCs
-- **Farewells**: warm and professional goodbyes
-- **Help & Assistance**: general assistance offers
-- **Commerce**: trade and merchant interactions
-- **Conversation**: fallback templates for maintaining conversation flow
-
-## Use Cases
-
-- NPC dialogue systems
-- Conversational AI training
-- Game narrative generation
-- Interactive fiction engines
-- Dialogue management systems
-
-## Attribution
-
-Part of **Warbler CDA** (Cognitive Development Architecture) - a production-ready RAG system featuring FractalStat multi-dimensional addressing.
-
-**Project**: [The Seed](https://github.com/tiny-walnut-games/the-seed)
-**Organization**: [Tiny Walnut Games](https://github.com/tiny-walnut-games)
-
-## Related Datasets
-
-- [warbler-pack-faction-politics](https://huggingface.co/datasets/tiny-walnut-games/warbler-pack-faction-politics) - Political intrigue templates
-- [warbler-pack-wisdom-scrolls](https://huggingface.co/datasets/tiny-walnut-games/warbler-pack-wisdom-scrolls) - Wisdom generation templates
-- [warbler-pack-hf-npc-dialogue](https://huggingface.co/datasets/tiny-walnut-games/warbler-pack-hf-npc-dialogue) - NPC dialogue from HuggingFace sources
-
-## License
-
-MIT License - See project LICENSE file for details.

packs/warbler-pack-faction-politics/README.md

@@ -1,267 +0,0 @@
-# Warbler Pack: Faction Politics
-
-Specialized conversation templates for political intrigue, faction diplomacy, and court machinations in the Warbler NPC conversation system.
-
-## Overview
-
-This content pack provides sophisticated dialogue templates for NPCs involved in political intrigue, diplomatic negotiations, and factional conflicts. Perfect for games and narratives featuring court politics, espionage, alliances, and betrayals.
-
-## Installation
-
-```bash
-npm install warbler-pack-faction-politics
-```
-
-## Usage
-
-### Basic Usage with Warbler Engine
-
-```typescript
-import { Warbler } from 'warbler-core';
-import politicsPackTemplates from 'warbler-pack-faction-politics';
-
-const warbler = new Warbler();
-
-// Register all politics pack templates
-warbler.registerTemplates(politicsPackTemplates.templates);
-
-// Or register specific templates
-warbler.registerTemplate(politicsPackTemplates.warningPoliticalThreat);
-warbler.registerTemplate(politicsPackTemplates.allianceProposal);
-```
-
-### Themed Template Sets
-
-```typescript
-import { 
-  warningPoliticalThreat,
-  intrigueInformationTrade,
-  betrayalRevelation 
-} from 'warbler-pack-faction-politics';
-
-// Create a spy/informant NPC
-const spyTemplates = [intrigueInformationTrade, betrayalRevelation];
-warbler.registerTemplates(spyTemplates);
-
-// Create a diplomatic NPC  
-import { allianceProposal, diplomaticImmunityClaim } from 'warbler-pack-faction-politics';
-const diplomatTemplates = [allianceProposal, diplomaticImmunityClaim];
-warbler.registerTemplates(diplomatTemplates);
-```
-
-## Template Categories
-
-### Threats & Warnings
-
-- **`warning_political_threat`**: Veiled warnings about faction displeasure and consequences
-
-### Information Trading
-
-- **`intrigue_information_trade`**: Offering to trade political secrets and intelligence
-
-### Diplomacy
-
-- **`alliance_proposal`**: Diplomatic overtures for political cooperation
-- **`diplomatic_immunity_claim`**: Claiming diplomatic protection and immunity
-
-### Betrayal & Conspiracy
-
-- **`betrayal_revelation`**: Revealing political betrayals and double-crosses
-- **`faction_loyalty_test`**: Testing political allegiance and commitment
-
-## Template Structure
-
-### Political Slots
-
-This pack introduces specialized slots for political scenarios:
-
-- `faction_name` (string): Name of political faction
-- `faction_leader` (string): Leader of the faction  
-- `faction_pronoun` (string): Pronouns for faction leader
-- `user_title` (string): Formal political title for the user
-- `diplomatic_title` (string): Official diplomatic rank
-- `target_faction` (string): Faction being discussed or targeted
-- `rival_faction` (string): Opposing or enemy faction
-- `betrayer_name` (string): Name of person committing betrayal
-- `threat_description` (string): Description of common threat or enemy
-
-### Common Usage Patterns
-
-Most templates support contextual political conversations:
-
-```typescript
-const politicalContext = {
-  npcId: 'court_advisor_001',
-  sceneId: 'royal_court',
-  worldState: {
-    current_faction: 'House Starwind',
-    rival_faction: 'House Blackmoor',
-    political_tension: 'high'
-  },
-  conversationHistory: []
-};
-
-const politicalSlots = {
-  faction_name: 'House Starwind',
-  faction_leader: 'Lord Commander Theron',
-  user_title: 'Honored Guest',
-  location: 'the Royal Court'
-};
-```
-
-## Advanced Examples
-
-### Political Intrigue Scene
-
-```typescript
-import { Warbler, WarblerContext } from 'warbler-core';
-import { warningPoliticalThreat, intrigueInformationTrade } from 'warbler-pack-faction-politics';
-
-const warbler = new Warbler();
-warbler.registerTemplate(warningPoliticalThreat);
-warbler.registerTemplate(intrigueInformationTrade);
-
-// Court advisor warns about faction consequences
-const threatContext: WarblerContext = {
-  npcId: 'advisor_suspicious',
-  sceneId: 'private_chamber',
-  previousUtterances: [],
-  worldState: { 
-    political_climate: 'tense',
-    player_faction_standing: 'negative'
-  },
-  conversationHistory: []
-};
-
-const result = warbler.processIntent(
-  { type: 'warning', confidence: 0.9, slots: {} },
-  threatContext,
-  {
-    user_name: 'Sir Blackwood',
-    faction_name: 'the Iron Circle',
-    faction_leader: 'Magistrate Vex',
-    faction_pronoun: 'them',
-    location: 'the merchant district'
-  }
-);
-
-console.log(result.utterance?.content);
-// Output: "Sir Blackwood, I would tread carefully if I were you. The Iron Circle has long memories, and Magistrate Vex does not forget those who cross them. Your recent actions in the merchant district have not gone unnoticed."
-```
-
-### Diplomatic Negotiation
-
-```typescript
-import { allianceProposal, factionLoyaltyTest } from 'warbler-pack-faction-politics';
-
-// Ambassador proposing alliance
-const diplomaticSlots = {
-  user_title: 'Your Lordship',
-  our_faction: 'the Northern Alliance',
-  threat_description: 'the growing shadow from the East'
-};
-
-const result = warbler.processIntent(
-  { type: 'alliance', confidence: 0.85, slots: {} },
-  context,
-  diplomaticSlots
-);
-
-// Output: "The times ahead will test us all, Your Lordship. The Northern Alliance and your people share common interests against the growing shadow from the East. Perhaps it is time we discussed a more... formal arrangement between our houses?"
-```
-
-### Information Broker Scenario
-
-```typescript
-import { intrigueInformationTrade, betrayalRevelation } from 'warbler-pack-faction-politics';
-
-// Spy offering information trade
-const spySlots = {
-  user_name: 'Captain',
-  location: 'the Capital',
-  target_faction: 'House Ravencrest'
-};
-
-const infoResult = warbler.processIntent(
-  { type: 'intrigue', confidence: 0.9, slots: {} },
-  context,
-  spySlots
-);
-
-// Later revealing betrayal
-const betrayalSlots = {
-  user_name: 'Captain',
-  betrayer_name: 'Lieutenant Hayes',
-  betrayer_pronoun: 'He',
-  rival_faction: 'the Shadow Syndicate',
-  location: 'the harbor'
-};
-
-const betrayalResult = warbler.processIntent(
-  { type: 'betrayal', confidence: 0.95, slots: {} },
-  context,
-  betrayalSlots
-);
-```
-
-## Content Guidelines
-
-This pack contains mature political themes suitable for:
-
-- ✅ Political intrigue and court drama
-- ✅ Diplomatic negotiations and alliance building
-- ✅ Espionage and information trading
-- ✅ Betrayal and conspiracy revelations
-- ✅ Faction-based conflicts and loyalty tests
-
-Content is designed for:
-- Fantasy/medieval political settings
-- Modern political thrillers
-- Sci-fi diplomatic scenarios
-- Any narrative requiring sophisticated political dialogue
-
-## Template Reference
-
-| Template ID | Intent Types | Primary Use | Key Slots |
-|-------------|--------------|-------------|-----------|
-| `warning_political_threat` | warning, politics | Faction warnings | faction_name*, faction_leader* |
-| `intrigue_information_trade` | intrigue, trade | Information trading | target_faction* |
-| `alliance_proposal` | alliance, diplomacy | Diplomatic overtures | our_faction*, threat_description* |
-| `betrayal_revelation` | betrayal, revelation | Conspiracy reveals | betrayer_name*, rival_faction* |
-| `faction_loyalty_test` | loyalty, test | Allegiance testing | faction_name*, faction_leader* |
-| `diplomatic_immunity_claim` | diplomacy, immunity | Legal protection | npc_name*, faction_name* |
-
-*Required slots for proper template function
-
-## Versioning & Compatibility
-
-- **Engine Compatibility**: Requires warbler-core ^0.1.0
-- **Content Rating**: Mature political themes
-- **Language**: Formal/elevated register appropriate for political discourse
-- **Character Limits**: All templates ≤ 320 characters for reasonable response lengths
-
-## Development & Contributing
-
-This pack follows political dialogue conventions:
-
-1. **Formal Register**: Uses elevated, courtly language
-2. **Implicit Threats**: Suggests consequences without explicit violence
-3. **Political Terminology**: Employs faction, diplomatic, and court language
-4. **Contextual Awareness**: References political relationships and power structures
-
-### Validation
-
-```bash
-npm run validate  # Validates template JSON structure
-npm run build     # Compiles TypeScript exports
-```
-
-## License
-
-MIT License - see LICENSE file for details.
-
-## Related Packages
-
-- [`warbler-core`](../warbler-core) - Core conversation engine
-- [`warbler-pack-core`](../warbler-pack-core) - Essential conversation templates
-- Additional specialized packs available in the Warbler ecosystem

packs/warbler-pack-faction-politics/README_HF_DATASET.md

@@ -1,88 +0,0 @@
----
-license: mit
-datasets:
-  - tiny-walnut-games/warbler-pack-faction-politics
-pretty_name: Warbler Pack Faction Politics - Political Dialogue Templates
-description: Political intrigue and faction interaction templates for the Warbler conversation system
-language:
-  - en
-tags:
-  - warbler
-  - conversation
-  - dialogue
-  - faction
-  - politics
-  - npc
-  - templates
-size_categories:
-  - n<1K
-source_datasets: []
----
-
-# Warbler Pack Faction Politics - Political Dialogue Templates
-
-Political intrigue and faction interaction templates for the Warbler conversation system.
-
-## Dataset Overview
-
-This dataset contains specialized conversation templates for handling faction politics, diplomatic negotiations, and politically-charged NPC interactions. It supports nuanced dialogue around loyalty, allegiance, political maneuvering, and factional relationships.
-
-**Documents**: ~15 templates
-**Language**: English
-**License**: MIT
-**Source**: Tiny Walnut Games - The Seed Project
-
-## Dataset Structure
-
-```
-{
-  "template_id": str,
-  "intent_types": [str],
-  "content": str,
-  "required_slots": [str],
-  "faction_tags": [str],
-  "tags": [str],
-  "max_length": int
-}
-```
-
-## Template Categories
-
-- **Faction Greetings**: faction-aware dialogue responses
-- **Political Negotiations**: diplomatic and negotiation templates
-- **Allegiance Responses**: loyalty and allegiance-related templates
-- **Conflict Resolution**: dispute and peace-making templates
-- **Factional Intrigue**: political maneuvering and espionage templates
-
-## Use Cases
-
-- Complex NPC dialogue systems with political dimensions
-- Faction-based game narratives
-- Diplomatic negotiation systems
-- Political simulation games
-- Interactive stories with factional conflicts
-
-## Features
-
-- Faction-aware response generation
-- Political alignment handling
-- Diplomatic tone management
-- Conflict/alliance tracking
-- FractalStat resonance optimization for political contexts
-
-## Attribution
-
-Part of **Warbler CDA** (Cognitive Development Architecture) - a production-ready RAG system featuring FractalStat multi-dimensional addressing.
-
-**Project**: [The Seed](https://github.com/tiny-walnut-games/the-seed)
-**Organization**: [Tiny Walnut Games](https://github.com/tiny-walnut-games)
-
-## Related Datasets
-
-- [warbler-pack-core](https://huggingface.co/datasets/tiny-walnut-games/warbler-pack-core) - Core conversation templates
-- [warbler-pack-wisdom-scrolls](https://huggingface.co/datasets/tiny-walnut-games/warbler-pack-wisdom-scrolls) - Wisdom generation templates
-- [warbler-pack-hf-npc-dialogue](https://huggingface.co/datasets/tiny-walnut-games/warbler-pack-hf-npc-dialogue) - NPC dialogue from HuggingFace sources
-
-## License
-
-MIT License - See project LICENSE file for details.

packs/warbler-pack-hf-arxiv/package.json

@@ -2,14 +2,14 @@
   "name": "warbler-pack-hf-arxiv",
   "version": "1.0.0",
   "description": "Warbler pack generated from HuggingFace datasets (chunked)",
-  "created_at": "2025-11-19T19:07:32.887499",
+  "created_at": "2025-12-02T10:48:41.412949",
   "document_count": 2549619,
   "source": "HuggingFace",
   "content_types": [
     "scholarly_discussion"
   ],
   "chunked": true,
-  "chunk_count": 255,
-  "docs_per_chunk": 10000,
-  "chunk_pattern": "warbler-pack-hf-arxiv-chunk-*_compressed.jsonl"
+  "chunk_count": 51,
+  "docs_per_chunk": 50000,
+  "chunk_pattern": "warbler-pack-hf-arxiv-chunk-*.jsonl"
 }