Git to HF

.gitattributes copy

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

.gitignore

@@ -0,0 +1,6 @@
+# Ignore compiled Python files
+__pycache__/
+scripts/__pycache__/
+config.py
+models/
+logs/

Dockerfile

@@ -0,0 +1,27 @@
+# Use lightweight Python base image
+FROM python:3.11-slim
+
+# Prevents Python from writing pyc files to disk & keeps output unbuffered
+ENV PYTHONDONTWRITEBYTECODE=1
+ENV PYTHONUNBUFFERED=1
+
+# Set working directory
+WORKDIR /app
+
+# Install system dependencies if needed (faiss, build tools, etc.)
+RUN apt-get update && apt-get install -y \
+    build-essential \
+    && rm -rf /var/lib/apt/lists/*
+
+# Install Python dependencies
+COPY requirements.txt .
+RUN pip install --no-cache-dir -r requirements.txt
+
+# Copy project files
+COPY . .
+
+# Expose app port (e.g., Gradio usually on 7860, FastAPI on 8000)
+EXPOSE 7860
+
+# Start your app (adjust command for your framework)
+CMD ["python", "main.py"]

README.md

@@ -1,16 +1,224 @@
----
-title: Code Compass
-emoji: 💬
-colorFrom: yellow
-colorTo: purple
-sdk: gradio
-sdk_version: 5.42.0
-app_file: app.py
-pinned: false
-hf_oauth: true
-hf_oauth_scopes:
-- inference-api
-short_description: An AI-powered tool for analyzing code repositories
----
-
-An example chatbot using [Gradio](https://gradio.app), [`huggingface_hub`](https://huggingface.co/docs/huggingface_hub/v0.22.2/en/index), and the [Hugging Face Inference API](https://huggingface.co/docs/api-inference/index).
+# 🔍code-compass
+
+An AI-powered tool for analyzing code repositories using hierarchical chunking and semantic search with Pinecone vector database.
+
+## 🚀 Features
+
+- **📥 Multiple Input Methods**: GitHub URLs or ZIP file uploads
+- **🧠 Hierarchical Chunking**: Smart code parsing at multiple levels (file → class → function → block)
+- **🔍 Semantic Search**: AI-powered natural language queries using Pinecone vector database
+- **🤖 Intelligent Analysis**: Local LLM integration with Qwen2.5-Coder-7B-Instruct
+- **💬 Conversation History**: Maintains context across multiple queries
+- **📊 Repository Analytics**: Comprehensive statistics and structure analysis
+- **🎯 Pinecone Integration**: Scalable vector database with automatic embedding generation
+- **⚡ Optimized Performance**: Quantized models for efficient local inference
+
+## 🛠️ Setup
+
+### Prerequisites
+
+1. **Python 3.8+**
+2. **Pinecone Account**: Create a free account at [Pinecone.io](https://www.pinecone.io/)
+3. **System Requirements** for LLM:
+   - **RAM**: 8GB minimum (16GB+ recommended)
+   - **Storage**: 5-8GB free space for model
+   - **CPU**: Multi-core processor (supports GPU acceleration if available)
+
+### Installation
+
+1. **Clone or download this project**
+   ```bash
+   git clone https://github.com/shahzeb171/code-compass.git
+   cd code-compass
+   ```
+
+2. **Install dependencies**
+   ```bash
+   pip install -r requirements.txt
+   ```
+
+3. **Download the LLM model**
+   ```
+   wget https://huggingface.co/bartowski/Qwen2.5-Coder-7B-Instruct-GGUF/resolve/main/Qwen2.5-Coder-7B-Instruct-Q4_K_M.gguf
+   ```
+   **Recommended**: Select Q4_K_M for the best balance of quality and performance.
+
+4. **Set up Pinecone API Key**
+   
+   Create `config.py` file:
+   ```
+   PINECONE_API_KEY=your-pinecone-api-key-here
+   PINECONE_INDEX_NAME=index_name(eg. code_compass_index)
+   PINECONE_EMBEDDING_MODEL=embedding_model(eg. llama-text-embed-v2 (check pinecone docs for more models))
+   MODEL_PATH=path_to_the_model
+   ```
+   
+### Getting Your Pinecone API Key
+
+1. Go to [Pinecone.io](https://www.pinecone.io/) and sign up for a free account
+2. Navigate to the "API Keys" section in your dashboard
+3. Create a new API key or copy an existing one
+4. The free tier includes:
+   - 1 index
+   - 5M vector dimensions
+   - Enough for most code analysis projects!
+
+## 🚀 Usage
+
+1. **Start the application**
+   ```bash
+   python main.py
+   ```
+
+2. **Open your browser** to `http://localhost:7860`
+
+3. **Load a repository**
+   - Enter a GitHub URL (e.g., `https://github.com/pallets/flask`)
+   - Or upload a ZIP file of your code
+   - Click "📁 Load Repository"
+
+4. **Process the repository**
+   - Click "🚀 Process Repository" to analyze and chunk your code
+   - This creates hierarchical chunks and stores them in Pinecone with automatic embedding generation
+   - Wait for processing to complete (may take 1-5 minutes depending on repo size)
+
+5. **Initialize the AI model** (Optional but recommended)
+   - Click "🚀 Initialize LLM" to start loading the local AI model
+   - This will load Qwen2.5-Coder-7B-Instruct for intelligent code analysis
+   - Initial loading takes 1-3 minutes
+
+6. **Query your code**
+   - Ask natural language questions like:
+     - "What does this repository do?"
+     - "Show me authentication functions"
+     - "How is error handling implemented?"
+     - "What are the main classes?"
+   - Toggle "Use AI Analysis" for intelligent responses vs basic search results
+   - The AI maintains conversation context for follow-up questions
+
+## 📊 How It Works
+
+### Hierarchical Chunking Strategy
+
+The system creates multiple levels of code chunks:
+
+**Level 1: File Context**
+- Complete file overview with imports and purpose
+- Metadata: file path, language, total lines
+
+**Level 2: Class Chunks** 
+- Full class definitions with inheritance and methods
+- Metadata: class name, methods list, relationships
+
+**Level 3: Function Chunks**
+- Individual function implementations with signatures
+- Metadata: function name, arguments, complexity score
+
+**Level 4: Code Block Chunks**
+- Sub-chunks for complex functions (loops, conditionals, error handling)
+- Metadata: block type, purpose, parent function
+
+### Vector Search Process
+
+1. **Embedding Generation**: Code chunks are converted to vector embeddings using SentenceTransformers
+2. **Vector Storage**: Embeddings stored in Pinecone with rich metadata
+3. **Semantic Search**: User queries are embedded and matched against stored vectors
+4. **Hybrid Filtering**: Results filtered by chunk type, file path, repository, etc.
+5. **Ranked Results**: Most relevant code sections returned with similarity scores
+
+## 🔧 Configuration Options
+
+### Supported Languages
+
+Currently optimized for Python with basic support for:
+- JavaScript/TypeScript
+- Java
+- C/C++
+- Go
+- Rust
+- PHP
+- Ruby
+
+## 📝 Example Repositories
+
+Try these public repositories:
+
+- **Flask**: `https://github.com/pallets/flask` - Web framework
+- **Requests**: `https://github.com/requests/requests` - HTTP library  
+- **FastAPI**: `https://github.com/tiangolo/fastapi` - Modern web framework
+- **Black**: `https://github.com/psf/black` - Code formatter
+
+## 🔍 Example Queries
+
+### General Repository Understanding
+- "What is the main purpose of this repository?"
+- "What are the core components and how do they interact?"
+- "Show me the project architecture overview"
+
+### Function & Class Discovery
+- "What are the main classes and their responsibilities?"
+- "Show me all authentication-related functions"
+- "Find functions that handle file operations"
+- "What utility functions are available?"
+
+### Implementation Analysis  
+- "How is error handling implemented?"
+- "Show me configuration management code"
+- "Find database-related functions"
+- "How does logging work in this project?"
+
+### Code Patterns
+- "Show me decorator implementations"
+- "Find async/await usage patterns"
+- "What design patterns are used?"
+- "How are tests structured?"
+
+## 🛟 Troubleshooting
+
+### Common Issues
+
+**"Pinecone API key is required"**
+- Make sure you've set the `PINECONE_API_KEY` environment variable
+- Or enter it in the Advanced Options section
+
+**"Error downloading repository"**
+- Check that the GitHub URL is correct and public
+- Ensure you have internet connection
+- Large repositories may timeout - try smaller repos first
+
+**"No chunks generated"**
+- Make sure the repository contains supported code files
+- Check that ZIP files aren't corrupted
+- Python files work best currently
+
+**"Vector store initialization failed"**
+- Verify your Pinecone API key is valid
+- Check your Pinecone account hasn't exceeded free tier limits
+- Try a different environment region if needed
+
+### Performance Tips
+
+- Start with smaller repositories (< 100 files) to test
+- Python repositories work best currently
+- Processing time scales with repository size
+- Queries are fast once processing is complete
+
+## 🔮 Future Enhancements
+
+- **More Language Support**: Better parsing for JavaScript, Java, etc.
+- **Code Generation**: AI-powered code completion and generation
+- **Diff Analysis**: Compare changes between repository versions
+- **Team Collaboration**: Share analyzed repositories
+- **Custom Embeddings**: Fine-tuned models for specific domains
+- **API Integration**: REST API for programmatic access
+
+## 🤝 Contributing
+
+Contributions welcome! Please open issues or submit pull requests.
+
+## 📞 Support
+
+For issues or questions:
+1. Check the troubleshooting section above
+2. Open a GitHub issue with detailed error messages
+3. Include your Python version and OS information

app.py

@@ -1,70 +1,462 @@
 import gradio as gr
-from huggingface_hub import InferenceClient
+import logging
+from datetime import datetime
+from pathlib import Path
+from scripts.RepositoryHandler import RepositoryHandler
+import os
 
+os.environ["CUDA_VISIBLE_DEVICES"] = "7"
 
-def respond(
-    message,
-    history: list[dict[str, str]],
-    system_message,
-    max_tokens,
-    temperature,
-    top_p,
-    hf_token: gr.OAuthToken,
-):
-    """
-    For more information on `huggingface_hub` Inference API support, please check the docs: https://huggingface.co/docs/huggingface_hub/v0.22.2/en/guides/inference
-    """
-    client = InferenceClient(token=hf_token.token, model="openai/gpt-oss-20b")
+# --- Setup Logging ---
+def setup_logger():
+    log_dir = Path("/data/home/sqamar/code-compass/logs")
+    log_dir.mkdir(parents=True, exist_ok=True)
+    timestamp = datetime.now().strftime("%Y%m%d_%H%M")
+    log_file = log_dir / f"{timestamp}_code_compass.log"
 
-    messages = [{"role": "system", "content": system_message}]
+    logger = logging.getLogger("code_compass")
+    logger.setLevel(logging.DEBUG)
 
-    messages.extend(history)
+    # Console handler
+    ch = logging.StreamHandler()
+    ch.setLevel(logging.INFO)
 
-    messages.append({"role": "user", "content": message})
+    # File handler
+    fh = logging.FileHandler(log_file)
+    fh.setLevel(logging.DEBUG)
 
-    response = ""
+    # Formatter
+    formatter = logging.Formatter("%(asctime)s [%(levelname)s] %(message)s")
+    ch.setFormatter(formatter)
+    fh.setFormatter(formatter)
 
-    for message in client.chat_completion(
-        messages,
-        max_tokens=max_tokens,
-        stream=True,
-        temperature=temperature,
-        top_p=top_p,
-    ):
-        choices = message.choices
-        token = ""
-        if len(choices) and choices[0].delta.content:
-            token = choices[0].delta.content
+    logger.addHandler(ch)
+    logger.addHandler(fh)
+    return logger
 
-        response += token
-        yield response
+setup_logger()
+logger = logging.getLogger("code_compass")
+# Global repository handler instance
+repo_handler = RepositoryHandler()
+if not os.path.exists("models/Qwen2.5-Coder-7B-Instruct-Q4_K_M.gguf"):
+    os.system("wget -q https://huggingface.co/bartowski/Qwen2.5-Coder-7B-Instruct-GGUF/resolve/main/Qwen2.5-Coder-7B-Instruct-Q4_K_M.gguf -O models/Qwen2.5-Coder-7B-Instruct-Q4_K_M.gguf")
 
 
+def process_repository(input_type, github_url, zip_file):
+    """Process repository based on input type"""
+    
+    # Clean up any previous repository
+    repo_handler.cleanup()
+    
+    if input_type == "GitHub URL":
+        if not github_url or not github_url.strip():
+            return "❌ Please enter a GitHub repository URL", "", "disabled", "disabled"
+        
+        if not repo_handler.validate_github_url(github_url.strip()):
+            return "❌ Invalid GitHub URL format. Please use: https://github.com/username/repository", "", "disabled", "disabled"
+        
+        success, message = repo_handler.download_github_repo(github_url.strip())
+        
+    else:  # ZIP File
+        if zip_file is None:
+            return "❌ Please upload a ZIP file", "", "disabled", "disabled"
+        
+        is_valid, validation_msg = repo_handler.validate_zip_file(zip_file)
+        if not is_valid:
+            return f"❌ {validation_msg}", "", "disabled", "disabled"
+        
+        success, message = repo_handler.extract_zip_file(zip_file)
+    
+    if success:
+        structure = repo_handler.get_repo_structure()
+        return message, structure, "🚀 Process Repository", "disabled"  # Enable process button, keep query disabled
+    else:
+        return message, "", "disabled", "disabled"
+
+def process_chunks():
+    """Process repository into chunks and store in vector database"""
+    if not repo_handler.is_loaded:
+        return "❌ No repository loaded", "disabled"
+    
+    # Run processing in background thread to avoid blocking UI
+    def background_processing():
+        return repo_handler.process_and_store_chunks()
+    
+    try:
+        success, message = background_processing()
+        if success:
+            return message, "Ask AI"  # Enable query functionality
+        else:
+            return message, "disabled"
+    except Exception as e:
+        return f"❌ Error processing chunks: {str(e)}", "disabled"
+
+def handle_query(query):
+    """Handle user queries about the repository"""
+    if not repo_handler.is_loaded:
+        return "❌ No repository loaded. Please load a repository first."
+    
+    if not repo_handler.chunks:
+        return "❌ Repository not processed yet. Please click 'Process Repository' first."
+    
+    if not query or not query.strip():
+        return "Please enter a query about the repository."
+    
+    return repo_handler.query_repository(query.strip())
+
+def get_repo_stats():
+    """Get repository statistics for display"""
+    if not repo_handler.is_loaded:
+        return "No repository loaded"
+    
+    if repo_handler.vector_store and repo_handler.chunks:
+        try:
+            # Get repository overview from vector store
+            overview = repo_handler.vector_store.get_repository_overview(repo_handler.repo_name)
+            logger.debug(f"Repository overview: {overview}")
+            if "error" not in overview:
+                stats = f"""📊 **Repository Statistics**
+
+🏷️ **Repository:** {overview['repo_name']}
+📦 **Total Chunks:** {overview['total_chunks']}
+📁 **Files:** {overview['files_count']}
+🏛️ **Classes:** {overview['classes_count']}  
+⚙️ **Functions:** {overview['functions_count']}
+💻 **Languages:** {', '.join(overview['languages'])}
+
+📋 **Chunk Distribution:**
 """
-For information on how to customize the ChatInterface, peruse the gradio docs: https://www.gradio.app/docs/chatinterface
-"""
-chatbot = gr.ChatInterface(
-    respond,
-    type="messages",
-    additional_inputs=[
-        gr.Textbox(value="You are a friendly Chatbot.", label="System message"),
-        gr.Slider(minimum=1, maximum=2048, value=512, step=1, label="Max new tokens"),
-        gr.Slider(minimum=0.1, maximum=4.0, value=0.7, step=0.1, label="Temperature"),
-        gr.Slider(
-            minimum=0.1,
-            maximum=1.0,
-            value=0.95,
-            step=0.05,
-            label="Top-p (nucleus sampling)",
-        ),
-    ],
-)
-
-with gr.Blocks() as demo:
-    with gr.Sidebar():
-        gr.LoginButton()
-    chatbot.render()
+                for chunk_type, count in overview['chunk_distribution'].items():
+                    stats += f"- {chunk_type.title()}: {count}\n"
+                
+                return stats
+            else:
+                return f"Error getting stats: {overview['error']}"
+        except Exception as e:
+            return f"Error getting repository stats: {str(e)}"
+    
+    return "Repository loaded but not processed yet"
+# Additional handler functions for LLM integration
+def initialize_llm():
+    """Initialize LLM model loading"""
+    return repo_handler.initialize_llm()
+
+def handle_query_with_llm(query, use_llm):
+    """Handle user queries with optional LLM processing"""
+    if not repo_handler.is_loaded:
+        return "❌ No repository loaded. Please load a repository first."
+    
+    if not repo_handler.chunks:
+        return "❌ Repository not processed yet. Please click 'Process Repository' first."
+    
+    if not query or not query.strip():
+        return "Please enter a query about the repository."
+    
+    return repo_handler.query_repository(query.strip(), use_llm=use_llm)
+
+def clear_conversation():
+    """Clear LLM conversation history"""
+    if repo_handler.llm:
+        repo_handler.llm.clear_conversation()
+        return "🗑️ Conversation history cleared!"
+    return "❌ LLM not initialized"
+
+def export_conversation():
+    """Export conversation history"""
+    if repo_handler.llm and repo_handler.llm.is_model_ready():
+        conversation = repo_handler.llm.export_conversation()
+        if conversation:
+            # Format for display
+            export_text = "# Conversation Export\n\n"
+            for msg in conversation:
+                role_emoji = {"system": "⚙️", "user": "👤", "assistant": "🤖"}.get(msg["role"], "💬")
+                export_text += f"## {role_emoji} {msg['role'].title()}\n"
+                export_text += f"**Time:** {msg['timestamp']}\n\n"
+                export_text += f"{msg['content']}\n\n---\n\n"
+            return export_text
+        else:
+            return "No conversation to export"
+    return "❌ LLM not ready or no conversation history"
+
+def get_llm_status():
+    """Get current LLM status"""
+    if not repo_handler.llm_loading_started:
+        return "🔄 LLM not initialized"
+    elif repo_handler.llm.is_model_ready():
+        model_info = repo_handler.llm.get_model_info()
+        conversation_summary = repo_handler.llm.get_conversation_summary()
+        return f"""✅ **LLM Ready!**
+        
+**Model:** Qwen2.5-Coder-7B-Instruct (Q4_K_M)
+**Context Window:** {model_info['context_window']} tokens
+**Temperature:** {model_info['temperature']}
+**Status:** {conversation_summary}
+
+🤖 Ready for intelligent code analysis!"""
+    else:
+        return "⏳ **LLM Loading...** Please wait for model initialization to complete."
 
+def create_interface():
+    """Create the Gradio interface"""
+    
+    with gr.Blocks(title="Code Compass", theme=gr.themes.Soft()) as demo:
+        
+        gr.Markdown("""
+        # 🔍 Code Compass
+        
+        Upload your repository via GitHub URL or ZIP file, process it with AI-powered chunking, and query your codebase using semantic search!
+        """)
+        
+        with gr.Row():
+            with gr.Column(scale=2):
+                
+                # Input section
+                with gr.Group():
+                    gr.Markdown("### 📥 Repository Input")
+                    
+                    input_type = gr.Dropdown(
+                        choices=["GitHub URL", "ZIP File"], 
+                        value="GitHub URL",
+                        label="Input Method",
+                        info="Choose how you want to provide your repository"
+                    )
+                    
+                    github_url = gr.Textbox(
+                        label="GitHub Repository URL",
+                        placeholder="https://github.com/username/repository",
+                        visible=True
+                    )
+                    
+                    zip_file = gr.File(
+                        label="Upload ZIP File",
+                        file_types=[".zip"],
+                        visible=False
+                    )
+                    
+                    load_btn = gr.Button("📁 Load Repository", variant="primary")
+                
+                # Processing section
+                with gr.Group():
+                    gr.Markdown("### ⚙️ Repository Processing")
+                    gr.Markdown("After loading, process your repository to enable AI-powered search")
+                    
+                    process_btn = gr.Button("🚀 Process Repository", interactive=False, variant="secondary")
+                    
+                # Status section
+                with gr.Group():
+                    gr.Markdown("### 📊 Status")
+                    status_output = gr.Textbox(
+                        label="Status",
+                        placeholder="Ready to load repository...",
+                        interactive=False,
+                        lines=3
+                    )
+            
+            with gr.Column(scale=1):
+                with gr.Group():
+                    gr.Markdown("### 📁 Repository Structure")
+                    structure_output = gr.Code(
+                        label="Directory Structure",
+                        # language="text",
+                        interactive=False,
+                        lines=10
+                    )
+                
+                with gr.Group():
+                    gr.Markdown("### 📊 Repository Stats")
+                    stats_output = gr.Markdown(
+                        value="Load and process a repository to see statistics"
+                    )
+                with gr.Group():
+                    gr.Markdown("### 🤖 LLM Status")
+                    llm_status = gr.Markdown(
+                        value="🔄 LLM not initialized"
+                    )
+                    init_llm_btn = gr.Button("🚀 Initialize LLM", variant="secondary")
+        # Query section
+        with gr.Row():
+            with gr.Column():
+                gr.Markdown("### 💬 Query Repository")
+                gr.Markdown("Ask questions about your code using natural language. The AI will search through your processed code chunks to find relevant information.")
+                
+                with gr.Row():
+                    query_input = gr.Textbox(
+                        label="Ask about your code",
+                        placeholder="e.g., 'What does this repository do?', 'Show me authentication functions', 'How is error handling implemented?'",
+                        lines=2,
+                        scale=4
+                    )
+                    query_btn = gr.Button("🔍 Ask Question", interactive=False, scale=1)
+                    use_llm_toggle = gr.Checkbox(
+                            label="Use AI Analysis",
+                            value=True,
+                            info="Get intelligent responses using LLM"
+                        )
+                    # Conversation controls
+                with gr.Row():
+                    clear_chat_btn = gr.Button("🗑️ Clear Chat History", variant="secondary", interactive=False)
+                    export_chat_btn = gr.Button("📥 Export Chat", variant="secondary", interactive=False)
+                query_output = gr.Markdown(
+                    value="Load and process a repository first to start querying...",
+                    height=400
+                )
+        
+        # Advanced options (collapsible)
+        # with gr.Accordion("🛠️ Advanced Options", open=False):
+        #     with gr.Row():
+        #         with gr.Column():
+        #             gr.Markdown("### 🔧 Pinecone Configuration")
+        #             api_key_input = gr.Textbox(
+        #                 label="Pinecone API Key",
+        #                 placeholder="Enter your Pinecone API key (or set PINECONE_API_KEY env var)",
+        #                 type="password"
+        #             )
+        #             environment_input = gr.Textbox(
+        #                 label="Pinecone Environment",
+        #                 value="us-west1-gcp-free",
+        #                 placeholder="e.g., us-west1-gcp-free"
+        #             )
+                
+        #         with gr.Column():
+        #             gr.Markdown("### 📈 Processing Options")
+        #             complexity_threshold = gr.Slider(
+        #                 minimum=5,
+        #                 maximum=50,
+        #                 value=20,
+        #                 step=5,
+        #                 label="Complexity Threshold",
+        #                 info="Functions above this complexity will be sub-chunked"
+        #             )
+        
+        # Event handlers
+        def toggle_inputs(choice):
+            return (
+                gr.update(visible=(choice == "GitHub URL")),
+                gr.update(visible=(choice == "ZIP File"))
+            )
+        
+        def update_buttons_after_load(status_text):
+            # Enable process button if repository is successfully loaded
+            is_loaded = "✅" in status_text and "successfully" in status_text.lower()
+            return gr.update(interactive=is_loaded)
+        
+        def update_query_button_after_process(status_text):
+            # Enable query button if processing is successful
+            is_processed = "✅" in status_text and "complete" in status_text.lower()
+            return gr.update(interactive=is_processed)
+        
+        def update_buttons_after_process(status_text):
+            # Enable query button if processing is successful
+            is_processed = "✅" in status_text and "complete" in status_text.lower()
+            return (
+                gr.update(interactive=is_processed),  # query_btn
+                gr.update(interactive=is_processed),  # clear_chat_btn  
+                gr.update(interactive=is_processed)   # export_chat_btn
+            )
+        
+        def update_llm_status():
+            return get_llm_status()
+        
+        def update_stats(status_output):
+            return get_repo_stats(), update_buttons_after_load(status_output), update_query_button_after_process(status_output)
+        
+        # Wire up the interface
+        input_type.change(
+            fn=toggle_inputs,
+            inputs=[input_type],
+            outputs=[github_url, zip_file]
+        )
+        
+        load_btn.click(
+            fn=process_repository,
+            inputs=[input_type, github_url, zip_file],
+            outputs=[status_output, structure_output, process_btn, query_btn]
+        ).then(
+            fn=update_stats,
+            inputs=[status_output],
+            outputs=[stats_output, process_btn, query_btn]
+        )
+        
+        process_btn.click(
+            fn=process_chunks,
+            outputs=[status_output, query_btn]
+        ).then(
+            fn=update_stats,
+            inputs=[status_output],
+            outputs=[stats_output, process_btn, query_btn]
+        )
+        
+        # Query handling
+        query_btn.click(
+            fn=handle_query_with_llm,
+            inputs=[query_input, use_llm_toggle],
+            outputs=[query_output]
+        ).then(
+            fn=update_llm_status,
+            outputs=[llm_status]
+        )
+        
+        # Chat management
+        clear_chat_btn.click(
+            fn=clear_conversation,
+            outputs=[query_output]
+        ).then(
+            fn=update_llm_status,
+            outputs=[llm_status]
+        )
+        
+        # Allow Enter key to submit query
+        query_input.submit(
+            fn=handle_query_with_llm,
+            inputs=[query_input, use_llm_toggle],
+            outputs=[query_output]
+        )
+         # LLM initialization
+        init_llm_btn.click(
+            fn=initialize_llm,
+            outputs=[llm_status]
+        ).then(
+            fn=update_llm_status,
+            outputs=[llm_status]
+        )
+        # Add some helpful examples
+        gr.Markdown("""
+        ### 📝 Example Repositories to Try:
+        - `https://github.com/pallets/flask` - Popular Python web framework
+        - `https://github.com/requests/requests` - HTTP library for Python
+        - `https://github.com/fastapi/fastapi` - Modern Python web framework
+        - `https://github.com/psf/black` - Python code formatter
+        
+        ### 💡 Example Queries:
+        - "What is the main purpose of this repository?"
+        - "Show me all the authentication functions"
+        - "How is error handling implemented?"
+        - "What are the main classes and their responsibilities?"  
+        - "Find functions that handle file operations"
+        - "Show me the configuration management code"
+        
+        ### ⚙️ Setup Requirements:
+        1. **Pinecone API Key**: Get a free API key from [Pinecone.io](https://www.pinecone.io/)
+        2. **Environment Variables**: Set `PINECONE_API_KEY` in your environment or enter it in Advanced Options
+        3. **Internet Connection**: Required for downloading repositories and accessing Pinecone
+        
+        ### 🚀 How It Works:
+        1. **Load**: Repository is downloaded/extracted and validated
+        2. **Process**: Code is analyzed and split into hierarchical chunks (file → class → function → block)
+        3. **Store**: Chunks are embedded using AI and stored in Pinecone vector database  
+        4. **Query**: Your questions are semantically matched against stored code chunks
+        """)
+    
+    return demo
 
 if __name__ == "__main__":
-    demo.launch()
+    # Create and launch the interface
+    demo = create_interface()
+    
+    # Launch with some nice settings
+    demo.launch(
+        server_name="0.0.0.0",  # Allow external access
+        server_port=7860,       # Standard port
+        share=False,            # Set to True to create public link
+        debug=True              # Enable debug mode for development
+    )

requirements.txt

@@ -0,0 +1,6 @@
+gradio>=4.0.0
+requests>=2.31.0
+pathlib2>=2.3.7
+pinecone>=3.0.0
+numpy>=1.21.0
+llama-cpp-python>=0.2.20

scripts/RepositoryHandler.py

@@ -0,0 +1,503 @@
+import gradio as gr
+import os
+import zipfile
+import tempfile
+import shutil
+import requests
+import re
+from pathlib import Path
+from urllib.parse import urlparse
+import subprocess
+import threading
+import time
+import logging
+# Import our custom modules
+from .chunker import HierarchicalChunker
+from .vectorstore import PineconeVectorStore
+from .llm_service import QwenCoderLLM
+from config import MODEL_PATH
+from typing import List, Dict, Any
+logger = logging.getLogger("code_compass")
+
+class RepositoryHandler:
+    def __init__(self):
+        self.temp_dir = None
+        self.repo_path = None
+        self.is_loaded = False
+        self.repo_name = None
+        self.chunks = []
+        
+        # Initialize chunker and vector store
+        self.chunker = HierarchicalChunker()
+        self.vector_store = None  # Will be initialized when needed
+        self.processing_status = {"status": "idle", "progress": 0, "message": ""}
+        
+        # Initialize LLM service
+        self.llm = QwenCoderLLM(model_path=MODEL_PATH, n_gpu_layers=-1)  # Adjust n_gpu_layers based on your GPU memory
+        self.llm_loading_started = False
+    
+    def validate_github_url(self, url):
+        """Validate if URL is a proper GitHub repository URL"""
+        github_pattern = r'https://github\.com/[\w\-\.]+/[\w\-\.]+'
+        return bool(re.match(github_pattern, url))
+    
+    def validate_zip_file(self, zip_file):
+        """Validate if uploaded file is a proper zip file"""
+        if zip_file is None:
+            return False, "No file uploaded"
+        
+        try:
+            # Check if file exists and has .zip extension
+            if not zip_file.name.lower().endswith('.zip'):
+                return False, "File must be a .zip file"
+            
+            # Try to open and validate the zip file
+            with zipfile.ZipFile(zip_file.name, 'r') as zip_ref:
+                # Test if zip file is valid
+                zip_ref.testzip()
+                
+                # Check if it contains at least one file
+                file_list = zip_ref.namelist()
+                if not file_list:
+                    return False, "Zip file is empty"
+                
+                # Check if it looks like a code repository
+                code_extensions = ['.py', '.js', '.java', '.cpp', '.c', '.go', '.rs', '.php', '.rb', '.ts']
+                has_code_files = any(
+                    any(fname.endswith(ext) for ext in code_extensions) 
+                    for fname in file_list
+                )
+                
+                if not has_code_files:
+                    return False, "Zip file doesn't appear to contain code files"
+                
+                return True, f"Valid zip file with {len(file_list)} files"
+                
+        except zipfile.BadZipFile:
+            return False, "Invalid or corrupted zip file"
+        except Exception as e:
+            return False, f"Error validating zip file: {str(e)}"
+    
+    def download_github_repo(self, github_url):
+        """Download GitHub repository using git clone"""
+        try:
+            # Create temporary directory
+            self.temp_dir = tempfile.mkdtemp(prefix="repo_")
+            
+            # Extract repo name for folder
+            self.repo_name = github_url.split('/')[-1].replace('.git', '')
+            self.repo_path = os.path.join(self.temp_dir, self.repo_name)
+            
+            # Clone the repository
+            result = subprocess.run([
+                'git', 'clone', github_url, self.repo_path
+            ], capture_output=True, text=True, timeout=300)
+            
+            if result.returncode != 0:
+                # If git clone fails, try downloading as zip
+                return self._download_repo_as_zip(github_url)
+            
+            # Count files in repository
+            total_files = sum(1 for _ in Path(self.repo_path).rglob('*') if _.is_file())
+            
+            self.is_loaded = True
+            return True, f"✅ Repository successfully cloned! Found {total_files} files in {self.repo_name}"
+            
+        except subprocess.TimeoutExpired:
+            return False, "❌ Download timeout - repository might be too large"
+        except FileNotFoundError:
+            # Git not installed, fallback to zip download
+            return self._download_repo_as_zip(github_url)
+        except Exception as e:
+            return False, f"❌ Error downloading repository: {str(e)}"
+    
+    def _download_repo_as_zip(self, github_url):
+        """Fallback method to download repo as zip if git is not available"""
+        try:
+            # Convert GitHub URL to zip download URL
+            zip_url = github_url.rstrip('/') + '/archive/refs/heads/main.zip'
+            
+            # Try main branch, if fails try master
+            for branch in ['main', 'master']:
+                try:
+                    zip_url = github_url.rstrip('/') + f'/archive/refs/heads/{branch}.zip'
+                    response = requests.get(zip_url, timeout=60)
+                    response.raise_for_status()
+                    break
+                except:
+                    continue
+            else:
+                return False, "❌ Could not download repository - check if it's public and accessible"
+            
+            # Create temp directory and save zip
+            self.temp_dir = tempfile.mkdtemp(prefix="repo_")
+            zip_path = os.path.join(self.temp_dir, "repo.zip")
+            
+            with open(zip_path, 'wb') as f:
+                f.write(response.content)
+            
+            # Extract zip
+            with zipfile.ZipFile(zip_path, 'r') as zip_ref:
+                zip_ref.extractall(self.temp_dir)
+            
+            # Find the extracted folder (usually repo-name-branch)
+            extracted_folders = [d for d in os.listdir(self.temp_dir) 
+                               if os.path.isdir(os.path.join(self.temp_dir, d))]
+            
+            if extracted_folders:
+                self.repo_path = os.path.join(self.temp_dir, extracted_folders[0])
+                total_files = sum(1 for _ in Path(self.repo_path).rglob('*') if _.is_file())
+                self.is_loaded = True
+                return True, f"✅ Repository successfully downloaded! Found {total_files} files"
+            else:
+                return False, "❌ Error extracting downloaded repository"
+                
+        except requests.RequestException as e:
+            return False, f"❌ Network error downloading repository: {str(e)}"
+        except Exception as e:
+            return False, f"❌ Error downloading repository: {str(e)}"
+    
+    def extract_zip_file(self, zip_file):
+        """Extract uploaded zip file"""
+        try:
+            # Create temporary directory
+            self.temp_dir = tempfile.mkdtemp(prefix="repo_")
+            
+            # Extract zip file
+            with zipfile.ZipFile(zip_file.name, 'r') as zip_ref:
+                zip_ref.extractall(self.temp_dir)
+            
+            # Find the main folder or use temp_dir if files are in root
+            extracted_items = os.listdir(self.temp_dir)
+            
+            # If there's only one folder, use it as repo_path
+            if len(extracted_items) == 1 and os.path.isdir(os.path.join(self.temp_dir, extracted_items[0])):
+                self.repo_path = os.path.join(self.temp_dir, extracted_items[0])
+                self.repo_name = os.path.basename(self.repo_path)
+            else:
+                # Files are in root of zip
+                self.repo_path = self.temp_dir
+            
+            # Count files
+            total_files = sum(1 for _ in Path(self.repo_path).rglob('*') if _.is_file())
+            
+            self.is_loaded = True
+            return True, f"✅ Zip file successfully extracted! Found {total_files} files"
+            
+        except Exception as e:
+            return False, f"❌ Error extracting zip file: {str(e)}"
+    
+    def initialize_vector_store(self, namespace):
+        """Initialize Pinecone vector store"""
+        try:
+            if self.vector_store is None:
+                print("🔄 Initializing vector store...")
+                self.vector_store = PineconeVectorStore(namespace=namespace)
+                print("✅ Vector store initialized!")
+            return True, "Vector store ready"
+        except Exception as e:
+            error_msg = f"❌ Error initializing vector store: {str(e)}"
+            print(error_msg)
+            return False, error_msg
+    
+    def process_and_store_chunks(self):
+        """Process repository into chunks and store in vector database"""
+        if not self.is_loaded or not self.repo_path:
+            return False, "❌ No repository loaded"
+        
+        try:
+            self.processing_status = {"status": "chunking", "progress": 10, "message": "Creating hierarchical chunks..."}
+            namespace = self.repo_name + "_namespace"
+            # Step 1: Create chunks
+            logger.info(f"🔄 Creating chunks for {self.repo_name}...")
+            self.chunks = self.chunker.chunk_repository(self.repo_path)
+            
+            if not self.chunks:
+                return False, "❌ No chunks generated from repository"
+            
+            # self.processing_status = {"status": "embedding", "progress": 40, "message": f"Generating embeddings for {len(self.chunks)} chunks..."}
+            
+            # Step 2: Initialize vector store
+            success, message = self.initialize_vector_store(namespace=namespace)
+            if not success:
+                return False, message
+            
+            # Step 3: Generate embeddings
+            # print("🔄 Generating embeddings...")
+            # self.chunks =  self.vector_store.generate_embeddings(self.chunks)
+            
+            self.processing_status = {"status": "storing", "progress": 70, "message": "Storing chunks in vector database..."}
+            
+            # Step 4: Store in Pinecone
+            logger.info("🔄 Storing chunks in vector database...")
+            result = self.vector_store.upsert_chunks(self.chunks)
+            
+            self.processing_status = {"status": "complete", "progress": 100, "message": "Processing complete!"}
+            
+            if result['status'] == 'success':
+                summary = f"""✅ Repository processing complete!
+                
+📊 **Processing Summary:**
+- Repository: {self.repo_name}
+- Total chunks created: {len(self.chunks)}
+- Successfully stored: {result['successful_upserts']}
+- Failed: {result['failed_upserts']}
+
+📁 **Chunk Distribution:**"""
+                
+                # Add chunk type distribution
+                chunk_types = {}
+                for chunk in self.chunks:
+                    chunk_type = chunk.chunk_type
+                    chunk_types[chunk_type] = chunk_types.get(chunk_type, 0) + 1
+                
+                for chunk_type, count in chunk_types.items():
+                    summary += f"\n- {chunk_type.title()}: {count}"
+                
+                summary += f"\n\n🔍 **Ready for queries!** You can now ask questions about your code."
+                
+                return True, summary
+            else:
+                return False, f"❌ Error storing chunks: {result.get('message', 'Unknown error')}"
+                
+        except Exception as e:
+            self.processing_status = {"status": "error", "progress": 0, "message": f"Error: {str(e)}"}
+            return False, f"❌ Error processing repository: {str(e)}"
+    
+    def query_repository(self, query_text, search_type="hybrid",use_llm=True):
+        """Query the repository using vector search"""
+        if not self.vector_store or not self.chunks:
+            return "❌ Repository not processed yet. Please load and process a repository first."
+        
+        if not query_text or not query_text.strip():
+            return "Please enter a query about the repository."
+        
+        try:
+            logger.info(f"🔍 Querying repository: {query_text}")
+            
+            # Perform hybrid search
+            results = self.vector_store.hybrid_search(
+                query_text=query_text.strip(),
+                repo_names=[self.repo_name],
+                top_k=10
+            )
+            
+            if not results:
+                return f"""🤖 No relevant results found for: "{query_text}"
+
+Try rephrasing your question or asking about:
+- Specific functions or classes
+- Code patterns or algorithms  
+- File structure or organization
+- Dependencies or imports"""
+            # Step 2: Use LLM for intelligent response if enabled and ready
+            if use_llm:
+                if not self.llm_loading_started:
+                    self.initialize_llm()
+                
+                if self.llm.is_model_ready():
+                    # Generate intelligent response using LLM
+                    llm_response = self.llm.generate_response(
+                        user_query=query_text.strip(),
+                        retrieved_chunks=results,
+                        use_history=True
+                    )
+                    
+                    if llm_response["status"] == "success":
+                        response = f"""🤖 **AI Analysis:**
+
+        {llm_response["response"]}
+
+        ---
+        📊 **Query Details:**
+        - Found {len(results)} relevant code sections
+        - Response generated in {llm_response["metadata"]["generation_time"]:.2f}s
+        - Conversation length: {llm_response["metadata"]["conversation_length"]} messages
+        """
+                        return response
+                    else:
+                        # Fall back to basic response if LLM fails
+                        return self._generate_basic_response(query_text, results) + f"\n\n⚠️ LLM Error: {llm_response.get('message', 'Unknown error')}"
+                else:
+                    # LLM not ready, provide basic response with loading status
+                    basic_response = self._generate_basic_response(query_text, results)
+                    return basic_response + "\n\n⏳ **Note:** AI model is still loading. You'll get smarter responses once it's ready!"
+            else:
+                # Basic response without LLM
+                return self._generate_basic_response(query_text, results)
+            
+        except Exception as e:
+            return f"❌ Error querying repository: {str(e)}"
+        # Format response
+#         response = f"""🔍 **Query Results for:** "{query_text}"
+
+#         📊 **Found {len(results)} relevant code sections:**
+
+#         """
+            
+#             for i, result in enumerate(results[:5], 1):  # Show top 5 results
+#                 metadata = result.get('metadata', {})
+#                 score = result.get('score', 0)
+                
+#                 chunk_type = metadata.get('chunk_type', 'unknown')
+#                 file_path = metadata.get('file_path', 'unknown')
+                
+#                 response += f"""**{i}. {chunk_type.title()} Match** (Similarity: {score:.2f})
+# 📄 File: `{file_path}`
+# """
+                
+#                 if chunk_type == 'function':
+#                     func_name = metadata.get('function_name', 'unknown')
+#                     class_name = metadata.get('class_name')
+#                     signature = metadata.get('signature', func_name)
+                    
+#                     response += f"🔧 Function: `{signature}`\n"
+#                     if class_name:
+#                         response += f"📦 Class: `{class_name}`\n"
+                        
+#                 elif chunk_type == 'class':
+#                     class_name = metadata.get('class_name', 'unknown')
+#                     methods = metadata.get('methods', [])
+#                     response += f"📦 Class: `{class_name}`\n"
+#                     if methods:
+#                         response += f"🔧 Methods: {', '.join(methods[:5])}\n"
+                        
+#                 elif chunk_type == 'file':
+#                     language = metadata.get('language', 'unknown')
+#                     total_lines = metadata.get('total_lines', 'unknown')
+#                     response += f"📝 Language: {language}, Lines: {total_lines}\n"
+                
+#                 response += "---\n\n"
+            
+#             # Add repository overview
+#             if len(results) > 5:
+#                 response += f"... and {len(results) - 5} more results available.\n\n"
+                
+#             response += f"""💡 **Suggestions:**
+# - Ask more specific questions about functions or classes
+# - Query about code patterns: "Show me error handling code"  
+# - Ask about structure: "What are the main components?"
+# - Request examples: "How is authentication implemented?"
+# """
+            
+#             return response
+            
+#         except Exception as e:
+#             return f"❌ Error querying repository: {str(e)}"
+    
+    def get_processing_status(self):
+        """Get current processing status"""
+        return self.processing_status
+    
+    def get_repo_structure(self):
+        """Get basic repository structure for display"""
+        if not self.is_loaded or not self.repo_path:
+            return "No repository loaded"
+        
+        try:
+            structure = []
+            for root, dirs, files in os.walk(self.repo_path):
+                # Skip hidden directories and common non-code directories
+                dirs[:] = [d for d in dirs if not d.startswith('.') and d not in ['node_modules', '__pycache__', 'venv', 'env']]
+                
+                level = root.replace(self.repo_path, '').count(os.sep)
+                indent = '  ' * level
+                structure.append(f"{indent}{os.path.basename(root)}/")
+                
+                # Limit files shown per directory
+                subindent = '  ' * (level + 1)
+                for file in files[:10]:  # Show max 10 files per directory
+                    if not file.startswith('.'):
+                        structure.append(f"{subindent}{file}")
+                
+                if len(files) > 10:
+                    structure.append(f"{subindent}... and {len(files) - 10} more files")
+                
+                # Limit depth to avoid too much output
+                if level > 3:
+                    dirs.clear()
+            
+            return '\n'.join(structure[:50])  # Limit total lines
+            
+        except Exception as e:
+            return f"Error reading repository structure: {str(e)}"
+    
+    def cleanup(self):
+        """Clean up temporary files"""
+        if self.temp_dir and os.path.exists(self.temp_dir):
+            try:
+                shutil.rmtree(self.temp_dir)
+                self.temp_dir = None
+                self.repo_path = None
+                self.is_loaded = False
+            except Exception as e:
+                print(f"Warning: Could not clean up temp directory: {e}")
+
+    def initialize_llm(self):
+        """Initialize LLM model loading"""
+        if not self.llm_loading_started:
+            print("🔄 Starting LLM model loading...")
+            self.llm.load_model_async()
+            self.llm_loading_started = True
+            return "🔄 LLM model loading started in background..."
+        elif self.llm.is_model_ready():
+            return "✅ LLM model is ready!"
+        else:
+            return "⏳ LLM model is still loading..."
+        
+    
+    
+    def _generate_basic_response(self, query_text: str, results: List[Dict[str, Any]]) -> str:
+        """Generate basic response without LLM"""
+        response = f"""🔍 **Search Results for:** "{query_text}"
+
+📊 **Found {len(results)} relevant code sections:**
+
+"""
+        
+        for i, result in enumerate(results[:5], 1):  # Show top 5 results
+            metadata = result.get('metadata', {})
+            score = result.get('score', 0)
+            
+            chunk_type = metadata.get('chunk_type', 'unknown')
+            file_path = metadata.get('file_path', 'unknown')
+            
+            response += f"""**{i}. {chunk_type.title()} Match** (Similarity: {score:.2f})
+📄 File: `{file_path}`
+"""
+            
+            if chunk_type == 'function':
+                func_name = metadata.get('function_name', 'unknown')
+                class_name = metadata.get('class_name')
+                signature = metadata.get('signature', func_name)
+                
+                response += f"🔧 Function: `{signature}`\n"
+                if class_name:
+                    response += f"📦 Class: `{class_name}`\n"
+                    
+            elif chunk_type == 'class':
+                class_name = metadata.get('class_name', 'unknown')
+                methods = metadata.get('methods', [])
+                response += f"📦 Class: `{class_name}`\n"
+                if methods:
+                    response += f"🔧 Methods: {', '.join(methods[:5])}\n"
+                    
+            elif chunk_type == 'file':
+                language = metadata.get('language', 'unknown')
+                total_lines = metadata.get('total_lines', 'unknown')
+                response += f"📝 Language: {language}, Lines: {total_lines}\n"
+            
+            response += "---\n\n"
+        
+        # Add suggestions
+        if len(results) > 5:
+            response += f"... and {len(results) - 5} more results available.\n\n"
+            
+        response += f"""💡 **Suggestions:**
+- Ask more specific questions about functions or classes
+- Query about code patterns: "Show me error handling code"  
+- Ask about structure: "What are the main components?"
+- Request examples: "How is authentication implemented?"
+"""
+        
+        return response

scripts/__init__.py

@@ -0,0 +1,4 @@
+from .RepositoryHandler import RepositoryHandler
+from .chunker import CodeChunk, HierarchicalChunker
+from .vectorstore import PineconeVectorStore
+from .llm_service import QwenCoderLLM

scripts/chunker.py

@@ -0,0 +1,578 @@
+import ast
+import os
+import hashlib
+from pathlib import Path
+from typing import List, Dict, Any, Optional, Tuple
+from dataclasses import dataclass
+from datetime import datetime
+import json
+import logging
+
+logger = logging.getLogger("code_compass")
+@dataclass
+class CodeChunk:
+    """Represents a hierarchical code chunk with metadata"""
+    id: str
+    content: str
+    chunk_type: str  # 'file', 'class', 'function', 'block'
+    metadata: Dict[str, Any]
+    embedding: Optional[List[float]] = None
+    
+    def to_dict(self) -> Dict[str, Any]:
+        """Convert chunk to dictionary for storage"""
+        return {
+            'id': self.id,
+            'content': self.content,
+            'chunk_type': self.chunk_type,
+            'metadata': self.metadata,
+            'embedding': self.embedding
+        }
+
+class HierarchicalChunker:
+    """
+    Advanced hierarchical code chunker that creates multiple levels of chunks:
+    Level 1: File-level context
+    Level 2: Class-level chunks  
+    Level 3: Function-level chunks
+    Level 4: Code block chunks (for complex functions)
+    """
+    
+    def __init__(self, complexity_threshold: int = 20):
+        self.complexity_threshold = complexity_threshold
+        self.supported_extensions = {
+            '.py': self._parse_python,
+            '.js': self._parse_javascript,
+            '.ts': self._parse_typescript,
+            '.java': self._parse_java,
+            '.cpp': self._parse_cpp,
+            '.c': self._parse_c,
+            '.go': self._parse_go,
+            '.rs': self._parse_rust,
+            # Add more as needed
+        }
+        
+    def chunk_repository(self, repo_path: str) -> List[CodeChunk]:
+        """
+        Main method to chunk entire repository hierarchically
+        """
+        chunks = []
+        repo_name = os.path.basename(repo_path)
+        
+        logger.info(f"🔄 Starting hierarchical chunking of {repo_name}...")
+        
+        # Walk through repository
+        for root, dirs, files in os.walk(repo_path):
+            # Skip common non-code directories
+            dirs[:] = [d for d in dirs if not d.startswith('.') and 
+                      d not in ['node_modules', '__pycache__', 'venv', 'env', 'dist', 'build']]
+            
+            for file in files:
+                if self._should_process_file(file):
+                    file_path = os.path.join(root, file)
+                    relative_path = os.path.relpath(file_path, repo_path)
+                    
+                    try:
+                        file_chunks = self._process_file(file_path, relative_path, repo_name)
+                        logger.debug(f"File chunks: {[chunk.to_dict() for chunk in file_chunks]}")
+                        chunks.extend(file_chunks)
+                        logger.info(f"✅ Processed {relative_path} -> {len(file_chunks)} chunks")
+                    except Exception as e:
+                        logger.info(f"❌ Error processing {relative_path}: {str(e)}")
+                        continue
+        
+        logger.info(f"🎉 Chunking complete! Generated {len(chunks)} total chunks")
+        return chunks
+    
+    def _should_process_file(self, filename: str) -> bool:
+        """Check if file should be processed for chunking"""
+        ext = Path(filename).suffix.lower()
+        
+        # Skip files that are too large or unwanted
+        unwanted_files = {
+            'package-lock.json', 'yarn.lock', 'poetry.lock',
+            'requirements.txt', '.gitignore', 'README.md',
+            'LICENSE', 'CHANGELOG.md'
+        }
+        
+        if filename in unwanted_files:
+            return False
+            
+        # Process code files
+        return ext in self.supported_extensions or ext in [
+            '.py', '.js', '.ts', '.jsx', '.tsx', '.java', '.cpp', '.c', '.h',
+            '.go', '.rs', '.php', '.rb', '.swift', '.kt', '.scala', '.cs'
+        ]
+    
+    def _process_file(self, file_path: str, relative_path: str, repo_name: str) -> List[CodeChunk]:
+        """Process a single file and generate hierarchical chunks"""
+        chunks = []
+        
+        try:
+            with open(file_path, 'r', encoding='utf-8', errors='ignore') as f:
+                content = f.read()
+        except Exception as e:
+            logger.info(f"❌ Could not read {relative_path}: {e}")
+            return chunks
+        
+        if not content.strip():
+            return chunks
+            
+        file_ext = Path(file_path).suffix.lower()
+        
+        # Level 1: File-level chunk
+        file_chunk = self._create_file_chunk(content, relative_path, repo_name)
+        chunks.append(file_chunk)
+        
+        # Language-specific parsing for deeper levels
+        if file_ext in self.supported_extensions:
+            try:
+                deeper_chunks = self.supported_extensions[file_ext](content, relative_path, repo_name)
+                chunks.extend(deeper_chunks)
+            except Exception as e:
+                logger.info(f"⚠��  Advanced parsing failed for {relative_path}, using basic chunking: {e}")
+                # Fallback to basic function extraction
+                basic_chunks = self._basic_function_extraction(content, relative_path, repo_name)
+                chunks.extend(basic_chunks)
+        else:
+            # For unsupported languages, do basic function/class detection
+            basic_chunks = self._basic_function_extraction(content, relative_path, repo_name)
+            chunks.extend(basic_chunks)
+        
+        return chunks
+    
+    def _create_file_chunk(self, content: str, relative_path: str, repo_name: str) -> CodeChunk:
+        """Create Level 1: File-level context chunk"""
+        
+        # Extract file summary info
+        lines = content.split('\n')
+        total_lines = len(lines)
+        
+        # Get imports/includes
+        imports = self._extract_imports(content, Path(relative_path).suffix)
+        
+        # Create condensed file overview
+        file_summary = f"""File: {relative_path}
+Lines: {total_lines}
+Language: {Path(relative_path).suffix}
+
+Imports/Dependencies:
+{chr(10).join(imports[:10])}  # Show first 10 imports
+
+File Purpose: {self._infer_file_purpose(relative_path, content)}
+
+Main Components:
+{self._extract_main_components_summary(content, Path(relative_path).suffix)}
+"""
+        
+        chunk_id = self._generate_chunk_id(repo_name, relative_path, "file", "")
+        
+        metadata = {
+            'repo_name': repo_name,
+            'file_path': relative_path,
+            'chunk_type': 'file',
+            'level': 1,
+            'language': Path(relative_path).suffix,
+            'total_lines': total_lines,
+            'imports': imports,
+            'timestamp': datetime.now().isoformat()
+        }
+        
+        return CodeChunk(
+            id=chunk_id,
+            content=file_summary,
+            chunk_type='file',
+            metadata=metadata
+        )
+    
+    def _parse_python(self, content: str, relative_path: str, repo_name: str) -> List[CodeChunk]:
+        """Parse Python files for classes and functions"""
+        chunks = []
+        
+        try:
+            tree = ast.parse(content)
+        except SyntaxError as e:
+            logger.info(f"⚠️  Python syntax error in {relative_path}: {e}")
+            return self._basic_function_extraction(content, relative_path, repo_name)
+        
+        # Level 2: Class chunks
+        for node in ast.walk(tree):
+            if isinstance(node, ast.ClassDef):
+                class_chunk = self._create_class_chunk(node, content, relative_path, repo_name)
+                chunks.append(class_chunk)
+                
+                # Level 3: Method chunks within class
+                for method in [n for n in node.body if isinstance(n, ast.FunctionDef)]:
+                    method_chunk = self._create_function_chunk(
+                        method, content, relative_path, repo_name, 
+                        parent_class=node.name
+                    )
+                    chunks.append(method_chunk)
+                    
+                    # Level 4: Complex method sub-chunks
+                    if self._calculate_complexity(method) > self.complexity_threshold:
+                        sub_chunks = self._create_sub_chunks(method, content, relative_path, repo_name)
+                        chunks.extend(sub_chunks)
+        
+        # Level 3: Standalone functions
+        for node in ast.walk(tree):
+            if isinstance(node, ast.FunctionDef):
+                # Skip if it's inside a class (already handled above)
+                parent_classes = [n for n in ast.walk(tree) if isinstance(n, ast.ClassDef) 
+                                if any(isinstance(child, ast.FunctionDef) and child.name == node.name 
+                                      for child in ast.walk(n))]
+                
+                if not parent_classes:
+                    func_chunk = self._create_function_chunk(node, content, relative_path, repo_name)
+                    chunks.append(func_chunk)
+                    
+                    # Level 4: Complex function sub-chunks
+                    if self._calculate_complexity(node) > self.complexity_threshold:
+                        sub_chunks = self._create_sub_chunks(node, content, relative_path, repo_name)
+                        chunks.extend(sub_chunks)
+        
+        return chunks
+    
+    def _create_class_chunk(self, class_node: ast.ClassDef, content: str, relative_path: str, repo_name: str) -> CodeChunk:
+        """Create Level 2: Class-level chunk"""
+        
+        lines = content.split('\n')
+        class_content = self._extract_node_content(class_node, lines)
+        
+        # Get class methods summary
+        methods = [n.name for n in class_node.body if isinstance(n, ast.FunctionDef)]
+        
+        # Get docstring
+        docstring = ast.get_docstring(class_node) or "No docstring available"
+        
+        # Get inheritance info
+        bases = [self._get_node_name(base) for base in class_node.bases] if class_node.bases else []
+        
+        class_summary = f"""Class: {class_node.name}
+File: {relative_path}
+Inheritance: {' -> '.join(bases) if bases else 'No inheritance'}
+
+Docstring:
+{docstring[:300]}...
+
+Methods ({len(methods)}):
+{', '.join(methods)}
+
+Full Class Definition:
+{class_content[:1000]}...  # Truncated for embedding
+"""
+        
+        chunk_id = self._generate_chunk_id(repo_name, relative_path, "class", class_node.name)
+        
+        metadata = {
+            'repo_name': repo_name,
+            'file_path': relative_path,
+            'chunk_type': 'class',
+            'level': 2,
+            'class_name': class_node.name,
+            'methods': methods,
+            'inheritance': bases,
+            'line_start': class_node.lineno,
+            'line_end': getattr(class_node, 'end_lineno', class_node.lineno),
+            'docstring': docstring,
+            'timestamp': datetime.now().isoformat()
+        }
+        
+        return CodeChunk(
+            id=chunk_id,
+            content=class_summary,
+            chunk_type='class',
+            metadata=metadata
+        )
+    
+    def _create_function_chunk(self, func_node: ast.FunctionDef, content: str, relative_path: str, 
+                             repo_name: str, parent_class: Optional[str] = None) -> CodeChunk:
+        """Create Level 3: Function-level chunk"""
+        
+        lines = content.split('\n')
+        func_content = self._extract_node_content(func_node, lines)
+        
+        # Get function signature
+        args = [arg.arg for arg in func_node.args.args]
+        signature = f"{func_node.name}({', '.join(args)})"
+        
+        # Get docstring
+        docstring = ast.get_docstring(func_node) or "No docstring available"
+        
+        # Calculate complexity
+        complexity = self._calculate_complexity(func_node)
+        
+        func_summary = f"""Function: {signature}
+File: {relative_path}
+Class: {parent_class or 'Standalone function'}
+Complexity Score: {complexity}
+
+Docstring:
+{docstring[:200]}...
+
+Function Implementation:
+{func_content}
+"""
+        
+        chunk_id = self._generate_chunk_id(
+            repo_name, relative_path, "function", 
+            f"{parent_class}.{func_node.name}" if parent_class else func_node.name
+        )
+        
+        metadata = {
+            'repo_name': repo_name,
+            'file_path': relative_path,
+            'chunk_type': 'function',
+            'level': 3,
+            'function_name': func_node.name,
+            'class_name': parent_class,
+            'signature': signature,
+            'arguments': args,
+            'complexity': complexity,
+            'line_start': func_node.lineno,
+            'line_end': getattr(func_node, 'end_lineno', func_node.lineno),
+            'docstring': docstring,
+            'timestamp': datetime.now().isoformat()
+        }
+        
+        return CodeChunk(
+            id=chunk_id,
+            content=func_summary,
+            chunk_type='function',
+            metadata=metadata
+        )
+    
+    def _create_sub_chunks(self, func_node: ast.FunctionDef, content: str, relative_path: str, repo_name: str) -> List[CodeChunk]:
+        """Create Level 4: Sub-chunks for complex functions"""
+        chunks = []
+        
+        # For now, create logical blocks based on control structures
+        lines = content.split('\n')
+        func_lines = lines[func_node.lineno-1:getattr(func_node, 'end_lineno', func_node.lineno)]
+        
+        # Simple block detection based on indentation and keywords
+        blocks = self._detect_code_blocks(func_lines, func_node.name)
+        
+        for i, block in enumerate(blocks):
+            if len(block['content']) > 50:  # Only create chunks for substantial blocks
+                chunk_id = self._generate_chunk_id(
+                    repo_name, relative_path, "block", f"{func_node.name}_block_{i}"
+                )
+                
+                block_summary = f"""Code Block {i+1} in {func_node.name}()
+Type: {block['type']}
+Purpose: {block['purpose']}
+
+Code:
+{block['content']}
+"""
+                
+                metadata = {
+                    'repo_name': repo_name,
+                    'file_path': relative_path,
+                    'chunk_type': 'block',
+                    'level': 4,
+                    'function_name': func_node.name,
+                    'block_index': i,
+                    'block_type': block['type'],
+                    'block_purpose': block['purpose'],
+                    'timestamp': datetime.now().isoformat()
+                }
+                
+                chunks.append(CodeChunk(
+                    id=chunk_id,
+                    content=block_summary,
+                    chunk_type='block',
+                    metadata=metadata
+                ))
+        
+        return chunks
+    
+    # Helper methods
+    def _extract_imports(self, content: str, file_ext: str) -> List[str]:
+        """Extract import statements based on language"""
+        imports = []
+        lines = content.split('\n')
+        
+        if file_ext == '.py':
+            for line in lines[:50]:  # Check first 50 lines
+                stripped = line.strip()
+                if stripped.startswith(('import ', 'from ')):
+                    imports.append(stripped)
+        elif file_ext in ['.js', '.ts']:
+            for line in lines[:50]:
+                stripped = line.strip()
+                if stripped.startswith(('import ', 'const ', 'require(')):
+                    imports.append(stripped)
+        
+        return imports
+    
+    def _infer_file_purpose(self, relative_path: str, content: str) -> str:
+        """Infer the purpose of a file based on path and content"""
+        filename = os.path.basename(relative_path).lower()
+        
+        if 'test' in filename:
+            return "Test file"
+        elif 'config' in filename:
+            return "Configuration file"
+        elif 'util' in filename or 'helper' in filename:
+            return "Utility/Helper functions"
+        elif '__init__' in filename:
+            return "Package initialization"
+        elif 'main' in filename:
+            return "Main entry point"
+        elif 'model' in filename:
+            return "Data model/schema definition"
+        elif 'view' in filename:
+            return "View/UI component"
+        elif 'controller' in filename:
+            return "Controller/Logic handler"
+        else:
+            # Analyze content for clues
+            if 'class ' in content and 'def __init__' in content:
+                return "Class definition file"
+            elif 'def ' in content:
+                return "Function definitions"
+            else:
+                return "Code file"
+    
+    def _extract_main_components_summary(self, content: str, file_ext: str) -> str:
+        """Extract summary of main components (classes, functions)"""
+        if file_ext != '.py':
+            return "Component analysis available for Python files only"
+        
+        try:
+            tree = ast.parse(content)
+            classes = [node.name for node in ast.walk(tree) if isinstance(node, ast.ClassDef)]
+            functions = [node.name for node in ast.walk(tree) if isinstance(node, ast.FunctionDef)]
+            
+            summary = ""
+            if classes:
+                summary += f"Classes: {', '.join(classes[:5])}\n"
+            if functions:
+                summary += f"Functions: {', '.join(functions[:10])}\n"
+            
+            return summary or "No major components detected"
+        except:
+            return "Could not analyze components"
+    
+    def _basic_function_extraction(self, content: str, relative_path: str, repo_name: str) -> List[CodeChunk]:
+        """Fallback function extraction using regex patterns"""
+        chunks = []
+        # This is a simplified fallback - you can enhance with regex patterns
+        # for different languages
+        return chunks
+    
+    def _extract_node_content(self, node: ast.AST, lines: List[str]) -> str:
+        """Extract the actual code content for an AST node"""
+        start_line = node.lineno - 1
+        end_line = getattr(node, 'end_lineno', node.lineno) - 1
+        
+        if end_line >= len(lines):
+            end_line = len(lines) - 1
+            
+        return '\n'.join(lines[start_line:end_line + 1])
+    
+    def _get_node_name(self, node: ast.AST) -> str:
+        """Get the name of an AST node"""
+        if hasattr(node, 'id'):
+            return node.id
+        elif hasattr(node, 'attr'):
+            return node.attr
+        else:
+            return str(node)
+    
+    def _calculate_complexity(self, node: ast.FunctionDef) -> int:
+        """Calculate cyclomatic complexity of a function"""
+        complexity = 1  # Base complexity
+        
+        for child in ast.walk(node):
+            if isinstance(child, (ast.If, ast.While, ast.For, ast.With, ast.Try)):
+                complexity += 1
+            elif isinstance(child, ast.ExceptHandler):
+                complexity += 1
+            elif isinstance(child, (ast.And, ast.Or)):
+                complexity += 1
+                
+        return complexity
+    
+    def _detect_code_blocks(self, func_lines: List[str], func_name: str) -> List[Dict[str, str]]:
+        """Detect logical code blocks within a function"""
+        blocks = []
+        current_block = []
+        block_type = "sequential"
+        
+        for line in func_lines:
+            stripped = line.strip()
+            
+            if any(keyword in stripped for keyword in ['if ', 'elif ', 'else:']):
+                if current_block:
+                    blocks.append({
+                        'content': '\n'.join(current_block),
+                        'type': block_type,
+                        'purpose': f"Logic block in {func_name}"
+                    })
+                    current_block = []
+                block_type = "conditional"
+            elif any(keyword in stripped for keyword in ['for ', 'while ']):
+                if current_block:
+                    blocks.append({
+                        'content': '\n'.join(current_block),
+                        'type': block_type,
+                        'purpose': f"Logic block in {func_name}"
+                    })
+                    current_block = []
+                block_type = "loop"
+            elif any(keyword in stripped for keyword in ['try:', 'except', 'finally:']):
+                if current_block:
+                    blocks.append({
+                        'content': '\n'.join(current_block),
+                        'type': block_type,
+                        'purpose': f"Logic block in {func_name}"
+                    })
+                    current_block = []
+                block_type = "exception_handling"
+            
+            current_block.append(line)
+        
+        if current_block:
+            blocks.append({
+                'content': '\n'.join(current_block),
+                'type': block_type,
+                'purpose': f"Final block in {func_name}"
+            })
+        
+        return blocks
+    
+    def _generate_chunk_id(self, repo_name: str, file_path: str, chunk_type: str, identifier: str) -> str:
+        """Generate unique chunk ID"""
+        unique_string = f"{repo_name}:{file_path}:{chunk_type}:{identifier}"
+        return hashlib.md5(unique_string.encode()).hexdigest()
+    
+    # Placeholder methods for other languages
+    def _parse_javascript(self, content: str, relative_path: str, repo_name: str) -> List[CodeChunk]:
+        """Parse JavaScript files - placeholder for now"""
+        return []
+    
+    def _parse_typescript(self, content: str, relative_path: str, repo_name: str) -> List[CodeChunk]:
+        """Parse TypeScript files - placeholder for now"""
+        return []
+    
+    def _parse_java(self, content: str, relative_path: str, repo_name: str) -> List[CodeChunk]:
+        """Parse Java files - placeholder for now"""
+        return []
+    
+    def _parse_cpp(self, content: str, relative_path: str, repo_name: str) -> List[CodeChunk]:
+        """Parse C++ files - placeholder for now"""
+        return []
+    
+    def _parse_c(self, content: str, relative_path: str, repo_name: str) -> List[CodeChunk]:
+        """Parse C files - placeholder for now"""
+        return []
+    
+    def _parse_go(self, content: str, relative_path: str, repo_name: str) -> List[CodeChunk]:
+        """Parse Go files - placeholder for now"""
+        return []
+    
+    def _parse_rust(self, content: str, relative_path: str, repo_name: str) -> List[CodeChunk]:
+        """Parse Rust files - placeholder for now"""
+        return []

scripts/download_model.py

@@ -0,0 +1,161 @@
+#!/usr/bin/env python3
+"""
+Script to download Qwen2.5-Coder-7B-Instruct quantized model
+"""
+
+import os
+import requests
+import sys
+from pathlib import Path
+from tqdm import tqdm
+
+import logging
+
+logger = logging.getLogger("code_compass")
+
+def download_file(url, filename):
+    """Download file with progress bar"""
+    logger.info(f"📥 Downloading {filename}...")
+    logger.info(f"🔗 URL: {url}")
+    
+    response = requests.get(url, stream=True)
+    total_size = int(response.headers.get('content-length', 0))
+    
+    if total_size == 0:
+        logger.info("❌ Could not determine file size")
+        return False
+    
+    logger.info(f"📊 File size: {total_size / (1024*1024*1024):.2f} GB")
+    
+    with open(filename, 'wb') as file, tqdm(
+        desc=filename,
+        total=total_size,
+        unit='B',
+        unit_scale=True,
+        unit_divisor=1024,
+    ) as progress_bar:
+        for chunk in response.iter_content(chunk_size=8192):
+            if chunk:
+                file.write(chunk)
+                progress_bar.update(len(chunk))
+    
+    logger.info(f"✅ Downloaded {filename} successfully!")
+    return True
+
+def main():
+    """Main download function"""
+    logger.info("🔍 Qwen2.5-Coder-7B-Instruct Model Downloader")
+    logger.info("=" * 50)
+    
+    # Available quantization options
+    models = {
+        "Q4_K_M": {
+            "url": "https://huggingface.co/bartowski/Qwen2.5-Coder-7B-Instruct-GGUF/resolve/main/Qwen2.5-Coder-7B-Instruct-Q4_K_M.gguf",
+            "filename": "qwen2.5-coder-7b-instruct-q4_k_m.gguf",
+            "size": "~4.5 GB",
+            "description": "4-bit quantization, best balance of quality and size (RECOMMENDED)"
+        },
+        "Q5_K_M": {
+            "url": "https://huggingface.co/bartowski/Qwen2.5-Coder-7B-Instruct-GGUF/resolve/main/Qwen2.5-Coder-7B-Instruct-Q5_K_M.gguf",
+            "filename": "qwen2.5-coder-7b-instruct-q5_k_m.gguf",
+            "size": "~5.5 GB",
+            "description": "5-bit quantization, higher quality than Q4"
+        },
+        "Q6_K": {
+            "url": "https://huggingface.co/bartowski/Qwen2.5-Coder-7B-Instruct-GGUF/resolve/main/Qwen2.5-Coder-7B-Instruct-Q6_K.gguf",
+            "filename": "qwen2.5-coder-7b-instruct-q6_k.gguf",
+            "size": "~6.5 GB",
+            "description": "6-bit quantization, highest quality"
+        },
+        "Q8_0": {
+            "url": "https://huggingface.co/bartowski/Qwen2.5-Coder-7B-Instruct-GGUF/resolve/main/Qwen2.5-Coder-7B-Instruct-Q8_0.gguf",
+            "filename": "qwen2.5-coder-7b-instruct-q8_0.gguf",
+            "size": "~7.5 GB",
+            "description": "8-bit quantization, near full precision"
+        }
+    }
+    
+    logger.info("📋 Available model variants:")
+    logger.info()
+    for i, (key, info) in enumerate(models.items(), 1):
+        marker = " ⭐ RECOMMENDED" if key == "Q4_K_M" else ""
+        logger.info(f"{i}. {key}{marker}")
+        logger.info(f"   Size: {info['size']}")
+        logger.info(f"   Description: {info['description']}")
+        logger.info()
+    
+    # Get user choice
+    while True:
+        try:
+            choice = input("Enter your choice (1-4) or 'q' to quit: ").strip()
+            
+            if choice.lower() == 'q':
+                logger.info("👋 Download cancelled.")
+                return
+            
+            choice_num = int(choice)
+            if 1 <= choice_num <= len(models):
+                selected_key = list(models.keys())[choice_num - 1]
+                selected_model = models[selected_key]
+                break
+            else:
+                logger.info("❌ Invalid choice. Please enter 1-4.")
+        except ValueError:
+            logger.info("❌ Invalid input. Please enter a number 1-4 or 'q'.")
+    
+    logger.info(f"📦 Selected: {selected_key}")
+    logger.info(f"📁 Filename: {selected_model['filename']}")
+    logger.info(f"📊 Size: {selected_model['size']}")
+    logger.info()
+    
+    # Check if file already exists
+    if os.path.exists(selected_model['filename']):
+        overwrite = input(f"⚠️  File {selected_model['filename']} already exists. Overwrite? (y/n): ")
+        if overwrite.lower() != 'y':
+            logger.info("👋 Download cancelled.")
+            return
+    
+    # Create models directory if it doesn't exist
+    models_dir = Path("models")
+    models_dir.mkdir(exist_ok=True)
+    
+    # Full path for the model
+    model_path = models_dir / selected_model['filename']
+    
+    # Download the model
+    try:
+        success = download_file(selected_model['url'], str(model_path))
+        
+        if success:
+            logger.info()
+            logger.info("🎉 Download completed successfully!")
+            logger.info(f"📁 Model saved to: {model_path}")
+            logger.info()
+            logger.info("🚀 To use the model:")
+            logger.info("   1. Make sure the model path in llm_service.py points to this file")
+            logger.info("   2. Run your main application: python main.py")
+            logger.info("   3. Click 'Initialize LLM' in the web interface")
+            logger.info()
+            logger.info("💡 System Requirements:")
+            logger.info("   - RAM: At least 8GB (16GB+ recommended)")
+            logger.info("   - Storage: Ensure you have enough free space")
+            logger.info("   - CPU: Modern multi-core processor recommended")
+        else:
+            logger.info("❌ Download failed!")
+            return 1
+            
+    except KeyboardInterrupt:
+        logger.info("\n🛑 Download interrupted by user")
+        # Clean up partial file
+        if os.path.exists(model_path):
+            os.remove(model_path)
+            logger.info(f"🗑️  Cleaned up partial file: {model_path}")
+        return 1
+    except Exception as e:
+        logger.info(f"❌ Error during download: {str(e)}")
+        return 1
+    
+    return 0
+
+if __name__ == "__main__":
+    sys.exit(main())

scripts/llm_service.py

@@ -0,0 +1,413 @@
+import os
+import json
+import time
+from typing import List, Dict, Any, Optional, Tuple
+from datetime import datetime
+from dataclasses import dataclass
+import threading
+from pathlib import Path
+
+# llama-cpp-python for quantized model inference
+from llama_cpp import Llama
+import logging
+import os
+
+os.environ["CUDA_VISIBLE_DEVICES"] = "7"
+
+logger = logging.getLogger("code_compass")
+@dataclass
+class ChatMessage:
+    """Represents a chat message in the conversation history"""
+    role: str  # 'system', 'user', 'assistant'
+    content: str
+    timestamp: str
+    metadata: Optional[Dict[str, Any]] = None
+
+class ConversationHistory:
+    """Manages conversation history with context window management"""
+    
+    def __init__(self, max_messages: int = 20, max_tokens: int = 4000):
+        self.messages: List[ChatMessage] = []
+        self.max_messages = max_messages
+        self.max_tokens = max_tokens
+        
+    def add_message(self, role: str, content: str, metadata: Optional[Dict[str, Any]] = None):
+        """Add a message to the conversation history"""
+        message = ChatMessage(
+            role=role,
+            content=content,
+            timestamp=datetime.now().isoformat(),
+            metadata=metadata or {}
+        )
+        self.messages.append(message)
+        self._trim_history()
+    
+    def _trim_history(self):
+        """Trim history to stay within limits"""
+        # Keep only the last max_messages
+        if len(self.messages) > self.max_messages:
+            # Always keep system messages
+            system_messages = [msg for msg in self.messages if msg.role == 'system']
+            recent_messages = [msg for msg in self.messages if msg.role != 'system'][-self.max_messages:]
+            self.messages = system_messages + recent_messages
+        
+        # Estimate token count and trim if needed
+        total_chars = sum(len(msg.content) for msg in self.messages)
+        # Rough estimate: 4 characters per token
+        estimated_tokens = total_chars // 4
+        
+        if estimated_tokens > self.max_tokens:
+            # Keep system messages and trim from the oldest user/assistant messages
+            system_messages = [msg for msg in self.messages if msg.role == 'system']
+            other_messages = [msg for msg in self.messages if msg.role != 'system']
+            
+            # Remove oldest messages until we're under the limit
+            while other_messages and (sum(len(msg.content) for msg in system_messages + other_messages) // 4) > self.max_tokens:
+                other_messages.pop(0)
+            
+            self.messages = system_messages + other_messages
+    
+    def get_messages_for_llm(self) -> List[Dict[str, str]]:
+        """Get messages in format expected by LLM"""
+        return [
+            {"role": msg.role, "content": msg.content}
+            for msg in self.messages
+        ]
+    
+    def clear(self):
+        """Clear conversation history"""
+        self.messages = []
+    
+    def get_summary(self) -> str:
+        """Get a summary of the conversation"""
+        if not self.messages:
+            return "No conversation history"
+        
+        user_msgs = len([msg for msg in self.messages if msg.role == 'user'])
+        assistant_msgs = len([msg for msg in self.messages if msg.role == 'assistant'])
+        
+        return f"Conversation: {user_msgs} questions, {assistant_msgs} responses"
+
+class QwenCoderLLM:
+    """
+    Qwen2.5-Coder-7B-Instruct integration using llama-cpp-python
+    """
+    
+    def __init__(self, 
+                 model_path: Optional[str] = None,
+                 n_ctx: int = 8192,  # Context window
+                 n_threads: int = -1,  # Auto-detect threads
+                 n_gpu_layers: int = 0,  # CPU-only by default
+                 temperature: float = 0.1,  # Low temperature for code tasks
+                 max_tokens: int = 1024):
+        
+        self.model_path = model_path or self._get_model_path()
+        self.n_ctx = n_ctx
+        self.n_threads = n_threads
+        self.n_gpu_layers = n_gpu_layers
+        self.temperature = temperature
+        self.max_tokens = max_tokens
+        
+        # Initialize conversation history
+        self.conversation = ConversationHistory()
+        
+        # Model loading
+        self.llm = None
+        self.is_loaded = False
+        self.loading_thread = None
+        
+        # System prompt for code analysis
+        self.system_prompt = self._create_system_prompt()
+        
+        # Initialize system message
+        self.conversation.add_message("system", self.system_prompt)
+    
+    def _get_model_path(self) -> str:
+        """Get model path, with instructions for download if not found"""
+        possible_paths = [
+            "./models/qwen2.5-coder-7b-instruct-q4_k_m.gguf",
+            "./qwen2.5-coder-7b-instruct-q4_k_m.gguf",
+            os.path.expanduser("~/models/qwen2.5-coder-7b-instruct-q4_k_m.gguf"),
+        ]
+        
+        for path in possible_paths:
+            if os.path.exists(path):
+                return path
+        
+        # Model not found - provide download instructions
+        logger.info("🔍 Qwen2.5-Coder model not found!")
+        logger.info("📥 Please download the quantized model:")
+        logger.info("   wget https://huggingface.co/bartowski/Qwen2.5-Coder-7B-Instruct-GGUF/resolve/main/Qwen2.5-Coder-7B-Instruct-Q4_K_M.gguf")
+        logger.info("   mv Qwen2.5-Coder-7B-Instruct-Q4_K_M.gguf qwen2.5-coder-7b-instruct-q4_k_m.gguf")
+        logger.info()
+        
+        # Return first path as placeholder
+        return possible_paths[0]
+    
+    def _create_system_prompt(self) -> str:
+        """Create system prompt for code analysis tasks"""
+        return """You are Qwen2.5-Coder, an expert AI assistant specialized in code analysis and software engineering. You have access to a codebase that has been analyzed and chunked hierarchically.
+
+**Your Role:**
+- Analyze code repositories with deep understanding
+- Provide accurate, helpful responses about code structure, functionality, and best practices
+- Maintain conversation context and refer to previous discussions
+- Give practical, actionable advice
+
+**Context Information:**
+When answering questions, you'll be provided with:
+1. **User Query**: The current question
+2. **Retrieved Code Chunks**: Relevant code sections from the repository
+3. **Conversation History**: Previous questions and answers in this session
+
+**Response Guidelines:**
+- Be concise but comprehensive
+- Use code examples from the retrieved chunks when relevant
+- Explain technical concepts clearly
+- Suggest improvements or alternatives when appropriate
+- If information is missing, say so rather than guessing
+- Format code snippets with proper syntax highlighting
+
+**Code Analysis Focus:**
+- Understand code architecture and patterns
+- Identify key functions, classes, and relationships  
+- Explain implementation details and design decisions
+- Highlight potential issues or improvements
+- Provide context about how components work together
+
+Always be helpful, accurate, and focused on the user's specific needs."""
+
+    def load_model_async(self):
+        """Load model asynchronously to avoid blocking the UI"""
+        def _load():
+            try:
+                logger.info(f"🔄 Loading Qwen2.5-Coder model from {self.model_path}...")
+                logger.info(f"⚙️  Configuration: n_ctx={self.n_ctx}, n_threads={self.n_threads}, n_gpu_layers={self.n_gpu_layers}")
+                
+                # self.llm = Llama(
+                #     model_path=self.model_path,
+                #     n_ctx=self.n_ctx,
+                #     n_threads=self.n_threads,
+                #     n_gpu_layers=self.n_gpu_layers,
+                #     verbose=False,
+                #     use_mlock=True,  # Keep model in memory
+                #     use_mmap=True,   # Memory-map the model file
+                # )
+                self.llm = Llama(
+                    model_path=self.model_path,
+                    cache_dir=Path('models'),
+                    seed=42,
+                    n_ctx=self.n_ctx,
+                    verbose=False,
+                    n_gpu_layers=self.n_gpu_layers,
+                    n_threads=self.n_threads,
+                )
+                
+                self.is_loaded = True
+                logger.info("✅ Qwen2.5-Coder model loaded successfully!")
+                
+                # Test the model with a simple query
+                # test_response = self.llm.create_chat_completion(
+                #     messages=[{"role": "user", "content": "Hello, are you working?"}],
+                #     max_tokens=50,
+                #     temperature=0.1
+                # )
+                # logger.info(f"🧪 Model test: {test_response['choices'][0]['message']['content'][:50]}...")
+                
+            except Exception as e:
+                logger.info(f"❌ Error loading model: {str(e)}")
+                self.is_loaded = False
+                
+        self.loading_thread = threading.Thread(target=_load)
+        self.loading_thread.start()
+    
+    def wait_for_model(self, timeout: int = 300) -> bool:
+        """Wait for model to load with timeout"""
+        if self.loading_thread:
+            self.loading_thread.join(timeout=timeout)
+        return self.is_loaded
+    
+    def is_model_ready(self) -> bool:
+        """Check if model is ready for inference"""
+        return self.is_loaded and self.llm is not None
+    
+    def generate_response(self, 
+                         user_query: str,
+                         retrieved_chunks: List[Dict[str, Any]] = None,
+                         use_history: bool = True) -> Dict[str, Any]:
+        """
+        Generate response using LLM with retrieved context and conversation history
+        
+        Args:
+            user_query: User's question
+            retrieved_chunks: Relevant code chunks from vector search
+            use_history: Whether to include conversation history
+            
+        Returns:
+            Dict with response and metadata
+        """
+        
+        if not self.is_model_ready():
+            return {
+                "status": "error",
+                "message": "❌ Model not loaded. Please wait for model initialization.",
+                "response": ""
+            }
+        
+        try:
+            # Build context from retrieved chunks
+            context = self._build_context_from_chunks(retrieved_chunks or [])
+            
+            # Create the current query with context
+            query_with_context = self._format_query_with_context(user_query, context)
+            
+            # Add user query to conversation history
+            self.conversation.add_message("user", user_query, {
+                "chunks_count": len(retrieved_chunks) if retrieved_chunks else 0,
+                "context_length": len(context)
+            })
+            
+            # Prepare messages for LLM
+            if use_history:
+                messages = self.conversation.get_messages_for_llm()
+                # Replace the last user message with the context-enhanced version
+                messages[-1]["content"] = query_with_context
+            else:
+                messages = [
+                    {"role": "system", "content": self.system_prompt},
+                    {"role": "user", "content": query_with_context}
+                ]
+            
+            logger.info(f"🤖 Generating response for query: '{user_query[:50]}...'")
+            logger.info(f"📊 Context: {len(retrieved_chunks) if retrieved_chunks else 0} chunks, History: {len(self.conversation.messages)} messages")
+            
+            # Generate response
+            start_time = time.time()
+            
+            response = self.llm.create_chat_completion(
+                messages=messages,
+                max_tokens=self.max_tokens,
+                temperature=self.temperature,
+                stream=False
+            )
+            
+            generation_time = time.time() - start_time
+            
+            # Extract response content
+            assistant_response = response['choices'][0]['message']['content']
+            
+            # Add assistant response to conversation history
+            self.conversation.add_message("assistant", assistant_response, {
+                "generation_time": generation_time,
+                "tokens_used": response.get('usage', {}).get('total_tokens', 0)
+            })
+            
+            logger.info(f"✅ Response generated in {generation_time:.2f}s")
+            
+            return {
+                "status": "success",
+                "response": assistant_response,
+                "metadata": {
+                    "generation_time": generation_time,
+                    "chunks_used": len(retrieved_chunks) if retrieved_chunks else 0,
+                    "conversation_length": len(self.conversation.messages),
+                    "tokens_used": response.get('usage', {}).get('total_tokens', 0)
+                }
+            }
+            
+        except Exception as e:
+            error_msg = f"❌ Error generating response: {str(e)}"
+            logger.info(error_msg)
+            
+            return {
+                "status": "error",
+                "message": error_msg,
+                "response": "I apologize, but I encountered an error while processing your request. Please try again."
+            }
+    
+    def _build_context_from_chunks(self, chunks: List[Dict[str, Any]]) -> str:
+        """Build context string from retrieved code chunks"""
+        if not chunks:
+            return ""
+        
+        context_parts = ["**Retrieved Code Context:**\n"]
+        
+        for i, chunk in enumerate(chunks[:5], 1):  # Limit to top 5 chunks
+            metadata = chunk.get('metadata', {})
+            score = chunk.get('score', 0)
+            
+            chunk_type = metadata.get('chunk_type', 'code')
+            file_path = metadata.get('file_path', 'unknown')
+            
+            context_parts.append(f"**{i}. {chunk_type.title()} from `{file_path}` (Similarity: {score:.2f})**")
+            
+            # Add specific context based on chunk type
+            if chunk_type == 'function':
+                func_name = metadata.get('function_name', 'unknown')
+                signature = metadata.get('signature', func_name)
+                class_name = metadata.get('class_name')
+                
+                if class_name:
+                    context_parts.append(f"Function: `{class_name}.{signature}`")
+                else:
+                    context_parts.append(f"Function: `{signature}`")
+                    
+            elif chunk_type == 'class':
+                class_name = metadata.get('class_name', 'unknown')
+                methods = metadata.get('methods', [])
+                context_parts.append(f"Class: `{class_name}`")
+                if methods:
+                    context_parts.append(f"Methods: {', '.join(methods[:5])}")
+                    
+            elif chunk_type == 'file':
+                language = metadata.get('language', '')
+                total_lines = metadata.get('total_lines', 'unknown')
+                context_parts.append(f"File overview: {language} ({total_lines} lines)")
+            
+            # Add a separator
+            context_parts.append("---\n")
+        
+        return "\n".join(context_parts)
+    
+    def _format_query_with_context(self, query: str, context: str) -> str:
+        """Format user query with retrieved context"""
+        if not context:
+            return query
+            
+        return f"""**User Question:** {query}
+
+{context}
+
+**Instructions:** Using the retrieved code context above, please provide a comprehensive answer to the user's question. Reference specific code snippets, functions, or classes when relevant. If the context doesn't contain enough information to fully answer the question, please mention what additional information would be helpful."""
+    
+    def clear_conversation(self):
+        """Clear conversation history but keep system prompt"""
+        self.conversation.clear()
+        self.conversation.add_message("system", self.system_prompt)
+    
+    def get_conversation_summary(self) -> str:
+        """Get summary of current conversation"""
+        return self.conversation.get_summary()
+    
+    def export_conversation(self) -> List[Dict[str, Any]]:
+        """Export conversation history"""
+        return [
+            {
+                "role": msg.role,
+                "content": msg.content,
+                "timestamp": msg.timestamp,
+                "metadata": msg.metadata
+            }
+            for msg in self.conversation.messages
+        ]
+    
+    def get_model_info(self) -> Dict[str, Any]:
+        """Get information about the loaded model"""
+        return {
+            "model_path": self.model_path,
+            "is_loaded": self.is_loaded,
+            "context_window": self.n_ctx,
+            "temperature": self.temperature,
+            "max_tokens": self.max_tokens,
+            "conversation_messages": len(self.conversation.messages)
+        }

scripts/vectorstore.py

@@ -0,0 +1,629 @@
+import os
+import time
+from typing import List, Dict, Any, Optional, Tuple
+import hashlib
+from datetime import datetime
+import json
+
+# Vector database and embedding imports
+from pinecone import Pinecone 
+# from sentence_transformers import SentenceTransformer
+import numpy as np
+import logging
+# Local imports
+from .chunker import CodeChunk
+from config import PINECONE_API_KEY, PINECONE_INDEX_NAME, PINECONE_EMBEDDING_MODEL
+
+
+logger = logging.getLogger("code_compass")
+class PineconeVectorStore:
+    """
+    Pinecone vector database integration with built-in embedding generation
+    """
+    
+    def __init__(self, 
+                 namespace
+                #  api_key: Optional[str] = None,
+                #  index_name: str = "code-compass",
+                #  embedding_model: str = "multilingual-e5-large"
+                 ):
+        """
+        Initialize Pinecone vector store with inference API for embeddings
+        
+        Args:
+            api_key: Pinecone API key (or set PINECONE_API_KEY env var)
+            index_name: Name of the Pinecone index
+            embedding_model: Pinecone's embedding model to use
+        """
+        
+        # Setup API key
+        self.api_key = PINECONE_API_KEY #api_key or os.getenv('PINECONE_API_KEY')
+        self.namespace = namespace
+        if not self.api_key:
+            raise ValueError("Pinecone API key is required. Set PINECONE_API_KEY env var or pass api_key parameter")
+        
+        self.index_name = PINECONE_INDEX_NAME #index_name
+        # self.embedding_model = embedding_model
+        
+        # Initialize Pinecone client
+        self.pc = Pinecone(api_key=self.api_key)
+        
+        # Initialize index
+        self._initialize_index()
+        
+    def _initialize_index(self):
+        """Initialize Pinecone index with inference API"""
+        try:
+            logger.info("🔄 Initializing Pinecone connection...")
+            
+            # Check if index exists
+            existing_indexes = [index.name for index in self.pc.list_indexes()]
+            
+            if self.index_name not in existing_indexes:
+                logger.info(f"🔄 Creating new Pinecone index: {self.index_name}")
+                
+                # Create index with inference API enabled
+                if not self.pc.has_index(self.index_name):
+                    self.pc.create_index_for_model(
+                        name=self.index_name,
+                        cloud="aws",
+                        region="us-east-1",
+                        embed={
+                            "model": PINECONE_EMBEDDING_MODEL,
+                            "field_map":{"text": "chunk_text", "metadata": "metadata", "id": "_id"}
+                        }
+                    )
+                
+                # Wait for index to be ready
+                logger.info("⏳ Waiting for index to be ready...")
+                while not self.pc.describe_index(self.index_name).status['ready']:
+                    time.sleep(1)
+            
+            # Connect to index
+            self.index = self.pc.Index(self.index_name)
+            logger.info(f"✅ Connected to Pinecone index: {self.index_name}")
+            
+            # Get index stats
+            stats = self.index.describe_index_stats()
+            logger.info(f"📊 Index stats: {stats.get('total_vector_count', 0)} vectors stored")
+            if self.namespace in stats.get('namespaces', {}):
+                logger.info(f"Namespace '{self.namespace}' exists. Proceeding with deletion...")
+                # 4. Delete all vectors in the namespace
+                self.index.delete_namespace(namespace=self.namespace)
+                logger.info(f"Successfully deleted all vectors in namespace '{self.namespace}'.")
+            else:
+                logger.info(f"Namespace '{self.namespace}' does not exist. No action needed.")
+
+        except Exception as e:
+            logger.info(f"❌ Error initializing Pinecone: {str(e)}")
+            raise
+    
+    def upsert_chunks(self, chunks: List[CodeChunk], batch_size: int = 96) -> Dict[str, Any]:
+        """
+        Upsert code chunks to Pinecone using inference API for embeddings
+        
+        Args:
+            chunks: List of code chunks (embeddings will be generated by Pinecone)
+            batch_size: Batch size for upsert operations
+            
+        Returns:
+            Dictionary with upsert results
+        """
+        logger.info(f"🔄 Upserting {len(chunks)} chunks to Pinecone with automatic embedding generation...")
+        
+        if not chunks:
+            return {"status": "error", "message": "No chunks provided"}
+        
+        # Prepare data for Pinecone inference API
+        data_to_upsert = []
+        
+        for chunk in chunks:
+            # Prepare metadata (Pinecone has limitations on metadata size)
+            metadata = self._prepare_metadata_for_pinecone(chunk.metadata)
+            
+            # For Pinecone inference API, we send the text content directly
+            data_to_upsert.append({
+                "_id": chunk.id,
+                "chunk_text": chunk.content,  # Pinecone will generate embeddings from this
+                "metadata": metadata
+            })
+        
+        if not data_to_upsert:
+            return {"status": "error", "message": "No valid data to upsert"}
+        
+        # Upsert in batches using Pinecone's inference API
+        successful_upserts = 0
+        failed_upserts = 0
+        
+        for i in range(0, len(data_to_upsert), batch_size):
+            batch = data_to_upsert[i:i + batch_size]
+            
+            try:
+                logger.info(f"📊 Upserting batch {i//batch_size + 1}/{(len(data_to_upsert)-1)//batch_size + 1} ({len(batch)} items)")
+                
+                # Debug: Print first item structure on first batch
+                if i == 0 and len(batch) > 0:
+                    logger.debug(f"🔍 Sample item structure:")
+                    sample_item = batch[0]
+                    logger.debug(f"  ID: {sample_item['_id']}")
+                    logger.debug(f"  Text length: {len(sample_item['chunk_text'])}")
+                    logger.debug(f"  Metadata keys: {sample_item['metadata']}")
+                
+                # Use Pinecone's inference API
+                upsert_response = self.index.upsert_records(
+                    self.namespace,batch
+                )
+                time.sleep(1)  # Slight delay to ensure consistency
+                successful_upserts += len(batch)
+                logger.info(f"✅ Batch {i//batch_size + 1} upserted successfully")
+                # if hasattr(upsert_response, 'upserted_count') and upsert_response.upserted_count > 0:
+                #     successful_upserts += upsert_response.upserted_count
+                # else:
+                #     # If no upserted_count, assume success based on batch size
+                #     successful_upserts += len(batch)
+            except Exception as e:
+                logger.info(f"❌ Error upserting batch {i//batch_size + 1}: {str(e)}")
+                
+                # Try alternative method if dataframe method fails
+                try:
+                    logger.info("🔄 Trying alternative upsert method...")
+                    
+                    # Convert to format expected by regular upsert
+                    vectors_batch = []
+                    for item in batch:
+                        vectors_batch.append({
+                            "_id": item["_id"],
+                            "chunk_text": item["chunk_text"],  # Let Pinecone handle embedding
+                            "metadata": item["metadata"]
+                        })
+                    
+                    # Use regular upsert with text (if supported)
+                    upsert_response = self.index.upsert_records(self.namespace, vectors_batch)
+                    # logger.debug("Upsert response: " + str(upsert_response))
+                    # if upsert_response.get('upserted_count', 0) > 0:
+                    #     successful_upserts += upsert_response['upserted_count']
+                    # else:
+                    #     failed_upserts += len(batch)
+                    time.sleep(10)
+                    successful_upserts += len(vectors_batch)
+                    logger.info(f"✅ Alternative upsert method succeeded for batch {i//batch_size + 1}")
+                except Exception as e2:
+                    logger.info(f"❌ Alternative upsert method also failed: {str(e2)}")
+                    failed_upserts += len(batch)
+                    continue
+        
+        # Final results
+        result = {
+            "status": "success" if successful_upserts > 0 else "error",
+            "successful_upserts": successful_upserts,
+            "failed_upserts": failed_upserts,
+            "total_chunks": len(chunks),
+            "timestamp": datetime.now().isoformat()
+        }
+        
+        logger.info(f"✅ Upsert complete! {successful_upserts} successful, {failed_upserts} failed")
+        return result
+    
+    def safe_json_store(self, final_metadata):
+        try:
+            return json.dumps(final_metadata, ensure_ascii=False)
+        except (TypeError, ValueError):
+            # fallback: force conversion to string and JSON-escape it
+            return json.dumps(str(final_metadata), ensure_ascii=False)
+        
+    def _prepare_metadata_for_pinecone(self, metadata: Dict[str, Any]) -> Dict[str, Any]:
+        """
+        Prepare metadata for Pinecone storage (handles size and type limitations)
+        """
+        # Pinecone metadata limitations:
+        # - Max 40KB per vector metadata
+        # - Only supports string, number, boolean, and list of strings
+        # - NO nested objects or complex data types
+        
+        cleaned_metadata = {}
+        
+        for key, value in metadata.items():
+            if value is None:
+                continue
+                
+            # Convert different types to Pinecone-compatible formats
+            if isinstance(value, (str, int, float, bool)):
+                # Limit string length to avoid size issues
+                if isinstance(value, str) and len(value) > 500:
+                    cleaned_metadata[key] = value[:500] + "..."
+                else:
+                    cleaned_metadata[key] = value
+                    
+            elif isinstance(value, list):
+                # Convert list to list of strings (Pinecone requirement)
+                if all(isinstance(item, str) for item in value):
+                    # Limit list size and string length
+                    limited_list = [str(item)[:100] for item in value[:5]]  # Max 5 items
+                    cleaned_metadata[key] = limited_list
+                else:
+                    # Convert non-string items to strings
+                    string_list = [str(item)[:100] for item in value[:5]]
+                    cleaned_metadata[key] = string_list
+                    
+            elif isinstance(value, dict):
+                # Pinecone doesn't support nested objects - flatten or convert to string
+                # Option 1: Flatten the dict
+                for sub_key, sub_value in value.items():
+                    flattened_key = f"{key}_{sub_key}"
+                    if isinstance(sub_value, (str, int, float, bool)):
+                        if isinstance(sub_value, str) and len(sub_value) > 200:
+                            cleaned_metadata[flattened_key] = str(sub_value)[:200] + "..."
+                        else:
+                            cleaned_metadata[flattened_key] = sub_value
+                    else:
+                        cleaned_metadata[flattened_key] = str(sub_value)[:200]
+                        
+            else:
+                # Convert other types to string
+                cleaned_metadata[key] = str(value)[:200]
+        
+        # Double-check that we don't have any complex types
+        final_metadata = {}
+        for key, value in cleaned_metadata.items():
+            if isinstance(value, (str, int, float, bool)):
+                final_metadata[key] = value
+            elif isinstance(value, list) and all(isinstance(item, str) for item in value):
+                final_metadata[key] = value
+            else:
+                # Last resort - convert to string
+                final_metadata[key] = str(value)[:200]
+        
+        return self.safe_json_store(final_metadata)#.replace("'", '"')  # Store as JSON string
+    
+    
+
+    def query_similar_chunks(self, 
+                           query_text: str, 
+                           top_k: int = 10,
+                           filter_dict: Optional[Dict[str, Any]] = None,
+                           include_metadata: bool = True) -> List[Dict[str, Any]]:
+        """
+        Query for similar chunks using Pinecone's inference API
+        
+        Args:
+            query_text: Text to search for (Pinecone will generate embeddings)
+            top_k: Number of similar chunks to return
+            filter_dict: Optional metadata filters
+            include_metadata: Whether to include metadata in results
+            
+        Returns:
+            List of similar chunks with scores
+        """
+        try:
+            logger.info(f"🔍 Searching for similar chunks to: '{query_text[:50]}...'")
+
+            # Use Pinecone's inference API for query
+            search_results = self.index.search(
+                namespace=self.namespace,
+                query={"inputs": {"text": query_text}, "top_k": top_k},
+            )
+            
+            
+            results = []
+            if 'result' not in search_results or 'hits' not in search_results['result']:
+                logger.info("⚠️  No results found in search response")
+                return []
+            for match in search_results['result']['hits']:
+                result = {
+                    'id': match['_id'],
+                    'chunk_text': match['fields']['chunk_text'],
+                    'score': float(match['_score']),
+                    'metadata': match['fields']['metadata'] if include_metadata else None
+                }
+                results.append(result)
+            
+            logger.info(f"✅ Found {len(results)} similar chunks")
+            logger.debug(f"Results: {results}")
+            return results
+            
+        except Exception as e:
+            logger.info(f"❌ Error querying similar chunks: {str(e)}")
+            
+            # Fallback to regular query if inference API fails
+            try:
+                logger.info("🔄 Trying fallback query method...")
+                # This would require manual embedding generation as fallback
+                # For now, return empty results
+                return []
+            except Exception as e2:
+                logger.info(f"❌ Fallback query also failed: {str(e2)}")
+                return []
+    
+    def query_by_metadata(self, 
+                         filter_dict: Dict[str, Any], 
+                         top_k: int = 100) -> List[Dict[str, Any]]:
+        """
+        Query chunks by metadata filters only
+        
+        Args:
+            filter_dict: Metadata filters
+            top_k: Maximum number of results
+            
+        Returns:
+            List of matching chunks
+        """
+        try:
+            logger.info(f"🔍 Querying by metadata: {filter_dict}")
+            
+            # Use a dummy vector for metadata-only search
+            dummy_vector = [0.0] *1024 #* self.dimension
+            
+            search_results = self.index.search(
+                namespace=self.namespace,
+                query={"inputs": {"text": filter_dict['repo_name']}, "top_k": top_k},
+            )
+            
+            
+            # self.index.query(
+            #     vector=dummy_vector,
+            #     namespace=self.namespace,
+            #     top_k=top_k,
+            #     filter=filter_dict,
+            #     include_metadata=True
+            # )
+            
+            results = []
+            if 'result' not in search_results or 'hits' not in search_results['result']:
+                logger.info("⚠️  No results found in search response")
+                return []
+            for match in search_results['result']['hits']:
+                result = {
+                    'id': match['_id'],
+                    'chunk_text': match['fields']['chunk_text'],
+                    'score': float(match['_score']),
+                    'metadata': json.loads(match['fields']['metadata']) #if include_metadata else None
+                }
+                results.append(result)
+            
+            logger.info(f"✅ Found {len(results)} chunks matching metadata filters")
+            return results
+            
+        except Exception as e:
+            logger.info(f"❌ Error querying by metadata: {str(e)}")
+            return []
+    
+    def get_chunk_by_id(self, chunk_id: str) -> Optional[Dict[str, Any]]:
+        """
+        Retrieve a specific chunk by its ID
+        
+        Args:
+            chunk_id: Unique chunk identifier
+            
+        Returns:
+            Chunk data or None if not found
+        """
+        try:
+            result = self.index.fetch(ids=[chunk_id])
+            
+            if chunk_id in result.vectors:
+                vector_data = result.vectors[chunk_id]
+                return {
+                    'id': chunk_id,
+                    'values': vector_data.values,
+                    'metadata': vector_data.metadata
+                }
+            else:
+                logger.info(f"⚠️  Chunk {chunk_id} not found")
+                return None
+                
+        except Exception as e:
+            logger.info(f"❌ Error fetching chunk {chunk_id}: {str(e)}")
+            return None
+    
+    def delete_chunks_by_repo(self, repo_name: str) -> Dict[str, Any]:
+        """
+        Delete all chunks belonging to a specific repository
+        
+        Args:
+            repo_name: Name of the repository to delete
+            
+        Returns:
+            Deletion results
+        """
+        try:
+            logger.info(f"🗑️  Deleting all chunks for repository: {repo_name}")
+            
+            # Query for all chunks from this repo
+            chunks_to_delete = self.query_by_metadata(
+                filter_dict={"repo_name": repo_name},
+                top_k=10000  # High number to get all chunks
+            )
+            
+            if not chunks_to_delete:
+                return {"status": "success", "message": "No chunks found for this repository"}
+            
+            # Extract IDs
+            chunk_ids = [chunk['id'] for chunk in chunks_to_delete]
+            
+            # Delete in batches
+            batch_size = 96
+            deleted_count = 0
+            
+            for i in range(0, len(chunk_ids), batch_size):
+                batch_ids = chunk_ids[i:i + batch_size]
+                
+                try:
+                    delete_response = self.index.delete(ids=batch_ids)
+                    deleted_count += len(batch_ids)
+                    logger.info(f"🗑️  Deleted batch {i//batch_size + 1} ({len(batch_ids)} chunks)")
+                    
+                except Exception as e:
+                    logger.info(f"❌ Error deleting batch: {str(e)}")
+            
+            result = {
+                "status": "success",
+                "deleted_count": deleted_count,
+                "repo_name": repo_name,
+                "timestamp": datetime.now().isoformat()
+            }
+            
+            logger.info(f"✅ Deleted {deleted_count} chunks for repository {repo_name}")
+            return result
+            
+        except Exception as e:
+            logger.info(f"❌ Error deleting chunks for repo {repo_name}: {str(e)}")
+            return {"status": "error", "message": str(e)}
+    
+    def get_index_stats(self) -> Dict[str, Any]:
+        """Get statistics about the Pinecone index"""
+        try:
+            stats = self.index.describe_index_stats()
+            return {
+                "total_vectors": stats.get('total_vector_count', 0),
+                "index_fullness": stats.get('index_fullness', 0),
+                "dimension": stats.get('dimension', self.dimension),
+                "namespaces": stats.get('namespaces', {}),
+                "timestamp": datetime.now().isoformat()
+            }
+        except Exception as e:
+            logger.info(f"❌ Error getting index stats: {str(e)}")
+            return {"error": str(e)}
+    
+    def hybrid_search(self, 
+                     query_text: str,
+                     chunk_types: Optional[List[str]] = None,
+                     repo_names: Optional[List[str]] = None,
+                     file_paths: Optional[List[str]] = None,
+                     top_k: int = 20) -> List[Dict[str, Any]]:
+        """
+        Perform hybrid search using Pinecone's inference API with metadata filters
+        
+        Args:
+            query_text: Text query for semantic search
+            chunk_types: Filter by chunk types (file, class, function, block)
+            repo_names: Filter by repository names
+            file_paths: Filter by specific file paths
+            top_k: Maximum number of results
+            
+        Returns:
+            List of relevant chunks ranked by similarity and filtered by metadata
+        """
+        try:
+            logger.info(f"🔍 Performing hybrid search for: '{query_text[:50]}...'")
+            
+            # Build metadata filter
+            filter_conditions = {}
+            
+            if chunk_types:
+                filter_conditions["chunk_type"] = {"$in": chunk_types}
+            if repo_names:
+                filter_conditions["repo_name"] = {"$in": repo_names}
+            if file_paths:
+                filter_conditions["file_path"] = {"$in": file_paths}
+            
+            # Perform semantic search with filters using inference API
+            results = self.query_similar_chunks(
+                query_text=query_text,
+                top_k=top_k,
+                filter_dict=filter_conditions if filter_conditions else None,
+                include_metadata=True
+            )
+            
+            # Post-process results to add relevance context
+            for result  in results:
+                result['search_type'] = 'hybrid'
+                result['query'] = query_text[:100]
+                logger.debug(f"Result metadata: {result.get('metadata', {})}")
+                result['metadata'] = json.loads(result.get('metadata', '{}'))
+                # Add relevance explanation based on chunk type
+                # logger.debug(f"Result metadata: {json.loads(result.get('metadata', {}))}")
+                chunk_type =  result["metadata"].get("chunk_type", "unknown")
+                if chunk_type == "file":
+                    result['relevance_context'] = 'File-level overview'
+                elif chunk_type == 'class':
+                    result['relevance_context'] = 'Class definition and structure'
+                elif chunk_type == 'function':
+                    result['relevance_context'] = 'Function implementation'
+                elif chunk_type == 'block':
+                    result['relevance_context'] = 'Code block logic'
+            
+            logger.info(f"✅ Hybrid search completed: {len(results)} relevant chunks found")
+            return results
+            
+        except Exception as e:
+            logger.info(f"❌ Error in hybrid search: {str(e)}")
+            return []
+    
+    def get_repository_overview(self, repo_name: str) -> Dict[str, Any]:
+        """
+        Get comprehensive overview of a repository's structure and content
+        
+        Args:
+            repo_name: Name of the repository
+            
+        Returns:
+            Repository overview with statistics and structure
+        """
+        try:
+            logger.info(f"📊 Getting overview for repository: {repo_name}")
+            
+            # Get all chunks for this repository
+            all_chunks = self.query_by_metadata(
+                filter_dict={"repo_name": repo_name},
+                top_k=10000
+            )
+            
+            if not all_chunks:
+                return {"error": f"No chunks found for repository {repo_name}"}
+            
+            # Analyze chunks by type
+            chunk_stats = {}
+            files = set()
+            classes = set()
+            functions = set()
+            languages = set()
+            
+            for chunk in all_chunks:
+                metadata = chunk.get('metadata', {})
+                chunk_type = metadata.get('chunk_type', 'unknown')
+                
+                chunk_stats[chunk_type] = chunk_stats.get(chunk_type, 0) + 1
+                
+                if 'file_path' in metadata:
+                    files.add(metadata['file_path'])
+                if 'language' in metadata:
+                    languages.add(metadata['language'])
+                if 'class_name' in metadata and metadata['class_name']:
+                    classes.add(metadata['class_name'])
+                if 'function_name' in metadata and metadata['function_name']:
+                    functions.add(metadata['function_name'])
+            
+            overview = {
+                "repo_name": repo_name,
+                "total_chunks": len(all_chunks),
+                "chunk_distribution": chunk_stats,
+                "files_count": len(files),
+                "classes_count": len(classes),
+                "functions_count": len(functions),
+                "languages": list(languages),
+                "sample_files": list(files)[:10],  # Show first 10 files
+                "sample_classes": list(classes)[:10],  # Show first 10 classes
+                "timestamp": datetime.now().isoformat()
+            }
+            
+            logger.info(f"✅ Repository overview generated for {repo_name}")
+            return overview
+            
+        except Exception as e:
+            logger.info(f"❌ Error getting repository overview: {str(e)}")
+            return {"error": str(e)}
+    
+    def cleanup_old_chunks(self, days_old: int = 30) -> Dict[str, Any]:
+        """
+        Clean up old chunks based on timestamp
+        
+        Args:
+            days_old: Delete chunks older than this many days
+            
+        Returns:
+            Cleanup results
+        """
+        # This would require storing timestamps in metadata and querying by date
+        # Implementation depends on your specific cleanup needs
+        logger.info(f"🧹 Cleanup functionality not implemented yet")
+        return {"status": "not_implemented"}