src/orchestrator/README.md

Stock Alchemist Orchestrator

Overview

The orchestrator is the central nervous system of Stock Alchemist, coordinating all components to work together seamlessly.

Architecture

Singleton Managers

All major components use thread-safe singleton patterns:

1. DatabaseManager: Manages all database operations

   • Provides unified access to calendar, news, and fundamental adapters
   • Thread-safe database connections

2. NewsScraperManager: Manages continuous news scraping

   • Alpaca WebSocket connection for real-time news
   • News processing pipeline with sentiment analysis
   • Automatic database storage

3. CalendarScraperManager: Manages daily calendar scraping

   • Fetches earnings, IPOs, stock splits, dividends, economic events
   • Runs once per day
   • Queues relevant tickers for analysis

4. FundamentalAnalysisManager: Manages on-demand analysis

   • Caches recent analyses (30-day validity)
   • Processes tickers from queue
   • Stores results with timestamps

Thread Communication

The coordinator manages 4 worker threads:

1. News Scraper Thread (Continuous)

   • Receives real-time news from Alpaca
   • Filters by sentiment (non-NEUTRAL)
   • Triggers fundamental analysis for relevant tickers
   • Saves all news to database

2. Calendar Scraper Thread (Daily)

   • Runs once every 24 hours
   • Fetches all calendar events for today
   • Saves events to database
   • Queues tickers for fundamental analysis

3. Fundamental Analysis Thread (On-Demand)

   • Processes tickers from analysis queue
   • Checks if analysis needed (>30 days old or missing)
   • Performs comprehensive stock analysis
   • Saves results with last_processed_datetime

4. Data Aggregation Thread (Periodic)

   • Runs every 5 minutes
   • Generates JSON overview files per ticker
   • Combines news, fundamentals, and calendar events

Data Flow

┌─────────────────┐
│  Alpaca News WS │ (Real-time)
└────────┬────────┘
         │
         v
┌────────────────────┐     ┌──────────────────┐
│ News Processor     │────>│ News Database    │
│ (Sentiment Filter) │     └──────────────────┘
└────────┬───────────┘
         │ (Non-NEUTRAL sentiment)
         v
┌────────────────────┐
│ Analysis Queue     │
└────────┬───────────┘
         │
         v
┌────────────────────┐     ┌──────────────────┐
│ Fundamental        │────>│ Analysis DB      │
│ Analysis Worker    │     │ (with timestamp) │
└────────────────────┘     └──────────────────┘
         │
         v
┌────────────────────┐
│ JSON Aggregator    │
│ (Per Ticker)       │
└────────────────────┘

┌─────────────────┐
│  NASDAQ API     │ (Daily)
└────────┬────────┘
         │
         v
┌────────────────────┐     ┌──────────────────┐
│ Calendar Scraper   │────>│ Calendar DB      │
│ (Earnings, IPOs,   │     └──────────────────┘
│  Splits, Divs)     │              │
└────────┬───────────┘              │
         │                          │
         └──────────>Analysis Queue<┘

Output Format

Ticker Overview JSON

Location: output/ticker_overviews/{TICKER}.json

{
  "ticker": "AAPL",
  "generated_at": "2025-11-10T12:00:00",
  "news": {
    "news_1": {
      "headline": "Apple announces new iPhone",
      "sentiment": "POSITIVE",
      "date": "2025-11-10"
    },
    "news_2": {
      "headline": "Apple stock downgraded",
      "sentiment": "NEGATIVE",
      "date": "2025-11-09"
    }
  },
  "fundamental_analysis": {
    "json_hash_reference": "a1b2c3d4e5f6...",
    "date_updated": "2025-11-10T10:00:00",
    "result": {
      "recommendation": "BUY",
      "final_score": 85,
      "confidence": "HIGH",
      "key_metrics": {...}
    }
  },
  "calendar_events": {
    "event_1": {
      "type": "earnings",
      "date": "2025-11-15",
      "data": {
        "eps_estimate": "1.50",
        "time": "after_close"
      },
      "importance": "high"
    }
  }
}

Database Schema

Fundamental Analysis Table

date: ISO date (YYYY-MM-DD)
data_type: "fundamental_analysis"
ticker: Stock symbol
data: {
    "analysis_type": "complete",
    "last_processed_datetime": "2025-11-10T12:00:00.123456",
    "recommendation": "BUY|SELL|HOLD",
    "final_score": 0-100,
    "confidence": "HIGH|MEDIUM|LOW",
    ...
}
created_at: ISO timestamp
updated_at: ISO timestamp
expiry_date: ISO date (30 days from creation)

Running the Orchestrator

From Project Root

python main.py

From Orchestrator Directory

cd src/orchestrator
python coordinator.py

Configuration

Timeouts and Intervals

• Calendar Scraping: Every 24 hours
• Analysis Cache: 30 days validity
• News Storage: 30 days TTL
• Economic Events: 7 days TTL
• Aggregation: Every 5 minutes

Queues

• news_queue: Real-time news items
• analysis_queue: Tickers needing fundamental analysis
• calendar_queue: Calendar events for processing

Error Handling

All threads have exception handling:

• Errors logged to console
• Threads continue running
• Failed operations don't crash system

Graceful Shutdown

Press Ctrl+C to initiate graceful shutdown:

1. Stops accepting new news
2. Finishes current analyses
3. Closes database connections
4. Terminates all threads

Monitoring

Console output shows:

• ✅ Component initialization
• 📰 News items received
• 📅 Calendar events fetched
• 🔍 Stocks being analyzed
• 📊 Aggregation runs
• ❌ Errors and warnings

Dependencies

• threading: Thread management
• queue: Inter-thread communication
• datetime: Timestamps
• json: Data serialization
• All Stock Alchemist modules

Future Enhancements

1. Web Dashboard: Real-time monitoring UI
2. Alerting System: Email/SMS for important events
3. ML Integration: Sentiment improvement
4. Backtesting: Historical performance analysis
5. API Server: REST API for external access