repos/Laddr/docs/migrations.md

Kafka Migration Guide

This document describes the migration from Redis to Kafka as the message broker for the Laddr multi-agent system.
Kafka provides persistent messaging, horizontal scaling, and higher throughput for production workloads.

---

Overview

Kafka offers durable, highly scalable messaging suitable for production.
This guide walks through architecture changes, configuration updates, deployment, verification, and rollback steps to migrate from Redis to Kafka.

---

Architecture Changes

Before (Redis)

• Message broker: Redis (pub/sub)
• Persistence: Ephemeral (lost on restart)
• Scaling: Limited by single instance
• Use case: Development and lightweight deployments

After (Kafka)

• Message broker: Apache Kafka with Zookeeper
• Persistence: Durable with configurable retention
• Scaling: Horizontal with partitions and consumer groups
• Use case: Production deployments with high throughput

Topic Structure

• laddr.tasks.<agent_name> – Task queue for each agent
• laddr.responses – Response messages

Consumer groups follow the pattern:
laddr-<agent_name>-workers

---

Configuration Changes

Environment Variables (.env)

# Message Broker Configuration (Kafka)
KAFKA_BOOTSTRAP=kafka:9092
QUEUE_BACKEND=kafka

# Previous Redis configuration (commented out)
# REDIS_URL=redis://redis:6379/0
# QUEUE_BACKEND=redis

Note:
Use kafka:9092 for client connections; kafka:29092 is for inter-broker communication only.

Docker Compose Services

• Zookeeper – Port 2181
• Kafka – Ports 9092 (client), 9101 (metrics)
• Kafka UI – Port 8080 (http://localhost:8080)

---

Deployment Instructions

Prerequisites

Install the Kafka dependencies:

pip install aiokafka>=0.11.0
# OR
pip install laddr[kafka]

Starting the System

cd /path/to/your/project
docker-compose down
docker-compose up -d

Monitor system status:

docker-compose ps
docker-compose logs -f kafka

---

Verification Steps

Check Kafka Topics

docker-compose exec kafka kafka-topics --list --bootstrap-server localhost:9092

Expected topics:

• laddr.tasks.coordinator
• laddr.tasks.researcher
• laddr.tasks.analyzer
• laddr.tasks.writer
• laddr.tasks.validator
• laddr.responses

Monitor Worker Logs

docker-compose logs -f coordinator_worker researcher_worker analyzer_worker writer_worker validator_worker

Test via API

curl -X POST http://localhost:8000/api/v1/run \
  -H "Content-Type: application/json" \
  -d '{
    "workflow_name": "test_workflow",
    "initial_task": {
      "agent": "researcher",
      "inputs": {"query": "test"}
    }
  }'

---

Scaling

To scale workers horizontally:

docker-compose up -d --scale coordinator_worker=3 --scale researcher_worker=3 --scale analyzer_worker=2 --scale writer_worker=2 --scale validator_worker=2

Kafka automatically balances partitions across consumers, enabling seamless horizontal scaling.

---

Rollback to Redis

REDIS_URL=redis://redis:6379/0
QUEUE_BACKEND=redis
# Comment out Kafka config
# KAFKA_BOOTSTRAP=kafka:9092

Update your docker-compose.yml to re-enable Redis, remove Kafka and Zookeeper, then restart:

docker-compose down && docker-compose up -d

---

Summary

• Kafka improves reliability, scaling, and persistence.
• Docker Compose now includes Kafka, Zookeeper, and Kafka UI.
• Redis remains available for quick rollback.

After migration, access Kafka UI at:
http://localhost:8080
to monitor system health and performance.