File size: 3,719 Bytes
4e909c7 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 |
# Kafka Migration Guide
This document describes the migration from **Redis** to **Kafka** as the message broker for the Laddr multi-agent system.
Kafka provides persistent messaging, horizontal scaling, and higher throughput for production workloads.
---
## Overview
Kafka offers durable, highly scalable messaging suitable for production.
This guide walks through **architecture changes**, **configuration updates**, **deployment**, **verification**, and **rollback** steps to migrate from Redis to Kafka.
---
## Architecture Changes
### Before (Redis)
- Message broker: Redis (pub/sub)
- Persistence: Ephemeral (lost on restart)
- Scaling: Limited by single instance
- Use case: Development and lightweight deployments
### After (Kafka)
- Message broker: Apache Kafka with Zookeeper
- Persistence: Durable with configurable retention
- Scaling: Horizontal with partitions and consumer groups
- Use case: Production deployments with high throughput
### Topic Structure
- `laddr.tasks.<agent_name>` – Task queue for each agent
- `laddr.responses` – Response messages
Consumer groups follow the pattern:
`laddr-<agent_name>-workers`
---
## Configuration Changes
### Environment Variables (`.env`)
```bash
# Message Broker Configuration (Kafka)
KAFKA_BOOTSTRAP=kafka:9092
QUEUE_BACKEND=kafka
# Previous Redis configuration (commented out)
# REDIS_URL=redis://redis:6379/0
# QUEUE_BACKEND=redis
```
> **Note:**
> Use `kafka:9092` for client connections; `kafka:29092` is for inter-broker communication only.
### Docker Compose Services
- **Zookeeper** – Port 2181
- **Kafka** – Ports 9092 (client), 9101 (metrics)
- **Kafka UI** – Port 8080 (`http://localhost:8080`)
---
## Deployment Instructions
### Prerequisites
Install the Kafka dependencies:
```bash
pip install aiokafka>=0.11.0
# OR
pip install laddr[kafka]
```
### Starting the System
```bash
cd /path/to/your/project
docker-compose down
docker-compose up -d
```
Monitor system status:
```bash
docker-compose ps
docker-compose logs -f kafka
```
---
## Verification Steps
### Check Kafka Topics
```bash
docker-compose exec kafka kafka-topics --list --bootstrap-server localhost:9092
```
Expected topics:
- `laddr.tasks.coordinator`
- `laddr.tasks.researcher`
- `laddr.tasks.analyzer`
- `laddr.tasks.writer`
- `laddr.tasks.validator`
- `laddr.responses`
### Monitor Worker Logs
```bash
docker-compose logs -f coordinator_worker researcher_worker analyzer_worker writer_worker validator_worker
```
### Test via API
```bash
curl -X POST http://localhost:8000/api/v1/run \
-H "Content-Type: application/json" \
-d '{
"workflow_name": "test_workflow",
"initial_task": {
"agent": "researcher",
"inputs": {"query": "test"}
}
}'
```
---
## Scaling
To scale workers horizontally:
```bash
docker-compose up -d --scale coordinator_worker=3 --scale researcher_worker=3 --scale analyzer_worker=2 --scale writer_worker=2 --scale validator_worker=2
```
Kafka automatically balances partitions across consumers, enabling seamless horizontal scaling.
---
## Rollback to Redis
```bash
REDIS_URL=redis://redis:6379/0
QUEUE_BACKEND=redis
# Comment out Kafka config
# KAFKA_BOOTSTRAP=kafka:9092
```
Update your `docker-compose.yml` to re-enable Redis, remove Kafka and Zookeeper, then restart:
```bash
docker-compose down && docker-compose up -d
```
---
## Summary
- Kafka improves **reliability**, **scaling**, and **persistence**.
- Docker Compose now includes **Kafka**, **Zookeeper**, and **Kafka UI**.
- Redis remains available for quick rollback.
After migration, access Kafka UI at:
`http://localhost:8080`
to monitor system health and performance.
|