Migrating Trails from Monolith to Microservices
At Trell, Trails (short videos up to 3 minutes) was part of a monolithic Python/Django application. As the platform scaled to millions of uploads, the monolith became a bottleneck. I led the redesign into a Go-based microservices architecture.
Before: The Monolith
┌─────────────────────────────────────────────┐
│ Django Monolith │
│ │
│ ┌─────────┐ ┌─────────┐ ┌───────────────┐ │
│ │ Auth │ │ Feed │ │ Video Upload │ │
│ │ │ │ │ │ + Processing │ │
│ └─────────┘ └─────────┘ └───────────────┘ │
│ ┌─────────┐ ┌─────────┐ ┌───────────────┐ │
│ │ Users │ │ Search │ │ Transcode │ │
│ └─────────┘ └─────────┘ └───────────────┘ │
└─────────────────────────────────────────────┘
│
Single DB
After: Microservices
Upload ──▶ ┌─────────────┐ ┌──────────────┐
Request │ API GW │───▶│ Upload Svc │
│ (Golang) │ │ (Golang) │
└─────────────┘ └──────┬───────┘
│
┌──────▼───────┐
│ Kafka Queue │
└──────┬───────┘
│
┌────────────┼────────────┐
│ │ │
┌──────▼──┐ ┌─────▼────┐ ┌───▼──────┐
│ Chunk 1 │ │ Chunk 2 │ │ Chunk 3 │
│ Process │ │ Process │ │ Process │
└────┬────┘ └────┬─────┘ └────┬─────┘
│ │ │
└────────────┼──────────────┘
│
┌──────▼───────┐
│ Merge & │
│ Transcode │
└──────┬───────┘
│
┌──────▼───────┐
│ CDN │
└──────────────┘
Video Chunking Strategy
The key insight was splitting uploaded videos into chunks and processing them concurrently. A 3-minute video gets split into ~18 ten-second chunks. Each chunk is processed in parallel on separate workers, then merged back. This gave us the 35% reduction in processing time.
Kafka for Sequential Ordering
Upload events are keyed by user ID and sent to Kafka. This ensures that if a user uploads multiple videos, they're processed in order. But videos from different users run in parallel across consumer group members.
Results
Metric Before After ───────────────────────────────────────────────── Upload Latency ~8s ~6.5s (-18%) Video Processing ~45s ~29s (-35%) System Responsiveness Baseline +20% Upload Efficiency Baseline +25%