Building a Golang Redis Stream Consumer

At CoinSwitch, our Order Management Service needed to process messages in real-time with sub-15ms latency. Amazon SQS was too slow — adding 50-100ms of overhead per message. So I built a stateless stream consumer library on top of Redis Streams that achieved a 92% reduction in retrieval latency.

The Problem

In a crypto exchange, order state changes need to propagate instantly. When a user places a buy order, the system needs to validate, match, and confirm within milliseconds. Our SQS-based pipeline was adding unacceptable delays.

Architecture

  ┌──────────────┐     ┌──────────────────┐     ┌──────────────┐
  │  Order API   │────▶│   Redis Stream    │────▶│  Consumer    │
  │  (Producer)  │     │  (Message Queue)  │     │  (Golang)    │
  └──────────────┘     └──────────────────┘     └──────┬───────┘
                                                       │
                              ┌─────────────────────────┤
                              │                         │
                       ┌──────▼──────┐          ┌──────▼───────┐
                       │  Handler 1  │          │  Handler 2   │
                       │  (Validate) │          │  (Execute)   │
                       └─────────────┘          └──────────────┘

Key Design Decisions

Stateless Consumer Groups

Each consumer instance is stateless — no local state, no coordination needed. Redis Stream consumer groups handle message distribution and acknowledgment natively. This means horizontal scaling is just spinning up more pods.

Batch Reading with XREADGROUP

Instead of reading one message at a time, the consumer reads in configurable batches. A batch of 10 messages takes roughly the same time as reading 1, giving us massive throughput gains.

Graceful Shutdown & Pending Recovery

On restart, the consumer first processes any pending (unacknowledged) messages from the PEL (Pending Entries List) before consuming new ones. This ensures exactly-once processing semantics without external coordination.

Results

  Metric                    SQS          Redis Stream
  ─────────────────────────────────────────────────────
  Avg Latency              80-120ms      10-15ms
  P99 Latency              250ms         25ms
  Throughput               ~1000 msg/s   ~15000 msg/s
  Cost (monthly)           $$$           $ (existing Redis)

92% reduction in message retrieval latency. The library is now used across multiple services at CoinSwitch for real-time event processing.