← All Articles
May 7, 202612 min readStreaming

Adaptive Kafka Consumer Concurrency with Lag Feedback

Dynamic worker scaling for consumers driven by lag slope and handler latency, with safeguards to avoid rebalance thrash.

Static consumer concurrency is almost always wrong by lunchtime.

Feedback Inputs

  • partition lag delta
  • handler p95
  • broker throttle signals
if (lagSlope > HIGH && p95 < 100) workers += 2;
if (p95 > 300) workers = Math.max(1, workers - 1);

Scale conservatively to avoid rebalance storms.

Next ArticleSLO Burn-Rate Alerting That Reduces NoiseReliability · 11 min read