Whenever the consumer lag on a Kafka topic spikes, the automatic reflex is pressing the “scale out” button. A beefier instance must process faster, and ten instances must do ten times the work, right? Well, not always.
This occurs when data is distributed unevenly across the partitions of a topic. Instead of a balanced workload, one or a few partitions receive a significantly higher volume of messages compared to others.
The most common cause is a message key with low cardinality. This often happens when using
To detect skew, monitor consumer lag across the group. A clear indicator of partition skew is when the maximum lag is significantly higher than the average lag, indicating that one consumer is struggling with a disproportionate workload.
Another clear indicator is disproportionate CPU usage. If you notice one consumer’s CPU usage is high, while the others are barely touched, there’s likely a bottleneck. This usually means the affected instance is struggling to keep up with a massive wave of messages, while the rest of the group is idling.
By now it’s probably clear that by adding more consumers the issue will remain, and it will remain as long as the messages are processed by a single consumer. Luckily the solution is pretty straightforward: improve the key cardinality.
This can be achieved by
A rebalance occurs when the group coordinator redistributes partition assignments among consumers. While this is a normal process used to handle scaling or recover from failed consumers, it becomes a "storm" when the group enters a constant, failing loop of reassignments.
During a rebalance, consumers typically stop fetching data. If the group loops through rebalances continuously, the entire consumer group stops processing altogether, turning a routine task into a "stop-the-world" event.
While on Partition Skew the culprit is a bad key selection, a Rebalancing Storm is most likely due to a bad configuration on the consumers properties. Although we want to detect failed consumers quickly, we must give them enough time between polls to actually process their data.
The first clue is an increasing consumer lag across the entire topic, rather than just a single partition.
Another metric we should keep an eye on is the group state. A healthy group stays Stable. If you see a constant switch between PreparingRebalance and CompletingRebalance, you are likely in the middle of a storm.
Finally, we must remain attentive of the logs, either on the broker or at your application, there are multiple hints you about any constant rebalance, these should resemble something like:
Scaling out isn't just ineffective here, it can actually make things worse. Adding more consumers to a Rebalancing Storm adds extra overhead for the coordinator, increases the total number of heartbeats, and leads to more frequent context switching. Instead of throwing more hardware at the problem, you should tune your consumer properties to ensure stability.
If your partitions are balanced and your group is stable, but the lag keeps climbing despite scaling out, you must be facing an issue within the consumer process itself.
There are countless potential causes, but a slow response from an external resource is often the root cause behind poor performance. External API calls, operations on a struggling database or cache, or even excessive logging, can add latency on every loop of the process contributing to the overall lag.
The main difference between this issue and the previous ones is that the causes and the mechanisms to detect them are nearly limitless. However, some good indicators are:
Adding more consumers, once again, could worsen the situation. A saturated database will struggle even more if more operations are executed simultaneously, and a rate-limited API returning 429 errors will not process more events just because you added more workers. Although there are a few cases where scaling up could help, there’s another factor to have in mind, the maximum number of consumers can’t be larger than the number of partitions.
In this scenario, scaling up is not the solution, nor is tweaking the consumer properties, or scrambling keys. The answer lies in diving into the code to improve performance. This could mean batching database/API operations, optimizing queries by adding indices, or using an internal thread pool to increase parallelism within the consumer itself.
We have gone through several scenarios where scaling out is not the answer when dealing with Kafka consumer lag. While adding more instances is often the easiest thing to do, it rarely addresses the underlying issues. Whether you are facing partition skew, a rebalancing storm, or inefficient processing logic, the solution requires a deeper look at your metrics, configuration, and code rather than just your infrastructure size.
By diagnosing the root cause first, you avoid scaling “just because” and ensure your system remains stable, efficient, and cost-effective.
