Understanding Kafka consumer rebalancing behavior during message processing in Java application

DevOps Kafka
#1

does kafka rebalancing of consumer matters on the poll we are calling from the application. I have 4 consumers for 4 different topics running in a java application and each of them are polling 500 messages and keeping them in memory for processing.

Another thread, reading these already fetched message one by one from all 4 consumer in round robin manner and once all 2000 messages are processed, the poll for the consumers is called again.

During this process, i see that kafka consumer rebalancing is happening and no messages are consumed for 3-4 minutes and then it resumes. This is happening very frequently, I have verified that
• My consumers are not increasing or decreasing - they are fixed
• No partitions are added/removed during the run
• The application CPU usage is minimal. Around 30% in both the pods
As per my assumption, rebalance should not rely on when i am calling the poll. Instead, heartbeat interval will tell the broker that consumer is alive and avoid rebalance. Please clarify this and help me on this understanding.

#2

heartbeat is needed before session timeout to avoid a rebalance, but a poll must also happen prior to <http://max.poll.interval.ms|max.poll.interval.ms> to also avoid a rebalance (the default is 5 minutes)

lets say the consumer processing the poll() is stuck (infinite loop, for example), if heartbeat was the only means to measure consumption, then this consumer would never consume again. So to ensure that the actual thread reading from kafka remains healthy.

#3

Why are you using 4 consumer and one processing threads? Sound unnecessarily complex?

Why not just have one threads, which just polls on all 4 topics and does processing, too?

#4

, its old architecture we are supporting. We are migrating to Kafka as part of infra changes.

#5

, i am calling the poll within this http://max.poll.interval.ms|max.poll.interval.ms window (from logs, i see every minute) but not getting any records even if i see the lag exists for the topic. In describe consumer group, i see the warn that consumer group is rebalancing.

#6

I would need to see more specifics at this point.

I assumed you are using the java kafka-clients.jar, but maybe this isn’t Java — if so, please provide specifics. If java, include any 3rd party libraries/frameworks you might be using.

#7

yes, i am using kafka-clients-3.5.1.jar. No frameworks, just simple consumers created. However, i observed that my code was creating multiple consumers but the actual parititions are 16 only.

in this case, there were many idle consumers which were created from the code. I have changed the logic and tried with singleton consumer - which means if the consumer is already created for a topic, it will not create another instead return the same.

After this, i don’t see rebalancing is happening however, i am concerned about using same consumer across different threads of my application because i read that the recommendation is using one consumer one thread. as this class in not thread safe and could lead to concurrency issues.

#8

yes, if you have one consumer, but you want multiple threads doing processing it becomes more complicated.

I have built apps that uses AdminClient to find the # of partitions + determines the number of cores on the machine and then create the # of consumers based on limit — I’ve had no problem with this working w/out any rebalancing going on.

Are you sharing a consumer group with another client (same group.id, different topic) ?

make sure you have a rebalance listener in your consumer and log the rebalancing. might help troubleshoot.

#9

no, i am not sharing the same consumer with other clients

#10

will check this rebalance-listener