Kafka - how to minimize lag?


We have been collecting around 1 billion events events from all the country in every day and all the events are sent to only one topic with 150 partition. While processing the events there is a lag and we cannot optimize the processing the events payload more since it is very simple. Every events has city fields to set which city it is sent to our system. Therefore, I can use this fields to send a city related topic and every city related topic has its own consumer or I can increase the partition size of the main topic, which is currently 150 in order to increase the consume speed and eradicate the lag. We have 5 broker and the replication factor is 5. Our main go is to overcome the lag so we are open to use more memory. Which method do you recommend us to apply? Thanks a lot.

Memory won’t solve lag. More/faster consumers will

How many consumer per consumer group do you have?. You can have up to 150 consumers per your consumer group to speed up receiving and processing your events

If cities are “assigned” specific partitions, then there is no main difference in specific topics vs. what you have now, as far as consumption is concerned unless you want find lag per city

. Agreed. If Iso still has issue with 150 consumer then the solution is to increase number of partitions in order to add more consumers

Thanks a lot for your attention. We have 5 consumer for the topic with 150 partition. we do not want to find the lag per city, we just want to consume, process and write the main topic as soon as possible. In order to do that we thought that it would be better to increase the parallelism while consuming.

Then we better increase partition number as mentioned, there is no main difference between city-separeted topic and the current topic.

As mentioned, to fix lag quickest, you can have up to 150 consumers in that group