Is it right thing to enable idempotent at producer level ?
Well, it depends. Does the producer write the same message twice? Or does the consumer read a single message two times?
For the first case, enabling idempotent producer may help (if the duplicates are introduced via internal producer retries).
Hey Thankyou , to your 1st question : I could see producer writes single message to Kafka topic , but within the topic , message is being posted to mutiple partitions and this is happening once in a while not everytime . Because of which for one message , duplicates are seen by consumer
> message is being posted to mutiple partitions
That is weird… Even if the producer retries internally, the duplicate should end up in the same partition (from my understanding).
Anyway: if the producer writes duplicates, enabling idempotente write may help to resolve the issue (it does not guard against producer crashes though)
But anything can be done at the topic/partition layer ?
Not sure if I understand the question? If the producer does write a duplicate, Kafka broker cannot know that it is a duplicate, so I guess the answer is “no” (if I understand the question correctly).
Producer is not writing duplicate 
once it is posted topic , due to tech or other server issues , one message is being sent to mutiple partitions
Sounds like a duplicate to me?
But I am still wondering, why the duplicate show up in a different partition.
If the producer retries internally, it would pick the partition first, and not change it later.
Exactly even am trying to understand at the topic layer
Thus, it sounds like that you might call send() twice?
So even if i enable idempotent at producer side , not much use right
I can see retry count in producer tables is 1
If you indeed call send() twice, than idempotent producer won’t helpl.
Where this send() twice ?
Let’s assume you call producer.send() and the write itself is successful, but your producer-client app crashed before the send() is acked. On restart/recovery of your producer-client, you might send the same message a second time by calling producer.send() again (because you app crashed before it could process the original ack from the broker).
Okay so you Mean in my producer code send () should be only once ?
When you call producer.send() the second time, and your message does not have a key (and thus, round-robin partitioning is used), a different partition could be picked.
But where to configure this ?
