Using transactional producer without transactions for fencing in k8s

DevOps Kafka
1

Is it possible to use a transactional producer without transactions? The goal is to have the producer get fenced if another (newer) instance pops up on another k8s pod. But with this particular producer, I won’t ever have a transaction with more than one related message, so I wouldn’t want the overhead of having to call commit() every time I produce with it.

2

Alternatively, I could have a transactional producer, and do things like:

producer.send(oneRecord, myCallback);
producer.commit();```
But there would be multiple threads producing at once.

Would all of those commit()'s cause a big performance hit?
3

Here’s another question—can multiple transactions be batched into one network request from the producer to the broker(s)?

Edit: it turns out they can’t, purusuant to: https://kafka.apache.org/23/javadoc/org/apache/kafka/clients/producer/KafkaProducer.html#commitTransaction–
> This method will flush any unsent records before actually committing the transaction.

4

Sounds like you want a distributed lock, not just a transaction boundary

5

A distributed lock would work, but introducing any systems outside of Kafka is not an option.

6

Zookeeper? https://programmer.ink/think/implementation-of-curator-distributed-lock.html

7

That’s not in Kafka anymore :wink:

8

The transactional producer solves a problem that is a superset of distributed lock on being able to produce with a certain ID. It just may be overkill