Hello, I have created 3 node cluster and started producing data and I m able consume it at all the nodes. But the thing is If I make two systems down my third system is throwing exception. Can I know what is the reason behind this and how to come out of this issue.
I want to create a cluster and in case if any two nodes goes down also, my third node should take care of the all the other two nodes work. How to achieve this?
You would have to either not publishing with acks=ALL or you would need to insure your min.isr=1.
while it is possible, it is not recommended. Typically if your system is running with just 1 broker (there is no replication) and you can run into a data loss. If you have 4 brokers you could do replication factor of 4 with a min.isr=2 which would give you durability with the desired availability you want (handle 2 brokers being down).
Now, if you are going down the path of wanting the system to function with 1 broker, you have to make sure your __consumer_offsets topic is also configured to have a min.isr setting of 1 (as well as others such as the transaction topic if you are using exactly once semantics).
The other thing is you are now expecting 1 instance to be able to handle the work of 3 — if it is being over provisioned you will most run into additional issues quickly.
Something got misconfigured. This can happen if broker settings that need to be the same are not and a topic is created. Or could be due to a reparation allocation.