A question regrading mirrormaker2, we are exploring mirror maker 2 to cross region data replication to replace our old mirror maker 1 setup. From a quick glance at the doc, looks like there is no way to ‘dynamically’ change topic list, if I want to add a new topic(without using the wildcard), I have to restart the mirrormaker 2 process. Is there anyway update the topic config without restart the process(connector)? We are trying the mirrormaker standalone cluster.
No. Unfortunately there isn’t a way to dynamically change the topic list
If I run a Kafka connect cluster, will the connect API make it more flexible? Any API I can update connector? Looks like it has to restart a connector with any configuration update anyway right?
Won’t be any more flexible. Dynamic config updates aren’t really a connect thing right now
I am asking since right now we are using some in house mirror maker logic that when the list of topic change, we only do a consumer.subscribe to the new list of topic. So it’s very minimal interruption to the running system other than a consumer group rebalance.
So only way for us now is to just stop connector and restart it with new config
You have any recommendation for how to deploy mirrormaker 2 in our case? we have some big cluster with a lot topic, so if user wants to add a new topic to the mirrormaker list, we have to restart all of mirrormaker/connect instances? Trying to minimize the overhead here.
Or we can do a topic level mirrormaker deployment, need some orchestration work on our side.
And/or use multiple instances of mm2
So the restart doesn’t hurt so bad
So the restart doesn't hurt so bad -> I think this depends on the throughput and the number of instances we are running. If we update all of them with one config it will be a wave of consume group rebalance. I think we will try multiple instances of mm2 first, so some isolation for different topics. Thanks for the help
