Hi all, I have a query related to KafkaStreams with multiple brokers. In versions 2.7.x it says that KafkaStreams does not support multiple brokers and will be addressed in future version. The current version is 3.9.x, has it addressed this feature?
I guess you mean multiple clusters?
There is still no multi-cluster support.
But given that you ask, it seems you are interested? Would be great if you would comment on the ticket expressing interest. That is how tickets are prioritized.
Where is it written it’ll be supported in the future? For the comments written in the ticket, Flink or Spark would be better
Where is it written it’ll be supported in the future?
There was a sentence about this in the docs at some point, as we originally intended to support this much earlier. But demand was not super strong. We remove this statement from the docs with 2.8 release.
Tip:
Kafka Streams applications can only communicate with a single Kafka cluster specified by this config value. Future versions of Kafka Streams will support connecting to different Kafka clusters for reading input streams and writing output streams.
For the comments written in the ticket, Flink or Spark would be better
Well, right now yes, as KS does not support in. But in general, I don’t see why Flink or Spark would be better?
Or you use MirrorMaker (or something similar) to replicate the result topic to the target cluster.
AFAIK, Flink/Spark handle EOS on their own and can poll, operate, and produce into N different clusters from one app, rather than simply mirror
Sure. They support this. – As said: right now, using Flink/Spark for cross cluster is a good option; agreed on this.
Maybe I did read your reply incorrectly, but I though you mean, using KS is never a good idea for this case – I and don’t see why you thing so? If we would add this feature to KS, it should be equally good as Flink/Spark for most cases (depending how we would build this feature – KS might not be able to easily support N input cluster, and M output clusters…)
I’ve seen people write in a KafkaProducer into the Processor API (or even forEach DSL), so there are obvious workarounds. I didn’t imply never, just that there are more suitable alternatives, should those be available
Yes, that’s a crude workaround
Mirrormaker seems like a viable option together with Kafka
