[jira] [Commented] (KAFKA-13326) Add multi-cluster support to Kafka Streams

["Matthias J. Sax (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-13326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17424298#comment-17424298 ] 

Matthias J. Sax commented on KAFKA-13326:
-----------------------------------------

Neither mirror-maker2 not replicator support EOS semantics while Kafka Streams does. EOS is based on Kafka's transaction that don't work cross-cluster.

Both, mirror-maker2 and replicator are "simple" copy-data tools, while Kafka Streams, that does processing, is much more complicated. For example, Kafka Streams might create additional topics to repartition data for processing, or create changelog topics for stateful processing. If there are two clusters, it's unclear in which those topics should be created. Also, it would be required not have a single consumer and producer, but to have multiple to read/write to both clusters. There is also complications with consumer group management if there are two cluster the application would connect to, as it's unclear which of both is really in charge (split brain problem).

While it's not impossible to built, it's rather complex.

> Add multi-cluster support to Kafka Streams
> ------------------------------------------
>
>                 Key: KAFKA-13326
>                 URL: https://issues.apache.org/jira/browse/KAFKA-13326
>             Project: Kafka
>          Issue Type: New Feature
>          Components: streams
>            Reporter: Guangyuan Wang
>            Priority: Major
>              Labels: needs-kip
>
> Dear Kafka Team,
> According to the link, https://kafka.apache.org/28/documentation/streams/developer-guide/config-streams.html#bootstrap-servers.
> Kafka Streams applications can only communicate with a single Kafka cluster specified by this config value. Future versions of Kafka Streams will support connecting to different Kafka clusters for reading input streams and writing output streams.
> Which version will this feature be added in the Kafka stream?  This is really a very good feature.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)