You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Guozhang Wang (Jira)" <ji...@apache.org> on 2021/05/20 05:32:00 UTC
[jira] [Commented] (KAFKA-12743) [Kafka Streams] - cluster failover
for stateful Kafka Streams applications
[ https://issues.apache.org/jira/browse/KAFKA-12743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17348063#comment-17348063 ]
Guozhang Wang commented on KAFKA-12743:
---------------------------------------
I think the statement should be "Streams does not have a good integration with MM2 for DR story yet", not that "Streams does not have a good DR story" :) There are other sync / async mechanisms for replication data between Kafka clusters which do not have prefix on the destination cluster's topic names, which have been used nicely with Streams for a while.
As for MM2, I think I agree that we can improve Streams' integration with this replicator better, for example, Streams can have a specific `mirror-maker-failover-enabled` config which 1) would expect the internal topic names to be prefixed while 2) still use the same topic names when fetching from committed offset topic, upon failover to the new cluster.
> [Kafka Streams] - cluster failover for stateful Kafka Streams applications
> --------------------------------------------------------------------------
>
> Key: KAFKA-12743
> URL: https://issues.apache.org/jira/browse/KAFKA-12743
> Project: Kafka
> Issue Type: Improvement
> Components: mirrormaker, streams
> Affects Versions: 2.8.0
> Reporter: Sergey Zyrianov
> Priority: Major
>
> Currently, when working with Kafka backed state stores in Kafka Streams, these log compacted topics are given a hardcoded name : _app_id-storename-changelog_
> {noformat}
> public static String storeChangelogTopic(String applicationId, String storeName) {
> return applicationId + "-" + storeName + STATE_CHANGELOG_TOPIC_SUFFIX;
> }{noformat}
>
> MirrorMaker2(mm2) copies these topics to remote cluster under the name _src-cluster-alias.app_id-storename-changelog_
>
> When streams app fails over to the remote cluster it has troubles to find changelog topic of its state store since it was renamed - given source cluster prefix by mm2.
> Whats the fix should be ? instruct mm2 to keep topic name or subscribe to regex *._app_id-storename-changelog_ topic name for the state's changelog.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)