You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "John Roesler (Jira)" <ji...@apache.org> on 2020/02/07 16:19:00 UTC

[jira] [Commented] (KAFKA-7669) Stream topology definition is not robust to the ordering changes

    [ https://issues.apache.org/jira/browse/KAFKA-7669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17032512#comment-17032512 ] 

John Roesler commented on KAFKA-7669:
-------------------------------------

This issue has been lurking for a while, and has been reported a number of different ways. It seems to take two forms:
1. changing the topology at all (in apparently compatible ways) can renumber operators and corrupt the application upon restart
2. changing the topology in combination with a rolling bounce results in members executing a different topology than the leader, which leads to extra problems (such as NPEs)

https://issues.apache.org/jira/browse/KAFKA-7669 is related, and seems to be more about just changing the topology at all
https://issues.apache.org/jira/browse/KAFKA-8307 proposes a fix
https://issues.apache.org/jira/browse/KAFKA-8810 seems to be a duplicate of KAFKA-8307
and https://issues.apache.org/jira/browse/KAFKA-9518 reports an exception that results from a rolling-bounce topology change.

> Stream topology definition is not robust to the ordering changes
> ----------------------------------------------------------------
>
>                 Key: KAFKA-7669
>                 URL: https://issues.apache.org/jira/browse/KAFKA-7669
>             Project: Kafka
>          Issue Type: Wish
>          Components: streams
>    Affects Versions: 2.0.0
>            Reporter: Mateusz Owczarek
>            Priority: Major
>
> It seems that if the user does not guarantee the order of the stream topology definition, he may end up with multiple stream branches having the same internal changelog (and repartition, if created) topic. 
> Let's assume:
> {code:java}
> val initialStream = new StreamsBuilder().stream(sth);
> val someStrings = (1 to 10).map(_.toString)
> val notGuaranteedOrderOfStreams: Map[String, KStream[...]] = someStrings.map(s => s -> initialStream.filter(...)).toMap{code}
> When the user defines now common aggregation logic for the notGuaranteedOrderOfStreams, and runs multiple instances of the application the KSTREAM-AGGREGATE-STATE-STORE topics names will not be unique and will contain results of the different streams from notGuaranteedOrderOfStreams map.
> All of this without a single warning that the topology (or just the order of the topology definition) differs in different instances of the Kafka Streams application.
> Also, I am concerned that ids in "KSTREAM-AGGREGATE-STATE-STORE-id-changelog " match so well for the different application instances (and different topologies).
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)