[jira] [Created] (KAFKA-5009) Globally Unique internal topic names when using Joins

["Mark Tranter (JIRA)" <ji...@apache.org>]

Mark Tranter created KAFKA-5009:
-----------------------------------

             Summary: Globally Unique internal topic names when using Joins 
                 Key: KAFKA-5009
                 URL: https://issues.apache.org/jira/browse/KAFKA-5009
             Project: Kafka
          Issue Type: Improvement
          Components: streams
            Reporter: Mark Tranter


We are using multiple different Kafka Streams applications on the back of a single kafka cluster. This allows each micro-service in our enterprise to consume & process kafka data to suit its own needs.

Currently when joining streams, an internal topic is created & named using KStreamsBuilder.newName(prefix);

```

            String thisWindowStreamName = topology.newName(WINDOWED_NAME);
            String otherWindowStreamName = topology.newName(WINDOWED_NAME);
            String joinThisName = rightOuter ? topology.newName(OUTERTHIS_NAME) : topology.newName(JOINTHIS_NAME);
            String joinOtherName = leftOuter ? topology.newName(OUTEROTHER_NAME) : topology.newName(JOINOTHER_NAME);
            String joinMergeName = topology.newName(MERGE_NAME);
```

This prefix is a constant, and internally an incrementor is used to generate a  unique (per KStreamBuilder instance) topic name for the topic.

In situations where multiple KStreamBuilders are in use (for example, multiple different Kafka Streams applications) we are seeing collisions in topic names.

Perhaps the join() methods should take a topic prefix overload to allow developers to provide unique names for these topics. Similar to the way other stateful processors work when having to provide a store name.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)