[jira] [Created] (SAMZA-1591) Planner for high-level applications should validate the partition number of intermediate topics

["Yi Pan (Data Infrastructure) (JIRA)" <ji...@apache.org>]

Yi Pan (Data Infrastructure) created SAMZA-1591:
---------------------------------------------------

             Summary: Planner for high-level applications should validate the partition number of intermediate topics
                 Key: SAMZA-1591
                 URL: https://issues.apache.org/jira/browse/SAMZA-1591
             Project: Samza
          Issue Type: Bug
    Affects Versions: 0.14.0
            Reporter: Yi Pan (Data Infrastructure)
             Fix For: 0.15.0


We have seen cases where the intermediate topic created by partitionBy() operator was created with partition number calculated by the planner. However, when input topic partition expands with traffic, usually that would have trickling effect to the downstream topic partition and task assignments (i.e. 128 partitions from the input topic got shuffled into only 8 partitions in the intermediate topic usually does not distribute the work load widely enough). At least, the planner needs to validation and print a warning message if the intermediate topic partition is less than the max of input/output topic partition number.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)