[jira] [Commented] (SAMZA-1989) SystemStreamGrouper interface change for SEP-5

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SAMZA-1989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16709309#comment-16709309 ] 

ASF GitHub Bot commented on SAMZA-1989:
---------------------------------------

Github user asfgit closed the pull request at:

    https://github.com/apache/samza/pull/803


> SystemStreamGrouper interface change for SEP-5
> ----------------------------------------------
>
>                 Key: SAMZA-1989
>                 URL: https://issues.apache.org/jira/browse/SAMZA-1989
>             Project: Samza
>          Issue Type: New Feature
>            Reporter: Shanthoosh Venkataraman
>            Assignee: Shanthoosh Venkataraman
>            Priority: Major
>
> Samza users may need to increase the partition count of the input streams of their stateful samza jobs. For example, Kafka needs to limit the maximum size of each partition to scale up its performance. Thus the number of partitions of a Kafka topic needs to be expanded to reduce the partition size if the average byte-in-rate or retention time of the Kafka topic has doubled.
> In order to perform a join between streams, stateful jobs generally route the partitions from the different input streams to same task of a container. However, when a input stream repartitioning happens, key space of a partition gets redistributed. This will make the stateful jobs to produce erroneous results.
> So if the partition count of input stream is increased then the users have to manually purge the changelog topics, local RocksDb state of their stateful jobs. This  results in an increased operational complexity and data loss.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)