[jira] [Commented] (SAMZA-1293) Enable partition expansion of input streams

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SAMZA-1293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16414764#comment-16414764 ] 

ASF GitHub Bot commented on SAMZA-1293:
---------------------------------------

Github user lindong28 closed the pull request at:

    https://github.com/apache/samza/pull/214


> Enable partition expansion of input streams
> -------------------------------------------
>
>                 Key: SAMZA-1293
>                 URL: https://issues.apache.org/jira/browse/SAMZA-1293
>             Project: Samza
>          Issue Type: New Feature
>            Reporter: Dong Lin
>            Priority: Major
>
> Right now, Samza does not allow partitions of the input stream to increase after a stateful job is created. This causes problem when Kafka is used as the input system, because we need to expand partitions of an existing topic as the byte-in-rate of the topic increases over time in order to limit the size of the maximum partition in Kafka. Kafka broker may have performance issue if the size of a given partition is too large.
> This patch provides a solution to increase partition number of the input streams of a stateful Samza job while still ensuring the correctness of Samze job output. The solution should work when Kafka is used as the input system. We expect this solution to work similarly with other input system as well. The motivation of increasing partition number of Kafka topic is 1) increase performance of Kafka broker and 2) increase throughput of Kafka consumer in the Samza container.
> See SEP-5 (https://cwiki.apache.org/confluence/display/SAMZA/SEP-5%3A+Enable+partition+expansion+of+input+streams) for the design and the interface change of this patch.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)