You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@kafka.apache.org by "Boyang Chen (JIRA)" <ji...@apache.org> on 2019/02/28 17:54:00 UTC

[jira] [Created] (KAFKA-8019) Better Scaling Experience for KStream

Boyang Chen created KAFKA-8019:
----------------------------------

             Summary: Better Scaling Experience for KStream
                 Key: KAFKA-8019
                 URL: https://issues.apache.org/jira/browse/KAFKA-8019
             Project: Kafka
          Issue Type: New Feature
            Reporter: Boyang Chen
            Assignee: Boyang Chen


In our day-to-day work, we found it really hard to scale up a stateful stream application when its state store is very heavy. The caveat is that when the newly spinned hosts take ownership of some active tasks, so that they need to use non-trivial amount of time to restore the state store from changelog topic. The reassigned tasks would be available for unpredicted long time, which is not favorable. Secondly the current global rebalance stops the entire application process, which in a rolling host swap scenario would suggest an infinite resource shuffling without actual progress.

Following the community's [cooperative rebalancing|https://cwiki.apache.org/confluence/display/KAFKA/Incremental+Cooperative+Rebalancing%3A+Support+and+Policies] proposal, we need to build something similar for KStream to better handle the auto scaling experience.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)