[jira] [Resolved] (KAFKA-8019) Better Scaling Experience for KStream

["Matthias J. Sax (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-8019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matthias J. Sax resolved KAFKA-8019.
------------------------------------
    Resolution: Duplicate

Closing this a duplicate of KAFKA-6145. Should be address via KIP-441.

> Better Scaling Experience for KStream
> -------------------------------------
>
>                 Key: KAFKA-8019
>                 URL: https://issues.apache.org/jira/browse/KAFKA-8019
>             Project: Kafka
>          Issue Type: New Feature
>          Components: streams
>            Reporter: Boyang Chen
>            Assignee: Boyang Chen
>            Priority: Major
>              Labels: kip
>
> In our day-to-day work, we found it really hard to scale up a stateful stream application when its state store is very heavy. The caveat is that when the newly spinned hosts take ownership of some active tasks, so that they need to use non-trivial amount of time to restore the state store from changelog topic. The reassigned tasks would be available for unpredicted long time, which is not favorable. Secondly the current global rebalance stops the entire application process, which in a rolling host swap scenario would suggest an infinite resource shuffling without actual progress.
> Following the community's [cooperative rebalancing|https://cwiki.apache.org/confluence/display/KAFKA/Incremental+Cooperative+Rebalancing%3A+Support+and+Policies] proposal, we need to build something similar for KStream to better handle the auto scaling experience.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)