[jira] [Comment Edited] (KAFKA-8207) StickyPartitionAssignor for KStream

["neeraj (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-8207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16816210#comment-16816210 ] 

neeraj edited comment on KAFKA-8207 at 4/12/19 12:26 PM:
---------------------------------------------------------

StreamPartitionAssignor sticky policy is for the stickiness of Tasks to partition. (StickyTaskAssignor) 

I am looking for Stickiness of consumer to partition like in normal Kafka Consumer

We have three consumers c1,c2, and c3 in the same consumer group using KStream (reading from a Kafka topic with multiple partitions). Each consumer is in a separate machine.  We are aggregating and saving it in the local state store at each node.

When node 1 goes down (for rolling deployment or for some other reason) rebalance kicks in and replica which is given by num.standby.replicas take over so it is fault tolerant, but when node 1 comes back again rebalance again kicks in  and node 1 processes different partitions and updates the state store for different partitions.

So Ideally, we want that node 1 consumer c1 should work on the same partitions as it was working earlier so that we can save replay time from change log and disk space as because of frequent rebalancing local store has data for all the partitions

problem similar to 

[https://tech.transferwise.com/achieving-high-availability-with-kafka-streams/]

 

In the past also https://issues.apache.org/jira/browse/KAFKA-7695 was raised and it was suggested to open a new enhancement

 

 

 


was (Author: neeraj.bhatt):
StreamPartitionAssignor sticky policy is for the stickiness of Tasks to partition. (StickyTaskAssignor) 

I am looking for Stickiness of consumer to partition like in normal Kafka Consumer

We have three consumers c1,c2, and c3 in the same consumer group using KStream (reading from a Kafka topic with multiple). Each consumer is in a separate machine.  We are aggregating and saving it in the local state store at each node.

When node 1 goes down (for rolling deployment or for some other reason) rebalance kicks in and replica which is given by num.standby.replicas take over so it is fault tolerant, but when node 1 comes back again rebalance again kicks in  and node 1 processes different partitions and updates the state store for different partitions.

So Ideally, we want that node 1 consumer c1 should work on the same partitions as it was working earlier so that we can save replay time from change log and disk space as because of frequent rebalancing local store has data for all the partitions

problem similar to 

[https://tech.transferwise.com/achieving-high-availability-with-kafka-streams/]

 

 

 

> StickyPartitionAssignor for KStream
> -----------------------------------
>
>                 Key: KAFKA-8207
>                 URL: https://issues.apache.org/jira/browse/KAFKA-8207
>             Project: Kafka
>          Issue Type: Improvement
>          Components: streams
>    Affects Versions: 2.0.0
>            Reporter: neeraj
>            Priority: Major
>
> In KStreams I am not able to give a sticky partition assignor or my custom partition assignor.
> Overriding the property while building stream does not work
> streams props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG, CustomAssignor.class.getName());
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)