You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "neeraj (JIRA)" <ji...@apache.org> on 2019/08/06 16:36:00 UTC
[jira] [Comment Edited] (KAFKA-8207) StickyPartitionAssignor for
KStream
[ https://issues.apache.org/jira/browse/KAFKA-8207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16901245#comment-16901245 ]
neeraj edited comment on KAFKA-8207 at 8/6/19 4:35 PM:
-------------------------------------------------------
Hi [~bchen225242] I have written you an email pasting content
–
Initially, we got stuck in the critical bug https://issues.apache.org/jira/browse/KAFKA-8715 but then we took source code and changed the GroupMetadata.scala to use UUID instead of currentStateTimestamp as mentioned in the pull request. We rebuild the Kafka and replaced the kafka_2.12-2.3.0 jar at all broker lib folder
We expected that when we restart our consumer cluster (20 machines)/or a single node we will be getting same partitions in the same node but that is not the case (consumer group give details). We need this because we plan to store a lot of aggregated data in rocks DB in our consumer cluster(20 machines)., different partitions to the different machine at restart means eventually each node will have all the data and it will take a lot of time to build the stores.
was (Author: neeraj.bhatt):
Hi [~bchen225242] I have written you an email pasting content
--
Initially, we got stuck in the critical bug https://issues.apache.org/jira/browse/KAFKA-8715 but then we took source code and changed the GroupMetadata.scala to use UUID instead of currentStateTimestamp (code snippet at the end of mail) as mentioned in the pull request. We rebuild the Kafka and replaced the kafka_2.12-2.3.0 jar at all broker lib folder
We expected that when we restart our consumer cluster (20 machines)/or a single node we will be getting same partitions in the same node but that is not the case (consumer group give details). We need this because we plan to store a lot of aggregated data in rocks DB in our consumer cluster(20 machines)., different partitions to the different machine at restart means eventually each node will have all the data and it will take a lot of time to build the stores.
> StickyPartitionAssignor for KStream
> -----------------------------------
>
> Key: KAFKA-8207
> URL: https://issues.apache.org/jira/browse/KAFKA-8207
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Affects Versions: 2.0.0
> Reporter: neeraj
> Priority: Major
>
> In KStreams I am not able to give a sticky partition assignor or my custom partition assignor.
> Overriding the property while building stream does not work
> streams props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG, CustomAssignor.class.getName());
>
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)