You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Clemens Valiente (JIRA)" <ji...@apache.org> on 2017/12/20 15:48:00 UTC
[jira] [Commented] (KAFKA-6391) output from ensure copartitioning
is not used for Cluster metadata, resulting in partitions without tasks
working on them
[ https://issues.apache.org/jira/browse/KAFKA-6391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16298657#comment-16298657 ]
Clemens Valiente commented on KAFKA-6391:
-----------------------------------------
I dunno if I am too dumb to understand it correctly because it seems to be a relatively basic thing to get wrong..
> output from ensure copartitioning is not used for Cluster metadata, resulting in partitions without tasks working on them
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-6391
> URL: https://issues.apache.org/jira/browse/KAFKA-6391
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 1.0.0
> Reporter: Clemens Valiente
> Assignee: Clemens Valiente
> Original Estimate: 20m
> Remaining Estimate: 20m
>
> https://github.com/apache/kafka/blob/1.0/streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamPartitionAssignor.java#L394
> Map<TopicPartition, PartitionInfo> allRepartitionTopicPartitions is created from repartitionTopicMetadata
> THEN we do ensureCoPartitioning on repartitionTopicMetadata
> THEN we create topics and partitions according to repartitionTopicMetadata
> THEN we use allRepartitionTopicPartitions to create our Cluster fullMetadata
> THEN we use fullMetadata to assign the tasks and no longer use repartitionTopicMetadata
> This results in any change to repartitionTopicMetadata in ensureCoPartitioning to be used for creating partitions but no tasks are ever created for any partition added by ensureCoPartitioning()
> the fix is easy: First ensureCoPartitioning() on repartitionTopicMetadata before creating allRepartitionTopicPartitions
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)