[jira] [Commented] (STORM-2719) Trident Kafka Spout Emitters do not get full partition information in getOrderedPartitions()

["Stig Rohde Døssing (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/STORM-2719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16151890#comment-16151890 ] 

Stig Rohde Døssing commented on STORM-2719:
-------------------------------------------

Hi [~janithkv]. This looks like a duplicate of https://issues.apache.org/jira/browse/STORM-2473 and https://issues.apache.org/jira/browse/STORM-2691, so I'd like to close this in favor of those. Please let me know if you disagree, or you feel that parts of this are missing from those issues.

> Trident Kafka Spout Emitters do not get full partition information in getOrderedPartitions()
> --------------------------------------------------------------------------------------------
>
>                 Key: STORM-2719
>                 URL: https://issues.apache.org/jira/browse/STORM-2719
>             Project: Apache Storm
>          Issue Type: Bug
>          Components: storm-kafka-client, trident
>    Affects Versions: 1.1.1
>            Reporter: Janith Kaiprath Valiyalappil
>
> The storm kakfa trident spout uses the KafkaTridentSpoutTopicPartitionRegistry, to get partition information. The coordinator calls the getTopicPartitions() method to get partition information and passes it to the emitters. But this partition information will not be accurate as all instances of KafkaTridentSpoutTopicPartitionRegistry will not be updated with full partition information.
> The update to the registry is done when the consumer subscribes using KafkaSpoutConsumerRebalanceListener. This calls the KafkaTridentSpoutTopicPartitionRegistry.INSTANCE.addAll(partitions); These calls would only update the registry in that particular worker with partition information for consumers in that worker.
> So when the coordinator calls the getOrderedPartitions() and passes it to each emitter by calling getOrderedPartitions(), the full partition information will not be present. The only probable case this would work is if the emitters and coordinators were on the same worker.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)