You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@storm.apache.org by "Stig Rohde Døssing (JIRA)" <ji...@apache.org> on 2017/08/08 22:58:00 UTC
[jira] [Updated] (STORM-2691) storm-kafka-client Trident spout
communicates with the Coordinator through a static field
[ https://issues.apache.org/jira/browse/STORM-2691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stig Rohde Døssing updated STORM-2691:
--------------------------------------
Description:
The Trident Kafka spout uses the KafkaTridentSpoutTopicPartitionRegistry enum to pass existing topic partitions from the spout to the coordinator. This only works when those components happen to be in the same JVM, because the coordinator gets the topic information from the KafkaConsumer started by KafkaTridentSpoutEmitter.
The coordinator runs in the TridentSpoutCoordinator bolt here https://github.com/apache/storm/blob/4c8a986f519cdf3e63bed47e9c4f723e4867267a/storm-client/src/jvm/org/apache/storm/trident/topology/TridentTopologyBuilder.java#L162, while the spout instances (emitters) run in TridentSpoutExecutors here https://github.com/apache/storm/blob/4c8a986f519cdf3e63bed47e9c4f723e4867267a/storm-client/src/jvm/org/apache/storm/trident/topology/TridentTopologyBuilder.java#L176.
We should replace the registry enum with writes to Zookeeper or something similar.
was:
The Trident Kafka spout uses the KafkaTridentSpoutTopicPartitionRegistry enum to pass existing topic partitions from the spout to the coordinator. This only works when those components happen to be in the same JVM.
The coordinator runs in the TridentSpoutCoordinator bolt here https://github.com/apache/storm/blob/4c8a986f519cdf3e63bed47e9c4f723e4867267a/storm-client/src/jvm/org/apache/storm/trident/topology/TridentTopologyBuilder.java#L162, while the spout instances run in TridentSpoutExecutors here https://github.com/apache/storm/blob/4c8a986f519cdf3e63bed47e9c4f723e4867267a/storm-client/src/jvm/org/apache/storm/trident/topology/TridentTopologyBuilder.java#L176.
We should replace the registry enum with writes to Zookeeper or something similar.
> storm-kafka-client Trident spout communicates with the Coordinator through a static field
> -----------------------------------------------------------------------------------------
>
> Key: STORM-2691
> URL: https://issues.apache.org/jira/browse/STORM-2691
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-kafka-client
> Affects Versions: 2.0.0, 1.1.1
> Reporter: Stig Rohde Døssing
>
> The Trident Kafka spout uses the KafkaTridentSpoutTopicPartitionRegistry enum to pass existing topic partitions from the spout to the coordinator. This only works when those components happen to be in the same JVM, because the coordinator gets the topic information from the KafkaConsumer started by KafkaTridentSpoutEmitter.
> The coordinator runs in the TridentSpoutCoordinator bolt here https://github.com/apache/storm/blob/4c8a986f519cdf3e63bed47e9c4f723e4867267a/storm-client/src/jvm/org/apache/storm/trident/topology/TridentTopologyBuilder.java#L162, while the spout instances (emitters) run in TridentSpoutExecutors here https://github.com/apache/storm/blob/4c8a986f519cdf3e63bed47e9c4f723e4867267a/storm-client/src/jvm/org/apache/storm/trident/topology/TridentTopologyBuilder.java#L176.
> We should replace the registry enum with writes to Zookeeper or something similar.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)