You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by "Mitchell Rathbun (BLOOMBERG/ 731 LEX)" <mr...@bloomberg.net> on 2017/10/09 23:27:55 UTC

Waiting for KafkaSpout Initialization

In our topology we have a KafkaSpout. The KafkaSpout's FirstPollOffsetStrategy is set to LATEST, since we only want the updates that occur after our topology is up and running. The reason we can do this is because we notify the server that we need a "recap" of certain information, so we don't need any messages before that update. The problem we are having is determining when exactly we should notify the server that we are "listening". Currently, we submit the topology by using a LocalCluster object and calling submitTopology. However, in looking through the documentation, the KafkaSpout isn't really initialized until "after the first call to  KafkaSpoutConsumerRebalanceListener.onPartitionsAssigned()". So my concern is that after we call submitTopology, we ask for a recap before the KafkaSpout is really ready to receive one, creating a race condition. Is there a way to wait for all of the spouts/bolts to be submitted and ready? Or more specifically to wait for KafkaSpout to be "initialized"? Or is there a different design approach that is recommended besides setting FirstPollOffsetStrategy to LATEST and asking for a recap?