You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Dominik Safaric <do...@gmail.com> on 2016/09/17 16:09:26 UTC
Storm 1.0.2 - KafkaSpout cannot find partition information
Hi,
I’ve deployed a topology consisting of a KafkaSpout using Kafka 0.10.0.1 and Zookeeper 3.4.6. All of the services, including the Nimbus and Supervisor, run on the same instance.
However, by examining the worker.log file, I’ve found that the KafkaSpout is unable to find partitions information of the given topic. The following log messages for each of the 12 partitions appear:
2016-09-17 17:37:33.333 o.a.s.k.PartitionManager [INFO] No partition information found, using configuration to determine offset
2016-09-17 17:37:33.333 o.a.s.k.PartitionManager [INFO] Last commit offset from zookeeper: 0
2016-09-17 17:37:33.333 o.a.s.k.PartitionManager [INFO] Commit offset 0 is more than 9223372036854775807 behind latest offset 0, resetting to startOffsetTime=-2
Whereas, the SpoutConf is instantiated as follows:
SpoutConfig spoutConfig = new SpoutConfig(hosts, "bytes", new String(), UUID.randomUUID().toString());
spoutConfig.scheme = new RawMultiScheme();
spoutConfig.zkServers = Arrays.asList("localhost");
spoutConfig.zkPort = 2181;
spoutConfig.zkRoot = "/brokers”;
By examining Zookeeper’s zNodes using the zkCli script, I’ve found that the partitions information is stored properly as prescribed by the default configuration: /brokers/topics/<topic_name>/partitions/…
What exactly might be the reason behind KafkaSpout not being able to consume messages from Kafka, i.e. find partition information? Is there a configuration mistake I’ve made?
Thanks in advance!
Dominik
Re: Storm 1.0.2 - KafkaSpout cannot find partition information
Posted by Ambud Sharma <as...@gmail.com>.
I believe you are overriding the value here:
spoutConfig.zkRoot = "/brokers”;
On Sep 17, 2016 9:29 AM, "Dominik Safaric" <do...@gmail.com> wrote:
> Already configured with an empty string, but unfortunately I keep getting
> the same message claiming no partitions can be found.
>
> Dominik Šafarić
>
> On 17 Sep 2016, at 18:11, Ambud Sharma <as...@gmail.com> wrote:
>
> The Zkroot should be empty string.
>
> On Sep 17, 2016 9:09 AM, "Dominik Safaric" <do...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I’ve deployed a topology consisting of a KafkaSpout using Kafka 0.10.0.1
>> and Zookeeper 3.4.6. All of the services, including the Nimbus and
>> Supervisor, run on the same instance.
>>
>> However, by examining the worker.log file, I’ve found that the KafkaSpout
>> is unable to find partitions information of the given topic. The following
>> log messages for each of the 12 partitions appear:
>>
>> 2016-09-17 17:37:33.333 o.a.s.k.PartitionManager [INFO] No partition
>> information found, using configuration to determine offset
>> 2016-09-17 17:37:33.333 o.a.s.k.PartitionManager [INFO] Last commit
>> offset from zookeeper: 0
>> 2016-09-17 17:37:33.333 o.a.s.k.PartitionManager [INFO] Commit offset 0
>> is more than 9223372036854775807 behind latest offset 0, resetting to
>> startOffsetTime=-2
>>
>> Whereas, the SpoutConf is instantiated as follows:
>>
>> SpoutConfig spoutConfig = new SpoutConfig(hosts, "bytes", new String(),
>> UUID.randomUUID().toString());
>> spoutConfig.scheme = new RawMultiScheme();
>> spoutConfig.zkServers = Arrays.asList("localhost");
>> spoutConfig.zkPort = 2181;
>> spoutConfig.zkRoot = "/brokers”;
>>
>> By examining Zookeeper’s zNodes using the zkCli script, I’ve found that
>> the partitions information is stored properly as prescribed by the default
>> configuration: /brokers/topics/<topic_name>/partitions/…
>>
>> What exactly might be the reason behind KafkaSpout not being able to
>> consume messages from Kafka, i.e. find partition information? Is there a
>> configuration mistake I’ve made?
>>
>> Thanks in advance!
>>
>> Dominik
>>
>
Re: Storm 1.0.2 - KafkaSpout cannot find partition information
Posted by Dominik Safaric <do...@gmail.com>.
Already configured with an empty string, but unfortunately I keep getting the same message claiming no partitions can be found.
Dominik Šafarić
> On 17 Sep 2016, at 18:11, Ambud Sharma <as...@gmail.com> wrote:
>
> The Zkroot should be empty string.
>
>
>> On Sep 17, 2016 9:09 AM, "Dominik Safaric" <do...@gmail.com> wrote:
>> Hi,
>>
>> I’ve deployed a topology consisting of a KafkaSpout using Kafka 0.10.0.1 and Zookeeper 3.4.6. All of the services, including the Nimbus and Supervisor, run on the same instance.
>>
>> However, by examining the worker.log file, I’ve found that the KafkaSpout is unable to find partitions information of the given topic. The following log messages for each of the 12 partitions appear:
>>
>> 2016-09-17 17:37:33.333 o.a.s.k.PartitionManager [INFO] No partition information found, using configuration to determine offset
>> 2016-09-17 17:37:33.333 o.a.s.k.PartitionManager [INFO] Last commit offset from zookeeper: 0
>> 2016-09-17 17:37:33.333 o.a.s.k.PartitionManager [INFO] Commit offset 0 is more than 9223372036854775807 behind latest offset 0, resetting to startOffsetTime=-2
>>
>> Whereas, the SpoutConf is instantiated as follows:
>>
>> SpoutConfig spoutConfig = new SpoutConfig(hosts, "bytes", new String(), UUID.randomUUID().toString());
>> spoutConfig.scheme = new RawMultiScheme();
>> spoutConfig.zkServers = Arrays.asList("localhost");
>> spoutConfig.zkPort = 2181;
>> spoutConfig.zkRoot = "/brokers”;
>>
>> By examining Zookeeper’s zNodes using the zkCli script, I’ve found that the partitions information is stored properly as prescribed by the default configuration: /brokers/topics/<topic_name>/partitions/…
>>
>> What exactly might be the reason behind KafkaSpout not being able to consume messages from Kafka, i.e. find partition information? Is there a configuration mistake I’ve made?
>>
>> Thanks in advance!
>>
>> Dominik
Re: Storm 1.0.2 - KafkaSpout cannot find partition information
Posted by Ambud Sharma <as...@gmail.com>.
The Zkroot should be empty string.
On Sep 17, 2016 9:09 AM, "Dominik Safaric" <do...@gmail.com> wrote:
> Hi,
>
> I’ve deployed a topology consisting of a KafkaSpout using Kafka 0.10.0.1
> and Zookeeper 3.4.6. All of the services, including the Nimbus and
> Supervisor, run on the same instance.
>
> However, by examining the worker.log file, I’ve found that the KafkaSpout
> is unable to find partitions information of the given topic. The following
> log messages for each of the 12 partitions appear:
>
> 2016-09-17 17:37:33.333 o.a.s.k.PartitionManager [INFO] No partition
> information found, using configuration to determine offset
> 2016-09-17 17:37:33.333 o.a.s.k.PartitionManager [INFO] Last commit
> offset from zookeeper: 0
> 2016-09-17 17:37:33.333 o.a.s.k.PartitionManager [INFO] Commit offset 0
> is more than 9223372036854775807 behind latest offset 0, resetting to
> startOffsetTime=-2
>
> Whereas, the SpoutConf is instantiated as follows:
>
> SpoutConfig spoutConfig = new SpoutConfig(hosts, "bytes", new String(),
> UUID.randomUUID().toString());
> spoutConfig.scheme = new RawMultiScheme();
> spoutConfig.zkServers = Arrays.asList("localhost");
> spoutConfig.zkPort = 2181;
> spoutConfig.zkRoot = "/brokers”;
>
> By examining Zookeeper’s zNodes using the zkCli script, I’ve found that
> the partitions information is stored properly as prescribed by the default
> configuration: /brokers/topics/<topic_name>/partitions/…
>
> What exactly might be the reason behind KafkaSpout not being able to
> consume messages from Kafka, i.e. find partition information? Is there a
> configuration mistake I’ve made?
>
> Thanks in advance!
>
> Dominik
>