You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by clay teahouse <cl...@gmail.com> on 2015/01/25 21:16:11 UTC
zookeeper questions
Hi All,
A couple of questions:
1)
What does storm.zookeeper.root point to? The definition says storm root
directory location. The default value is /storm. But I am not sure I
understand what this means. If I have external zookeeper running, what
should this configuration point to?
2) I am running my topology in cluster mode, and I said with external
zookeeper (its configuration is specified in storm.yaml). But I am seeing
the following lines in my worker logs:
s.k.DynamicBrokersReader [INFO] Read partition info from zookeeper:
GlobalPartitionInformation{partitionMap={0=localhost:9092,
1=localhost:9092}}
..
..
..
s.k.PartitionManager [INFO] No partition information found, using
configuration to determine offset
I've read that the second message means that storm is not using the
external zookeeper. If that is the case, what would be the reason for that.
thanks,
Clay
Re: zookeeper questions
Posted by clay teahouse <cl...@gmail.com>.
The zookeeper is running in port 2181. The kafka broker is running in port
9092.
On Sun, Jan 25, 2015 at 3:21 PM, Margus Roo <ma...@roo.ee> wrote:
> is there any zookeepers running in your localhost in 9092 port?
>
> Margus (margusja) Roohttp://margus.roo.ee
> skype: margusja
> +372 51 480
>
> On 25/01/15 23:14, clay teahouse wrote:
>
> I meant zookeeper is external to storm. I running the zookeeper that comes
> with kafka.
>
> On Sun, Jan 25, 2015 at 3:11 PM, Margus Roo <ma...@roo.ee> wrote:
>
>> So your storm's zookeeper is external. But how about Kafka's?
>>
>> Margus (margusja) Roohttp://margus.roo.ee
>> skype: margusja
>> +372 51 480
>>
>> On 25/01/15 23:08, clay teahouse wrote:
>>
>> Thanks Margus.
>> 1) I don't do anything (at least explicitly) with zookeeper settings or
>> the offsets. All I do in the topology, I specify the address of the
>> zookeeper (ZkHosts). Here is some the lines in between:
>> 2015-01-25T13:26:52.288-0600 s.k.PartitionManager [INFO] Read last
>> commit offset from zookeeper: 7; old topology_id:
>> a001b24c-d99c-49f1-8b60-573aac41937e - new topology_id:
>> ad09f254-ca58-4298-8f57-09741389da56
>> 2015-01-25T13:26:52.288-0600 s.k.PartitionManager [INFO] Starting Kafka
>> localhost:1 from offset 7
>> 2015-01-25T13:26:52.297-0600 s.k.PartitionManager [INFO] Read partition
>> information from: /storm/partition_0 --> null
>> 2015-01-25T13:26:52.302-0600 s.k.PartitionManager [INFO] No partition
>> information found, using configuration to determine offset
>>
>> 2) So /storm actually is an absolute path?
>>
>> On Sun, Jan 25, 2015 at 2:45 PM, Margus Roo <ma...@roo.ee> wrote:
>>
>>> Maybe you overwrote it in your topology code?
>>> zk path in example /storm is a place where storm keeps statuses about
>>> storm in example.
>>>
>>> Margus (margusja) Roohttp://margus.roo.ee
>>> skype: margusja
>>> +372 51 480
>>>
>>> On 25/01/15 22:16, clay teahouse wrote:
>>>
>>> Hi All,
>>>
>>> A couple of questions:
>>> 1)
>>> What does storm.zookeeper.root point to? The definition says storm root
>>> directory location. The default value is /storm. But I am not sure I
>>> understand what this means. If I have external zookeeper running, what
>>> should this configuration point to?
>>>
>>> 2) I am running my topology in cluster mode, and I said with external
>>> zookeeper (its configuration is specified in storm.yaml). But I am seeing
>>> the following lines in my worker logs:
>>>
>>> s.k.DynamicBrokersReader [INFO] Read partition info from zookeeper:
>>> GlobalPartitionInformation{partitionMap={0=localhost:9092,
>>> 1=localhost:9092}}
>>> ..
>>> ..
>>> ..
>>> s.k.PartitionManager [INFO] No partition information found, using
>>> configuration to determine offset
>>>
>>> I've read that the second message means that storm is not using the
>>> external zookeeper. If that is the case, what would be the reason for that.
>>> thanks,
>>> Clay
>>>
>>>
>>>
>>
>>
>
>
Re: zookeeper questions
Posted by Margus Roo <ma...@roo.ee>.
is there any zookeepers running in your localhost in 9092 port?
Margus (margusja) Roo
http://margus.roo.ee
skype: margusja
+372 51 480
On 25/01/15 23:14, clay teahouse wrote:
> I meant zookeeper is external to storm. I running the zookeeper that
> comes with kafka.
>
> On Sun, Jan 25, 2015 at 3:11 PM, Margus Roo <margus@roo.ee
> <ma...@roo.ee>> wrote:
>
> So your storm's zookeeper is external. But how about Kafka's?
>
> Margus (margusja) Roo
> http://margus.roo.ee
> skype: margusja
> +372 51 480
>
> On 25/01/15 23:08, clay teahouse wrote:
>> Thanks Margus.
>> 1) I don't do anything (at least explicitly) with zookeeper
>> settings or the offsets. All I do in the topology, I specify the
>> address of the zookeeper (ZkHosts). Here is some the lines in
>> between:
>> 2015-01-25T13:26:52.288-0600 s.k.PartitionManager [INFO] Read
>> last commit offset from zookeeper: 7; old topology_id:
>> a001b24c-d99c-49f1-8b60-573aac41937e - new topology_id:
>> ad09f254-ca58-4298-8f57-09741389da56
>> 2015-01-25T13:26:52.288-0600 s.k.PartitionManager [INFO] Starting
>> Kafka localhost:1 from offset 7
>> 2015-01-25T13:26:52.297-0600 s.k.PartitionManager [INFO] Read
>> partition information from: /storm/partition_0 --> null
>> 2015-01-25T13:26:52.302-0600 s.k.PartitionManager [INFO] No
>> partition information found, using configuration to determine offset
>>
>> 2) So /storm actually is an absolute path?
>>
>> On Sun, Jan 25, 2015 at 2:45 PM, Margus Roo <margus@roo.ee
>> <ma...@roo.ee>> wrote:
>>
>> Maybe you overwrote it in your topology code?
>> zk path in example /storm is a place where storm keeps
>> statuses about storm in example.
>>
>> Margus (margusja) Roo
>> http://margus.roo.ee
>> skype: margusja
>> +372 51 480
>>
>> On 25/01/15 22:16, clay teahouse wrote:
>>> Hi All,
>>>
>>> A couple of questions:
>>> 1)
>>> What does storm.zookeeper.root point to? The definition says
>>> storm root directory location. The default value is /storm.
>>> But I am not sure I understand what this means. If I have
>>> external zookeeper running, what should this configuration
>>> point to?
>>>
>>> 2) I am running my topology in cluster mode, and I said with
>>> external zookeeper (its configuration is specified in
>>> storm.yaml). But I am seeing the following lines in my
>>> worker logs:
>>>
>>> s.k.DynamicBrokersReader [INFO] Read partition info from
>>> zookeeper:
>>> GlobalPartitionInformation{partitionMap={0=localhost:9092,
>>> 1=localhost:9092}}
>>> ..
>>> ..
>>> ..
>>> s.k.PartitionManager [INFO] No partition information found,
>>> using configuration to determine offset
>>>
>>> I've read that the second message means that storm is not
>>> using the external zookeeper. If that is the case, what
>>> would be the reason for that.
>>> thanks,
>>> Clay
>>>
>>
>>
>
>
Re: zookeeper questions
Posted by clay teahouse <cl...@gmail.com>.
I meant zookeeper is external to storm. I running the zookeeper that comes
with kafka.
On Sun, Jan 25, 2015 at 3:11 PM, Margus Roo <ma...@roo.ee> wrote:
> So your storm's zookeeper is external. But how about Kafka's?
>
> Margus (margusja) Roohttp://margus.roo.ee
> skype: margusja
> +372 51 480
>
> On 25/01/15 23:08, clay teahouse wrote:
>
> Thanks Margus.
> 1) I don't do anything (at least explicitly) with zookeeper settings or
> the offsets. All I do in the topology, I specify the address of the
> zookeeper (ZkHosts). Here is some the lines in between:
> 2015-01-25T13:26:52.288-0600 s.k.PartitionManager [INFO] Read last
> commit offset from zookeeper: 7; old topology_id:
> a001b24c-d99c-49f1-8b60-573aac41937e - new topology_id:
> ad09f254-ca58-4298-8f57-09741389da56
> 2015-01-25T13:26:52.288-0600 s.k.PartitionManager [INFO] Starting Kafka
> localhost:1 from offset 7
> 2015-01-25T13:26:52.297-0600 s.k.PartitionManager [INFO] Read partition
> information from: /storm/partition_0 --> null
> 2015-01-25T13:26:52.302-0600 s.k.PartitionManager [INFO] No partition
> information found, using configuration to determine offset
>
> 2) So /storm actually is an absolute path?
>
> On Sun, Jan 25, 2015 at 2:45 PM, Margus Roo <ma...@roo.ee> wrote:
>
>> Maybe you overwrote it in your topology code?
>> zk path in example /storm is a place where storm keeps statuses about
>> storm in example.
>>
>> Margus (margusja) Roohttp://margus.roo.ee
>> skype: margusja
>> +372 51 480
>>
>> On 25/01/15 22:16, clay teahouse wrote:
>>
>> Hi All,
>>
>> A couple of questions:
>> 1)
>> What does storm.zookeeper.root point to? The definition says storm root
>> directory location. The default value is /storm. But I am not sure I
>> understand what this means. If I have external zookeeper running, what
>> should this configuration point to?
>>
>> 2) I am running my topology in cluster mode, and I said with external
>> zookeeper (its configuration is specified in storm.yaml). But I am seeing
>> the following lines in my worker logs:
>>
>> s.k.DynamicBrokersReader [INFO] Read partition info from zookeeper:
>> GlobalPartitionInformation{partitionMap={0=localhost:9092,
>> 1=localhost:9092}}
>> ..
>> ..
>> ..
>> s.k.PartitionManager [INFO] No partition information found, using
>> configuration to determine offset
>>
>> I've read that the second message means that storm is not using the
>> external zookeeper. If that is the case, what would be the reason for that.
>> thanks,
>> Clay
>>
>>
>>
>
>
Re: zookeeper questions
Posted by Margus Roo <ma...@roo.ee>.
So your storm's zookeeper is external. But how about Kafka's?
Margus (margusja) Roo
http://margus.roo.ee
skype: margusja
+372 51 480
On 25/01/15 23:08, clay teahouse wrote:
> Thanks Margus.
> 1) I don't do anything (at least explicitly) with zookeeper settings
> or the offsets. All I do in the topology, I specify the address of the
> zookeeper (ZkHosts). Here is some the lines in between:
> 2015-01-25T13:26:52.288-0600 s.k.PartitionManager [INFO] Read last
> commit offset from zookeeper: 7; old topology_id:
> a001b24c-d99c-49f1-8b60-573aac41937e - new topology_id:
> ad09f254-ca58-4298-8f57-09741389da56
> 2015-01-25T13:26:52.288-0600 s.k.PartitionManager [INFO] Starting
> Kafka localhost:1 from offset 7
> 2015-01-25T13:26:52.297-0600 s.k.PartitionManager [INFO] Read
> partition information from: /storm/partition_0 --> null
> 2015-01-25T13:26:52.302-0600 s.k.PartitionManager [INFO] No partition
> information found, using configuration to determine offset
>
> 2) So /storm actually is an absolute path?
>
> On Sun, Jan 25, 2015 at 2:45 PM, Margus Roo <margus@roo.ee
> <ma...@roo.ee>> wrote:
>
> Maybe you overwrote it in your topology code?
> zk path in example /storm is a place where storm keeps statuses
> about storm in example.
>
> Margus (margusja) Roo
> http://margus.roo.ee
> skype: margusja
> +372 51 480
>
> On 25/01/15 22:16, clay teahouse wrote:
>> Hi All,
>>
>> A couple of questions:
>> 1)
>> What does storm.zookeeper.root point to? The definition says
>> storm root directory location. The default value is /storm. But I
>> am not sure I understand what this means. If I have external
>> zookeeper running, what should this configuration point to?
>>
>> 2) I am running my topology in cluster mode, and I said with
>> external zookeeper (its configuration is specified in
>> storm.yaml). But I am seeing the following lines in my worker logs:
>>
>> s.k.DynamicBrokersReader [INFO] Read partition info from
>> zookeeper:
>> GlobalPartitionInformation{partitionMap={0=localhost:9092,
>> 1=localhost:9092}}
>> ..
>> ..
>> ..
>> s.k.PartitionManager [INFO] No partition information found, using
>> configuration to determine offset
>>
>> I've read that the second message means that storm is not using
>> the external zookeeper. If that is the case, what would be the
>> reason for that.
>> thanks,
>> Clay
>>
>
>
Re: zookeeper questions
Posted by clay teahouse <cl...@gmail.com>.
Thanks Margus.
1) I don't do anything (at least explicitly) with zookeeper settings or
the offsets. All I do in the topology, I specify the address of the
zookeeper (ZkHosts). Here is some the lines in between:
2015-01-25T13:26:52.288-0600 s.k.PartitionManager [INFO] Read last commit
offset from zookeeper: 7; old topology_id:
a001b24c-d99c-49f1-8b60-573aac41937e - new topology_id:
ad09f254-ca58-4298-8f57-09741389da56
2015-01-25T13:26:52.288-0600 s.k.PartitionManager [INFO] Starting Kafka
localhost:1 from offset 7
2015-01-25T13:26:52.297-0600 s.k.PartitionManager [INFO] Read partition
information from: /storm/partition_0 --> null
2015-01-25T13:26:52.302-0600 s.k.PartitionManager [INFO] No partition
information found, using configuration to determine offset
2) So /storm actually is an absolute path?
On Sun, Jan 25, 2015 at 2:45 PM, Margus Roo <ma...@roo.ee> wrote:
> Maybe you overwrote it in your topology code?
> zk path in example /storm is a place where storm keeps statuses about
> storm in example.
>
> Margus (margusja) Roohttp://margus.roo.ee
> skype: margusja
> +372 51 480
>
> On 25/01/15 22:16, clay teahouse wrote:
>
> Hi All,
>
> A couple of questions:
> 1)
> What does storm.zookeeper.root point to? The definition says storm root
> directory location. The default value is /storm. But I am not sure I
> understand what this means. If I have external zookeeper running, what
> should this configuration point to?
>
> 2) I am running my topology in cluster mode, and I said with external
> zookeeper (its configuration is specified in storm.yaml). But I am seeing
> the following lines in my worker logs:
>
> s.k.DynamicBrokersReader [INFO] Read partition info from zookeeper:
> GlobalPartitionInformation{partitionMap={0=localhost:9092,
> 1=localhost:9092}}
> ..
> ..
> ..
> s.k.PartitionManager [INFO] No partition information found, using
> configuration to determine offset
>
> I've read that the second message means that storm is not using the
> external zookeeper. If that is the case, what would be the reason for that.
> thanks,
> Clay
>
>
>
Re: zookeeper questions
Posted by Margus Roo <ma...@roo.ee>.
Maybe you overwrote it in your topology code?
zk path in example /storm is a place where storm keeps statuses about
storm in example.
Margus (margusja) Roo
http://margus.roo.ee
skype: margusja
+372 51 480
On 25/01/15 22:16, clay teahouse wrote:
> Hi All,
>
> A couple of questions:
> 1)
> What does storm.zookeeper.root point to? The definition says storm
> root directory location. The default value is /storm. But I am not
> sure I understand what this means. If I have external zookeeper
> running, what should this configuration point to?
>
> 2) I am running my topology in cluster mode, and I said with external
> zookeeper (its configuration is specified in storm.yaml). But I am
> seeing the following lines in my worker logs:
>
> s.k.DynamicBrokersReader [INFO] Read partition info from zookeeper:
> GlobalPartitionInformation{partitionMap={0=localhost:9092,
> 1=localhost:9092}}
> ..
> ..
> ..
> s.k.PartitionManager [INFO] No partition information found, using
> configuration to determine offset
>
> I've read that the second message means that storm is not using the
> external zookeeper. If that is the case, what would be the reason for
> that.
> thanks,
> Clay
>