You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Kashyap Mhaisekar <ka...@gmail.com> on 2015/09/25 01:58:16 UTC
Nodes dont talk to each other
HI,
I am now facing a unique issue where the nodes of a cluster do not talk to
each other. The spout does not accept messages and even if there are some
messages, they are not sent to other workers on the same node or different
nodes.
Logs also dont indicate much. Has anyone faced this issue before?
I am on Storm 0.9.5 and have 8 machines with 8 cores and 32 GB each.
Thanks for the time.
Kashyap
Re: Nodes dont talk to each other
Posted by Srinath C <sr...@gmail.com>.
I would suggest that you take a thread dump and analyze what the executor
threads are doing. Make sure that you are not blocking the executor threads.
Regards,
Srinath
On Sep 25, 2015 5:28 AM, "Kashyap Mhaisekar" <ka...@gmail.com> wrote:
> HI,
> I am now facing a unique issue where the nodes of a cluster do not talk to
> each other. The spout does not accept messages and even if there are some
> messages, they are not sent to other workers on the same node or different
> nodes.
>
> Logs also dont indicate much. Has anyone faced this issue before?
>
> I am on Storm 0.9.5 and have 8 machines with 8 cores and 32 GB each.
>
> Thanks for the time.
>
> Kashyap
>
Re: Nodes dont talk to each other
Posted by Kashyap Mhaisekar <ka...@gmail.com>.
Thanks Onur. the problem is resolved.
What I noticed is that the supervisor ids are not synced between the
zookeeper cluster nodes. Some of the supervisors were registered and some
were not.
Basically, i stopped the storm cluster, delete the storm-data, cleaned up
zookeeper zpath and restarted and it works as of now.
Thanks
Kashyap
On Fri, Sep 25, 2015 at 9:56 AM, Onur Yalazı <on...@8digits.com>
wrote:
> I mean in size. I had to introduce a compressing serializer to lower our
> bw usage a few days ago.
>
> SDN is software defined network, like the networking used via cloud
> providers or openstack like tools.
>
> I would start by looking into dns issues first. Try resolving each nodes
> hostname on each and every node of the system. Use hostnames as it's
> reported on the stormmui.
>
> Does the cluster start and then stop or never fire any tuple? Also the
> issue may be that the spout having problems generating tuples.
>
> If the cluster starts and stops a while later, start using some sort of
> network monitoring and watch for bw usage between supervisor nodes.
> On Sep 25, 2015 5:47 PM, "Kashyap Mhaisekar" <ka...@gmail.com> wrote:
>
>> Onur,
>> By huge tuples - do you mean in number or size? And what is SDN?
>>
>> There are no firewalls for this and all zookeepers are correctly listed
>> on storm.yaml. Any advices on how I should go about looking for the root
>> cause?
>>
>> Thanks
>> kashyap
>>
>> On Fri, Sep 25, 2015 at 9:00 AM, Onur Yalazı <on...@8digits.com>
>> wrote:
>>
>>> That situation happens even if a single node's hostname can not be
>>> resolved.
>>>
>>> Also if you are running with huge tuples like we do (big mistake I
>>> know), and if the network has low capacity compared to what you need, like
>>> a soft limit on sdn kicking in, cluster stops to communicate and run.
>>> On Sep 25, 2015 4:45 PM, "Srividhya Shanmugam" <
>>> SrividhyaShanmugam@fico.com> wrote:
>>>
>>>> Are the zookeeper servers listed in storm.yaml file running properly?
>>>>
>>>> -----Original Message-----
>>>> From: Derek Dagit [mailto:derekd@yahoo-inc.com]
>>>> Sent: Friday, September 25, 2015 9:39 AM
>>>> To: user@storm.apache.org
>>>> Subject: Re: Nodes dont talk to each other
>>>>
>>>> I saw it happen before when the firewalls were not configured correctly.
>>>>
>>>> --
>>>> Derek
>>>>
>>>>
>>>> ________________________________
>>>> From: Kashyap Mhaisekar <ka...@gmail.com>
>>>> To: user@storm.apache.org
>>>> Sent: Thursday, September 24, 2015 6:58 PM
>>>> Subject: Nodes dont talk to each other
>>>>
>>>>
>>>>
>>>> HI,
>>>> I am now facing a unique issue where the nodes of a cluster do not talk
>>>> to each other. The spout does not accept messages and even if there are
>>>> some messages, they are not sent to other workers on the same node or
>>>> different nodes.
>>>>
>>>> Logs also dont indicate much. Has anyone faced this issue before?
>>>>
>>>> I am on Storm 0.9.5 and have 8 machines with 8 cores and 32 GB each.
>>>>
>>>> Thanks for the time.
>>>>
>>>> Kashyap
>>>>
>>>> This email and any files transmitted with it are confidential,
>>>> proprietary and intended solely for the individual or entity to whom they
>>>> are addressed. If you have received this email in error please delete it
>>>> immediately.
>>>>
>>>
>>
Re: Nodes dont talk to each other
Posted by Onur Yalazı <on...@8digits.com>.
I mean in size. I had to introduce a compressing serializer to lower our bw
usage a few days ago.
SDN is software defined network, like the networking used via cloud
providers or openstack like tools.
I would start by looking into dns issues first. Try resolving each nodes
hostname on each and every node of the system. Use hostnames as it's
reported on the stormmui.
Does the cluster start and then stop or never fire any tuple? Also the
issue may be that the spout having problems generating tuples.
If the cluster starts and stops a while later, start using some sort of
network monitoring and watch for bw usage between supervisor nodes.
On Sep 25, 2015 5:47 PM, "Kashyap Mhaisekar" <ka...@gmail.com> wrote:
> Onur,
> By huge tuples - do you mean in number or size? And what is SDN?
>
> There are no firewalls for this and all zookeepers are correctly listed on
> storm.yaml. Any advices on how I should go about looking for the root cause?
>
> Thanks
> kashyap
>
> On Fri, Sep 25, 2015 at 9:00 AM, Onur Yalazı <on...@8digits.com>
> wrote:
>
>> That situation happens even if a single node's hostname can not be
>> resolved.
>>
>> Also if you are running with huge tuples like we do (big mistake I know),
>> and if the network has low capacity compared to what you need, like a soft
>> limit on sdn kicking in, cluster stops to communicate and run.
>> On Sep 25, 2015 4:45 PM, "Srividhya Shanmugam" <
>> SrividhyaShanmugam@fico.com> wrote:
>>
>>> Are the zookeeper servers listed in storm.yaml file running properly?
>>>
>>> -----Original Message-----
>>> From: Derek Dagit [mailto:derekd@yahoo-inc.com]
>>> Sent: Friday, September 25, 2015 9:39 AM
>>> To: user@storm.apache.org
>>> Subject: Re: Nodes dont talk to each other
>>>
>>> I saw it happen before when the firewalls were not configured correctly.
>>>
>>> --
>>> Derek
>>>
>>>
>>> ________________________________
>>> From: Kashyap Mhaisekar <ka...@gmail.com>
>>> To: user@storm.apache.org
>>> Sent: Thursday, September 24, 2015 6:58 PM
>>> Subject: Nodes dont talk to each other
>>>
>>>
>>>
>>> HI,
>>> I am now facing a unique issue where the nodes of a cluster do not talk
>>> to each other. The spout does not accept messages and even if there are
>>> some messages, they are not sent to other workers on the same node or
>>> different nodes.
>>>
>>> Logs also dont indicate much. Has anyone faced this issue before?
>>>
>>> I am on Storm 0.9.5 and have 8 machines with 8 cores and 32 GB each.
>>>
>>> Thanks for the time.
>>>
>>> Kashyap
>>>
>>> This email and any files transmitted with it are confidential,
>>> proprietary and intended solely for the individual or entity to whom they
>>> are addressed. If you have received this email in error please delete it
>>> immediately.
>>>
>>
>
Re: Nodes dont talk to each other
Posted by Kashyap Mhaisekar <ka...@gmail.com>.
Onur,
By huge tuples - do you mean in number or size? And what is SDN?
There are no firewalls for this and all zookeepers are correctly listed on
storm.yaml. Any advices on how I should go about looking for the root cause?
Thanks
kashyap
On Fri, Sep 25, 2015 at 9:00 AM, Onur Yalazı <on...@8digits.com>
wrote:
> That situation happens even if a single node's hostname can not be
> resolved.
>
> Also if you are running with huge tuples like we do (big mistake I know),
> and if the network has low capacity compared to what you need, like a soft
> limit on sdn kicking in, cluster stops to communicate and run.
> On Sep 25, 2015 4:45 PM, "Srividhya Shanmugam" <
> SrividhyaShanmugam@fico.com> wrote:
>
>> Are the zookeeper servers listed in storm.yaml file running properly?
>>
>> -----Original Message-----
>> From: Derek Dagit [mailto:derekd@yahoo-inc.com]
>> Sent: Friday, September 25, 2015 9:39 AM
>> To: user@storm.apache.org
>> Subject: Re: Nodes dont talk to each other
>>
>> I saw it happen before when the firewalls were not configured correctly.
>>
>> --
>> Derek
>>
>>
>> ________________________________
>> From: Kashyap Mhaisekar <ka...@gmail.com>
>> To: user@storm.apache.org
>> Sent: Thursday, September 24, 2015 6:58 PM
>> Subject: Nodes dont talk to each other
>>
>>
>>
>> HI,
>> I am now facing a unique issue where the nodes of a cluster do not talk
>> to each other. The spout does not accept messages and even if there are
>> some messages, they are not sent to other workers on the same node or
>> different nodes.
>>
>> Logs also dont indicate much. Has anyone faced this issue before?
>>
>> I am on Storm 0.9.5 and have 8 machines with 8 cores and 32 GB each.
>>
>> Thanks for the time.
>>
>> Kashyap
>>
>> This email and any files transmitted with it are confidential,
>> proprietary and intended solely for the individual or entity to whom they
>> are addressed. If you have received this email in error please delete it
>> immediately.
>>
>
RE: Nodes dont talk to each other
Posted by Onur Yalazı <on...@8digits.com>.
That situation happens even if a single node's hostname can not be
resolved.
Also if you are running with huge tuples like we do (big mistake I know),
and if the network has low capacity compared to what you need, like a soft
limit on sdn kicking in, cluster stops to communicate and run.
On Sep 25, 2015 4:45 PM, "Srividhya Shanmugam" <Sr...@fico.com>
wrote:
> Are the zookeeper servers listed in storm.yaml file running properly?
>
> -----Original Message-----
> From: Derek Dagit [mailto:derekd@yahoo-inc.com]
> Sent: Friday, September 25, 2015 9:39 AM
> To: user@storm.apache.org
> Subject: Re: Nodes dont talk to each other
>
> I saw it happen before when the firewalls were not configured correctly.
>
> --
> Derek
>
>
> ________________________________
> From: Kashyap Mhaisekar <ka...@gmail.com>
> To: user@storm.apache.org
> Sent: Thursday, September 24, 2015 6:58 PM
> Subject: Nodes dont talk to each other
>
>
>
> HI,
> I am now facing a unique issue where the nodes of a cluster do not talk to
> each other. The spout does not accept messages and even if there are some
> messages, they are not sent to other workers on the same node or different
> nodes.
>
> Logs also dont indicate much. Has anyone faced this issue before?
>
> I am on Storm 0.9.5 and have 8 machines with 8 cores and 32 GB each.
>
> Thanks for the time.
>
> Kashyap
>
> This email and any files transmitted with it are confidential, proprietary
> and intended solely for the individual or entity to whom they are
> addressed. If you have received this email in error please delete it
> immediately.
>
RE: Nodes dont talk to each other
Posted by Srividhya Shanmugam <Sr...@fico.com>.
Are the zookeeper servers listed in storm.yaml file running properly?
-----Original Message-----
From: Derek Dagit [mailto:derekd@yahoo-inc.com]
Sent: Friday, September 25, 2015 9:39 AM
To: user@storm.apache.org
Subject: Re: Nodes dont talk to each other
I saw it happen before when the firewalls were not configured correctly.
--
Derek
________________________________
From: Kashyap Mhaisekar <ka...@gmail.com>
To: user@storm.apache.org
Sent: Thursday, September 24, 2015 6:58 PM
Subject: Nodes dont talk to each other
HI,
I am now facing a unique issue where the nodes of a cluster do not talk to each other. The spout does not accept messages and even if there are some messages, they are not sent to other workers on the same node or different nodes.
Logs also dont indicate much. Has anyone faced this issue before?
I am on Storm 0.9.5 and have 8 machines with 8 cores and 32 GB each.
Thanks for the time.
Kashyap
This email and any files transmitted with it are confidential, proprietary and intended solely for the individual or entity to whom they are addressed. If you have received this email in error please delete it immediately.
Re: Nodes dont talk to each other
Posted by Derek Dagit <de...@yahoo-inc.com>.
I saw it happen before when the firewalls were not configured correctly.
--
Derek
________________________________
From: Kashyap Mhaisekar <ka...@gmail.com>
To: user@storm.apache.org
Sent: Thursday, September 24, 2015 6:58 PM
Subject: Nodes dont talk to each other
HI,
I am now facing a unique issue where the nodes of a cluster do not talk to each other. The spout does not accept messages and even if there are some messages, they are not sent to other workers on the same node or different nodes.
Logs also dont indicate much. Has anyone faced this issue before?
I am on Storm 0.9.5 and have 8 machines with 8 cores and 32 GB each.
Thanks for the time.
Kashyap