You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Maurice Lawler <ma...@me.com> on 2013/04/10 01:27:07 UTC

Cluster-Heartbeat Error

The errors below are flooding my /var/log/cloud/management/management-cloud-log.

Please advise.

- Maurice




2013-04-09 17:22:12,183 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:2, nodeIP:0.0.0.0
2013-04-09 17:22:12,183 INFO  [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to 0.0.0.0
2013-04-09 17:22:12,184 INFO  [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 2 is detected inactive by timestamp but is pingable
2013-04-09 17:22:13,682 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:2, nodeIP0.0.0.0
2013-04-09 17:22:13,682 INFO  [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to 0.0.0.0
2013-04-09 17:22:13,683 INFO  [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 2 is detected inactive by timestamp but is pingable
2013-04-09 17:22:15,184 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:2, nodeIP0.0.0.0
2013-04-09 17:22:15,184 INFO  [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to0.0.0.0
2013-04-09 17:22:15,184 INFO  [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 2 is detected inactive by timestamp but is pingable
2013-04-09 17:22:16,684 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:2, nodeIP:0.0.0.0
2013-04-09 17:22:16,684 INFO  [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to 0.0.0.0
2013-04-09 17:22:16,684 INFO  [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 2 is detected inactive by timestamp but is pingable
2013-04-09 17:22:18,183 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:2, nodeIP0.0.0.0
2013-04-09 17:22:18,183 INFO  [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to0.0.0.0
2013-04-09 17:22:18,183 INFO  [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 2 is detected inactive by timestamp but is pingable


Re: Cluster-Heartbeat Error

Posted by Maurice Lawler <ma...@me.com>.
I changed my IP address to 0.0.0.0 as to not provide that information; my configuration is not set to 0.0.0.0 -- I am certain I would have more errors then described. :-) 


As far as my IPtables, by default those ports are already opened. 

- Maurice


On Apr 9, 2013, at 7:50 PM, Mathias Mullins <ma...@citrix.com> wrote:

> Better yet, did you open your iptables for port 8135 and 9090? That's usually when see those errors. 
> 
> Thank you,
> Matt 
> 
> On Apr 9, 2013, at 7:54 PM, "Ahmad Emneina" <ae...@gmail.com> wrote:
> 
>> did you change your host parameter in the global config to 0.0.0.0? it
>> should be set to the actual ip of the host, hosting the cloustack
>> management service.
>> 
>> 0.0.0.0, bad!
>> 
>> 
>> On Tue, Apr 9, 2013 at 4:27 PM, Maurice Lawler <ma...@me.com>wrote:
>> 
>>> The errors below are flooding my
>>> /var/log/cloud/management/management-cloud-log.
>>> 
>>> Please advise.
>>> 
>>> - Maurice
>>> 
>>> 
>>> 
>>> 
>>> 2013-04-09 17:22:12,183 DEBUG [cloud.cluster.ClusterManagerImpl]
>>> (Cluster-Heartbeat-1:null) Detected management node left, id:2,
>>> nodeIP:0.0.0.0
>>> 2013-04-09 17:22:12,183 INFO  [cloud.cluster.ClusterManagerImpl]
>>> (Cluster-Heartbeat-1:null) Trying to connect to 0.0.0.0
>>> 2013-04-09 17:22:12,184 INFO  [cloud.cluster.ClusterManagerImpl]
>>> (Cluster-Heartbeat-1:null) Management node 2 is detected inactive by
>>> timestamp but is pingable
>>> 2013-04-09 17:22:13,682 DEBUG [cloud.cluster.ClusterManagerImpl]
>>> (Cluster-Heartbeat-1:null) Detected management node left, id:2,
>>> nodeIP0.0.0.0
>>> 2013-04-09 17:22:13,682 INFO  [cloud.cluster.ClusterManagerImpl]
>>> (Cluster-Heartbeat-1:null) Trying to connect to 0.0.0.0
>>> 2013-04-09 17:22:13,683 INFO  [cloud.cluster.ClusterManagerImpl]
>>> (Cluster-Heartbeat-1:null) Management node 2 is detected inactive by
>>> timestamp but is pingable
>>> 2013-04-09 17:22:15,184 DEBUG [cloud.cluster.ClusterManagerImpl]
>>> (Cluster-Heartbeat-1:null) Detected management node left, id:2,
>>> nodeIP0.0.0.0
>>> 2013-04-09 17:22:15,184 INFO  [cloud.cluster.ClusterManagerImpl]
>>> (Cluster-Heartbeat-1:null) Trying to connect to0.0.0.0
>>> 2013-04-09 17:22:15,184 INFO  [cloud.cluster.ClusterManagerImpl]
>>> (Cluster-Heartbeat-1:null) Management node 2 is detected inactive by
>>> timestamp but is pingable
>>> 2013-04-09 17:22:16,684 DEBUG [cloud.cluster.ClusterManagerImpl]
>>> (Cluster-Heartbeat-1:null) Detected management node left, id:2,
>>> nodeIP:0.0.0.0
>>> 2013-04-09 17:22:16,684 INFO  [cloud.cluster.ClusterManagerImpl]
>>> (Cluster-Heartbeat-1:null) Trying to connect to 0.0.0.0
>>> 2013-04-09 17:22:16,684 INFO  [cloud.cluster.ClusterManagerImpl]
>>> (Cluster-Heartbeat-1:null) Management node 2 is detected inactive by
>>> timestamp but is pingable
>>> 2013-04-09 17:22:18,183 DEBUG [cloud.cluster.ClusterManagerImpl]
>>> (Cluster-Heartbeat-1:null) Detected management node left, id:2,
>>> nodeIP0.0.0.0
>>> 2013-04-09 17:22:18,183 INFO  [cloud.cluster.ClusterManagerImpl]
>>> (Cluster-Heartbeat-1:null) Trying to connect to0.0.0.0
>>> 2013-04-09 17:22:18,183 INFO  [cloud.cluster.ClusterManagerImpl]
>>> (Cluster-Heartbeat-1:null) Management node 2 is detected inactive by
>>> timestamp but is pingable
>>> 
>>>

Re: Cluster-Heartbeat Error

Posted by Mathias Mullins <ma...@citrix.com>.
Better yet, did you open your iptables for port 8135 and 9090? That's usually when see those errors. 

Thank you,
Matt 

On Apr 9, 2013, at 7:54 PM, "Ahmad Emneina" <ae...@gmail.com> wrote:

> did you change your host parameter in the global config to 0.0.0.0? it
> should be set to the actual ip of the host, hosting the cloustack
> management service.
> 
> 0.0.0.0, bad!
> 
> 
> On Tue, Apr 9, 2013 at 4:27 PM, Maurice Lawler <ma...@me.com>wrote:
> 
>> The errors below are flooding my
>> /var/log/cloud/management/management-cloud-log.
>> 
>> Please advise.
>> 
>> - Maurice
>> 
>> 
>> 
>> 
>> 2013-04-09 17:22:12,183 DEBUG [cloud.cluster.ClusterManagerImpl]
>> (Cluster-Heartbeat-1:null) Detected management node left, id:2,
>> nodeIP:0.0.0.0
>> 2013-04-09 17:22:12,183 INFO  [cloud.cluster.ClusterManagerImpl]
>> (Cluster-Heartbeat-1:null) Trying to connect to 0.0.0.0
>> 2013-04-09 17:22:12,184 INFO  [cloud.cluster.ClusterManagerImpl]
>> (Cluster-Heartbeat-1:null) Management node 2 is detected inactive by
>> timestamp but is pingable
>> 2013-04-09 17:22:13,682 DEBUG [cloud.cluster.ClusterManagerImpl]
>> (Cluster-Heartbeat-1:null) Detected management node left, id:2,
>> nodeIP0.0.0.0
>> 2013-04-09 17:22:13,682 INFO  [cloud.cluster.ClusterManagerImpl]
>> (Cluster-Heartbeat-1:null) Trying to connect to 0.0.0.0
>> 2013-04-09 17:22:13,683 INFO  [cloud.cluster.ClusterManagerImpl]
>> (Cluster-Heartbeat-1:null) Management node 2 is detected inactive by
>> timestamp but is pingable
>> 2013-04-09 17:22:15,184 DEBUG [cloud.cluster.ClusterManagerImpl]
>> (Cluster-Heartbeat-1:null) Detected management node left, id:2,
>> nodeIP0.0.0.0
>> 2013-04-09 17:22:15,184 INFO  [cloud.cluster.ClusterManagerImpl]
>> (Cluster-Heartbeat-1:null) Trying to connect to0.0.0.0
>> 2013-04-09 17:22:15,184 INFO  [cloud.cluster.ClusterManagerImpl]
>> (Cluster-Heartbeat-1:null) Management node 2 is detected inactive by
>> timestamp but is pingable
>> 2013-04-09 17:22:16,684 DEBUG [cloud.cluster.ClusterManagerImpl]
>> (Cluster-Heartbeat-1:null) Detected management node left, id:2,
>> nodeIP:0.0.0.0
>> 2013-04-09 17:22:16,684 INFO  [cloud.cluster.ClusterManagerImpl]
>> (Cluster-Heartbeat-1:null) Trying to connect to 0.0.0.0
>> 2013-04-09 17:22:16,684 INFO  [cloud.cluster.ClusterManagerImpl]
>> (Cluster-Heartbeat-1:null) Management node 2 is detected inactive by
>> timestamp but is pingable
>> 2013-04-09 17:22:18,183 DEBUG [cloud.cluster.ClusterManagerImpl]
>> (Cluster-Heartbeat-1:null) Detected management node left, id:2,
>> nodeIP0.0.0.0
>> 2013-04-09 17:22:18,183 INFO  [cloud.cluster.ClusterManagerImpl]
>> (Cluster-Heartbeat-1:null) Trying to connect to0.0.0.0
>> 2013-04-09 17:22:18,183 INFO  [cloud.cluster.ClusterManagerImpl]
>> (Cluster-Heartbeat-1:null) Management node 2 is detected inactive by
>> timestamp but is pingable
>> 
>>

Re: Cluster-Heartbeat Error

Posted by Ahmad Emneina <ae...@gmail.com>.
did you change your host parameter in the global config to 0.0.0.0? it
should be set to the actual ip of the host, hosting the cloustack
management service.

0.0.0.0, bad!


On Tue, Apr 9, 2013 at 4:27 PM, Maurice Lawler <ma...@me.com>wrote:

> The errors below are flooding my
> /var/log/cloud/management/management-cloud-log.
>
> Please advise.
>
> - Maurice
>
>
>
>
> 2013-04-09 17:22:12,183 DEBUG [cloud.cluster.ClusterManagerImpl]
> (Cluster-Heartbeat-1:null) Detected management node left, id:2,
> nodeIP:0.0.0.0
> 2013-04-09 17:22:12,183 INFO  [cloud.cluster.ClusterManagerImpl]
> (Cluster-Heartbeat-1:null) Trying to connect to 0.0.0.0
> 2013-04-09 17:22:12,184 INFO  [cloud.cluster.ClusterManagerImpl]
> (Cluster-Heartbeat-1:null) Management node 2 is detected inactive by
> timestamp but is pingable
> 2013-04-09 17:22:13,682 DEBUG [cloud.cluster.ClusterManagerImpl]
> (Cluster-Heartbeat-1:null) Detected management node left, id:2,
> nodeIP0.0.0.0
> 2013-04-09 17:22:13,682 INFO  [cloud.cluster.ClusterManagerImpl]
> (Cluster-Heartbeat-1:null) Trying to connect to 0.0.0.0
> 2013-04-09 17:22:13,683 INFO  [cloud.cluster.ClusterManagerImpl]
> (Cluster-Heartbeat-1:null) Management node 2 is detected inactive by
> timestamp but is pingable
> 2013-04-09 17:22:15,184 DEBUG [cloud.cluster.ClusterManagerImpl]
> (Cluster-Heartbeat-1:null) Detected management node left, id:2,
> nodeIP0.0.0.0
> 2013-04-09 17:22:15,184 INFO  [cloud.cluster.ClusterManagerImpl]
> (Cluster-Heartbeat-1:null) Trying to connect to0.0.0.0
> 2013-04-09 17:22:15,184 INFO  [cloud.cluster.ClusterManagerImpl]
> (Cluster-Heartbeat-1:null) Management node 2 is detected inactive by
> timestamp but is pingable
> 2013-04-09 17:22:16,684 DEBUG [cloud.cluster.ClusterManagerImpl]
> (Cluster-Heartbeat-1:null) Detected management node left, id:2,
> nodeIP:0.0.0.0
> 2013-04-09 17:22:16,684 INFO  [cloud.cluster.ClusterManagerImpl]
> (Cluster-Heartbeat-1:null) Trying to connect to 0.0.0.0
> 2013-04-09 17:22:16,684 INFO  [cloud.cluster.ClusterManagerImpl]
> (Cluster-Heartbeat-1:null) Management node 2 is detected inactive by
> timestamp but is pingable
> 2013-04-09 17:22:18,183 DEBUG [cloud.cluster.ClusterManagerImpl]
> (Cluster-Heartbeat-1:null) Detected management node left, id:2,
> nodeIP0.0.0.0
> 2013-04-09 17:22:18,183 INFO  [cloud.cluster.ClusterManagerImpl]
> (Cluster-Heartbeat-1:null) Trying to connect to0.0.0.0
> 2013-04-09 17:22:18,183 INFO  [cloud.cluster.ClusterManagerImpl]
> (Cluster-Heartbeat-1:null) Management node 2 is detected inactive by
> timestamp but is pingable
>
>

Re: Cluster-Heartbeat Error

Posted by Kelven Yang <ke...@citrix.com>.
If it is not 0.0.0.0 IP address issue. You may need to check if you have a orphan/stuck MS server running at node 2, your MS on node 2 may be shown as running but for some reason it is stuck and stopped to tick the database heartbeat

Kelven

From: Maurice Lawler <ma...@me.com>>
Reply-To: "users@cloudstack.apache.org<ma...@cloudstack.apache.org>" <us...@cloudstack.apache.org>>
Date: Tuesday, April 9, 2013 4:27 PM
To: "users@cloudstack.apache.org<ma...@cloudstack.apache.org>" <us...@cloudstack.apache.org>>, "users@cloudstack.apache.org<ma...@cloudstack.apache.org>" <us...@cloudstack.apache.org>>
Cc: Cloud Dev <de...@cloudstack.apache.org>>
Subject: Cluster-Heartbeat Error

The errors below are flooding my /var/log/cloud/management/management-cloud-log.

Please advise.

- Maurice




2013-04-09 17:22:12,183 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:2, nodeIP:0.0.0.0
2013-04-09 17:22:12,183 INFO  [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to 0.0.0.0
2013-04-09 17:22:12,184 INFO  [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 2 is detected inactive by timestamp but is pingable
2013-04-09 17:22:13,682 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:2, nodeIP0.0.0.0
2013-04-09 17:22:13,682 INFO  [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to 0.0.0.0
2013-04-09 17:22:13,683 INFO  [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 2 is detected inactive by timestamp but is pingable
2013-04-09 17:22:15,184 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:2, nodeIP0.0.0.0
2013-04-09 17:22:15,184 INFO  [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to0.0.0.0
2013-04-09 17:22:15,184 INFO  [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 2 is detected inactive by timestamp but is pingable
2013-04-09 17:22:16,684 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:2, nodeIP:0.0.0.0
2013-04-09 17:22:16,684 INFO  [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to 0.0.0.0
2013-04-09 17:22:16,684 INFO  [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 2 is detected inactive by timestamp but is pingable
2013-04-09 17:22:18,183 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:2, nodeIP0.0.0.0
2013-04-09 17:22:18,183 INFO  [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to0.0.0.0
2013-04-09 17:22:18,183 INFO  [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 2 is detected inactive by timestamp but is pingable

Re: Cluster-Heartbeat Error

Posted by Kelven Yang <ke...@citrix.com>.
If it is not 0.0.0.0 IP address issue. You may need to check if you have a orphan/stuck MS server running at node 2, your MS on node 2 may be shown as running but for some reason it is stuck and stopped to tick the database heartbeat

Kelven

From: Maurice Lawler <ma...@me.com>>
Reply-To: "users@cloudstack.apache.org<ma...@cloudstack.apache.org>" <us...@cloudstack.apache.org>>
Date: Tuesday, April 9, 2013 4:27 PM
To: "users@cloudstack.apache.org<ma...@cloudstack.apache.org>" <us...@cloudstack.apache.org>>, "users@cloudstack.apache.org<ma...@cloudstack.apache.org>" <us...@cloudstack.apache.org>>
Cc: Cloud Dev <de...@cloudstack.apache.org>>
Subject: Cluster-Heartbeat Error

The errors below are flooding my /var/log/cloud/management/management-cloud-log.

Please advise.

- Maurice




2013-04-09 17:22:12,183 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:2, nodeIP:0.0.0.0
2013-04-09 17:22:12,183 INFO  [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to 0.0.0.0
2013-04-09 17:22:12,184 INFO  [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 2 is detected inactive by timestamp but is pingable
2013-04-09 17:22:13,682 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:2, nodeIP0.0.0.0
2013-04-09 17:22:13,682 INFO  [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to 0.0.0.0
2013-04-09 17:22:13,683 INFO  [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 2 is detected inactive by timestamp but is pingable
2013-04-09 17:22:15,184 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:2, nodeIP0.0.0.0
2013-04-09 17:22:15,184 INFO  [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to0.0.0.0
2013-04-09 17:22:15,184 INFO  [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 2 is detected inactive by timestamp but is pingable
2013-04-09 17:22:16,684 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:2, nodeIP:0.0.0.0
2013-04-09 17:22:16,684 INFO  [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to 0.0.0.0
2013-04-09 17:22:16,684 INFO  [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 2 is detected inactive by timestamp but is pingable
2013-04-09 17:22:18,183 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:2, nodeIP0.0.0.0
2013-04-09 17:22:18,183 INFO  [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to0.0.0.0
2013-04-09 17:22:18,183 INFO  [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 2 is detected inactive by timestamp but is pingable