You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Maurice Lawler <ma...@me.com> on 2013/04/26 05:53:35 UTC
Cluster Heartbeat
Greetings,
Within my log of /var/log/cloud/management/management-cloud.log I am seeing the following:
2013-04-25 22:50:11,081 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
2013-04-25 22:50:11,081 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
2013-04-25 22:50:11,081 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
2013-04-25 22:50:12,581 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
2013-04-25 22:50:12,582 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
2013-04-25 22:50:12,582 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
2013-04-25 22:50:14,080 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
2013-04-25 22:50:14,080 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
2013-04-25 22:50:14,080 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
2013-04-25 22:50:15,579 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
2013-04-25 22:50:15,580 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
2013-04-25 22:50:15,580 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
2013-04-25 22:50:17,080 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
2013-04-25 22:50:17,081 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
2013-04-25 22:50:17,081 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
2013-04-25 22:50:18,581 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
2013-04-25 22:50:18,581 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
2013-04-25 22:50:18,581 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
It was preivously told to me to remove the "down" enttries within the table mshost; however, when I do that the error is no longer, but rebooting cloud-management if I were to make a change seems to revert whatever was done via table and the error reappears.
Anyone able to help me resolve this once and for all?
- Maurice
Re: Cluster Heartbeat
Posted by Kelven Yang <ke...@citrix.com>.
I remember I saw a discussion thread for password-enabled-template issues, you may start a question thread anyway, since the subject "Cluster Heartbeat" may not get attention from developers at the field on this password issue. In your case, you need some help from KVM developer to find out the root cause
Kelven
From: Maurice Lawler <ma...@me.com>>
Date: Friday, April 26, 2013 5:22 PM
To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>" <de...@cloudstack.apache.org>>
Cc: Kelven Yang <ke...@citrix.com>>
Subject: Re: Cluster Heartbeat
Sure, however, not sure where this e-mail thread is?
On Apr 26, 2013, at 08:13 PM, Kelven Yang <ke...@citrix.com>> wrote:
Please pass the log information to the password enabled template email thread, hopefully someone else in the field can check and help you from there.
Kelven
From: Maurice Lawler <ma...@me.com>>>
Date: Friday, April 26, 2013 5:06 PM
To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>>" <de...@cloudstack.apache.org>>>
Cc: Kelven Yang <ke...@citrix.com>>>, Kelven Yang <ke...@citrix.com>>>
Subject: Re: Cluster Heartbeat
It seems communication is possible, I tested it via telnet on port 8080, from within the guest VM to the DHCP IP that I found within /var/log/messages
Reviewing the logs, I see:
Apr 26 15:19:38 CentOS63 cloud: VM has already saved a password from the password server at dhcpip
Apr 26 15:19:38 CentOS63 cloud: Could not find password server IP in /var/lib/dhclient/resolv.conf.predhclient.eth0
Apr 26 15:19:38 CentOS63 cloud: Did not need to change password.
Apr 26 18:37:51 CentOS63 cloud: Found password server IP dhcpip in /var/lib/dhclient/dhclient-eth0.leases
Apr 26 18:37:51 CentOS63 cloud: Sending request to password server at dhcpip
Apr 26 18:37:51 CentOS63 cloud: Got response from server at dhcpip
Apr 26 18:37:51 CentOS63 cloud: VM has already saved a password from the password server at dhcpip
Apr 26 18:37:51 CentOS63 cloud: Could not find password server IP in /var/lib/dhclient/resolv.conf.predhclient.eth0
Apr 26 18:37:51 CentOS63 cloud: Did not need to change password.
What else would you need to know?
On Apr 26, 2013, at 07:58 PM, Kelven Yang <ke...@citrix.com>>> wrote:
You can either try out devCloud, or running management server in its own machine (could be a VM on your KVM host) to get rid of this KVM limitation.
For your password enabled template, depends on which guest OS you are using, to trouble shoot the problem, need a few steps to determine the root cause, but first of all, you need to check the connectivity from within guest VM to DHCP server that you see from inside the guest VM on port 8080.
With limited information, it is hard to tell for now.
Kelven
From: Maurice Lawler <ma...@me.com>>>>
Date: Friday, April 26, 2013 1:30 PM
To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>>>" <de...@cloudstack.apache.org>>>>
Cc: Kelven Yang <ke...@citrix.com>>>>
Subject: Re: Cluster Heartbeat
Kelven,
Thanks for the quick response, yes, I have seen no performance issues due to the error; you hit the nail on the head in stating it's annoyance. However, I'll live with the limitation KVM provide; as long as my instances remain running as well as they are.
However, the second part of mine; not sure if you have seen my mailing on the password enabled templates?
- Maurice
On Apr 26, 2013, at 04:27 PM, Kelven Yang <ke...@citrix.com>>>> wrote:
Thanks for the info and this is the explanation,
When you add KVM host into management server, it creates network bridge that affects CloudStack management to identify itself (we use MAC address to derive management server ID). This is a limitation of running KVM host and management server in one machine.
The message may be annoying, but it should not affect CloudStack from normal functioning
Kelven
From: Maurice Lawler <ma...@me.com>>>>>
Date: Friday, April 26, 2013 1:17 PM
To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>>>>" <de...@cloudstack.apache.org>>>>>
Cc: Kelven Yang <ke...@citrix.com>>>>>
Subject: Re: Cluster Heartbeat
Hello Kelven,
I am using KVM | CentOS 6.3 | CS 4.0.1
All under ONE server. However, I was as mentioned told to remove the 'Down' instances from mshost; restart and the error continues.
- Maurice
On Apr 26, 2013, at 01:13 PM, Kelven Yang <ke...@citrix.com>>>>> wrote:
What is your management server cluster setup look like? If it is a one-node setup, how many network interfaces do you have in the management server node?
Kelven
From: Maurice Lawler <ma...@me.com>>>>>>
Reply-To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>>>>>" <de...@cloudstack.apache.org>>>>>>
Date: Thursday, April 25, 2013 8:53 PM
To: Cloud Dev <de...@cloudstack.apache.org>>>>>>, "users@cloudstack.apache.org<ma...@cloudstack.apache.org>>>>>" <us...@cloudstack.apache.org>>>>>>, "users@cloudstack.apache.org<ma...@cloudstack.apache.org>>>>>" <us...@cloudstack.apache.org>>>>>>
Cc: "cloudstack-dev@incubator.apache.org<ma...@incubator.apache.org>>>>>" <cl...@incubator.apache.org>>>>>>, "cloudstack-dev@incubator.apache.org<ma...@incubator.apache.org>>>>>" <cl...@incubator.apache.org>>>>>>
Subject: Cluster Heartbeat
Greetings,
Within my log of /var/log/cloud/management/management-cloud.log I am seeing the following:
2013-04-25 22:50:11,081 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
2013-04-25 22:50:11,081 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
2013-04-25 22:50:11,081 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
2013-04-25 22:50:12,581 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
2013-04-25 22:50:12,582 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
2013-04-25 22:50:12,582 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
2013-04-25 22:50:14,080 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
2013-04-25 22:50:14,080 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
2013-04-25 22:50:14,080 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
2013-04-25 22:50:15,579 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
2013-04-25 22:50:15,580 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
2013-04-25 22:50:15,580 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
2013-04-25 22:50:17,080 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
2013-04-25 22:50:17,081 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
2013-04-25 22:50:17,081 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
2013-04-25 22:50:18,581 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
2013-04-25 22:50:18,581 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
2013-04-25 22:50:18,581 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
It was preivously told to me to remove the "down" enttries within the table mshost; however, when I do that the error is no longer, but rebooting cloud-management if I were to make a change seems to revert whatever was done via table and the error reappears.
Anyone able to help me resolve this once and for all?
- Maurice
Re: Cluster Heartbeat
Posted by Maurice Lawler <ma...@me.com>.
Sure, however, not sure where this e-mail thread is?
On Apr 26, 2013, at 08:13 PM, Kelven Yang <ke...@citrix.com> wrote:
> Please pass the log information to the password enabled template email thread, hopefully someone else in the field can check and help you from there.
>
> Kelven
>
> From: Maurice Lawler <ma...@me.com>>
> Date: Friday, April 26, 2013 5:06 PM
> To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>" <de...@cloudstack.apache.org>>
> Cc: Kelven Yang <ke...@citrix.com>>, Kelven Yang <ke...@citrix.com>>
> Subject: Re: Cluster Heartbeat
>
> It seems communication is possible, I tested it via telnet on port 8080, from within the guest VM to the DHCP IP that I found within /var/log/messages
>
> Reviewing the logs, I see:
>
> Apr 26 15:19:38 CentOS63 cloud: VM has already saved a password from the password server at dhcpip
> Apr 26 15:19:38 CentOS63 cloud: Could not find password server IP in /var/lib/dhclient/resolv.conf.predhclient.eth0
> Apr 26 15:19:38 CentOS63 cloud: Did not need to change password.
> Apr 26 18:37:51 CentOS63 cloud: Found password server IP dhcpip in /var/lib/dhclient/dhclient-eth0.leases
> Apr 26 18:37:51 CentOS63 cloud: Sending request to password server at dhcpip
> Apr 26 18:37:51 CentOS63 cloud: Got response from server at dhcpip
> Apr 26 18:37:51 CentOS63 cloud: VM has already saved a password from the password server at dhcpip
> Apr 26 18:37:51 CentOS63 cloud: Could not find password server IP in /var/lib/dhclient/resolv.conf.predhclient.eth0
> Apr 26 18:37:51 CentOS63 cloud: Did not need to change password.
>
> What else would you need to know?
>
>
> On Apr 26, 2013, at 07:58 PM, Kelven Yang <ke...@citrix.com>> wrote:
>
> You can either try out devCloud, or running management server in its own machine (could be a VM on your KVM host) to get rid of this KVM limitation.
>
> For your password enabled template, depends on which guest OS you are using, to trouble shoot the problem, need a few steps to determine the root cause, but first of all, you need to check the connectivity from within guest VM to DHCP server that you see from inside the guest VM on port 8080.
>
> With limited information, it is hard to tell for now.
>
> Kelven
>
>
> From: Maurice Lawler <ma...@me.com>>>
> Date: Friday, April 26, 2013 1:30 PM
> To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>>" <de...@cloudstack.apache.org>>>
> Cc: Kelven Yang <ke...@citrix.com>>>
> Subject: Re: Cluster Heartbeat
>
> Kelven,
>
> Thanks for the quick response, yes, I have seen no performance issues due to the error; you hit the nail on the head in stating it's annoyance. However, I'll live with the limitation KVM provide; as long as my instances remain running as well as they are.
>
> However, the second part of mine; not sure if you have seen my mailing on the password enabled templates?
>
> - Maurice
>
> On Apr 26, 2013, at 04:27 PM, Kelven Yang <ke...@citrix.com>>> wrote:
>
> Thanks for the info and this is the explanation,
>
> When you add KVM host into management server, it creates network bridge that affects CloudStack management to identify itself (we use MAC address to derive management server ID). This is a limitation of running KVM host and management server in one machine.
>
> The message may be annoying, but it should not affect CloudStack from normal functioning
>
> Kelven
>
> From: Maurice Lawler <ma...@me.com>>>>
> Date: Friday, April 26, 2013 1:17 PM
> To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>>>" <de...@cloudstack.apache.org>>>>
> Cc: Kelven Yang <ke...@citrix.com>>>>
> Subject: Re: Cluster Heartbeat
>
> Hello Kelven,
>
> I am using KVM | CentOS 6.3 | CS 4.0.1
>
> All under ONE server. However, I was as mentioned told to remove the 'Down' instances from mshost; restart and the error continues.
>
> - Maurice
>
> On Apr 26, 2013, at 01:13 PM, Kelven Yang <ke...@citrix.com>>>> wrote:
>
> What is your management server cluster setup look like? If it is a one-node setup, how many network interfaces do you have in the management server node?
>
> Kelven
>
> From: Maurice Lawler <ma...@me.com>>>>>
> Reply-To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>>>>" <de...@cloudstack.apache.org>>>>>
> Date: Thursday, April 25, 2013 8:53 PM
> To: Cloud Dev <de...@cloudstack.apache.org>>>>>, "users@cloudstack.apache.org<ma...@cloudstack.apache.org>>>>" <us...@cloudstack.apache.org>>>>>, "users@cloudstack.apache.org<ma...@cloudstack.apache.org>>>>" <us...@cloudstack.apache.org>>>>>
> Cc: "cloudstack-dev@incubator.apache.org<ma...@incubator.apache.org>>>>" <cl...@incubator.apache.org>>>>>, "cloudstack-dev@incubator.apache.org<ma...@incubator.apache.org>>>>" <cl...@incubator.apache.org>>>>>
> Subject: Cluster Heartbeat
>
> Greetings,
>
> Within my log of /var/log/cloud/management/management-cloud.log I am seeing the following:
>
> 2013-04-25 22:50:11,081 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
> 2013-04-25 22:50:11,081 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
> 2013-04-25 22:50:11,081 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
> 2013-04-25 22:50:12,581 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
> 2013-04-25 22:50:12,582 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
> 2013-04-25 22:50:12,582 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
> 2013-04-25 22:50:14,080 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
> 2013-04-25 22:50:14,080 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
> 2013-04-25 22:50:14,080 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
> 2013-04-25 22:50:15,579 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
> 2013-04-25 22:50:15,580 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
> 2013-04-25 22:50:15,580 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
> 2013-04-25 22:50:17,080 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
> 2013-04-25 22:50:17,081 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
> 2013-04-25 22:50:17,081 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
> 2013-04-25 22:50:18,581 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
> 2013-04-25 22:50:18,581 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
> 2013-04-25 22:50:18,581 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
>
>
> It was preivously told to me to remove the "down" enttries within the table mshost; however, when I do that the error is no longer, but rebooting cloud-management if I were to make a change seems to revert whatever was done via table and the error reappears.
>
> Anyone able to help me resolve this once and for all?
>
> - Maurice
Re: Cluster Heartbeat
Posted by Kelven Yang <ke...@citrix.com>.
Please pass the log information to the password enabled template email thread, hopefully someone else in the field can check and help you from there.
Kelven
From: Maurice Lawler <ma...@me.com>>
Date: Friday, April 26, 2013 5:06 PM
To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>" <de...@cloudstack.apache.org>>
Cc: Kelven Yang <ke...@citrix.com>>, Kelven Yang <ke...@citrix.com>>
Subject: Re: Cluster Heartbeat
It seems communication is possible, I tested it via telnet on port 8080, from within the guest VM to the DHCP IP that I found within /var/log/messages
Reviewing the logs, I see:
Apr 26 15:19:38 CentOS63 cloud: VM has already saved a password from the password server at dhcpip
Apr 26 15:19:38 CentOS63 cloud: Could not find password server IP in /var/lib/dhclient/resolv.conf.predhclient.eth0
Apr 26 15:19:38 CentOS63 cloud: Did not need to change password.
Apr 26 18:37:51 CentOS63 cloud: Found password server IP dhcpip in /var/lib/dhclient/dhclient-eth0.leases
Apr 26 18:37:51 CentOS63 cloud: Sending request to password server at dhcpip
Apr 26 18:37:51 CentOS63 cloud: Got response from server at dhcpip
Apr 26 18:37:51 CentOS63 cloud: VM has already saved a password from the password server at dhcpip
Apr 26 18:37:51 CentOS63 cloud: Could not find password server IP in /var/lib/dhclient/resolv.conf.predhclient.eth0
Apr 26 18:37:51 CentOS63 cloud: Did not need to change password.
What else would you need to know?
On Apr 26, 2013, at 07:58 PM, Kelven Yang <ke...@citrix.com>> wrote:
You can either try out devCloud, or running management server in its own machine (could be a VM on your KVM host) to get rid of this KVM limitation.
For your password enabled template, depends on which guest OS you are using, to trouble shoot the problem, need a few steps to determine the root cause, but first of all, you need to check the connectivity from within guest VM to DHCP server that you see from inside the guest VM on port 8080.
With limited information, it is hard to tell for now.
Kelven
From: Maurice Lawler <ma...@me.com>>>
Date: Friday, April 26, 2013 1:30 PM
To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>>" <de...@cloudstack.apache.org>>>
Cc: Kelven Yang <ke...@citrix.com>>>
Subject: Re: Cluster Heartbeat
Kelven,
Thanks for the quick response, yes, I have seen no performance issues due to the error; you hit the nail on the head in stating it's annoyance. However, I'll live with the limitation KVM provide; as long as my instances remain running as well as they are.
However, the second part of mine; not sure if you have seen my mailing on the password enabled templates?
- Maurice
On Apr 26, 2013, at 04:27 PM, Kelven Yang <ke...@citrix.com>>> wrote:
Thanks for the info and this is the explanation,
When you add KVM host into management server, it creates network bridge that affects CloudStack management to identify itself (we use MAC address to derive management server ID). This is a limitation of running KVM host and management server in one machine.
The message may be annoying, but it should not affect CloudStack from normal functioning
Kelven
From: Maurice Lawler <ma...@me.com>>>>
Date: Friday, April 26, 2013 1:17 PM
To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>>>" <de...@cloudstack.apache.org>>>>
Cc: Kelven Yang <ke...@citrix.com>>>>
Subject: Re: Cluster Heartbeat
Hello Kelven,
I am using KVM | CentOS 6.3 | CS 4.0.1
All under ONE server. However, I was as mentioned told to remove the 'Down' instances from mshost; restart and the error continues.
- Maurice
On Apr 26, 2013, at 01:13 PM, Kelven Yang <ke...@citrix.com>>>> wrote:
What is your management server cluster setup look like? If it is a one-node setup, how many network interfaces do you have in the management server node?
Kelven
From: Maurice Lawler <ma...@me.com>>>>>
Reply-To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>>>>" <de...@cloudstack.apache.org>>>>>
Date: Thursday, April 25, 2013 8:53 PM
To: Cloud Dev <de...@cloudstack.apache.org>>>>>, "users@cloudstack.apache.org<ma...@cloudstack.apache.org>>>>" <us...@cloudstack.apache.org>>>>>, "users@cloudstack.apache.org<ma...@cloudstack.apache.org>>>>" <us...@cloudstack.apache.org>>>>>
Cc: "cloudstack-dev@incubator.apache.org<ma...@incubator.apache.org>>>>" <cl...@incubator.apache.org>>>>>, "cloudstack-dev@incubator.apache.org<ma...@incubator.apache.org>>>>" <cl...@incubator.apache.org>>>>>
Subject: Cluster Heartbeat
Greetings,
Within my log of /var/log/cloud/management/management-cloud.log I am seeing the following:
2013-04-25 22:50:11,081 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
2013-04-25 22:50:11,081 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
2013-04-25 22:50:11,081 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
2013-04-25 22:50:12,581 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
2013-04-25 22:50:12,582 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
2013-04-25 22:50:12,582 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
2013-04-25 22:50:14,080 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
2013-04-25 22:50:14,080 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
2013-04-25 22:50:14,080 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
2013-04-25 22:50:15,579 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
2013-04-25 22:50:15,580 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
2013-04-25 22:50:15,580 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
2013-04-25 22:50:17,080 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
2013-04-25 22:50:17,081 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
2013-04-25 22:50:17,081 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
2013-04-25 22:50:18,581 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
2013-04-25 22:50:18,581 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
2013-04-25 22:50:18,581 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
It was preivously told to me to remove the "down" enttries within the table mshost; however, when I do that the error is no longer, but rebooting cloud-management if I were to make a change seems to revert whatever was done via table and the error reappears.
Anyone able to help me resolve this once and for all?
- Maurice
Re: Cluster Heartbeat
Posted by Maurice Lawler <ma...@me.com>.
It seems communication is possible, I tested it via telnet on port 8080, from within the guest VM to the DHCP IP that I found within /var/log/messages
Reviewing the logs, I see:
Apr 26 15:19:38 CentOS63 cloud: VM has already saved a password from the password server at dhcpip
Apr 26 15:19:38 CentOS63 cloud: Could not find password server IP in /var/lib/dhclient/resolv.conf.predhclient.eth0
Apr 26 15:19:38 CentOS63 cloud: Did not need to change password.
Apr 26 18:37:51 CentOS63 cloud: Found password server IP dhcpip in /var/lib/dhclient/dhclient-eth0.leases
Apr 26 18:37:51 CentOS63 cloud: Sending request to password server at dhcpip
Apr 26 18:37:51 CentOS63 cloud: Got response from server at dhcpip
Apr 26 18:37:51 CentOS63 cloud: VM has already saved a password from the password server at dhcpip
Apr 26 18:37:51 CentOS63 cloud: Could not find password server IP in /var/lib/dhclient/resolv.conf.predhclient.eth0
Apr 26 18:37:51 CentOS63 cloud: Did not need to change password.
What else would you need to know?
On Apr 26, 2013, at 07:58 PM, Kelven Yang <ke...@citrix.com> wrote:
> You can either try out devCloud, or running management server in its own machine (could be a VM on your KVM host) to get rid of this KVM limitation.
>
> For your password enabled template, depends on which guest OS you are using, to trouble shoot the problem, need a few steps to determine the root cause, but first of all, you need to check the connectivity from within guest VM to DHCP server that you see from inside the guest VM on port 8080.
>
> With limited information, it is hard to tell for now.
>
> Kelven
>
>
> From: Maurice Lawler <ma...@me.com>>
> Date: Friday, April 26, 2013 1:30 PM
> To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>" <de...@cloudstack.apache.org>>
> Cc: Kelven Yang <ke...@citrix.com>>
> Subject: Re: Cluster Heartbeat
>
> Kelven,
>
> Thanks for the quick response, yes, I have seen no performance issues due to the error; you hit the nail on the head in stating it's annoyance. However, I'll live with the limitation KVM provide; as long as my instances remain running as well as they are.
>
> However, the second part of mine; not sure if you have seen my mailing on the password enabled templates?
>
> - Maurice
>
> On Apr 26, 2013, at 04:27 PM, Kelven Yang <ke...@citrix.com>> wrote:
>
> Thanks for the info and this is the explanation,
>
> When you add KVM host into management server, it creates network bridge that affects CloudStack management to identify itself (we use MAC address to derive management server ID). This is a limitation of running KVM host and management server in one machine.
>
> The message may be annoying, but it should not affect CloudStack from normal functioning
>
> Kelven
>
> From: Maurice Lawler <ma...@me.com>>>
> Date: Friday, April 26, 2013 1:17 PM
> To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>>" <de...@cloudstack.apache.org>>>
> Cc: Kelven Yang <ke...@citrix.com>>>
> Subject: Re: Cluster Heartbeat
>
> Hello Kelven,
>
> I am using KVM | CentOS 6.3 | CS 4.0.1
>
> All under ONE server. However, I was as mentioned told to remove the 'Down' instances from mshost; restart and the error continues.
>
> - Maurice
>
> On Apr 26, 2013, at 01:13 PM, Kelven Yang <ke...@citrix.com>>> wrote:
>
> What is your management server cluster setup look like? If it is a one-node setup, how many network interfaces do you have in the management server node?
>
> Kelven
>
> From: Maurice Lawler <ma...@me.com>>>>
> Reply-To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>>>" <de...@cloudstack.apache.org>>>>
> Date: Thursday, April 25, 2013 8:53 PM
> To: Cloud Dev <de...@cloudstack.apache.org>>>>, "users@cloudstack.apache.org<ma...@cloudstack.apache.org>>>" <us...@cloudstack.apache.org>>>>, "users@cloudstack.apache.org<ma...@cloudstack.apache.org>>>" <us...@cloudstack.apache.org>>>>
> Cc: "cloudstack-dev@incubator.apache.org<ma...@incubator.apache.org>>>" <cl...@incubator.apache.org>>>>, "cloudstack-dev@incubator.apache.org<ma...@incubator.apache.org>>>" <cl...@incubator.apache.org>>>>
> Subject: Cluster Heartbeat
>
> Greetings,
>
> Within my log of /var/log/cloud/management/management-cloud.log I am seeing the following:
>
> 2013-04-25 22:50:11,081 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
> 2013-04-25 22:50:11,081 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
> 2013-04-25 22:50:11,081 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
> 2013-04-25 22:50:12,581 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
> 2013-04-25 22:50:12,582 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
> 2013-04-25 22:50:12,582 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
> 2013-04-25 22:50:14,080 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
> 2013-04-25 22:50:14,080 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
> 2013-04-25 22:50:14,080 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
> 2013-04-25 22:50:15,579 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
> 2013-04-25 22:50:15,580 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
> 2013-04-25 22:50:15,580 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
> 2013-04-25 22:50:17,080 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
> 2013-04-25 22:50:17,081 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
> 2013-04-25 22:50:17,081 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
> 2013-04-25 22:50:18,581 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
> 2013-04-25 22:50:18,581 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
> 2013-04-25 22:50:18,581 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
>
>
> It was preivously told to me to remove the "down" enttries within the table mshost; however, when I do that the error is no longer, but rebooting cloud-management if I were to make a change seems to revert whatever was done via table and the error reappears.
>
> Anyone able to help me resolve this once and for all?
>
> - Maurice
Re: Cluster Heartbeat
Posted by Kelven Yang <ke...@citrix.com>.
You can either try out devCloud, or running management server in its own machine (could be a VM on your KVM host) to get rid of this KVM limitation.
For your password enabled template, depends on which guest OS you are using, to trouble shoot the problem, need a few steps to determine the root cause, but first of all, you need to check the connectivity from within guest VM to DHCP server that you see from inside the guest VM on port 8080.
With limited information, it is hard to tell for now.
Kelven
From: Maurice Lawler <ma...@me.com>>
Date: Friday, April 26, 2013 1:30 PM
To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>" <de...@cloudstack.apache.org>>
Cc: Kelven Yang <ke...@citrix.com>>
Subject: Re: Cluster Heartbeat
Kelven,
Thanks for the quick response, yes, I have seen no performance issues due to the error; you hit the nail on the head in stating it's annoyance. However, I'll live with the limitation KVM provide; as long as my instances remain running as well as they are.
However, the second part of mine; not sure if you have seen my mailing on the password enabled templates?
- Maurice
On Apr 26, 2013, at 04:27 PM, Kelven Yang <ke...@citrix.com>> wrote:
Thanks for the info and this is the explanation,
When you add KVM host into management server, it creates network bridge that affects CloudStack management to identify itself (we use MAC address to derive management server ID). This is a limitation of running KVM host and management server in one machine.
The message may be annoying, but it should not affect CloudStack from normal functioning
Kelven
From: Maurice Lawler <ma...@me.com>>>
Date: Friday, April 26, 2013 1:17 PM
To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>>" <de...@cloudstack.apache.org>>>
Cc: Kelven Yang <ke...@citrix.com>>>
Subject: Re: Cluster Heartbeat
Hello Kelven,
I am using KVM | CentOS 6.3 | CS 4.0.1
All under ONE server. However, I was as mentioned told to remove the 'Down' instances from mshost; restart and the error continues.
- Maurice
On Apr 26, 2013, at 01:13 PM, Kelven Yang <ke...@citrix.com>>> wrote:
What is your management server cluster setup look like? If it is a one-node setup, how many network interfaces do you have in the management server node?
Kelven
From: Maurice Lawler <ma...@me.com>>>>
Reply-To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>>>" <de...@cloudstack.apache.org>>>>
Date: Thursday, April 25, 2013 8:53 PM
To: Cloud Dev <de...@cloudstack.apache.org>>>>, "users@cloudstack.apache.org<ma...@cloudstack.apache.org>>>" <us...@cloudstack.apache.org>>>>, "users@cloudstack.apache.org<ma...@cloudstack.apache.org>>>" <us...@cloudstack.apache.org>>>>
Cc: "cloudstack-dev@incubator.apache.org<ma...@incubator.apache.org>>>" <cl...@incubator.apache.org>>>>, "cloudstack-dev@incubator.apache.org<ma...@incubator.apache.org>>>" <cl...@incubator.apache.org>>>>
Subject: Cluster Heartbeat
Greetings,
Within my log of /var/log/cloud/management/management-cloud.log I am seeing the following:
2013-04-25 22:50:11,081 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
2013-04-25 22:50:11,081 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
2013-04-25 22:50:11,081 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
2013-04-25 22:50:12,581 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
2013-04-25 22:50:12,582 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
2013-04-25 22:50:12,582 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
2013-04-25 22:50:14,080 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
2013-04-25 22:50:14,080 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
2013-04-25 22:50:14,080 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
2013-04-25 22:50:15,579 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
2013-04-25 22:50:15,580 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
2013-04-25 22:50:15,580 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
2013-04-25 22:50:17,080 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
2013-04-25 22:50:17,081 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
2013-04-25 22:50:17,081 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
2013-04-25 22:50:18,581 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
2013-04-25 22:50:18,581 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
2013-04-25 22:50:18,581 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
It was preivously told to me to remove the "down" enttries within the table mshost; however, when I do that the error is no longer, but rebooting cloud-management if I were to make a change seems to revert whatever was done via table and the error reappears.
Anyone able to help me resolve this once and for all?
- Maurice
Re: Cluster Heartbeat
Posted by Maurice Lawler <ma...@me.com>.
Kelven,
Thanks for the quick response, yes, I have seen no performance issues due to the error; you hit the nail on the head in stating it's annoyance. However, I'll live with the limitation KVM provide; as long as my instances remain running as well as they are.
However, the second part of mine; not sure if you have seen my mailing on the password enabled templates?
- Maurice
On Apr 26, 2013, at 04:27 PM, Kelven Yang <ke...@citrix.com> wrote:
> Thanks for the info and this is the explanation,
>
> When you add KVM host into management server, it creates network bridge that affects CloudStack management to identify itself (we use MAC address to derive management server ID). This is a limitation of running KVM host and management server in one machine.
>
> The message may be annoying, but it should not affect CloudStack from normal functioning
>
> Kelven
>
> From: Maurice Lawler <ma...@me.com>>
> Date: Friday, April 26, 2013 1:17 PM
> To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>" <de...@cloudstack.apache.org>>
> Cc: Kelven Yang <ke...@citrix.com>>
> Subject: Re: Cluster Heartbeat
>
> Hello Kelven,
>
> I am using KVM | CentOS 6.3 | CS 4.0.1
>
> All under ONE server. However, I was as mentioned told to remove the 'Down' instances from mshost; restart and the error continues.
>
> - Maurice
>
> On Apr 26, 2013, at 01:13 PM, Kelven Yang <ke...@citrix.com>> wrote:
>
> What is your management server cluster setup look like? If it is a one-node setup, how many network interfaces do you have in the management server node?
>
> Kelven
>
> From: Maurice Lawler <ma...@me.com>>>
> Reply-To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>>" <de...@cloudstack.apache.org>>>
> Date: Thursday, April 25, 2013 8:53 PM
> To: Cloud Dev <de...@cloudstack.apache.org>>>, "users@cloudstack.apache.org<ma...@cloudstack.apache.org>>" <us...@cloudstack.apache.org>>>, "users@cloudstack.apache.org<ma...@cloudstack.apache.org>>" <us...@cloudstack.apache.org>>>
> Cc: "cloudstack-dev@incubator.apache.org<ma...@incubator.apache.org>>" <cl...@incubator.apache.org>>>, "cloudstack-dev@incubator.apache.org<ma...@incubator.apache.org>>" <cl...@incubator.apache.org>>>
> Subject: Cluster Heartbeat
>
> Greetings,
>
> Within my log of /var/log/cloud/management/management-cloud.log I am seeing the following:
>
> 2013-04-25 22:50:11,081 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
> 2013-04-25 22:50:11,081 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
> 2013-04-25 22:50:11,081 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
> 2013-04-25 22:50:12,581 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
> 2013-04-25 22:50:12,582 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
> 2013-04-25 22:50:12,582 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
> 2013-04-25 22:50:14,080 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
> 2013-04-25 22:50:14,080 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
> 2013-04-25 22:50:14,080 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
> 2013-04-25 22:50:15,579 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
> 2013-04-25 22:50:15,580 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
> 2013-04-25 22:50:15,580 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
> 2013-04-25 22:50:17,080 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
> 2013-04-25 22:50:17,081 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
> 2013-04-25 22:50:17,081 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
> 2013-04-25 22:50:18,581 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
> 2013-04-25 22:50:18,581 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
> 2013-04-25 22:50:18,581 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
>
>
> It was preivously told to me to remove the "down" enttries within the table mshost; however, when I do that the error is no longer, but rebooting cloud-management if I were to make a change seems to revert whatever was done via table and the error reappears.
>
> Anyone able to help me resolve this once and for all?
>
> - Maurice
Re: Cluster Heartbeat
Posted by Kelven Yang <ke...@citrix.com>.
Thanks for the info and this is the explanation,
When you add KVM host into management server, it creates network bridge that affects CloudStack management to identify itself (we use MAC address to derive management server ID). This is a limitation of running KVM host and management server in one machine.
The message may be annoying, but it should not affect CloudStack from normal functioning
Kelven
From: Maurice Lawler <ma...@me.com>>
Date: Friday, April 26, 2013 1:17 PM
To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>" <de...@cloudstack.apache.org>>
Cc: Kelven Yang <ke...@citrix.com>>
Subject: Re: Cluster Heartbeat
Hello Kelven,
I am using KVM | CentOS 6.3 | CS 4.0.1
All under ONE server. However, I was as mentioned told to remove the 'Down' instances from mshost; restart and the error continues.
- Maurice
On Apr 26, 2013, at 01:13 PM, Kelven Yang <ke...@citrix.com>> wrote:
What is your management server cluster setup look like? If it is a one-node setup, how many network interfaces do you have in the management server node?
Kelven
From: Maurice Lawler <ma...@me.com>>>
Reply-To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>>" <de...@cloudstack.apache.org>>>
Date: Thursday, April 25, 2013 8:53 PM
To: Cloud Dev <de...@cloudstack.apache.org>>>, "users@cloudstack.apache.org<ma...@cloudstack.apache.org>>" <us...@cloudstack.apache.org>>>, "users@cloudstack.apache.org<ma...@cloudstack.apache.org>>" <us...@cloudstack.apache.org>>>
Cc: "cloudstack-dev@incubator.apache.org<ma...@incubator.apache.org>>" <cl...@incubator.apache.org>>>, "cloudstack-dev@incubator.apache.org<ma...@incubator.apache.org>>" <cl...@incubator.apache.org>>>
Subject: Cluster Heartbeat
Greetings,
Within my log of /var/log/cloud/management/management-cloud.log I am seeing the following:
2013-04-25 22:50:11,081 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
2013-04-25 22:50:11,081 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
2013-04-25 22:50:11,081 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
2013-04-25 22:50:12,581 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
2013-04-25 22:50:12,582 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
2013-04-25 22:50:12,582 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
2013-04-25 22:50:14,080 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
2013-04-25 22:50:14,080 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
2013-04-25 22:50:14,080 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
2013-04-25 22:50:15,579 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
2013-04-25 22:50:15,580 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
2013-04-25 22:50:15,580 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
2013-04-25 22:50:17,080 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
2013-04-25 22:50:17,081 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
2013-04-25 22:50:17,081 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
2013-04-25 22:50:18,581 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
2013-04-25 22:50:18,581 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
2013-04-25 22:50:18,581 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
It was preivously told to me to remove the "down" enttries within the table mshost; however, when I do that the error is no longer, but rebooting cloud-management if I were to make a change seems to revert whatever was done via table and the error reappears.
Anyone able to help me resolve this once and for all?
- Maurice
Re: Cluster Heartbeat
Posted by Maurice Lawler <ma...@me.com>.
Hello Kelven,
I am using KVM | CentOS 6.3 | CS 4.0.1
All under ONE server. However, I was as mentioned told to remove the 'Down' instances from mshost; restart and the error continues.
- Maurice
On Apr 26, 2013, at 01:13 PM, Kelven Yang <ke...@citrix.com> wrote:
> What is your management server cluster setup look like? If it is a one-node setup, how many network interfaces do you have in the management server node?
>
> Kelven
>
> From: Maurice Lawler <ma...@me.com>>
> Reply-To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>" <de...@cloudstack.apache.org>>
> Date: Thursday, April 25, 2013 8:53 PM
> To: Cloud Dev <de...@cloudstack.apache.org>>, "users@cloudstack.apache.org<ma...@cloudstack.apache.org>" <us...@cloudstack.apache.org>>, "users@cloudstack.apache.org<ma...@cloudstack.apache.org>" <us...@cloudstack.apache.org>>
> Cc: "cloudstack-dev@incubator.apache.org<ma...@incubator.apache.org>" <cl...@incubator.apache.org>>, "cloudstack-dev@incubator.apache.org<ma...@incubator.apache.org>" <cl...@incubator.apache.org>>
> Subject: Cluster Heartbeat
>
> Greetings,
>
> Within my log of /var/log/cloud/management/management-cloud.log I am seeing the following:
>
> 2013-04-25 22:50:11,081 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
> 2013-04-25 22:50:11,081 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
> 2013-04-25 22:50:11,081 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
> 2013-04-25 22:50:12,581 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
> 2013-04-25 22:50:12,582 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
> 2013-04-25 22:50:12,582 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
> 2013-04-25 22:50:14,080 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
> 2013-04-25 22:50:14,080 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
> 2013-04-25 22:50:14,080 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
> 2013-04-25 22:50:15,579 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
> 2013-04-25 22:50:15,580 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
> 2013-04-25 22:50:15,580 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
> 2013-04-25 22:50:17,080 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
> 2013-04-25 22:50:17,081 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
> 2013-04-25 22:50:17,081 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
> 2013-04-25 22:50:18,581 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
> 2013-04-25 22:50:18,581 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
> 2013-04-25 22:50:18,581 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
>
>
> It was preivously told to me to remove the "down" enttries within the table mshost; however, when I do that the error is no longer, but rebooting cloud-management if I were to make a change seems to revert whatever was done via table and the error reappears.
>
> Anyone able to help me resolve this once and for all?
>
> - Maurice
Re: Cluster Heartbeat
Posted by Kelven Yang <ke...@citrix.com>.
What is your management server cluster setup look like? If it is a one-node setup, how many network interfaces do you have in the management server node?
Kelven
From: Maurice Lawler <ma...@me.com>>
Reply-To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>" <de...@cloudstack.apache.org>>
Date: Thursday, April 25, 2013 8:53 PM
To: Cloud Dev <de...@cloudstack.apache.org>>, "users@cloudstack.apache.org<ma...@cloudstack.apache.org>" <us...@cloudstack.apache.org>>, "users@cloudstack.apache.org<ma...@cloudstack.apache.org>" <us...@cloudstack.apache.org>>
Cc: "cloudstack-dev@incubator.apache.org<ma...@incubator.apache.org>" <cl...@incubator.apache.org>>, "cloudstack-dev@incubator.apache.org<ma...@incubator.apache.org>" <cl...@incubator.apache.org>>
Subject: Cluster Heartbeat
Greetings,
Within my log of /var/log/cloud/management/management-cloud.log I am seeing the following:
2013-04-25 22:50:11,081 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
2013-04-25 22:50:11,081 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
2013-04-25 22:50:11,081 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
2013-04-25 22:50:12,581 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
2013-04-25 22:50:12,582 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
2013-04-25 22:50:12,582 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
2013-04-25 22:50:14,080 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
2013-04-25 22:50:14,080 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
2013-04-25 22:50:14,080 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
2013-04-25 22:50:15,579 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
2013-04-25 22:50:15,580 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
2013-04-25 22:50:15,580 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
2013-04-25 22:50:17,080 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
2013-04-25 22:50:17,081 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
2013-04-25 22:50:17,081 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
2013-04-25 22:50:18,581 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
2013-04-25 22:50:18,581 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
2013-04-25 22:50:18,581 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
It was preivously told to me to remove the "down" enttries within the table mshost; however, when I do that the error is no longer, but rebooting cloud-management if I were to make a change seems to revert whatever was done via table and the error reappears.
Anyone able to help me resolve this once and for all?
- Maurice
Re: Cluster Heartbeat
Posted by Kelven Yang <ke...@citrix.com>.
What is your management server cluster setup look like? If it is a one-node setup, how many network interfaces do you have in the management server node?
Kelven
From: Maurice Lawler <ma...@me.com>>
Reply-To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>" <de...@cloudstack.apache.org>>
Date: Thursday, April 25, 2013 8:53 PM
To: Cloud Dev <de...@cloudstack.apache.org>>, "users@cloudstack.apache.org<ma...@cloudstack.apache.org>" <us...@cloudstack.apache.org>>, "users@cloudstack.apache.org<ma...@cloudstack.apache.org>" <us...@cloudstack.apache.org>>
Cc: "cloudstack-dev@incubator.apache.org<ma...@incubator.apache.org>" <cl...@incubator.apache.org>>, "cloudstack-dev@incubator.apache.org<ma...@incubator.apache.org>" <cl...@incubator.apache.org>>
Subject: Cluster Heartbeat
Greetings,
Within my log of /var/log/cloud/management/management-cloud.log I am seeing the following:
2013-04-25 22:50:11,081 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
2013-04-25 22:50:11,081 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
2013-04-25 22:50:11,081 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
2013-04-25 22:50:12,581 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
2013-04-25 22:50:12,582 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
2013-04-25 22:50:12,582 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
2013-04-25 22:50:14,080 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
2013-04-25 22:50:14,080 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
2013-04-25 22:50:14,080 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
2013-04-25 22:50:15,579 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
2013-04-25 22:50:15,580 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
2013-04-25 22:50:15,580 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
2013-04-25 22:50:17,080 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
2013-04-25 22:50:17,081 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
2013-04-25 22:50:17,081 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
2013-04-25 22:50:18,581 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Detected management node left, id:1, nodeIP:MYIPADDRESS
2013-04-25 22:50:18,581 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Trying to connect to MYIPADDRESS
2013-04-25 22:50:18,581 INFO [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null) Management node 1 is detected inactive by timestamp but is pingable
It was preivously told to me to remove the "down" enttries within the table mshost; however, when I do that the error is no longer, but rebooting cloud-management if I were to make a change seems to revert whatever was done via table and the error reappears.
Anyone able to help me resolve this once and for all?
- Maurice