You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Carlos Reategui <ca...@reategui.com> on 2014/07/25 00:35:57 UTC
Change IP subnet of cluster
Hi All,
Had to move one of my clusters to a new subnet but it is not working (e.g.
192.168.1.0/24 to 10.100.1.0/24). These are the steps I took:
Environment: CS 4.1.1 on Ubuntu 12.04, XenServer 6.1, Shared NFS SR.
1) stopped all instances using cloudstack UI
2) stop cloudstack-management service on MS
3) Used XenCenter to kill the system VMs (no other instances running)
4) Created backup of cloud db.
5) Followed http://support.citrix.com/article/CTX123477 and successfully
changed the IP of hosts. According to XenCenter everything is good
including SR.
6) Changed IP of MS
7) verified communication between MS and Hosts using ssh and ping with new
IPs.
8) used sed to search and replace all old IPs with new IPs in cloud backup
sql file (e.g. sed -i.bak 's/192.168.1./10.100.1./g' clouddb.sql).
9) visually verified all diffs in the sql file and made sure no references
to 192.168 left.
10) loaded up new sql
11) search all files under /etc on MS for old IP. found and
edited: /etc/cloudstack/management/db.properties
12) start cloudstack-management service on MS
Unfortunately things are not working. The MS is apparently unable to
connect to the hosts but I can not figure out why from the logs.
Logs here: https://www.dropbox.com/s/s5glxrbyatmsoug/management-server.log
Any help recovering is appreciated. I do not want to have to re-install
and create/import template for each of the instance VHDs.
thank you,
-Carlos
Re: Change IP subnet of cluster [SOLVED]
Posted by Daan Hoogland <da...@gmail.com>.
H Carlos, glad you figured it out. A colleague had a similar issue but
his finding was that the host table included a timestamp to identify
the management server.
You are right about the replacement of the management server whether
it is mac or timestamp, this will pose a problem.
On Fri, Jul 25, 2014 at 10:43 PM, Carlos Reátegui <cr...@gmail.com> wrote:
> My system is back up and running.
>
> As I suspected in my second email the problem was related to the msid in the mshost table. Upon bringing up my system a new mshost entry was being created for the same MS and for some reason it was unable to connect to my XenServer hosts.
>
> I decided to go back to my edited sql with the new IPs and change the existing mshost entry to have the new msid value:
>
> sed -i.bak4 's/159090355471823/159090355471825/g' cloudstack_cloud-newips.sql
>
> I did a global replace since there are foreign key constraints on the msid that I saw in the host and async_job tables
>
> After reloading this new sql and starting the MS everything is back to normal, but with a new subnet for my hosts and guests (this is a basic network).
>
>
> Question for the Devs:
>
> Lets say the machine my MS was running on crashed and I replaced it with a new machine. Since the msid is derived from the MAC wouldn’t I have encountered this same problem and not been able to have the new machine connect to the hosts?
>
> thanks,
> Carlos
>
>
>
>
> On Jul 25, 2014, at 8:46 AM, Carlos Reátegui <cr...@gmail.com> wrote:
>
>> Any thoughts out there?
>>
>> It keeps trying to connect to the hosts but it is unable to and there are no clues in the logs as to why. I am successfully connected with XenCenter to the pool and also am able to ssh to all the hosts from the MS.
>>
>> What does “Disable Cluster” or “Unmanage Cluster” do? Should I try that and re-enable/manage?
>>
>> From the UI, things appear ok but starting any instance fails.
>>
>> Thanks,
>> Carlos
>>
>> Log snippet form this am:
>>
>> 2014-07-25 21:03:19,599 DEBUG [agent.manager.ClusteredAgentAttache] (StatsCollector-2:null) Seq 2-65931749: Forwarding null to 159090355471823
>> 2014-07-25 21:03:19,599 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-13:null) Seq 2-65931749: Routing from 159090355471825
>> 2014-07-25 21:03:19,599 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-13:null) Seq 2-65931749: Link is closed
>> 2014-07-25 21:03:19,600 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-13:null) Seq 2-65931749: MgmtId 159090355471825: Req: Resource [Host:2] is
>> unreachable: Host 2: Link is closed
>> 2014-07-25 21:03:19,600 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-13:null) Seq 2--1: MgmtId 159090355471825: Req: Routing to peer
>> 2014-07-25 21:03:19,601 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-14:null) Seq 2--1: MgmtId 159090355471825: Req: Cancel request received
>> 2014-07-25 21:03:19,601 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-14:null) Seq 2-65931749: Cancelling.
>> 2014-07-25 21:03:19,601 DEBUG [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 2-65931749: Waiting some more time because this is the current command
>> 2014-07-25 21:03:19,601 DEBUG [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 2-65931749: Waiting some more time because this is the current command
>> 2014-07-25 21:03:19,601 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in
>> error code list for exceptions
>> 2014-07-25 21:03:19,601 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 2-65931749: Timed out on null
>> 2014-07-25 21:03:19,601 DEBUG [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 2-65931749: Cancelling.
>> 2014-07-25 21:03:19,601 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 65931749 to Host 2 timed out after 3600
>> 2014-07-25 21:03:19,601 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 2 statistics.
>> 2014-07-25 21:03:19,601 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host: 2
>> 2014-07-25 21:03:19,606 DEBUG [agent.manager.ClusteredAgentAttache] (StatsCollector-2:null) Seq 3-602278373: Forwarding null to 159090355471823
>> 2014-07-25 21:03:19,607 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-15:null) Seq 3-602278373: Routing from 159090355471825
>> 2014-07-25 21:03:19,607 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-15:null) Seq 3-602278373: Link is closed
>> 2014-07-25 21:03:19,607 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-15:null) Seq 3-602278373: MgmtId 159090355471825: Req: Resource [Host:3] is unreachable: Host 3: Link is closed
>> 2014-07-25 21:03:19,608 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-15:null) Seq 3--1: MgmtId 159090355471825: Req: Routing to peer
>> 2014-07-25 21:03:19,608 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-4:null) Seq 3--1: MgmtId 159090355471825: Req: Cancel request received
>> 2014-07-25 21:03:19,609 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-4:null) Seq 3-602278373: Cancelling.
>> 2014-07-25 21:03:19,609 DEBUG [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-602278373: Waiting some more time because this is the current command
>> 2014-07-25 21:03:19,609 DEBUG [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-602278373: Waiting some more time because this is the current command
>> 2014-07-25 21:03:19,609 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
>> 2014-07-25 21:03:19,609 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-602278373: Timed out on null
>> 2014-07-25 21:03:19,609 DEBUG [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-602278373: Cancelling.
>> 2014-07-25 21:03:19,609 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 602278373 to Host 3 timed out after 3600
>> 2014-07-25 21:03:19,609 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 3 statistics.
>> 2014-07-25 21:03:19,609 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host: 3
>> 2014-07-25 21:03:19,614 DEBUG [agent.manager.ClusteredAgentAttache] (StatsCollector-2:null) Seq 5-1311574501: Forwarding null to 159090355471823
>> 2014-07-25 21:03:19,617 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-1:null) Seq 5-1311574501: Routing from 159090355471825
>> 2014-07-25 21:03:19,617 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-1:null) Seq 5-1311574501: Link is closed
>> 2014-07-25 21:03:19,617 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-1:null) Seq 5-1311574501: MgmtId 159090355471825: Req: Resource [Host:5] is unreachable: Host 5: Link is closed
>>
>>
>>
>> On Jul 24, 2014, at 10:59 PM, Carlos Reátegui <ca...@reategui.com> wrote:
>>
>>> Not sure if it is related but I see 2 entries in the mshost for my same server but with different msid. Both show as ‘Up’. In reading the table comments it seems the msid is based on the MAC. I am guessing this may be due to using a bond and that it may be have selected a different NIC to get the bond MAC from. Is it ok to have both of these entries? Should I mark the old one as Down?
>>>
>>> Along these lines is there something similar with the hosts and that is why the MS is having problems connecting to them, ie. the MACs don’t match?
>>>
>>> thanks,
>>> Carlos
>>>
>>>
>>> On Jul 24, 2014, at 3:35 PM, Carlos Reategui <ca...@reategui.com> wrote:
>>>
>>>> Hi All,
>>>>
>>>> Had to move one of my clusters to a new subnet but it is not working (e.g. 192.168.1.0/24 to 10.100.1.0/24). These are the steps I took:
>>>>
>>>> Environment: CS 4.1.1 on Ubuntu 12.04, XenServer 6.1, Shared NFS SR.
>>>>
>>>> 1) stopped all instances using cloudstack UI
>>>> 2) stop cloudstack-management service on MS
>>>> 3) Used XenCenter to kill the system VMs (no other instances running)
>>>> 4) Created backup of cloud db.
>>>> 5) Followed http://support.citrix.com/article/CTX123477 and successfully changed the IP of hosts. According to XenCenter everything is good including SR.
>>>> 6) Changed IP of MS
>>>> 7) verified communication between MS and Hosts using ssh and ping with new IPs.
>>>> 8) used sed to search and replace all old IPs with new IPs in cloud backup sql file (e.g. sed -i.bak 's/192.168.1./10.100.1./g' clouddb.sql).
>>>> 9) visually verified all diffs in the sql file and made sure no references to 192.168 left.
>>>> 10) loaded up new sql
>>>> 11) search all files under /etc on MS for old IP. found and edited: /etc/cloudstack/management/db.properties
>>>> 12) start cloudstack-management service on MS
>>>>
>>>> Unfortunately things are not working. The MS is apparently unable to connect to the hosts but I can not figure out why from the logs.
>>>>
>>>> Logs here: https://www.dropbox.com/s/s5glxrbyatmsoug/management-server.log
>>>>
>>>> Any help recovering is appreciated. I do not want to have to re-install and create/import template for each of the instance VHDs.
>>>>
>>>> thank you,
>>>> -Carlos
>>>
>>
>
--
Daan
Re: Change IP subnet of cluster [SOLVED]
Posted by Daan Hoogland <da...@gmail.com>.
H Carlos, glad you figured it out. A colleague had a similar issue but
his finding was that the host table included a timestamp to identify
the management server.
You are right about the replacement of the management server whether
it is mac or timestamp, this will pose a problem.
On Fri, Jul 25, 2014 at 10:43 PM, Carlos Reátegui <cr...@gmail.com> wrote:
> My system is back up and running.
>
> As I suspected in my second email the problem was related to the msid in the mshost table. Upon bringing up my system a new mshost entry was being created for the same MS and for some reason it was unable to connect to my XenServer hosts.
>
> I decided to go back to my edited sql with the new IPs and change the existing mshost entry to have the new msid value:
>
> sed -i.bak4 's/159090355471823/159090355471825/g' cloudstack_cloud-newips.sql
>
> I did a global replace since there are foreign key constraints on the msid that I saw in the host and async_job tables
>
> After reloading this new sql and starting the MS everything is back to normal, but with a new subnet for my hosts and guests (this is a basic network).
>
>
> Question for the Devs:
>
> Lets say the machine my MS was running on crashed and I replaced it with a new machine. Since the msid is derived from the MAC wouldn’t I have encountered this same problem and not been able to have the new machine connect to the hosts?
>
> thanks,
> Carlos
>
>
>
>
> On Jul 25, 2014, at 8:46 AM, Carlos Reátegui <cr...@gmail.com> wrote:
>
>> Any thoughts out there?
>>
>> It keeps trying to connect to the hosts but it is unable to and there are no clues in the logs as to why. I am successfully connected with XenCenter to the pool and also am able to ssh to all the hosts from the MS.
>>
>> What does “Disable Cluster” or “Unmanage Cluster” do? Should I try that and re-enable/manage?
>>
>> From the UI, things appear ok but starting any instance fails.
>>
>> Thanks,
>> Carlos
>>
>> Log snippet form this am:
>>
>> 2014-07-25 21:03:19,599 DEBUG [agent.manager.ClusteredAgentAttache] (StatsCollector-2:null) Seq 2-65931749: Forwarding null to 159090355471823
>> 2014-07-25 21:03:19,599 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-13:null) Seq 2-65931749: Routing from 159090355471825
>> 2014-07-25 21:03:19,599 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-13:null) Seq 2-65931749: Link is closed
>> 2014-07-25 21:03:19,600 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-13:null) Seq 2-65931749: MgmtId 159090355471825: Req: Resource [Host:2] is
>> unreachable: Host 2: Link is closed
>> 2014-07-25 21:03:19,600 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-13:null) Seq 2--1: MgmtId 159090355471825: Req: Routing to peer
>> 2014-07-25 21:03:19,601 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-14:null) Seq 2--1: MgmtId 159090355471825: Req: Cancel request received
>> 2014-07-25 21:03:19,601 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-14:null) Seq 2-65931749: Cancelling.
>> 2014-07-25 21:03:19,601 DEBUG [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 2-65931749: Waiting some more time because this is the current command
>> 2014-07-25 21:03:19,601 DEBUG [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 2-65931749: Waiting some more time because this is the current command
>> 2014-07-25 21:03:19,601 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in
>> error code list for exceptions
>> 2014-07-25 21:03:19,601 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 2-65931749: Timed out on null
>> 2014-07-25 21:03:19,601 DEBUG [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 2-65931749: Cancelling.
>> 2014-07-25 21:03:19,601 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 65931749 to Host 2 timed out after 3600
>> 2014-07-25 21:03:19,601 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 2 statistics.
>> 2014-07-25 21:03:19,601 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host: 2
>> 2014-07-25 21:03:19,606 DEBUG [agent.manager.ClusteredAgentAttache] (StatsCollector-2:null) Seq 3-602278373: Forwarding null to 159090355471823
>> 2014-07-25 21:03:19,607 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-15:null) Seq 3-602278373: Routing from 159090355471825
>> 2014-07-25 21:03:19,607 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-15:null) Seq 3-602278373: Link is closed
>> 2014-07-25 21:03:19,607 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-15:null) Seq 3-602278373: MgmtId 159090355471825: Req: Resource [Host:3] is unreachable: Host 3: Link is closed
>> 2014-07-25 21:03:19,608 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-15:null) Seq 3--1: MgmtId 159090355471825: Req: Routing to peer
>> 2014-07-25 21:03:19,608 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-4:null) Seq 3--1: MgmtId 159090355471825: Req: Cancel request received
>> 2014-07-25 21:03:19,609 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-4:null) Seq 3-602278373: Cancelling.
>> 2014-07-25 21:03:19,609 DEBUG [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-602278373: Waiting some more time because this is the current command
>> 2014-07-25 21:03:19,609 DEBUG [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-602278373: Waiting some more time because this is the current command
>> 2014-07-25 21:03:19,609 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
>> 2014-07-25 21:03:19,609 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-602278373: Timed out on null
>> 2014-07-25 21:03:19,609 DEBUG [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-602278373: Cancelling.
>> 2014-07-25 21:03:19,609 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 602278373 to Host 3 timed out after 3600
>> 2014-07-25 21:03:19,609 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 3 statistics.
>> 2014-07-25 21:03:19,609 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host: 3
>> 2014-07-25 21:03:19,614 DEBUG [agent.manager.ClusteredAgentAttache] (StatsCollector-2:null) Seq 5-1311574501: Forwarding null to 159090355471823
>> 2014-07-25 21:03:19,617 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-1:null) Seq 5-1311574501: Routing from 159090355471825
>> 2014-07-25 21:03:19,617 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-1:null) Seq 5-1311574501: Link is closed
>> 2014-07-25 21:03:19,617 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-1:null) Seq 5-1311574501: MgmtId 159090355471825: Req: Resource [Host:5] is unreachable: Host 5: Link is closed
>>
>>
>>
>> On Jul 24, 2014, at 10:59 PM, Carlos Reátegui <ca...@reategui.com> wrote:
>>
>>> Not sure if it is related but I see 2 entries in the mshost for my same server but with different msid. Both show as ‘Up’. In reading the table comments it seems the msid is based on the MAC. I am guessing this may be due to using a bond and that it may be have selected a different NIC to get the bond MAC from. Is it ok to have both of these entries? Should I mark the old one as Down?
>>>
>>> Along these lines is there something similar with the hosts and that is why the MS is having problems connecting to them, ie. the MACs don’t match?
>>>
>>> thanks,
>>> Carlos
>>>
>>>
>>> On Jul 24, 2014, at 3:35 PM, Carlos Reategui <ca...@reategui.com> wrote:
>>>
>>>> Hi All,
>>>>
>>>> Had to move one of my clusters to a new subnet but it is not working (e.g. 192.168.1.0/24 to 10.100.1.0/24). These are the steps I took:
>>>>
>>>> Environment: CS 4.1.1 on Ubuntu 12.04, XenServer 6.1, Shared NFS SR.
>>>>
>>>> 1) stopped all instances using cloudstack UI
>>>> 2) stop cloudstack-management service on MS
>>>> 3) Used XenCenter to kill the system VMs (no other instances running)
>>>> 4) Created backup of cloud db.
>>>> 5) Followed http://support.citrix.com/article/CTX123477 and successfully changed the IP of hosts. According to XenCenter everything is good including SR.
>>>> 6) Changed IP of MS
>>>> 7) verified communication between MS and Hosts using ssh and ping with new IPs.
>>>> 8) used sed to search and replace all old IPs with new IPs in cloud backup sql file (e.g. sed -i.bak 's/192.168.1./10.100.1./g' clouddb.sql).
>>>> 9) visually verified all diffs in the sql file and made sure no references to 192.168 left.
>>>> 10) loaded up new sql
>>>> 11) search all files under /etc on MS for old IP. found and edited: /etc/cloudstack/management/db.properties
>>>> 12) start cloudstack-management service on MS
>>>>
>>>> Unfortunately things are not working. The MS is apparently unable to connect to the hosts but I can not figure out why from the logs.
>>>>
>>>> Logs here: https://www.dropbox.com/s/s5glxrbyatmsoug/management-server.log
>>>>
>>>> Any help recovering is appreciated. I do not want to have to re-install and create/import template for each of the instance VHDs.
>>>>
>>>> thank you,
>>>> -Carlos
>>>
>>
>
--
Daan
Re: Change IP subnet of cluster [SOLVED]
Posted by Carlos Reátegui <cr...@gmail.com>.
My system is back up and running.
As I suspected in my second email the problem was related to the msid in the mshost table. Upon bringing up my system a new mshost entry was being created for the same MS and for some reason it was unable to connect to my XenServer hosts.
I decided to go back to my edited sql with the new IPs and change the existing mshost entry to have the new msid value:
sed -i.bak4 's/159090355471823/159090355471825/g' cloudstack_cloud-newips.sql
I did a global replace since there are foreign key constraints on the msid that I saw in the host and async_job tables
After reloading this new sql and starting the MS everything is back to normal, but with a new subnet for my hosts and guests (this is a basic network).
Question for the Devs:
Lets say the machine my MS was running on crashed and I replaced it with a new machine. Since the msid is derived from the MAC wouldn’t I have encountered this same problem and not been able to have the new machine connect to the hosts?
thanks,
Carlos
On Jul 25, 2014, at 8:46 AM, Carlos Reátegui <cr...@gmail.com> wrote:
> Any thoughts out there?
>
> It keeps trying to connect to the hosts but it is unable to and there are no clues in the logs as to why. I am successfully connected with XenCenter to the pool and also am able to ssh to all the hosts from the MS.
>
> What does “Disable Cluster” or “Unmanage Cluster” do? Should I try that and re-enable/manage?
>
> From the UI, things appear ok but starting any instance fails.
>
> Thanks,
> Carlos
>
> Log snippet form this am:
>
> 2014-07-25 21:03:19,599 DEBUG [agent.manager.ClusteredAgentAttache] (StatsCollector-2:null) Seq 2-65931749: Forwarding null to 159090355471823
> 2014-07-25 21:03:19,599 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-13:null) Seq 2-65931749: Routing from 159090355471825
> 2014-07-25 21:03:19,599 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-13:null) Seq 2-65931749: Link is closed
> 2014-07-25 21:03:19,600 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-13:null) Seq 2-65931749: MgmtId 159090355471825: Req: Resource [Host:2] is
> unreachable: Host 2: Link is closed
> 2014-07-25 21:03:19,600 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-13:null) Seq 2--1: MgmtId 159090355471825: Req: Routing to peer
> 2014-07-25 21:03:19,601 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-14:null) Seq 2--1: MgmtId 159090355471825: Req: Cancel request received
> 2014-07-25 21:03:19,601 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-14:null) Seq 2-65931749: Cancelling.
> 2014-07-25 21:03:19,601 DEBUG [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 2-65931749: Waiting some more time because this is the current command
> 2014-07-25 21:03:19,601 DEBUG [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 2-65931749: Waiting some more time because this is the current command
> 2014-07-25 21:03:19,601 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in
> error code list for exceptions
> 2014-07-25 21:03:19,601 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 2-65931749: Timed out on null
> 2014-07-25 21:03:19,601 DEBUG [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 2-65931749: Cancelling.
> 2014-07-25 21:03:19,601 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 65931749 to Host 2 timed out after 3600
> 2014-07-25 21:03:19,601 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 2 statistics.
> 2014-07-25 21:03:19,601 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host: 2
> 2014-07-25 21:03:19,606 DEBUG [agent.manager.ClusteredAgentAttache] (StatsCollector-2:null) Seq 3-602278373: Forwarding null to 159090355471823
> 2014-07-25 21:03:19,607 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-15:null) Seq 3-602278373: Routing from 159090355471825
> 2014-07-25 21:03:19,607 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-15:null) Seq 3-602278373: Link is closed
> 2014-07-25 21:03:19,607 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-15:null) Seq 3-602278373: MgmtId 159090355471825: Req: Resource [Host:3] is unreachable: Host 3: Link is closed
> 2014-07-25 21:03:19,608 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-15:null) Seq 3--1: MgmtId 159090355471825: Req: Routing to peer
> 2014-07-25 21:03:19,608 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-4:null) Seq 3--1: MgmtId 159090355471825: Req: Cancel request received
> 2014-07-25 21:03:19,609 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-4:null) Seq 3-602278373: Cancelling.
> 2014-07-25 21:03:19,609 DEBUG [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-602278373: Waiting some more time because this is the current command
> 2014-07-25 21:03:19,609 DEBUG [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-602278373: Waiting some more time because this is the current command
> 2014-07-25 21:03:19,609 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
> 2014-07-25 21:03:19,609 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-602278373: Timed out on null
> 2014-07-25 21:03:19,609 DEBUG [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-602278373: Cancelling.
> 2014-07-25 21:03:19,609 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 602278373 to Host 3 timed out after 3600
> 2014-07-25 21:03:19,609 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 3 statistics.
> 2014-07-25 21:03:19,609 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host: 3
> 2014-07-25 21:03:19,614 DEBUG [agent.manager.ClusteredAgentAttache] (StatsCollector-2:null) Seq 5-1311574501: Forwarding null to 159090355471823
> 2014-07-25 21:03:19,617 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-1:null) Seq 5-1311574501: Routing from 159090355471825
> 2014-07-25 21:03:19,617 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-1:null) Seq 5-1311574501: Link is closed
> 2014-07-25 21:03:19,617 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-1:null) Seq 5-1311574501: MgmtId 159090355471825: Req: Resource [Host:5] is unreachable: Host 5: Link is closed
>
>
>
> On Jul 24, 2014, at 10:59 PM, Carlos Reátegui <ca...@reategui.com> wrote:
>
>> Not sure if it is related but I see 2 entries in the mshost for my same server but with different msid. Both show as ‘Up’. In reading the table comments it seems the msid is based on the MAC. I am guessing this may be due to using a bond and that it may be have selected a different NIC to get the bond MAC from. Is it ok to have both of these entries? Should I mark the old one as Down?
>>
>> Along these lines is there something similar with the hosts and that is why the MS is having problems connecting to them, ie. the MACs don’t match?
>>
>> thanks,
>> Carlos
>>
>>
>> On Jul 24, 2014, at 3:35 PM, Carlos Reategui <ca...@reategui.com> wrote:
>>
>>> Hi All,
>>>
>>> Had to move one of my clusters to a new subnet but it is not working (e.g. 192.168.1.0/24 to 10.100.1.0/24). These are the steps I took:
>>>
>>> Environment: CS 4.1.1 on Ubuntu 12.04, XenServer 6.1, Shared NFS SR.
>>>
>>> 1) stopped all instances using cloudstack UI
>>> 2) stop cloudstack-management service on MS
>>> 3) Used XenCenter to kill the system VMs (no other instances running)
>>> 4) Created backup of cloud db.
>>> 5) Followed http://support.citrix.com/article/CTX123477 and successfully changed the IP of hosts. According to XenCenter everything is good including SR.
>>> 6) Changed IP of MS
>>> 7) verified communication between MS and Hosts using ssh and ping with new IPs.
>>> 8) used sed to search and replace all old IPs with new IPs in cloud backup sql file (e.g. sed -i.bak 's/192.168.1./10.100.1./g' clouddb.sql).
>>> 9) visually verified all diffs in the sql file and made sure no references to 192.168 left.
>>> 10) loaded up new sql
>>> 11) search all files under /etc on MS for old IP. found and edited: /etc/cloudstack/management/db.properties
>>> 12) start cloudstack-management service on MS
>>>
>>> Unfortunately things are not working. The MS is apparently unable to connect to the hosts but I can not figure out why from the logs.
>>>
>>> Logs here: https://www.dropbox.com/s/s5glxrbyatmsoug/management-server.log
>>>
>>> Any help recovering is appreciated. I do not want to have to re-install and create/import template for each of the instance VHDs.
>>>
>>> thank you,
>>> -Carlos
>>
>
Re: Change IP subnet of cluster [SOLVED]
Posted by Carlos Reátegui <cr...@gmail.com>.
My system is back up and running.
As I suspected in my second email the problem was related to the msid in the mshost table. Upon bringing up my system a new mshost entry was being created for the same MS and for some reason it was unable to connect to my XenServer hosts.
I decided to go back to my edited sql with the new IPs and change the existing mshost entry to have the new msid value:
sed -i.bak4 's/159090355471823/159090355471825/g' cloudstack_cloud-newips.sql
I did a global replace since there are foreign key constraints on the msid that I saw in the host and async_job tables
After reloading this new sql and starting the MS everything is back to normal, but with a new subnet for my hosts and guests (this is a basic network).
Question for the Devs:
Lets say the machine my MS was running on crashed and I replaced it with a new machine. Since the msid is derived from the MAC wouldn’t I have encountered this same problem and not been able to have the new machine connect to the hosts?
thanks,
Carlos
On Jul 25, 2014, at 8:46 AM, Carlos Reátegui <cr...@gmail.com> wrote:
> Any thoughts out there?
>
> It keeps trying to connect to the hosts but it is unable to and there are no clues in the logs as to why. I am successfully connected with XenCenter to the pool and also am able to ssh to all the hosts from the MS.
>
> What does “Disable Cluster” or “Unmanage Cluster” do? Should I try that and re-enable/manage?
>
> From the UI, things appear ok but starting any instance fails.
>
> Thanks,
> Carlos
>
> Log snippet form this am:
>
> 2014-07-25 21:03:19,599 DEBUG [agent.manager.ClusteredAgentAttache] (StatsCollector-2:null) Seq 2-65931749: Forwarding null to 159090355471823
> 2014-07-25 21:03:19,599 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-13:null) Seq 2-65931749: Routing from 159090355471825
> 2014-07-25 21:03:19,599 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-13:null) Seq 2-65931749: Link is closed
> 2014-07-25 21:03:19,600 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-13:null) Seq 2-65931749: MgmtId 159090355471825: Req: Resource [Host:2] is
> unreachable: Host 2: Link is closed
> 2014-07-25 21:03:19,600 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-13:null) Seq 2--1: MgmtId 159090355471825: Req: Routing to peer
> 2014-07-25 21:03:19,601 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-14:null) Seq 2--1: MgmtId 159090355471825: Req: Cancel request received
> 2014-07-25 21:03:19,601 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-14:null) Seq 2-65931749: Cancelling.
> 2014-07-25 21:03:19,601 DEBUG [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 2-65931749: Waiting some more time because this is the current command
> 2014-07-25 21:03:19,601 DEBUG [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 2-65931749: Waiting some more time because this is the current command
> 2014-07-25 21:03:19,601 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in
> error code list for exceptions
> 2014-07-25 21:03:19,601 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 2-65931749: Timed out on null
> 2014-07-25 21:03:19,601 DEBUG [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 2-65931749: Cancelling.
> 2014-07-25 21:03:19,601 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 65931749 to Host 2 timed out after 3600
> 2014-07-25 21:03:19,601 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 2 statistics.
> 2014-07-25 21:03:19,601 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host: 2
> 2014-07-25 21:03:19,606 DEBUG [agent.manager.ClusteredAgentAttache] (StatsCollector-2:null) Seq 3-602278373: Forwarding null to 159090355471823
> 2014-07-25 21:03:19,607 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-15:null) Seq 3-602278373: Routing from 159090355471825
> 2014-07-25 21:03:19,607 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-15:null) Seq 3-602278373: Link is closed
> 2014-07-25 21:03:19,607 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-15:null) Seq 3-602278373: MgmtId 159090355471825: Req: Resource [Host:3] is unreachable: Host 3: Link is closed
> 2014-07-25 21:03:19,608 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-15:null) Seq 3--1: MgmtId 159090355471825: Req: Routing to peer
> 2014-07-25 21:03:19,608 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-4:null) Seq 3--1: MgmtId 159090355471825: Req: Cancel request received
> 2014-07-25 21:03:19,609 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-4:null) Seq 3-602278373: Cancelling.
> 2014-07-25 21:03:19,609 DEBUG [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-602278373: Waiting some more time because this is the current command
> 2014-07-25 21:03:19,609 DEBUG [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-602278373: Waiting some more time because this is the current command
> 2014-07-25 21:03:19,609 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
> 2014-07-25 21:03:19,609 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-602278373: Timed out on null
> 2014-07-25 21:03:19,609 DEBUG [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-602278373: Cancelling.
> 2014-07-25 21:03:19,609 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 602278373 to Host 3 timed out after 3600
> 2014-07-25 21:03:19,609 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 3 statistics.
> 2014-07-25 21:03:19,609 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host: 3
> 2014-07-25 21:03:19,614 DEBUG [agent.manager.ClusteredAgentAttache] (StatsCollector-2:null) Seq 5-1311574501: Forwarding null to 159090355471823
> 2014-07-25 21:03:19,617 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-1:null) Seq 5-1311574501: Routing from 159090355471825
> 2014-07-25 21:03:19,617 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-1:null) Seq 5-1311574501: Link is closed
> 2014-07-25 21:03:19,617 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-1:null) Seq 5-1311574501: MgmtId 159090355471825: Req: Resource [Host:5] is unreachable: Host 5: Link is closed
>
>
>
> On Jul 24, 2014, at 10:59 PM, Carlos Reátegui <ca...@reategui.com> wrote:
>
>> Not sure if it is related but I see 2 entries in the mshost for my same server but with different msid. Both show as ‘Up’. In reading the table comments it seems the msid is based on the MAC. I am guessing this may be due to using a bond and that it may be have selected a different NIC to get the bond MAC from. Is it ok to have both of these entries? Should I mark the old one as Down?
>>
>> Along these lines is there something similar with the hosts and that is why the MS is having problems connecting to them, ie. the MACs don’t match?
>>
>> thanks,
>> Carlos
>>
>>
>> On Jul 24, 2014, at 3:35 PM, Carlos Reategui <ca...@reategui.com> wrote:
>>
>>> Hi All,
>>>
>>> Had to move one of my clusters to a new subnet but it is not working (e.g. 192.168.1.0/24 to 10.100.1.0/24). These are the steps I took:
>>>
>>> Environment: CS 4.1.1 on Ubuntu 12.04, XenServer 6.1, Shared NFS SR.
>>>
>>> 1) stopped all instances using cloudstack UI
>>> 2) stop cloudstack-management service on MS
>>> 3) Used XenCenter to kill the system VMs (no other instances running)
>>> 4) Created backup of cloud db.
>>> 5) Followed http://support.citrix.com/article/CTX123477 and successfully changed the IP of hosts. According to XenCenter everything is good including SR.
>>> 6) Changed IP of MS
>>> 7) verified communication between MS and Hosts using ssh and ping with new IPs.
>>> 8) used sed to search and replace all old IPs with new IPs in cloud backup sql file (e.g. sed -i.bak 's/192.168.1./10.100.1./g' clouddb.sql).
>>> 9) visually verified all diffs in the sql file and made sure no references to 192.168 left.
>>> 10) loaded up new sql
>>> 11) search all files under /etc on MS for old IP. found and edited: /etc/cloudstack/management/db.properties
>>> 12) start cloudstack-management service on MS
>>>
>>> Unfortunately things are not working. The MS is apparently unable to connect to the hosts but I can not figure out why from the logs.
>>>
>>> Logs here: https://www.dropbox.com/s/s5glxrbyatmsoug/management-server.log
>>>
>>> Any help recovering is appreciated. I do not want to have to re-install and create/import template for each of the instance VHDs.
>>>
>>> thank you,
>>> -Carlos
>>
>
Re: Change IP subnet of cluster
Posted by Carlos Reátegui <cr...@gmail.com>.
Any thoughts out there?
It keeps trying to connect to the hosts but it is unable to and there are no clues in the logs as to why. I am successfully connected with XenCenter to the pool and also am able to ssh to all the hosts from the MS.
What does “Disable Cluster” or “Unmanage Cluster” do? Should I try that and re-enable/manage?
From the UI, things appear ok but starting any instance fails.
Thanks,
Carlos
Log snippet form this am:
2014-07-25 21:03:19,599 DEBUG [agent.manager.ClusteredAgentAttache] (StatsCollector-2:null) Seq 2-65931749: Forwarding null to 159090355471823
2014-07-25 21:03:19,599 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-13:null) Seq 2-65931749: Routing from 159090355471825
2014-07-25 21:03:19,599 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-13:null) Seq 2-65931749: Link is closed
2014-07-25 21:03:19,600 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-13:null) Seq 2-65931749: MgmtId 159090355471825: Req: Resource [Host:2] is
unreachable: Host 2: Link is closed
2014-07-25 21:03:19,600 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-13:null) Seq 2--1: MgmtId 159090355471825: Req: Routing to peer
2014-07-25 21:03:19,601 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-14:null) Seq 2--1: MgmtId 159090355471825: Req: Cancel request received
2014-07-25 21:03:19,601 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-14:null) Seq 2-65931749: Cancelling.
2014-07-25 21:03:19,601 DEBUG [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 2-65931749: Waiting some more time because this is the current command
2014-07-25 21:03:19,601 DEBUG [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 2-65931749: Waiting some more time because this is the current command
2014-07-25 21:03:19,601 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in
error code list for exceptions
2014-07-25 21:03:19,601 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 2-65931749: Timed out on null
2014-07-25 21:03:19,601 DEBUG [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 2-65931749: Cancelling.
2014-07-25 21:03:19,601 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 65931749 to Host 2 timed out after 3600
2014-07-25 21:03:19,601 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 2 statistics.
2014-07-25 21:03:19,601 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host: 2
2014-07-25 21:03:19,606 DEBUG [agent.manager.ClusteredAgentAttache] (StatsCollector-2:null) Seq 3-602278373: Forwarding null to 159090355471823
2014-07-25 21:03:19,607 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-15:null) Seq 3-602278373: Routing from 159090355471825
2014-07-25 21:03:19,607 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-15:null) Seq 3-602278373: Link is closed
2014-07-25 21:03:19,607 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-15:null) Seq 3-602278373: MgmtId 159090355471825: Req: Resource [Host:3] is unreachable: Host 3: Link is closed
2014-07-25 21:03:19,608 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-15:null) Seq 3--1: MgmtId 159090355471825: Req: Routing to peer
2014-07-25 21:03:19,608 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-4:null) Seq 3--1: MgmtId 159090355471825: Req: Cancel request received
2014-07-25 21:03:19,609 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-4:null) Seq 3-602278373: Cancelling.
2014-07-25 21:03:19,609 DEBUG [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-602278373: Waiting some more time because this is the current command
2014-07-25 21:03:19,609 DEBUG [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-602278373: Waiting some more time because this is the current command
2014-07-25 21:03:19,609 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-07-25 21:03:19,609 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-602278373: Timed out on null
2014-07-25 21:03:19,609 DEBUG [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-602278373: Cancelling.
2014-07-25 21:03:19,609 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 602278373 to Host 3 timed out after 3600
2014-07-25 21:03:19,609 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 3 statistics.
2014-07-25 21:03:19,609 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host: 3
2014-07-25 21:03:19,614 DEBUG [agent.manager.ClusteredAgentAttache] (StatsCollector-2:null) Seq 5-1311574501: Forwarding null to 159090355471823
2014-07-25 21:03:19,617 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-1:null) Seq 5-1311574501: Routing from 159090355471825
2014-07-25 21:03:19,617 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-1:null) Seq 5-1311574501: Link is closed
2014-07-25 21:03:19,617 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-1:null) Seq 5-1311574501: MgmtId 159090355471825: Req: Resource [Host:5] is unreachable: Host 5: Link is closed
On Jul 24, 2014, at 10:59 PM, Carlos Reátegui <ca...@reategui.com> wrote:
> Not sure if it is related but I see 2 entries in the mshost for my same server but with different msid. Both show as ‘Up’. In reading the table comments it seems the msid is based on the MAC. I am guessing this may be due to using a bond and that it may be have selected a different NIC to get the bond MAC from. Is it ok to have both of these entries? Should I mark the old one as Down?
>
> Along these lines is there something similar with the hosts and that is why the MS is having problems connecting to them, ie. the MACs don’t match?
>
> thanks,
> Carlos
>
>
> On Jul 24, 2014, at 3:35 PM, Carlos Reategui <ca...@reategui.com> wrote:
>
>> Hi All,
>>
>> Had to move one of my clusters to a new subnet but it is not working (e.g. 192.168.1.0/24 to 10.100.1.0/24). These are the steps I took:
>>
>> Environment: CS 4.1.1 on Ubuntu 12.04, XenServer 6.1, Shared NFS SR.
>>
>> 1) stopped all instances using cloudstack UI
>> 2) stop cloudstack-management service on MS
>> 3) Used XenCenter to kill the system VMs (no other instances running)
>> 4) Created backup of cloud db.
>> 5) Followed http://support.citrix.com/article/CTX123477 and successfully changed the IP of hosts. According to XenCenter everything is good including SR.
>> 6) Changed IP of MS
>> 7) verified communication between MS and Hosts using ssh and ping with new IPs.
>> 8) used sed to search and replace all old IPs with new IPs in cloud backup sql file (e.g. sed -i.bak 's/192.168.1./10.100.1./g' clouddb.sql).
>> 9) visually verified all diffs in the sql file and made sure no references to 192.168 left.
>> 10) loaded up new sql
>> 11) search all files under /etc on MS for old IP. found and edited: /etc/cloudstack/management/db.properties
>> 12) start cloudstack-management service on MS
>>
>> Unfortunately things are not working. The MS is apparently unable to connect to the hosts but I can not figure out why from the logs.
>>
>> Logs here: https://www.dropbox.com/s/s5glxrbyatmsoug/management-server.log
>>
>> Any help recovering is appreciated. I do not want to have to re-install and create/import template for each of the instance VHDs.
>>
>> thank you,
>> -Carlos
>
Re: Change IP subnet of cluster
Posted by Carlos Reátegui <ca...@reategui.com>.
Not sure if it is related but I see 2 entries in the mshost for my same server but with different msid. Both show as ‘Up’. In reading the table comments it seems the msid is based on the MAC. I am guessing this may be due to using a bond and that it may be have selected a different NIC to get the bond MAC from. Is it ok to have both of these entries? Should I mark the old one as Down?
Along these lines is there something similar with the hosts and that is why the MS is having problems connecting to them, ie. the MACs don’t match?
thanks,
Carlos
On Jul 24, 2014, at 3:35 PM, Carlos Reategui <ca...@reategui.com> wrote:
> Hi All,
>
> Had to move one of my clusters to a new subnet but it is not working (e.g. 192.168.1.0/24 to 10.100.1.0/24). These are the steps I took:
>
> Environment: CS 4.1.1 on Ubuntu 12.04, XenServer 6.1, Shared NFS SR.
>
> 1) stopped all instances using cloudstack UI
> 2) stop cloudstack-management service on MS
> 3) Used XenCenter to kill the system VMs (no other instances running)
> 4) Created backup of cloud db.
> 5) Followed http://support.citrix.com/article/CTX123477 and successfully changed the IP of hosts. According to XenCenter everything is good including SR.
> 6) Changed IP of MS
> 7) verified communication between MS and Hosts using ssh and ping with new IPs.
> 8) used sed to search and replace all old IPs with new IPs in cloud backup sql file (e.g. sed -i.bak 's/192.168.1./10.100.1./g' clouddb.sql).
> 9) visually verified all diffs in the sql file and made sure no references to 192.168 left.
> 10) loaded up new sql
> 11) search all files under /etc on MS for old IP. found and edited: /etc/cloudstack/management/db.properties
> 12) start cloudstack-management service on MS
>
> Unfortunately things are not working. The MS is apparently unable to connect to the hosts but I can not figure out why from the logs.
>
> Logs here: https://www.dropbox.com/s/s5glxrbyatmsoug/management-server.log
>
> Any help recovering is appreciated. I do not want to have to re-install and create/import template for each of the instance VHDs.
>
> thank you,
> -Carlos