You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by benoit lair <ku...@gmail.com> on 2013/02/11 17:17:16 UTC

Production recovery procedure for VPC Virtual router into advanced zone - CS 4.0.0

Hello,

I am currently working on a model of preproduction  of cs 4.0.0 on a centos
6.3 server management.

I have deployed my zones, networks, primary iscsi storages onto my array,
configured my clusters with xcp 1.1.

Now i'm looking for deploying the VPC features.

I have configured my vpc, defined my CIDR, created 3 tiers networks,
deployed 1 vm onto each tier network.

All worked very great.

Now i've tried to force crash of the vpc virtual router in order to
troubleshooting the recovery of production of the vpc.

I got some problems in certain cases, after crashing my vpc router, my tier
networks didn't mount onto the virtual router.

This caused lossing the avalibility to recover production of my tiered
networking vms.

So i'm looking for differents recovery procedures in order to recover a
full fonctionnal vpc cloud.

If somebody got best practices in order to recover a vpc router, i would
appreciate any help.


Thanks.

Benoit Lair.

Re: Production recovery procedure for VPC Virtual router into advanced zone - CS 4.0.0

Posted by Ahmad Emneina <ae...@gmail.com>.
Going to loop in Anthony here, try to get his attention.


On Thu, Feb 14, 2013 at 2:10 AM, benoit lair <ku...@gmail.com> wrote:

> Hello,
>
>
> Also when i try to reboot a missing network (id 240, 172.28.1.1 as vpc
> gateway) here are the log entries concerning this action :
>
> 2013-02-14 10:53:39,703 DEBUG [agent.transport.Request]
> (DirectAgent-259:null) Seq 14-1794713198: Processing:  { Ans: , MgmtId:
> 165013918770818, via: 14, Ver: v1, Flags: 110,
> [{"SetupGuestNetworkAnswer":{"result":false,"details":"Creating guest
> network failed due to com.cloud.utils.exception.CloudRuntimeException:
> callHostPlugin failed for cmd: routerProxy with args args: vpc_guestnw.sh
> 169.254.3.22 -D -d eth2 -i 172.28.2.1 -g 172.28.2.1 -m 24 -n 172.28.2.0 -s
> 195.7.111.30,91.151.119.67 -e z03.cloud.web-et-solutions.com,  due to
> Failed to create input stream: Read timed out","wait":0}}] }
> 2013-02-14 10:53:39,703 DEBUG [agent.manager.AgentAttache]
> (DirectAgent-259:null) Seq 14-1794713202: Sending now.  is current
> sequence.
> 2013-02-14 10:53:39,703 DEBUG [agent.transport.Request]
> (Network-Scavenger-1:null) Seq 14-1794713198: Received:  { Ans: , MgmtId:
> 165013918770818, via: 14, Ver: v1, Flags: 110, { SetupGuestNetworkAnswer }
> }
> 2013-02-14 10:53:39,703 DEBUG [agent.transport.Request]
> (DirectAgent-259:null) Seq 14-1794713202: Executing:  { Cmd , MgmtId:
> 165013918770818, via: 14, Ver: v1, Flags: 100111,
> [{"SetupGuestNetworkCommand":{"dhcpRange":"172.28.1.1","networkDomain":"
> z03.cloud.web-et-solutions.com
>
> ","defaultDns1":"195.7.111.30","defaultDns2":"91.151.119.67","isRedundant":false,"add":false,"nic":{"deviceId":3,"networkRateMbps":200,"defaultNic":false,"uuid":"f1c50f06-db19-4293-9842-d37bb4d8bba2","ip":"172.28.1.1","netmask":"255.255.255.0","mac":"02:00:20:cc:00:08","dns1":"195.7.111.30","dns2":"91.151.119.67","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://2510","isolationUri":"vlan://2510","isSecurityGroupEnabled":false,"name":"ground1"},"accessDetails":{"router.guest.ip":"172.28.1.1","guest.vlan.tag":"2510","guest.network.gateway":"172.28.1.1","guest.bridge":"172.28.1.255","
> router.name":"r-18183-VM","router.ip":"169.254.3.22"},"wait":0}}] }
> 2013-02-14 10:53:39,703 WARN
> [network.router.VpcVirtualNetworkApplianceManagerImpl]
> (Network-Scavenger-1:null) Unable to destroy guest network on router
> VM[DomainRouter|r-18183-VM]
> 2013-02-14 10:53:39,703 WARN
> [network.router.VpcVirtualNetworkApplianceManagerImpl]
> (Network-Scavenger-1:null) Failed to destroy guest network config
> Ntwk[239|Guest|12] on router VM[DomainRouter|r-18183-VM]
> 2013-02-14 10:53:39,704 WARN  [network.element.VpcVirtualRouterElement]
> (Network-Scavenger-1:null) Failed to unplug nic in network
> Ntwk[239|Guest|12] for virtual router VM[DomainRouter|r-18183-VM]
> 2013-02-14 10:53:39,704 WARN  [cloud.network.NetworkManagerImpl]
> (Network-Scavenger-1:null) Unable to complete shutdown of the network
> elements due to element: VpcVirtualRouter
> 2013-02-14 10:53:39,703 DEBUG [agent.manager.DirectAgentAttache]
> (DirectAgent-20:null) Seq 14-1794713202: Executing request
> 2013-02-14 10:53:39,876 DEBUG [agent.manager.AgentAttache]
> (RouterStatusMonitor-1:null) Seq 14-1794713256: Waited too long.
> 2013-02-14 10:53:39,876 INFO  [utils.exception.CSExceptionErrorCode]
> (RouterStatusMonitor-1:null) Could not find exception:
> com.cloud.exception.OperationTimedoutException in error code list for
> exceptions
> 2013-02-14 10:53:39,876 WARN  [agent.manager.AgentAttache]
> (RouterStatusMonitor-1:null) Seq 14-1794713256: Timed out on Seq
> 14-1794713256:  { Cmd , MgmtId: 165013918770818, via: 14, Ver: v1, Flags:
> 100111,
>
> [{"CheckS2SVpnConnectionsCommand":{"vpnIps":[],"accessDetails":{"router.ip":"169.254.3.191","
> router.name":"r-18165-VM"},"wait":30}}] }
> 2013-02-14 10:53:39,876 DEBUG [agent.manager.AgentAttache]
> (RouterStatusMonitor-1:null) Seq 14-1794713256: Cancelling.
> 2013-02-14 10:53:39,877 WARN  [agent.manager.AgentManagerImpl]
> (RouterStatusMonitor-1:null) Operation timed out: Commands 1794713256 to
> Host 14 timed out after 30
> 2013-02-14 10:53:39,877 WARN
> [network.router.VirtualNetworkApplianceManagerImpl]
> (RouterStatusMonitor-1:null) Unable to update router r-18165-VM's VPN
> connection status
> 2013-02-14 10:53:39,879 DEBUG [agent.manager.DirectAgentAttache]
> (DirectAgent-20:null) Seq 14-1794713202: Response Received:
> 2013-02-14 10:53:39,880 DEBUG [agent.transport.Request]
> (DirectAgent-20:null) Seq 14-1794713202: Processing:  { Ans: , MgmtId:
> 165013918770818, via: 14, Ver: v1, Flags: 110,
> [{"SetupGuestNetworkAnswer":{"result":false,"details":"Can not find vif
> with mac 02:00:20:cc:00:08 for VM r-18183-VM","wait":0}}]
>
> For info :
> 172.28.1.1 is the LB tier network (id 240)
> 172.28.2.1 is a tier network without LB (id 239)
> 172.28.3.1 is a tier network without LB (id 241)
>
> Finally, i get the network 239 remounted as eth2 (172.25.2.1) into vpc vr,
> but no way to get back the others (240 and 241)
>
> The log entries show errors when trying to reboot others networks (also
> when i rebooted the network with the UI dashboard, i finaly got an error
> pop message).
>
> Regards, Benoit Lair.
>
>
>
> 2013/2/11 benoit lair <ku...@gmail.com>
>
> > Hello,
> >
> > I am currently working on a model of preproduction  of cs 4.0.0 on a
> centos
> > 6.3 server management.
> >
> > I have deployed my zones, networks, primary iscsi storages onto my array,
> > configured my clusters with xcp 1.1.
> >
> > Now i'm looking for deploying the VPC features.
> >
> > I have configured my vpc, defined my CIDR, created 3 tiers networks,
> > deployed 1 vm onto each tier network.
> >
> > All worked very great.
> >
> > Now i've tried to force crash of the vpc virtual router in order to
> > troubleshooting the recovery of production of the vpc.
> >
> > I got some problems in certain cases, after crashing my vpc router, my
> > tier networks didn't mount onto the virtual router.
> >
> > This caused lossing the avalibility to recover production of my tiered
> > networking vms.
> >
> > So i'm looking for differents recovery procedures in order to recover a
> > full fonctionnal vpc cloud.
> >
> > If somebody got best practices in order to recover a vpc router, i would
> > appreciate any help.
> >
> >
> > Thanks.
> >
> > Benoit Lair.
> >
>

Re: Production recovery procedure for VPC Virtual router into advanced zone - CS 4.0.0

Posted by benoit lair <ku...@gmail.com>.
Hello,


Also when i try to reboot a missing network (id 240, 172.28.1.1 as vpc
gateway) here are the log entries concerning this action :

2013-02-14 10:53:39,703 DEBUG [agent.transport.Request]
(DirectAgent-259:null) Seq 14-1794713198: Processing:  { Ans: , MgmtId:
165013918770818, via: 14, Ver: v1, Flags: 110,
[{"SetupGuestNetworkAnswer":{"result":false,"details":"Creating guest
network failed due to com.cloud.utils.exception.CloudRuntimeException:
callHostPlugin failed for cmd: routerProxy with args args: vpc_guestnw.sh
169.254.3.22 -D -d eth2 -i 172.28.2.1 -g 172.28.2.1 -m 24 -n 172.28.2.0 -s
195.7.111.30,91.151.119.67 -e z03.cloud.web-et-solutions.com,  due to
Failed to create input stream: Read timed out","wait":0}}] }
2013-02-14 10:53:39,703 DEBUG [agent.manager.AgentAttache]
(DirectAgent-259:null) Seq 14-1794713202: Sending now.  is current sequence.
2013-02-14 10:53:39,703 DEBUG [agent.transport.Request]
(Network-Scavenger-1:null) Seq 14-1794713198: Received:  { Ans: , MgmtId:
165013918770818, via: 14, Ver: v1, Flags: 110, { SetupGuestNetworkAnswer } }
2013-02-14 10:53:39,703 DEBUG [agent.transport.Request]
(DirectAgent-259:null) Seq 14-1794713202: Executing:  { Cmd , MgmtId:
165013918770818, via: 14, Ver: v1, Flags: 100111,
[{"SetupGuestNetworkCommand":{"dhcpRange":"172.28.1.1","networkDomain":"
z03.cloud.web-et-solutions.com
","defaultDns1":"195.7.111.30","defaultDns2":"91.151.119.67","isRedundant":false,"add":false,"nic":{"deviceId":3,"networkRateMbps":200,"defaultNic":false,"uuid":"f1c50f06-db19-4293-9842-d37bb4d8bba2","ip":"172.28.1.1","netmask":"255.255.255.0","mac":"02:00:20:cc:00:08","dns1":"195.7.111.30","dns2":"91.151.119.67","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://2510","isolationUri":"vlan://2510","isSecurityGroupEnabled":false,"name":"ground1"},"accessDetails":{"router.guest.ip":"172.28.1.1","guest.vlan.tag":"2510","guest.network.gateway":"172.28.1.1","guest.bridge":"172.28.1.255","
router.name":"r-18183-VM","router.ip":"169.254.3.22"},"wait":0}}] }
2013-02-14 10:53:39,703 WARN
[network.router.VpcVirtualNetworkApplianceManagerImpl]
(Network-Scavenger-1:null) Unable to destroy guest network on router
VM[DomainRouter|r-18183-VM]
2013-02-14 10:53:39,703 WARN
[network.router.VpcVirtualNetworkApplianceManagerImpl]
(Network-Scavenger-1:null) Failed to destroy guest network config
Ntwk[239|Guest|12] on router VM[DomainRouter|r-18183-VM]
2013-02-14 10:53:39,704 WARN  [network.element.VpcVirtualRouterElement]
(Network-Scavenger-1:null) Failed to unplug nic in network
Ntwk[239|Guest|12] for virtual router VM[DomainRouter|r-18183-VM]
2013-02-14 10:53:39,704 WARN  [cloud.network.NetworkManagerImpl]
(Network-Scavenger-1:null) Unable to complete shutdown of the network
elements due to element: VpcVirtualRouter
2013-02-14 10:53:39,703 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-20:null) Seq 14-1794713202: Executing request
2013-02-14 10:53:39,876 DEBUG [agent.manager.AgentAttache]
(RouterStatusMonitor-1:null) Seq 14-1794713256: Waited too long.
2013-02-14 10:53:39,876 INFO  [utils.exception.CSExceptionErrorCode]
(RouterStatusMonitor-1:null) Could not find exception:
com.cloud.exception.OperationTimedoutException in error code list for
exceptions
2013-02-14 10:53:39,876 WARN  [agent.manager.AgentAttache]
(RouterStatusMonitor-1:null) Seq 14-1794713256: Timed out on Seq
14-1794713256:  { Cmd , MgmtId: 165013918770818, via: 14, Ver: v1, Flags:
100111,
[{"CheckS2SVpnConnectionsCommand":{"vpnIps":[],"accessDetails":{"router.ip":"169.254.3.191","
router.name":"r-18165-VM"},"wait":30}}] }
2013-02-14 10:53:39,876 DEBUG [agent.manager.AgentAttache]
(RouterStatusMonitor-1:null) Seq 14-1794713256: Cancelling.
2013-02-14 10:53:39,877 WARN  [agent.manager.AgentManagerImpl]
(RouterStatusMonitor-1:null) Operation timed out: Commands 1794713256 to
Host 14 timed out after 30
2013-02-14 10:53:39,877 WARN
[network.router.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:null) Unable to update router r-18165-VM's VPN
connection status
2013-02-14 10:53:39,879 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-20:null) Seq 14-1794713202: Response Received:
2013-02-14 10:53:39,880 DEBUG [agent.transport.Request]
(DirectAgent-20:null) Seq 14-1794713202: Processing:  { Ans: , MgmtId:
165013918770818, via: 14, Ver: v1, Flags: 110,
[{"SetupGuestNetworkAnswer":{"result":false,"details":"Can not find vif
with mac 02:00:20:cc:00:08 for VM r-18183-VM","wait":0}}]

For info :
172.28.1.1 is the LB tier network (id 240)
172.28.2.1 is a tier network without LB (id 239)
172.28.3.1 is a tier network without LB (id 241)

Finally, i get the network 239 remounted as eth2 (172.25.2.1) into vpc vr,
but no way to get back the others (240 and 241)

The log entries show errors when trying to reboot others networks (also
when i rebooted the network with the UI dashboard, i finaly got an error
pop message).

Regards, Benoit Lair.



2013/2/11 benoit lair <ku...@gmail.com>

> Hello,
>
> I am currently working on a model of preproduction  of cs 4.0.0 on a centos
> 6.3 server management.
>
> I have deployed my zones, networks, primary iscsi storages onto my array,
> configured my clusters with xcp 1.1.
>
> Now i'm looking for deploying the VPC features.
>
> I have configured my vpc, defined my CIDR, created 3 tiers networks,
> deployed 1 vm onto each tier network.
>
> All worked very great.
>
> Now i've tried to force crash of the vpc virtual router in order to
> troubleshooting the recovery of production of the vpc.
>
> I got some problems in certain cases, after crashing my vpc router, my
> tier networks didn't mount onto the virtual router.
>
> This caused lossing the avalibility to recover production of my tiered
> networking vms.
>
> So i'm looking for differents recovery procedures in order to recover a
> full fonctionnal vpc cloud.
>
> If somebody got best practices in order to recover a vpc router, i would
> appreciate any help.
>
>
> Thanks.
>
> Benoit Lair.
>

Re: Production recovery procedure for VPC Virtual router into advanced zone - CS 4.0.0

Posted by benoit lair <ku...@gmail.com>.
Hello Ahmad,

I've tried so to restart VPC (made call with the UI dashboard), so my vpc
vr has been recreated, i've seen onto management-server.log the request for
recreating my vpc which is :

2013-02-14 10:17:01,008 DEBUG [agent.transport.Request]
(DirectAgent-289:null) Seq 14-1794712962: Executing:  { Cmd , MgmtId:
165013918770818, via: 14, Ver: v1, Flags: 100111,
[{"StartCommand":{"vm":{"id":18183,"name":"r-18183-VM","bootloader":"PyGrub","type":"DomainRouter","cpus":1,"speed":500,"minRam":134217728,"maxRam":134217728,"arch":"x86_64","os":"Debian
GNU/Linux 6(64-bit)","bootArgs":" vpccidr=172.28.0.0/16 domain=
z03.cloud.web-et-solutions.com dns1=195.7.111.30 dns2=91.151.119.67
template=domP name=r-18183-VM eth0ip=169.254.3.22 eth0mask=255.255.0.0
type=vpcrouter
disable_rp_filter=true","rebootOnCrash":false,"enableHA":true,"limitCpuUse":false,"vncPassword":"9c5468337802250c","params":{},"uuid":"02f8704c-77dc-487c-b070-408c6e6c1060","disks":[{"id":8069,"name":"ROOT-18183","mountPoint":"/iqn.1984-05.com.dell:powervault.md3200i.6d4ae52000b3a400000000005019e097/6","path":"294080ab-1f18-48e7-9941-d8b449e2fecb","size":2147483648,"type":"ROOT","storagePoolType":"IscsiLUN","storagePoolUuid":"49ddaea6-0f2c-327a-844f-fd3f64f6963c","deviceId":0}],"nics":[{"deviceId":0,"networkRateMbps":-1,"defaultNic":false,"uuid":"c558916e-95f7-415d-9533-88a84b35d8ab","ip":"169.254.3.22","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:03:16","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false}]},"wait":0}},{"check.CheckSshCommand":{"ip":"169.254.3.22","port":3922,"interval":6,"retries":100,"name":"r-18183-VM","wait":0}},{"GetDomRVersionCmd":{"accessDetails":{"router.ip":"169.254.3.22","
router.name
":"r-18183-VM"},"wait":0}},{},{"PlugNicCommand":{"nic":{"deviceId":1,"networkRateMbps":200,"defaultNic":true,"uuid":"a4a6aed0-1d0d-457d-8df2-478b462354c4","ip":"10.14.10.36","netmask":"255.254.0.0","gateway":"10.14.0.1","mac":"06:b7:74:00:07:8f","dns1":"195.7.111.30","dns2":"91.151.119.67","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://2020","isolationUri":"vlan://2020","isSecurityGroupEnabled":false,"name":"ground1"},"instanceName":"r-18183-VM","wait":0}},{"routing.IpAssocVpcCommand":{"ipAddresses":[{"accountId":2,"publicIp":"10.14.10.36","sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":false,"vlanId":"2020","vlanGateway":"10.14.0.1","vlanNetmask":"255.254.0.0","vifMacAddress":"06:b7:74:00:07:8f","networkRate":200,"trafficType":"Public","networkName":"ground1"}],"accessDetails":{"router.guest.ip":"10.14.10.36","zone.network.type":"Advanced","router.ip":"169.254.3.22","
router.name
":"r-18183-VM"},"wait":0}},{"routing.SetSourceNatCommand":{"ipAddress":{"accountId":2,"publicIp":"10.14.10.36","sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":false,"vlanId":"2020","vlanGateway":"10.14.0.1","vlanNetmask":"255.254.0.0","vifMacAddress":"06:b7:74:00:07:8f","networkRate":200,"trafficType":"Public","networkName":"ground1"},"add":true,"accessDetails":{"zone.network.type":"Advanced","router.ip":"169.254.3.22","
router.name
":"r-18183-VM"},"wait":0}},{"PlugNicCommand":{"nic":{"deviceId":2,"networkRateMbps":200,"defaultNic":false,"uuid":"eb0254f5-d484-4d56-86cb-0a6b0c844e6d","ip":"172.28.2.1","netmask":"255.255.255.0","mac":"02:00:6e:c8:00:06","dns1":"195.7.111.30","dns2":"91.151.119.67","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://2502","isolationUri":"vlan://2502","isSecurityGroupEnabled":false,"name":"ground1"},"instanceName":"r-18183-VM","wait":0}},{"SetupGuestNetworkCommand":{"dhcpRange":"172.28.2.1","networkDomain":"
z03.cloud.web-et-solutions.com
","defaultDns1":"195.7.111.30","defaultDns2":"91.151.119.67","isRedundant":false,"add":true,"nic":
{"deviceId":2,"networkRateMbps":200,"defaultNic":false,"uuid":"eb0254f5-d484-4d56-86cb-0a6b0c844e6d","ip":"172.28.2.1","netmask":"255.255.255.0","mac":"02:00:6e:c8:00:06","dns1":"195.7.111.30","dns2":"91.151.119.67","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://2502","isolationUri":"vlan://2502","isSecurityGroupEnabled":false,"name":"ground1"},"accessDetails":{"router.guest.ip":"172.28.2.1","guest.vlan.tag":"2502","guest.network.gateway":"172.28.2.1","guest.bridge":"172.28.2.255","
router.name":"r-18183-VM","router.ip":"169.254.3.22"},"wait":0}},
{"PlugNicCommand":{"nic":{"deviceId":3,"networkRateMbps":200,"defaultNic":false,"uuid":"f1c50f06-db19-4293-9842-d37bb4d8bba2","ip":"172.28.1.1","netmask":"255.255.255.0","mac":"02:00:20:cc:00:08","dns1":"195.7.111.30","dns2":"91.151.119.67","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://2510","isolationUri":"vlan://2510","isSecurityGroupEnabled":false,"name":"ground1"},"instanceName":"r-18183-VM","wait":0}},{"SetupGuestNetworkCommand":{"dhcpRange":"172.28.1.1","networkDomain":"
z03.cloud.web-et-solutions.com
","defaultDns1":"195.7.111.30","defaultDns2":"91.151.119.67","isRedundant":false,"add":true,"nic":{"deviceId":3,"networkRateMbps":200,"defaultNic":false,"uuid":"f1c50f06-db19-4293-9842-d37bb4d8bba2","ip":"172.28.1.1","netmask":"255.255.255.0","mac":"02:00:20:cc:00:08","dns1":"195.7.111.30","dns2":"91.151.119.67","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://2510","isolationUri":"vlan://2510","isSecurityGroupEnabled":false,"name":"ground1"},"accessDetails":{"router.guest.ip":"172.28.1.1","guest.vlan.tag":"2510","guest.network.gateway":"172.28.1.1","guest.bridge":"172.28.1.255","
router.name
":"r-18183-VM","router.ip":"169.254.3.22"},"wait":0}},{"PlugNicCommand":{"nic":{"deviceId":4,"networkRateMbps":200,"defaultNic":false,"uuid":"f969ed36-9a5e-4683-8107-b17232347e42","ip":"172.28.3.1","netmask":"255.255.255.0","mac":"02:00:40:42:00:08","dns1":"195.7.111.30","dns2":"91.151.119.67","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://2501","isolationUri":"vlan://2501","isSecurityGroupEnabled":false,"name":"ground1"},"instanceName":"r-18183-VM","wait":0}},{"SetupGuestNetworkCommand":{"dhcpRange":"172.28.3.1","networkDomain":"
z03.cloud.web-et-solutions.com
","defaultDns1":"195.7.111.30","defaultDns2":"91.151.119.67","isRedundant":false,"add":true,"nic":{"deviceId":4,"networkRateMbps":200,"defaultNic":false,"uuid":"f969ed36-9a5e-4683-8107-b17232347e42","ip":"172.28.3.1","netmask":"255.255.255.0","mac":"02:00:40:42:00:08","dns1":"195.7.111.30","dns2":"91.151.119.67","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://2501","isolationUri":"vlan://2501","isSecurityGroupEnabled":false,"name":"ground1"},"accessDetails":{"router.guest.ip":"172.28.3.1","guest.vlan.tag":"2501","guest.network.gateway":"172.28.3.1","guest.bridge":"172.28.3.255","
router.name
":"r-18183-VM","router.ip":"169.254.3.22"},"wait":0}},{"routing.DhcpEntryCommand":{"vmMac":"02:00:19:85:00:06","vmIpAddress":"172.28.1.241","vmName":"tier-frontal2-test2","defaultRouter":"172.28.1.1","accessDetails":{"router.guest.ip":"172.28.1.1","zone.network.type":"Advanced","
router.name
":"r-18183-VM","router.ip":"169.254.3.22"},"wait":0}},{"routing.VmDataCommand":{"vmIpAddress":"172.28.1.241","vmName":"tier-frontal2-test2","accessDetails":{"router.guest.ip":"172.28.1.1","zone.network.type":"Advanced","router.ip":"169.254.3.22","
router.name":"r-18183-VM"},"wait":0}},{}] }


So I wait for the recreation of my vpc, but i do not have my vlan
interfaces :

I took a look onto my management-server.log and saw this :

2013-02-14 10:31:29,529 DEBUG [xen.resource.XenServerConnectionPool]
(DirectAgent-430:null) XmlRpcException for method: host.call_plugin due to
Failed to create input stream: Read timed out.  Reconnecting...retry=1
2013-02-14 10:31:29,529 DEBUG [xen.resource.CitrixResourceBase]
(DirectAgent-430:null) callHostPlugin failed for cmd: routerProxy with args
args: vpc_guestnw.sh 169.254.3.22 -C -d eth2 -i 172.28.2.1 -g 172.28.2.1 -m
24 -n 172.28.2.0 -s 195.7.111.30,91.151.119.67 -e
z03.cloud.web-et-solutions.com,  due to Failed to create input stream: Read
timed out
2013-02-14 10:31:29,529 WARN  [xen.resource.CitrixResourceBase]
(DirectAgent-430:null) Creating guest network failed due to
com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for
cmd: routerProxy with args args: vpc_guestnw.sh 169.254.3.22 -C -d eth2 -i
172.28.2.1 -g 172.28.2.1 -m 24 -n 172.28.2.0 -s 195.7.111.30,91.151.119.67
-e z03.cloud.web-et-solutions.com,  due to Failed to create input stream:
Read timed out
com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for
cmd: routerProxy with args args: vpc_guestnw.sh 169.254.3.22 -C -d eth2 -i
172.28.2.1 -g 172.28.2.1 -m 24 -n 172.28.2.0 -s 195.7.111.30,91.151.119.67
-e z03.cloud.web-et-solutions.com,  due to Failed to create input stream:
Read timed out
        at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.callHostPlugin(CitrixResourceBase.java:3749)
        at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:7375)
        at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:541)
        at
com.cloud.hypervisor.xen.resource.XcpServerResource.executeRequest(XcpServerResource.java:52)
        at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:191)
        at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:679)
2013-02-14 10:31:29,530 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-430:null) Seq 14-1794712962: Cancelling because one of the
answers is false and it is stop on error.
2013-02-14 10:31:29,530 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-430:null) Seq 14-1794712962: Response Received:
2013-02-14 10:31:29,531 DEBUG [agent.transport.Request]
(DirectAgent-430:null) Seq 14-1794712962: Processing:  { Ans: , MgmtId:
165013918770818, via: 14, Ver: v1, Flags: 110,
[{"StartAnswer":{"vm":{"id":18183,"name":"r-18183-VM","bootloader":"PyGrub","type":"DomainRouter","cpus":1,"speed":500,"minRam":134217728,"maxRam":134217728,"arch":"x86_64","os":"Debian
GNU/Linux 6(64-bit)","bootArgs":" vpccidr=172.28.0.0/16 domain=
z03.cloud.web-et-solutions.com dns1=195.7.111.30 dns2=91.151.119.67
template=domP name=r-18183-VM eth0ip=169.254.3.22 eth0mask=255.255.0.0
type=vpcrouter
disable_rp_filter=true","rebootOnCrash":false,"enableHA":true,"limitCpuUse":false,"vncPassword":"9c5468337802250c","params":{},"uuid":"02f8704c-77dc-487c-b070-408c6e6c1060","disks":[{"id":8069,"name":"ROOT-18183","mountPoint":"/iqn.1984-05.com.dell:powervault.md3200i.6d4ae52000b3a400000000005019e097/6","path":"294080ab-1f18-48e7-9941-d8b449e2fecb","size":2147483648,"type":"ROOT","storagePoolType":"IscsiLUN","storagePoolUuid":"49ddaea6-0f2c-327a-844f-fd3f64f6963c","deviceId":0}],"nics":[{"deviceId":0,"networkRateMbps":-1,"defaultNic":false,"uuid":"c558916e-95f7-415d-9533-88a84b35d8ab","ip":"169.254.3.22","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:03:16","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false}]},"result":true,"wait":0}},{"check.CheckSshAnswer":{"result":true,"wait":0}},{"GetDomRVersionAnswer":{"templateVersion":"Cloudstack
Release 3.0 Mon Feb 6 15:10:04 PST
2012","scriptsVersion":"cd49bf69b25051c4dd7751c571c9e6f9","result":true,"details":"Cloudstack
Release 3.0 Mon Feb 6 15:10:04 PST
2012&cd49bf69b25051c4dd7751c571c9e6f9","wait":0}},{"NetworkUsageAnswer":{"routerName":"r-18183-VM","bytesSent":0,"bytesReceived":0,"result":true,"wait":0}},{"PlugNicAnswer":{"result":true,"details":"success","wait":0}},{"routing.IpAssocAnswer":{"results":["10.14.10.36
-
success"],"result":true,"wait":0}},{"routing.SetSourceNatAnswer":{"result":true,"details":"success","wait":0}},{"PlugNicAnswer":{"result":true,"details":"success","wait":0}},{"SetupGuestNetworkAnswer":{"result":false,"details":"Creating
guest network failed due to
com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for
cmd: routerProxy with args args: vpc_guestnw.sh 169.254.3.22 -C -d eth2 -i
172.28.2.1 -g 172.28.2.1 -m 24 -n 172.28.2.0 -s 195.7.111.30,91.151.119.67
-e z03.cloud.web-et-solutions.com,  due to Failed to create input stream:
Read timed out","wait":0}}] }


How can i troubleshoot this ?


Thanks a lot for your help.

Regards, Benoit Lair.

2013/2/13 Ahmad Emneina <ae...@gmail.com>

> Hey Benoit,
>
> I queried some engineers inside the citrix qa team. They told me to try the
> restartVPC api call. This should recreate the router. If it doesnt we
> should log a bug, there should be a way to recover from this easily.
>
>
> https://incubator.apache.org/cloudstack/docs/api/apidocs-4.0.0/root_admin/restartVPC.html
>
>
> On Wed, Feb 13, 2013 at 9:53 AM, benoit lair <ku...@gmail.com>
> wrote:
>
> > Hello Ahmad,
> >
> > I shuttted down the vpc vr during an update of acls sent by the mgmt
> > server, so mgmt server sais me the vpc vr has 3 rules, in reality, an
> > ipatbles -L give me only 2 acls.
> >
> > More simply, when i create a new vpc, create 3 tiers netoworks, deployed
> a
> > vm into each tier network, all seems to be ok. Problems begin when i
> simply
> > reboot the vpc vr.
> >
> > After a reboot, i have no more the interfaces of my tier (vlans)
> interfaces
> > mounted.
> >
> > Even if i reboot each tier network, i can't recover access to my vlans so
> > no access to my vms.
> >
> > In fact, i only have the control interface (1698.254.x.x) the public
> > interface (in my case 10.14.10.36) and the localhost one.
> >
> > My master cidr is 172.28.0.0/16, my tiers are 172.28.1/24,
> > 172.28.2.1/24and
> > 172.28.3.1/24, before reboot i got 3 interfaces : 172.28.1.1, 172.28.2.1
> > and 172.28.3.1 after reboot no more interfaces...
> >
> > I have already tried to destroy the vpc vr, triggered the creation of a
> new
> > vm to force cs to detect that vpc vr is missing and so relaunch the
> > installation of the vpc vr.
> >
> > Even in this case, the vpc is redeployed with only mgmt , public and
> > loopback interface, no vlan interfaces.
> >
> > Have you already encountered this problem ?
> >
> > 2013/2/11 Anthony Xu <Xu...@citrix.com>
> >
> > > You can try restart network with clean checked in UI, it will try to
> > > destroy Virtual router on this network and create a new one.
> > >
> > >
> > > Anthony
> > >
> > > > -----Original Message-----
> > > > From: Ahmad Emneina [mailto:aemneina@gmail.com]
> > > > Sent: Monday, February 11, 2013 10:48 AM
> > > > To: Cloudstack users
> > > > Subject: Re: Production recovery procedure for VPC Virtual router
> into
> > > > advanced zone - CS 4.0.0
> > > >
> > > > how did you crash the vpc router?
> > > >
> > > > did spinning up a vm, or restarting a vm, in the affected network
> > > > restore
> > > > the router? (this is probably the easiest way to correct the issue,
> if
> > > > cloudstack understands the router is gone.)
> > > >
> > > > I would only spin up a vm after a destroy router is called, insuring
> > > > cloudstack understands that router needs to be recreated.
> > > >
> > > >
> > > > On Mon, Feb 11, 2013 at 8:17 AM, benoit lair <ku...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hello,
> > > > >
> > > > > I am currently working on a model of preproduction  of cs 4.0.0 on
> a
> > > > centos
> > > > > 6.3 server management.
> > > > >
> > > > > I have deployed my zones, networks, primary iscsi storages onto my
> > > > array,
> > > > > configured my clusters with xcp 1.1.
> > > > >
> > > > > Now i'm looking for deploying the VPC features.
> > > > >
> > > > > I have configured my vpc, defined my CIDR, created 3 tiers
> networks,
> > > > > deployed 1 vm onto each tier network.
> > > > >
> > > > > All worked very great.
> > > > >
> > > > > Now i've tried to force crash of the vpc virtual router in order to
> > > > > troubleshooting the recovery of production of the vpc.
> > > > >
> > > > > I got some problems in certain cases, after crashing my vpc router,
> > > > my tier
> > > > > networks didn't mount onto the virtual router.
> > > > >
> > > > > This caused lossing the avalibility to recover production of my
> > > > tiered
> > > > > networking vms.
> > > > >
> > > > > So i'm looking for differents recovery procedures in order to
> recover
> > > > a
> > > > > full fonctionnal vpc cloud.
> > > > >
> > > > > If somebody got best practices in order to recover a vpc router, i
> > > > would
> > > > > appreciate any help.
> > > > >
> > > > >
> > > > > Thanks.
> > > > >
> > > > > Benoit Lair.
> > > > >
> > >
> >
>

Re: Production recovery procedure for VPC Virtual router into advanced zone - CS 4.0.0

Posted by Ahmad Emneina <ae...@gmail.com>.
Hey Benoit,

I queried some engineers inside the citrix qa team. They told me to try the
restartVPC api call. This should recreate the router. If it doesnt we
should log a bug, there should be a way to recover from this easily.

https://incubator.apache.org/cloudstack/docs/api/apidocs-4.0.0/root_admin/restartVPC.html


On Wed, Feb 13, 2013 at 9:53 AM, benoit lair <ku...@gmail.com> wrote:

> Hello Ahmad,
>
> I shuttted down the vpc vr during an update of acls sent by the mgmt
> server, so mgmt server sais me the vpc vr has 3 rules, in reality, an
> ipatbles -L give me only 2 acls.
>
> More simply, when i create a new vpc, create 3 tiers netoworks, deployed a
> vm into each tier network, all seems to be ok. Problems begin when i simply
> reboot the vpc vr.
>
> After a reboot, i have no more the interfaces of my tier (vlans) interfaces
> mounted.
>
> Even if i reboot each tier network, i can't recover access to my vlans so
> no access to my vms.
>
> In fact, i only have the control interface (1698.254.x.x) the public
> interface (in my case 10.14.10.36) and the localhost one.
>
> My master cidr is 172.28.0.0/16, my tiers are 172.28.1/24,
> 172.28.2.1/24and
> 172.28.3.1/24, before reboot i got 3 interfaces : 172.28.1.1, 172.28.2.1
> and 172.28.3.1 after reboot no more interfaces...
>
> I have already tried to destroy the vpc vr, triggered the creation of a new
> vm to force cs to detect that vpc vr is missing and so relaunch the
> installation of the vpc vr.
>
> Even in this case, the vpc is redeployed with only mgmt , public and
> loopback interface, no vlan interfaces.
>
> Have you already encountered this problem ?
>
> 2013/2/11 Anthony Xu <Xu...@citrix.com>
>
> > You can try restart network with clean checked in UI, it will try to
> > destroy Virtual router on this network and create a new one.
> >
> >
> > Anthony
> >
> > > -----Original Message-----
> > > From: Ahmad Emneina [mailto:aemneina@gmail.com]
> > > Sent: Monday, February 11, 2013 10:48 AM
> > > To: Cloudstack users
> > > Subject: Re: Production recovery procedure for VPC Virtual router into
> > > advanced zone - CS 4.0.0
> > >
> > > how did you crash the vpc router?
> > >
> > > did spinning up a vm, or restarting a vm, in the affected network
> > > restore
> > > the router? (this is probably the easiest way to correct the issue, if
> > > cloudstack understands the router is gone.)
> > >
> > > I would only spin up a vm after a destroy router is called, insuring
> > > cloudstack understands that router needs to be recreated.
> > >
> > >
> > > On Mon, Feb 11, 2013 at 8:17 AM, benoit lair <ku...@gmail.com>
> > > wrote:
> > >
> > > > Hello,
> > > >
> > > > I am currently working on a model of preproduction  of cs 4.0.0 on a
> > > centos
> > > > 6.3 server management.
> > > >
> > > > I have deployed my zones, networks, primary iscsi storages onto my
> > > array,
> > > > configured my clusters with xcp 1.1.
> > > >
> > > > Now i'm looking for deploying the VPC features.
> > > >
> > > > I have configured my vpc, defined my CIDR, created 3 tiers networks,
> > > > deployed 1 vm onto each tier network.
> > > >
> > > > All worked very great.
> > > >
> > > > Now i've tried to force crash of the vpc virtual router in order to
> > > > troubleshooting the recovery of production of the vpc.
> > > >
> > > > I got some problems in certain cases, after crashing my vpc router,
> > > my tier
> > > > networks didn't mount onto the virtual router.
> > > >
> > > > This caused lossing the avalibility to recover production of my
> > > tiered
> > > > networking vms.
> > > >
> > > > So i'm looking for differents recovery procedures in order to recover
> > > a
> > > > full fonctionnal vpc cloud.
> > > >
> > > > If somebody got best practices in order to recover a vpc router, i
> > > would
> > > > appreciate any help.
> > > >
> > > >
> > > > Thanks.
> > > >
> > > > Benoit Lair.
> > > >
> >
>

Re: Production recovery procedure for VPC Virtual router into advanced zone - CS 4.0.0

Posted by benoit lair <ku...@gmail.com>.
Hello Ahmad,

I shuttted down the vpc vr during an update of acls sent by the mgmt
server, so mgmt server sais me the vpc vr has 3 rules, in reality, an
ipatbles -L give me only 2 acls.

More simply, when i create a new vpc, create 3 tiers netoworks, deployed a
vm into each tier network, all seems to be ok. Problems begin when i simply
reboot the vpc vr.

After a reboot, i have no more the interfaces of my tier (vlans) interfaces
mounted.

Even if i reboot each tier network, i can't recover access to my vlans so
no access to my vms.

In fact, i only have the control interface (1698.254.x.x) the public
interface (in my case 10.14.10.36) and the localhost one.

My master cidr is 172.28.0.0/16, my tiers are 172.28.1/24, 172.28.2.1/24and
172.28.3.1/24, before reboot i got 3 interfaces : 172.28.1.1, 172.28.2.1
and 172.28.3.1 after reboot no more interfaces...

I have already tried to destroy the vpc vr, triggered the creation of a new
vm to force cs to detect that vpc vr is missing and so relaunch the
installation of the vpc vr.

Even in this case, the vpc is redeployed with only mgmt , public and
loopback interface, no vlan interfaces.

Have you already encountered this problem ?

2013/2/11 Anthony Xu <Xu...@citrix.com>

> You can try restart network with clean checked in UI, it will try to
> destroy Virtual router on this network and create a new one.
>
>
> Anthony
>
> > -----Original Message-----
> > From: Ahmad Emneina [mailto:aemneina@gmail.com]
> > Sent: Monday, February 11, 2013 10:48 AM
> > To: Cloudstack users
> > Subject: Re: Production recovery procedure for VPC Virtual router into
> > advanced zone - CS 4.0.0
> >
> > how did you crash the vpc router?
> >
> > did spinning up a vm, or restarting a vm, in the affected network
> > restore
> > the router? (this is probably the easiest way to correct the issue, if
> > cloudstack understands the router is gone.)
> >
> > I would only spin up a vm after a destroy router is called, insuring
> > cloudstack understands that router needs to be recreated.
> >
> >
> > On Mon, Feb 11, 2013 at 8:17 AM, benoit lair <ku...@gmail.com>
> > wrote:
> >
> > > Hello,
> > >
> > > I am currently working on a model of preproduction  of cs 4.0.0 on a
> > centos
> > > 6.3 server management.
> > >
> > > I have deployed my zones, networks, primary iscsi storages onto my
> > array,
> > > configured my clusters with xcp 1.1.
> > >
> > > Now i'm looking for deploying the VPC features.
> > >
> > > I have configured my vpc, defined my CIDR, created 3 tiers networks,
> > > deployed 1 vm onto each tier network.
> > >
> > > All worked very great.
> > >
> > > Now i've tried to force crash of the vpc virtual router in order to
> > > troubleshooting the recovery of production of the vpc.
> > >
> > > I got some problems in certain cases, after crashing my vpc router,
> > my tier
> > > networks didn't mount onto the virtual router.
> > >
> > > This caused lossing the avalibility to recover production of my
> > tiered
> > > networking vms.
> > >
> > > So i'm looking for differents recovery procedures in order to recover
> > a
> > > full fonctionnal vpc cloud.
> > >
> > > If somebody got best practices in order to recover a vpc router, i
> > would
> > > appreciate any help.
> > >
> > >
> > > Thanks.
> > >
> > > Benoit Lair.
> > >
>

RE: Production recovery procedure for VPC Virtual router into advanced zone - CS 4.0.0

Posted by Anthony Xu <Xu...@citrix.com>.
You can try restart network with clean checked in UI, it will try to destroy Virtual router on this network and create a new one.


Anthony

> -----Original Message-----
> From: Ahmad Emneina [mailto:aemneina@gmail.com]
> Sent: Monday, February 11, 2013 10:48 AM
> To: Cloudstack users
> Subject: Re: Production recovery procedure for VPC Virtual router into
> advanced zone - CS 4.0.0
> 
> how did you crash the vpc router?
> 
> did spinning up a vm, or restarting a vm, in the affected network
> restore
> the router? (this is probably the easiest way to correct the issue, if
> cloudstack understands the router is gone.)
> 
> I would only spin up a vm after a destroy router is called, insuring
> cloudstack understands that router needs to be recreated.
> 
> 
> On Mon, Feb 11, 2013 at 8:17 AM, benoit lair <ku...@gmail.com>
> wrote:
> 
> > Hello,
> >
> > I am currently working on a model of preproduction  of cs 4.0.0 on a
> centos
> > 6.3 server management.
> >
> > I have deployed my zones, networks, primary iscsi storages onto my
> array,
> > configured my clusters with xcp 1.1.
> >
> > Now i'm looking for deploying the VPC features.
> >
> > I have configured my vpc, defined my CIDR, created 3 tiers networks,
> > deployed 1 vm onto each tier network.
> >
> > All worked very great.
> >
> > Now i've tried to force crash of the vpc virtual router in order to
> > troubleshooting the recovery of production of the vpc.
> >
> > I got some problems in certain cases, after crashing my vpc router,
> my tier
> > networks didn't mount onto the virtual router.
> >
> > This caused lossing the avalibility to recover production of my
> tiered
> > networking vms.
> >
> > So i'm looking for differents recovery procedures in order to recover
> a
> > full fonctionnal vpc cloud.
> >
> > If somebody got best practices in order to recover a vpc router, i
> would
> > appreciate any help.
> >
> >
> > Thanks.
> >
> > Benoit Lair.
> >

Re: Production recovery procedure for VPC Virtual router into advanced zone - CS 4.0.0

Posted by Ahmad Emneina <ae...@gmail.com>.
how did you crash the vpc router?

did spinning up a vm, or restarting a vm, in the affected network restore
the router? (this is probably the easiest way to correct the issue, if
cloudstack understands the router is gone.)

I would only spin up a vm after a destroy router is called, insuring
cloudstack understands that router needs to be recreated.


On Mon, Feb 11, 2013 at 8:17 AM, benoit lair <ku...@gmail.com> wrote:

> Hello,
>
> I am currently working on a model of preproduction  of cs 4.0.0 on a centos
> 6.3 server management.
>
> I have deployed my zones, networks, primary iscsi storages onto my array,
> configured my clusters with xcp 1.1.
>
> Now i'm looking for deploying the VPC features.
>
> I have configured my vpc, defined my CIDR, created 3 tiers networks,
> deployed 1 vm onto each tier network.
>
> All worked very great.
>
> Now i've tried to force crash of the vpc virtual router in order to
> troubleshooting the recovery of production of the vpc.
>
> I got some problems in certain cases, after crashing my vpc router, my tier
> networks didn't mount onto the virtual router.
>
> This caused lossing the avalibility to recover production of my tiered
> networking vms.
>
> So i'm looking for differents recovery procedures in order to recover a
> full fonctionnal vpc cloud.
>
> If somebody got best practices in order to recover a vpc router, i would
> appreciate any help.
>
>
> Thanks.
>
> Benoit Lair.
>