You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@cloudstack.apache.org by "Jayapal Reddy (JIRA)" <ji...@apache.org> on 2013/12/18 08:01:15 UTC
[jira] [Resolved] (CLOUDSTACK-5505) [Automation] Private gateway
not getting programmed in VPC router
[ https://issues.apache.org/jira/browse/CLOUDSTACK-5505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jayapal Reddy resolved CLOUDSTACK-5505.
---------------------------------------
Resolution: Fixed
> [Automation] Private gateway not getting programmed in VPC router
> ------------------------------------------------------------------
>
> Key: CLOUDSTACK-5505
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5505
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the default.)
> Components: Network Controller
> Affects Versions: 4.3.0
> Environment: KVM
> Branch : 4.3
> Reporter: Rayees Namathponnan
> Assignee: Jayapal Reddy
> Priority: Critical
> Fix For: 4.3.0
>
> Attachments: CLOUDSTACK-5505.rar
>
>
> This issue found as part of regression automation
> integration.component.test_vpc_vms_deployment.TestVMDeployVPC.test_07_delete_network_with_rules
> Test cases performing below steps
> 1726 # Validate the following
> 1727 # 1. Create a VPC with cidr - 10.1.1.1/16
> 1728 # 2. Add network1(10.1.1.1/24) and network2(10.1.2.1/24) to this VPC.
> 1729 # 3. Deploy vm1 and vm2 in network1 and vm3 and vm4 in network2.
> 1730 # 4. Create a PF /Static Nat/LB rule for vms in network1.
> 1731 # 5. Create a PF /Static Nat/LB rule for vms in network2.
> 1732 # 6. Create ingress network ACL for allowing all the above rules from
> 1733 # public ip range on network1 and network2.
> 1734 # 7. Create egress network ACL for network1 and network2 to access
> 1735 # google.com.
> 1736 # 8. Create a private gateway for this VPC and add a static route to
> 1737 # this gateway
> 1738 # 9. Create a VPN gateway for this VPC and add a static route to this
> 1739 # gateway.
> 1740 # 10. Make sure that all the PF,LB, Static NAT rules work as expected
> 1741 # 11. Make sure that we are able to access google from all user Vms
> 1742 # 12. Make sure that the newly added private gateway's and VPN
> 1743 # gateway's static routes work as expected.
> 1744 # Steps:
> 1745 # 1. Delete the 1st network.
> 1746 # 2. Delete account
> 1747 # Validations:
> 1748 # 1. As part of network deletion all the resources attached with
> 1749 # network should get deleted. All other VMs and rules shall work as
> 1750 # expected
> 1751 # 2. All the resources associated with account should be deleted
> Steps to reproduce
> Step 1 : Create advanced zone in KVM
> Step 2 : Create VPC
> Step 3 : create 2 network inside
> Step 4 : Create Private Gateway
> Step 5 : Add static route
> Result
> Failed to add static route with error "Failed to create static route"
> MS log (look lob Job-Executor-102:ctx-12de6228)
> -------------------------------------------------------------------
> 2013-12-13 15:23:12,952 DEBUG [c.c.a.ApiServlet] (catalina-exec-7:ctx-0c9be854) ===START=== 10.214.5.33 -- GET command=createStaticRoute&response=json&sessionkey=RP7BqoAP0Roa1gkg68z4YhvFAO4%3D&g
> atewayid=528f0ba2-bd3d-4e76-9d45-d567cbd47d67&cidr=10.2.3.0%2F24&_=1386976993256
> 2013-12-13 15:23:12,964 DEBUG [c.c.n.v.VpcManagerImpl] (catalina-exec-7:ctx-0c9be854 ctx-e80dd4b0) Adding static route StaticRoute[7c159911-d38a-4dbf-a98d-708b099c7389|10.2.3.0/24|4]
> 2013-12-13 15:23:13,049 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-102:ctx-12de6228) Add job-2928 into job monitoring
> 2013-12-13 15:23:13,049 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-102:ctx-12de6228) Executing AsyncJobVO {id:2928, userId: 2, accountId: 2, instanceType: StaticRoute, instanceId: 4, cm
> d: org.apache.cloudstack.api.command.user.vpc.CreateStaticRouteCmd, cmdInfo: {"id":"4","response":"json","sessionkey":"RP7BqoAP0Roa1gkg68z4YhvFAO4\u003d","cmdEventType":"STATIC.ROUTE.CREATE","ctxU
> serId":"2","gatewayid":"528f0ba2-bd3d-4e76-9d45-d567cbd47d67","httpmethod":"GET","_":"1386976993256","ctxAccountId":"2","ctxStartEventId":"14430","cidr":"10.2.3.0/24"}, cmdVersion: 0, status: IN_P
> ROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 29066118877352, completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
> 2013-12-13 15:23:13,053 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (catalina-exec-7:ctx-0c9be854 ctx-e80dd4b0) submit async job-2928, details: AsyncJobVO {id:2928, userId: 2, accountId: 2, instanceTy
> pe: StaticRoute, instanceId: 4, cmd: org.apache.cloudstack.api.command.user.vpc.CreateStaticRouteCmd, cmdInfo: {"id":"4","response":"json","sessionkey":"RP7BqoAP0Roa1gkg68z4YhvFAO4\u003d","cmdEven
> tType":"STATIC.ROUTE.CREATE","ctxUserId":"2","gatewayid":"528f0ba2-bd3d-4e76-9d45-d567cbd47d67","httpmethod":"GET","_":"1386976993256","ctxAccountId":"2","ctxStartEventId":"14430","cidr":"10.2.3.0
> /24"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 29066118877352, completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
> 2013-12-13 15:23:13,055 DEBUG [c.c.a.ApiServlet] (catalina-exec-7:ctx-0c9be854 ctx-e80dd4b0) ===END=== 10.214.5.33 -- GET command=createStaticRoute&response=json&sessionkey=RP7BqoAP0Roa1gkg68z4Y
> hvFAO4%3D&gatewayid=528f0ba2-bd3d-4e76-9d45-d567cbd47d67&cidr=10.2.3.0%2F24&_=1386976993256
> 2013-12-13 15:23:13,060 DEBUG [c.c.n.v.VpcManagerImpl] (Job-Executor-102:ctx-12de6228 ctx-e80dd4b0) Applying static routes for vpc [VPC [77-RayeesVPC]
> 2013-12-13 15:23:13,070 DEBUG [c.c.a.t.Request] (Job-Executor-102:ctx-12de6228 ctx-e80dd4b0) Seq 2-1828592525: Sending { Cmd , MgmtId: 29066118877352, via: 2(Rack2Host12.lab.vmops.com), Ver: v1,
> Flags: 100001, [{"com.cloud.agent.api.routing.SetStaticRouteCommand":{"staticRoutes":[{"id":4,"uuid":"7c159911-d38a-4dbf-a98d-708b099c7389","targetCidr":"10.2.3.0/24","accountId":2,"domainId":1,"g
> atewayId":4,"state":"Add","vpcId":77,"vlanTag":"2364","gateway":"10.2.3.1","netmask":"255.255.255.0","ipAddress":"10.2.3.2"}],"accessDetails":{"zone.network.type":"Advanced","router.ip":"169.254.2
> .108","router.name":"r-706-QA"},"wait":0}}] }
> 2013-12-13 15:23:13,308 DEBUG [c.c.a.t.Request] (AgentManager-Handler-12:null) Seq 2-1828592525: Processing: { Ans: , MgmtId: 29066118877352, via: 2, Ver: v1, Flags: 0, [{"com.cloud.agent.api.rou
> ting.SetStaticRouteAnswer":{"results":["Failed"],"result":false,"wait":0}}] }
> 2013-12-13 15:23:13,308 DEBUG [c.c.a.t.Request] (Job-Executor-102:ctx-12de6228 ctx-e80dd4b0) Seq 2-1828592525: Received: { Ans: , MgmtId: 29066118877352, via: 2, Ver: v1, Flags: 0, { SetStaticRou
> teAnswer } }
> 2013-12-13 15:23:13,309 DEBUG [c.c.a.ApiServlet] (catalina-exec-14:ctx-75d4f4a5) ===START=== 10.223.240.194 -- GET signature=m9opIJ%2BNeEGTeQ0NEl5GZnZtitU%3D&apiKey=LepyvCSNC55jvLUEoHU3GP9oI4QiT
> vgt9dsCdALVIf3NxlNBq7dmcHdCb_6NenlyV3QwRmEn5DnGp__U-5UwXA&command=queryAsyncJobResult&response=json&jobid=08d6fc64-19d4-433b-bcd2-20d487e43dd5
> 2013-12-13 15:23:13,314 DEBUG [c.c.n.v.VpcManagerImpl] (Job-Executor-102:ctx-12de6228 ctx-e80dd4b0) Revoking static route StaticRoute[7c159911-d38a-4dbf-a98d-708b099c7389|10.2.3.0/24|4]
> 2013-12-13 15:23:13,319 DEBUG [c.c.n.v.VpcManagerImpl] (Job-Executor-102:ctx-12de6228 ctx-e80dd4b0) Marked static route StaticRoute[7c159911-d38a-4dbf-a98d-708b099c7389|10.2.3.0/24|4] with state R
> evoke
> 2013-12-13 15:23:13,322 DEBUG [c.c.n.v.VpcManagerImpl] (Job-Executor-102:ctx-12de6228 ctx-e80dd4b0) Applying static routes for vpc [VPC [77-RayeesVPC]
> 2013-12-13 15:23:13,331 DEBUG [c.c.a.ApiServlet] (catalina-exec-14:ctx-75d4f4a5 ctx-e2e99bdf ctx-a1b9efd2) ===END=== 10.223.240.194 -- GET signature=m9opIJ%2BNeEGTeQ0NEl5GZnZtitU%3D&apiKey=Lepyv
> CSNC55jvLUEoHU3GP9oI4QiTvgt9dsCdALVIf3NxlNBq7dmcHdCb_6NenlyV3QwRmEn5DnGp__U-5UwXA&command=queryAsyncJobResult&response=json&jobid=08d6fc64-19d4-433b-bcd2-20d487e43dd5
> 2013-12-13 15:23:13,340 DEBUG [c.c.a.t.Request] (Job-Executor-102:ctx-12de6228 ctx-e80dd4b0) Seq 2-1828592526: Sending { Cmd , MgmtId: 29066118877352, via: 2(Rack2Host12.lab.vmops.com), Ver: v1,
> Flags: 100001, [{"com.cloud.agent.api.routing.SetStaticRouteCommand":{"staticRoutes":[{"id":4,"uuid":"7c159911-d38a-4dbf-a98d-708b099c7389","targetCidr":"10.2.3.0/24","accountId":2,"domainId":1,"gatewayId":4,"state":"Revoke","vpcId":77,"vlanTag":"2364","gateway":"10.2.3.1","netmask":"255.255.255.0","ipAddress":"10.2.3.2"}],"accessDetails":{"zone.network.type":"Advanced","router.ip":"169.254.2.108","router.name":"r-706-QA"},"wait":0}}] }
> 2013-12-13 15:23:13,563 DEBUG [c.c.a.t.Request] (Job-Executor-102:ctx-12de6228 ctx-e80dd4b0) Seq 2-1828592526: Received: { Ans: , MgmtId: 29066118877352, via: 2, Ver: v1, Flags: 0, { SetStaticRouteAnswer } }
> 2013-12-13 15:23:13,563 DEBUG [c.c.n.e.VpcVirtualRouterElement] (Job-Executor-102:ctx-12de6228 ctx-e80dd4b0) Applied static routes on vpc [VPC [77-RayeesVPC]
> 2013-12-13 15:23:13,563 DEBUG [c.c.n.v.VpcManagerImpl] (Job-Executor-102:ctx-12de6228 ctx-e80dd4b0) Applied static routes for vpc [VPC [77-RayeesVPC]
> 2013-12-13 15:23:13,567 DEBUG [c.c.n.NetworkModelImpl] (Job-Executor-15:ctx-f757750b ctx-4a7c2e5e) Service SecurityGroup is not supported in the network id=572
> 2013-12-13 15:23:13,574 DEBUG [c.c.n.v.VpcManagerImpl] (Job-Executor-102:ctx-12de6228 ctx-e80dd4b0) Removed route StaticRoute[7c159911-d38a-4dbf-a98d-708b099c7389|10.2.3.0/24|4] from the DB
> 2013-12-13 15:23:13,578 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-102:ctx-12de6228) Complete async job-2928, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed to create static route"}
> 2013-12-13 15:23:13,584 DEBUG [c.c.n.NetworkModelImpl] (Job-Executor-15:ctx-f757750b ctx-4a7c2e5e) Service SecurityGroup is not supported in the network id=572
> 2013-12-13 15:23:13,589 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (Job-Executor-15:ctx-f757750b ctx-4a7c2e5e) Applying dhcp entry in network Ntwk[572|Guest|156]
> 2013-12-13 15:23:13,597 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-102:ctx-12de6228) Done executing org.apache.cloudstack.api.command.user.vpc.CreateStaticRouteCmd for job-2928
> 2013-12-13 15:23:13,601 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-102:ctx-12de6228) Remove job-2928 from job monitoring
> 2013-12-13 15:23:13,603 DEBUG [c.c.a.t.Request] (Job-Executor-15:ctx-f757750b ctx-4a7c2e5e) Seq 1-1935872252: Sending { Cmd , MgmtId: 29066118877352, via: 1(Rack2Host11.lab.vmops.com), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.DhcpEntryCommand":{"vmMac":"02:00:42:c8:00:01","vmIpAddress":"10.1.1.122","vmName":"QA-2ac35f13-8ced-4c76-b983-5bdea778e276","defaultRouter":"10.1.1.1","duid":"00:03:00:01:02:00:42:c8:00:01","isDefault":true,"executeInSequence":false,"accessDetails":{"router.guest.ip":"10.1.1.1","zone.network.type":"Advanced","router.name":"r-732-QA","router.ip":"169.254.1.101"},"wait":0}}] }
> Router Log
> -------------
> if you look the message.log in side the VPC router, you can see Private Gateway itself not programmed (attaching all log, please check router log r-706-QA)
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.160401] pci 0000:00:0b.0: BAR 6: assigned [mem 0x08040000-0x0804ffff pref]
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.160406] pci 0000:00:0b.0: BAR 1: assigned [mem 0x08013000-0x08013fff]
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.160443] pci 0000:00:0b.0: BAR 0: assigned [io 0x1060-0x107f]
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.160483] pci 0000:00:00.0: no hotplug settings from platform
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.160486] pci 0000:00:00.0: using default PCI settings
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.160536] pci 0000:00:01.0: no hotplug settings from platform
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.160539] pci 0000:00:01.0: using default PCI settings
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.160584] ata_piix 0000:00:01.1: no hotplug settings from platform
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.160587] ata_piix 0000:00:01.1: using default PCI settings
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.160637] uhci_hcd 0000:00:01.2: no hotplug settings from platform
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.160641] uhci_hcd 0000:00:01.2: using default PCI settings
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.160818] piix4_smbus 0000:00:01.3: no hotplug settings from platform
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.160822] piix4_smbus 0000:00:01.3: using default PCI settings
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.160872] pci 0000:00:02.0: no hotplug settings from platform
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.160875] pci 0000:00:02.0: using default PCI settings
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.163389] virtio-pci 0000:00:03.0: no hotplug settings from platform
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.163393] virtio-pci 0000:00:03.0: using default PCI settings
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.163430] virtio-pci 0000:00:04.0: no hotplug settings from platform
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.163433] virtio-pci 0000:00:04.0: using default PCI settings
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.163488] virtio-pci 0000:00:05.0: no hotplug settings from platform
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.163491] virtio-pci 0000:00:05.0: using default PCI settings
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.163532] virtio-pci 0000:00:06.0: no hotplug settings from platform
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.163535] virtio-pci 0000:00:06.0: using default PCI settings
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.163581] virtio-pci 0000:00:07.0: no hotplug settings from platform
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.163584] virtio-pci 0000:00:07.0: using default PCI settings
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.163640] virtio-pci 0000:00:08.0: no hotplug settings from platform
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.163644] virtio-pci 0000:00:08.0: using default PCI settings
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.163795] virtio-pci 0000:00:09.0: no hotplug settings from platform
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.163798] virtio-pci 0000:00:09.0: using default PCI settings
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.163855] pci 0000:00:0b.0: no hotplug settings from platform
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.163858] pci 0000:00:0b.0: using default PCI settings
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.166368] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)
> Dec 13 23:45:39 r-706-QA cloud: vpc_ipassoc.sh:Waiting for interface ethnull to appear, 0 seconds
> Dec 13 23:45:40 r-706-QA cloud: vpc_ipassoc.sh:Waiting for interface ethnull to appear, 1 seconds
> Dec 13 23:45:41 r-706-QA cloud: vpc_ipassoc.sh:Waiting for interface ethnull to appear, 2 seconds
> Dec 13 23:45:42 r-706-QA cloud: vpc_ipassoc.sh:Waiting for interface ethnull to appear, 3 seconds
> Dec 13 23:45:43 r-706-QA cloud: vpc_ipassoc.sh:Waiting for interface ethnull to appear, 4 seconds
> Dec 13 23:45:44 r-706-QA cloud: vpc_ipassoc.sh:Waiting for interface ethnull to appear, 5 seconds
> Dec 13 23:45:45 r-706-QA cloud: vpc_ipassoc.sh:Waiting for interface ethnull to appear, 6 seconds
> Dec 13 23:45:46 r-706-QA cloud: vpc_ipassoc.sh:Waiting for interface ethnull to appear, 7 seconds
> Dec 13 23:45:47 r-706-QA cloud: vpc_ipassoc.sh:Waiting for interface ethnull to appear, 8 seconds
> Dec 13 23:45:48 r-706-QA cloud: vpc_ipassoc.sh:Waiting for interface ethnull to appear, 9 seconds
> Dec 13 23:45:49 r-706-QA cloud: vpc_ipassoc.sh:Waiting for interface ethnull to appear, 10 seconds
> Dec 13 23:45:50 r-706-QA cloud: vpc_ipassoc.sh:Waiting for interface ethnull to appear, 11 seconds
> Dec 13 23:45:51 r-706-QA cloud: vpc_ipassoc.sh:Waiting for interface ethnull to appear, 12 seconds
> Dec 13 23:45:52 r-706-QA cloud: vpc_ipassoc.sh:Waiting for interface ethnull to appear, 13 seconds
> Dec 13 23:45:53 r-706-QA cloud: vpc_ipassoc.sh:Waiting for interface ethnull to appear, 14 seconds
> Dec 13 23:45:54 r-706-QA cloud: vpc_ipassoc.sh:Waiting for interface ethnull to appear, 15 seconds
> Dec 13 23:45:55 r-706-QA cloud: vpc_ipassoc.sh:Waiting for interface ethnull to appear, 16 seconds
> Dec 13 23:45:56 r-706-QA cloud: vpc_ipassoc.sh:interface ethnull never appeared
> Dec 13 23:45:56 r-706-QA cloud: vpc_ipassoc.sh:Adding ip 10.2.3.2 on interface ethnull
> Dec 13 23:45:56 r-706-QA cloud: vpc_ipassoc.sh:Add routing 10.2.3.2 on interface ethnull
> Dec 13 23:45:57 r-706-QA cloud: vpc_privategw_acl.sh: enter apply acl rules on private gateway interface : eth4, inbound::all:0:0:0.0.0.0/0
> Dec 13 23:45:57 r-706-QA cloud: vpc_privategw_acl.sh: exit apply acl rules for private gw interface : eth4
> Dec 13 23:45:57 r-706-QA cloud: vpc_privategw_acl.sh: successful in applying acl rules on private gateway interface : eth4
> Dec 13 23:45:57 r-706-QA cloud: vpc_privategw_acl.sh: enter apply acl rules on private gateway interface : eth4, inbound::all:0:0:0.0.0.0/0
> Dec 13 23:45:57 r-706-QA cloud: vpc_privategw_acl.sh: exit apply acl rules for private gw interface : eth4
> Dec 13 23:45:57 r-706-QA cloud: vpc_privategw_acl.sh: successful in applying acl rules on private gateway interface : eth4
> Dec 13 23:45:57 r-706-QA cloud: vpc_privategw_acl.sh: deleting backup on private gateway interface : eth4
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)