You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@cloudstack.apache.org by "Sanjeev N (JIRA)" <ji...@apache.org> on 2014/05/28 09:23:01 UTC
[jira] [Updated] (CLOUDSTACK-6796) [OVS]Failure in network update
does not change network offering to original offering
[ https://issues.apache.org/jira/browse/CLOUDSTACK-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sanjeev N updated CLOUDSTACK-6796:
----------------------------------
Attachment: management-server.rar
Attached MS log file
> [OVS]Failure in network update does not change network offering to original offering
> ------------------------------------------------------------------------------------
>
> Key: CLOUDSTACK-6796
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-6796
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the default.)
> Components: Management Server, Network Controller
> Affects Versions: 4.4.0
> Environment: Latest build from 4.4 with commit d130530bd3e1cd6d8249d5045e00e4e4e2201521
> Reporter: Sanjeev N
> Assignee: Murali Reddy
> Priority: Critical
> Labels: ovs
> Fix For: 4.4.0
>
> Attachments: management-server.rar
>
>
> [OVS]Failure in network update does not change network offering to original offering hence starting vms would fail in the network
> Steps to Reproduce:
> ===============
> 1.Bring up CS in advanced zone with xen cluster
> 2.Create physical network with GRE isolation
> 3.Create network with default offering "DefaultIsolatedNetworkOfferingWithSourceNatService"
> 4.Deploy few vms in the above netwrok
> 5.Create another network offering with virtual networking service and OVS as the connectivity service provider
> 6.Stop all the vms in the network
> 7.Update network with new offering created at step5
> Results:
> ======
> Network update will fail from vlan isolation to connectivity service due to bug CS-6795. However the network offering id for the network is changing to new network offering. It is not setting back to default isolated network offering.
> mysql> select * from ntwk_offering_service_map where network_offering_id=15;
> +----+---------------------+----------------+---------------+---------------------+
> | id | network_offering_id | service | provider | created |
> +----+---------------------+----------------+---------------+---------------------+
> | 60 | 15 | Connectivity | Ovs | 2014-05-26 12:51:34 |
> | 55 | 15 | Dhcp | VirtualRouter | 2014-05-26 12:51:34 |
> | 54 | 15 | Dns | VirtualRouter | 2014-05-26 12:51:34 |
> | 61 | 15 | Firewall | VirtualRouter | 2014-05-26 12:51:34 |
> | 58 | 15 | Lb | VirtualRouter | 2014-05-26 12:51:34 |
> | 57 | 15 | PortForwarding | VirtualRouter | 2014-05-26 12:51:34 |
> | 56 | 15 | SourceNat | VirtualRouter | 2014-05-26 12:51:34 |
> | 59 | 15 | StaticNat | VirtualRouter | 2014-05-26 12:51:34 |
> | 53 | 15 | UserData | VirtualRouter | 2014-05-26 12:51:34 |
> +----+---------------------+----------------+---------------+---------------------+
> 9 rows in set (0.00 sec)
> Following is the network created with default isolated network offering but after network update failure the offering still shows the new offering:
> mysql> select * from networks where id=211\G;
> *************************** 1. row ***************************
> id: 211
> name: vlan1
> uuid: f803e17f-b59b-4229-9e70-5bb4fcfc2570
> display_text: vlan1
> traffic_type: Guest
> broadcast_domain_type: Vlan
> broadcast_uri: vlan://986
> gateway: 10.1.1.1
> cidr: 10.1.1.0/24
> mode: Dhcp
> network_offering_id: 15
> physical_network_id: 200
> data_center_id: 1
> guru_name: ExternalGuestNetworkGuru
> state: Shutdown
> related: 211
> domain_id: 1
> account_id: 2
> dns1: NULL
> dns2: NULL
> guru_data: NULL
> set_fields: 0
> acl_type: Account
> network_domain: cs2cloud.internal
> reservation_id: c2b3cb64-adfd-4722-9aed-8d2d7710e32f
> guest_type: Isolated
> restart_required: 0
> created: 2014-05-28 11:09:16
> removed: NULL
> specify_ip_ranges: 0
> vpc_id: NULL
> ip6_gateway: NULL
> ip6_cidr: NULL
> network_cidr: NULL
> display_network: 1
> network_acl_id: NULL
> streched_l2: 0
> 1 row in set (0.00 sec)
> ERROR:
> No query specified
> Impact of this:
> ===========
> Since the network offering is with connectivity service , CS is failed to implement the network and vm start is failing.
> 2014-05-28 07:52:28,188 DEBUG [c.c.n.e.OvsElement] (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Checking if OvsElement can handle service SourceNat on network vlan1
> 2014-05-28 07:52:28,189 DEBUG [c.c.n.e.OvsElement] (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Virtual router element doesn't need to associate ip addresses on the backend; virtual router doesn't exist in the network 211
> 2014-05-28 07:52:28,193 DEBUG [c.c.n.e.VirtualRouterElement] (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Virtual router elemnt doesn't need to associate ip addresses on the backend; virtual router doesn't exist in the network 211
> 2014-05-28 07:52:28,195 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Sending network shutdown to Ovs
> 2014-05-28 07:52:28,195 DEBUG [c.c.n.e.OvsElement] (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Checking if OvsElement can handle service Connectivity on network vlan1
> 2014-05-28 07:52:28,195 WARN [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Unable to complete shutdown of the network elements due to element: Ovs
> 2014-05-28 07:52:28,197 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Sending network shutdown to VirtualRouter
> 2014-05-28 07:52:28,201 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Lock is released for network Ntwk[f803e17f-b59b-4229-9e70-5bb4fcfc2570|Guest|15] as a part of network shutdown
> 2014-05-28 07:52:28,202 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Lock is released for network id 211 as a part of network implement
> 2014-05-28 07:52:28,202 ERROR [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Failed to start instance VM[User|i-2-25-VM]
> java.lang.NullPointerException
> at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepare(NetworkOrchestrator.java:1273)
> at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:983)
> at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5180)
> at sun.reflect.GeneratedMethodAccessor436.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5325)
> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> 2014-05-28 07:52:28,206 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Cleaning up resources for the vm VM[User|i-2-25-VM] in Starting state
> 2014-05-28 07:52:28,209 DEBUG [c.c.a.t.Request] (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Seq 1-8670555182595055942: Sending { Cmd , MgmtId: 7332683579487, via: 1(Rack1Pod1Host13), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"checkBeforeCleanup":false,"vmName":"i-2-25-VM","wait":0}}] }
> 2014-05-28 07:52:28,209 DEBUG [c.c.a.t.Request] (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Seq 1-8670555182595055942: Executing: { Cmd , MgmtId: 7332683579487, via: 1(Rack1Pod1Host13), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"checkBeforeCleanup":false,"vmName":"i-2-25-VM","wait":0}}] }
> 2014-05-28 07:52:28,209 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-304:ctx-21497aaa) Seq 1-8670555182595055942: Executing request
> 2014-05-28 07:52:28,215 INFO [c.c.h.x.r.CitrixResourceBase] (DirectAgent-304:ctx-21497aaa) VM does not exist on XenServer693592c3-3a6d-4025-84f2-64c52b733568
> 2014-05-28 07:52:28,215 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-304:ctx-21497aaa) Seq 1-8670555182595055942: Response Received:
> 2014-05-28 07:52:28,215 DEBUG [c.c.a.t.Request] (DirectAgent-304:ctx-21497aaa) Seq 1-8670555182595055942: Processing: { Ans: , MgmtId: 7332683579487, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.StopAnswer":{"result":true,"details":"VM does not exist","wait":0}}] }
> 2014-05-28 07:52:28,215 DEBUG [c.c.a.t.Request] (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Seq 1-8670555182595055942: Received: { Ans: , MgmtId: 7332683579487, via: 1, Ver: v1, Flags: 10, { StopAnswer } }
> 2014-05-28 07:52:28,221 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Successfully released network resources for the vm VM[User|i-2-25-VM]
> 2014-05-28 07:52:28,221 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Successfully cleanued up resources for the vm VM[User|i-2-25-VM] in Starting state
> 2014-05-28 07:52:28,229 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) VM state transitted from :Starting to Stopped with event: OperationFailedvm's original host id: 1 new host id: null host id before state transition: 1
> 2014-05-28 07:52:28,248 INFO [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Rethrow exception com.cloud.exception.AgentUnavailableException: Resource [Host:1] is unreachable: Host 1: Unable to start instance due to null
> 2014-05-28 07:52:28,248 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123) Done with run of VM work job: com.cloud.vm.VmWorkStart for VM 25, job origin: 122
> 2014-05-28 07:52:28,248 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123) Unable to complete AsyncJobVO {id:123, userId: 2, accountId: 2, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo: rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAACAAAAAAAAAAIAAAAAAAAAGXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAXBzcgAOamF2YS5sYW5nLkxvbmc7i-SQzI8j3wIAAUoABXZhbHVleHIAEGphdmEubGFuZy5OdW1iZXKGrJUdC5TgiwIAAHhwAAAAAAAAAAFxAH4ACnBwcHEAfgAKcHNyABFqYXZhLnV0aWwuSGFzaE1hcAUH2sHDFmDRAwACRgAKbG9hZEZhY3RvckkACXRocmVzaG9sZHhwP0AAAAAAAAx3CAAAABAAAAABdAAKVm1QYXNzd29yZHQAHHJPMEFCWFFBRG5OaGRtVmtYM0JoYzNOM2IzSmt4cA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 7332683579487, completeMsid: null, lastUpdated: null, lastPolled: null, created: Wed May 28 07:52:26 EDT 2014}, job origin:122
> com.cloud.exception.AgentUnavailableException: Resource [Host:1] is unreachable: Host 1: Unable to start instance due to null
> at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1103)
> at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5180)
> at sun.reflect.GeneratedMethodAccessor436.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5325)
> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: java.lang.NullPointerException
> at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepare(NetworkOrchestrator.java:1273)
> at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:983)
> ... 19 more
> 014-05-28 07:52:30,312 DEBUG [c.c.n.e.OvsElement] (Work-Job-Executor-48:ctx-56a17636 job-122/job-124 ctx-bd415b93) Checking if OvsElement can handle service SourceNat on network vlan1
> 2014-05-28 07:52:30,314 DEBUG [c.c.n.e.OvsElement] (Work-Job-Executor-48:ctx-56a17636 job-122/job-124 ctx-bd415b93) Virtual router element doesn't need to associate ip addresses on the backend; virtual router doesn't exist in the network 211
> 2014-05-28 07:52:30,317 DEBUG [c.c.n.e.VirtualRouterElement] (Work-Job-Executor-48:ctx-56a17636 job-122/job-124 ctx-bd415b93) Virtual router elemnt doesn't need to associate ip addresses on the backend; virtual router doesn't exist in the network 211
> 2014-05-28 07:52:30,319 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-48:ctx-56a17636 job-122/job-124 ctx-bd415b93) Sending network shutdown to Ovs
> 2014-05-28 07:52:30,320 DEBUG [c.c.n.e.OvsElement] (Work-Job-Executor-48:ctx-56a17636 job-122/job-124 ctx-bd415b93) Checking if OvsElement can handle service Connectivity on network vlan1
> 2014-05-28 07:52:30,320 WARN [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-48:ctx-56a17636 job-122/job-124 ctx-bd415b93) Unable to complete shutdown of the network elements due to element: Ovs
> 2014-05-28 07:52:30,322 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-48:ctx-56a17636 job-122/job-124 ctx-bd415b93) Sending network shutdown to VirtualRouter
> 2014-05-28 07:52:30,326 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-48:ctx-56a17636 job-122/job-124 ctx-bd415b93) Lock is released for network Ntwk[f803e17f-b59b-4229-9e70-5bb4fcfc2570|Guest|15] as a part of network shutdown
> 2014-05-28 07:52:30,326 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-48:ctx-56a17636 job-122/job-124 ctx-bd415b93) Lock is released for network id 211 as a part of network implement
> 2014-05-28 07:52:30,326 ERROR [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-48:ctx-56a17636 job-122/job-124 ctx-bd415b93) Failed to start instance VM[User|i-2-25-VM]
> java.lang.NullPointerException
> at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepare(NetworkOrchestrator.java:1273)
> at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:983)
> at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5180)
> at sun.reflect.GeneratedMethodAccessor436.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5325)
> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> 2014-05-28 07:52:30,331 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-48:ctx-56a17636 job-122/job-124 ctx-bd415b93) Cleaning up resources for the vm VM[User|i-2-25-VM] in Starting state
--
This message was sent by Atlassian JIRA
(v6.2#6252)