You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@cloudstack.apache.org by "Sanjeev N (JIRA)" <ji...@apache.org> on 2014/05/28 09:23:01 UTC
[jira] [Created] (CLOUDSTACK-6796) [OVS]Failure in network update
does not change network offering to original offering
Sanjeev N created CLOUDSTACK-6796:
-------------------------------------
Summary: [OVS]Failure in network update does not change network offering to original offering
Key: CLOUDSTACK-6796
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-6796
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Management Server, Network Controller
Affects Versions: 4.4.0
Environment: Latest build from 4.4 with commit d130530bd3e1cd6d8249d5045e00e4e4e2201521
Reporter: Sanjeev N
Assignee: Murali Reddy
Priority: Critical
Fix For: 4.4.0
Attachments: management-server.rar
[OVS]Failure in network update does not change network offering to original offering hence starting vms would fail in the network
Steps to Reproduce:
===============
1.Bring up CS in advanced zone with xen cluster
2.Create physical network with GRE isolation
3.Create network with default offering "DefaultIsolatedNetworkOfferingWithSourceNatService"
4.Deploy few vms in the above netwrok
5.Create another network offering with virtual networking service and OVS as the connectivity service provider
6.Stop all the vms in the network
7.Update network with new offering created at step5
Results:
======
Network update will fail from vlan isolation to connectivity service due to bug CS-6795. However the network offering id for the network is changing to new network offering. It is not setting back to default isolated network offering.
mysql> select * from ntwk_offering_service_map where network_offering_id=15;
+----+---------------------+----------------+---------------+---------------------+
| id | network_offering_id | service | provider | created |
+----+---------------------+----------------+---------------+---------------------+
| 60 | 15 | Connectivity | Ovs | 2014-05-26 12:51:34 |
| 55 | 15 | Dhcp | VirtualRouter | 2014-05-26 12:51:34 |
| 54 | 15 | Dns | VirtualRouter | 2014-05-26 12:51:34 |
| 61 | 15 | Firewall | VirtualRouter | 2014-05-26 12:51:34 |
| 58 | 15 | Lb | VirtualRouter | 2014-05-26 12:51:34 |
| 57 | 15 | PortForwarding | VirtualRouter | 2014-05-26 12:51:34 |
| 56 | 15 | SourceNat | VirtualRouter | 2014-05-26 12:51:34 |
| 59 | 15 | StaticNat | VirtualRouter | 2014-05-26 12:51:34 |
| 53 | 15 | UserData | VirtualRouter | 2014-05-26 12:51:34 |
+----+---------------------+----------------+---------------+---------------------+
9 rows in set (0.00 sec)
Following is the network created with default isolated network offering but after network update failure the offering still shows the new offering:
mysql> select * from networks where id=211\G;
*************************** 1. row ***************************
id: 211
name: vlan1
uuid: f803e17f-b59b-4229-9e70-5bb4fcfc2570
display_text: vlan1
traffic_type: Guest
broadcast_domain_type: Vlan
broadcast_uri: vlan://986
gateway: 10.1.1.1
cidr: 10.1.1.0/24
mode: Dhcp
network_offering_id: 15
physical_network_id: 200
data_center_id: 1
guru_name: ExternalGuestNetworkGuru
state: Shutdown
related: 211
domain_id: 1
account_id: 2
dns1: NULL
dns2: NULL
guru_data: NULL
set_fields: 0
acl_type: Account
network_domain: cs2cloud.internal
reservation_id: c2b3cb64-adfd-4722-9aed-8d2d7710e32f
guest_type: Isolated
restart_required: 0
created: 2014-05-28 11:09:16
removed: NULL
specify_ip_ranges: 0
vpc_id: NULL
ip6_gateway: NULL
ip6_cidr: NULL
network_cidr: NULL
display_network: 1
network_acl_id: NULL
streched_l2: 0
1 row in set (0.00 sec)
ERROR:
No query specified
Impact of this:
===========
Since the network offering is with connectivity service , CS is failed to implement the network and vm start is failing.
2014-05-28 07:52:28,188 DEBUG [c.c.n.e.OvsElement] (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Checking if OvsElement can handle service SourceNat on network vlan1
2014-05-28 07:52:28,189 DEBUG [c.c.n.e.OvsElement] (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Virtual router element doesn't need to associate ip addresses on the backend; virtual router doesn't exist in the network 211
2014-05-28 07:52:28,193 DEBUG [c.c.n.e.VirtualRouterElement] (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Virtual router elemnt doesn't need to associate ip addresses on the backend; virtual router doesn't exist in the network 211
2014-05-28 07:52:28,195 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Sending network shutdown to Ovs
2014-05-28 07:52:28,195 DEBUG [c.c.n.e.OvsElement] (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Checking if OvsElement can handle service Connectivity on network vlan1
2014-05-28 07:52:28,195 WARN [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Unable to complete shutdown of the network elements due to element: Ovs
2014-05-28 07:52:28,197 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Sending network shutdown to VirtualRouter
2014-05-28 07:52:28,201 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Lock is released for network Ntwk[f803e17f-b59b-4229-9e70-5bb4fcfc2570|Guest|15] as a part of network shutdown
2014-05-28 07:52:28,202 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Lock is released for network id 211 as a part of network implement
2014-05-28 07:52:28,202 ERROR [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Failed to start instance VM[User|i-2-25-VM]
java.lang.NullPointerException
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepare(NetworkOrchestrator.java:1273)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:983)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5180)
at sun.reflect.GeneratedMethodAccessor436.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5325)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-05-28 07:52:28,206 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Cleaning up resources for the vm VM[User|i-2-25-VM] in Starting state
2014-05-28 07:52:28,209 DEBUG [c.c.a.t.Request] (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Seq 1-8670555182595055942: Sending { Cmd , MgmtId: 7332683579487, via: 1(Rack1Pod1Host13), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"checkBeforeCleanup":false,"vmName":"i-2-25-VM","wait":0}}] }
2014-05-28 07:52:28,209 DEBUG [c.c.a.t.Request] (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Seq 1-8670555182595055942: Executing: { Cmd , MgmtId: 7332683579487, via: 1(Rack1Pod1Host13), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"checkBeforeCleanup":false,"vmName":"i-2-25-VM","wait":0}}] }
2014-05-28 07:52:28,209 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-304:ctx-21497aaa) Seq 1-8670555182595055942: Executing request
2014-05-28 07:52:28,215 INFO [c.c.h.x.r.CitrixResourceBase] (DirectAgent-304:ctx-21497aaa) VM does not exist on XenServer693592c3-3a6d-4025-84f2-64c52b733568
2014-05-28 07:52:28,215 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-304:ctx-21497aaa) Seq 1-8670555182595055942: Response Received:
2014-05-28 07:52:28,215 DEBUG [c.c.a.t.Request] (DirectAgent-304:ctx-21497aaa) Seq 1-8670555182595055942: Processing: { Ans: , MgmtId: 7332683579487, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.StopAnswer":{"result":true,"details":"VM does not exist","wait":0}}] }
2014-05-28 07:52:28,215 DEBUG [c.c.a.t.Request] (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Seq 1-8670555182595055942: Received: { Ans: , MgmtId: 7332683579487, via: 1, Ver: v1, Flags: 10, { StopAnswer } }
2014-05-28 07:52:28,221 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Successfully released network resources for the vm VM[User|i-2-25-VM]
2014-05-28 07:52:28,221 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Successfully cleanued up resources for the vm VM[User|i-2-25-VM] in Starting state
2014-05-28 07:52:28,229 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) VM state transitted from :Starting to Stopped with event: OperationFailedvm's original host id: 1 new host id: null host id before state transition: 1
2014-05-28 07:52:28,248 INFO [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Rethrow exception com.cloud.exception.AgentUnavailableException: Resource [Host:1] is unreachable: Host 1: Unable to start instance due to null
2014-05-28 07:52:28,248 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123) Done with run of VM work job: com.cloud.vm.VmWorkStart for VM 25, job origin: 122
2014-05-28 07:52:28,248 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123) Unable to complete AsyncJobVO {id:123, userId: 2, accountId: 2, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo: rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAACAAAAAAAAAAIAAAAAAAAAGXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAXBzcgAOamF2YS5sYW5nLkxvbmc7i-SQzI8j3wIAAUoABXZhbHVleHIAEGphdmEubGFuZy5OdW1iZXKGrJUdC5TgiwIAAHhwAAAAAAAAAAFxAH4ACnBwcHEAfgAKcHNyABFqYXZhLnV0aWwuSGFzaE1hcAUH2sHDFmDRAwACRgAKbG9hZEZhY3RvckkACXRocmVzaG9sZHhwP0AAAAAAAAx3CAAAABAAAAABdAAKVm1QYXNzd29yZHQAHHJPMEFCWFFBRG5OaGRtVmtYM0JoYzNOM2IzSmt4cA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 7332683579487, completeMsid: null, lastUpdated: null, lastPolled: null, created: Wed May 28 07:52:26 EDT 2014}, job origin:122
com.cloud.exception.AgentUnavailableException: Resource [Host:1] is unreachable: Host 1: Unable to start instance due to null
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1103)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5180)
at sun.reflect.GeneratedMethodAccessor436.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5325)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.NullPointerException
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepare(NetworkOrchestrator.java:1273)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:983)
... 19 more
014-05-28 07:52:30,312 DEBUG [c.c.n.e.OvsElement] (Work-Job-Executor-48:ctx-56a17636 job-122/job-124 ctx-bd415b93) Checking if OvsElement can handle service SourceNat on network vlan1
2014-05-28 07:52:30,314 DEBUG [c.c.n.e.OvsElement] (Work-Job-Executor-48:ctx-56a17636 job-122/job-124 ctx-bd415b93) Virtual router element doesn't need to associate ip addresses on the backend; virtual router doesn't exist in the network 211
2014-05-28 07:52:30,317 DEBUG [c.c.n.e.VirtualRouterElement] (Work-Job-Executor-48:ctx-56a17636 job-122/job-124 ctx-bd415b93) Virtual router elemnt doesn't need to associate ip addresses on the backend; virtual router doesn't exist in the network 211
2014-05-28 07:52:30,319 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-48:ctx-56a17636 job-122/job-124 ctx-bd415b93) Sending network shutdown to Ovs
2014-05-28 07:52:30,320 DEBUG [c.c.n.e.OvsElement] (Work-Job-Executor-48:ctx-56a17636 job-122/job-124 ctx-bd415b93) Checking if OvsElement can handle service Connectivity on network vlan1
2014-05-28 07:52:30,320 WARN [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-48:ctx-56a17636 job-122/job-124 ctx-bd415b93) Unable to complete shutdown of the network elements due to element: Ovs
2014-05-28 07:52:30,322 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-48:ctx-56a17636 job-122/job-124 ctx-bd415b93) Sending network shutdown to VirtualRouter
2014-05-28 07:52:30,326 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-48:ctx-56a17636 job-122/job-124 ctx-bd415b93) Lock is released for network Ntwk[f803e17f-b59b-4229-9e70-5bb4fcfc2570|Guest|15] as a part of network shutdown
2014-05-28 07:52:30,326 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-48:ctx-56a17636 job-122/job-124 ctx-bd415b93) Lock is released for network id 211 as a part of network implement
2014-05-28 07:52:30,326 ERROR [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-48:ctx-56a17636 job-122/job-124 ctx-bd415b93) Failed to start instance VM[User|i-2-25-VM]
java.lang.NullPointerException
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepare(NetworkOrchestrator.java:1273)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:983)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5180)
at sun.reflect.GeneratedMethodAccessor436.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5325)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-05-28 07:52:30,331 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-48:ctx-56a17636 job-122/job-124 ctx-bd415b93) Cleaning up resources for the vm VM[User|i-2-25-VM] in Starting state
--
This message was sent by Atlassian JIRA
(v6.2#6252)