You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Mason Donahue <md...@backstopsolutions.com> on 2016/09/19 16:56:07 UTC
Reestablishing a VR when the VR was deleted
Hi there,
We’re in a bit of a pickle with our Cloudstack 4.4.1 install. (Yes, I know it’s outdated; we were hoping to upgrade soon and then this happened.) We had an issue where the VR for one of our Networks went down, and I mistakenly missed the ‘cleanup’ checkbox in the ‘Restart network’ menu of the UI. We use the VR solely for DNS. We now have no VRs in our setup, and the network can no longer be restarted due to a check that routers are running. Unfortunately, to delete and re-add the network, the log output states that we’d have to expunge all of our machines, which I am hoping to avoid doing. What are my other options? Can I un-mark the router as deleted in the DB, and will that allow it to limp along to the point where it can at least rebuild the network?
Thanks,
--Mason
(relevant logs below)
2016-09-19 11:46:50,890 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141) Executing AsyncJobVO {id:10141, userId: 2, accountId: 2, instanceType: None, instanceId: null, cmd: org.apache.cloudstack.api.command.user.network.RestartNetworkCmd, cmdInfo: {"response":"json","id":"448deced-7
223-4549-98bd-5acafc811f05","sessionkey":"985SpsvETZfPpiNyTjNsOgXWVQI\u003d","cleanup":"true","ctxDetails":"{\"com.cloud.network.Network\":\"448deced-7223-4549-98bd-5acafc811f05\"}","cmdEventType":"NETWORK.RESTART","ctxUserId":"2","httpmethod":"GET","_":"1474303610834","uuid":"448deced-7223-4549-98bd-5acafc811f05","c
txAccountId":"2","ctxStartEventId":"21061"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 144343483243, completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
2016-09-19 11:46:50,891 DEBUG [c.c.a.ApiServlet] (catalina-exec-22:ctx-94ea893d ctx-69ffa947) ===END=== 192.168.42.156 -- GET command=restartNetwork&id=448deced-7223-4549-98bd-5acafc811f05&cleanup=true&response=json&sessionkey=985SpsvETZfPpiNyTjNsOgXWVQI%3D&_=1474303610834
2016-09-19 11:46:50,925 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Restarting network 204...
2016-09-19 11:46:50,925 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Shutting down the network id=204 as a part of network restart
2016-09-19 11:46:50,929 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Releasing 0 port forwarding rules for network id=204 as a part of shutdownNetworkRules
2016-09-19 11:46:50,930 DEBUG [c.c.n.f.FirewallManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) There are no rules to forward to the network elements
2016-09-19 11:46:50,932 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Releasing 0 static nat rules for network id=204 as a part of shutdownNetworkRules
2016-09-19 11:46:50,932 DEBUG [c.c.n.f.FirewallManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) There are no rules to forward to the network elements
2016-09-19 11:46:50,934 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Revoking 0 Public load balancing rules for network id=204
2016-09-19 11:46:50,934 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) There are no Load Balancing Rules to forward to the network elements
2016-09-19 11:46:50,936 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Revoking 0 Internal load balancing rules for network id=204
2016-09-19 11:46:50,936 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) There are no Load Balancing Rules to forward to the network elements
2016-09-19 11:46:50,937 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Releasing 0 firewall ingress rules for network id=204 as a part of shutdownNetworkRules
2016-09-19 11:46:50,937 DEBUG [c.c.n.f.FirewallManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) There are no rules to forward to the network elements
2016-09-19 11:46:50,938 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Releasing 0 firewall egress rules for network id=204 as a part of shutdownNetworkRules
2016-09-19 11:46:50,939 DEBUG [c.c.n.f.FirewallManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) There are no rules to forward to the network elements
2016-09-19 11:46:50,941 DEBUG [c.c.n.r.RulesManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Found 0 static nat rules to apply for network id 204
2016-09-19 11:46:51,029 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Sending network shutdown to SecurityGroupProvider
2016-09-19 11:46:51,032 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Sending network shutdown to VirtualRouter
2016-09-19 11:46:51,034 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Implementing the network Ntwk[448deced-7223-4549-98bd-5acafc811f05|Guest|6] elements and resources as a part of network restart
2016-09-19 11:46:51,040 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Asking SecurityGroupProvider to implemenet Ntwk[448deced-7223-4549-98bd-5acafc811f05|Guest|6]
2016-09-19 11:46:51,043 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Asking VirtualRouter to implemenet Ntwk[448deced-7223-4549-98bd-5acafc811f05|Guest|6]
2016-09-19 11:46:51,049 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Lock is acquired for network id 204 as a part of router startup in Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(null)-Cluster(null)-H
ost(null)-Storage()]
2016-09-19 11:46:51,051 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Lock is released for network id 204 as a part of router startup in Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(null)-Cluster(null)-H
ost(null)-Storage()]
2016-09-19 11:46:51,053 WARN [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Failed to implement network Ntwk[448deced-7223-4549-98bd-5acafc811f05|Guest|6] elements and resources as a part of network restart due to
com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is unreachable: Can't find all necessary running routers!
at com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:199)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElementsAndResources(NetworkOrchestrator.java:1080)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:2430)
at com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1892)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy156.restartNetwork(Unknown Source)
at org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:95)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-09-19 11:46:51,054 WARN [c.c.n.NetworkServiceImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Network id=204 failed to restart.
2016-09-19 11:46:51,072 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141) Complete async job-10141, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed to restart network"}
2016-09-19 11:46:51,087 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141) Done executing org.apache.cloudstack.api.command.user.network.RestartNetworkCmd for job-10141
2016-09-19 11:46:51,107 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-5:ctx-8a42cc36 job-10141) Remove job-10141 from job monitoring
RE: Reestablishing a VR when the VR was deleted
Posted by Kirk Kosinski <ki...@shapeblue.com>.
Hi, it's good to hear it's resolved. The document you linked is correct in the context of an upgrade. The upgrade process looks at the name and description of the templates to find the new system VM template, and will mark it as such in the database. If, after an upgrade, the system VM template is somehow removed I'm not sure how to fix it besides a database hack.
Best regards,
Kirk
kirk.kosinski@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HS
@shapeblue
-----Original Message-----
From: Mason Donahue [mailto:mdonahue@backstopsolutions.com]
Sent: Monday, September 19, 2016 3:05 PM
To: users@cloudstack.apache.org
Subject: Re: Reestablishing a VR when the VR was deleted
Well, that was the bit of information that finally got us back up and running.
The original routerVM failed to start due to the template not being there, so I had redownloaded the template according to the directions at https://docs.cloudstack.apache.org/projects/cloudstack-release-notes/en/4.4.1/upgrade/upgrade-4.3.html and that seems to have been the source of the problem.
Within the errors that accompanied the failure to start of the random VM I picked there was an attempt to start a VR:
2016-09-19 15:29:48,962 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (Work-Job-Executor-4:ctx-21dac544 job-10167/job-10169 ctx-d441bcf6) Lock is acquired for network id 204 as a part of router startup in Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(1)-Cluster(7)-Host(25)-Storage()]
2016-09-19 15:29:48,972 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (Work-Job-Executor-4:ctx-21dac544 job-10167/job-10169 ctx-d441bcf6) Adding nic for Virtual Router in Guest network Ntwk[448deced-7223-4549-98bd-5acafc811f05|Guest|6]
2016-09-19 15:29:48,977 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (Work-Job-Executor-4:ctx-21dac544 job-10167/job-10169 ctx-d441bcf6) Adding nic for Virtual Router in Control network
2016-09-19 15:29:48,980 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-4:ctx-21dac544 job-10167/job-10169 ctx-d441bcf6) Found existing network configuration for offering [Network Offering [3-Control-System-Control-Network]: Ntwk[28d967d8-3b75-4362-9234-b0d029b0d21b|Control|3]
2016-09-19 15:29:48,980 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-4:ctx-21dac544 job-10167/job-10169 ctx-d441bcf6) Releasing lock for Acct[7500fc58-dcf6-11e2-b492-00219b9585d4-system]
2016-09-19 15:29:48,984 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (Work-Job-Executor-4:ctx-21dac544 job-10167/job-10169 ctx-d441bcf6) Allocating the VR i=1703 in datacenter com.cloud.dc.DataCenterVO$$EnhancerByCGLIB$$9732e921@1with the hypervisor type KVM
2016-09-19 15:29:48,988 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (Work-Job-Executor-4:ctx-21dac544 job-10167/job-10169 ctx-d441bcf6) KVM won't support system vm, skip it
2016-09-19 15:29:48,989 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (Work-Job-Executor-4:ctx-21dac544 job-10167/job-10169 ctx-d441bcf6) Lock is released for network id 204 as a part of router startup in Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(1)-Cluster(7)-Host(25)-Storage()]
2016-09-19 15:29:48,989 INFO [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-4:ctx-21dac544 job-10167/job-10169 ctx-d441bcf6) Unable to contact resource.
com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is unreachable: Can't find at least one running router!
The line ‘KVM won’t support system vm, skip it’ was the key, as googling it led to other cloudstack-users mailing list posts with other people who had failed to add systemvm templates back in. Shutting down the management server, updating the TYPE column in the database for the systemvm to ‘SYSTEM’ instead of ‘USER’, and starting a VM again fixed it, and everything came back up happily.
It seems like the documentation on importing systemVMs (for instance, https://docs.cloudstack.apache.org/projects/cloudstack-release-notes/en/4.9.0/upgrade/upgrade-4.4.html) is incorrect, as it says that the template should be imported as ‘Routing: no’. Or is there some other method in which a template is supposed to be set to SYSTEM instead of USER?
Thanks!
--Mason
On 9/19/16, 3:22 PM, "Kirk Kosinski" <ki...@shapeblue.com> wrote:
Hi, if you start a VM in a network that has no VR, the VR will be recreated. So you can stop/start an existing VM in the network, or deploy a new VM to the network.
Best regards,
kirk.kosinski@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HS
@shapeblue
-----Original Message-----
From: Mason Donahue [mailto:mdonahue@backstopsolutions.com]
Sent: Monday, September 19, 2016 9:56 AM
To: users@cloudstack.apache.org
Subject: Reestablishing a VR when the VR was deleted
Hi there,
We’re in a bit of a pickle with our Cloudstack 4.4.1 install. (Yes, I know it’s outdated; we were hoping to upgrade soon and then this happened.) We had an issue where the VR for one of our Networks went down, and I mistakenly missed the ‘cleanup’ checkbox in the ‘Restart network’ menu of the UI. We use the VR solely for DNS. We now have no VRs in our setup, and the network can no longer be restarted due to a check that routers are running. Unfortunately, to delete and re-add the network, the log output states that we’d have to expunge all of our machines, which I am hoping to avoid doing. What are my other options? Can I un-mark the router as deleted in the DB, and will that allow it to limp along to the point where it can at least rebuild the network?
Thanks,
--Mason
(relevant logs below)
2016-09-19 11:46:50,890 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141) Executing AsyncJobVO {id:10141, userId: 2, accountId: 2, instanceType: None, instanceId: null, cmd: org.apache.cloudstack.api.command.user.network.RestartNetworkCmd, cmdInfo: {"response":"json","id":"448deced-7
223-4549-98bd-5acafc811f05","sessionkey":"985SpsvETZfPpiNyTjNsOgXWVQI\u003d","cleanup":"true","ctxDetails":"{\"com.cloud.network.Network\":\"448deced-7223-4549-98bd-5acafc811f05\"}","cmdEventType":"NETWORK.RESTART","ctxUserId":"2","httpmethod":"GET","_":"1474303610834","uuid":"448deced-7223-4549-98bd-5acafc811f05","c
txAccountId":"2","ctxStartEventId":"21061"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 144343483243, completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
2016-09-19 11:46:50,891 DEBUG [c.c.a.ApiServlet] (catalina-exec-22:ctx-94ea893d ctx-69ffa947) ===END=== 192.168.42.156 -- GET command=restartNetwork&id=448deced-7223-4549-98bd-5acafc811f05&cleanup=true&response=json&sessionkey=985SpsvETZfPpiNyTjNsOgXWVQI%3D&_=1474303610834
2016-09-19 11:46:50,925 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Restarting network 204...
2016-09-19 11:46:50,925 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Shutting down the network id=204 as a part of network restart
2016-09-19 11:46:50,929 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Releasing 0 port forwarding rules for network id=204 as a part of shutdownNetworkRules
2016-09-19 11:46:50,930 DEBUG [c.c.n.f.FirewallManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) There are no rules to forward to the network elements
2016-09-19 11:46:50,932 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Releasing 0 static nat rules for network id=204 as a part of shutdownNetworkRules
2016-09-19 11:46:50,932 DEBUG [c.c.n.f.FirewallManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) There are no rules to forward to the network elements
2016-09-19 11:46:50,934 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Revoking 0 Public load balancing rules for network id=204
2016-09-19 11:46:50,934 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) There are no Load Balancing Rules to forward to the network elements
2016-09-19 11:46:50,936 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Revoking 0 Internal load balancing rules for network id=204
2016-09-19 11:46:50,936 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) There are no Load Balancing Rules to forward to the network elements
2016-09-19 11:46:50,937 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Releasing 0 firewall ingress rules for network id=204 as a part of shutdownNetworkRules
2016-09-19 11:46:50,937 DEBUG [c.c.n.f.FirewallManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) There are no rules to forward to the network elements
2016-09-19 11:46:50,938 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Releasing 0 firewall egress rules for network id=204 as a part of shutdownNetworkRules
2016-09-19 11:46:50,939 DEBUG [c.c.n.f.FirewallManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) There are no rules to forward to the network elements
2016-09-19 11:46:50,941 DEBUG [c.c.n.r.RulesManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Found 0 static nat rules to apply for network id 204
2016-09-19 11:46:51,029 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Sending network shutdown to SecurityGroupProvider
2016-09-19 11:46:51,032 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Sending network shutdown to VirtualRouter
2016-09-19 11:46:51,034 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Implementing the network Ntwk[448deced-7223-4549-98bd-5acafc811f05|Guest|6] elements and resources as a part of network restart
2016-09-19 11:46:51,040 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Asking SecurityGroupProvider to implemenet Ntwk[448deced-7223-4549-98bd-5acafc811f05|Guest|6]
2016-09-19 11:46:51,043 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Asking VirtualRouter to implemenet Ntwk[448deced-7223-4549-98bd-5acafc811f05|Guest|6]
2016-09-19 11:46:51,049 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Lock is acquired for network id 204 as a part of router startup in Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(null)-Cluster(null)-H
ost(null)-Storage()]
2016-09-19 11:46:51,051 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Lock is released for network id 204 as a part of router startup in Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(null)-Cluster(null)-H
ost(null)-Storage()]
2016-09-19 11:46:51,053 WARN [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Failed to implement network Ntwk[448deced-7223-4549-98bd-5acafc811f05|Guest|6] elements and resources as a part of network restart due to
com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is unreachable: Can't find all necessary running routers!
at com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:199)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElementsAndResources(NetworkOrchestrator.java:1080)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:2430)
at com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1892)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy156.restartNetwork(Unknown Source)
at org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:95)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-09-19 11:46:51,054 WARN [c.c.n.NetworkServiceImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Network id=204 failed to restart.
2016-09-19 11:46:51,072 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141) Complete async job-10141, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed to restart network"}
2016-09-19 11:46:51,087 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141) Done executing org.apache.cloudstack.api.command.user.network.RestartNetworkCmd for job-10141
2016-09-19 11:46:51,107 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-5:ctx-8a42cc36 job-10141) Remove job-10141 from job monitoring
Re: Reestablishing a VR when the VR was deleted
Posted by Mason Donahue <md...@backstopsolutions.com>.
Well, that was the bit of information that finally got us back up and running.
The original routerVM failed to start due to the template not being there, so I had redownloaded the template according to the directions at https://docs.cloudstack.apache.org/projects/cloudstack-release-notes/en/4.4.1/upgrade/upgrade-4.3.html and that seems to have been the source of the problem.
Within the errors that accompanied the failure to start of the random VM I picked there was an attempt to start a VR:
2016-09-19 15:29:48,962 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (Work-Job-Executor-4:ctx-21dac544 job-10167/job-10169 ctx-d441bcf6) Lock is acquired for network id 204 as a part of router startup in Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(1)-Cluster(7)-Host(25)-Storage()]
2016-09-19 15:29:48,972 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (Work-Job-Executor-4:ctx-21dac544 job-10167/job-10169 ctx-d441bcf6) Adding nic for Virtual Router in Guest network Ntwk[448deced-7223-4549-98bd-5acafc811f05|Guest|6]
2016-09-19 15:29:48,977 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (Work-Job-Executor-4:ctx-21dac544 job-10167/job-10169 ctx-d441bcf6) Adding nic for Virtual Router in Control network
2016-09-19 15:29:48,980 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-4:ctx-21dac544 job-10167/job-10169 ctx-d441bcf6) Found existing network configuration for offering [Network Offering [3-Control-System-Control-Network]: Ntwk[28d967d8-3b75-4362-9234-b0d029b0d21b|Control|3]
2016-09-19 15:29:48,980 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-4:ctx-21dac544 job-10167/job-10169 ctx-d441bcf6) Releasing lock for Acct[7500fc58-dcf6-11e2-b492-00219b9585d4-system]
2016-09-19 15:29:48,984 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (Work-Job-Executor-4:ctx-21dac544 job-10167/job-10169 ctx-d441bcf6) Allocating the VR i=1703 in datacenter com.cloud.dc.DataCenterVO$$EnhancerByCGLIB$$9732e921@1with the hypervisor type KVM
2016-09-19 15:29:48,988 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (Work-Job-Executor-4:ctx-21dac544 job-10167/job-10169 ctx-d441bcf6) KVM won't support system vm, skip it
2016-09-19 15:29:48,989 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (Work-Job-Executor-4:ctx-21dac544 job-10167/job-10169 ctx-d441bcf6) Lock is released for network id 204 as a part of router startup in Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(1)-Cluster(7)-Host(25)-Storage()]
2016-09-19 15:29:48,989 INFO [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-4:ctx-21dac544 job-10167/job-10169 ctx-d441bcf6) Unable to contact resource.
com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is unreachable: Can't find at least one running router!
The line ‘KVM won’t support system vm, skip it’ was the key, as googling it led to other cloudstack-users mailing list posts with other people who had failed to add systemvm templates back in. Shutting down the management server, updating the TYPE column in the database for the systemvm to ‘SYSTEM’ instead of ‘USER’, and starting a VM again fixed it, and everything came back up happily.
It seems like the documentation on importing systemVMs (for instance, https://docs.cloudstack.apache.org/projects/cloudstack-release-notes/en/4.9.0/upgrade/upgrade-4.4.html) is incorrect, as it says that the template should be imported as ‘Routing: no’. Or is there some other method in which a template is supposed to be set to SYSTEM instead of USER?
Thanks!
--Mason
On 9/19/16, 3:22 PM, "Kirk Kosinski" <ki...@shapeblue.com> wrote:
Hi, if you start a VM in a network that has no VR, the VR will be recreated. So you can stop/start an existing VM in the network, or deploy a new VM to the network.
Best regards,
kirk.kosinski@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HS
@shapeblue
-----Original Message-----
From: Mason Donahue [mailto:mdonahue@backstopsolutions.com]
Sent: Monday, September 19, 2016 9:56 AM
To: users@cloudstack.apache.org
Subject: Reestablishing a VR when the VR was deleted
Hi there,
We’re in a bit of a pickle with our Cloudstack 4.4.1 install. (Yes, I know it’s outdated; we were hoping to upgrade soon and then this happened.) We had an issue where the VR for one of our Networks went down, and I mistakenly missed the ‘cleanup’ checkbox in the ‘Restart network’ menu of the UI. We use the VR solely for DNS. We now have no VRs in our setup, and the network can no longer be restarted due to a check that routers are running. Unfortunately, to delete and re-add the network, the log output states that we’d have to expunge all of our machines, which I am hoping to avoid doing. What are my other options? Can I un-mark the router as deleted in the DB, and will that allow it to limp along to the point where it can at least rebuild the network?
Thanks,
--Mason
(relevant logs below)
2016-09-19 11:46:50,890 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141) Executing AsyncJobVO {id:10141, userId: 2, accountId: 2, instanceType: None, instanceId: null, cmd: org.apache.cloudstack.api.command.user.network.RestartNetworkCmd, cmdInfo: {"response":"json","id":"448deced-7
223-4549-98bd-5acafc811f05","sessionkey":"985SpsvETZfPpiNyTjNsOgXWVQI\u003d","cleanup":"true","ctxDetails":"{\"com.cloud.network.Network\":\"448deced-7223-4549-98bd-5acafc811f05\"}","cmdEventType":"NETWORK.RESTART","ctxUserId":"2","httpmethod":"GET","_":"1474303610834","uuid":"448deced-7223-4549-98bd-5acafc811f05","c
txAccountId":"2","ctxStartEventId":"21061"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 144343483243, completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
2016-09-19 11:46:50,891 DEBUG [c.c.a.ApiServlet] (catalina-exec-22:ctx-94ea893d ctx-69ffa947) ===END=== 192.168.42.156 -- GET command=restartNetwork&id=448deced-7223-4549-98bd-5acafc811f05&cleanup=true&response=json&sessionkey=985SpsvETZfPpiNyTjNsOgXWVQI%3D&_=1474303610834
2016-09-19 11:46:50,925 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Restarting network 204...
2016-09-19 11:46:50,925 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Shutting down the network id=204 as a part of network restart
2016-09-19 11:46:50,929 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Releasing 0 port forwarding rules for network id=204 as a part of shutdownNetworkRules
2016-09-19 11:46:50,930 DEBUG [c.c.n.f.FirewallManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) There are no rules to forward to the network elements
2016-09-19 11:46:50,932 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Releasing 0 static nat rules for network id=204 as a part of shutdownNetworkRules
2016-09-19 11:46:50,932 DEBUG [c.c.n.f.FirewallManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) There are no rules to forward to the network elements
2016-09-19 11:46:50,934 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Revoking 0 Public load balancing rules for network id=204
2016-09-19 11:46:50,934 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) There are no Load Balancing Rules to forward to the network elements
2016-09-19 11:46:50,936 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Revoking 0 Internal load balancing rules for network id=204
2016-09-19 11:46:50,936 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) There are no Load Balancing Rules to forward to the network elements
2016-09-19 11:46:50,937 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Releasing 0 firewall ingress rules for network id=204 as a part of shutdownNetworkRules
2016-09-19 11:46:50,937 DEBUG [c.c.n.f.FirewallManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) There are no rules to forward to the network elements
2016-09-19 11:46:50,938 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Releasing 0 firewall egress rules for network id=204 as a part of shutdownNetworkRules
2016-09-19 11:46:50,939 DEBUG [c.c.n.f.FirewallManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) There are no rules to forward to the network elements
2016-09-19 11:46:50,941 DEBUG [c.c.n.r.RulesManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Found 0 static nat rules to apply for network id 204
2016-09-19 11:46:51,029 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Sending network shutdown to SecurityGroupProvider
2016-09-19 11:46:51,032 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Sending network shutdown to VirtualRouter
2016-09-19 11:46:51,034 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Implementing the network Ntwk[448deced-7223-4549-98bd-5acafc811f05|Guest|6] elements and resources as a part of network restart
2016-09-19 11:46:51,040 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Asking SecurityGroupProvider to implemenet Ntwk[448deced-7223-4549-98bd-5acafc811f05|Guest|6]
2016-09-19 11:46:51,043 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Asking VirtualRouter to implemenet Ntwk[448deced-7223-4549-98bd-5acafc811f05|Guest|6]
2016-09-19 11:46:51,049 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Lock is acquired for network id 204 as a part of router startup in Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(null)-Cluster(null)-H
ost(null)-Storage()]
2016-09-19 11:46:51,051 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Lock is released for network id 204 as a part of router startup in Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(null)-Cluster(null)-H
ost(null)-Storage()]
2016-09-19 11:46:51,053 WARN [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Failed to implement network Ntwk[448deced-7223-4549-98bd-5acafc811f05|Guest|6] elements and resources as a part of network restart due to
com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is unreachable: Can't find all necessary running routers!
at com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:199)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElementsAndResources(NetworkOrchestrator.java:1080)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:2430)
at com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1892)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy156.restartNetwork(Unknown Source)
at org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:95)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-09-19 11:46:51,054 WARN [c.c.n.NetworkServiceImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Network id=204 failed to restart.
2016-09-19 11:46:51,072 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141) Complete async job-10141, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed to restart network"}
2016-09-19 11:46:51,087 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141) Done executing org.apache.cloudstack.api.command.user.network.RestartNetworkCmd for job-10141
2016-09-19 11:46:51,107 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-5:ctx-8a42cc36 job-10141) Remove job-10141 from job monitoring
RE: Reestablishing a VR when the VR was deleted
Posted by Kirk Kosinski <ki...@shapeblue.com>.
Hi, if you start a VM in a network that has no VR, the VR will be recreated. So you can stop/start an existing VM in the network, or deploy a new VM to the network.
Best regards,
kirk.kosinski@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HS
@shapeblue
-----Original Message-----
From: Mason Donahue [mailto:mdonahue@backstopsolutions.com]
Sent: Monday, September 19, 2016 9:56 AM
To: users@cloudstack.apache.org
Subject: Reestablishing a VR when the VR was deleted
Hi there,
We’re in a bit of a pickle with our Cloudstack 4.4.1 install. (Yes, I know it’s outdated; we were hoping to upgrade soon and then this happened.) We had an issue where the VR for one of our Networks went down, and I mistakenly missed the ‘cleanup’ checkbox in the ‘Restart network’ menu of the UI. We use the VR solely for DNS. We now have no VRs in our setup, and the network can no longer be restarted due to a check that routers are running. Unfortunately, to delete and re-add the network, the log output states that we’d have to expunge all of our machines, which I am hoping to avoid doing. What are my other options? Can I un-mark the router as deleted in the DB, and will that allow it to limp along to the point where it can at least rebuild the network?
Thanks,
--Mason
(relevant logs below)
2016-09-19 11:46:50,890 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141) Executing AsyncJobVO {id:10141, userId: 2, accountId: 2, instanceType: None, instanceId: null, cmd: org.apache.cloudstack.api.command.user.network.RestartNetworkCmd, cmdInfo: {"response":"json","id":"448deced-7
223-4549-98bd-5acafc811f05","sessionkey":"985SpsvETZfPpiNyTjNsOgXWVQI\u003d","cleanup":"true","ctxDetails":"{\"com.cloud.network.Network\":\"448deced-7223-4549-98bd-5acafc811f05\"}","cmdEventType":"NETWORK.RESTART","ctxUserId":"2","httpmethod":"GET","_":"1474303610834","uuid":"448deced-7223-4549-98bd-5acafc811f05","c
txAccountId":"2","ctxStartEventId":"21061"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 144343483243, completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
2016-09-19 11:46:50,891 DEBUG [c.c.a.ApiServlet] (catalina-exec-22:ctx-94ea893d ctx-69ffa947) ===END=== 192.168.42.156 -- GET command=restartNetwork&id=448deced-7223-4549-98bd-5acafc811f05&cleanup=true&response=json&sessionkey=985SpsvETZfPpiNyTjNsOgXWVQI%3D&_=1474303610834
2016-09-19 11:46:50,925 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Restarting network 204...
2016-09-19 11:46:50,925 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Shutting down the network id=204 as a part of network restart
2016-09-19 11:46:50,929 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Releasing 0 port forwarding rules for network id=204 as a part of shutdownNetworkRules
2016-09-19 11:46:50,930 DEBUG [c.c.n.f.FirewallManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) There are no rules to forward to the network elements
2016-09-19 11:46:50,932 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Releasing 0 static nat rules for network id=204 as a part of shutdownNetworkRules
2016-09-19 11:46:50,932 DEBUG [c.c.n.f.FirewallManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) There are no rules to forward to the network elements
2016-09-19 11:46:50,934 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Revoking 0 Public load balancing rules for network id=204
2016-09-19 11:46:50,934 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) There are no Load Balancing Rules to forward to the network elements
2016-09-19 11:46:50,936 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Revoking 0 Internal load balancing rules for network id=204
2016-09-19 11:46:50,936 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) There are no Load Balancing Rules to forward to the network elements
2016-09-19 11:46:50,937 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Releasing 0 firewall ingress rules for network id=204 as a part of shutdownNetworkRules
2016-09-19 11:46:50,937 DEBUG [c.c.n.f.FirewallManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) There are no rules to forward to the network elements
2016-09-19 11:46:50,938 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Releasing 0 firewall egress rules for network id=204 as a part of shutdownNetworkRules
2016-09-19 11:46:50,939 DEBUG [c.c.n.f.FirewallManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) There are no rules to forward to the network elements
2016-09-19 11:46:50,941 DEBUG [c.c.n.r.RulesManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Found 0 static nat rules to apply for network id 204
2016-09-19 11:46:51,029 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Sending network shutdown to SecurityGroupProvider
2016-09-19 11:46:51,032 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Sending network shutdown to VirtualRouter
2016-09-19 11:46:51,034 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Implementing the network Ntwk[448deced-7223-4549-98bd-5acafc811f05|Guest|6] elements and resources as a part of network restart
2016-09-19 11:46:51,040 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Asking SecurityGroupProvider to implemenet Ntwk[448deced-7223-4549-98bd-5acafc811f05|Guest|6]
2016-09-19 11:46:51,043 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Asking VirtualRouter to implemenet Ntwk[448deced-7223-4549-98bd-5acafc811f05|Guest|6]
2016-09-19 11:46:51,049 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Lock is acquired for network id 204 as a part of router startup in Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(null)-Cluster(null)-H
ost(null)-Storage()]
2016-09-19 11:46:51,051 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Lock is released for network id 204 as a part of router startup in Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(null)-Cluster(null)-H
ost(null)-Storage()]
2016-09-19 11:46:51,053 WARN [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Failed to implement network Ntwk[448deced-7223-4549-98bd-5acafc811f05|Guest|6] elements and resources as a part of network restart due to
com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is unreachable: Can't find all necessary running routers!
at com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:199)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElementsAndResources(NetworkOrchestrator.java:1080)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:2430)
at com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1892)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy156.restartNetwork(Unknown Source)
at org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:95)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-09-19 11:46:51,054 WARN [c.c.n.NetworkServiceImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Network id=204 failed to restart.
2016-09-19 11:46:51,072 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141) Complete async job-10141, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed to restart network"}
2016-09-19 11:46:51,087 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-5:ctx-8a42cc36 job-10141) Done executing org.apache.cloudstack.api.command.user.network.RestartNetworkCmd for job-10141
2016-09-19 11:46:51,107 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-5:ctx-8a42cc36 job-10141) Remove job-10141 from job monitoring