Unable to restart some networks / virtual routers

[Andrei Mikhailovsky <an...@arhont.com>]

Hello guys, 

I am having an issue with restarting couple of networks and their corresponding virtual routers. The majority of networks can be restarted apart from a few. 

I am running ACS 4.4.2 with KVM hypervisors all running on Ubuntu 14.04. 

Here is the log from the management server: 


2015-03-16 15:34:01,343 DEBUG [c.c.a.t.Request] (AgentManager-Handler-7:null) Seq 8-4763401030874431513: Processing: { Ans: , MgmtId: 115129173025118, via: 8, Ver: v1, Flags: 110, [{"com.cloud.agent.api.Answer":{"result":false,"details":"java.lang.NullPointerException\n\tat com.cloud.network.Networks$BroadcastDomainType.getSchemeValue(Networks.java:159)\n\tat com.cloud.network.Networks$BroadcastDomainType.getValue(Networks.java:213)\n\tat com.cloud.hypervisor.kvm.resource.BridgeVifDriver.plug(BridgeVifDriver.java:96)\n\tat com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVif(LibvirtComputingResource.java:4010)\n\tat com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVifs(LibvirtComputingResource.java:3766)\n\tat com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:3795)\n\tat com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1333)\n\tat com.cloud.agent.Agent.processRequest(Agent.java:501)\n\tat com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:808)\n\tat com.cloud.utils.nio.Task.run(Task.java:84)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat java.lang.Thread.run(Thread.java:745)\n","wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous failure","wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous failure","wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous failure","wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous failure","wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous failure","wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous failure","wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous failure","wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous failure","wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous failure","wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous failure","wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous failure","wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous failure","wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous failure","wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous failure","wait":0}}] } 
2015-03-16 15:34:01,343 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-7:null) Seq 8-4763401030874431513: No more commands found 
2015-03-16 15:34:01,344 DEBUG [c.c.a.t.Request] (Work-Job-Executor-7:ctx-4fca30aa job-13975/job-13978 ctx-0d61e6c8) Seq 8-4763401030874431513: Received: { Ans: , MgmtId: 115129173025118, via: 8, Ver: v1, Flags: 110, { Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer } } 
2015-03-16 15:34:01,350 ERROR [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-7:ctx-4fca30aa job-13975/job-13978 ctx-0d61e6c8) Failed to start instance VM[DomainRouter|r-1192-VM] 
com.cloud.utils.exception.CloudRuntimeException: Unable to get answer that is of class com.cloud.agent.api.StartAnswer 
at com.cloud.agent.manager.Commands.getAnswer(Commands.java:80) 
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1020) 
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5195) 
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
at java.lang.reflect.Method.invoke(Method.java:606) 
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) 
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5340) 
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) 
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503) 
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) 
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) 
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) 
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) 
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) 
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460) 
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) 
at java.util.concurrent.FutureTask.run(FutureTask.java:262) 
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
at java.lang.Thread.run(Thread.java:745) 
2015-03-16 15:34:01,354 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-7:ctx-4fca30aa job-13975/job-13978 ctx-0d61e6c8) Cleaning up resources for the vm VM[DomainRouter|r-1192-VM] in Starting state 



The host server have the following logs (please note that the timestamps show different times as they are from two different start attempts, but the entries are the same every time I attempt to start the virtual router): 

2015-03-16 16:03:36,781 WARN [cloud.agent.Agent] (agentRequest-Handler-2:null) Caught: 
java.lang.NullPointerException 
at com.cloud.network.Networks$BroadcastDomainType.getSchemeValue(Networks.java:159) 
at com.cloud.network.Networks$BroadcastDomainType.getValue(Networks.java:213) 
at com.cloud.hypervisor.kvm.resource.BridgeVifDriver.plug(BridgeVifDriver.java:96) 
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVif(LibvirtComputingResource.java:4010) 
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVifs(LibvirtComputingResource.java:3766) 
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:3795) 
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1333) 
at com.cloud.agent.Agent.processRequest(Agent.java:501) 
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:808) 
at com.cloud.utils.nio.Task.run(Task.java:84) 
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
at java.lang.Thread.run(Thread.java:745) 
2015-03-16 16:03:36,784 DEBUG [cloud.agent.Agent] (agentRequest-Handler-2:null) Seq 8-4763401030874431681: { Ans: , MgmtId: 115129173025118, via: 8, Ver: v1, Flags: 110, [{"com.cloud.agent.api.Answer":{"result":false,"details":"java.lang.NullPointerException\n\tat com.cloud.network.Networks$BroadcastDomainType.getSchemeValue(Networks.java:159)\n\tat com.cloud.network.Networks$BroadcastDomainType.getValue(Networks.java:213)\n\tat com.cloud.hypervisor.kvm.resource.BridgeVifDriver.plug(BridgeVifDriver.java:96)\n\tat com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVif(LibvirtComputingResource.java:4010)\n\tat com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVifs(LibvirtComputingResource.java:3766)\n\tat com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:3795)\n\tat com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1333)\n\tat com.cloud.agent.Agent.processRequest(Agent.java:501)\n\tat com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:808)\n\tat com.cloud.utils.nio.Task.run(Task.java:84)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat java.lang.Thread.run(Thread.java:745)\n","wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous failure","wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous failure","wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous failure","wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous failure","wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous failure","wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous failure","wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous failure","wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous failure","wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous failure","wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous failure","wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous failure","wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous failure","wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous failure","wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous failure","wait":0}}] } 



Does anyone have any idea what's the problem? 

Thanks 

Andrei