You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Sonali Jadhav <so...@servercentralen.se> on 2016/08/24 14:52:43 UTC
VR unable to reach gateway
Hi,
I am running ACS 4.8.0. One cluster with two xenserver 6.5 hosts.
One of the VM's network went unreachable so when I checked VR it was unable to reach gateway. So I tried to restart VR and did cleanup. But issue is still same. When it rebooted VR, I saw ping command to gateway which ending with this ,
PING 123.23.24.25 (123.23.24.25): 48 data bytes
/etc/init.d/cloud-early-config: line 574: 2009 Terminated ping -n -c 3 $GW
But then it did give root login prompt on console. But problem is I am unable to login to VR anymore since cloudstack just shutdown VR vm from xen and I can't even log into VR console so I can't check more logs.
Cloudstack shut down VR automatically with this mail alert -
Command: com.cloud.agent.api.Command failed while starting virtual router
Details: Timed out in waiting SSH execution result com.cloud.agent.api.Answer
And on cloudstack management logs I get this error,
2016-08-24 14:32:41,548 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-112:ctx-3f255d1a job-17880) (logid:0e364177) Unexpected exception while executing org.apache.cloudstack.api.command.admin.router.RebootRouterCmd
com.cloud.exception.AgentUnavailableException: Resource [Host:1] is unreachable: Host 1: Unable to start instance due to Unable to start VM:772fed1c-bb75-4778-9811-7a0a3633cb5b due to error in finalizeStart, not retrying
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1120)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4580)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.cloud.utils.exception.ExecutionException: Unable to start VM:772fed1c-bb75-4778-9811-7a0a3633cb5b due to error in finalizeStart, not retrying
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1084)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4580)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
... 18 more
2016-08-24 14:32:41,550 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-112:ctx-3f255d1a job-17880) (logid:0e364177) Complete async job-17880, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Resource [Host:1] is unreachable: Host 1: Unable to start instance due to Unable to start VM:772fed1c-bb75-4778-9811-7a0a3633cb5b due to error in finalizeStart, not retrying"}
2016-08-24 14:32:41,553 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-112:ctx-3f255d1a job-17880) (logid:0e364177) Publish async job-17880 complete on message bus
Network I am trying to get up is production network, so I need it up asap. What could be the problem here?
I tried to create new VR, and that seems to reach gateway. Its problem with only this VR. What if I destroy router, from cloudstack panel? Would it recreate VR automatically? Or any other suggestion on this issue?
/sonali
RE: VR unable to reach gateway
Posted by Sonali Jadhav <so...@servercentralen.se>.
That did not helped, I ended up creating new network, so we changed dns. That was very weird issue.
/Sonali
-----Original Message-----
From: Glenn Wagner [mailto:glenn.wagner@shapeblue.com]
Sent: Wednesday, August 24, 2016 8:59 PM
To: users@cloudstack.apache.org
Subject: RE: VR unable to reach gateway
Ok if you destroy the VR , cloudstack will create a new one.
Glenn
glenn.wagner@shapeblue.com
www.shapeblue.com
First Floor, Victoria Centre, 7 Victoria Street, Somerset West, Cape Town 7129South Africa @shapeblue
-----Original Message-----
From: Sonali Jadhav [mailto:sonali@servercentralen.se]
Sent: Wednesday, 24 August 2016 5:20 PM
To: users@cloudstack.apache.org
Subject: RE: VR unable to reach gateway
Hi,
Yes I tried to move VR to another Host, that also gave same issue.
And no Hosts are unreachable, that was weird error. I have tried making new VR, for test that worked.
/Sonali
-----Original Message-----
From: Glenn Wagner [mailto:glenn.wagner@shapeblue.com]
Sent: Wednesday, August 24, 2016 8:42 PM
To: users@cloudstack.apache.org
Subject: RE: VR unable to reach gateway
Hi.
The other VR that works which host is it on?
The error seems to be a HOST 1 problem - [Host:1] is unreachable:
Can you migrate the VR to the other host and see if it works ?
Glenn
glenn.wagner@shapeblue.com
www.shapeblue.com
First Floor, Victoria Centre, 7 Victoria Street, Somerset West, Cape Town 7129South Africa @shapeblue
-----Original Message-----
From: Sonali Jadhav [mailto:sonali@servercentralen.se]
Sent: Wednesday, 24 August 2016 4:53 PM
To: users@cloudstack.apache.org
Subject: VR unable to reach gateway
Hi,
I am running ACS 4.8.0. One cluster with two xenserver 6.5 hosts.
One of the VM's network went unreachable so when I checked VR it was unable to reach gateway. So I tried to restart VR and did cleanup. But issue is still same. When it rebooted VR, I saw ping command to gateway which ending with this ,
PING 123.23.24.25 (123.23.24.25): 48 data bytes
/etc/init.d/cloud-early-config: line 574: 2009 Terminated ping -n -c 3 $GW
But then it did give root login prompt on console. But problem is I am unable to login to VR anymore since cloudstack just shutdown VR vm from xen and I can't even log into VR console so I can't check more logs.
Cloudstack shut down VR automatically with this mail alert -
Command: com.cloud.agent.api.Command failed while starting virtual router
Details: Timed out in waiting SSH execution result com.cloud.agent.api.Answer
And on cloudstack management logs I get this error,
2016-08-24 14:32:41,548 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-112:ctx-3f255d1a job-17880) (logid:0e364177) Unexpected exception while executing org.apache.cloudstack.api.command.admin.router.RebootRouterCmd
com.cloud.exception.AgentUnavailableException: Resource [Host:1] is unreachable: Host 1: Unable to start instance due to Unable to start VM:772fed1c-bb75-4778-9811-7a0a3633cb5b due to error in finalizeStart, not retrying
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1120)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4580)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.cloud.utils.exception.ExecutionException: Unable to start VM:772fed1c-bb75-4778-9811-7a0a3633cb5b due to error in finalizeStart, not retrying
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1084)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4580)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
... 18 more
2016-08-24 14:32:41,550 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-112:ctx-3f255d1a job-17880) (logid:0e364177) Complete async job-17880, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Resource [Host:1] is unreachable: Host 1: Unable to start instance due to Unable to start VM:772fed1c-bb75-4778-9811-7a0a3633cb5b due to error in finalizeStart, not retrying"}
2016-08-24 14:32:41,553 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-112:ctx-3f255d1a job-17880) (logid:0e364177) Publish async job-17880 complete on message bus
Network I am trying to get up is production network, so I need it up asap. What could be the problem here?
I tried to create new VR, and that seems to reach gateway. Its problem with only this VR. What if I destroy router, from cloudstack panel? Would it recreate VR automatically? Or any other suggestion on this issue?
/sonali
RE: VR unable to reach gateway
Posted by Glenn Wagner <gl...@shapeblue.com>.
Ok if you destroy the VR , cloudstack will create a new one.
Glenn
glenn.wagner@shapeblue.com
www.shapeblue.com
First Floor, Victoria Centre, 7 Victoria Street, Somerset West, Cape Town 7129South Africa
@shapeblue
-----Original Message-----
From: Sonali Jadhav [mailto:sonali@servercentralen.se]
Sent: Wednesday, 24 August 2016 5:20 PM
To: users@cloudstack.apache.org
Subject: RE: VR unable to reach gateway
Hi,
Yes I tried to move VR to another Host, that also gave same issue.
And no Hosts are unreachable, that was weird error. I have tried making new VR, for test that worked.
/Sonali
-----Original Message-----
From: Glenn Wagner [mailto:glenn.wagner@shapeblue.com]
Sent: Wednesday, August 24, 2016 8:42 PM
To: users@cloudstack.apache.org
Subject: RE: VR unable to reach gateway
Hi.
The other VR that works which host is it on?
The error seems to be a HOST 1 problem - [Host:1] is unreachable:
Can you migrate the VR to the other host and see if it works ?
Glenn
glenn.wagner@shapeblue.com
www.shapeblue.com
First Floor, Victoria Centre, 7 Victoria Street, Somerset West, Cape Town 7129South Africa @shapeblue
-----Original Message-----
From: Sonali Jadhav [mailto:sonali@servercentralen.se]
Sent: Wednesday, 24 August 2016 4:53 PM
To: users@cloudstack.apache.org
Subject: VR unable to reach gateway
Hi,
I am running ACS 4.8.0. One cluster with two xenserver 6.5 hosts.
One of the VM's network went unreachable so when I checked VR it was unable to reach gateway. So I tried to restart VR and did cleanup. But issue is still same. When it rebooted VR, I saw ping command to gateway which ending with this ,
PING 123.23.24.25 (123.23.24.25): 48 data bytes
/etc/init.d/cloud-early-config: line 574: 2009 Terminated ping -n -c 3 $GW
But then it did give root login prompt on console. But problem is I am unable to login to VR anymore since cloudstack just shutdown VR vm from xen and I can't even log into VR console so I can't check more logs.
Cloudstack shut down VR automatically with this mail alert -
Command: com.cloud.agent.api.Command failed while starting virtual router
Details: Timed out in waiting SSH execution result com.cloud.agent.api.Answer
And on cloudstack management logs I get this error,
2016-08-24 14:32:41,548 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-112:ctx-3f255d1a job-17880) (logid:0e364177) Unexpected exception while executing org.apache.cloudstack.api.command.admin.router.RebootRouterCmd
com.cloud.exception.AgentUnavailableException: Resource [Host:1] is unreachable: Host 1: Unable to start instance due to Unable to start VM:772fed1c-bb75-4778-9811-7a0a3633cb5b due to error in finalizeStart, not retrying
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1120)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4580)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.cloud.utils.exception.ExecutionException: Unable to start VM:772fed1c-bb75-4778-9811-7a0a3633cb5b due to error in finalizeStart, not retrying
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1084)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4580)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
... 18 more
2016-08-24 14:32:41,550 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-112:ctx-3f255d1a job-17880) (logid:0e364177) Complete async job-17880, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Resource [Host:1] is unreachable: Host 1: Unable to start instance due to Unable to start VM:772fed1c-bb75-4778-9811-7a0a3633cb5b due to error in finalizeStart, not retrying"}
2016-08-24 14:32:41,553 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-112:ctx-3f255d1a job-17880) (logid:0e364177) Publish async job-17880 complete on message bus
Network I am trying to get up is production network, so I need it up asap. What could be the problem here?
I tried to create new VR, and that seems to reach gateway. Its problem with only this VR. What if I destroy router, from cloudstack panel? Would it recreate VR automatically? Or any other suggestion on this issue?
/sonali
RE: VR unable to reach gateway
Posted by Sonali Jadhav <so...@servercentralen.se>.
Hi,
Yes I tried to move VR to another Host, that also gave same issue.
And no Hosts are unreachable, that was weird error. I have tried making new VR, for test that worked.
/Sonali
-----Original Message-----
From: Glenn Wagner [mailto:glenn.wagner@shapeblue.com]
Sent: Wednesday, August 24, 2016 8:42 PM
To: users@cloudstack.apache.org
Subject: RE: VR unable to reach gateway
Hi.
The other VR that works which host is it on?
The error seems to be a HOST 1 problem - [Host:1] is unreachable:
Can you migrate the VR to the other host and see if it works ?
Glenn
glenn.wagner@shapeblue.com
www.shapeblue.com
First Floor, Victoria Centre, 7 Victoria Street, Somerset West, Cape Town 7129South Africa @shapeblue
-----Original Message-----
From: Sonali Jadhav [mailto:sonali@servercentralen.se]
Sent: Wednesday, 24 August 2016 4:53 PM
To: users@cloudstack.apache.org
Subject: VR unable to reach gateway
Hi,
I am running ACS 4.8.0. One cluster with two xenserver 6.5 hosts.
One of the VM's network went unreachable so when I checked VR it was unable to reach gateway. So I tried to restart VR and did cleanup. But issue is still same. When it rebooted VR, I saw ping command to gateway which ending with this ,
PING 123.23.24.25 (123.23.24.25): 48 data bytes
/etc/init.d/cloud-early-config: line 574: 2009 Terminated ping -n -c 3 $GW
But then it did give root login prompt on console. But problem is I am unable to login to VR anymore since cloudstack just shutdown VR vm from xen and I can't even log into VR console so I can't check more logs.
Cloudstack shut down VR automatically with this mail alert -
Command: com.cloud.agent.api.Command failed while starting virtual router
Details: Timed out in waiting SSH execution result com.cloud.agent.api.Answer
And on cloudstack management logs I get this error,
2016-08-24 14:32:41,548 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-112:ctx-3f255d1a job-17880) (logid:0e364177) Unexpected exception while executing org.apache.cloudstack.api.command.admin.router.RebootRouterCmd
com.cloud.exception.AgentUnavailableException: Resource [Host:1] is unreachable: Host 1: Unable to start instance due to Unable to start VM:772fed1c-bb75-4778-9811-7a0a3633cb5b due to error in finalizeStart, not retrying
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1120)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4580)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.cloud.utils.exception.ExecutionException: Unable to start VM:772fed1c-bb75-4778-9811-7a0a3633cb5b due to error in finalizeStart, not retrying
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1084)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4580)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
... 18 more
2016-08-24 14:32:41,550 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-112:ctx-3f255d1a job-17880) (logid:0e364177) Complete async job-17880, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Resource [Host:1] is unreachable: Host 1: Unable to start instance due to Unable to start VM:772fed1c-bb75-4778-9811-7a0a3633cb5b due to error in finalizeStart, not retrying"}
2016-08-24 14:32:41,553 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-112:ctx-3f255d1a job-17880) (logid:0e364177) Publish async job-17880 complete on message bus
Network I am trying to get up is production network, so I need it up asap. What could be the problem here?
I tried to create new VR, and that seems to reach gateway. Its problem with only this VR. What if I destroy router, from cloudstack panel? Would it recreate VR automatically? Or any other suggestion on this issue?
/sonali
RE: VR unable to reach gateway
Posted by Glenn Wagner <gl...@shapeblue.com>.
Hi.
The other VR that works which host is it on?
The error seems to be a HOST 1 problem - [Host:1] is unreachable:
Can you migrate the VR to the other host and see if it works ?
Glenn
glenn.wagner@shapeblue.com
www.shapeblue.com
First Floor, Victoria Centre, 7 Victoria Street, Somerset West, Cape Town 7129South Africa
@shapeblue
-----Original Message-----
From: Sonali Jadhav [mailto:sonali@servercentralen.se]
Sent: Wednesday, 24 August 2016 4:53 PM
To: users@cloudstack.apache.org
Subject: VR unable to reach gateway
Hi,
I am running ACS 4.8.0. One cluster with two xenserver 6.5 hosts.
One of the VM's network went unreachable so when I checked VR it was unable to reach gateway. So I tried to restart VR and did cleanup. But issue is still same. When it rebooted VR, I saw ping command to gateway which ending with this ,
PING 123.23.24.25 (123.23.24.25): 48 data bytes
/etc/init.d/cloud-early-config: line 574: 2009 Terminated ping -n -c 3 $GW
But then it did give root login prompt on console. But problem is I am unable to login to VR anymore since cloudstack just shutdown VR vm from xen and I can't even log into VR console so I can't check more logs.
Cloudstack shut down VR automatically with this mail alert -
Command: com.cloud.agent.api.Command failed while starting virtual router
Details: Timed out in waiting SSH execution result com.cloud.agent.api.Answer
And on cloudstack management logs I get this error,
2016-08-24 14:32:41,548 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-112:ctx-3f255d1a job-17880) (logid:0e364177) Unexpected exception while executing org.apache.cloudstack.api.command.admin.router.RebootRouterCmd
com.cloud.exception.AgentUnavailableException: Resource [Host:1] is unreachable: Host 1: Unable to start instance due to Unable to start VM:772fed1c-bb75-4778-9811-7a0a3633cb5b due to error in finalizeStart, not retrying
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1120)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4580)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.cloud.utils.exception.ExecutionException: Unable to start VM:772fed1c-bb75-4778-9811-7a0a3633cb5b due to error in finalizeStart, not retrying
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1084)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4580)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
... 18 more
2016-08-24 14:32:41,550 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-112:ctx-3f255d1a job-17880) (logid:0e364177) Complete async job-17880, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Resource [Host:1] is unreachable: Host 1: Unable to start instance due to Unable to start VM:772fed1c-bb75-4778-9811-7a0a3633cb5b due to error in finalizeStart, not retrying"}
2016-08-24 14:32:41,553 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-112:ctx-3f255d1a job-17880) (logid:0e364177) Publish async job-17880 complete on message bus
Network I am trying to get up is production network, so I need it up asap. What could be the problem here?
I tried to create new VR, and that seems to reach gateway. Its problem with only this VR. What if I destroy router, from cloudstack panel? Would it recreate VR automatically? Or any other suggestion on this issue?
/sonali