You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by JS...@h2l.com on 2013/03/18 21:02:32 UTC
System VMs won't start
I know others have had problems with System VMs not starting, but my logs
appears to be different.
I'm running CS 4.0.1 on a flat network with 2 hosts, a management server,
and a shared storage server. All on CentOS 6.x machines.
After installation it shows me that the primary storage is up, and my
secondary storage is enabled. But neither of the system vms (Console
Proxy, or Secondary Storage) start successfully.
Here's what the management log has to say repeatedly:
2013-03-18 14:55:20,609 WARN
[storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null)
Exception while trying to start secondary storage vm
com.cloud.exception.AgentUnavailableException: Resource [Host:3] is
unreachable: Host 3: Unable to start instance due to Unable to get answer
that is of class com.cloud.agent.api.StartAnswer
at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:847)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:472)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:465)
at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:257)
at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:684)
at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1310)
at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:119)
at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:106)
at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:34)
at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:83)
at
com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:73)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to get
answer that is of class com.cloud.agent.api.StartAnswer
at com.cloud.agent.manager.Commands.getAnswer(Commands.java:80)
at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:783)
... 20 more
2013-03-18 14:55:20,610 INFO
[storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) Unable
to start secondary storage vm for standby capacity, secStorageVm vm Id :
10157, will recycle it and start a new one
2013-03-18 14:55:20,610 INFO
[cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:null)
Primary secondary storage is not even started, wait until next turn
2013-03-18 14:55:22,497 DEBUG [cloud.server.StatsCollector]
(StatsCollector-1:null) HostStatsCollector is running...
2013-03-18 14:55:23,142 DEBUG [agent.transport.Request]
(StatsCollector-1:null) Seq 1-891290142: Received: { Ans: , MgmtId:
161336797896, via: 1, Ver: v1, Flags: 10, { GetHostStatsAnswer } }
2013-03-18 14:55:23,829 DEBUG [agent.transport.Request]
(StatsCollector-1:null) Seq 3-773128182: Received: { Ans: , MgmtId:
161336797896, via: 3, Ver: v1, Flags: 10, { GetHostStatsAnswer } }
2013-03-18 14:55:29,320 DEBUG [cloud.server.StatsCollector]
(StatsCollector-2:null) StorageCollector is running...
2013-03-18 14:55:29,324 DEBUG [cloud.server.StatsCollector]
(StatsCollector-2:null) There is no secondary storage VM for secondary
storage host nfs://192.168.100.239/Storage/secondary
Any thoughts?
Thanks,
Jeremy
Re: System VMs won't start
Posted by Ahmad Emneina <ae...@gmail.com>.
possible dns resolution issue. can you resolve the hosts provided to
cloudstack from the management server?
On Mon, Mar 18, 2013 at 1:54 PM, <JS...@h2l.com> wrote:
> I don't even know what 'host 3' would be. My two hosts servers have fully
> qualified domain names (and are registered in CS with those names), and
> I'm using IP addressing for the NFS share.
>
> The management server can connect to the NFS share as I did that to
> download the template as specified in the install instructions.
>
> I wonder if it has something to do with the NFS server though as I see
> this in it's log file:
>
> Mar 18 16:32:15 H2L-NAS1 rpc.idmapd[1504]: nss_getpwnam: name
> 'root@h2l.com' does not map into domain 'localdomain'
> Mar 18 16:34:45 H2L-NAS1 rpc.idmapd[1504]: nss_getpwnam: name
> 'root@h2l.com' does not map into domain 'localdomain'
> Mar 18 16:37:44 H2L-NAS1 rpc.idmapd[1504]: nss_getpwnam: name
> 'root@h2l.com' does not map into domain 'localdomain'
> Mar 18 16:40:45 H2L-NAS1 rpc.idmapd[1504]: nss_getpwnam: name
> 'root@h2l.com' does not map into domain 'localdomain'
> Mar 18 16:42:45 H2L-NAS1 rpc.idmapd[1504]: nss_getpwnam: name
> 'root@h2l.com' does not map into domain 'localdomain'
>
> There's not much more in the management server log, it just keeps
> repeating those same lines I posted earlier.
>
>
>
> From: Ahmad Emneina <ae...@gmail.com>
> To: Cloudstack users <cl...@incubator.apache.org>,
> Date: 03/18/2013 04:16 PM
> Subject: Re: System VMs won't start
>
>
>
> At a glance, its seems the management server cant connect to 'host 3'.
> Maybe a more full log might surface the exact issue. I'd start by trying
> to
> connect to the host in question from the management server.
>
>
> On Mon, Mar 18, 2013 at 1:02 PM, <JS...@h2l.com> wrote:
>
> > I know others have had problems with System VMs not starting, but my
> logs
> > appears to be different.
> >
> > I'm running CS 4.0.1 on a flat network with 2 hosts, a management
> server,
> > and a shared storage server. All on CentOS 6.x machines.
> >
> > After installation it shows me that the primary storage is up, and my
> > secondary storage is enabled. But neither of the system vms (Console
> > Proxy, or Secondary Storage) start successfully.
> >
> > Here's what the management log has to say repeatedly:
> >
> > 2013-03-18 14:55:20,609 WARN
> > [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null)
> > Exception while trying to start secondary storage vm
> > com.cloud.exception.AgentUnavailableException: Resource [Host:3] is
> > unreachable: Host 3: Unable to start instance due to Unable to get
> answer
> > that is of class com.cloud.agent.api.StartAnswer
> > at
> >
> >
>
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:847)
> > at
> >
> >
>
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:472)
> > at
> >
> >
>
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:465)
> > at
> >
> >
>
> com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:257)
> > at
> >
> >
>
> com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:684)
> > at
> >
> >
>
> com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1310)
> > at
> >
> >
>
> com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:119)
> > at
> >
> >
>
> com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
> > at
> > com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:106)
> > at
> > com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:34)
> > at
> >
> com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:83)
> > at
> > com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:73)
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > at
> >
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
> > at
> > java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
> > at
> >
> >
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
> > at
> >
> >
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
> > at
> >
> >
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
> > at
> >
> >
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > at java.lang.Thread.run(Thread.java:679)
> > Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to
> get
> > answer that is of class com.cloud.agent.api.StartAnswer
> > at com.cloud.agent.manager.Commands.getAnswer(Commands.java:80)
> > at
> >
> >
>
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:783)
> > ... 20 more
> > 2013-03-18 14:55:20,610 INFO
> > [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null)
> Unable
> > to start secondary storage vm for standby capacity, secStorageVm vm Id :
> > 10157, will recycle it and start a new one
> > 2013-03-18 14:55:20,610 INFO
> > [cloud.secstorage.PremiumSecondaryStorageManagerImpl]
> (secstorage-1:null)
> > Primary secondary storage is not even started, wait until next turn
> > 2013-03-18 14:55:22,497 DEBUG [cloud.server.StatsCollector]
> > (StatsCollector-1:null) HostStatsCollector is running...
> > 2013-03-18 14:55:23,142 DEBUG [agent.transport.Request]
> > (StatsCollector-1:null) Seq 1-891290142: Received: { Ans: , MgmtId:
> > 161336797896, via: 1, Ver: v1, Flags: 10, { GetHostStatsAnswer } }
> > 2013-03-18 14:55:23,829 DEBUG [agent.transport.Request]
> > (StatsCollector-1:null) Seq 3-773128182: Received: { Ans: , MgmtId:
> > 161336797896, via: 3, Ver: v1, Flags: 10, { GetHostStatsAnswer } }
> > 2013-03-18 14:55:29,320 DEBUG [cloud.server.StatsCollector]
> > (StatsCollector-2:null) StorageCollector is running...
> > 2013-03-18 14:55:29,324 DEBUG [cloud.server.StatsCollector]
> > (StatsCollector-2:null) There is no secondary storage VM for secondary
> > storage host nfs://192.168.100.239/Storage/secondary
> >
> > Any thoughts?
> >
> > Thanks,
> > Jeremy
>
>
Re: System VMs won't start
Posted by Carlos Reategui <ca...@reategui.com>.
Not sure if I ran into the same thing you did or not. I am finally up and
running but there are a few variables at play so I am not sure which one
did it for me.
Since you mention you are on CentOS on your hosts I am assuming KVM. I am
using Xen via xcp-xapi on Ubuntu so not sure how much of this applies to
you.
Things that have changed ordered in most likely to have made the difference
first:
- Disable STP and enable multiple host per port on my switch per this:
http://support.citrix.com/article/CTX123158
- Use separate nic for management and guest traffic. Previously I was
bonding the 2 nics
- Set Global Settings->secstorage.allowed.internal.sites to my network
192.168.1.0/24. I have everything (CS Management, hosts, primary and
secondary nfs, etc) on this network.
- Enable local SR for system vms (Global
Settings->system.vm.use.local.storage->true).
One quick thing you may want to try is to add your hosts via IP instead of
DNS name.
On Mon, Mar 18, 2013 at 1:54 PM, <JS...@h2l.com> wrote:
> I don't even know what 'host 3' would be. My two hosts servers have fully
> qualified domain names (and are registered in CS with those names), and
> I'm using IP addressing for the NFS share.
>
> The management server can connect to the NFS share as I did that to
> download the template as specified in the install instructions.
>
> I wonder if it has something to do with the NFS server though as I see
> this in it's log file:
>
> Mar 18 16:32:15 H2L-NAS1 rpc.idmapd[1504]: nss_getpwnam: name
> 'root@h2l.com' does not map into domain 'localdomain'
> Mar 18 16:34:45 H2L-NAS1 rpc.idmapd[1504]: nss_getpwnam: name
> 'root@h2l.com' does not map into domain 'localdomain'
> Mar 18 16:37:44 H2L-NAS1 rpc.idmapd[1504]: nss_getpwnam: name
> 'root@h2l.com' does not map into domain 'localdomain'
> Mar 18 16:40:45 H2L-NAS1 rpc.idmapd[1504]: nss_getpwnam: name
> 'root@h2l.com' does not map into domain 'localdomain'
> Mar 18 16:42:45 H2L-NAS1 rpc.idmapd[1504]: nss_getpwnam: name
> 'root@h2l.com' does not map into domain 'localdomain'
>
> There's not much more in the management server log, it just keeps
> repeating those same lines I posted earlier.
>
>
>
> From: Ahmad Emneina <ae...@gmail.com>
> To: Cloudstack users <cl...@incubator.apache.org>,
> Date: 03/18/2013 04:16 PM
> Subject: Re: System VMs won't start
>
>
>
> At a glance, its seems the management server cant connect to 'host 3'.
> Maybe a more full log might surface the exact issue. I'd start by trying
> to
> connect to the host in question from the management server.
>
>
> On Mon, Mar 18, 2013 at 1:02 PM, <JS...@h2l.com> wrote:
>
> > I know others have had problems with System VMs not starting, but my
> logs
> > appears to be different.
> >
> > I'm running CS 4.0.1 on a flat network with 2 hosts, a management
> server,
> > and a shared storage server. All on CentOS 6.x machines.
> >
> > After installation it shows me that the primary storage is up, and my
> > secondary storage is enabled. But neither of the system vms (Console
> > Proxy, or Secondary Storage) start successfully.
> >
> > Here's what the management log has to say repeatedly:
> >
> > 2013-03-18 14:55:20,609 WARN
> > [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null)
> > Exception while trying to start secondary storage vm
> > com.cloud.exception.AgentUnavailableException: Resource [Host:3] is
> > unreachable: Host 3: Unable to start instance due to Unable to get
> answer
> > that is of class com.cloud.agent.api.StartAnswer
> > at
> >
> >
>
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:847)
> > at
> >
> >
>
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:472)
> > at
> >
> >
>
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:465)
> > at
> >
> >
>
> com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:257)
> > at
> >
> >
>
> com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:684)
> > at
> >
> >
>
> com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1310)
> > at
> >
> >
>
> com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:119)
> > at
> >
> >
>
> com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
> > at
> > com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:106)
> > at
> > com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:34)
> > at
> >
> com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:83)
> > at
> > com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:73)
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > at
> >
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
> > at
> > java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
> > at
> >
> >
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
> > at
> >
> >
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
> > at
> >
> >
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
> > at
> >
> >
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > at java.lang.Thread.run(Thread.java:679)
> > Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to
> get
> > answer that is of class com.cloud.agent.api.StartAnswer
> > at com.cloud.agent.manager.Commands.getAnswer(Commands.java:80)
> > at
> >
> >
>
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:783)
> > ... 20 more
> > 2013-03-18 14:55:20,610 INFO
> > [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null)
> Unable
> > to start secondary storage vm for standby capacity, secStorageVm vm Id :
> > 10157, will recycle it and start a new one
> > 2013-03-18 14:55:20,610 INFO
> > [cloud.secstorage.PremiumSecondaryStorageManagerImpl]
> (secstorage-1:null)
> > Primary secondary storage is not even started, wait until next turn
> > 2013-03-18 14:55:22,497 DEBUG [cloud.server.StatsCollector]
> > (StatsCollector-1:null) HostStatsCollector is running...
> > 2013-03-18 14:55:23,142 DEBUG [agent.transport.Request]
> > (StatsCollector-1:null) Seq 1-891290142: Received: { Ans: , MgmtId:
> > 161336797896, via: 1, Ver: v1, Flags: 10, { GetHostStatsAnswer } }
> > 2013-03-18 14:55:23,829 DEBUG [agent.transport.Request]
> > (StatsCollector-1:null) Seq 3-773128182: Received: { Ans: , MgmtId:
> > 161336797896, via: 3, Ver: v1, Flags: 10, { GetHostStatsAnswer } }
> > 2013-03-18 14:55:29,320 DEBUG [cloud.server.StatsCollector]
> > (StatsCollector-2:null) StorageCollector is running...
> > 2013-03-18 14:55:29,324 DEBUG [cloud.server.StatsCollector]
> > (StatsCollector-2:null) There is no secondary storage VM for secondary
> > storage host nfs://192.168.100.239/Storage/secondary
> >
> > Any thoughts?
> >
> > Thanks,
> > Jeremy
>
>
Re: System VMs won't start
Posted by JS...@h2l.com.
I don't even know what 'host 3' would be. My two hosts servers have fully
qualified domain names (and are registered in CS with those names), and
I'm using IP addressing for the NFS share.
The management server can connect to the NFS share as I did that to
download the template as specified in the install instructions.
I wonder if it has something to do with the NFS server though as I see
this in it's log file:
Mar 18 16:32:15 H2L-NAS1 rpc.idmapd[1504]: nss_getpwnam: name
'root@h2l.com' does not map into domain 'localdomain'
Mar 18 16:34:45 H2L-NAS1 rpc.idmapd[1504]: nss_getpwnam: name
'root@h2l.com' does not map into domain 'localdomain'
Mar 18 16:37:44 H2L-NAS1 rpc.idmapd[1504]: nss_getpwnam: name
'root@h2l.com' does not map into domain 'localdomain'
Mar 18 16:40:45 H2L-NAS1 rpc.idmapd[1504]: nss_getpwnam: name
'root@h2l.com' does not map into domain 'localdomain'
Mar 18 16:42:45 H2L-NAS1 rpc.idmapd[1504]: nss_getpwnam: name
'root@h2l.com' does not map into domain 'localdomain'
There's not much more in the management server log, it just keeps
repeating those same lines I posted earlier.
From: Ahmad Emneina <ae...@gmail.com>
To: Cloudstack users <cl...@incubator.apache.org>,
Date: 03/18/2013 04:16 PM
Subject: Re: System VMs won't start
At a glance, its seems the management server cant connect to 'host 3'.
Maybe a more full log might surface the exact issue. I'd start by trying
to
connect to the host in question from the management server.
On Mon, Mar 18, 2013 at 1:02 PM, <JS...@h2l.com> wrote:
> I know others have had problems with System VMs not starting, but my
logs
> appears to be different.
>
> I'm running CS 4.0.1 on a flat network with 2 hosts, a management
server,
> and a shared storage server. All on CentOS 6.x machines.
>
> After installation it shows me that the primary storage is up, and my
> secondary storage is enabled. But neither of the system vms (Console
> Proxy, or Secondary Storage) start successfully.
>
> Here's what the management log has to say repeatedly:
>
> 2013-03-18 14:55:20,609 WARN
> [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null)
> Exception while trying to start secondary storage vm
> com.cloud.exception.AgentUnavailableException: Resource [Host:3] is
> unreachable: Host 3: Unable to start instance due to Unable to get
answer
> that is of class com.cloud.agent.api.StartAnswer
> at
>
>
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:847)
> at
>
>
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:472)
> at
>
>
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:465)
> at
>
>
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:257)
> at
>
>
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:684)
> at
>
>
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1310)
> at
>
>
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:119)
> at
>
>
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
> at
> com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:106)
> at
> com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:34)
> at
>
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:83)
> at
> com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:73)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at
>
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
> at
> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
> at
>
>
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
> at
>
>
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
> at
>
>
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
> at
>
>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to
get
> answer that is of class com.cloud.agent.api.StartAnswer
> at com.cloud.agent.manager.Commands.getAnswer(Commands.java:80)
> at
>
>
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:783)
> ... 20 more
> 2013-03-18 14:55:20,610 INFO
> [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null)
Unable
> to start secondary storage vm for standby capacity, secStorageVm vm Id :
> 10157, will recycle it and start a new one
> 2013-03-18 14:55:20,610 INFO
> [cloud.secstorage.PremiumSecondaryStorageManagerImpl]
(secstorage-1:null)
> Primary secondary storage is not even started, wait until next turn
> 2013-03-18 14:55:22,497 DEBUG [cloud.server.StatsCollector]
> (StatsCollector-1:null) HostStatsCollector is running...
> 2013-03-18 14:55:23,142 DEBUG [agent.transport.Request]
> (StatsCollector-1:null) Seq 1-891290142: Received: { Ans: , MgmtId:
> 161336797896, via: 1, Ver: v1, Flags: 10, { GetHostStatsAnswer } }
> 2013-03-18 14:55:23,829 DEBUG [agent.transport.Request]
> (StatsCollector-1:null) Seq 3-773128182: Received: { Ans: , MgmtId:
> 161336797896, via: 3, Ver: v1, Flags: 10, { GetHostStatsAnswer } }
> 2013-03-18 14:55:29,320 DEBUG [cloud.server.StatsCollector]
> (StatsCollector-2:null) StorageCollector is running...
> 2013-03-18 14:55:29,324 DEBUG [cloud.server.StatsCollector]
> (StatsCollector-2:null) There is no secondary storage VM for secondary
> storage host nfs://192.168.100.239/Storage/secondary
>
> Any thoughts?
>
> Thanks,
> Jeremy
Re: System VMs won't start
Posted by Ahmad Emneina <ae...@gmail.com>.
At a glance, its seems the management server cant connect to 'host 3'.
Maybe a more full log might surface the exact issue. I'd start by trying to
connect to the host in question from the management server.
On Mon, Mar 18, 2013 at 1:02 PM, <JS...@h2l.com> wrote:
> I know others have had problems with System VMs not starting, but my logs
> appears to be different.
>
> I'm running CS 4.0.1 on a flat network with 2 hosts, a management server,
> and a shared storage server. All on CentOS 6.x machines.
>
> After installation it shows me that the primary storage is up, and my
> secondary storage is enabled. But neither of the system vms (Console
> Proxy, or Secondary Storage) start successfully.
>
> Here's what the management log has to say repeatedly:
>
> 2013-03-18 14:55:20,609 WARN
> [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null)
> Exception while trying to start secondary storage vm
> com.cloud.exception.AgentUnavailableException: Resource [Host:3] is
> unreachable: Host 3: Unable to start instance due to Unable to get answer
> that is of class com.cloud.agent.api.StartAnswer
> at
>
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:847)
> at
>
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:472)
> at
>
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:465)
> at
>
> com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:257)
> at
>
> com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:684)
> at
>
> com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1310)
> at
>
> com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:119)
> at
>
> com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
> at
> com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:106)
> at
> com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:34)
> at
> com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:83)
> at
> com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:73)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
> at
> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
> at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
> at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to get
> answer that is of class com.cloud.agent.api.StartAnswer
> at com.cloud.agent.manager.Commands.getAnswer(Commands.java:80)
> at
>
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:783)
> ... 20 more
> 2013-03-18 14:55:20,610 INFO
> [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) Unable
> to start secondary storage vm for standby capacity, secStorageVm vm Id :
> 10157, will recycle it and start a new one
> 2013-03-18 14:55:20,610 INFO
> [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:null)
> Primary secondary storage is not even started, wait until next turn
> 2013-03-18 14:55:22,497 DEBUG [cloud.server.StatsCollector]
> (StatsCollector-1:null) HostStatsCollector is running...
> 2013-03-18 14:55:23,142 DEBUG [agent.transport.Request]
> (StatsCollector-1:null) Seq 1-891290142: Received: { Ans: , MgmtId:
> 161336797896, via: 1, Ver: v1, Flags: 10, { GetHostStatsAnswer } }
> 2013-03-18 14:55:23,829 DEBUG [agent.transport.Request]
> (StatsCollector-1:null) Seq 3-773128182: Received: { Ans: , MgmtId:
> 161336797896, via: 3, Ver: v1, Flags: 10, { GetHostStatsAnswer } }
> 2013-03-18 14:55:29,320 DEBUG [cloud.server.StatsCollector]
> (StatsCollector-2:null) StorageCollector is running...
> 2013-03-18 14:55:29,324 DEBUG [cloud.server.StatsCollector]
> (StatsCollector-2:null) There is no secondary storage VM for secondary
> storage host nfs://192.168.100.239/Storage/secondary
>
> Any thoughts?
>
> Thanks,
> Jeremy