You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Andrei Mikhailovsky <an...@arhont.com> on 2014/08/09 19:29:36 UTC
HELP - unable to start systemvms on KVM after upgrade to 4.4 with
java 1.7 installed
Hello guys,
I've done an upgrade to ACS 4.4 from version 4.2.1 following the release notes and upgrade guide. I've also read several threads regarding issues with the systemvm template. I've downloaded the template from http://cloudstack.apt-get.eu/systemvm/4.4/systemvm64template-4.4.0-6-kvm.qcow2.bz2
I've also upgraded java to version 1.7 on host servers as well as the acs management server.
Following the guide, I've executed the script that restarts all system vms with the following output:
Stopping and starting 1 secondary storage vm(s)...
Done stopping and starting secondary storage vm(s)
Stopping and starting 1 console proxy vm(s)...
ERROR: Failed to start console proxy vm with id 847
Done stopping and starting console proxy vm(s) .
Stopping and starting 15 running routing vm(s)...
ERROR: Failed to restart domainRouter with id 864
ERROR: Failed to restart domainRouter with id 852
......
I've noticed and all of my system vms are Stopped and ACS is unable to start SSVM / CPVM and any of the virtual routers.
Looking into the issue i've noticed that the host servers have the following in agent.log file:
2014-08-09 18:00:12,093 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) Failed to perform a RBD action on the Ceph cluster, the error was: Failed to clone image 7ec0a208-6593-420c-9437-79279a5e3f03@cloudstack-base-snap to 566b003a-de59-419a-a1f6-7cf77cc95378
The management server log shows this error:
2014-08-09 18:12:32,524 WARN [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-7bb24cc7) Exception while trying to start secondary storage vm
java.lang.RuntimeException: Job failed due to exception Resource [Host:7] is unreachable: Host 7: Unable to start instance due to Unable to get answer that is of class com.cloud.agent.api.StartAnswer
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-08-09 18:12:32,525 INFO [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-7bb24cc7) Unable to start secondary storage vm for standby capacity, secStorageVm vm Id : 940, will recycle it and start a new one
This error keeps cycling every minute or so.
Could someone please help me with sorting out the system vms.
Many thanks
Andrei
Re: HELP - unable to start systemvms on KVM after upgrade to 4.4 with
java 1.7 installed
Posted by Daan Hoogland <da...@gmail.com>.
yes
On Sun, Aug 10, 2014 at 6:44 PM, Andrei Mikhailovsky <an...@arhont.com> wrote:
> Daan,
>
> sorry, my head is not thinking straight after spending second day on the upgrades issues. Is this the URL you are referring to:
>
>
> http://jenkins.buildacloud.org/view/4.4/job/cloudstack-4.4-systemvm64/lastSuccessfulBuild/artifact/tools/appliance/dist/systemvm64template-4.4-2014-08-10-kvm.qcow2.bz2
>
>
> Thanks
>
> Andrei
>
> ----- Original Message -----
> From: "Daan Hoogland" <da...@gmail.com>
> To: users@cloudstack.apache.org
> Sent: Sunday, 10 August, 2014 5:39:20 PM
> Subject: Re: HELP - unable to start systemvms on KVM after upgrade to 4.4 with java 1.7 installed
>
> Andrei,
>
> Try the sysvm templates VROM Jenkins.buildacloud.org. the 4.4 chapter
> contains a build for 64bit templates. These are reported as working. You
> are bot the only onze having problems with the released templates.
>
> biligual spelling checker used.read at your own risk
> Op 10 aug. 2014 17:19 schreef "Andrei Mikhailovsky" <an...@arhont.com>:
>
>> I've got another proble, still relating to the systemvms. This time it is:
>> https://issues.apache.org/jira/browse/CLOUDSTACK-6893
>>
>> Anyone have an idea on how to fix it on 4.4.0?
>>
>> As in my previous post, all my system vms are down - no connectivity to
>> the vms. I was hoping to fix this on 4.4.0 without the need to downgrade
>> for the 2nd time in 2 days.
>>
>> Thanks
>>
>> Andrei
>>
>> ----- Original Message -----
>> From: "Andrei Mikhailovsky" <an...@arhont.com>
>> To: users@cloudstack.apache.org
>> Sent: Saturday, 9 August, 2014 6:29:36 PM
>> Subject: HELP - unable to start systemvms on KVM after upgrade to 4.4 with
>> java 1.7 installed
>>
>> Hello guys,
>>
>> I've done an upgrade to ACS 4.4 from version 4.2.1 following the release
>> notes and upgrade guide. I've also read several threads regarding issues
>> with the systemvm template. I've downloaded the template from
>> http://cloudstack.apt-get.eu/systemvm/4.4/systemvm64template-4.4.0-6-kvm.qcow2.bz2
>>
>> I've also upgraded java to version 1.7 on host servers as well as the acs
>> management server.
>>
>> Following the guide, I've executed the script that restarts all system vms
>> with the following output:
>>
>>
>>
>> Stopping and starting 1 secondary storage vm(s)...
>>
>>
>>
>>
>> Done stopping and starting secondary storage vm(s)
>>
>>
>>
>>
>> Stopping and starting 1 console proxy vm(s)...
>>
>> ERROR: Failed to start console proxy vm with id 847
>>
>>
>>
>>
>> Done stopping and starting console proxy vm(s) .
>>
>>
>>
>>
>> Stopping and starting 15 running routing vm(s)...
>>
>> ERROR: Failed to restart domainRouter with id 864
>>
>>
>>
>>
>> ERROR: Failed to restart domainRouter with id 852
>>
>>
>>
>>
>> ......
>>
>>
>>
>>
>> I've noticed and all of my system vms are Stopped and ACS is unable to
>> start SSVM / CPVM and any of the virtual routers.
>>
>>
>>
>>
>> Looking into the issue i've noticed that the host servers have the
>> following in agent.log file:
>>
>>
>>
>>
>> 2014-08-09 18:00:12,093 ERROR [kvm.storage.LibvirtStorageAdaptor]
>> (agentRequest-Handler-4:null) Failed to perform a RBD action on the Ceph
>> cluster, the error was: Failed to clone image
>> 7ec0a208-6593-420c-9437-79279a5e3f03@cloudstack-base-snap to
>> 566b003a-de59-419a-a1f6-7cf77cc95378
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> The management server log shows this error:
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> 2014-08-09 18:12:32,524 WARN [o.a.c.s.SecondaryStorageManagerImpl]
>> (secstorage-1:ctx-7bb24cc7) Exception while trying to start secondary
>> storage vm
>>
>> java.lang.RuntimeException: Job failed due to exception Resource [Host:7]
>> is unreachable: Host 7: Unable to start instance due to Unable to get
>> answer that is of class com.cloud.agent.api.StartAnswer
>>
>> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114)
>>
>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
>>
>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>
>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>
>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
>>
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>
>> at java.lang.Thread.run(Thread.java:744)
>>
>> 2014-08-09 18:12:32,525 INFO [o.a.c.s.SecondaryStorageManagerImpl]
>> (secstorage-1:ctx-7bb24cc7) Unable to start secondary storage vm for
>> standby capacity, secStorageVm vm Id : 940, will recycle it and start a new
>> one
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> This error keeps cycling every minute or so.
>>
>>
>>
>>
>> Could someone please help me with sorting out the system vms.
>>
>>
>>
>>
>> Many thanks
>>
>>
>>
>>
>> Andrei
>>
--
Daan
Re: HELP - unable to start systemvms on KVM after upgrade to 4.4
with java 1.7 installed
Posted by Andrei Mikhailovsky <an...@arhont.com>.
Daan,
sorry, my head is not thinking straight after spending second day on the upgrades issues. Is this the URL you are referring to:
http://jenkins.buildacloud.org/view/4.4/job/cloudstack-4.4-systemvm64/lastSuccessfulBuild/artifact/tools/appliance/dist/systemvm64template-4.4-2014-08-10-kvm.qcow2.bz2
Thanks
Andrei
----- Original Message -----
From: "Daan Hoogland" <da...@gmail.com>
To: users@cloudstack.apache.org
Sent: Sunday, 10 August, 2014 5:39:20 PM
Subject: Re: HELP - unable to start systemvms on KVM after upgrade to 4.4 with java 1.7 installed
Andrei,
Try the sysvm templates VROM Jenkins.buildacloud.org. the 4.4 chapter
contains a build for 64bit templates. These are reported as working. You
are bot the only onze having problems with the released templates.
biligual spelling checker used.read at your own risk
Op 10 aug. 2014 17:19 schreef "Andrei Mikhailovsky" <an...@arhont.com>:
> I've got another proble, still relating to the systemvms. This time it is:
> https://issues.apache.org/jira/browse/CLOUDSTACK-6893
>
> Anyone have an idea on how to fix it on 4.4.0?
>
> As in my previous post, all my system vms are down - no connectivity to
> the vms. I was hoping to fix this on 4.4.0 without the need to downgrade
> for the 2nd time in 2 days.
>
> Thanks
>
> Andrei
>
> ----- Original Message -----
> From: "Andrei Mikhailovsky" <an...@arhont.com>
> To: users@cloudstack.apache.org
> Sent: Saturday, 9 August, 2014 6:29:36 PM
> Subject: HELP - unable to start systemvms on KVM after upgrade to 4.4 with
> java 1.7 installed
>
> Hello guys,
>
> I've done an upgrade to ACS 4.4 from version 4.2.1 following the release
> notes and upgrade guide. I've also read several threads regarding issues
> with the systemvm template. I've downloaded the template from
> http://cloudstack.apt-get.eu/systemvm/4.4/systemvm64template-4.4.0-6-kvm.qcow2.bz2
>
> I've also upgraded java to version 1.7 on host servers as well as the acs
> management server.
>
> Following the guide, I've executed the script that restarts all system vms
> with the following output:
>
>
>
> Stopping and starting 1 secondary storage vm(s)...
>
>
>
>
> Done stopping and starting secondary storage vm(s)
>
>
>
>
> Stopping and starting 1 console proxy vm(s)...
>
> ERROR: Failed to start console proxy vm with id 847
>
>
>
>
> Done stopping and starting console proxy vm(s) .
>
>
>
>
> Stopping and starting 15 running routing vm(s)...
>
> ERROR: Failed to restart domainRouter with id 864
>
>
>
>
> ERROR: Failed to restart domainRouter with id 852
>
>
>
>
> ......
>
>
>
>
> I've noticed and all of my system vms are Stopped and ACS is unable to
> start SSVM / CPVM and any of the virtual routers.
>
>
>
>
> Looking into the issue i've noticed that the host servers have the
> following in agent.log file:
>
>
>
>
> 2014-08-09 18:00:12,093 ERROR [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) Failed to perform a RBD action on the Ceph
> cluster, the error was: Failed to clone image
> 7ec0a208-6593-420c-9437-79279a5e3f03@cloudstack-base-snap to
> 566b003a-de59-419a-a1f6-7cf77cc95378
>
>
>
>
>
>
>
>
>
>
>
>
>
> The management server log shows this error:
>
>
>
>
>
>
>
>
>
>
> 2014-08-09 18:12:32,524 WARN [o.a.c.s.SecondaryStorageManagerImpl]
> (secstorage-1:ctx-7bb24cc7) Exception while trying to start secondary
> storage vm
>
> java.lang.RuntimeException: Job failed due to exception Resource [Host:7]
> is unreachable: Host 7: Unable to start instance due to Unable to get
> answer that is of class com.cloud.agent.api.StartAnswer
>
> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114)
>
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
>
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
>
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>
> at java.lang.Thread.run(Thread.java:744)
>
> 2014-08-09 18:12:32,525 INFO [o.a.c.s.SecondaryStorageManagerImpl]
> (secstorage-1:ctx-7bb24cc7) Unable to start secondary storage vm for
> standby capacity, secStorageVm vm Id : 940, will recycle it and start a new
> one
>
>
>
>
>
>
>
>
>
>
> This error keeps cycling every minute or so.
>
>
>
>
> Could someone please help me with sorting out the system vms.
>
>
>
>
> Many thanks
>
>
>
>
> Andrei
>
Re: HELP - unable to start systemvms on KVM after upgrade to 4.4
with java 1.7 installed
Posted by Andrei Mikhailovsky <an...@arhont.com>.
Daan,
Thanks for your reply. Would you happen to know the URL to download these templates? I did try the latest templates from: http://cloudstack.apt-get.eu/systemvm/4.4/systemvm64template-4.4.0-6-kvm.qcow2.bz2, but they are producing 6893 bug ((
Thanks
Andrei
----- Original Message -----
From: "Daan Hoogland" <da...@gmail.com>
To: users@cloudstack.apache.org
Sent: Sunday, 10 August, 2014 5:39:20 PM
Subject: Re: HELP - unable to start systemvms on KVM after upgrade to 4.4 with java 1.7 installed
Andrei,
Try the sysvm templates VROM Jenkins.buildacloud.org. the 4.4 chapter
contains a build for 64bit templates. These are reported as working. You
are bot the only onze having problems with the released templates.
biligual spelling checker used.read at your own risk
Op 10 aug. 2014 17:19 schreef "Andrei Mikhailovsky" <an...@arhont.com>:
> I've got another proble, still relating to the systemvms. This time it is:
> https://issues.apache.org/jira/browse/CLOUDSTACK-6893
>
> Anyone have an idea on how to fix it on 4.4.0?
>
> As in my previous post, all my system vms are down - no connectivity to
> the vms. I was hoping to fix this on 4.4.0 without the need to downgrade
> for the 2nd time in 2 days.
>
> Thanks
>
> Andrei
>
> ----- Original Message -----
> From: "Andrei Mikhailovsky" <an...@arhont.com>
> To: users@cloudstack.apache.org
> Sent: Saturday, 9 August, 2014 6:29:36 PM
> Subject: HELP - unable to start systemvms on KVM after upgrade to 4.4 with
> java 1.7 installed
>
> Hello guys,
>
> I've done an upgrade to ACS 4.4 from version 4.2.1 following the release
> notes and upgrade guide. I've also read several threads regarding issues
> with the systemvm template. I've downloaded the template from
> http://cloudstack.apt-get.eu/systemvm/4.4/systemvm64template-4.4.0-6-kvm.qcow2.bz2
>
> I've also upgraded java to version 1.7 on host servers as well as the acs
> management server.
>
> Following the guide, I've executed the script that restarts all system vms
> with the following output:
>
>
>
> Stopping and starting 1 secondary storage vm(s)...
>
>
>
>
> Done stopping and starting secondary storage vm(s)
>
>
>
>
> Stopping and starting 1 console proxy vm(s)...
>
> ERROR: Failed to start console proxy vm with id 847
>
>
>
>
> Done stopping and starting console proxy vm(s) .
>
>
>
>
> Stopping and starting 15 running routing vm(s)...
>
> ERROR: Failed to restart domainRouter with id 864
>
>
>
>
> ERROR: Failed to restart domainRouter with id 852
>
>
>
>
> ......
>
>
>
>
> I've noticed and all of my system vms are Stopped and ACS is unable to
> start SSVM / CPVM and any of the virtual routers.
>
>
>
>
> Looking into the issue i've noticed that the host servers have the
> following in agent.log file:
>
>
>
>
> 2014-08-09 18:00:12,093 ERROR [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) Failed to perform a RBD action on the Ceph
> cluster, the error was: Failed to clone image
> 7ec0a208-6593-420c-9437-79279a5e3f03@cloudstack-base-snap to
> 566b003a-de59-419a-a1f6-7cf77cc95378
>
>
>
>
>
>
>
>
>
>
>
>
>
> The management server log shows this error:
>
>
>
>
>
>
>
>
>
>
> 2014-08-09 18:12:32,524 WARN [o.a.c.s.SecondaryStorageManagerImpl]
> (secstorage-1:ctx-7bb24cc7) Exception while trying to start secondary
> storage vm
>
> java.lang.RuntimeException: Job failed due to exception Resource [Host:7]
> is unreachable: Host 7: Unable to start instance due to Unable to get
> answer that is of class com.cloud.agent.api.StartAnswer
>
> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114)
>
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
>
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
>
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>
> at java.lang.Thread.run(Thread.java:744)
>
> 2014-08-09 18:12:32,525 INFO [o.a.c.s.SecondaryStorageManagerImpl]
> (secstorage-1:ctx-7bb24cc7) Unable to start secondary storage vm for
> standby capacity, secStorageVm vm Id : 940, will recycle it and start a new
> one
>
>
>
>
>
>
>
>
>
>
> This error keeps cycling every minute or so.
>
>
>
>
> Could someone please help me with sorting out the system vms.
>
>
>
>
> Many thanks
>
>
>
>
> Andrei
>
Re: HELP - unable to start systemvms on KVM after upgrade to 4.4 with
java 1.7 installed
Posted by "Mitch.Fiorentini" <mi...@gmail.com>.
Thanks Erik: But, I'm running Ubuntu. Can I just remove CS and re-install
4.3?
On Sat, Sep 27, 2014 at 3:26 PM, Erik Weber <te...@gmail.com> wrote:
> You may give these rpms a try:
> http://jenkins.huug.no:8080/job/cloudstack-4.4.0-with-fix-noredist-rpm/
> That's 4.4.0 + the patch.
>
> Or you can wait for 4.4.1
>
> --
> Erik
>
> On Sat, Sep 27, 2014 at 9:07 PM, Mitch.Fiorentini <
> mitch.fiorentini@gmail.com> wrote:
>
> > Hello: Looks like I'm running into this after a 4.3 to 4.4 upgrade.
> Whats
> > the fix / recommend course of action for this?
> >
> > On Sun, Aug 10, 2014 at 3:10 PM, Erik Weber <te...@gmail.com> wrote:
> >
> > > That's 6893, fixed in 4.4.1.
> > >
> > > Erik
> > > 10. aug. 2014 19:35 skrev "Andrei Mikhailovsky" <an...@arhont.com>
> > > følgende:
> > >
> > > > Daan,
> > > >
> > > > Just tried the template from Jenkins as per your suggestion and that
> > > > didn't help. Still seeing:
> > > >
> > > > 2014-08-10 18:33:06,538 DEBUG [c.c.a.t.Request]
> > > > (AgentManager-Handler-9:null) Seq 8-191684459139956780: Processing:
> {
> > > Ans:
> > > > , MgmtId: 115129173025118, via: 8, Ver: v1, Flags: 10,
> > > > [{"com.cloud.agent.api.Answer":{"result":false,"details":"No enum
> > > constant
> > > >
> > >
> >
> com.cloud.hypervisor.kvm.resource.LibvirtVMDef.DiskDef.diskCacheMode.none","wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped
> > > > by previous failure","wait":0}}] }
> > > >
> > > >
> > > >
> > > > Any other ideas?
> > > >
> > > > Thanks
> > > >
> > > >
> > > > ----- Original Message -----
> > > > From: "Daan Hoogland" <da...@gmail.com>
> > > > To: users@cloudstack.apache.org
> > > > Sent: Sunday, 10 August, 2014 5:39:20 PM
> > > > Subject: Re: HELP - unable to start systemvms on KVM after upgrade to
> > 4.4
> > > > with java 1.7 installed
> > > >
> > > > Andrei,
> > > >
> > > > Try the sysvm templates VROM Jenkins.buildacloud.org. the 4.4
> chapter
> > > > contains a build for 64bit templates. These are reported as working.
> > You
> > > > are bot the only onze having problems with the released templates.
> > > >
> > > > biligual spelling checker used.read at your own risk
> > > > Op 10 aug. 2014 17:19 schreef "Andrei Mikhailovsky" <
> andrei@arhont.com
> > >:
> > > >
> > > > > I've got another proble, still relating to the systemvms. This time
> > it
> > > > is:
> > > > > https://issues.apache.org/jira/browse/CLOUDSTACK-6893
> > > > >
> > > > > Anyone have an idea on how to fix it on 4.4.0?
> > > > >
> > > > > As in my previous post, all my system vms are down - no
> connectivity
> > to
> > > > > the vms. I was hoping to fix this on 4.4.0 without the need to
> > > downgrade
> > > > > for the 2nd time in 2 days.
> > > > >
> > > > > Thanks
> > > > >
> > > > > Andrei
> > > > >
> > > > > ----- Original Message -----
> > > > > From: "Andrei Mikhailovsky" <an...@arhont.com>
> > > > > To: users@cloudstack.apache.org
> > > > > Sent: Saturday, 9 August, 2014 6:29:36 PM
> > > > > Subject: HELP - unable to start systemvms on KVM after upgrade to
> 4.4
> > > > with
> > > > > java 1.7 installed
> > > > >
> > > > > Hello guys,
> > > > >
> > > > > I've done an upgrade to ACS 4.4 from version 4.2.1 following the
> > > release
> > > > > notes and upgrade guide. I've also read several threads regarding
> > > issues
> > > > > with the systemvm template. I've downloaded the template from
> > > > >
> > > >
> > >
> >
> http://cloudstack.apt-get.eu/systemvm/4.4/systemvm64template-4.4.0-6-kvm.qcow2.bz2
> > > > >
> > > > > I've also upgraded java to version 1.7 on host servers as well as
> the
> > > acs
> > > > > management server.
> > > > >
> > > > > Following the guide, I've executed the script that restarts all
> > system
> > > > vms
> > > > > with the following output:
> > > > >
> > > > >
> > > > >
> > > > > Stopping and starting 1 secondary storage vm(s)...
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Done stopping and starting secondary storage vm(s)
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Stopping and starting 1 console proxy vm(s)...
> > > > >
> > > > > ERROR: Failed to start console proxy vm with id 847
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Done stopping and starting console proxy vm(s) .
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Stopping and starting 15 running routing vm(s)...
> > > > >
> > > > > ERROR: Failed to restart domainRouter with id 864
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > ERROR: Failed to restart domainRouter with id 852
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > ......
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > I've noticed and all of my system vms are Stopped and ACS is unable
> > to
> > > > > start SSVM / CPVM and any of the virtual routers.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Looking into the issue i've noticed that the host servers have the
> > > > > following in agent.log file:
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > 2014-08-09 18:00:12,093 ERROR [kvm.storage.LibvirtStorageAdaptor]
> > > > > (agentRequest-Handler-4:null) Failed to perform a RBD action on the
> > > Ceph
> > > > > cluster, the error was: Failed to clone image
> > > > > 7ec0a208-6593-420c-9437-79279a5e3f03@cloudstack-base-snap to
> > > > > 566b003a-de59-419a-a1f6-7cf77cc95378
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > The management server log shows this error:
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > 2014-08-09 18:12:32,524 WARN [o.a.c.s.SecondaryStorageManagerImpl]
> > > > > (secstorage-1:ctx-7bb24cc7) Exception while trying to start
> secondary
> > > > > storage vm
> > > > >
> > > > > java.lang.RuntimeException: Job failed due to exception Resource
> > > [Host:7]
> > > > > is unreachable: Host 7: Unable to start instance due to Unable to
> get
> > > > > answer that is of class com.cloud.agent.api.StartAnswer
> > > > >
> > > > > at
> > > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114)
> > > > >
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
> > > > >
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> > > > >
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> > > > >
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> > > > >
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> > > > >
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> > > > >
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
> > > > >
> > > > > at
> > > >
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > > > >
> > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> > > > >
> > > > > at
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > > > >
> > > > > at
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > > > >
> > > > > at java.lang.Thread.run(Thread.java:744)
> > > > >
> > > > > 2014-08-09 18:12:32,525 INFO [o.a.c.s.SecondaryStorageManagerImpl]
> > > > > (secstorage-1:ctx-7bb24cc7) Unable to start secondary storage vm
> for
> > > > > standby capacity, secStorageVm vm Id : 940, will recycle it and
> > start a
> > > > new
> > > > > one
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > This error keeps cycling every minute or so.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Could someone please help me with sorting out the system vms.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Many thanks
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Andrei
> > > > >
> > > >
> > >
> >
>
Re: HELP - unable to start systemvms on KVM after upgrade to 4.4 with
java 1.7 installed
Posted by Erik Weber <te...@gmail.com>.
You may give these rpms a try:
http://jenkins.huug.no:8080/job/cloudstack-4.4.0-with-fix-noredist-rpm/
That's 4.4.0 + the patch.
Or you can wait for 4.4.1
--
Erik
On Sat, Sep 27, 2014 at 9:07 PM, Mitch.Fiorentini <
mitch.fiorentini@gmail.com> wrote:
> Hello: Looks like I'm running into this after a 4.3 to 4.4 upgrade. Whats
> the fix / recommend course of action for this?
>
> On Sun, Aug 10, 2014 at 3:10 PM, Erik Weber <te...@gmail.com> wrote:
>
> > That's 6893, fixed in 4.4.1.
> >
> > Erik
> > 10. aug. 2014 19:35 skrev "Andrei Mikhailovsky" <an...@arhont.com>
> > følgende:
> >
> > > Daan,
> > >
> > > Just tried the template from Jenkins as per your suggestion and that
> > > didn't help. Still seeing:
> > >
> > > 2014-08-10 18:33:06,538 DEBUG [c.c.a.t.Request]
> > > (AgentManager-Handler-9:null) Seq 8-191684459139956780: Processing: {
> > Ans:
> > > , MgmtId: 115129173025118, via: 8, Ver: v1, Flags: 10,
> > > [{"com.cloud.agent.api.Answer":{"result":false,"details":"No enum
> > constant
> > >
> >
> com.cloud.hypervisor.kvm.resource.LibvirtVMDef.DiskDef.diskCacheMode.none","wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped
> > > by previous failure","wait":0}}] }
> > >
> > >
> > >
> > > Any other ideas?
> > >
> > > Thanks
> > >
> > >
> > > ----- Original Message -----
> > > From: "Daan Hoogland" <da...@gmail.com>
> > > To: users@cloudstack.apache.org
> > > Sent: Sunday, 10 August, 2014 5:39:20 PM
> > > Subject: Re: HELP - unable to start systemvms on KVM after upgrade to
> 4.4
> > > with java 1.7 installed
> > >
> > > Andrei,
> > >
> > > Try the sysvm templates VROM Jenkins.buildacloud.org. the 4.4 chapter
> > > contains a build for 64bit templates. These are reported as working.
> You
> > > are bot the only onze having problems with the released templates.
> > >
> > > biligual spelling checker used.read at your own risk
> > > Op 10 aug. 2014 17:19 schreef "Andrei Mikhailovsky" <andrei@arhont.com
> >:
> > >
> > > > I've got another proble, still relating to the systemvms. This time
> it
> > > is:
> > > > https://issues.apache.org/jira/browse/CLOUDSTACK-6893
> > > >
> > > > Anyone have an idea on how to fix it on 4.4.0?
> > > >
> > > > As in my previous post, all my system vms are down - no connectivity
> to
> > > > the vms. I was hoping to fix this on 4.4.0 without the need to
> > downgrade
> > > > for the 2nd time in 2 days.
> > > >
> > > > Thanks
> > > >
> > > > Andrei
> > > >
> > > > ----- Original Message -----
> > > > From: "Andrei Mikhailovsky" <an...@arhont.com>
> > > > To: users@cloudstack.apache.org
> > > > Sent: Saturday, 9 August, 2014 6:29:36 PM
> > > > Subject: HELP - unable to start systemvms on KVM after upgrade to 4.4
> > > with
> > > > java 1.7 installed
> > > >
> > > > Hello guys,
> > > >
> > > > I've done an upgrade to ACS 4.4 from version 4.2.1 following the
> > release
> > > > notes and upgrade guide. I've also read several threads regarding
> > issues
> > > > with the systemvm template. I've downloaded the template from
> > > >
> > >
> >
> http://cloudstack.apt-get.eu/systemvm/4.4/systemvm64template-4.4.0-6-kvm.qcow2.bz2
> > > >
> > > > I've also upgraded java to version 1.7 on host servers as well as the
> > acs
> > > > management server.
> > > >
> > > > Following the guide, I've executed the script that restarts all
> system
> > > vms
> > > > with the following output:
> > > >
> > > >
> > > >
> > > > Stopping and starting 1 secondary storage vm(s)...
> > > >
> > > >
> > > >
> > > >
> > > > Done stopping and starting secondary storage vm(s)
> > > >
> > > >
> > > >
> > > >
> > > > Stopping and starting 1 console proxy vm(s)...
> > > >
> > > > ERROR: Failed to start console proxy vm with id 847
> > > >
> > > >
> > > >
> > > >
> > > > Done stopping and starting console proxy vm(s) .
> > > >
> > > >
> > > >
> > > >
> > > > Stopping and starting 15 running routing vm(s)...
> > > >
> > > > ERROR: Failed to restart domainRouter with id 864
> > > >
> > > >
> > > >
> > > >
> > > > ERROR: Failed to restart domainRouter with id 852
> > > >
> > > >
> > > >
> > > >
> > > > ......
> > > >
> > > >
> > > >
> > > >
> > > > I've noticed and all of my system vms are Stopped and ACS is unable
> to
> > > > start SSVM / CPVM and any of the virtual routers.
> > > >
> > > >
> > > >
> > > >
> > > > Looking into the issue i've noticed that the host servers have the
> > > > following in agent.log file:
> > > >
> > > >
> > > >
> > > >
> > > > 2014-08-09 18:00:12,093 ERROR [kvm.storage.LibvirtStorageAdaptor]
> > > > (agentRequest-Handler-4:null) Failed to perform a RBD action on the
> > Ceph
> > > > cluster, the error was: Failed to clone image
> > > > 7ec0a208-6593-420c-9437-79279a5e3f03@cloudstack-base-snap to
> > > > 566b003a-de59-419a-a1f6-7cf77cc95378
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > The management server log shows this error:
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > 2014-08-09 18:12:32,524 WARN [o.a.c.s.SecondaryStorageManagerImpl]
> > > > (secstorage-1:ctx-7bb24cc7) Exception while trying to start secondary
> > > > storage vm
> > > >
> > > > java.lang.RuntimeException: Job failed due to exception Resource
> > [Host:7]
> > > > is unreachable: Host 7: Unable to start instance due to Unable to get
> > > > answer that is of class com.cloud.agent.api.StartAnswer
> > > >
> > > > at
> > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114)
> > > >
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
> > > >
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> > > >
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> > > >
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> > > >
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> > > >
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> > > >
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
> > > >
> > > > at
> > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > > >
> > > > at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> > > >
> > > > at
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > > >
> > > > at
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > > >
> > > > at java.lang.Thread.run(Thread.java:744)
> > > >
> > > > 2014-08-09 18:12:32,525 INFO [o.a.c.s.SecondaryStorageManagerImpl]
> > > > (secstorage-1:ctx-7bb24cc7) Unable to start secondary storage vm for
> > > > standby capacity, secStorageVm vm Id : 940, will recycle it and
> start a
> > > new
> > > > one
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > This error keeps cycling every minute or so.
> > > >
> > > >
> > > >
> > > >
> > > > Could someone please help me with sorting out the system vms.
> > > >
> > > >
> > > >
> > > >
> > > > Many thanks
> > > >
> > > >
> > > >
> > > >
> > > > Andrei
> > > >
> > >
> >
>
Re: HELP - unable to start systemvms on KVM after upgrade to 4.4 with
java 1.7 installed
Posted by "Mitch.Fiorentini" <mi...@gmail.com>.
Hello: Looks like I'm running into this after a 4.3 to 4.4 upgrade. Whats
the fix / recommend course of action for this?
On Sun, Aug 10, 2014 at 3:10 PM, Erik Weber <te...@gmail.com> wrote:
> That's 6893, fixed in 4.4.1.
>
> Erik
> 10. aug. 2014 19:35 skrev "Andrei Mikhailovsky" <an...@arhont.com>
> følgende:
>
> > Daan,
> >
> > Just tried the template from Jenkins as per your suggestion and that
> > didn't help. Still seeing:
> >
> > 2014-08-10 18:33:06,538 DEBUG [c.c.a.t.Request]
> > (AgentManager-Handler-9:null) Seq 8-191684459139956780: Processing: {
> Ans:
> > , MgmtId: 115129173025118, via: 8, Ver: v1, Flags: 10,
> > [{"com.cloud.agent.api.Answer":{"result":false,"details":"No enum
> constant
> >
> com.cloud.hypervisor.kvm.resource.LibvirtVMDef.DiskDef.diskCacheMode.none","wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped
> > by previous failure","wait":0}}] }
> >
> >
> >
> > Any other ideas?
> >
> > Thanks
> >
> >
> > ----- Original Message -----
> > From: "Daan Hoogland" <da...@gmail.com>
> > To: users@cloudstack.apache.org
> > Sent: Sunday, 10 August, 2014 5:39:20 PM
> > Subject: Re: HELP - unable to start systemvms on KVM after upgrade to 4.4
> > with java 1.7 installed
> >
> > Andrei,
> >
> > Try the sysvm templates VROM Jenkins.buildacloud.org. the 4.4 chapter
> > contains a build for 64bit templates. These are reported as working. You
> > are bot the only onze having problems with the released templates.
> >
> > biligual spelling checker used.read at your own risk
> > Op 10 aug. 2014 17:19 schreef "Andrei Mikhailovsky" <an...@arhont.com>:
> >
> > > I've got another proble, still relating to the systemvms. This time it
> > is:
> > > https://issues.apache.org/jira/browse/CLOUDSTACK-6893
> > >
> > > Anyone have an idea on how to fix it on 4.4.0?
> > >
> > > As in my previous post, all my system vms are down - no connectivity to
> > > the vms. I was hoping to fix this on 4.4.0 without the need to
> downgrade
> > > for the 2nd time in 2 days.
> > >
> > > Thanks
> > >
> > > Andrei
> > >
> > > ----- Original Message -----
> > > From: "Andrei Mikhailovsky" <an...@arhont.com>
> > > To: users@cloudstack.apache.org
> > > Sent: Saturday, 9 August, 2014 6:29:36 PM
> > > Subject: HELP - unable to start systemvms on KVM after upgrade to 4.4
> > with
> > > java 1.7 installed
> > >
> > > Hello guys,
> > >
> > > I've done an upgrade to ACS 4.4 from version 4.2.1 following the
> release
> > > notes and upgrade guide. I've also read several threads regarding
> issues
> > > with the systemvm template. I've downloaded the template from
> > >
> >
> http://cloudstack.apt-get.eu/systemvm/4.4/systemvm64template-4.4.0-6-kvm.qcow2.bz2
> > >
> > > I've also upgraded java to version 1.7 on host servers as well as the
> acs
> > > management server.
> > >
> > > Following the guide, I've executed the script that restarts all system
> > vms
> > > with the following output:
> > >
> > >
> > >
> > > Stopping and starting 1 secondary storage vm(s)...
> > >
> > >
> > >
> > >
> > > Done stopping and starting secondary storage vm(s)
> > >
> > >
> > >
> > >
> > > Stopping and starting 1 console proxy vm(s)...
> > >
> > > ERROR: Failed to start console proxy vm with id 847
> > >
> > >
> > >
> > >
> > > Done stopping and starting console proxy vm(s) .
> > >
> > >
> > >
> > >
> > > Stopping and starting 15 running routing vm(s)...
> > >
> > > ERROR: Failed to restart domainRouter with id 864
> > >
> > >
> > >
> > >
> > > ERROR: Failed to restart domainRouter with id 852
> > >
> > >
> > >
> > >
> > > ......
> > >
> > >
> > >
> > >
> > > I've noticed and all of my system vms are Stopped and ACS is unable to
> > > start SSVM / CPVM and any of the virtual routers.
> > >
> > >
> > >
> > >
> > > Looking into the issue i've noticed that the host servers have the
> > > following in agent.log file:
> > >
> > >
> > >
> > >
> > > 2014-08-09 18:00:12,093 ERROR [kvm.storage.LibvirtStorageAdaptor]
> > > (agentRequest-Handler-4:null) Failed to perform a RBD action on the
> Ceph
> > > cluster, the error was: Failed to clone image
> > > 7ec0a208-6593-420c-9437-79279a5e3f03@cloudstack-base-snap to
> > > 566b003a-de59-419a-a1f6-7cf77cc95378
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > The management server log shows this error:
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > 2014-08-09 18:12:32,524 WARN [o.a.c.s.SecondaryStorageManagerImpl]
> > > (secstorage-1:ctx-7bb24cc7) Exception while trying to start secondary
> > > storage vm
> > >
> > > java.lang.RuntimeException: Job failed due to exception Resource
> [Host:7]
> > > is unreachable: Host 7: Unable to start instance due to Unable to get
> > > answer that is of class com.cloud.agent.api.StartAnswer
> > >
> > > at
> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114)
> > >
> > > at
> > >
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
> > >
> > > at
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> > >
> > > at
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> > >
> > > at
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> > >
> > > at
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> > >
> > > at
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> > >
> > > at
> > >
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
> > >
> > > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > >
> > > at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> > >
> > > at
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > >
> > > at
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > >
> > > at java.lang.Thread.run(Thread.java:744)
> > >
> > > 2014-08-09 18:12:32,525 INFO [o.a.c.s.SecondaryStorageManagerImpl]
> > > (secstorage-1:ctx-7bb24cc7) Unable to start secondary storage vm for
> > > standby capacity, secStorageVm vm Id : 940, will recycle it and start a
> > new
> > > one
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > This error keeps cycling every minute or so.
> > >
> > >
> > >
> > >
> > > Could someone please help me with sorting out the system vms.
> > >
> > >
> > >
> > >
> > > Many thanks
> > >
> > >
> > >
> > >
> > > Andrei
> > >
> >
>
Re: HELP - unable to start systemvms on KVM after upgrade to 4.4 with
java 1.7 installed
Posted by Erik Weber <te...@gmail.com>.
That's 6893, fixed in 4.4.1.
Erik
10. aug. 2014 19:35 skrev "Andrei Mikhailovsky" <an...@arhont.com>
følgende:
> Daan,
>
> Just tried the template from Jenkins as per your suggestion and that
> didn't help. Still seeing:
>
> 2014-08-10 18:33:06,538 DEBUG [c.c.a.t.Request]
> (AgentManager-Handler-9:null) Seq 8-191684459139956780: Processing: { Ans:
> , MgmtId: 115129173025118, via: 8, Ver: v1, Flags: 10,
> [{"com.cloud.agent.api.Answer":{"result":false,"details":"No enum constant
> com.cloud.hypervisor.kvm.resource.LibvirtVMDef.DiskDef.diskCacheMode.none","wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped
> by previous failure","wait":0}}] }
>
>
>
> Any other ideas?
>
> Thanks
>
>
> ----- Original Message -----
> From: "Daan Hoogland" <da...@gmail.com>
> To: users@cloudstack.apache.org
> Sent: Sunday, 10 August, 2014 5:39:20 PM
> Subject: Re: HELP - unable to start systemvms on KVM after upgrade to 4.4
> with java 1.7 installed
>
> Andrei,
>
> Try the sysvm templates VROM Jenkins.buildacloud.org. the 4.4 chapter
> contains a build for 64bit templates. These are reported as working. You
> are bot the only onze having problems with the released templates.
>
> biligual spelling checker used.read at your own risk
> Op 10 aug. 2014 17:19 schreef "Andrei Mikhailovsky" <an...@arhont.com>:
>
> > I've got another proble, still relating to the systemvms. This time it
> is:
> > https://issues.apache.org/jira/browse/CLOUDSTACK-6893
> >
> > Anyone have an idea on how to fix it on 4.4.0?
> >
> > As in my previous post, all my system vms are down - no connectivity to
> > the vms. I was hoping to fix this on 4.4.0 without the need to downgrade
> > for the 2nd time in 2 days.
> >
> > Thanks
> >
> > Andrei
> >
> > ----- Original Message -----
> > From: "Andrei Mikhailovsky" <an...@arhont.com>
> > To: users@cloudstack.apache.org
> > Sent: Saturday, 9 August, 2014 6:29:36 PM
> > Subject: HELP - unable to start systemvms on KVM after upgrade to 4.4
> with
> > java 1.7 installed
> >
> > Hello guys,
> >
> > I've done an upgrade to ACS 4.4 from version 4.2.1 following the release
> > notes and upgrade guide. I've also read several threads regarding issues
> > with the systemvm template. I've downloaded the template from
> >
> http://cloudstack.apt-get.eu/systemvm/4.4/systemvm64template-4.4.0-6-kvm.qcow2.bz2
> >
> > I've also upgraded java to version 1.7 on host servers as well as the acs
> > management server.
> >
> > Following the guide, I've executed the script that restarts all system
> vms
> > with the following output:
> >
> >
> >
> > Stopping and starting 1 secondary storage vm(s)...
> >
> >
> >
> >
> > Done stopping and starting secondary storage vm(s)
> >
> >
> >
> >
> > Stopping and starting 1 console proxy vm(s)...
> >
> > ERROR: Failed to start console proxy vm with id 847
> >
> >
> >
> >
> > Done stopping and starting console proxy vm(s) .
> >
> >
> >
> >
> > Stopping and starting 15 running routing vm(s)...
> >
> > ERROR: Failed to restart domainRouter with id 864
> >
> >
> >
> >
> > ERROR: Failed to restart domainRouter with id 852
> >
> >
> >
> >
> > ......
> >
> >
> >
> >
> > I've noticed and all of my system vms are Stopped and ACS is unable to
> > start SSVM / CPVM and any of the virtual routers.
> >
> >
> >
> >
> > Looking into the issue i've noticed that the host servers have the
> > following in agent.log file:
> >
> >
> >
> >
> > 2014-08-09 18:00:12,093 ERROR [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-4:null) Failed to perform a RBD action on the Ceph
> > cluster, the error was: Failed to clone image
> > 7ec0a208-6593-420c-9437-79279a5e3f03@cloudstack-base-snap to
> > 566b003a-de59-419a-a1f6-7cf77cc95378
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > The management server log shows this error:
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > 2014-08-09 18:12:32,524 WARN [o.a.c.s.SecondaryStorageManagerImpl]
> > (secstorage-1:ctx-7bb24cc7) Exception while trying to start secondary
> > storage vm
> >
> > java.lang.RuntimeException: Job failed due to exception Resource [Host:7]
> > is unreachable: Host 7: Unable to start instance due to Unable to get
> > answer that is of class com.cloud.agent.api.StartAnswer
> >
> > at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114)
> >
> > at
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
> >
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> >
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> >
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> >
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> >
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> >
> > at
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
> >
> > at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> >
> > at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> >
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >
> > at java.lang.Thread.run(Thread.java:744)
> >
> > 2014-08-09 18:12:32,525 INFO [o.a.c.s.SecondaryStorageManagerImpl]
> > (secstorage-1:ctx-7bb24cc7) Unable to start secondary storage vm for
> > standby capacity, secStorageVm vm Id : 940, will recycle it and start a
> new
> > one
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > This error keeps cycling every minute or so.
> >
> >
> >
> >
> > Could someone please help me with sorting out the system vms.
> >
> >
> >
> >
> > Many thanks
> >
> >
> >
> >
> > Andrei
> >
>
Re: HELP - unable to start systemvms on KVM after upgrade to 4.4 with
java 1.7 installed
Posted by Daan Hoogland <da...@gmail.com>.
It sounds like a genuine kvm-plugin bug. Have a look if you recognise
it in jira or add a ticket. I can't really tell whether it is code or
configuration.
On Sun, Aug 10, 2014 at 7:35 PM, Andrei Mikhailovsky <an...@arhont.com> wrote:
> Daan,
>
> Just tried the template from Jenkins as per your suggestion and that didn't help. Still seeing:
>
> 2014-08-10 18:33:06,538 DEBUG [c.c.a.t.Request] (AgentManager-Handler-9:null) Seq 8-191684459139956780: Processing: { Ans: , MgmtId: 115129173025118, via: 8, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":false,"details":"No enum constant com.cloud.hypervisor.kvm.resource.LibvirtVMDef.DiskDef.diskCacheMode.none","wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous failure","wait":0}}] }
>
>
>
> Any other ideas?
>
> Thanks
>
>
> ----- Original Message -----
> From: "Daan Hoogland" <da...@gmail.com>
> To: users@cloudstack.apache.org
> Sent: Sunday, 10 August, 2014 5:39:20 PM
> Subject: Re: HELP - unable to start systemvms on KVM after upgrade to 4.4 with java 1.7 installed
>
> Andrei,
>
> Try the sysvm templates VROM Jenkins.buildacloud.org. the 4.4 chapter
> contains a build for 64bit templates. These are reported as working. You
> are bot the only onze having problems with the released templates.
>
> biligual spelling checker used.read at your own risk
> Op 10 aug. 2014 17:19 schreef "Andrei Mikhailovsky" <an...@arhont.com>:
>
>> I've got another proble, still relating to the systemvms. This time it is:
>> https://issues.apache.org/jira/browse/CLOUDSTACK-6893
>>
>> Anyone have an idea on how to fix it on 4.4.0?
>>
>> As in my previous post, all my system vms are down - no connectivity to
>> the vms. I was hoping to fix this on 4.4.0 without the need to downgrade
>> for the 2nd time in 2 days.
>>
>> Thanks
>>
>> Andrei
>>
>> ----- Original Message -----
>> From: "Andrei Mikhailovsky" <an...@arhont.com>
>> To: users@cloudstack.apache.org
>> Sent: Saturday, 9 August, 2014 6:29:36 PM
>> Subject: HELP - unable to start systemvms on KVM after upgrade to 4.4 with
>> java 1.7 installed
>>
>> Hello guys,
>>
>> I've done an upgrade to ACS 4.4 from version 4.2.1 following the release
>> notes and upgrade guide. I've also read several threads regarding issues
>> with the systemvm template. I've downloaded the template from
>> http://cloudstack.apt-get.eu/systemvm/4.4/systemvm64template-4.4.0-6-kvm.qcow2.bz2
>>
>> I've also upgraded java to version 1.7 on host servers as well as the acs
>> management server.
>>
>> Following the guide, I've executed the script that restarts all system vms
>> with the following output:
>>
>>
>>
>> Stopping and starting 1 secondary storage vm(s)...
>>
>>
>>
>>
>> Done stopping and starting secondary storage vm(s)
>>
>>
>>
>>
>> Stopping and starting 1 console proxy vm(s)...
>>
>> ERROR: Failed to start console proxy vm with id 847
>>
>>
>>
>>
>> Done stopping and starting console proxy vm(s) .
>>
>>
>>
>>
>> Stopping and starting 15 running routing vm(s)...
>>
>> ERROR: Failed to restart domainRouter with id 864
>>
>>
>>
>>
>> ERROR: Failed to restart domainRouter with id 852
>>
>>
>>
>>
>> ......
>>
>>
>>
>>
>> I've noticed and all of my system vms are Stopped and ACS is unable to
>> start SSVM / CPVM and any of the virtual routers.
>>
>>
>>
>>
>> Looking into the issue i've noticed that the host servers have the
>> following in agent.log file:
>>
>>
>>
>>
>> 2014-08-09 18:00:12,093 ERROR [kvm.storage.LibvirtStorageAdaptor]
>> (agentRequest-Handler-4:null) Failed to perform a RBD action on the Ceph
>> cluster, the error was: Failed to clone image
>> 7ec0a208-6593-420c-9437-79279a5e3f03@cloudstack-base-snap to
>> 566b003a-de59-419a-a1f6-7cf77cc95378
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> The management server log shows this error:
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> 2014-08-09 18:12:32,524 WARN [o.a.c.s.SecondaryStorageManagerImpl]
>> (secstorage-1:ctx-7bb24cc7) Exception while trying to start secondary
>> storage vm
>>
>> java.lang.RuntimeException: Job failed due to exception Resource [Host:7]
>> is unreachable: Host 7: Unable to start instance due to Unable to get
>> answer that is of class com.cloud.agent.api.StartAnswer
>>
>> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114)
>>
>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
>>
>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>
>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>
>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
>>
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>
>> at java.lang.Thread.run(Thread.java:744)
>>
>> 2014-08-09 18:12:32,525 INFO [o.a.c.s.SecondaryStorageManagerImpl]
>> (secstorage-1:ctx-7bb24cc7) Unable to start secondary storage vm for
>> standby capacity, secStorageVm vm Id : 940, will recycle it and start a new
>> one
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> This error keeps cycling every minute or so.
>>
>>
>>
>>
>> Could someone please help me with sorting out the system vms.
>>
>>
>>
>>
>> Many thanks
>>
>>
>>
>>
>> Andrei
>>
--
Daan
Re: HELP - unable to start systemvms on KVM after upgrade to 4.4
with java 1.7 installed
Posted by Andrei Mikhailovsky <an...@arhont.com>.
Daan,
Just tried the template from Jenkins as per your suggestion and that didn't help. Still seeing:
2014-08-10 18:33:06,538 DEBUG [c.c.a.t.Request] (AgentManager-Handler-9:null) Seq 8-191684459139956780: Processing: { Ans: , MgmtId: 115129173025118, via: 8, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":false,"details":"No enum constant com.cloud.hypervisor.kvm.resource.LibvirtVMDef.DiskDef.diskCacheMode.none","wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous failure","wait":0}}] }
Any other ideas?
Thanks
----- Original Message -----
From: "Daan Hoogland" <da...@gmail.com>
To: users@cloudstack.apache.org
Sent: Sunday, 10 August, 2014 5:39:20 PM
Subject: Re: HELP - unable to start systemvms on KVM after upgrade to 4.4 with java 1.7 installed
Andrei,
Try the sysvm templates VROM Jenkins.buildacloud.org. the 4.4 chapter
contains a build for 64bit templates. These are reported as working. You
are bot the only onze having problems with the released templates.
biligual spelling checker used.read at your own risk
Op 10 aug. 2014 17:19 schreef "Andrei Mikhailovsky" <an...@arhont.com>:
> I've got another proble, still relating to the systemvms. This time it is:
> https://issues.apache.org/jira/browse/CLOUDSTACK-6893
>
> Anyone have an idea on how to fix it on 4.4.0?
>
> As in my previous post, all my system vms are down - no connectivity to
> the vms. I was hoping to fix this on 4.4.0 without the need to downgrade
> for the 2nd time in 2 days.
>
> Thanks
>
> Andrei
>
> ----- Original Message -----
> From: "Andrei Mikhailovsky" <an...@arhont.com>
> To: users@cloudstack.apache.org
> Sent: Saturday, 9 August, 2014 6:29:36 PM
> Subject: HELP - unable to start systemvms on KVM after upgrade to 4.4 with
> java 1.7 installed
>
> Hello guys,
>
> I've done an upgrade to ACS 4.4 from version 4.2.1 following the release
> notes and upgrade guide. I've also read several threads regarding issues
> with the systemvm template. I've downloaded the template from
> http://cloudstack.apt-get.eu/systemvm/4.4/systemvm64template-4.4.0-6-kvm.qcow2.bz2
>
> I've also upgraded java to version 1.7 on host servers as well as the acs
> management server.
>
> Following the guide, I've executed the script that restarts all system vms
> with the following output:
>
>
>
> Stopping and starting 1 secondary storage vm(s)...
>
>
>
>
> Done stopping and starting secondary storage vm(s)
>
>
>
>
> Stopping and starting 1 console proxy vm(s)...
>
> ERROR: Failed to start console proxy vm with id 847
>
>
>
>
> Done stopping and starting console proxy vm(s) .
>
>
>
>
> Stopping and starting 15 running routing vm(s)...
>
> ERROR: Failed to restart domainRouter with id 864
>
>
>
>
> ERROR: Failed to restart domainRouter with id 852
>
>
>
>
> ......
>
>
>
>
> I've noticed and all of my system vms are Stopped and ACS is unable to
> start SSVM / CPVM and any of the virtual routers.
>
>
>
>
> Looking into the issue i've noticed that the host servers have the
> following in agent.log file:
>
>
>
>
> 2014-08-09 18:00:12,093 ERROR [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) Failed to perform a RBD action on the Ceph
> cluster, the error was: Failed to clone image
> 7ec0a208-6593-420c-9437-79279a5e3f03@cloudstack-base-snap to
> 566b003a-de59-419a-a1f6-7cf77cc95378
>
>
>
>
>
>
>
>
>
>
>
>
>
> The management server log shows this error:
>
>
>
>
>
>
>
>
>
>
> 2014-08-09 18:12:32,524 WARN [o.a.c.s.SecondaryStorageManagerImpl]
> (secstorage-1:ctx-7bb24cc7) Exception while trying to start secondary
> storage vm
>
> java.lang.RuntimeException: Job failed due to exception Resource [Host:7]
> is unreachable: Host 7: Unable to start instance due to Unable to get
> answer that is of class com.cloud.agent.api.StartAnswer
>
> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114)
>
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
>
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
>
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>
> at java.lang.Thread.run(Thread.java:744)
>
> 2014-08-09 18:12:32,525 INFO [o.a.c.s.SecondaryStorageManagerImpl]
> (secstorage-1:ctx-7bb24cc7) Unable to start secondary storage vm for
> standby capacity, secStorageVm vm Id : 940, will recycle it and start a new
> one
>
>
>
>
>
>
>
>
>
>
> This error keeps cycling every minute or so.
>
>
>
>
> Could someone please help me with sorting out the system vms.
>
>
>
>
> Many thanks
>
>
>
>
> Andrei
>
Re: HELP - unable to start systemvms on KVM after upgrade to 4.4 with
java 1.7 installed
Posted by Daan Hoogland <da...@gmail.com>.
Andrei,
Try the sysvm templates VROM Jenkins.buildacloud.org. the 4.4 chapter
contains a build for 64bit templates. These are reported as working. You
are bot the only onze having problems with the released templates.
biligual spelling checker used.read at your own risk
Op 10 aug. 2014 17:19 schreef "Andrei Mikhailovsky" <an...@arhont.com>:
> I've got another proble, still relating to the systemvms. This time it is:
> https://issues.apache.org/jira/browse/CLOUDSTACK-6893
>
> Anyone have an idea on how to fix it on 4.4.0?
>
> As in my previous post, all my system vms are down - no connectivity to
> the vms. I was hoping to fix this on 4.4.0 without the need to downgrade
> for the 2nd time in 2 days.
>
> Thanks
>
> Andrei
>
> ----- Original Message -----
> From: "Andrei Mikhailovsky" <an...@arhont.com>
> To: users@cloudstack.apache.org
> Sent: Saturday, 9 August, 2014 6:29:36 PM
> Subject: HELP - unable to start systemvms on KVM after upgrade to 4.4 with
> java 1.7 installed
>
> Hello guys,
>
> I've done an upgrade to ACS 4.4 from version 4.2.1 following the release
> notes and upgrade guide. I've also read several threads regarding issues
> with the systemvm template. I've downloaded the template from
> http://cloudstack.apt-get.eu/systemvm/4.4/systemvm64template-4.4.0-6-kvm.qcow2.bz2
>
> I've also upgraded java to version 1.7 on host servers as well as the acs
> management server.
>
> Following the guide, I've executed the script that restarts all system vms
> with the following output:
>
>
>
> Stopping and starting 1 secondary storage vm(s)...
>
>
>
>
> Done stopping and starting secondary storage vm(s)
>
>
>
>
> Stopping and starting 1 console proxy vm(s)...
>
> ERROR: Failed to start console proxy vm with id 847
>
>
>
>
> Done stopping and starting console proxy vm(s) .
>
>
>
>
> Stopping and starting 15 running routing vm(s)...
>
> ERROR: Failed to restart domainRouter with id 864
>
>
>
>
> ERROR: Failed to restart domainRouter with id 852
>
>
>
>
> ......
>
>
>
>
> I've noticed and all of my system vms are Stopped and ACS is unable to
> start SSVM / CPVM and any of the virtual routers.
>
>
>
>
> Looking into the issue i've noticed that the host servers have the
> following in agent.log file:
>
>
>
>
> 2014-08-09 18:00:12,093 ERROR [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) Failed to perform a RBD action on the Ceph
> cluster, the error was: Failed to clone image
> 7ec0a208-6593-420c-9437-79279a5e3f03@cloudstack-base-snap to
> 566b003a-de59-419a-a1f6-7cf77cc95378
>
>
>
>
>
>
>
>
>
>
>
>
>
> The management server log shows this error:
>
>
>
>
>
>
>
>
>
>
> 2014-08-09 18:12:32,524 WARN [o.a.c.s.SecondaryStorageManagerImpl]
> (secstorage-1:ctx-7bb24cc7) Exception while trying to start secondary
> storage vm
>
> java.lang.RuntimeException: Job failed due to exception Resource [Host:7]
> is unreachable: Host 7: Unable to start instance due to Unable to get
> answer that is of class com.cloud.agent.api.StartAnswer
>
> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114)
>
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
>
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
>
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>
> at java.lang.Thread.run(Thread.java:744)
>
> 2014-08-09 18:12:32,525 INFO [o.a.c.s.SecondaryStorageManagerImpl]
> (secstorage-1:ctx-7bb24cc7) Unable to start secondary storage vm for
> standby capacity, secStorageVm vm Id : 940, will recycle it and start a new
> one
>
>
>
>
>
>
>
>
>
>
> This error keeps cycling every minute or so.
>
>
>
>
> Could someone please help me with sorting out the system vms.
>
>
>
>
> Many thanks
>
>
>
>
> Andrei
>
Re: HELP - unable to start systemvms on KVM after upgrade to 4.4
with java 1.7 installed
Posted by Andrei Mikhailovsky <an...@arhont.com>.
I've got another proble, still relating to the systemvms. This time it is: https://issues.apache.org/jira/browse/CLOUDSTACK-6893
Anyone have an idea on how to fix it on 4.4.0?
As in my previous post, all my system vms are down - no connectivity to the vms. I was hoping to fix this on 4.4.0 without the need to downgrade for the 2nd time in 2 days.
Thanks
Andrei
----- Original Message -----
From: "Andrei Mikhailovsky" <an...@arhont.com>
To: users@cloudstack.apache.org
Sent: Saturday, 9 August, 2014 6:29:36 PM
Subject: HELP - unable to start systemvms on KVM after upgrade to 4.4 with java 1.7 installed
Hello guys,
I've done an upgrade to ACS 4.4 from version 4.2.1 following the release notes and upgrade guide. I've also read several threads regarding issues with the systemvm template. I've downloaded the template from http://cloudstack.apt-get.eu/systemvm/4.4/systemvm64template-4.4.0-6-kvm.qcow2.bz2
I've also upgraded java to version 1.7 on host servers as well as the acs management server.
Following the guide, I've executed the script that restarts all system vms with the following output:
Stopping and starting 1 secondary storage vm(s)...
Done stopping and starting secondary storage vm(s)
Stopping and starting 1 console proxy vm(s)...
ERROR: Failed to start console proxy vm with id 847
Done stopping and starting console proxy vm(s) .
Stopping and starting 15 running routing vm(s)...
ERROR: Failed to restart domainRouter with id 864
ERROR: Failed to restart domainRouter with id 852
......
I've noticed and all of my system vms are Stopped and ACS is unable to start SSVM / CPVM and any of the virtual routers.
Looking into the issue i've noticed that the host servers have the following in agent.log file:
2014-08-09 18:00:12,093 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) Failed to perform a RBD action on the Ceph cluster, the error was: Failed to clone image 7ec0a208-6593-420c-9437-79279a5e3f03@cloudstack-base-snap to 566b003a-de59-419a-a1f6-7cf77cc95378
The management server log shows this error:
2014-08-09 18:12:32,524 WARN [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-7bb24cc7) Exception while trying to start secondary storage vm
java.lang.RuntimeException: Job failed due to exception Resource [Host:7] is unreachable: Host 7: Unable to start instance due to Unable to get answer that is of class com.cloud.agent.api.StartAnswer
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-08-09 18:12:32,525 INFO [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-7bb24cc7) Unable to start secondary storage vm for standby capacity, secStorageVm vm Id : 940, will recycle it and start a new one
This error keeps cycling every minute or so.
Could someone please help me with sorting out the system vms.
Many thanks
Andrei