You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Andrei Mikhailovsky <an...@arhont.com> on 2013/12/24 14:06:44 UTC
Snapshot and Template problems in ACS 4.2.0 with KVM
Hi guys,
I am wondering if anyone is experiencing problems with creating Snapshot, converting it to the Template and deploying a vm from that template?
I am having a problem doing this on my newly installed ACS 4.2.0 with KVM hypervisor with NFS secondary storage. I get the following error:
2013-12-23 15:54:11,088 DEBUG [agent.transport.Request] (AgentManager-Handler-11:null) Seq 1-2082880489 : Processing: { Ans: , MgmtId: 238402986947284, via: 1, Ver: v1, Flags: 110, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException: Failed to copy /mnt/08452456-d626-307b-9cbe-c028ae633435/59dfab5d-832e-4160-8f8e-47efb2d23d3e.raw to 0756278c-5abc-4e6c-b9ae-37033194d31a","wait":0}}] }
2013-12-23 15:54:11,088 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-11:null) Seq 1-2082880489 : No more commands found
2013-12-23 15:54:11,088 DEBUG [agent.transport.Request] (Job-Executor-113:job-548 = [ dc47c0ce-eb8b-4230-b58a-047ab55950fb ]) Seq 1-2082880489 : Received: { Ans: , MgmtId: 238402986947284, via: 1, Ver: v1, Flags: 110, { CopyCmdAnswer } }
2013-12-23 15:54:11,093 INFO [storage.volume.VolumeServiceImpl] (Job-Executor-113:job-548 = [ dc47c0ce-eb8b-4230-b58a-047ab55950fb ]) releasing lock for VMTemplateStoragePool 12
2013-12-23 15:54:11,093 WARN [utils.db.Merovingian2] (Job-Executor-113:job-548 = [ dc47c0ce-eb8b-4230-b58a-047ab55950fb ]) Was unable to find lock for the key template_spool_ref12 and thread id 291724899
2013-12-23 15:54:11,093 DEBUG [cloud.storage.VolumeManagerImpl] (Job-Executor-113:job-548 = [ dc47c0ce-eb8b-4230-b58a-047ab55950fb ]) Unable to create Vol[504|vm=508|ROOT]:com.cloud.utils.exception.CloudRuntimeException: Failed to copy /mnt/08452456-d626-307b-9cbe-c028ae633435/59dfab5d-832e-4160-8f8e-47efb2d23d3e.raw to 0756278c-5abc-4e6c-b9ae-37033194d31a
2013-12-23 15:54:11,093 INFO [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-113:job-548 = [ dc47c0ce-eb8b-4230-b58a-047ab55950fb ]) Unable to contact resource.
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:2] is unreachable: Unable to create Vol[504|vm=508|ROOT]:com.cloud.utils.exception.CloudRuntimeException: Failed to copy /mnt/08452456-d626-307b-9cbe-c028ae633435/59dfab5d-832e-4160-8f8e-47efb2d23d3e.raw to 0756278c-5abc-4e6c-b9ae-37033194d31a
at com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2544)
at com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2592)
at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:889)
at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:578)
at org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:227)
at org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209)
at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3406)
at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2966)
at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2952)
at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:420)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)
The template file is there:
# find ./ |grep 59dfab5d-832e-4160-8f8e-47efb2d23d3e.raw
./template/tmpl/5/217/59dfab5d-832e-4160-8f8e-47efb2d23d3e.raw
What I've noticed is that I can stop a vm, create a template from its volume and roll out a new vm without any issues. The problem is when you create a snapshot and create a template from that snapshot.
An even stranger thing is that I have another installation of ACS 4.2.0, which has been upgraded many times from version 3 to 4.0 to 4.1 to 4.2.0 instead of a clean install. I do not have this issue on that installation of ACS, but I do with a clean one. I've tried reinstalling a clean version many times and the problem is reproducible for me.
Any help or thoughts would be highly welcome.
Many thanks
Andrei
Re: Snapshot and Template problems in ACS 4.2.0 with KVM
Posted by BJO ERN <bj...@gmail.com>.
Snapshots are working on RAW volumes if you have RBD or CLVM running.
Cloudstack executes a qemu-img convert command to convert the raw
partition/volume to a qcow2 file.
In this case I would look at the cloudstack-agent.err and .log at
/var/log/cloustack/agent for suspicious errors.
Usually I had issues with mounting those volumes on the hypervisor or
issues writing into the files
Bjoern
On Tue, Dec 24, 2013 at 8:28 AM, Andrei Mikhailovsky <an...@arhont.com>wrote:
>
>
> I think the RAW format is used for RBD primary storage, which I have.
> Having said this, the design documents, as far as I understood, show that
> all operations with snapshots and templates are done in QCOW2 for
> compatibility purposes, as you've said .
>
> So, it seems, there is an issue with snapshotting from the RBD primary
> storage.
>
> Wido, do you have any idea what could go wrong here?
>
> Thanks
>
> Andrei
>
> ----- Original Message -----
>
> From: "Andrija Panic" <an...@gmail.com>
> To: users@cloudstack.apache.org
> Sent: Tuesday, 24 December, 2013 4:12:41 PM
> Subject: Re: Snapshot and Template problems in ACS 4.2.0 with KVM
>
> ACS officialy only works with qcow2, so this is strange that you see RAW as
> a format...
>
>
> On 24 December 2013 16:58, Andrei Mikhailovsky <an...@arhont.com> wrote:
>
> >
> > Andrija,
> >
> > Thanks for your answer, but I would think that if there is an issue with
> > the secondary storage it would also effect the template creation. I do
> not
> > have any issues with directly creating the templates. The issues seems to
> > be when the snapshots are involved.
> >
> > Also, i've noticed that when you do root_volume > template > vm - the
> > template format in the database is shown as QCOW2. When I am doing
> > root_volume > snapshot > template > vm the template format is shown as
> RAW.
> >
> > It seems that there is a problem using RAW format. Has anyone else
> > encountered this issue?
> >
> > Andre i
> > ----- Original Message -----
> >
> > From: "Andrija Panic" <an...@gmail.com>
> > To: users@cloudstack.apache.org
> > Sent: Tuesday, 24 December, 2013 1:13:14 PM
> > Subject: Re: Snapshot and Template problems in ACS 4.2.0 with KVM
> >
> > Make sure that hypervisor node, can mount and write to the storage, you
> can
> > check this with primary storage, there is KVMHA folders inside, and each
> of
> > them has files that should be modified each minute, so if any of those
> > lags, you might have issues with connecting to storage - I had that case,
> > so forced umount of the problematic NFS storage, and restarted CS agent
> on
> > problematic node.
> >
> > Hope it helps
> >
> >
> > On 24 December 2013 14:06, Andrei Mikhailovsky <an...@arhont.com>
> wrote:
> >
> > > Hi guys,
> > >
> > >
> > > I am wondering if anyone is experiencing problems with creating
> Snapshot,
> > > converting it to the Template and deploying a vm from that template?
> > >
> > > I am having a problem doing this on my newly installed ACS 4.2.0 with
> KVM
> > > hypervisor with NFS secondary storage. I get the following error:
> > >
> > >
> > > 2013-12-23 15:54:11,088 DEBUG [agent.transport.Request]
> > > (AgentManager-Handler-11:null) Seq 1-2082880489 : Processing: { Ans: ,
> > > MgmtId: 238402986947284, via: 1, Ver: v1, Flags: 110,
> > >
> >
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException:
> > > Failed to copy
> > >
> >
> /mnt/08452456-d626-307b-9cbe-c028ae633435/59dfab5d-832e-4160-8f8e-47efb2d23d3e.raw
> > > to 0756278c-5abc-4e6c-b9ae-37033194d31a","wait":0}}] }
> > > 2013-12-23 15:54:11,088 DEBUG [agent.manager.AgentAttache]
> > > (AgentManager-Handler-11:null) Seq 1-2082880489 : No more commands
> found
> > > 2013-12-23 15:54:11,088 DEBUG [agent.transport.Request]
> > > (Job-Executor-113:job-548 = [ dc47c0ce-eb8b-4230-b58a-047ab55950fb ])
> Seq
> > > 1-2082880489 : Received: { Ans: , MgmtId: 238402986947284, via: 1, Ver:
> > v1,
> > > Flags: 110, { CopyCmdAnswer } }
> > > 2013-12-23 15:54:11,093 INFO [storage.volume.VolumeServiceImpl]
> > > (Job-Executor-113:job-548 = [ dc47c0ce-eb8b-4230-b58a-047ab55950fb ])
> > > releasing lock for VMTemplateStoragePool 12
> > > 2013-12-23 15:54:11,093 WARN [utils.db.Merovingian2]
> > > (Job-Executor-113:job-548 = [ dc47c0ce-eb8b-4230-b58a-047ab55950fb ])
> Was
> > > unable to find lock for the key template_spool_ref12 and thread id
> > 291724899
> > > 2013-12-23 15:54:11,093 DEBUG [cloud.storage.VolumeManagerImpl]
> > > (Job-Executor-113:job-548 = [ dc47c0ce-eb8b-4230-b58a-047ab55950fb ])
> > > Unable to create
> > > Vol[504|vm=508|ROOT]:com.cloud.utils.exception.CloudRuntimeException:
> > > Failed to copy
> > >
> >
> /mnt/08452456-d626-307b-9cbe-c028ae633435/59dfab5d-832e-4160-8f8e-47efb2d23d3e.raw
> > > to 0756278c-5abc-4e6c-b9ae-37033194d31a
> > > 2013-12-23 15:54:11,093 INFO [cloud.vm.VirtualMachineManagerImpl]
> > > (Job-Executor-113:job-548 = [ dc47c0ce-eb8b-4230-b58a-047ab55950fb ])
> > > Unable to contact resource.
> > > com.cloud.exception.StorageUnavailableException: Resource
> [StoragePool:2]
> > > is unreachable: Unable to create
> > > Vol[504|vm=508|ROOT]:com.cloud.utils.exception.CloudRuntimeException:
> > > Failed to copy
> > >
> >
> /mnt/08452456-d626-307b-9cbe-c028ae633435/59dfab5d-832e-4160-8f8e-47efb2d23d3e.raw
> > > to 0756278c-5abc-4e6c-b9ae-37033194d31a
> > > at
> > >
> >
> com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2544)
> > > at
> > com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2592)
> > > at
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:889)
> > > at
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:578)
> > > at
> > >
> >
> org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:227)
> > > at
> > >
> >
> org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209)
> > > at
> > >
> >
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3406)
> > > at
> > >
> >
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2966)
> > > at
> > >
> >
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2952)
> > > at
> > >
> >
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> > > at
> > >
> >
> org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:420)
> > > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
> > > at
> > com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
> > > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> > > at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> > > at
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
> > > at
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > > at java.lang.Thread.run(Thread.java:701)
> > >
> > >
> > > The template file is there:
> > >
> > > # find ./ |grep 59dfab5d-832e-4160-8f8e-47efb2d23d3e.raw
> > > ./template/tmpl/5/217/59dfab5d-832e-4160-8f8e-47efb2d23d3e.raw
> > >
> > >
> > > What I've noticed is that I can stop a vm, create a template from its
> > > volume and roll out a new vm without any issues. The problem is when
> you
> > > create a snapshot and create a template from that snapshot.
> > >
> > >
> > > An even stranger thing is that I have another installation of ACS
> 4.2.0,
> > > which has been upgraded many times from version 3 to 4.0 to 4.1 to
> 4.2.0
> > > instead of a clean install. I do not have this issue on that
> installation
> > > of ACS, but I do with a clean one. I've tried reinstalling a clean
> > version
> > > many times and the problem is reproducible for me.
> > >
> > > Any help or thoughts would be highly welcome.
> > >
> > > Many thanks
> > >
> > > Andrei
> > >
> >
> >
> >
> > --
> >
> > Andrija Panić
> > --------------------------------------
> > http://admintweets.com
> > --------------------------------------
> >
> >
>
>
> --
>
> Andrija Panić
> --------------------------------------
> http://admintweets.com
> --------------------------------------
>
>
Re: Snapshot and Template problems in ACS 4.2.0 with KVM
Posted by Andrei Mikhailovsky <an...@arhont.com>.
I think the RAW format is used for RBD primary storage, which I have. Having said this, the design documents, as far as I understood, show that all operations with snapshots and templates are done in QCOW2 for compatibility purposes, as you've said .
So, it seems, there is an issue with snapshotting from the RBD primary storage.
Wido, do you have any idea what could go wrong here?
Thanks
Andrei
----- Original Message -----
From: "Andrija Panic" <an...@gmail.com>
To: users@cloudstack.apache.org
Sent: Tuesday, 24 December, 2013 4:12:41 PM
Subject: Re: Snapshot and Template problems in ACS 4.2.0 with KVM
ACS officialy only works with qcow2, so this is strange that you see RAW as
a format...
On 24 December 2013 16:58, Andrei Mikhailovsky <an...@arhont.com> wrote:
>
> Andrija,
>
> Thanks for your answer, but I would think that if there is an issue with
> the secondary storage it would also effect the template creation. I do not
> have any issues with directly creating the templates. The issues seems to
> be when the snapshots are involved.
>
> Also, i've noticed that when you do root_volume > template > vm - the
> template format in the database is shown as QCOW2. When I am doing
> root_volume > snapshot > template > vm the template format is shown as RAW.
>
> It seems that there is a problem using RAW format. Has anyone else
> encountered this issue?
>
> Andre i
> ----- Original Message -----
>
> From: "Andrija Panic" <an...@gmail.com>
> To: users@cloudstack.apache.org
> Sent: Tuesday, 24 December, 2013 1:13:14 PM
> Subject: Re: Snapshot and Template problems in ACS 4.2.0 with KVM
>
> Make sure that hypervisor node, can mount and write to the storage, you can
> check this with primary storage, there is KVMHA folders inside, and each of
> them has files that should be modified each minute, so if any of those
> lags, you might have issues with connecting to storage - I had that case,
> so forced umount of the problematic NFS storage, and restarted CS agent on
> problematic node.
>
> Hope it helps
>
>
> On 24 December 2013 14:06, Andrei Mikhailovsky <an...@arhont.com> wrote:
>
> > Hi guys,
> >
> >
> > I am wondering if anyone is experiencing problems with creating Snapshot,
> > converting it to the Template and deploying a vm from that template?
> >
> > I am having a problem doing this on my newly installed ACS 4.2.0 with KVM
> > hypervisor with NFS secondary storage. I get the following error:
> >
> >
> > 2013-12-23 15:54:11,088 DEBUG [agent.transport.Request]
> > (AgentManager-Handler-11:null) Seq 1-2082880489 : Processing: { Ans: ,
> > MgmtId: 238402986947284, via: 1, Ver: v1, Flags: 110,
> >
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException:
> > Failed to copy
> >
> /mnt/08452456-d626-307b-9cbe-c028ae633435/59dfab5d-832e-4160-8f8e-47efb2d23d3e.raw
> > to 0756278c-5abc-4e6c-b9ae-37033194d31a","wait":0}}] }
> > 2013-12-23 15:54:11,088 DEBUG [agent.manager.AgentAttache]
> > (AgentManager-Handler-11:null) Seq 1-2082880489 : No more commands found
> > 2013-12-23 15:54:11,088 DEBUG [agent.transport.Request]
> > (Job-Executor-113:job-548 = [ dc47c0ce-eb8b-4230-b58a-047ab55950fb ]) Seq
> > 1-2082880489 : Received: { Ans: , MgmtId: 238402986947284, via: 1, Ver:
> v1,
> > Flags: 110, { CopyCmdAnswer } }
> > 2013-12-23 15:54:11,093 INFO [storage.volume.VolumeServiceImpl]
> > (Job-Executor-113:job-548 = [ dc47c0ce-eb8b-4230-b58a-047ab55950fb ])
> > releasing lock for VMTemplateStoragePool 12
> > 2013-12-23 15:54:11,093 WARN [utils.db.Merovingian2]
> > (Job-Executor-113:job-548 = [ dc47c0ce-eb8b-4230-b58a-047ab55950fb ]) Was
> > unable to find lock for the key template_spool_ref12 and thread id
> 291724899
> > 2013-12-23 15:54:11,093 DEBUG [cloud.storage.VolumeManagerImpl]
> > (Job-Executor-113:job-548 = [ dc47c0ce-eb8b-4230-b58a-047ab55950fb ])
> > Unable to create
> > Vol[504|vm=508|ROOT]:com.cloud.utils.exception.CloudRuntimeException:
> > Failed to copy
> >
> /mnt/08452456-d626-307b-9cbe-c028ae633435/59dfab5d-832e-4160-8f8e-47efb2d23d3e.raw
> > to 0756278c-5abc-4e6c-b9ae-37033194d31a
> > 2013-12-23 15:54:11,093 INFO [cloud.vm.VirtualMachineManagerImpl]
> > (Job-Executor-113:job-548 = [ dc47c0ce-eb8b-4230-b58a-047ab55950fb ])
> > Unable to contact resource.
> > com.cloud.exception.StorageUnavailableException: Resource [StoragePool:2]
> > is unreachable: Unable to create
> > Vol[504|vm=508|ROOT]:com.cloud.utils.exception.CloudRuntimeException:
> > Failed to copy
> >
> /mnt/08452456-d626-307b-9cbe-c028ae633435/59dfab5d-832e-4160-8f8e-47efb2d23d3e.raw
> > to 0756278c-5abc-4e6c-b9ae-37033194d31a
> > at
> >
> com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2544)
> > at
> com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2592)
> > at
> >
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:889)
> > at
> >
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:578)
> > at
> >
> org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:227)
> > at
> >
> org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209)
> > at
> >
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3406)
> > at
> >
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2966)
> > at
> >
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2952)
> > at
> >
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> > at
> >
> org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:420)
> > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
> > at
> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
> > at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > at java.lang.Thread.run(Thread.java:701)
> >
> >
> > The template file is there:
> >
> > # find ./ |grep 59dfab5d-832e-4160-8f8e-47efb2d23d3e.raw
> > ./template/tmpl/5/217/59dfab5d-832e-4160-8f8e-47efb2d23d3e.raw
> >
> >
> > What I've noticed is that I can stop a vm, create a template from its
> > volume and roll out a new vm without any issues. The problem is when you
> > create a snapshot and create a template from that snapshot.
> >
> >
> > An even stranger thing is that I have another installation of ACS 4.2.0,
> > which has been upgraded many times from version 3 to 4.0 to 4.1 to 4.2.0
> > instead of a clean install. I do not have this issue on that installation
> > of ACS, but I do with a clean one. I've tried reinstalling a clean
> version
> > many times and the problem is reproducible for me.
> >
> > Any help or thoughts would be highly welcome.
> >
> > Many thanks
> >
> > Andrei
> >
>
>
>
> --
>
> Andrija Panić
> --------------------------------------
> http://admintweets.com
> --------------------------------------
>
>
--
Andrija Panić
--------------------------------------
http://admintweets.com
--------------------------------------
Re: Snapshot and Template problems in ACS 4.2.0 with KVM
Posted by Andrija Panic <an...@gmail.com>.
ACS officialy only works with qcow2, so this is strange that you see RAW as
a format...
On 24 December 2013 16:58, Andrei Mikhailovsky <an...@arhont.com> wrote:
>
> Andrija,
>
> Thanks for your answer, but I would think that if there is an issue with
> the secondary storage it would also effect the template creation. I do not
> have any issues with directly creating the templates. The issues seems to
> be when the snapshots are involved.
>
> Also, i've noticed that when you do root_volume > template > vm - the
> template format in the database is shown as QCOW2. When I am doing
> root_volume > snapshot > template > vm the template format is shown as RAW.
>
> It seems that there is a problem using RAW format. Has anyone else
> encountered this issue?
>
> Andre i
> ----- Original Message -----
>
> From: "Andrija Panic" <an...@gmail.com>
> To: users@cloudstack.apache.org
> Sent: Tuesday, 24 December, 2013 1:13:14 PM
> Subject: Re: Snapshot and Template problems in ACS 4.2.0 with KVM
>
> Make sure that hypervisor node, can mount and write to the storage, you can
> check this with primary storage, there is KVMHA folders inside, and each of
> them has files that should be modified each minute, so if any of those
> lags, you might have issues with connecting to storage - I had that case,
> so forced umount of the problematic NFS storage, and restarted CS agent on
> problematic node.
>
> Hope it helps
>
>
> On 24 December 2013 14:06, Andrei Mikhailovsky <an...@arhont.com> wrote:
>
> > Hi guys,
> >
> >
> > I am wondering if anyone is experiencing problems with creating Snapshot,
> > converting it to the Template and deploying a vm from that template?
> >
> > I am having a problem doing this on my newly installed ACS 4.2.0 with KVM
> > hypervisor with NFS secondary storage. I get the following error:
> >
> >
> > 2013-12-23 15:54:11,088 DEBUG [agent.transport.Request]
> > (AgentManager-Handler-11:null) Seq 1-2082880489 : Processing: { Ans: ,
> > MgmtId: 238402986947284, via: 1, Ver: v1, Flags: 110,
> >
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException:
> > Failed to copy
> >
> /mnt/08452456-d626-307b-9cbe-c028ae633435/59dfab5d-832e-4160-8f8e-47efb2d23d3e.raw
> > to 0756278c-5abc-4e6c-b9ae-37033194d31a","wait":0}}] }
> > 2013-12-23 15:54:11,088 DEBUG [agent.manager.AgentAttache]
> > (AgentManager-Handler-11:null) Seq 1-2082880489 : No more commands found
> > 2013-12-23 15:54:11,088 DEBUG [agent.transport.Request]
> > (Job-Executor-113:job-548 = [ dc47c0ce-eb8b-4230-b58a-047ab55950fb ]) Seq
> > 1-2082880489 : Received: { Ans: , MgmtId: 238402986947284, via: 1, Ver:
> v1,
> > Flags: 110, { CopyCmdAnswer } }
> > 2013-12-23 15:54:11,093 INFO [storage.volume.VolumeServiceImpl]
> > (Job-Executor-113:job-548 = [ dc47c0ce-eb8b-4230-b58a-047ab55950fb ])
> > releasing lock for VMTemplateStoragePool 12
> > 2013-12-23 15:54:11,093 WARN [utils.db.Merovingian2]
> > (Job-Executor-113:job-548 = [ dc47c0ce-eb8b-4230-b58a-047ab55950fb ]) Was
> > unable to find lock for the key template_spool_ref12 and thread id
> 291724899
> > 2013-12-23 15:54:11,093 DEBUG [cloud.storage.VolumeManagerImpl]
> > (Job-Executor-113:job-548 = [ dc47c0ce-eb8b-4230-b58a-047ab55950fb ])
> > Unable to create
> > Vol[504|vm=508|ROOT]:com.cloud.utils.exception.CloudRuntimeException:
> > Failed to copy
> >
> /mnt/08452456-d626-307b-9cbe-c028ae633435/59dfab5d-832e-4160-8f8e-47efb2d23d3e.raw
> > to 0756278c-5abc-4e6c-b9ae-37033194d31a
> > 2013-12-23 15:54:11,093 INFO [cloud.vm.VirtualMachineManagerImpl]
> > (Job-Executor-113:job-548 = [ dc47c0ce-eb8b-4230-b58a-047ab55950fb ])
> > Unable to contact resource.
> > com.cloud.exception.StorageUnavailableException: Resource [StoragePool:2]
> > is unreachable: Unable to create
> > Vol[504|vm=508|ROOT]:com.cloud.utils.exception.CloudRuntimeException:
> > Failed to copy
> >
> /mnt/08452456-d626-307b-9cbe-c028ae633435/59dfab5d-832e-4160-8f8e-47efb2d23d3e.raw
> > to 0756278c-5abc-4e6c-b9ae-37033194d31a
> > at
> >
> com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2544)
> > at
> com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2592)
> > at
> >
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:889)
> > at
> >
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:578)
> > at
> >
> org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:227)
> > at
> >
> org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209)
> > at
> >
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3406)
> > at
> >
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2966)
> > at
> >
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2952)
> > at
> >
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> > at
> >
> org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:420)
> > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
> > at
> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
> > at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > at java.lang.Thread.run(Thread.java:701)
> >
> >
> > The template file is there:
> >
> > # find ./ |grep 59dfab5d-832e-4160-8f8e-47efb2d23d3e.raw
> > ./template/tmpl/5/217/59dfab5d-832e-4160-8f8e-47efb2d23d3e.raw
> >
> >
> > What I've noticed is that I can stop a vm, create a template from its
> > volume and roll out a new vm without any issues. The problem is when you
> > create a snapshot and create a template from that snapshot.
> >
> >
> > An even stranger thing is that I have another installation of ACS 4.2.0,
> > which has been upgraded many times from version 3 to 4.0 to 4.1 to 4.2.0
> > instead of a clean install. I do not have this issue on that installation
> > of ACS, but I do with a clean one. I've tried reinstalling a clean
> version
> > many times and the problem is reproducible for me.
> >
> > Any help or thoughts would be highly welcome.
> >
> > Many thanks
> >
> > Andrei
> >
>
>
>
> --
>
> Andrija Panić
> --------------------------------------
> http://admintweets.com
> --------------------------------------
>
>
--
Andrija Panić
--------------------------------------
http://admintweets.com
--------------------------------------
Re: Snapshot and Template problems in ACS 4.2.0 with KVM
Posted by Andrei Mikhailovsky <an...@arhont.com>.
Andrija,
Thanks for your answer, but I would think that if there is an issue with the secondary storage it would also effect the template creation. I do not have any issues with directly creating the templates. The issues seems to be when the snapshots are involved.
Also, i've noticed that when you do root_volume > template > vm - the template format in the database is shown as QCOW2. When I am doing root_volume > snapshot > template > vm the template format is shown as RAW.
It seems that there is a problem using RAW format. Has anyone else encountered this issue?
Andre i
----- Original Message -----
From: "Andrija Panic" <an...@gmail.com>
To: users@cloudstack.apache.org
Sent: Tuesday, 24 December, 2013 1:13:14 PM
Subject: Re: Snapshot and Template problems in ACS 4.2.0 with KVM
Make sure that hypervisor node, can mount and write to the storage, you can
check this with primary storage, there is KVMHA folders inside, and each of
them has files that should be modified each minute, so if any of those
lags, you might have issues with connecting to storage - I had that case,
so forced umount of the problematic NFS storage, and restarted CS agent on
problematic node.
Hope it helps
On 24 December 2013 14:06, Andrei Mikhailovsky <an...@arhont.com> wrote:
> Hi guys,
>
>
> I am wondering if anyone is experiencing problems with creating Snapshot,
> converting it to the Template and deploying a vm from that template?
>
> I am having a problem doing this on my newly installed ACS 4.2.0 with KVM
> hypervisor with NFS secondary storage. I get the following error:
>
>
> 2013-12-23 15:54:11,088 DEBUG [agent.transport.Request]
> (AgentManager-Handler-11:null) Seq 1-2082880489 : Processing: { Ans: ,
> MgmtId: 238402986947284, via: 1, Ver: v1, Flags: 110,
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException:
> Failed to copy
> /mnt/08452456-d626-307b-9cbe-c028ae633435/59dfab5d-832e-4160-8f8e-47efb2d23d3e.raw
> to 0756278c-5abc-4e6c-b9ae-37033194d31a","wait":0}}] }
> 2013-12-23 15:54:11,088 DEBUG [agent.manager.AgentAttache]
> (AgentManager-Handler-11:null) Seq 1-2082880489 : No more commands found
> 2013-12-23 15:54:11,088 DEBUG [agent.transport.Request]
> (Job-Executor-113:job-548 = [ dc47c0ce-eb8b-4230-b58a-047ab55950fb ]) Seq
> 1-2082880489 : Received: { Ans: , MgmtId: 238402986947284, via: 1, Ver: v1,
> Flags: 110, { CopyCmdAnswer } }
> 2013-12-23 15:54:11,093 INFO [storage.volume.VolumeServiceImpl]
> (Job-Executor-113:job-548 = [ dc47c0ce-eb8b-4230-b58a-047ab55950fb ])
> releasing lock for VMTemplateStoragePool 12
> 2013-12-23 15:54:11,093 WARN [utils.db.Merovingian2]
> (Job-Executor-113:job-548 = [ dc47c0ce-eb8b-4230-b58a-047ab55950fb ]) Was
> unable to find lock for the key template_spool_ref12 and thread id 291724899
> 2013-12-23 15:54:11,093 DEBUG [cloud.storage.VolumeManagerImpl]
> (Job-Executor-113:job-548 = [ dc47c0ce-eb8b-4230-b58a-047ab55950fb ])
> Unable to create
> Vol[504|vm=508|ROOT]:com.cloud.utils.exception.CloudRuntimeException:
> Failed to copy
> /mnt/08452456-d626-307b-9cbe-c028ae633435/59dfab5d-832e-4160-8f8e-47efb2d23d3e.raw
> to 0756278c-5abc-4e6c-b9ae-37033194d31a
> 2013-12-23 15:54:11,093 INFO [cloud.vm.VirtualMachineManagerImpl]
> (Job-Executor-113:job-548 = [ dc47c0ce-eb8b-4230-b58a-047ab55950fb ])
> Unable to contact resource.
> com.cloud.exception.StorageUnavailableException: Resource [StoragePool:2]
> is unreachable: Unable to create
> Vol[504|vm=508|ROOT]:com.cloud.utils.exception.CloudRuntimeException:
> Failed to copy
> /mnt/08452456-d626-307b-9cbe-c028ae633435/59dfab5d-832e-4160-8f8e-47efb2d23d3e.raw
> to 0756278c-5abc-4e6c-b9ae-37033194d31a
> at
> com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2544)
> at com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2592)
> at
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:889)
> at
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:578)
> at
> org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:227)
> at
> org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209)
> at
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3406)
> at
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2966)
> at
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2952)
> at
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at
> org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:420)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
> at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:701)
>
>
> The template file is there:
>
> # find ./ |grep 59dfab5d-832e-4160-8f8e-47efb2d23d3e.raw
> ./template/tmpl/5/217/59dfab5d-832e-4160-8f8e-47efb2d23d3e.raw
>
>
> What I've noticed is that I can stop a vm, create a template from its
> volume and roll out a new vm without any issues. The problem is when you
> create a snapshot and create a template from that snapshot.
>
>
> An even stranger thing is that I have another installation of ACS 4.2.0,
> which has been upgraded many times from version 3 to 4.0 to 4.1 to 4.2.0
> instead of a clean install. I do not have this issue on that installation
> of ACS, but I do with a clean one. I've tried reinstalling a clean version
> many times and the problem is reproducible for me.
>
> Any help or thoughts would be highly welcome.
>
> Many thanks
>
> Andrei
>
--
Andrija Panić
--------------------------------------
http://admintweets.com
--------------------------------------
Re: Snapshot and Template problems in ACS 4.2.0 with KVM
Posted by Andrija Panic <an...@gmail.com>.
Make sure that hypervisor node, can mount and write to the storage, you can
check this with primary storage, there is KVMHA folders inside, and each of
them has files that should be modified each minute, so if any of those
lags, you might have issues with connecting to storage - I had that case,
so forced umount of the problematic NFS storage, and restarted CS agent on
problematic node.
Hope it helps
On 24 December 2013 14:06, Andrei Mikhailovsky <an...@arhont.com> wrote:
> Hi guys,
>
>
> I am wondering if anyone is experiencing problems with creating Snapshot,
> converting it to the Template and deploying a vm from that template?
>
> I am having a problem doing this on my newly installed ACS 4.2.0 with KVM
> hypervisor with NFS secondary storage. I get the following error:
>
>
> 2013-12-23 15:54:11,088 DEBUG [agent.transport.Request]
> (AgentManager-Handler-11:null) Seq 1-2082880489 : Processing: { Ans: ,
> MgmtId: 238402986947284, via: 1, Ver: v1, Flags: 110,
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException:
> Failed to copy
> /mnt/08452456-d626-307b-9cbe-c028ae633435/59dfab5d-832e-4160-8f8e-47efb2d23d3e.raw
> to 0756278c-5abc-4e6c-b9ae-37033194d31a","wait":0}}] }
> 2013-12-23 15:54:11,088 DEBUG [agent.manager.AgentAttache]
> (AgentManager-Handler-11:null) Seq 1-2082880489 : No more commands found
> 2013-12-23 15:54:11,088 DEBUG [agent.transport.Request]
> (Job-Executor-113:job-548 = [ dc47c0ce-eb8b-4230-b58a-047ab55950fb ]) Seq
> 1-2082880489 : Received: { Ans: , MgmtId: 238402986947284, via: 1, Ver: v1,
> Flags: 110, { CopyCmdAnswer } }
> 2013-12-23 15:54:11,093 INFO [storage.volume.VolumeServiceImpl]
> (Job-Executor-113:job-548 = [ dc47c0ce-eb8b-4230-b58a-047ab55950fb ])
> releasing lock for VMTemplateStoragePool 12
> 2013-12-23 15:54:11,093 WARN [utils.db.Merovingian2]
> (Job-Executor-113:job-548 = [ dc47c0ce-eb8b-4230-b58a-047ab55950fb ]) Was
> unable to find lock for the key template_spool_ref12 and thread id 291724899
> 2013-12-23 15:54:11,093 DEBUG [cloud.storage.VolumeManagerImpl]
> (Job-Executor-113:job-548 = [ dc47c0ce-eb8b-4230-b58a-047ab55950fb ])
> Unable to create
> Vol[504|vm=508|ROOT]:com.cloud.utils.exception.CloudRuntimeException:
> Failed to copy
> /mnt/08452456-d626-307b-9cbe-c028ae633435/59dfab5d-832e-4160-8f8e-47efb2d23d3e.raw
> to 0756278c-5abc-4e6c-b9ae-37033194d31a
> 2013-12-23 15:54:11,093 INFO [cloud.vm.VirtualMachineManagerImpl]
> (Job-Executor-113:job-548 = [ dc47c0ce-eb8b-4230-b58a-047ab55950fb ])
> Unable to contact resource.
> com.cloud.exception.StorageUnavailableException: Resource [StoragePool:2]
> is unreachable: Unable to create
> Vol[504|vm=508|ROOT]:com.cloud.utils.exception.CloudRuntimeException:
> Failed to copy
> /mnt/08452456-d626-307b-9cbe-c028ae633435/59dfab5d-832e-4160-8f8e-47efb2d23d3e.raw
> to 0756278c-5abc-4e6c-b9ae-37033194d31a
> at
> com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2544)
> at com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2592)
> at
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:889)
> at
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:578)
> at
> org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:227)
> at
> org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209)
> at
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3406)
> at
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2966)
> at
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2952)
> at
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at
> org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:420)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
> at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:701)
>
>
> The template file is there:
>
> # find ./ |grep 59dfab5d-832e-4160-8f8e-47efb2d23d3e.raw
> ./template/tmpl/5/217/59dfab5d-832e-4160-8f8e-47efb2d23d3e.raw
>
>
> What I've noticed is that I can stop a vm, create a template from its
> volume and roll out a new vm without any issues. The problem is when you
> create a snapshot and create a template from that snapshot.
>
>
> An even stranger thing is that I have another installation of ACS 4.2.0,
> which has been upgraded many times from version 3 to 4.0 to 4.1 to 4.2.0
> instead of a clean install. I do not have this issue on that installation
> of ACS, but I do with a clean one. I've tried reinstalling a clean version
> many times and the problem is reproducible for me.
>
> Any help or thoughts would be highly welcome.
>
> Many thanks
>
> Andrei
>
--
Andrija Panić
--------------------------------------
http://admintweets.com
--------------------------------------