You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Nux! <nu...@li.nux.ro> on 2013/12/21 00:04:19 UTC
KVM snapshots fail on GlusterFS shared mount point
Hi,
I know there's a vote for RC, maybe this has some importance. I'm
running KVM with a GlusterFS shared mount point on
cloudstack-4.2-223b272, snapshots consistently fail to be backed up.
They get stuck in "CreatedOnPrimary" status.
The shared mount point is more than functional and the 2ndary storage
can be mounted manually on both hypervisor and SSVM without problems.
GUI says:
"Failed to create snapshot due to an internal error creating snapshot
for volume 15"
Libvirt says:
2083: warning : qemuDomainObjBeginJobInternal:839 : Cannot start job
(query, none) for domain i-2-12-VM; current job is (modify, none) owned
by (2081, 0)
2083: error : qemuDomainObjBeginJobInternal:843 : Timed out during
operation: cannot acquire state change lock
Management logs says http://tmp.nux.ro/T94-ACSgluster.txt
Has anyone seen this before?
--
Sent from the Delta quadrant using Borg technology!
Nux!
www.nux.ro
Re: KVM snapshots fail on GlusterFS shared mount point (NFS too)
Posted by Nux! <nu...@li.nux.ro>.
On 20.12.2013 23:25, Nux! wrote:
> On 20.12.2013 23:04, Nux! wrote:
>> Hi,
>> I know there's a vote for RC, maybe this has some importance. I'm
>> running KVM with a GlusterFS shared mount point on
>
> In fact the snapshots fail with NFS primary storage as well, though
> no more complaints from libvirt in this case.
Tried on another test setup with Basic zone and NFS primary storage,
same outcome, even today's 4.2 from github fails with the same error.
2013-12-21 10:17:02,913 DEBUG [agent.manager.AgentAttache]
(AgentManager-Handler-6:null) Seq 1-1784545298: No more commands found
2013-12-21 10:17:02,913 DEBUG [agent.transport.Request]
(Job-Executor-1:job-12 = [ 7a568f3d-35a3-4c6c-ae11-691e6f4ab328 ]) Seq
1-1784545298: Received: { Ans: , MgmtId: 165340524328661, via: 1, Ver:
v1, Flags: 110, { CopyCmdAnswer } }
2013-12-21 10:17:02,923 DEBUG [storage.snapshot.SnapshotManagerImpl]
(Job-Executor-1:job-12 = [ 7a568f3d-35a3-4c6c-ae11-691e6f4ab328 ])
Failed to create snapshot
com.cloud.utils.exception.CloudRuntimeException: Failed to backup
98f25c16-fb79-4b0e-8443-4e7093d60555 for disk
/mnt/0740a8bf-c482-38a6-ad95-a2ef2ce4c1d5/739db255-5fc3-42c2-ae46-c7cd5047b8e1
to /mnt/e166eefb-d51e-33fe-80ab-9fec36914865/snapshots/2/3
at
org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:280)
at
org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:136)
at
org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:279)
at
com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1007)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1311)
at
com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2773)
at
org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:170)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at
com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)
2013-12-21 10:17:02,950 DEBUG [storage.volume.VolumeServiceImpl]
(Job-Executor-1:job-12 = [ 7a568f3d-35a3-4c6c-ae11-691e6f4ab328 ]) Take
snapshot: 3 failed
com.cloud.utils.exception.CloudRuntimeException: Failed to create
snapshot
at
com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1034)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1311)
at
com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2773)
at
org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:170)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at
com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)
Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to
backup 98f25c16-fb79-4b0e-8443-4e7093d60555 for disk
/mnt/0740a8bf-c482-38a6-ad95-a2ef2ce4c1d5/739db255-5fc3-42c2-ae46-c7cd5047b8e1
to /mnt/e166eefb-d51e-33fe-80ab-9fec36914865/snapshots/2/3
at
org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:280)
at
org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:136)
at
org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:279)
at
com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1007)
... 16 more
2013-12-21 10:17:02,950 DEBUG [user.snapshot.CreateSnapshotCmd]
(Job-Executor-1:job-12 = [ 7a568f3d-35a3-4c6c-ae11-691e6f4ab328 ])
Failed to create snapshot
org.apache.cloudstack.api.ServerApiException
at
org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:176)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at
com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)
--
Sent from the Delta quadrant using Borg technology!
Nux!
www.nux.ro
Re: KVM snapshots fail on GlusterFS shared mount point
Posted by Nux! <nu...@li.nux.ro>.
On 20.12.2013 23:04, Nux! wrote:
> Hi,
>
> I know there's a vote for RC, maybe this has some importance. I'm
> running KVM with a GlusterFS shared mount point on
In fact the snapshots fail with NFS primary storage as well, though no
more complaints from libvirt in this case.
--
Sent from the Delta quadrant using Borg technology!
Nux!
www.nux.ro
Re: KVM snapshots fail on GlusterFS shared mount point
Posted by Nux! <nu...@li.nux.ro>.
On 20.12.2013 23:04, Nux! wrote:
> Hi,
>
> I know there's a vote for RC, maybe this has some importance. I'm
> running KVM with a GlusterFS shared mount point on
> cloudstack-4.2-223b272, snapshots consistently fail to be backed up.
> They get stuck in "CreatedOnPrimary" status.
>
> The shared mount point is more than functional and the 2ndary storage
> can be mounted manually on both hypervisor and SSVM without problems.
>
> GUI says:
> "Failed to create snapshot due to an internal error creating snapshot
> for volume 15"
>
> Libvirt says:
> 2083: warning : qemuDomainObjBeginJobInternal:839 : Cannot start job
> (query, none) for domain i-2-12-VM; current job is (modify, none)
> owned by (2081, 0)
> 2083: error : qemuDomainObjBeginJobInternal:843 : Timed out during
> operation: cannot acquire state change lock
>
> Management logs says http://tmp.nux.ro/T94-ACSgluster.txt
>
> Has anyone seen this before?
It could be this bug. Unless I'm doing something really stupid this is
quite a release blocker.
https://issues.apache.org/jira/browse/CLOUDSTACK-5393
Happy to test if someone comes can patch it.
Lucian
--
Sent from the Delta quadrant using Borg technology!
Nux!
www.nux.ro