You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Mārtiņš Jakubovičs <ma...@vertigs.lv> on 2014/08/26 12:17:07 UTC
Volume snapshots in same time
Hello,
I test schaduled volume snapshots and when I create in same time rules
for few machines (in my case I got 5 VM's) they all fails except one.
http://pastebin.com/E60emjuJ
How I can solve this?
INFO [c.c.h.HighAvailabilityManagerImpl] (HA-4:ctx-125a21bc) checking
health of usage server
INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-49:ctx-1bf25e45
job-9783) Add job-9783 into job monitoring
INFO [o.a.c.a.c.u.s.CreateSnapshotCmd]
(API-Job-Executor-49:ctx-1bf25e45 job-9783 ctx-9d7e935b) VOLSS:
createSnapshotCmd starts:1409047550501
WARN [c.c.u.d.Merovingian2] (API-Job-Executor-49:ctx-1bf25e45 job-9783
ctx-9d7e935b) Was unable to find lock for the key vm_instance1173 and
thread id 158740505
INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-50:ctx-bb4641db
job-9785) Add job-9785 into job monitoring
INFO [o.a.c.a.c.u.s.CreateSnapshotCmd]
(API-Job-Executor-50:ctx-bb4641db job-9785 ctx-26da1a0f) VOLSS:
createSnapshotCmd starts:1409047550587
WARN [c.c.u.d.Merovingian2] (API-Job-Executor-50:ctx-bb4641db job-9785
ctx-26da1a0f) Was unable to find lock for the key vm_instance1172 and
thread id 736777923
INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-51:ctx-1f378859
job-9787) Add job-9787 into job monitoring
INFO [o.a.c.a.c.u.s.CreateSnapshotCmd]
(API-Job-Executor-51:ctx-1f378859 job-9787 ctx-ae2be6e3) VOLSS:
createSnapshotCmd starts:1409047550660
WARN [c.c.u.d.Merovingian2] (API-Job-Executor-51:ctx-1f378859 job-9787
ctx-ae2be6e3) Was unable to find lock for the key vm_instance1171 and
thread id 142175984
INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-52:ctx-197114de
job-9788) Add job-9788 into job monitoring
INFO [o.a.c.a.c.u.s.CreateSnapshotCmd]
(API-Job-Executor-52:ctx-197114de job-9788 ctx-f29734ce) VOLSS:
createSnapshotCmd starts:1409047550750
INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-53:ctx-1e698b15
job-9790) Add job-9790 into job monitoring
WARN [c.c.u.d.Merovingian2] (API-Job-Executor-52:ctx-197114de job-9788
ctx-f29734ce) Was unable to find lock for the key vm_instance1170 and
thread id 1643232589
INFO [o.a.c.a.c.u.s.CreateSnapshotCmd]
(API-Job-Executor-53:ctx-1e698b15 job-9790 ctx-97cf6519) VOLSS:
createSnapshotCmd starts:1409047550795
WARN [c.c.u.d.Merovingian2] (API-Job-Executor-53:ctx-1e698b15 job-9790
ctx-97cf6519) Was unable to find lock for the key vm_instance1168 and
thread id 1037433024
INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-68:ctx-6e51f025
job-9783/job-9784) Add job-9784 into job monitoring
INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-69:ctx-cd90b5b2
job-9785/job-9786) Add job-9786 into job monitoring
INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-70:ctx-d44cfd70
job-9787/job-9789) Add job-9789 into job monitoring
INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-71:ctx-cf2e5ec2
job-9788/job-9791) Add job-9791 into job monitoring
INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-72:ctx-48cef8a6
job-9790/job-9792) Add job-9792 into job monitoring
WARN [o.a.c.h.x.XenServerResourceNewBase]
(DirectAgent-303:ctx-c4ae6319) No event for task
OpaqueRef:d7e9133b-f9bb-59e6-720c-ab4e79e0feff
WARN [o.a.c.h.x.XenServerResourceNewBase]
(DirectAgent-119:ctx-cac68540) No event for task
OpaqueRef:04cc04cf-f676-d187-25f7-801572973d7a
WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-119:ctx-cac68540) Host
10.10.10.66 OpaqueRef:18b050f6-a1db-1ba5-f290-bcd58811caad: Unable to
remove SR
WARN [c.c.h.x.r.XenServerStorageProcessor]
(DirectAgent-119:ctx-cac68540) BackupSnapshot Failed due to No event for
task OpaqueRef:04cc04cf-f676-d187-25f7-801572973d7a
java.util.concurrent.TimeoutException: No event for task
OpaqueRef:04cc04cf-f676-d187-25f7-801572973d7a
at
org.apache.cloudstack.hypervisor.xenserver.XenServerResourceNewBase.waitForTask(XenServerResourceNewBase.java:124)
at
com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.backupSnapshot(Xenserver625StorageProcessor.java:430)
at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:94)
at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:52)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:546)
at
com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:61)
at
com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:102)
at
com.cloud.hypervisor.xen.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)
at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-70:ctx-d44cfd70
job-9787/job-9789) Remove job-9789 from job monitoring
INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-51:ctx-1f378859
job-9787) Remove job-9787 from job monitoring
WARN [c.c.h.x.r.XenServerStorageProcessor]
(DirectAgent-303:ctx-c4ae6319) BackupSnapshot Failed due to No event for
task OpaqueRef:d7e9133b-f9bb-59e6-720c-ab4e79e0feff
java.util.concurrent.TimeoutException: No event for task
OpaqueRef:d7e9133b-f9bb-59e6-720c-ab4e79e0feff
at
org.apache.cloudstack.hypervisor.xenserver.XenServerResourceNewBase.waitForTask(XenServerResourceNewBase.java:124)
at
com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.backupSnapshot(Xenserver625StorageProcessor.java:430)
at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:94)
at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:52)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:546)
at
com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:61)
at
com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:102)
at
com.cloud.hypervisor.xen.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)
at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-68:ctx-6e51f025
job-9783/job-9784) Remove job-9784 from job monitoring
INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-49:ctx-1bf25e45
job-9783) Remove job-9783 from job monitoring
WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-176:ctx-830c05b9)
callHostPlugin failed for cmd: mountNfsSecondaryStorage with args
remoteDir: 10.10.10.137:/vol/cs_xs8_sec_vol_04082014_173817, localDir:
/var/cloud_mount/543e60ad-5a48-3644-a8c5-5b3ebd4e0ae3, due to You gave
an invalid session reference. It may have been invalidated by a server
restart, or timed out. You should get a new session handle, using one
of the session.login_ calls. This error does not invalidate the current
connection. The handle parameter echoes the bad value given.
You gave an invalid session reference. It may have been invalidated by
a server restart, or timed out. You should get a new session handle,
using one of the session.login_ calls. This error does not invalidate
the current connection. The handle parameter echoes the bad value given.
at com.xensource.xenapi.Types.checkResponse(Types.java:218)
at com.xensource.xenapi.Connection.dispatch(Connection.java:350)
at
com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:456)
at com.xensource.xenapi.Event.properFrom(Event.java:310)
at
org.apache.cloudstack.hypervisor.xenserver.XenServerResourceNewBase.waitForTask(XenServerResourceNewBase.java:116)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.callHostPluginAsync(CitrixResourceBase.java:3506)
at
com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.mountNfs(Xenserver625StorageProcessor.java:68)
at
com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.backupSnapshot(Xenserver625StorageProcessor.java:418)
at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:94)
at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:52)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:546)
at
com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:61)
at
com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:102)
at
com.cloud.hypervisor.xen.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)
at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
WARN [c.c.h.x.r.XenServerStorageProcessor]
(DirectAgent-176:ctx-830c05b9) Could not mount secondary storage
10.10.10.137:/vol/cs_xs8_sec_vol_04082014_173817 on host
/var/cloud_mount/543e60ad-5a48-3644-a8c5-5b3ebd4e0ae3
WARN [c.c.h.x.r.XenServerStorageProcessor]
(DirectAgent-176:ctx-830c05b9) BackupSnapshot Failed due to Could not
mount secondary storage 10.10.10.137:/vol/cs_xs8_sec_vol_04082014_173817
on host /var/cloud_mount/543e60ad-5a48-3644-a8c5-5b3ebd4e0ae3
com.cloud.utils.exception.CloudRuntimeException: Could not mount
secondary storage 10.10.10.137:/vol/cs_xs8_sec_vol_04082014_173817 on
host /var/cloud_mount/543e60ad-5a48-3644-a8c5-5b3ebd4e0ae3
at
com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.mountNfs(Xenserver625StorageProcessor.java:73)
at
com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.backupSnapshot(Xenserver625StorageProcessor.java:418)
at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:94)
at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:52)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:546)
at
com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:61)
at
com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:102)
at
com.cloud.hypervisor.xen.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)
at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-288:ctx-0efa2c48) Task
failed! Task record: uuid:
cd20da12-7cbc-18cf-9629-c74dd1e3fa53
nameLabel: Async.VDI.copy
nameDescription:
allowedOperations: [cancel]
currentOperations: {}
created: Tue Aug 26 13:07:15 EEST 2014
finished: Thu Jan 01 03:00:00 EET 1970
status: pending
residentOn: com.xensource.xenapi.Host@40960732
progress: 0.0
type: <none/>
result:
errorInfo: []
otherConfig: {}
subtaskOf: com.xensource.xenapi.Task@aaf13f6f
subtasks: [com.xensource.xenapi.Task@1e6e981e]
WARN [c.c.h.x.r.XenServerStorageProcessor]
(DirectAgent-288:ctx-0efa2c48) unable to destroy
task(OpaqueRef:be099f81-ab1b-0808-6652-af015577a575) due to You gave an
invalid object reference. The object may have recently been deleted.
The class parameter gives the type of reference given, and the handle
parameter echoes the bad value given.
INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-72:ctx-48cef8a6
job-9790/job-9792) Remove job-9792 from job monitoring
INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-53:ctx-1e698b15
job-9790) Remove job-9790 from job monitoring
WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-288:ctx-0efa2c48) Host
10.10.10.66 OpaqueRef:bf007bb1-b83c-9e14-75f7-6790b52de61b: Unable to
remove SR
WARN [c.c.h.x.r.XenServerStorageProcessor]
(DirectAgent-288:ctx-0efa2c48) BackupSnapshot Failed due to Task failed!
Task record: uuid: cd20da12-7cbc-18cf-9629-c74dd1e3fa53
nameLabel: Async.VDI.copy
nameDescription:
allowedOperations: [cancel]
currentOperations: {}
created: Tue Aug 26 13:07:15 EEST 2014
finished: Thu Jan 01 03:00:00 EET 1970
status: pending
residentOn: com.xensource.xenapi.Host@40960732
progress: 0.0
type: <none/>
result:
errorInfo: []
otherConfig: {}
subtaskOf: com.xensource.xenapi.Task@aaf13f6f
subtasks: [com.xensource.xenapi.Task@1e6e981e]
Task failed! Task record: uuid:
cd20da12-7cbc-18cf-9629-c74dd1e3fa53
nameLabel: Async.VDI.copy
nameDescription:
allowedOperations: [cancel]
currentOperations: {}
created: Tue Aug 26 13:07:15 EEST 2014
finished: Thu Jan 01 03:00:00 EET 1970
status: pending
residentOn: com.xensource.xenapi.Host@40960732
progress: 0.0
type: <none/>
result:
errorInfo: []
otherConfig: {}
subtaskOf: com.xensource.xenapi.Task@aaf13f6f
subtasks: [com.xensource.xenapi.Task@1e6e981e]
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.checkForSuccess(CitrixResourceBase.java:3293)
at
com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.backupSnapshot(Xenserver625StorageProcessor.java:431)
at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:94)
at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:52)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:546)
at
com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:61)
at
com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:102)
at
com.cloud.hypervisor.xen.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)
at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-71:ctx-cf2e5ec2
job-9788/job-9791) Remove job-9791 from job monitoring
INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-52:ctx-197114de
job-9788) Remove job-9788 from job monitoring
WARN [o.a.c.alerts] (CapacityChecker:ctx-3116e57f) alertType:: 3 //
dataCenterId:: 5 // podId:: 5 // clusterId:: null // message:: System
Alert: Remaining unallocated Storage is low in cluster DEMO Cluster-1
pod Demo-Rack-1 of availability zone Demo
WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-2ca2a925) Task
(job-9785) has been pending for 168 seconds
WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-2ca2a925) Task
(job-9786) has been pending for 166 seconds
WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-bd10b11d) Task
(job-9785) has been pending for 168 seconds
WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-bd10b11d) Task
(job-9786) has been pending for 166 seconds
WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-423be8e1) Task
(job-9785) has been pending for 222 seconds
WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-423be8e1) Task
(job-9786) has been pending for 221 seconds
INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-69:ctx-cd90b5b2
job-9785/job-9786) Remove job-9786 from job monitoring
INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-50:ctx-bb4641db
job-9785) Remove job-9785 from job monitoring
Re: Volume snapshots in same time
Posted by Vihar <vi...@gmail.com>.
Hi Martin,
This happens when multiple threads tries to access the same table at same
time. Its a race case situation.
Regards
Vihar K
On Fri, Aug 29, 2014 at 5:08 AM, Kirk Kosinski <ki...@gmail.com>
wrote:
> Hi, Mārtiņš. Unfortunately CloudStack doesn't support this yet.
>
> Best regards,
> Kirk
>
> On 08/26/2014 03:17 AM, Mārtiņš Jakubovičs wrote:
> > Hello,
> >
> > I test schaduled volume snapshots and when I create in same time rules
> > for few machines (in my case I got 5 VM's) they all fails except one.
> >
> > http://pastebin.com/E60emjuJ
> >
> > How I can solve this?
> >
> > INFO [c.c.h.HighAvailabilityManagerImpl] (HA-4:ctx-125a21bc) checking
> > health of usage server
> > INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-49:ctx-1bf25e45
> > job-9783) Add job-9783 into job monitoring
> > INFO [o.a.c.a.c.u.s.CreateSnapshotCmd]
> > (API-Job-Executor-49:ctx-1bf25e45 job-9783 ctx-9d7e935b) VOLSS:
> > createSnapshotCmd starts:1409047550501
> > WARN [c.c.u.d.Merovingian2] (API-Job-Executor-49:ctx-1bf25e45 job-9783
> > ctx-9d7e935b) Was unable to find lock for the key vm_instance1173 and
> > thread id 158740505
> > INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-50:ctx-bb4641db
> > job-9785) Add job-9785 into job monitoring
> > INFO [o.a.c.a.c.u.s.CreateSnapshotCmd]
> > (API-Job-Executor-50:ctx-bb4641db job-9785 ctx-26da1a0f) VOLSS:
> > createSnapshotCmd starts:1409047550587
> > WARN [c.c.u.d.Merovingian2] (API-Job-Executor-50:ctx-bb4641db job-9785
> > ctx-26da1a0f) Was unable to find lock for the key vm_instance1172 and
> > thread id 736777923
> > INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-51:ctx-1f378859
> > job-9787) Add job-9787 into job monitoring
> > INFO [o.a.c.a.c.u.s.CreateSnapshotCmd]
> > (API-Job-Executor-51:ctx-1f378859 job-9787 ctx-ae2be6e3) VOLSS:
> > createSnapshotCmd starts:1409047550660
> > WARN [c.c.u.d.Merovingian2] (API-Job-Executor-51:ctx-1f378859 job-9787
> > ctx-ae2be6e3) Was unable to find lock for the key vm_instance1171 and
> > thread id 142175984
> > INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-52:ctx-197114de
> > job-9788) Add job-9788 into job monitoring
> > INFO [o.a.c.a.c.u.s.CreateSnapshotCmd]
> > (API-Job-Executor-52:ctx-197114de job-9788 ctx-f29734ce) VOLSS:
> > createSnapshotCmd starts:1409047550750
> > INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-53:ctx-1e698b15
> > job-9790) Add job-9790 into job monitoring
> > WARN [c.c.u.d.Merovingian2] (API-Job-Executor-52:ctx-197114de job-9788
> > ctx-f29734ce) Was unable to find lock for the key vm_instance1170 and
> > thread id 1643232589
> > INFO [o.a.c.a.c.u.s.CreateSnapshotCmd]
> > (API-Job-Executor-53:ctx-1e698b15 job-9790 ctx-97cf6519) VOLSS:
> > createSnapshotCmd starts:1409047550795
> > WARN [c.c.u.d.Merovingian2] (API-Job-Executor-53:ctx-1e698b15 job-9790
> > ctx-97cf6519) Was unable to find lock for the key vm_instance1168 and
> > thread id 1037433024
> > INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-68:ctx-6e51f025
> > job-9783/job-9784) Add job-9784 into job monitoring
> > INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-69:ctx-cd90b5b2
> > job-9785/job-9786) Add job-9786 into job monitoring
> > INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-70:ctx-d44cfd70
> > job-9787/job-9789) Add job-9789 into job monitoring
> > INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-71:ctx-cf2e5ec2
> > job-9788/job-9791) Add job-9791 into job monitoring
> > INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-72:ctx-48cef8a6
> > job-9790/job-9792) Add job-9792 into job monitoring
> > WARN [o.a.c.h.x.XenServerResourceNewBase]
> > (DirectAgent-303:ctx-c4ae6319) No event for task
> > OpaqueRef:d7e9133b-f9bb-59e6-720c-ab4e79e0feff
> > WARN [o.a.c.h.x.XenServerResourceNewBase]
> > (DirectAgent-119:ctx-cac68540) No event for task
> > OpaqueRef:04cc04cf-f676-d187-25f7-801572973d7a
> > WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-119:ctx-cac68540) Host
> > 10.10.10.66 OpaqueRef:18b050f6-a1db-1ba5-f290-bcd58811caad: Unable to
> > remove SR
> > WARN [c.c.h.x.r.XenServerStorageProcessor]
> > (DirectAgent-119:ctx-cac68540) BackupSnapshot Failed due to No event for
> > task OpaqueRef:04cc04cf-f676-d187-25f7-801572973d7a
> > java.util.concurrent.TimeoutException: No event for task
> > OpaqueRef:04cc04cf-f676-d187-25f7-801572973d7a
> > at
> >
> org.apache.cloudstack.hypervisor.xenserver.XenServerResourceNewBase.waitForTask(XenServerResourceNewBase.java:124)
> > at
> >
> com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.backupSnapshot(Xenserver625StorageProcessor.java:430)
> > at
> >
> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:94)
> > at
> >
> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:52)
> > at
> >
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:546)
> > at
> >
> com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:61)
> > at
> >
> com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:102)
> > at
> >
> com.cloud.hypervisor.xen.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)
> > at
> >
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216)
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> > at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > at java.lang.Thread.run(Thread.java:745)
> > INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-70:ctx-d44cfd70
> > job-9787/job-9789) Remove job-9789 from job monitoring
> > INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-51:ctx-1f378859
> > job-9787) Remove job-9787 from job monitoring
> > WARN [c.c.h.x.r.XenServerStorageProcessor]
> > (DirectAgent-303:ctx-c4ae6319) BackupSnapshot Failed due to No event for
> > task OpaqueRef:d7e9133b-f9bb-59e6-720c-ab4e79e0feff
> > java.util.concurrent.TimeoutException: No event for task
> > OpaqueRef:d7e9133b-f9bb-59e6-720c-ab4e79e0feff
> > at
> >
> org.apache.cloudstack.hypervisor.xenserver.XenServerResourceNewBase.waitForTask(XenServerResourceNewBase.java:124)
> > at
> >
> com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.backupSnapshot(Xenserver625StorageProcessor.java:430)
> > at
> >
> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:94)
> > at
> >
> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:52)
> > at
> >
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:546)
> > at
> >
> com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:61)
> > at
> >
> com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:102)
> > at
> >
> com.cloud.hypervisor.xen.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)
> > at
> >
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216)
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> > at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > at java.lang.Thread.run(Thread.java:745)
> > INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-68:ctx-6e51f025
> > job-9783/job-9784) Remove job-9784 from job monitoring
> > INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-49:ctx-1bf25e45
> > job-9783) Remove job-9783 from job monitoring
> > WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-176:ctx-830c05b9)
> > callHostPlugin failed for cmd: mountNfsSecondaryStorage with args
> > remoteDir: 10.10.10.137:/vol/cs_xs8_sec_vol_04082014_173817, localDir:
> > /var/cloud_mount/543e60ad-5a48-3644-a8c5-5b3ebd4e0ae3, due to You gave
> > an invalid session reference. It may have been invalidated by a server
> > restart, or timed out. You should get a new session handle, using one
> > of the session.login_ calls. This error does not invalidate the current
> > connection. The handle parameter echoes the bad value given.
> > You gave an invalid session reference. It may have been invalidated by
> > a server restart, or timed out. You should get a new session handle,
> > using one of the session.login_ calls. This error does not invalidate
> > the current connection. The handle parameter echoes the bad value given.
> > at com.xensource.xenapi.Types.checkResponse(Types.java:218)
> > at com.xensource.xenapi.Connection.dispatch(Connection.java:350)
> > at
> >
> com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:456)
> > at com.xensource.xenapi.Event.properFrom(Event.java:310)
> > at
> >
> org.apache.cloudstack.hypervisor.xenserver.XenServerResourceNewBase.waitForTask(XenServerResourceNewBase.java:116)
> > at
> >
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.callHostPluginAsync(CitrixResourceBase.java:3506)
> > at
> >
> com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.mountNfs(Xenserver625StorageProcessor.java:68)
> > at
> >
> com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.backupSnapshot(Xenserver625StorageProcessor.java:418)
> > at
> >
> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:94)
> > at
> >
> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:52)
> > at
> >
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:546)
> > at
> >
> com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:61)
> > at
> >
> com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:102)
> > at
> >
> com.cloud.hypervisor.xen.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)
> > at
> >
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216)
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> > at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > at java.lang.Thread.run(Thread.java:745)
> > WARN [c.c.h.x.r.XenServerStorageProcessor]
> > (DirectAgent-176:ctx-830c05b9) Could not mount secondary storage
> > 10.10.10.137:/vol/cs_xs8_sec_vol_04082014_173817 on host
> > /var/cloud_mount/543e60ad-5a48-3644-a8c5-5b3ebd4e0ae3
> > WARN [c.c.h.x.r.XenServerStorageProcessor]
> > (DirectAgent-176:ctx-830c05b9) BackupSnapshot Failed due to Could not
> > mount secondary storage 10.10.10.137:/vol/cs_xs8_sec_vol_04082014_173817
> > on host /var/cloud_mount/543e60ad-5a48-3644-a8c5-5b3ebd4e0ae3
> > com.cloud.utils.exception.CloudRuntimeException: Could not mount
> > secondary storage 10.10.10.137:/vol/cs_xs8_sec_vol_04082014_173817 on
> > host /var/cloud_mount/543e60ad-5a48-3644-a8c5-5b3ebd4e0ae3
> > at
> >
> com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.mountNfs(Xenserver625StorageProcessor.java:73)
> > at
> >
> com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.backupSnapshot(Xenserver625StorageProcessor.java:418)
> > at
> >
> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:94)
> > at
> >
> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:52)
> > at
> >
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:546)
> > at
> >
> com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:61)
> > at
> >
> com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:102)
> > at
> >
> com.cloud.hypervisor.xen.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)
> > at
> >
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216)
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> > at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > at java.lang.Thread.run(Thread.java:745)
> > WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-288:ctx-0efa2c48) Task
> > failed! Task record: uuid:
> > cd20da12-7cbc-18cf-9629-c74dd1e3fa53
> > nameLabel: Async.VDI.copy
> > nameDescription:
> > allowedOperations: [cancel]
> > currentOperations: {}
> > created: Tue Aug 26 13:07:15 EEST 2014
> > finished: Thu Jan 01 03:00:00 EET 1970
> > status: pending
> > residentOn: com.xensource.xenapi.Host@40960732
> > progress: 0.0
> > type: <none/>
> > result:
> > errorInfo: []
> > otherConfig: {}
> > subtaskOf: com.xensource.xenapi.Task@aaf13f6f
> > subtasks: [com.xensource.xenapi.Task@1e6e981e]
> >
> > WARN [c.c.h.x.r.XenServerStorageProcessor]
> > (DirectAgent-288:ctx-0efa2c48) unable to destroy
> > task(OpaqueRef:be099f81-ab1b-0808-6652-af015577a575) due to You gave an
> > invalid object reference. The object may have recently been deleted.
> > The class parameter gives the type of reference given, and the handle
> > parameter echoes the bad value given.
> > INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-72:ctx-48cef8a6
> > job-9790/job-9792) Remove job-9792 from job monitoring
> > INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-53:ctx-1e698b15
> > job-9790) Remove job-9790 from job monitoring
> > WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-288:ctx-0efa2c48) Host
> > 10.10.10.66 OpaqueRef:bf007bb1-b83c-9e14-75f7-6790b52de61b: Unable to
> > remove SR
> > WARN [c.c.h.x.r.XenServerStorageProcessor]
> > (DirectAgent-288:ctx-0efa2c48) BackupSnapshot Failed due to Task failed!
> > Task record: uuid: cd20da12-7cbc-18cf-9629-c74dd1e3fa53
> > nameLabel: Async.VDI.copy
> > nameDescription:
> > allowedOperations: [cancel]
> > currentOperations: {}
> > created: Tue Aug 26 13:07:15 EEST 2014
> > finished: Thu Jan 01 03:00:00 EET 1970
> > status: pending
> > residentOn: com.xensource.xenapi.Host@40960732
> > progress: 0.0
> > type: <none/>
> > result:
> > errorInfo: []
> > otherConfig: {}
> > subtaskOf: com.xensource.xenapi.Task@aaf13f6f
> > subtasks: [com.xensource.xenapi.Task@1e6e981e]
> >
> > Task failed! Task record: uuid:
> > cd20da12-7cbc-18cf-9629-c74dd1e3fa53
> > nameLabel: Async.VDI.copy
> > nameDescription:
> > allowedOperations: [cancel]
> > currentOperations: {}
> > created: Tue Aug 26 13:07:15 EEST 2014
> > finished: Thu Jan 01 03:00:00 EET 1970
> > status: pending
> > residentOn: com.xensource.xenapi.Host@40960732
> > progress: 0.0
> > type: <none/>
> > result:
> > errorInfo: []
> > otherConfig: {}
> > subtaskOf: com.xensource.xenapi.Task@aaf13f6f
> > subtasks: [com.xensource.xenapi.Task@1e6e981e]
> >
> > at
> >
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.checkForSuccess(CitrixResourceBase.java:3293)
> > at
> >
> com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.backupSnapshot(Xenserver625StorageProcessor.java:431)
> > at
> >
> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:94)
> > at
> >
> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:52)
> > at
> >
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:546)
> > at
> >
> com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:61)
> > at
> >
> com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:102)
> > at
> >
> com.cloud.hypervisor.xen.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)
> > at
> >
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216)
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> > at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > at java.lang.Thread.run(Thread.java:745)
> > INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-71:ctx-cf2e5ec2
> > job-9788/job-9791) Remove job-9791 from job monitoring
> > INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-52:ctx-197114de
> > job-9788) Remove job-9788 from job monitoring
> > WARN [o.a.c.alerts] (CapacityChecker:ctx-3116e57f) alertType:: 3 //
> > dataCenterId:: 5 // podId:: 5 // clusterId:: null // message:: System
> > Alert: Remaining unallocated Storage is low in cluster DEMO Cluster-1
> > pod Demo-Rack-1 of availability zone Demo
> > WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-2ca2a925) Task
> > (job-9785) has been pending for 168 seconds
> > WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-2ca2a925) Task
> > (job-9786) has been pending for 166 seconds
> > WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-bd10b11d) Task
> > (job-9785) has been pending for 168 seconds
> > WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-bd10b11d) Task
> > (job-9786) has been pending for 166 seconds
> > WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-423be8e1) Task
> > (job-9785) has been pending for 222 seconds
> > WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-423be8e1) Task
> > (job-9786) has been pending for 221 seconds
> > INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-69:ctx-cd90b5b2
> > job-9785/job-9786) Remove job-9786 from job monitoring
> > INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-50:ctx-bb4641db
> > job-9785) Remove job-9785 from job monitoring
> >
>
>
Re: Volume snapshots in same time
Posted by Kirk Kosinski <ki...@gmail.com>.
Hi, Mārtiņš. Unfortunately CloudStack doesn't support this yet.
Best regards,
Kirk
On 08/26/2014 03:17 AM, Mārtiņš Jakubovičs wrote:
> Hello,
>
> I test schaduled volume snapshots and when I create in same time rules
> for few machines (in my case I got 5 VM's) they all fails except one.
>
> http://pastebin.com/E60emjuJ
>
> How I can solve this?
>
> INFO [c.c.h.HighAvailabilityManagerImpl] (HA-4:ctx-125a21bc) checking
> health of usage server
> INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-49:ctx-1bf25e45
> job-9783) Add job-9783 into job monitoring
> INFO [o.a.c.a.c.u.s.CreateSnapshotCmd]
> (API-Job-Executor-49:ctx-1bf25e45 job-9783 ctx-9d7e935b) VOLSS:
> createSnapshotCmd starts:1409047550501
> WARN [c.c.u.d.Merovingian2] (API-Job-Executor-49:ctx-1bf25e45 job-9783
> ctx-9d7e935b) Was unable to find lock for the key vm_instance1173 and
> thread id 158740505
> INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-50:ctx-bb4641db
> job-9785) Add job-9785 into job monitoring
> INFO [o.a.c.a.c.u.s.CreateSnapshotCmd]
> (API-Job-Executor-50:ctx-bb4641db job-9785 ctx-26da1a0f) VOLSS:
> createSnapshotCmd starts:1409047550587
> WARN [c.c.u.d.Merovingian2] (API-Job-Executor-50:ctx-bb4641db job-9785
> ctx-26da1a0f) Was unable to find lock for the key vm_instance1172 and
> thread id 736777923
> INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-51:ctx-1f378859
> job-9787) Add job-9787 into job monitoring
> INFO [o.a.c.a.c.u.s.CreateSnapshotCmd]
> (API-Job-Executor-51:ctx-1f378859 job-9787 ctx-ae2be6e3) VOLSS:
> createSnapshotCmd starts:1409047550660
> WARN [c.c.u.d.Merovingian2] (API-Job-Executor-51:ctx-1f378859 job-9787
> ctx-ae2be6e3) Was unable to find lock for the key vm_instance1171 and
> thread id 142175984
> INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-52:ctx-197114de
> job-9788) Add job-9788 into job monitoring
> INFO [o.a.c.a.c.u.s.CreateSnapshotCmd]
> (API-Job-Executor-52:ctx-197114de job-9788 ctx-f29734ce) VOLSS:
> createSnapshotCmd starts:1409047550750
> INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-53:ctx-1e698b15
> job-9790) Add job-9790 into job monitoring
> WARN [c.c.u.d.Merovingian2] (API-Job-Executor-52:ctx-197114de job-9788
> ctx-f29734ce) Was unable to find lock for the key vm_instance1170 and
> thread id 1643232589
> INFO [o.a.c.a.c.u.s.CreateSnapshotCmd]
> (API-Job-Executor-53:ctx-1e698b15 job-9790 ctx-97cf6519) VOLSS:
> createSnapshotCmd starts:1409047550795
> WARN [c.c.u.d.Merovingian2] (API-Job-Executor-53:ctx-1e698b15 job-9790
> ctx-97cf6519) Was unable to find lock for the key vm_instance1168 and
> thread id 1037433024
> INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-68:ctx-6e51f025
> job-9783/job-9784) Add job-9784 into job monitoring
> INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-69:ctx-cd90b5b2
> job-9785/job-9786) Add job-9786 into job monitoring
> INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-70:ctx-d44cfd70
> job-9787/job-9789) Add job-9789 into job monitoring
> INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-71:ctx-cf2e5ec2
> job-9788/job-9791) Add job-9791 into job monitoring
> INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-72:ctx-48cef8a6
> job-9790/job-9792) Add job-9792 into job monitoring
> WARN [o.a.c.h.x.XenServerResourceNewBase]
> (DirectAgent-303:ctx-c4ae6319) No event for task
> OpaqueRef:d7e9133b-f9bb-59e6-720c-ab4e79e0feff
> WARN [o.a.c.h.x.XenServerResourceNewBase]
> (DirectAgent-119:ctx-cac68540) No event for task
> OpaqueRef:04cc04cf-f676-d187-25f7-801572973d7a
> WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-119:ctx-cac68540) Host
> 10.10.10.66 OpaqueRef:18b050f6-a1db-1ba5-f290-bcd58811caad: Unable to
> remove SR
> WARN [c.c.h.x.r.XenServerStorageProcessor]
> (DirectAgent-119:ctx-cac68540) BackupSnapshot Failed due to No event for
> task OpaqueRef:04cc04cf-f676-d187-25f7-801572973d7a
> java.util.concurrent.TimeoutException: No event for task
> OpaqueRef:04cc04cf-f676-d187-25f7-801572973d7a
> at
> org.apache.cloudstack.hypervisor.xenserver.XenServerResourceNewBase.waitForTask(XenServerResourceNewBase.java:124)
> at
> com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.backupSnapshot(Xenserver625StorageProcessor.java:430)
> at
> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:94)
> at
> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:52)
> at
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:546)
> at
> com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:61)
> at
> com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:102)
> at
> com.cloud.hypervisor.xen.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-70:ctx-d44cfd70
> job-9787/job-9789) Remove job-9789 from job monitoring
> INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-51:ctx-1f378859
> job-9787) Remove job-9787 from job monitoring
> WARN [c.c.h.x.r.XenServerStorageProcessor]
> (DirectAgent-303:ctx-c4ae6319) BackupSnapshot Failed due to No event for
> task OpaqueRef:d7e9133b-f9bb-59e6-720c-ab4e79e0feff
> java.util.concurrent.TimeoutException: No event for task
> OpaqueRef:d7e9133b-f9bb-59e6-720c-ab4e79e0feff
> at
> org.apache.cloudstack.hypervisor.xenserver.XenServerResourceNewBase.waitForTask(XenServerResourceNewBase.java:124)
> at
> com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.backupSnapshot(Xenserver625StorageProcessor.java:430)
> at
> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:94)
> at
> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:52)
> at
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:546)
> at
> com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:61)
> at
> com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:102)
> at
> com.cloud.hypervisor.xen.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-68:ctx-6e51f025
> job-9783/job-9784) Remove job-9784 from job monitoring
> INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-49:ctx-1bf25e45
> job-9783) Remove job-9783 from job monitoring
> WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-176:ctx-830c05b9)
> callHostPlugin failed for cmd: mountNfsSecondaryStorage with args
> remoteDir: 10.10.10.137:/vol/cs_xs8_sec_vol_04082014_173817, localDir:
> /var/cloud_mount/543e60ad-5a48-3644-a8c5-5b3ebd4e0ae3, due to You gave
> an invalid session reference. It may have been invalidated by a server
> restart, or timed out. You should get a new session handle, using one
> of the session.login_ calls. This error does not invalidate the current
> connection. The handle parameter echoes the bad value given.
> You gave an invalid session reference. It may have been invalidated by
> a server restart, or timed out. You should get a new session handle,
> using one of the session.login_ calls. This error does not invalidate
> the current connection. The handle parameter echoes the bad value given.
> at com.xensource.xenapi.Types.checkResponse(Types.java:218)
> at com.xensource.xenapi.Connection.dispatch(Connection.java:350)
> at
> com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:456)
> at com.xensource.xenapi.Event.properFrom(Event.java:310)
> at
> org.apache.cloudstack.hypervisor.xenserver.XenServerResourceNewBase.waitForTask(XenServerResourceNewBase.java:116)
> at
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.callHostPluginAsync(CitrixResourceBase.java:3506)
> at
> com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.mountNfs(Xenserver625StorageProcessor.java:68)
> at
> com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.backupSnapshot(Xenserver625StorageProcessor.java:418)
> at
> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:94)
> at
> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:52)
> at
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:546)
> at
> com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:61)
> at
> com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:102)
> at
> com.cloud.hypervisor.xen.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> WARN [c.c.h.x.r.XenServerStorageProcessor]
> (DirectAgent-176:ctx-830c05b9) Could not mount secondary storage
> 10.10.10.137:/vol/cs_xs8_sec_vol_04082014_173817 on host
> /var/cloud_mount/543e60ad-5a48-3644-a8c5-5b3ebd4e0ae3
> WARN [c.c.h.x.r.XenServerStorageProcessor]
> (DirectAgent-176:ctx-830c05b9) BackupSnapshot Failed due to Could not
> mount secondary storage 10.10.10.137:/vol/cs_xs8_sec_vol_04082014_173817
> on host /var/cloud_mount/543e60ad-5a48-3644-a8c5-5b3ebd4e0ae3
> com.cloud.utils.exception.CloudRuntimeException: Could not mount
> secondary storage 10.10.10.137:/vol/cs_xs8_sec_vol_04082014_173817 on
> host /var/cloud_mount/543e60ad-5a48-3644-a8c5-5b3ebd4e0ae3
> at
> com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.mountNfs(Xenserver625StorageProcessor.java:73)
> at
> com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.backupSnapshot(Xenserver625StorageProcessor.java:418)
> at
> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:94)
> at
> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:52)
> at
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:546)
> at
> com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:61)
> at
> com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:102)
> at
> com.cloud.hypervisor.xen.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-288:ctx-0efa2c48) Task
> failed! Task record: uuid:
> cd20da12-7cbc-18cf-9629-c74dd1e3fa53
> nameLabel: Async.VDI.copy
> nameDescription:
> allowedOperations: [cancel]
> currentOperations: {}
> created: Tue Aug 26 13:07:15 EEST 2014
> finished: Thu Jan 01 03:00:00 EET 1970
> status: pending
> residentOn: com.xensource.xenapi.Host@40960732
> progress: 0.0
> type: <none/>
> result:
> errorInfo: []
> otherConfig: {}
> subtaskOf: com.xensource.xenapi.Task@aaf13f6f
> subtasks: [com.xensource.xenapi.Task@1e6e981e]
>
> WARN [c.c.h.x.r.XenServerStorageProcessor]
> (DirectAgent-288:ctx-0efa2c48) unable to destroy
> task(OpaqueRef:be099f81-ab1b-0808-6652-af015577a575) due to You gave an
> invalid object reference. The object may have recently been deleted.
> The class parameter gives the type of reference given, and the handle
> parameter echoes the bad value given.
> INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-72:ctx-48cef8a6
> job-9790/job-9792) Remove job-9792 from job monitoring
> INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-53:ctx-1e698b15
> job-9790) Remove job-9790 from job monitoring
> WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-288:ctx-0efa2c48) Host
> 10.10.10.66 OpaqueRef:bf007bb1-b83c-9e14-75f7-6790b52de61b: Unable to
> remove SR
> WARN [c.c.h.x.r.XenServerStorageProcessor]
> (DirectAgent-288:ctx-0efa2c48) BackupSnapshot Failed due to Task failed!
> Task record: uuid: cd20da12-7cbc-18cf-9629-c74dd1e3fa53
> nameLabel: Async.VDI.copy
> nameDescription:
> allowedOperations: [cancel]
> currentOperations: {}
> created: Tue Aug 26 13:07:15 EEST 2014
> finished: Thu Jan 01 03:00:00 EET 1970
> status: pending
> residentOn: com.xensource.xenapi.Host@40960732
> progress: 0.0
> type: <none/>
> result:
> errorInfo: []
> otherConfig: {}
> subtaskOf: com.xensource.xenapi.Task@aaf13f6f
> subtasks: [com.xensource.xenapi.Task@1e6e981e]
>
> Task failed! Task record: uuid:
> cd20da12-7cbc-18cf-9629-c74dd1e3fa53
> nameLabel: Async.VDI.copy
> nameDescription:
> allowedOperations: [cancel]
> currentOperations: {}
> created: Tue Aug 26 13:07:15 EEST 2014
> finished: Thu Jan 01 03:00:00 EET 1970
> status: pending
> residentOn: com.xensource.xenapi.Host@40960732
> progress: 0.0
> type: <none/>
> result:
> errorInfo: []
> otherConfig: {}
> subtaskOf: com.xensource.xenapi.Task@aaf13f6f
> subtasks: [com.xensource.xenapi.Task@1e6e981e]
>
> at
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.checkForSuccess(CitrixResourceBase.java:3293)
> at
> com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.backupSnapshot(Xenserver625StorageProcessor.java:431)
> at
> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:94)
> at
> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:52)
> at
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:546)
> at
> com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:61)
> at
> com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:102)
> at
> com.cloud.hypervisor.xen.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-71:ctx-cf2e5ec2
> job-9788/job-9791) Remove job-9791 from job monitoring
> INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-52:ctx-197114de
> job-9788) Remove job-9788 from job monitoring
> WARN [o.a.c.alerts] (CapacityChecker:ctx-3116e57f) alertType:: 3 //
> dataCenterId:: 5 // podId:: 5 // clusterId:: null // message:: System
> Alert: Remaining unallocated Storage is low in cluster DEMO Cluster-1
> pod Demo-Rack-1 of availability zone Demo
> WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-2ca2a925) Task
> (job-9785) has been pending for 168 seconds
> WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-2ca2a925) Task
> (job-9786) has been pending for 166 seconds
> WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-bd10b11d) Task
> (job-9785) has been pending for 168 seconds
> WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-bd10b11d) Task
> (job-9786) has been pending for 166 seconds
> WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-423be8e1) Task
> (job-9785) has been pending for 222 seconds
> WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-423be8e1) Task
> (job-9786) has been pending for 221 seconds
> INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-69:ctx-cd90b5b2
> job-9785/job-9786) Remove job-9786 from job monitoring
> INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-50:ctx-bb4641db
> job-9785) Remove job-9785 from job monitoring
>