You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@cloudstack.apache.org by "Sangeetha Hariharan (JIRA)" <ji...@apache.org> on 2013/12/04 02:20:35 UTC

[jira] [Updated] (CLOUDSTACK-5356) Xenserver - Failed to create snapshot when secondary store was made unavaibale for about 1.5 hour leaving behind snapshot in " CreatedOnPrimary" state. The subsequent scheduled snapshot also failed

     [ https://issues.apache.org/jira/browse/CLOUDSTACK-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sangeetha Hariharan updated CLOUDSTACK-5356:
--------------------------------------------

    Attachment: Snaposhotfailures.rar

> Xenserver - Failed to create snapshot when secondary store was made unavaibale for about 1.5 hour leaving behind snapshot in " CreatedOnPrimary" state. The subsequent scheduled snapshot  also failed
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CLOUDSTACK-5356
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5356
>             Project: CloudStack
>          Issue Type: Bug
>      Security Level: Public(Anyone can view this level - this is the default.) 
>          Components: Management Server
>    Affects Versions: 4.3.0
>         Environment: Build from 4.3
>            Reporter: Sangeetha Hariharan
>            Priority: Critical
>             Fix For: 4.3.0
>
>         Attachments: Snaposhotfailures.rar
>
>
> Xenserver - Failed to create snapshot when secondary store was made unavaibale for about 1.5 hour leaving behind snapshot in " CreatedOnPrimary" state. The subsequent scheduled snapshot  also failed
> Set up:
> Advanced Zone with 2 Xenserver 6.2 hosts:
> Steps to reproduce the problem:
> 1. Deploy 5 Vms in each of the hosts with 10 GB ROOT volume size , so we start with 10 Vms.
> 2. Start concurrent hourly snapshots for ROOT volumes of all the Vms.
> 3. Shutdown the Secondary storage server when the snapshots are in the progress.
> 4. Bring the Secondary storage server up after 1and 1/2 hour.
> We see that the snapshot creation was halted when the Secondary storage server was down. When  it was brought up , snapshots creation continued for all the volumes except for 1 volume.
> Following exception was encountered after about 45 minutes:
> 2013-12-03 17:53:46,745 DEBUG [c.c.h.x.r.XenServerConnectionPool] (DirectAgent-258:ctx-981656b0) XmlRpcExceptio
> n for method: PBD.unplug due to Failed to create input stream: Read timed out.  Reconnecting...retry=1
> 2013-12-03 17:53:46,745 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-258:ctx-981656b0) Host 10.223.59.67 O
> paqueRef:710e7848-b118-cd69-d8c2-d3e8ffad8429: Catch Exception: Failed to create input stream: Read timed out
> 2013-12-03 17:53:46,745 WARN  [c.c.h.x.r.CitrixResourceBase] (DirectAgent-258:ctx-981656b0) Host 10.223.59.67 O
> paqueRef:710e7848-b118-cd69-d8c2-d3e8ffad8429: Unable to remove SR
> 2013-12-03 17:53:46,746 WARN  [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-258:ctx-981656b0) BackupSnapsh
> ot Failed due to Task failed! Task record:                 uuid: 99a9ee1f-645b-8893-5e13-4e16006c57ef
>            nameLabel: Async.VDI.copy
>      nameDescription:
>    allowedOperations: []
>    currentOperations: {}
>              created: Tue Dec 03 17:11:16 EST 2013
>             finished: Tue Dec 03 13:25:23 EST 2013
>               status: failure
>           residentOn: com.xensource.xenapi.Host@72f2fe96
>             progress: 1.0
>                 type: <none/>
>               result:
>            errorInfo: [SR_BACKEND_FAILURE, non-zero exit, , Traceback (most recent call last):
>   File "/opt/xensource/sm/NFSSR", line 280, in ?
>     SRCommand.run(NFSSR, DRIVER_INFO)
>   File "/opt/xensource/sm/SRCommand.py", line 327, in run
>     ret = cmd.run(sr)
>   File "/opt/xensource/sm/SRCommand.py", line 106, in run
>     return self._run_locked(sr)
>   File "/opt/xensource/sm/SRCommand.py", line 153, in _run_locked
>     return self._run(sr, target)
>   File "/opt/xensource/sm/SRCommand.py", line 197, in _run
>     return target.create(self.params['sr_uuid'], self.vdi_uuid, long(self.params['args'][0]))
>   File "/opt/xensource/sm/FileSR.py", line 487, in create
>     self.sr._update(self.sr.uuid, self.size)
>   File "/opt/xensource/sm/FileSR.py", line 185, in _update
>     self.physical_utilisation  = self._getutilisation()
>   File "/opt/xensource/sm/FileSR.py", line 270, in _getutilisation
>     return util.get_fs_utilisation(self.path)
>   File "/opt/xensource/sm/util.py", line 445, in get_fs_utilisation
>     st = ioretry_stat(lambda: os.statvfs(path))
>   File "/opt/xensource/sm/util.py", line 302, in ioretry_stat
>     stat = f()
>   File "/opt/xensource/sm/util.py", line 445, in <lambda>
>     st = ioretry_stat(lambda: os.statvfs(path))
> OSError: [Errno 5] Input/output error: '/var/run/sr-mount/b9dffc14-d14e-aec2-5609-4c8adee168fa'
> ]
>          otherConfig: {}
>            subtaskOf: com.xensource.xenapi.Task@aaf13f6f
>             subtasks: []
> Task failed! Task record:                 uuid: 99a9ee1f-645b-8893-5e13-4e16006c57ef
>            nameLabel: Async.VDI.copy
>      nameDescription:
>    allowedOperations: []
>    currentOperations: {}
>              created: Tue Dec 03 17:11:16 EST 2013
>             finished: Tue Dec 03 13:25:23 EST 2013
>               status: failure
>           residentOn: com.xensource.xenapi.Host@72f2fe96
>             progress: 1.0
>                 type: <none/>
>               result:
>            errorInfo: [SR_BACKEND_FAILURE, non-zero exit, , Traceback (most recent call last):
>   File "/opt/xensource/sm/NFSSR", line 280, in ?
>     SRCommand.run(NFSSR, DRIVER_INFO)
>   File "/opt/xensource/sm/SRCommand.py", line 327, in run
>     ret = cmd.run(sr)
>   File "/opt/xensource/sm/SRCommand.py", line 106, in run
>     return self._run_locked(sr)
>   File "/opt/xensource/sm/SRCommand.py", line 153, in _run_locked
>     return self._run(sr, target)
>   File "/opt/xensource/sm/SRCommand.py", line 197, in _run
>     return target.create(self.params['sr_uuid'], self.vdi_uuid, long(self.params['args'][0]))
>   File "/opt/xensource/sm/FileSR.py", line 487, in create
>     self.sr._update(self.sr.uuid, self.size)
>   File "/opt/xensource/sm/FileSR.py", line 185, in _update
>     self.physical_utilisation  = self._getutilisation()
>   File "/opt/xensource/sm/FileSR.py", line 270, in _getutilisation
>     return util.get_fs_utilisation(self.path)
>   File "/opt/xensource/sm/util.py", line 445, in get_fs_utilisation
>     st = ioretry_stat(lambda: os.statvfs(path))
>   File "/opt/xensource/sm/util.py", line 302, in ioretry_stat
>     stat = f()
>   File "/opt/xensource/sm/util.py", line 445, in <lambda>
>     st = ioretry_stat(lambda: os.statvfs(path))
> OSError: [Errno 5] Input/output error: '/var/run/sr-mount/b9dffc14-d14e-aec2-5609-4c8adee168fa'
> ]
>          otherConfig: {}
>            subtaskOf: com.xensource.xenapi.Task@aaf13f6f
>             subtasks: []
>         at com.cloud.hypervisor.xen.resource.CitrixResourceBase.checkForSuccess(CitrixResourceBase.java:3736)
>         at com.cloud.hypervisor.xen.resource.CitrixResourceBase.cloudVDIcopy(CitrixResourceBase.java:3914)
>         at com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.backupSnapshot(XenServerStorageProcessor.java:1268)
>         at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:90)
>         at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:50)
>         at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:613)
>         at com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:59)
>         at com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:106)
>         at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216)
>         at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>         at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:722)
> 2013-12-03 17:53:46,747 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-258:ctx-981656b0) Seq 2-1599668375: Response Received:
> 2013-12-03 17:53:46,747 DEBUG [c.c.a.t.Request] (DirectAgent-258:ctx-981656b0) Seq 2-1599668375: Processing:  { Ans: , MgmtId: 112516401760401, via: 2, Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"BackupSnapshot Failed due to Task failed! Task record:                 uuid: 99a9ee1f-645b-8893-5e13-4e16006c57ef\n           nameLabel: Async.VDI.copy\n     nameDescription: \n   allowedOperations: []\n   currentOperations: {}\n             created: Tue Dec 03 17:11:16 EST 2013\n            finished: Tue Dec 03 13:25:23 EST 2013\n              status: failure\n          residentOn: com.xensource.xenapi.Host@72f2fe96\n            progress: 1.0\n                type: <none/>\n              result: \n           errorInfo: [SR_BACKEND_FAILURE, non-zero exit, , Traceback (most recent call last):\n  File \"/opt/xensource/sm/NFSSR\", line 280, in ?\n    SRCommand.run(NFSSR, DRIVER_INFO)\n  File \"/opt/xensource/sm/SRCommand.py\", line 327, in run\n    ret = cmd.run(sr)\n  File \"/opt/xensource/sm/SRCommand.py\", line 106, in run\n    return self._run_locked(sr)\n  File \"/opt/xensource/sm/SRCommand.py\", line 153, in _run_locked\n    return self._run(sr, target)\n  File \"/opt/xensource/sm/SRCommand.py\", line 197, in _run\n    return target.create(self.params['sr_uuid'], self.vdi_uuid, long(self.params['args'][0]))\n  File \"/opt/xensource/sm/FileSR.py\", line 487, in create\n    self.sr._update(self.sr.uuid, self.size)\n  File \"/opt/xensource/sm/FileSR.py\", line 185, in _update\n    self.physical_utilisation  = self._getutilisation()\n  File \"/opt/xensource/sm/FileSR.py\", line 270, in _getutilisation\n    return util.get_fs_utilisation(self.path)\n  File \"/opt/xensource/sm/util.py\", line 445, in get_fs_utilisation\n    st = ioretry_stat(lambda: os.statvfs(path))\n  File \"/opt/xensource/sm/util.py\", line 302, in ioretry_stat\n    stat = f()\n  File \"/opt/xensource/sm/util.py\", line 445, in <lambda>\n    st = ioretry_stat(lambda: os.statvfs(path))\nOSError: [Errno 5] Input/output error: '/var/run/sr-mount/b9dffc14-d14e-aec2-5609-4c8adee168fa'\n]\n         otherConfig: {}\n           subtaskOf: com.xensource.xenapi.Task@aaf13f6f\n            subtasks: []\n","wait":0}}] }
> 2013-12-03 17:53:46,748 DEBUG [c.c.a.t.Request] (Job-Executor-81:ctx-45d54d9d ctx-5b62ab1e) Seq 2-1599668375: Received:  { Ans: , MgmtId: 112516401760401, via: 2, Ver: v1, Flags: 10, { CopyCmdAnswer } }
> 2013-12-03 17:53:46,760 DEBUG [c.c.s.s.SnapshotManagerImpl] (Job-Executor-81:ctx-45d54d9d ctx-5b62ab1e) Failed to create snapshot
> This failure left behind the snapshot in "CreatedOnPrimary" state.
> The subsequent hourly snapshot also failed , due to following exception:
> 2013-12-03 18:10:15,382 WARN  [c.c.h.x.r.CitrixResourceBase] (DirectAgent-183:ctx-6020b49b) callHostPlugin failed for cmd
> : create_secondary_storage_folder with args remoteMountPath: 10.223.57.195:/export/home/secondary, newFolder: snapshots/3
> /13,  due to There was a failure communicating with the plugin.
> 2013-12-03 18:10:15,382 WARN  [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-183:ctx-6020b49b) BackupSnapshot Failed
> due to callHostPlugin failed for cmd: create_secondary_storage_folder with args remoteMountPath: 10.223.57.195:/export/ho
> me/secondary, newFolder: snapshots/3/13,  due to There was a failure communicating with the plugin.
> com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for cmd: create_secondary_storage_folder with args
>  remoteMountPath: 10.223.57.195:/export/home/secondary, newFolder: snapshots/3/13,  due to There was a failure communicat
> ing with the plugin.
>         at com.cloud.hypervisor.xen.resource.CitrixResourceBase.callHostPlugin(CitrixResourceBase.java:4340)
>         at com.cloud.hypervisor.xen.resource.CitrixResourceBase.createSecondaryStorageFolder(CitrixResourceBase.java:7847
> )
>         at com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.backupSnapshot(XenServerStorageProcessor.java:1259
> )
>         at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:
> 90)
>         at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHan
> dlerBase.java:50)
>         at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:613)
>         at com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:59)
>         at com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:106)
>         at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216)
>         at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:10
> 3)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>         at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.ja
> va:178)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:722)
> 2013-12-03 18:10:15,383 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-183:ctx-6020b49b) Seq 1-2071593441: Response Received:
> 2013-12-03 18:10:15,383 DEBUG [c.c.a.t.Request] (DirectAgent-183:ctx-6020b49b) Seq 1-2071593441: Processing:  { Ans: , MgmtId: 112516401760401, via: 1, Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"BackupSnapshot Failed due to callHostPlugin failed for cmd: create_secondary_storage_folder with args remoteMountPath: 10.223.57.195:/export/home/secondary, newFolder: snapshots/3/13,  due to There was a failure communicating with the plugin.","wait":0}}] }
> 2013-12-03 18:10:15,383 DEBUG [c.c.a.t.Request] (Job-Executor-91:ctx-51ac8719 ctx-6eac2069) Seq 1-2071593441: Received:  { Ans: , MgmtId: 112516401760401, via: 1, Ver: v1, Flags: 10, { CopyCmdAnswer } }
> 2013-12-03 18:10:15,406 DEBUG [c.c.s.s.SnapshotManagerImpl] (Job-Executor-91:ctx-51ac8719 ctx-6eac2069) Failed to create snapshot
> com.cloud.utils.exception.CloudRuntimeException: BackupSnapshot Failed due to callHostPlugin failed for cmd: create_secondary_storage_folder with args remoteMountPath: 10.223.57.195:/export/home/secondary, newFolder: snapshots/3/13,  due to There was a failure communicating with the plugin.
>         at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:275)
>         at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:135)
>         at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:294)
>         at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:951)
>         at sun.reflect.GeneratedMethodAccessor334.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:601)
>         at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
>         at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>         at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
>         at $Proxy160.takeSnapshot(Unknown Source)
>         at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1342)
>         at com.cloud.storage.VolumeApiServiceImpl.takeSnapshot(VolumeApiServiceImpl.java:1402)
>         at sun.reflect.GeneratedMethodAccessor333.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:601)
>         at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
>         at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>         at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
>         at $Proxy232.takeSnapshot(Unknown Source)
>         at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:181)
>         at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)
>         at com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109)
>         at com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>         at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63)
>         at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:520)
>         at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>         at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:722)
> snapshot entries for volume that encountered this issue:
> | 13 |              1 |          3 |         1 |        13 |               13 | CreatedOnPrimary | NULL | TestVM-tiny-host-1ps-0-3_ROOT-13_20131203221013 | c7b68a48-d55b-4cef-8c54-85fca5f684ce |             3 | HOURLY           | 21474836480 | 2013-12-03 22:10:13 | NULL    | NULL           |     NULL |       NULL |         NULL | XenServer       | 2.2     |  NULL |
> | 23 |              1 |          3 |         1 |        13 |               13 | CreatedOnPrimary | NULL | TestVM-tiny-host-1ps-0-3_ROOT-13_20131203231013 | 667dcdfd-7f47-4ce5-8abb-54cd7df8d9a4 |             3 | HOURLY           | 21474836480 | 2013-12-03 23:10:13 | NULL    | NULL           |     NULL |       NULL |         NULL | XenServer       | 2.2     |  NULL |
> | 24 |              1 |          3 |         1 |        13 |               13 | BackingUp        | NULL | TestVM-tiny-host-1ps-0-3_ROOT-13_20131204001013 | 5de9e37a-2009-4e56-8fdb-8615ee4fe427 |             3 | HOURLY           | 21474836480 | 2013-12-04 00:10:13 | NULL    | NULL           |     NULL |       NULL |         NULL | XenServer       | 2.2     |  NULL |
> +----+----------------+------------+-----------+-----------+------------------+------------------+------+-------------------------------------------------+--------------------------------------+---------------+------------------+-------------+---------------------+---------+----------------+----------+------------+--------------+-----------------+---------+-------+
> 4 rows in set (0.00 sec)



--
This message was sent by Atlassian JIRA
(v6.1#6144)