You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Sean Lair <sl...@ippathways.com> on 2016/04/20 02:54:33 UTC
Bug in Snapshot Retention?
Hi all,
I'm running Cloudstack 4.8 on XenServer 6.5. I have one volume that I'm taking snapshots of, it is set to keep a total of 29 snapshots, but I have close to 100 snapshots in the state of "BackedUp". Am I misinterpreting the scheduled snapshot screen or am I running into a bug? Please see the output below for more detail:
[cid:image004.jpg@01D19A75.2310A6C0]
MariaDB [cloud]> select count(status) from snapshots where volume_id = 71 and status = 'backedup';
+---------------+
| count(status) |
+---------------+
| 98 |
+---------------+
1 row in set (0.00 sec)
Thanks
Sean
Re: Bug in Snapshot Retention?
Posted by Simon Weller <sw...@ena.com>.
That's a good question. I think the argument is that it can only occur when a Mgmt Server is restarted, hence it checks upon startup.
- Si
________________________________________
From: Sean Lair <sl...@ippathways.com>
Sent: Tuesday, April 26, 2016 11:32 AM
To: users@cloudstack.apache.org
Subject: RE: Bug in Snapshot Retention?
Thanks! That PR does look like it will fix our issue and allow the retention thread to delete the allocated snapshot.
A bigger question is do you think the management server should be checking for these orphaned allocated snapshots periodically or at startup?
-----Original Message-----
From: Simon Weller [mailto:sweller@ena.com]
Sent: Monday, April 25, 2016 10:01 AM
To: users@cloudstack.apache.org
Subject: Re: Bug in Snapshot Retention?
There is an open issue (and associated PR) that covers this I think....
https://issues.apache.org/jira/browse/CLOUDSTACK-9200
and PR: https://github.com/apache/cloudstack/pull/1282
- Si
________________________________________
From: Anshul Gangwar <an...@accelerite.com>
Sent: Sunday, April 24, 2016 11:48 PM
To: users@cloudstack.apache.org
Subject: Re: Bug in Snapshot Retention?
They can stuck in Allocated state if the management server is restarted just before the snapshot transitioned to new state. Time period when this can happen will depend on the jobs scheduled on VM to which this volume is attached.
This sounds like a bug that snapshot which is stuck in Allocated state is failing to delete. Can you create a ticket to track this?
Regards,
Anshul
> On 20-Apr-2016, at 8:00 PM, Sean Lair <sl...@ippathways.com> wrote:
>
> Thanks for the responses all. The "Removed Field" for the snapshots with the status of "BackedUp" is NULL.
>
> I combed the logs and found the exception below. It was successfully deleting snapshots before that log entry, then errored on the "Allocated" snapshot and stopped any further deletions. I'm not sure what allocated mean, but will start researching.
>
> 2016-04-20 00:44:38,830 DEBUG [c.c.s.s.SnapshotManagerImpl] (Work-Job-Executor-1:ctx-954cbd99 job-7396/job-7401 ctx-d0261c6c) (logid:bafb5d42) post process snapshot failed
> com.cloud.utils.exception.CloudRuntimeException: Failed to delete snapshot:com.cloud.exception.InvalidParameterValueException: Can't delete snapshotshot 1351 due to it is in Allocated Status
> at com.cloud.storage.snapshot.SnapshotManagerImpl.deleteSnapshot(SnapshotManagerImpl.java:478)
> at com.cloud.storage.snapshot.SnapshotManagerImpl.postCreateRecurringSnapshotForPolicy(SnapshotManagerImpl.java:420)
> at com.cloud.storage.snapshot.SnapshotManagerImpl.postCreateSnapshot(SnapshotManagerImpl.java:399)
> at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1010)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at com.sun.proxy.$Proxy191.takeSnapshot(Unknown Source)
> at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1591)
> at com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(VolumeApiServiceImpl.java:2107)
> at com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(VolumeApiServiceImpl.java:2899)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> at com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:2907)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at com.sun.proxy.$Proxy196.handleVmWorkJob(Unknown Source)
> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
>
> -----Original Message-----
> From: Anshul Gangwar [mailto:anshul.gangwar@accelerite.com]
> Sent: Wednesday, April 20, 2016 12:05 AM
> To: users@cloudstack.apache.org
> Subject: Re: Bug in Snapshot Retention?
>
> Are you getting any exception in logs after the completion of snapshot for this volume?
>
> If so try changing "job.cancel.threshold" global setting to appropriate timeout. This will make sure that job is not completed before the snapshot process is finished and will also make sure that cleanup process is not failing. This setting will affect all the jobs.
>
> Regards,
> Anshul
>
>
>
>> On 20-Apr-2016, at 6:24 AM, Sean Lair <sl...@ippathways.com> wrote:
>>
>> Hi all,
>>
>> I'm running Cloudstack 4.8 on XenServer 6.5. I have one volume that I'm taking snapshots of, it is set to keep a total of 29 snapshots, but I have close to 100 snapshots in the state of "BackedUp". Am I misinterpreting the scheduled snapshot screen or am I running into a bug? Please see the output below for more detail:
>>
>> [cid:image004.jpg@01D19A75.2310A6C0]
>>
>> MariaDB [cloud]> select count(status) from snapshots where volume_id = 71 and status = 'backedup';
>> +---------------+
>> | count(status) |
>> +---------------+
>> | 98 |
>> +---------------+
>> 1 row in set (0.00 sec)
>>
>> Thanks
>> Sean
>>
>
>
>
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
RE: Bug in Snapshot Retention?
Posted by Sean Lair <sl...@ippathways.com>.
Thanks! That PR does look like it will fix our issue and allow the retention thread to delete the allocated snapshot.
A bigger question is do you think the management server should be checking for these orphaned allocated snapshots periodically or at startup?
-----Original Message-----
From: Simon Weller [mailto:sweller@ena.com]
Sent: Monday, April 25, 2016 10:01 AM
To: users@cloudstack.apache.org
Subject: Re: Bug in Snapshot Retention?
There is an open issue (and associated PR) that covers this I think....
https://issues.apache.org/jira/browse/CLOUDSTACK-9200
and PR: https://github.com/apache/cloudstack/pull/1282
- Si
________________________________________
From: Anshul Gangwar <an...@accelerite.com>
Sent: Sunday, April 24, 2016 11:48 PM
To: users@cloudstack.apache.org
Subject: Re: Bug in Snapshot Retention?
They can stuck in Allocated state if the management server is restarted just before the snapshot transitioned to new state. Time period when this can happen will depend on the jobs scheduled on VM to which this volume is attached.
This sounds like a bug that snapshot which is stuck in Allocated state is failing to delete. Can you create a ticket to track this?
Regards,
Anshul
> On 20-Apr-2016, at 8:00 PM, Sean Lair <sl...@ippathways.com> wrote:
>
> Thanks for the responses all. The "Removed Field" for the snapshots with the status of "BackedUp" is NULL.
>
> I combed the logs and found the exception below. It was successfully deleting snapshots before that log entry, then errored on the "Allocated" snapshot and stopped any further deletions. I'm not sure what allocated mean, but will start researching.
>
> 2016-04-20 00:44:38,830 DEBUG [c.c.s.s.SnapshotManagerImpl] (Work-Job-Executor-1:ctx-954cbd99 job-7396/job-7401 ctx-d0261c6c) (logid:bafb5d42) post process snapshot failed
> com.cloud.utils.exception.CloudRuntimeException: Failed to delete snapshot:com.cloud.exception.InvalidParameterValueException: Can't delete snapshotshot 1351 due to it is in Allocated Status
> at com.cloud.storage.snapshot.SnapshotManagerImpl.deleteSnapshot(SnapshotManagerImpl.java:478)
> at com.cloud.storage.snapshot.SnapshotManagerImpl.postCreateRecurringSnapshotForPolicy(SnapshotManagerImpl.java:420)
> at com.cloud.storage.snapshot.SnapshotManagerImpl.postCreateSnapshot(SnapshotManagerImpl.java:399)
> at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1010)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at com.sun.proxy.$Proxy191.takeSnapshot(Unknown Source)
> at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1591)
> at com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(VolumeApiServiceImpl.java:2107)
> at com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(VolumeApiServiceImpl.java:2899)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> at com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:2907)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at com.sun.proxy.$Proxy196.handleVmWorkJob(Unknown Source)
> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
>
> -----Original Message-----
> From: Anshul Gangwar [mailto:anshul.gangwar@accelerite.com]
> Sent: Wednesday, April 20, 2016 12:05 AM
> To: users@cloudstack.apache.org
> Subject: Re: Bug in Snapshot Retention?
>
> Are you getting any exception in logs after the completion of snapshot for this volume?
>
> If so try changing "job.cancel.threshold" global setting to appropriate timeout. This will make sure that job is not completed before the snapshot process is finished and will also make sure that cleanup process is not failing. This setting will affect all the jobs.
>
> Regards,
> Anshul
>
>
>
>> On 20-Apr-2016, at 6:24 AM, Sean Lair <sl...@ippathways.com> wrote:
>>
>> Hi all,
>>
>> I'm running Cloudstack 4.8 on XenServer 6.5. I have one volume that I'm taking snapshots of, it is set to keep a total of 29 snapshots, but I have close to 100 snapshots in the state of "BackedUp". Am I misinterpreting the scheduled snapshot screen or am I running into a bug? Please see the output below for more detail:
>>
>> [cid:image004.jpg@01D19A75.2310A6C0]
>>
>> MariaDB [cloud]> select count(status) from snapshots where volume_id = 71 and status = 'backedup';
>> +---------------+
>> | count(status) |
>> +---------------+
>> | 98 |
>> +---------------+
>> 1 row in set (0.00 sec)
>>
>> Thanks
>> Sean
>>
>
>
>
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
Re: Bug in Snapshot Retention?
Posted by Simon Weller <sw...@ena.com>.
There is an open issue (and associated PR) that covers this I think....
https://issues.apache.org/jira/browse/CLOUDSTACK-9200
and PR: https://github.com/apache/cloudstack/pull/1282
- Si
________________________________________
From: Anshul Gangwar <an...@accelerite.com>
Sent: Sunday, April 24, 2016 11:48 PM
To: users@cloudstack.apache.org
Subject: Re: Bug in Snapshot Retention?
They can stuck in Allocated state if the management server is restarted just before the snapshot transitioned to new state. Time period when this can happen will depend on the jobs scheduled on VM to which this volume is attached.
This sounds like a bug that snapshot which is stuck in Allocated state is failing to delete. Can you create a ticket to track this?
Regards,
Anshul
> On 20-Apr-2016, at 8:00 PM, Sean Lair <sl...@ippathways.com> wrote:
>
> Thanks for the responses all. The "Removed Field" for the snapshots with the status of "BackedUp" is NULL.
>
> I combed the logs and found the exception below. It was successfully deleting snapshots before that log entry, then errored on the "Allocated" snapshot and stopped any further deletions. I'm not sure what allocated mean, but will start researching.
>
> 2016-04-20 00:44:38,830 DEBUG [c.c.s.s.SnapshotManagerImpl] (Work-Job-Executor-1:ctx-954cbd99 job-7396/job-7401 ctx-d0261c6c) (logid:bafb5d42) post process snapshot failed
> com.cloud.utils.exception.CloudRuntimeException: Failed to delete snapshot:com.cloud.exception.InvalidParameterValueException: Can't delete snapshotshot 1351 due to it is in Allocated Status
> at com.cloud.storage.snapshot.SnapshotManagerImpl.deleteSnapshot(SnapshotManagerImpl.java:478)
> at com.cloud.storage.snapshot.SnapshotManagerImpl.postCreateRecurringSnapshotForPolicy(SnapshotManagerImpl.java:420)
> at com.cloud.storage.snapshot.SnapshotManagerImpl.postCreateSnapshot(SnapshotManagerImpl.java:399)
> at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1010)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at com.sun.proxy.$Proxy191.takeSnapshot(Unknown Source)
> at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1591)
> at com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(VolumeApiServiceImpl.java:2107)
> at com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(VolumeApiServiceImpl.java:2899)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> at com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:2907)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at com.sun.proxy.$Proxy196.handleVmWorkJob(Unknown Source)
> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
>
> -----Original Message-----
> From: Anshul Gangwar [mailto:anshul.gangwar@accelerite.com]
> Sent: Wednesday, April 20, 2016 12:05 AM
> To: users@cloudstack.apache.org
> Subject: Re: Bug in Snapshot Retention?
>
> Are you getting any exception in logs after the completion of snapshot for this volume?
>
> If so try changing “job.cancel.threshold” global setting to appropriate timeout. This will make sure that job is not completed before the snapshot process is finished and will also make sure that cleanup process is not failing. This setting will affect all the jobs.
>
> Regards,
> Anshul
>
>
>
>> On 20-Apr-2016, at 6:24 AM, Sean Lair <sl...@ippathways.com> wrote:
>>
>> Hi all,
>>
>> I'm running Cloudstack 4.8 on XenServer 6.5. I have one volume that I'm taking snapshots of, it is set to keep a total of 29 snapshots, but I have close to 100 snapshots in the state of "BackedUp". Am I misinterpreting the scheduled snapshot screen or am I running into a bug? Please see the output below for more detail:
>>
>> [cid:image004.jpg@01D19A75.2310A6C0]
>>
>> MariaDB [cloud]> select count(status) from snapshots where volume_id = 71 and status = 'backedup';
>> +---------------+
>> | count(status) |
>> +---------------+
>> | 98 |
>> +---------------+
>> 1 row in set (0.00 sec)
>>
>> Thanks
>> Sean
>>
>
>
>
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
Re: Bug in Snapshot Retention?
Posted by Anshul Gangwar <an...@accelerite.com>.
They can stuck in Allocated state if the management server is restarted just before the snapshot transitioned to new state. Time period when this can happen will depend on the jobs scheduled on VM to which this volume is attached.
This sounds like a bug that snapshot which is stuck in Allocated state is failing to delete. Can you create a ticket to track this?
Regards,
Anshul
> On 20-Apr-2016, at 8:00 PM, Sean Lair <sl...@ippathways.com> wrote:
>
> Thanks for the responses all. The "Removed Field" for the snapshots with the status of "BackedUp" is NULL.
>
> I combed the logs and found the exception below. It was successfully deleting snapshots before that log entry, then errored on the "Allocated" snapshot and stopped any further deletions. I'm not sure what allocated mean, but will start researching.
>
> 2016-04-20 00:44:38,830 DEBUG [c.c.s.s.SnapshotManagerImpl] (Work-Job-Executor-1:ctx-954cbd99 job-7396/job-7401 ctx-d0261c6c) (logid:bafb5d42) post process snapshot failed
> com.cloud.utils.exception.CloudRuntimeException: Failed to delete snapshot:com.cloud.exception.InvalidParameterValueException: Can't delete snapshotshot 1351 due to it is in Allocated Status
> at com.cloud.storage.snapshot.SnapshotManagerImpl.deleteSnapshot(SnapshotManagerImpl.java:478)
> at com.cloud.storage.snapshot.SnapshotManagerImpl.postCreateRecurringSnapshotForPolicy(SnapshotManagerImpl.java:420)
> at com.cloud.storage.snapshot.SnapshotManagerImpl.postCreateSnapshot(SnapshotManagerImpl.java:399)
> at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1010)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at com.sun.proxy.$Proxy191.takeSnapshot(Unknown Source)
> at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1591)
> at com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(VolumeApiServiceImpl.java:2107)
> at com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(VolumeApiServiceImpl.java:2899)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> at com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:2907)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at com.sun.proxy.$Proxy196.handleVmWorkJob(Unknown Source)
> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
>
> -----Original Message-----
> From: Anshul Gangwar [mailto:anshul.gangwar@accelerite.com]
> Sent: Wednesday, April 20, 2016 12:05 AM
> To: users@cloudstack.apache.org
> Subject: Re: Bug in Snapshot Retention?
>
> Are you getting any exception in logs after the completion of snapshot for this volume?
>
> If so try changing “job.cancel.threshold” global setting to appropriate timeout. This will make sure that job is not completed before the snapshot process is finished and will also make sure that cleanup process is not failing. This setting will affect all the jobs.
>
> Regards,
> Anshul
>
>
>
>> On 20-Apr-2016, at 6:24 AM, Sean Lair <sl...@ippathways.com> wrote:
>>
>> Hi all,
>>
>> I'm running Cloudstack 4.8 on XenServer 6.5. I have one volume that I'm taking snapshots of, it is set to keep a total of 29 snapshots, but I have close to 100 snapshots in the state of "BackedUp". Am I misinterpreting the scheduled snapshot screen or am I running into a bug? Please see the output below for more detail:
>>
>> [cid:image004.jpg@01D19A75.2310A6C0]
>>
>> MariaDB [cloud]> select count(status) from snapshots where volume_id = 71 and status = 'backedup';
>> +---------------+
>> | count(status) |
>> +---------------+
>> | 98 |
>> +---------------+
>> 1 row in set (0.00 sec)
>>
>> Thanks
>> Sean
>>
>
>
>
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
RE: Bug in Snapshot Retention?
Posted by Sean Lair <sl...@ippathways.com>.
Hi all,
Any insight in why some snapshots could be stuck in "allocated" state?
Thanks
Sean
-----Original Message-----
From: Sean Lair
Sent: Wednesday, April 20, 2016 9:31 AM
To: users@cloudstack.apache.org
Subject: RE: Bug in Snapshot Retention?
Thanks for the responses all. The "Removed Field" for the snapshots with the status of "BackedUp" is NULL.
I combed the logs and found the exception below. It was successfully deleting snapshots before that log entry, then errored on the "Allocated" snapshot and stopped any further deletions. I'm not sure what allocated mean, but will start researching.
2016-04-20 00:44:38,830 DEBUG [c.c.s.s.SnapshotManagerImpl] (Work-Job-Executor-1:ctx-954cbd99 job-7396/job-7401 ctx-d0261c6c) (logid:bafb5d42) post process snapshot failed
com.cloud.utils.exception.CloudRuntimeException: Failed to delete snapshot:com.cloud.exception.InvalidParameterValueException: Can't delete snapshotshot 1351 due to it is in Allocated Status
at com.cloud.storage.snapshot.SnapshotManagerImpl.deleteSnapshot(SnapshotManagerImpl.java:478)
at com.cloud.storage.snapshot.SnapshotManagerImpl.postCreateRecurringSnapshotForPolicy(SnapshotManagerImpl.java:420)
at com.cloud.storage.snapshot.SnapshotManagerImpl.postCreateSnapshot(SnapshotManagerImpl.java:399)
at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1010)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy191.takeSnapshot(Unknown Source)
at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1591)
at com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(VolumeApiServiceImpl.java:2107)
at com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(VolumeApiServiceImpl.java:2899)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:2907)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy196.handleVmWorkJob(Unknown Source)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
-----Original Message-----
From: Anshul Gangwar [mailto:anshul.gangwar@accelerite.com]
Sent: Wednesday, April 20, 2016 12:05 AM
To: users@cloudstack.apache.org
Subject: Re: Bug in Snapshot Retention?
Are you getting any exception in logs after the completion of snapshot for this volume?
If so try changing “job.cancel.threshold” global setting to appropriate timeout. This will make sure that job is not completed before the snapshot process is finished and will also make sure that cleanup process is not failing. This setting will affect all the jobs.
Regards,
Anshul
> On 20-Apr-2016, at 6:24 AM, Sean Lair <sl...@ippathways.com> wrote:
>
> Hi all,
>
> I'm running Cloudstack 4.8 on XenServer 6.5. I have one volume that I'm taking snapshots of, it is set to keep a total of 29 snapshots, but I have close to 100 snapshots in the state of "BackedUp". Am I misinterpreting the scheduled snapshot screen or am I running into a bug? Please see the output below for more detail:
>
> [cid:image004.jpg@01D19A75.2310A6C0]
>
> MariaDB [cloud]> select count(status) from snapshots where volume_id = 71 and status = 'backedup';
> +---------------+
> | count(status) |
> +---------------+
> | 98 |
> +---------------+
> 1 row in set (0.00 sec)
>
> Thanks
> Sean
>
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
RE: Bug in Snapshot Retention?
Posted by Sean Lair <sl...@ippathways.com>.
Thanks for the responses all. The "Removed Field" for the snapshots with the status of "BackedUp" is NULL.
I combed the logs and found the exception below. It was successfully deleting snapshots before that log entry, then errored on the "Allocated" snapshot and stopped any further deletions. I'm not sure what allocated mean, but will start researching.
2016-04-20 00:44:38,830 DEBUG [c.c.s.s.SnapshotManagerImpl] (Work-Job-Executor-1:ctx-954cbd99 job-7396/job-7401 ctx-d0261c6c) (logid:bafb5d42) post process snapshot failed
com.cloud.utils.exception.CloudRuntimeException: Failed to delete snapshot:com.cloud.exception.InvalidParameterValueException: Can't delete snapshotshot 1351 due to it is in Allocated Status
at com.cloud.storage.snapshot.SnapshotManagerImpl.deleteSnapshot(SnapshotManagerImpl.java:478)
at com.cloud.storage.snapshot.SnapshotManagerImpl.postCreateRecurringSnapshotForPolicy(SnapshotManagerImpl.java:420)
at com.cloud.storage.snapshot.SnapshotManagerImpl.postCreateSnapshot(SnapshotManagerImpl.java:399)
at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1010)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy191.takeSnapshot(Unknown Source)
at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1591)
at com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(VolumeApiServiceImpl.java:2107)
at com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(VolumeApiServiceImpl.java:2899)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:2907)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy196.handleVmWorkJob(Unknown Source)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
-----Original Message-----
From: Anshul Gangwar [mailto:anshul.gangwar@accelerite.com]
Sent: Wednesday, April 20, 2016 12:05 AM
To: users@cloudstack.apache.org
Subject: Re: Bug in Snapshot Retention?
Are you getting any exception in logs after the completion of snapshot for this volume?
If so try changing “job.cancel.threshold” global setting to appropriate timeout. This will make sure that job is not completed before the snapshot process is finished and will also make sure that cleanup process is not failing. This setting will affect all the jobs.
Regards,
Anshul
> On 20-Apr-2016, at 6:24 AM, Sean Lair <sl...@ippathways.com> wrote:
>
> Hi all,
>
> I'm running Cloudstack 4.8 on XenServer 6.5. I have one volume that I'm taking snapshots of, it is set to keep a total of 29 snapshots, but I have close to 100 snapshots in the state of "BackedUp". Am I misinterpreting the scheduled snapshot screen or am I running into a bug? Please see the output below for more detail:
>
> [cid:image004.jpg@01D19A75.2310A6C0]
>
> MariaDB [cloud]> select count(status) from snapshots where volume_id = 71 and status = 'backedup';
> +---------------+
> | count(status) |
> +---------------+
> | 98 |
> +---------------+
> 1 row in set (0.00 sec)
>
> Thanks
> Sean
>
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
Re: Bug in Snapshot Retention?
Posted by Anshul Gangwar <an...@accelerite.com>.
Are you getting any exception in logs after the completion of snapshot for this volume?
If so try changing “job.cancel.threshold” global setting to appropriate timeout. This will make sure that job is not completed before the snapshot process is finished and will also make sure that cleanup process is not failing. This setting will affect all the jobs.
Regards,
Anshul
> On 20-Apr-2016, at 6:24 AM, Sean Lair <sl...@ippathways.com> wrote:
>
> Hi all,
>
> I'm running Cloudstack 4.8 on XenServer 6.5. I have one volume that I'm taking snapshots of, it is set to keep a total of 29 snapshots, but I have close to 100 snapshots in the state of "BackedUp". Am I misinterpreting the scheduled snapshot screen or am I running into a bug? Please see the output below for more detail:
>
> [cid:image004.jpg@01D19A75.2310A6C0]
>
> MariaDB [cloud]> select count(status) from snapshots where volume_id = 71 and status = 'backedup';
> +---------------+
> | count(status) |
> +---------------+
> | 98 |
> +---------------+
> 1 row in set (0.00 sec)
>
> Thanks
> Sean
>
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
Re: Bug in Snapshot Retention?
Posted by Shweta Agarwal <sh...@accelerite.com>.
Hi Sean,
Can you also Check Removed field for these snapshots in snapshots table
Thanks
Shweta
Principal Product Engineer, CloudPlatform
Accelerite
________________________________________
From: Sean Lair <sl...@ippathways.com>
Sent: Wednesday, April 20, 2016 6:24 AM
To: users@cloudstack.apache.org
Subject: Bug in Snapshot Retention?
Hi all,
I'm running Cloudstack 4.8 on XenServer 6.5. I have one volume that I'm taking snapshots of, it is set to keep a total of 29 snapshots, but I have close to 100 snapshots in the state of "BackedUp". Am I misinterpreting the scheduled snapshot screen or am I running into a bug? Please see the output below for more detail:
[cid:image004.jpg@01D19A75.2310A6C0]
MariaDB [cloud]> select count(status) from snapshots where volume_id = 71 and status = 'backedup';
+---------------+
| count(status) |
+---------------+
| 98 |
+---------------+
1 row in set (0.00 sec)
Thanks
Sean
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.