You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Yong Chen <Yo...@coredesktop.com> on 2013/07/15 02:43:50 UTC
The snapshot chain is too long
Hi,
I'm using CS 4.0.1 and XS 6.0.2. There is a problem with taking snapshots of instances.
Some of VMs have scheduled snapshots (e.g. weekly, monthly) so eventually there will be quite some number of snapshots on secondary storage. This is fine as I read CloudStack should copy snapshots to secondary storage and then coalesce snapshots on primary storage so can keep the chain not too long (like 3-4 child normally).
However I found after snapshot been taken for a while it starts failing due to error logged on CS Management log 'SR_BACKEND_FAILURE_109The snapshot chain is too long'. I verified it by checking with vhd-util and it does show the affected VHD has exactly 30 snapshots in chain which is the limit for XenServer.
I tried to search for info why the chain can grow to 30 and only found this article for CloudStack 2.2 http://support.citrix.com/article/CTX133470 seems to be related. It mentioned there maybe problem preventing XenServer from cleaning up snapshots. However I can't find any more error info other than CS Management log 'SR_BACKEND_FAILURE_109The snapshot chain is too long'.
What can cause XenServer or CloudStack not cleaning up snapshots from primary storage? Thanks in advance.
Yong
Re: The snapshot chain is too long
Posted by France <ma...@isg.si>.
Hi Yong,
I'm hitting the same problem nowdays. I guess i have left over snapshots
from failed NFS storage, which was since replaced.
Did you coalesce snapshot chain by hand or did you delete the snapshots
manually? If so, do you remember how?
Here are my snapshots, and there are currently no snapshot creation
processes:
xe vdi-list is-a-snapshot=true | grep name-label
name-label ( RW): XYZ_ROOT-385_20140125020342
name-label ( RW): Template c2b3a07f-d16f-4abb-9162-55e4130a417c
name-label ( RW): Template e80af9a4-e087-4220-977b-868fa4ec75b6
name-label ( RW): XYZ_ROOT-385_20140121020342
name-label ( RW): XYZ_ROOT-385_20140121020342
name-label ( RW): XBBBC_20140206040441
name-label ( RW): XYZ_ROOT-385_20140124020342
name-label ( RW): OCWWW_20140112000011
name-label ( RW): XYZ_ROOT-385_20140122020342
name-label ( RW): Template routing-1
name-label ( RW): Template aa0bcd7c-4b03-4778-a038-da80fdfb7a43
name-label ( RW): OCWWWXXXXX_ROOT-330_20140201130342
name-label ( RW): Template e80af9a4-e087-4220-977b-868fa4ec75b6
name-label ( RW): XYZ_ROOT-385_20140125020342
name-label ( RW): XYZ_ROOT-385_20140123020342
name-label ( RW): XYZ_ROOT-385_20140122020342
name-label ( RW): Template 58e13a51-affa-4fe2-a66b-19e89091290d
name-label ( RW): ABCCCDDDD_ROOT-334_20140201160342
name-label ( RW): ANON_ROOT-324_20131121124532
name-label ( RW): Template fc0262f2-7609-498b-a1ac-ed71e1ebe7f9
name-label ( RW): XYZ_ROOT-385_20140125020342
name-label ( RW): Template d768db3f-6d42-48f9-bdfb-7dceccef9f3e
name-label ( RW): XYZ_ROOT-385_20140124020342
name-label ( RW): Template fc0262f2-7609-498b-a1ac-ed71e1ebe7f9
name-label ( RW): XYZ_ROOT-385_20140120020341
name-label ( RW): detached_hrosci_20130513190437
name-label ( RW): XYZ_ROOT-385_20140123020342
name-label ( RW): XYZ_ROOT-385_20140124020342
name-label ( RW): Template routing-1
name-label ( RW): NGGQQQ_ROOT-423_20140202030342
name-label ( RW): XYZ_ROOT-385_20140121020342
name-label ( RW): SOME_work_ROOT-295_20130322150148
name-label ( RW): XYZ_ROOT-385_20140122020342
name-label ( RW): XYZ_ROOT-385_20140120020341
name-label ( RW): Template 57d7c73c-ca06-4225-8a9f-7cc5776c5610
name-label ( RW): XYZ_ROOT-385_20140122020342
name-label ( RW): Template e80af9a4-e087-4220-977b-868fa4ec75b6
name-label ( RW): Template 90d42566-b956-4d9d-9685-91e19b693f86
name-label ( RW): Template c2b3a07f-d16f-4abb-9162-55e4130a417c
name-label ( RW): XYZ_ROOT-385_20140120020341
name-label ( RW): DDGGWW_ROOT-234_20140118030341
name-label ( RW): Template 8f99eaf7-6d33-4097-8d19-9cafd681f124
name-label ( RW): Template afa9d0a0-8242-443b-ac53-b0b1c760559c
name-label ( RW): XYZ_ROOT-385_20140123020342
name-label ( RW): NNGGNN_ROOT-351_20140205020342
name-label ( RW): OOIIOO_ROOT-313_20131203084833
name-label ( RW): Template routing-1
name-label ( RW): Template 61d4df5b-bccf-4457-ad08-0ae57ea16a7e
name-label ( RW): XYZ_ROOT-385_20140123020342
name-label ( RW): TTGGTT_ROOT-233_20121213090209
name-label ( RW): XYZ_ROOT-385_20140122020342
name-label ( RW): XYZ_ROOT-385_20140120020341
name-label ( RW): XYZ_ROOT-385_20140123020342
name-label ( RW): Template 1feab759-b573-4227-8b12-8e9846ee4bd6
name-label ( RW): Template 4e444f15-ac78-4b53-9899-c406478f99b2
name-label ( RW): Template 690fa285-3317-45d6-a563-35ddd5af493e
name-label ( RW): XYZ_ROOT-385_20140124020342
name-label ( RW): XYZ_ROOT-385_20140121020342
name-label ( RW): ABCCCDDDD_DATA-334_20140207020441
name-label ( RW): Template 4e444f15-ac78-4b53-9899-c406478f99b2
name-label ( RW): XYZ_ROOT-385_20140124020342
name-label ( RW): Template c2b3a07f-d16f-4abb-9162-55e4130a417c
name-label ( RW): Template c1e4d036-bd10-4b33-803a-9e34f2c755fe
name-label ( RW): XYZ_ROOT-385_20140121020342
name-label ( RW): XYZ_ROOT-385_20140120020341
On 16/7/13 12:29 PM, Yong Chen wrote:
> Can anyone help please? Is below correct?
>
> Yong
>
> ________________________________________
> From: Yong Chen
> Sent: Monday, 15 July 2013 11:58 AM
> To: Cloudstack users mailing list
> Subject: RE: The snapshot chain is too long
>
> My understand of how snapshot works (XenServer as hypervisor) in CloudStack is:
>
> 1. User takes a snapshot of volume either manually or by schedule
> 2. CloudStack does API call to XS for taking a snapshot of the particular VHD - this happens on primary storage
> 3. CloudStack use SSVM to copy the newly created snapshot (VHD) from primary to secondary storage
> 4. CloudStack make API call to coalesce snapshot chain for the volume on primary storage
>
> This way CloudStack can keep snapshot chain length minimum on primary storage. It should never grow near the XenServer limit of 30 snapshots.
>
> As of VHD snapshot chain on secondary storage, it is different from primary storage's. There is a global setting 'snapshot.delta.max' (default to 16) to determine how often should CS start a new chain with full copy of VHD.
>
> Above is what my understanding is. If it's not accurate please let me know.
>
> And my problem is some VHD's snapshot chain does hit to the limit of 30 on XenServer.
>
> Regards,
>
> Yong
>
> -----Original Message-----
> From: Kirk Jantzer [mailto:kirk.jantzer@gmail.com]
> Sent: Monday, 15 July 2013 10:47 AM
> To: Cloudstack users mailing list
> Subject: Re: The snapshot chain is too long
>
> Don't think of CS snapshots the same way you do as VMWare snapshots.
> CloudStack snapshots are WHOLE COPY BACKUPS of the instance. Also, in regards to snapshot chains, I know VMWare has a default setting per VM of
> 32 snapshots. If you exceed this, it will stop you from creating a new one.
> You can, however, modify that setting so that you can create more. I am not sure if XenServer has the same type of setting.
>
>
> On Sun, Jul 14, 2013 at 8:43 PM, Yong Chen <Yo...@coredesktop.com> wrote:
>
>> Hi,
>>
>> I'm using CS 4.0.1 and XS 6.0.2. There is a problem with taking
>> snapshots of instances.
>>
>> Some of VMs have scheduled snapshots (e.g. weekly, monthly) so
>> eventually there will be quite some number of snapshots on secondary
>> storage. This is fine as I read CloudStack should copy snapshots to
>> secondary storage and then coalesce snapshots on primary storage so
>> can keep the chain not too long (like 3-4 child normally).
>>
>> However I found after snapshot been taken for a while it starts
>> failing due to error logged on CS Management log
>> 'SR_BACKEND_FAILURE_109The snapshot chain is too long'. I verified it
>> by checking with vhd-util and it does show the affected VHD has
>> exactly 30 snapshots in chain which is the limit for XenServer.
>>
>> I tried to search for info why the chain can grow to 30 and only found
>> this article for CloudStack 2.2
>> http://support.citrix.com/article/CTX133470 seems to be related. It
>> mentioned there maybe problem preventing XenServer from cleaning up
>> snapshots. However I can't find any more error info other than CS
>> Management log 'SR_BACKEND_FAILURE_109The snapshot chain is too long'.
>>
>> What can cause XenServer or CloudStack not cleaning up snapshots from
>> primary storage? Thanks in advance.
>>
>> Yong
>>
>
>
> --
> Regards,
>
> Kirk Jantzer
> c: (678) 561-5475
> http://about.met/kirkjantzer
RE: The snapshot chain is too long
Posted by Yong Chen <Yo...@coredesktop.com>.
Can anyone help please? Is below correct?
Yong
________________________________________
From: Yong Chen
Sent: Monday, 15 July 2013 11:58 AM
To: Cloudstack users mailing list
Subject: RE: The snapshot chain is too long
My understand of how snapshot works (XenServer as hypervisor) in CloudStack is:
1. User takes a snapshot of volume either manually or by schedule
2. CloudStack does API call to XS for taking a snapshot of the particular VHD - this happens on primary storage
3. CloudStack use SSVM to copy the newly created snapshot (VHD) from primary to secondary storage
4. CloudStack make API call to coalesce snapshot chain for the volume on primary storage
This way CloudStack can keep snapshot chain length minimum on primary storage. It should never grow near the XenServer limit of 30 snapshots.
As of VHD snapshot chain on secondary storage, it is different from primary storage's. There is a global setting 'snapshot.delta.max' (default to 16) to determine how often should CS start a new chain with full copy of VHD.
Above is what my understanding is. If it's not accurate please let me know.
And my problem is some VHD's snapshot chain does hit to the limit of 30 on XenServer.
Regards,
Yong
-----Original Message-----
From: Kirk Jantzer [mailto:kirk.jantzer@gmail.com]
Sent: Monday, 15 July 2013 10:47 AM
To: Cloudstack users mailing list
Subject: Re: The snapshot chain is too long
Don't think of CS snapshots the same way you do as VMWare snapshots.
CloudStack snapshots are WHOLE COPY BACKUPS of the instance. Also, in regards to snapshot chains, I know VMWare has a default setting per VM of
32 snapshots. If you exceed this, it will stop you from creating a new one.
You can, however, modify that setting so that you can create more. I am not sure if XenServer has the same type of setting.
On Sun, Jul 14, 2013 at 8:43 PM, Yong Chen <Yo...@coredesktop.com> wrote:
> Hi,
>
> I'm using CS 4.0.1 and XS 6.0.2. There is a problem with taking
> snapshots of instances.
>
> Some of VMs have scheduled snapshots (e.g. weekly, monthly) so
> eventually there will be quite some number of snapshots on secondary
> storage. This is fine as I read CloudStack should copy snapshots to
> secondary storage and then coalesce snapshots on primary storage so
> can keep the chain not too long (like 3-4 child normally).
>
> However I found after snapshot been taken for a while it starts
> failing due to error logged on CS Management log
> 'SR_BACKEND_FAILURE_109The snapshot chain is too long'. I verified it
> by checking with vhd-util and it does show the affected VHD has
> exactly 30 snapshots in chain which is the limit for XenServer.
>
> I tried to search for info why the chain can grow to 30 and only found
> this article for CloudStack 2.2
> http://support.citrix.com/article/CTX133470 seems to be related. It
> mentioned there maybe problem preventing XenServer from cleaning up
> snapshots. However I can't find any more error info other than CS
> Management log 'SR_BACKEND_FAILURE_109The snapshot chain is too long'.
>
> What can cause XenServer or CloudStack not cleaning up snapshots from
> primary storage? Thanks in advance.
>
> Yong
>
--
Regards,
Kirk Jantzer
c: (678) 561-5475
http://about.met/kirkjantzer
RE: The snapshot chain is too long
Posted by Yong Chen <Yo...@coredesktop.com>.
My understand of how snapshot works (XenServer as hypervisor) in CloudStack is:
1. User takes a snapshot of volume either manually or by schedule
2. CloudStack does API call to XS for taking a snapshot of the particular VHD - this happens on primary storage
3. CloudStack use SSVM to copy the newly created snapshot (VHD) from primary to secondary storage
4. CloudStack make API call to coalesce snapshot chain for the volume on primary storage
This way CloudStack can keep snapshot chain length minimum on primary storage. It should never grow near the XenServer limit of 30 snapshots.
As of VHD snapshot chain on secondary storage, it is different from primary storage's. There is a global setting 'snapshot.delta.max' (default to 16) to determine how often should CS start a new chain with full copy of VHD.
Above is what my understanding is. If it's not accurate please let me know.
And my problem is some VHD's snapshot chain does hit to the limit of 30 on XenServer.
Regards,
Yong
-----Original Message-----
From: Kirk Jantzer [mailto:kirk.jantzer@gmail.com]
Sent: Monday, 15 July 2013 10:47 AM
To: Cloudstack users mailing list
Subject: Re: The snapshot chain is too long
Don't think of CS snapshots the same way you do as VMWare snapshots.
CloudStack snapshots are WHOLE COPY BACKUPS of the instance. Also, in regards to snapshot chains, I know VMWare has a default setting per VM of
32 snapshots. If you exceed this, it will stop you from creating a new one.
You can, however, modify that setting so that you can create more. I am not sure if XenServer has the same type of setting.
On Sun, Jul 14, 2013 at 8:43 PM, Yong Chen <Yo...@coredesktop.com> wrote:
> Hi,
>
> I'm using CS 4.0.1 and XS 6.0.2. There is a problem with taking
> snapshots of instances.
>
> Some of VMs have scheduled snapshots (e.g. weekly, monthly) so
> eventually there will be quite some number of snapshots on secondary
> storage. This is fine as I read CloudStack should copy snapshots to
> secondary storage and then coalesce snapshots on primary storage so
> can keep the chain not too long (like 3-4 child normally).
>
> However I found after snapshot been taken for a while it starts
> failing due to error logged on CS Management log
> 'SR_BACKEND_FAILURE_109The snapshot chain is too long'. I verified it
> by checking with vhd-util and it does show the affected VHD has
> exactly 30 snapshots in chain which is the limit for XenServer.
>
> I tried to search for info why the chain can grow to 30 and only found
> this article for CloudStack 2.2
> http://support.citrix.com/article/CTX133470 seems to be related. It
> mentioned there maybe problem preventing XenServer from cleaning up
> snapshots. However I can't find any more error info other than CS
> Management log 'SR_BACKEND_FAILURE_109The snapshot chain is too long'.
>
> What can cause XenServer or CloudStack not cleaning up snapshots from
> primary storage? Thanks in advance.
>
> Yong
>
--
Regards,
Kirk Jantzer
c: (678) 561-5475
http://about.met/kirkjantzer
Re: The snapshot chain is too long
Posted by Kirk Jantzer <ki...@gmail.com>.
Don't think of CS snapshots the same way you do as VMWare snapshots.
CloudStack snapshots are WHOLE COPY BACKUPS of the instance. Also, in
regards to snapshot chains, I know VMWare has a default setting per VM of
32 snapshots. If you exceed this, it will stop you from creating a new one.
You can, however, modify that setting so that you can create more. I am not
sure if XenServer has the same type of setting.
On Sun, Jul 14, 2013 at 8:43 PM, Yong Chen <Yo...@coredesktop.com> wrote:
> Hi,
>
> I'm using CS 4.0.1 and XS 6.0.2. There is a problem with taking snapshots
> of instances.
>
> Some of VMs have scheduled snapshots (e.g. weekly, monthly) so eventually
> there will be quite some number of snapshots on secondary storage. This is
> fine as I read CloudStack should copy snapshots to secondary storage and
> then coalesce snapshots on primary storage so can keep the chain not too
> long (like 3-4 child normally).
>
> However I found after snapshot been taken for a while it starts failing
> due to error logged on CS Management log 'SR_BACKEND_FAILURE_109The
> snapshot chain is too long'. I verified it by checking with vhd-util and it
> does show the affected VHD has exactly 30 snapshots in chain which is the
> limit for XenServer.
>
> I tried to search for info why the chain can grow to 30 and only found
> this article for CloudStack 2.2
> http://support.citrix.com/article/CTX133470 seems to be related. It
> mentioned there maybe problem preventing XenServer from cleaning up
> snapshots. However I can't find any more error info other than CS
> Management log 'SR_BACKEND_FAILURE_109The snapshot chain is too long'.
>
> What can cause XenServer or CloudStack not cleaning up snapshots from
> primary storage? Thanks in advance.
>
> Yong
>
--
Regards,
Kirk Jantzer
c: (678) 561-5475
http://about.met/kirkjantzer