You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by "S. Brüseke - proIO GmbH" <s....@proio.com> on 2016/04/05 09:51:05 UTC
snapshot cleanup on hypervisor primary storage
Hey guys,
we are using CS 4.5 with XenServer 6.5 SP1 and observed a behavior with volume-snapshots that will fill up your primary storage over time:
Workflow:
1. create an instance
2. create a volume-snapshot of the ROOT-disk of that instance
3. delete instance and expunge it
4. check primary storage of XenServer. The latest snapshot of each volume-snapshot will stay on primary storage and is not being deleted even after waiting for storage.cleanup.interval
Can somebody reproduce this?
As far as I understand the workflow of volume-snapshots CS will XenServer ask to do a snapshot of the vm and then copy this snapshot to secondary storage. But why CS is not deleting the snapshot on primary storage after a success copy to secondary storage? Is this a "broken" workflow or is there a reason for that?
Is this the same behavior in newer releases of CS?
Mit freundlichen Grüßen / With kind regards,
Swen
- proIO GmbH -
Geschäftsführer: Swen Brüseke
Sitz der Gesellschaft: Frankfurt am Main
USt-IdNr. DE 267 075 918
Registergericht: Frankfurt am Main - HRB 86239
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen.
Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben,
informieren Sie bitte sofort den Absender und vernichten Sie diese Mail.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail sind nicht gestattet.
This e-mail may contain confidential and/or privileged information.
If you are not the intended recipient (or have received this e-mail in error) please notify
the sender immediately and destroy this e-mail.
Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.
AW: snapshot cleanup on hypervisor primary storage
Posted by "S. Brüseke - proIO GmbH" <s....@proio.com>.
XenSever is not deleting snapshots automatically when you delete a vm. XenCenter is asking you what snapshot you want to delete.
I do not think that is a problem of our environment, I really think this is happening in every CS / XenServer installation.
Can somebody confirm this?
Mit freundlichen Grüßen / With kind regards,
Swen
-----Ursprüngliche Nachricht-----
Von: Pavan Bandarupally [mailto:pavan.bandarupally@accelerite.com]
Gesendet: Dienstag, 5. April 2016 14:03
An: S. Brüseke - proIO GmbH; users@cloudstack.apache.org
Betreff: RE: snapshot cleanup on hypervisor primary storage
Hi Swen,
Cloud Stack just hands over the snapshot operation to XenServer via XenServer commands. As mentioned earlier if the volume (VDI/VBD) is deleted all the associated snapshots will be cleaned up by Xenserver.
My assumption here is that the volume that you have deleted from cloud stack is not deleted in the XenServer and hence the snapshots are not getting deleted ? Can you please check on XenServer if the VDI chain (for the deleted volume) is still present in the SR ?
I am not exactly aware of any changes in functionality from 4.5.1 but you can try one thing to isolate if the issue is with CS version or XenServer. Can you please create a standalone VM on Xenserver and take a snapshot of the VDI ? Delete the VDI and check if the snapshots are getting deleted ?
Regards,
Pavan
-----Original Message-----
From: S. Brüseke - proIO GmbH [mailto:s.brueseke@proio.com]
Sent: Tuesday, April 05, 2016 4:54 PM
To: Pavan Bandarupally; users@cloudstack.apache.org
Subject: AW: snapshot cleanup on hypervisor primary storage
Hi Pavan,
XenServer is not deleting the snapshots even after do a rescan:
Apr 5 13:20:59 cp-test-xs-2 SMGC: [24074] No work, exiting Apr 5 13:20:59 cp-test-xs-2 SMGC: [24074] In cleanup
I am really not sure why he should do it in first place. Can you please tell me more about it? Please have in mind that we are using CS 4.5.1 and not the newest version.
Mit freundlichen Grüßen / With kind regards,
Swen
-----Ursprüngliche Nachricht-----
Von: Pavan Bandarupally [mailto:pavan.bandarupally@accelerite.com]
Gesendet: Dienstag, 5. April 2016 13:14
An: users@cloudstack.apache.org; S. Brüseke - proIO GmbH
Betreff: RE: snapshot cleanup on hypervisor primary storage
Hi Swen,
Yes , if you have deleted the volume, the snapshots of that volume should be cleaned from primary storage by Xenserver.
Can you please run sr scan and check if they are getting cleaned up (running sr scan will manually trigger the GC on Xenserver which should clean up these if GC is not run for some reason) ?
Regards,
Pavan
-----Original Message-----
From: S. Brüseke - proIO GmbH [mailto:s.brueseke@proio.com]
Sent: Tuesday, April 05, 2016 4:05 PM
To: users@cloudstack.apache.org
Subject: AW: snapshot cleanup on hypervisor primary storage
Hi Pavan,
snapshots are working fine. Are you sure that snapshots on primary storage should be deleted?
I did some testing and observed the following:
First volume-snapshot of an instance
1. you create a volume-snapshot in CS UI 2. XenServer is taking a snapshot of the vm 3. XenServer is mounting secondary storage 4. XenServer is copying snapshot to secondary storage 5. XenServer is unmounting secondary storage
Second (or more) volume-snapshot of an instance 1. you create a volume-snapshot in CS UI 2. XenServer is taking a snapshot of the vm and uses the first snapshot as parent (you are unable to see this in XenCenter because the parent snapshot will be hidden) 3. XenServer is mounting secondary storage 4. XenServer is copying only delta of snapshot to secondary storage into the same directory as the first snapshot 5. XenServer is unmounting secondary storage
If you delete the a snapshot via XenCenter all following volume-snapshots are going to fail because of missing parent!
If you migrate vm to another XenServer host and create a volume-snapshot it will work. Reason for that is that CS is starting from the beginning here and handles this as a first snapshot. CS also uses a new folder on secondary storage for this volume-snapshot.
The last snapshot on XenServer will always be there, because CS needs it as parent for the next one and XenServer is creating a vhd chain.
The questions here are:
1. When I delete an instance is the snapshot on XenServer still needed?
I think now, it can be deleted even the volume-snapshot is still on secondary storage.
2. When I delete all volume-snapshots in CS UI of the snapshot-chain will CS delete the snapshots on XenServer?
As far as I can see, no.
3. Who is cleaning up the snapshots on XenServer's primary storage when you do a storage migration (normal and live) to another primary storage?
Right now, nobody is doing this and if you are using storage migration a lot (if you are using local storage you will do it a lot) than you end up with GBs of unwanted data on your primary storages.
Mit freundlichen Grüßen / With kind regards,
Swen
-----Ursprüngliche Nachricht-----
Von: Pavan Bandarupally [mailto:pavan.bandarupally@accelerite.com]
Gesendet: Dienstag, 5. April 2016 11:17
An: users@cloudstack.apache.org; S. Brüseke - proIO GmbH
Betreff: RE: snapshot cleanup on hypervisor primary storage
Hi Swen,
I don't think this is expected. The snapshots should get cleaned from Primary Storage by XenServer. Can you please check if the snapshots are usable or corrupted ?
Regarding deletion of snapshots on expunging the VM is not expected, because we do keep the snapshots (in secondary store) for further usage as templates / volumes can be created from the snapshots irrespective of whether the VM from whose disk the snapshot was created exists or not.
Regards,
Pavan
-----Original Message-----
From: S. Brüseke - proIO GmbH [mailto:s.brueseke@proio.com]
Sent: Tuesday, April 05, 2016 1:21 PM
To: users@cloudstack.apache.org
Subject: snapshot cleanup on hypervisor primary storage
Hey guys,
we are using CS 4.5 with XenServer 6.5 SP1 and observed a behavior with volume-snapshots that will fill up your primary storage over time:
Workflow:
1. create an instance
2. create a volume-snapshot of the ROOT-disk of that instance 3. delete instance and expunge it 4. check primary storage of XenServer. The latest snapshot of each volume-snapshot will stay on primary storage and is not being deleted even after waiting for storage.cleanup.interval
Can somebody reproduce this?
As far as I understand the workflow of volume-snapshots CS will XenServer ask to do a snapshot of the vm and then copy this snapshot to secondary storage. But why CS is not deleting the snapshot on primary storage after a success copy to secondary storage? Is this a "broken" workflow or is there a reason for that?
Is this the same behavior in newer releases of CS?
Mit freundlichen Grüßen / With kind regards,
Swen
- proIO GmbH -
Geschäftsführer: Swen Brüseke
Sitz der Gesellschaft: Frankfurt am Main
USt-IdNr. DE 267 075 918
Registergericht: Frankfurt am Main - HRB 86239
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen.
Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail sind nicht gestattet.
This e-mail may contain confidential and/or privileged information.
If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail.
Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
- proIO GmbH -
Geschäftsführer: Swen Brüseke
Sitz der Gesellschaft: Frankfurt am Main
USt-IdNr. DE 267 075 918
Registergericht: Frankfurt am Main - HRB 86239
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen.
Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail sind nicht gestattet.
This e-mail may contain confidential and/or privileged information.
If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail.
Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
- proIO GmbH -
Geschäftsführer: Swen Brüseke
Sitz der Gesellschaft: Frankfurt am Main
USt-IdNr. DE 267 075 918
Registergericht: Frankfurt am Main - HRB 86239
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen.
Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail sind nicht gestattet.
This e-mail may contain confidential and/or privileged information.
If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail.
Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
- proIO GmbH -
Geschäftsführer: Swen Brüseke
Sitz der Gesellschaft: Frankfurt am Main
USt-IdNr. DE 267 075 918
Registergericht: Frankfurt am Main - HRB 86239
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen.
Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben,
informieren Sie bitte sofort den Absender und vernichten Sie diese Mail.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail sind nicht gestattet.
This e-mail may contain confidential and/or privileged information.
If you are not the intended recipient (or have received this e-mail in error) please notify
the sender immediately and destroy this e-mail.
Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.
RE: snapshot cleanup on hypervisor primary storage
Posted by Pavan Bandarupally <pa...@accelerite.com>.
Hi Swen,
Cloud Stack just hands over the snapshot operation to XenServer via XenServer commands. As mentioned earlier if the volume (VDI/VBD) is deleted all the associated snapshots will be cleaned up by Xenserver.
My assumption here is that the volume that you have deleted from cloud stack is not deleted in the XenServer and hence the snapshots are not getting deleted ? Can you please check on XenServer if the VDI chain (for the deleted volume) is still present in the SR ?
I am not exactly aware of any changes in functionality from 4.5.1 but you can try one thing to isolate if the issue is with CS version or XenServer. Can you please create a standalone VM on Xenserver and take a snapshot of the VDI ? Delete the VDI and check if the snapshots are getting deleted ?
Regards,
Pavan
-----Original Message-----
From: S. Brüseke - proIO GmbH [mailto:s.brueseke@proio.com]
Sent: Tuesday, April 05, 2016 4:54 PM
To: Pavan Bandarupally; users@cloudstack.apache.org
Subject: AW: snapshot cleanup on hypervisor primary storage
Hi Pavan,
XenServer is not deleting the snapshots even after do a rescan:
Apr 5 13:20:59 cp-test-xs-2 SMGC: [24074] No work, exiting Apr 5 13:20:59 cp-test-xs-2 SMGC: [24074] In cleanup
I am really not sure why he should do it in first place. Can you please tell me more about it? Please have in mind that we are using CS 4.5.1 and not the newest version.
Mit freundlichen Grüßen / With kind regards,
Swen
-----Ursprüngliche Nachricht-----
Von: Pavan Bandarupally [mailto:pavan.bandarupally@accelerite.com]
Gesendet: Dienstag, 5. April 2016 13:14
An: users@cloudstack.apache.org; S. Brüseke - proIO GmbH
Betreff: RE: snapshot cleanup on hypervisor primary storage
Hi Swen,
Yes , if you have deleted the volume, the snapshots of that volume should be cleaned from primary storage by Xenserver.
Can you please run sr scan and check if they are getting cleaned up (running sr scan will manually trigger the GC on Xenserver which should clean up these if GC is not run for some reason) ?
Regards,
Pavan
-----Original Message-----
From: S. Brüseke - proIO GmbH [mailto:s.brueseke@proio.com]
Sent: Tuesday, April 05, 2016 4:05 PM
To: users@cloudstack.apache.org
Subject: AW: snapshot cleanup on hypervisor primary storage
Hi Pavan,
snapshots are working fine. Are you sure that snapshots on primary storage should be deleted?
I did some testing and observed the following:
First volume-snapshot of an instance
1. you create a volume-snapshot in CS UI 2. XenServer is taking a snapshot of the vm 3. XenServer is mounting secondary storage 4. XenServer is copying snapshot to secondary storage 5. XenServer is unmounting secondary storage
Second (or more) volume-snapshot of an instance 1. you create a volume-snapshot in CS UI 2. XenServer is taking a snapshot of the vm and uses the first snapshot as parent (you are unable to see this in XenCenter because the parent snapshot will be hidden) 3. XenServer is mounting secondary storage 4. XenServer is copying only delta of snapshot to secondary storage into the same directory as the first snapshot 5. XenServer is unmounting secondary storage
If you delete the a snapshot via XenCenter all following volume-snapshots are going to fail because of missing parent!
If you migrate vm to another XenServer host and create a volume-snapshot it will work. Reason for that is that CS is starting from the beginning here and handles this as a first snapshot. CS also uses a new folder on secondary storage for this volume-snapshot.
The last snapshot on XenServer will always be there, because CS needs it as parent for the next one and XenServer is creating a vhd chain.
The questions here are:
1. When I delete an instance is the snapshot on XenServer still needed?
I think now, it can be deleted even the volume-snapshot is still on secondary storage.
2. When I delete all volume-snapshots in CS UI of the snapshot-chain will CS delete the snapshots on XenServer?
As far as I can see, no.
3. Who is cleaning up the snapshots on XenServer's primary storage when you do a storage migration (normal and live) to another primary storage?
Right now, nobody is doing this and if you are using storage migration a lot (if you are using local storage you will do it a lot) than you end up with GBs of unwanted data on your primary storages.
Mit freundlichen Grüßen / With kind regards,
Swen
-----Ursprüngliche Nachricht-----
Von: Pavan Bandarupally [mailto:pavan.bandarupally@accelerite.com]
Gesendet: Dienstag, 5. April 2016 11:17
An: users@cloudstack.apache.org; S. Brüseke - proIO GmbH
Betreff: RE: snapshot cleanup on hypervisor primary storage
Hi Swen,
I don't think this is expected. The snapshots should get cleaned from Primary Storage by XenServer. Can you please check if the snapshots are usable or corrupted ?
Regarding deletion of snapshots on expunging the VM is not expected, because we do keep the snapshots (in secondary store) for further usage as templates / volumes can be created from the snapshots irrespective of whether the VM from whose disk the snapshot was created exists or not.
Regards,
Pavan
-----Original Message-----
From: S. Brüseke - proIO GmbH [mailto:s.brueseke@proio.com]
Sent: Tuesday, April 05, 2016 1:21 PM
To: users@cloudstack.apache.org
Subject: snapshot cleanup on hypervisor primary storage
Hey guys,
we are using CS 4.5 with XenServer 6.5 SP1 and observed a behavior with volume-snapshots that will fill up your primary storage over time:
Workflow:
1. create an instance
2. create a volume-snapshot of the ROOT-disk of that instance 3. delete instance and expunge it 4. check primary storage of XenServer. The latest snapshot of each volume-snapshot will stay on primary storage and is not being deleted even after waiting for storage.cleanup.interval
Can somebody reproduce this?
As far as I understand the workflow of volume-snapshots CS will XenServer ask to do a snapshot of the vm and then copy this snapshot to secondary storage. But why CS is not deleting the snapshot on primary storage after a success copy to secondary storage? Is this a "broken" workflow or is there a reason for that?
Is this the same behavior in newer releases of CS?
Mit freundlichen Grüßen / With kind regards,
Swen
- proIO GmbH -
Geschäftsführer: Swen Brüseke
Sitz der Gesellschaft: Frankfurt am Main
USt-IdNr. DE 267 075 918
Registergericht: Frankfurt am Main - HRB 86239
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen.
Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail sind nicht gestattet.
This e-mail may contain confidential and/or privileged information.
If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail.
Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
- proIO GmbH -
Geschäftsführer: Swen Brüseke
Sitz der Gesellschaft: Frankfurt am Main
USt-IdNr. DE 267 075 918
Registergericht: Frankfurt am Main - HRB 86239
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen.
Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail sind nicht gestattet.
This e-mail may contain confidential and/or privileged information.
If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail.
Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
- proIO GmbH -
Geschäftsführer: Swen Brüseke
Sitz der Gesellschaft: Frankfurt am Main
USt-IdNr. DE 267 075 918
Registergericht: Frankfurt am Main - HRB 86239
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen.
Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail sind nicht gestattet.
This e-mail may contain confidential and/or privileged information.
If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail.
Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
AW: snapshot cleanup on hypervisor primary storage
Posted by "S. Brüseke - proIO GmbH" <s....@proio.com>.
Hi Pavan,
XenServer is not deleting the snapshots even after do a rescan:
Apr 5 13:20:59 cp-test-xs-2 SMGC: [24074] No work, exiting
Apr 5 13:20:59 cp-test-xs-2 SMGC: [24074] In cleanup
I am really not sure why he should do it in first place. Can you please tell me more about it? Please have in mind that we are using CS 4.5.1 and not the newest version.
Mit freundlichen Grüßen / With kind regards,
Swen
-----Ursprüngliche Nachricht-----
Von: Pavan Bandarupally [mailto:pavan.bandarupally@accelerite.com]
Gesendet: Dienstag, 5. April 2016 13:14
An: users@cloudstack.apache.org; S. Brüseke - proIO GmbH
Betreff: RE: snapshot cleanup on hypervisor primary storage
Hi Swen,
Yes , if you have deleted the volume, the snapshots of that volume should be cleaned from primary storage by Xenserver.
Can you please run sr scan and check if they are getting cleaned up (running sr scan will manually trigger the GC on Xenserver which should clean up these if GC is not run for some reason) ?
Regards,
Pavan
-----Original Message-----
From: S. Brüseke - proIO GmbH [mailto:s.brueseke@proio.com]
Sent: Tuesday, April 05, 2016 4:05 PM
To: users@cloudstack.apache.org
Subject: AW: snapshot cleanup on hypervisor primary storage
Hi Pavan,
snapshots are working fine. Are you sure that snapshots on primary storage should be deleted?
I did some testing and observed the following:
First volume-snapshot of an instance
1. you create a volume-snapshot in CS UI 2. XenServer is taking a snapshot of the vm 3. XenServer is mounting secondary storage 4. XenServer is copying snapshot to secondary storage 5. XenServer is unmounting secondary storage
Second (or more) volume-snapshot of an instance 1. you create a volume-snapshot in CS UI 2. XenServer is taking a snapshot of the vm and uses the first snapshot as parent (you are unable to see this in XenCenter because the parent snapshot will be hidden) 3. XenServer is mounting secondary storage 4. XenServer is copying only delta of snapshot to secondary storage into the same directory as the first snapshot 5. XenServer is unmounting secondary storage
If you delete the a snapshot via XenCenter all following volume-snapshots are going to fail because of missing parent!
If you migrate vm to another XenServer host and create a volume-snapshot it will work. Reason for that is that CS is starting from the beginning here and handles this as a first snapshot. CS also uses a new folder on secondary storage for this volume-snapshot.
The last snapshot on XenServer will always be there, because CS needs it as parent for the next one and XenServer is creating a vhd chain.
The questions here are:
1. When I delete an instance is the snapshot on XenServer still needed?
I think now, it can be deleted even the volume-snapshot is still on secondary storage.
2. When I delete all volume-snapshots in CS UI of the snapshot-chain will CS delete the snapshots on XenServer?
As far as I can see, no.
3. Who is cleaning up the snapshots on XenServer's primary storage when you do a storage migration (normal and live) to another primary storage?
Right now, nobody is doing this and if you are using storage migration a lot (if you are using local storage you will do it a lot) than you end up with GBs of unwanted data on your primary storages.
Mit freundlichen Grüßen / With kind regards,
Swen
-----Ursprüngliche Nachricht-----
Von: Pavan Bandarupally [mailto:pavan.bandarupally@accelerite.com]
Gesendet: Dienstag, 5. April 2016 11:17
An: users@cloudstack.apache.org; S. Brüseke - proIO GmbH
Betreff: RE: snapshot cleanup on hypervisor primary storage
Hi Swen,
I don't think this is expected. The snapshots should get cleaned from Primary Storage by XenServer. Can you please check if the snapshots are usable or corrupted ?
Regarding deletion of snapshots on expunging the VM is not expected, because we do keep the snapshots (in secondary store) for further usage as templates / volumes can be created from the snapshots irrespective of whether the VM from whose disk the snapshot was created exists or not.
Regards,
Pavan
-----Original Message-----
From: S. Brüseke - proIO GmbH [mailto:s.brueseke@proio.com]
Sent: Tuesday, April 05, 2016 1:21 PM
To: users@cloudstack.apache.org
Subject: snapshot cleanup on hypervisor primary storage
Hey guys,
we are using CS 4.5 with XenServer 6.5 SP1 and observed a behavior with volume-snapshots that will fill up your primary storage over time:
Workflow:
1. create an instance
2. create a volume-snapshot of the ROOT-disk of that instance 3. delete instance and expunge it 4. check primary storage of XenServer. The latest snapshot of each volume-snapshot will stay on primary storage and is not being deleted even after waiting for storage.cleanup.interval
Can somebody reproduce this?
As far as I understand the workflow of volume-snapshots CS will XenServer ask to do a snapshot of the vm and then copy this snapshot to secondary storage. But why CS is not deleting the snapshot on primary storage after a success copy to secondary storage? Is this a "broken" workflow or is there a reason for that?
Is this the same behavior in newer releases of CS?
Mit freundlichen Grüßen / With kind regards,
Swen
- proIO GmbH -
Geschäftsführer: Swen Brüseke
Sitz der Gesellschaft: Frankfurt am Main
USt-IdNr. DE 267 075 918
Registergericht: Frankfurt am Main - HRB 86239
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen.
Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail sind nicht gestattet.
This e-mail may contain confidential and/or privileged information.
If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail.
Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
- proIO GmbH -
Geschäftsführer: Swen Brüseke
Sitz der Gesellschaft: Frankfurt am Main
USt-IdNr. DE 267 075 918
Registergericht: Frankfurt am Main - HRB 86239
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen.
Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail sind nicht gestattet.
This e-mail may contain confidential and/or privileged information.
If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail.
Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
- proIO GmbH -
Geschäftsführer: Swen Brüseke
Sitz der Gesellschaft: Frankfurt am Main
USt-IdNr. DE 267 075 918
Registergericht: Frankfurt am Main - HRB 86239
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen.
Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben,
informieren Sie bitte sofort den Absender und vernichten Sie diese Mail.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail sind nicht gestattet.
This e-mail may contain confidential and/or privileged information.
If you are not the intended recipient (or have received this e-mail in error) please notify
the sender immediately and destroy this e-mail.
Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.
RE: snapshot cleanup on hypervisor primary storage
Posted by Pavan Bandarupally <pa...@accelerite.com>.
Hi Swen,
Yes , if you have deleted the volume, the snapshots of that volume should be cleaned from primary storage by Xenserver.
Can you please run sr scan and check if they are getting cleaned up (running sr scan will manually trigger the GC on Xenserver which should clean up these if GC is not run for some reason) ?
Regards,
Pavan
-----Original Message-----
From: S. Brüseke - proIO GmbH [mailto:s.brueseke@proio.com]
Sent: Tuesday, April 05, 2016 4:05 PM
To: users@cloudstack.apache.org
Subject: AW: snapshot cleanup on hypervisor primary storage
Hi Pavan,
snapshots are working fine. Are you sure that snapshots on primary storage should be deleted?
I did some testing and observed the following:
First volume-snapshot of an instance
1. you create a volume-snapshot in CS UI 2. XenServer is taking a snapshot of the vm 3. XenServer is mounting secondary storage 4. XenServer is copying snapshot to secondary storage 5. XenServer is unmounting secondary storage
Second (or more) volume-snapshot of an instance 1. you create a volume-snapshot in CS UI 2. XenServer is taking a snapshot of the vm and uses the first snapshot as parent (you are unable to see this in XenCenter because the parent snapshot will be hidden) 3. XenServer is mounting secondary storage 4. XenServer is copying only delta of snapshot to secondary storage into the same directory as the first snapshot 5. XenServer is unmounting secondary storage
If you delete the a snapshot via XenCenter all following volume-snapshots are going to fail because of missing parent!
If you migrate vm to another XenServer host and create a volume-snapshot it will work. Reason for that is that CS is starting from the beginning here and handles this as a first snapshot. CS also uses a new folder on secondary storage for this volume-snapshot.
The last snapshot on XenServer will always be there, because CS needs it as parent for the next one and XenServer is creating a vhd chain.
The questions here are:
1. When I delete an instance is the snapshot on XenServer still needed?
I think now, it can be deleted even the volume-snapshot is still on secondary storage.
2. When I delete all volume-snapshots in CS UI of the snapshot-chain will CS delete the snapshots on XenServer?
As far as I can see, no.
3. Who is cleaning up the snapshots on XenServer's primary storage when you do a storage migration (normal and live) to another primary storage?
Right now, nobody is doing this and if you are using storage migration a lot (if you are using local storage you will do it a lot) than you end up with GBs of unwanted data on your primary storages.
Mit freundlichen Grüßen / With kind regards,
Swen
-----Ursprüngliche Nachricht-----
Von: Pavan Bandarupally [mailto:pavan.bandarupally@accelerite.com]
Gesendet: Dienstag, 5. April 2016 11:17
An: users@cloudstack.apache.org; S. Brüseke - proIO GmbH
Betreff: RE: snapshot cleanup on hypervisor primary storage
Hi Swen,
I don't think this is expected. The snapshots should get cleaned from Primary Storage by XenServer. Can you please check if the snapshots are usable or corrupted ?
Regarding deletion of snapshots on expunging the VM is not expected, because we do keep the snapshots (in secondary store) for further usage as templates / volumes can be created from the snapshots irrespective of whether the VM from whose disk the snapshot was created exists or not.
Regards,
Pavan
-----Original Message-----
From: S. Brüseke - proIO GmbH [mailto:s.brueseke@proio.com]
Sent: Tuesday, April 05, 2016 1:21 PM
To: users@cloudstack.apache.org
Subject: snapshot cleanup on hypervisor primary storage
Hey guys,
we are using CS 4.5 with XenServer 6.5 SP1 and observed a behavior with volume-snapshots that will fill up your primary storage over time:
Workflow:
1. create an instance
2. create a volume-snapshot of the ROOT-disk of that instance 3. delete instance and expunge it 4. check primary storage of XenServer. The latest snapshot of each volume-snapshot will stay on primary storage and is not being deleted even after waiting for storage.cleanup.interval
Can somebody reproduce this?
As far as I understand the workflow of volume-snapshots CS will XenServer ask to do a snapshot of the vm and then copy this snapshot to secondary storage. But why CS is not deleting the snapshot on primary storage after a success copy to secondary storage? Is this a "broken" workflow or is there a reason for that?
Is this the same behavior in newer releases of CS?
Mit freundlichen Grüßen / With kind regards,
Swen
- proIO GmbH -
Geschäftsführer: Swen Brüseke
Sitz der Gesellschaft: Frankfurt am Main
USt-IdNr. DE 267 075 918
Registergericht: Frankfurt am Main - HRB 86239
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen.
Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail sind nicht gestattet.
This e-mail may contain confidential and/or privileged information.
If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail.
Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
- proIO GmbH -
Geschäftsführer: Swen Brüseke
Sitz der Gesellschaft: Frankfurt am Main
USt-IdNr. DE 267 075 918
Registergericht: Frankfurt am Main - HRB 86239
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen.
Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail sind nicht gestattet.
This e-mail may contain confidential and/or privileged information.
If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail.
Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
AW: snapshot cleanup on hypervisor primary storage
Posted by "S. Brüseke - proIO GmbH" <s....@proio.com>.
Hi Pavan,
snapshots are working fine. Are you sure that snapshots on primary storage should be deleted?
I did some testing and observed the following:
First volume-snapshot of an instance
1. you create a volume-snapshot in CS UI
2. XenServer is taking a snapshot of the vm
3. XenServer is mounting secondary storage
4. XenServer is copying snapshot to secondary storage
5. XenServer is unmounting secondary storage
Second (or more) volume-snapshot of an instance
1. you create a volume-snapshot in CS UI
2. XenServer is taking a snapshot of the vm and uses the first snapshot as parent (you are unable to see this in XenCenter because the parent snapshot will be hidden)
3. XenServer is mounting secondary storage
4. XenServer is copying only delta of snapshot to secondary storage into the same directory as the first snapshot
5. XenServer is unmounting secondary storage
If you delete the a snapshot via XenCenter all following volume-snapshots are going to fail because of missing parent!
If you migrate vm to another XenServer host and create a volume-snapshot it will work. Reason for that is that CS is starting from the beginning here and handles this as a first snapshot. CS also uses a new folder on secondary storage for this volume-snapshot.
The last snapshot on XenServer will always be there, because CS needs it as parent for the next one and XenServer is creating a vhd chain.
The questions here are:
1. When I delete an instance is the snapshot on XenServer still needed?
I think now, it can be deleted even the volume-snapshot is still on secondary storage.
2. When I delete all volume-snapshots in CS UI of the snapshot-chain will CS delete the snapshots on XenServer?
As far as I can see, no.
3. Who is cleaning up the snapshots on XenServer's primary storage when you do a storage migration (normal and live) to another primary storage?
Right now, nobody is doing this and if you are using storage migration a lot (if you are using local storage you will do it a lot) than you end up with GBs of unwanted data on your primary storages.
Mit freundlichen Grüßen / With kind regards,
Swen
-----Ursprüngliche Nachricht-----
Von: Pavan Bandarupally [mailto:pavan.bandarupally@accelerite.com]
Gesendet: Dienstag, 5. April 2016 11:17
An: users@cloudstack.apache.org; S. Brüseke - proIO GmbH
Betreff: RE: snapshot cleanup on hypervisor primary storage
Hi Swen,
I don't think this is expected. The snapshots should get cleaned from Primary Storage by XenServer. Can you please check if the snapshots are usable or corrupted ?
Regarding deletion of snapshots on expunging the VM is not expected, because we do keep the snapshots (in secondary store) for further usage as templates / volumes can be created from the snapshots irrespective of whether the VM from whose disk the snapshot was created exists or not.
Regards,
Pavan
-----Original Message-----
From: S. Brüseke - proIO GmbH [mailto:s.brueseke@proio.com]
Sent: Tuesday, April 05, 2016 1:21 PM
To: users@cloudstack.apache.org
Subject: snapshot cleanup on hypervisor primary storage
Hey guys,
we are using CS 4.5 with XenServer 6.5 SP1 and observed a behavior with volume-snapshots that will fill up your primary storage over time:
Workflow:
1. create an instance
2. create a volume-snapshot of the ROOT-disk of that instance 3. delete instance and expunge it 4. check primary storage of XenServer. The latest snapshot of each volume-snapshot will stay on primary storage and is not being deleted even after waiting for storage.cleanup.interval
Can somebody reproduce this?
As far as I understand the workflow of volume-snapshots CS will XenServer ask to do a snapshot of the vm and then copy this snapshot to secondary storage. But why CS is not deleting the snapshot on primary storage after a success copy to secondary storage? Is this a "broken" workflow or is there a reason for that?
Is this the same behavior in newer releases of CS?
Mit freundlichen Grüßen / With kind regards,
Swen
- proIO GmbH -
Geschäftsführer: Swen Brüseke
Sitz der Gesellschaft: Frankfurt am Main
USt-IdNr. DE 267 075 918
Registergericht: Frankfurt am Main - HRB 86239
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen.
Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail sind nicht gestattet.
This e-mail may contain confidential and/or privileged information.
If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail.
Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
- proIO GmbH -
Geschäftsführer: Swen Brüseke
Sitz der Gesellschaft: Frankfurt am Main
USt-IdNr. DE 267 075 918
Registergericht: Frankfurt am Main - HRB 86239
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen.
Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben,
informieren Sie bitte sofort den Absender und vernichten Sie diese Mail.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail sind nicht gestattet.
This e-mail may contain confidential and/or privileged information.
If you are not the intended recipient (or have received this e-mail in error) please notify
the sender immediately and destroy this e-mail.
Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.
RE: snapshot cleanup on hypervisor primary storage
Posted by Pavan Bandarupally <pa...@accelerite.com>.
Hi Swen,
I don't think this is expected. The snapshots should get cleaned from Primary Storage by XenServer. Can you please check if the snapshots are usable or corrupted ?
Regarding deletion of snapshots on expunging the VM is not expected, because we do keep the snapshots (in secondary store) for further usage as templates / volumes can be created from the snapshots irrespective of whether the VM from whose disk the snapshot was created exists or not.
Regards,
Pavan
-----Original Message-----
From: S. Brüseke - proIO GmbH [mailto:s.brueseke@proio.com]
Sent: Tuesday, April 05, 2016 1:21 PM
To: users@cloudstack.apache.org
Subject: snapshot cleanup on hypervisor primary storage
Hey guys,
we are using CS 4.5 with XenServer 6.5 SP1 and observed a behavior with volume-snapshots that will fill up your primary storage over time:
Workflow:
1. create an instance
2. create a volume-snapshot of the ROOT-disk of that instance 3. delete instance and expunge it 4. check primary storage of XenServer. The latest snapshot of each volume-snapshot will stay on primary storage and is not being deleted even after waiting for storage.cleanup.interval
Can somebody reproduce this?
As far as I understand the workflow of volume-snapshots CS will XenServer ask to do a snapshot of the vm and then copy this snapshot to secondary storage. But why CS is not deleting the snapshot on primary storage after a success copy to secondary storage? Is this a "broken" workflow or is there a reason for that?
Is this the same behavior in newer releases of CS?
Mit freundlichen Grüßen / With kind regards,
Swen
- proIO GmbH -
Geschäftsführer: Swen Brüseke
Sitz der Gesellschaft: Frankfurt am Main
USt-IdNr. DE 267 075 918
Registergericht: Frankfurt am Main - HRB 86239
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen.
Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail sind nicht gestattet.
This e-mail may contain confidential and/or privileged information.
If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail.
Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.