You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Bryan Whitehead <dr...@megahappy.net> on 2013/02/02 01:02:46 UTC
SystemVM offline causes extreme snapshot bloat
For some reason my storageVM was in a state where things "worked" but
automated snapshots were not working.
Basically the step where libvirt created a stapshot on the primary storage
worked - but the step to have the snapshot parent copied to the secondary
storage was not working. I got automated snapshots working again by
rebooting the storage systemVM.
However, when i look at libvirt I can see there are many snapshots on the
primary storage. A 12G filesystem has a qcow2 size of 220GB.
virsh snapshot-list --parent i-3-14-VM
Name Creation Time State Parent
------------------------------------------------------------
24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130125025313 2013-01-25
02:53:13 +0000 running
24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130126025313 2013-01-26
02:53:13 +0000 running
24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130125025313
24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130127025313 2013-01-27
02:53:13 +0000 running
24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130126025313
24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130128025313 2013-01-28
02:53:13 +0000 running
24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130127025313
24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130129025313 2013-01-29
02:53:13 +0000 running
24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130128025313
24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130130025313 2013-01-30
02:53:13 +0000 running
24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130129025313
24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130131025313 2013-01-31
02:53:13 +0000 running
24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130130025313
24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130201025313 2013-02-01
02:53:13 +0000 running
24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130131025313
24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130201223643 2013-02-01
22:36:43 +0000 running
24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130201025313
The actual qcow2 file looks like a much bigger wreck:
qemu-img info /gluster/qcow2/images/7ec192c1-ecee-4f5e-8e3d-567f7878ceb1
image: /gluster/qcow2/images/7ec192c1-ecee-4f5e-8e3d-567f7878ceb1
file format: qcow2
virtual size: 100G (107374182400 bytes)
disk size: 211G
cluster_size: 65536
backing file: /gluster/qcow2/images/cb151441-209c-4f43-a2a4-b390c9bb9768
(actual path: /gluster/qcow2/images/cb151441-209c-4f43-a2a4-b390c9bb9768)
Snapshot list:
ID TAG VM SIZE DATE VM CLOCK
1 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20121230025415
1.0G 2012-12-30 02:54:16 785:39:01.680
2 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20121231025415
1.0G 2012-12-31 02:54:16 809:35:35.417
3 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130101025415
1.0G 2013-01-01 02:54:16 833:24:20.046
4 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130102025415
1.0G 2013-01-02 02:54:15 857:13:02.766
5 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130103025415
1.0G 2013-01-03 02:54:15 881:01:42.467
6 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130104025415
1.0G 2013-01-04 02:54:15 904:50:29.840
7 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130105025415
1.0G 2013-01-05 02:54:16 928:39:17.670
8 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130106025415
1.0G 2013-01-06 02:54:16 952:28:07.139
9 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130107025415
1.1G 2013-01-07 02:54:16 976:16:58.396
10 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130108025415
1.1G 2013-01-08 02:54:15 1000:05:45.624
11 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130109025415
1.1G 2013-01-09 02:54:15 1023:54:25.482
12 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130110025415
1.1G 2013-01-10 02:54:16 1047:43:17.503
13 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130111025415
1.1G 2013-01-11 02:54:16 1071:32:03.747
14 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130112025415
1.1G 2013-01-12 02:54:16 1095:20:52.220
15 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130113025415
1.1G 2013-01-13 02:54:16 1119:09:43.387
16 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130114025415
1.1G 2013-01-14 02:54:16 1142:58:32.657
17 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130115025415
1.1G 2013-01-15 02:54:16 1166:47:20.084
18 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130116025416
1.1G 2013-01-16 02:54:16 1190:36:06.729
19 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130117025416
1.1G 2013-01-17 02:54:16 1214:24:53.904
20 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130118025415
1.1G 2013-01-18 02:54:16 1238:13:35.640
21 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130119025415
1.1G 2013-01-19 02:54:16 1262:02:17.839
22 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130120025415
1.1G 2013-01-20 02:54:16 1285:51:01.114
23 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130121025415
1.1G 2013-01-21 02:54:16 1309:39:49.956
24 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130122025416
1.1G 2013-01-22 02:54:16 1333:28:39.703
25 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130123025415
1.1G 2013-01-23 02:54:16 1357:17:25.426
26 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130124025416
1.1G 2013-01-24 02:54:16 1381:06:04.731
27 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130125025313
517M 2013-01-25 02:53:13 04:46:52.307
28 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130126025313
678M 2013-01-26 02:53:13 28:44:35.134
29 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130127025313
918M 2013-01-27 02:53:13 52:42:00.249
30 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130128025313
1.1G 2013-01-28 02:53:13 76:38:55.428
31 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130129025313
1.2G 2013-01-29 02:53:13 100:35:27.978
32 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130130025313
1.2G 2013-01-30 02:53:13 124:31:55.179
33 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130131025313
1.3G 2013-01-31 02:53:13 148:28:17.744
34 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130201025313
1.3G 2013-02-01 02:53:13 172:24:33.630
35 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130201223643
1.3G 2013-02-01 22:36:43 192:04:17.498
Suggestions on how I can clean up these extra snapshots?
-Bryan
Re: SystemVM offline causes extreme snapshot bloat
Posted by Ahmad Emneina <ae...@gmail.com>.
sounds painful. thanks for sharing this though. Hopefully someone can gain
value from this, and incorporate this into their operations.
On Thu, Feb 7, 2013 at 1:19 PM, Bryan Whitehead <dr...@megahappy.net>wrote:
> This is on cloudstack 3.0.2. I had already removed all the snapshots in
> the cloudstack UI. So from the cloudstack point of view there was zero
> snapshots when in fact there was many more.
>
> The libvirt guys in #virt IRC were able to help me clean up all the
> internal snapshots on the qcow2 file via "virsh snapshot-delete [...]
> [...]". Unfortunately I cannot recover the 200G of space the qcow2 file has
> grown to without shutting down the VM since I'm running qemu/libvirt that
> is too old (CentOS 6.3 + some patches).
>
> According to libvirt there is 0 snapshots left but directly running
> "qemu-img info [qcow2 file]" shows there are still ~10 snapshots. (look at
> my original email - you can see the difference between "qemu-img info"
> output and "virsh snapshot-list"
>
> Anyhow, scheduled snapshots is working again just fine... including
> auto-rotation of a limit of 3-backups. But I will need to more closely
> monitor logs for systemvm storage problems... I can deal with a 220G qcow2
> file. Since many internal snapshots have been deleted there is a lot of
> "free" space in the qcow2 file - so hopefully it won't grow again if this
> happens again.
>
>
> On Tue, Feb 5, 2013 at 1:33 PM, Ahmad Emneina <ae...@gmail.com> wrote:
>
>> Bryan* ... sorry
>>
>>
>> On Tue, Feb 5, 2013 at 1:33 PM, Ahmad Emneina <ae...@gmail.com> wrote:
>>
>> > I'd also file a bug for this Brian. The process was failing, and not
>> > recovering gracefully, by cleaning up after itself.
>> >
>> >
>> > On Tue, Feb 5, 2013 at 1:18 PM, Ahmad Emneina <ae...@gmail.com>
>> wrote:
>> >
>> >> I think in 4.0 and master this was broken. Marcus or Edison could
>> >> probably comment to that. Can you try deleting the snapshots from the
>> >> cloudstack UI? If that doesnt work, we can work to devise a method
>> where we
>> >> query the db, identify valid snapshots, then using virsh commands
>> delete
>> >> the failed snapshots.
>> >>
>> >>
>> >> On Fri, Feb 1, 2013 at 4:02 PM, Bryan Whitehead <driver@megahappy.net
>> >wrote:
>> >>
>> >>> For some reason my storageVM was in a state where things "worked" but
>> >>> automated snapshots were not working.
>> >>>
>> >>> Basically the step where libvirt created a stapshot on the primary
>> >>> storage
>> >>> worked - but the step to have the snapshot parent copied to the
>> secondary
>> >>> storage was not working. I got automated snapshots working again by
>> >>> rebooting the storage systemVM.
>> >>>
>> >>> However, when i look at libvirt I can see there are many snapshots on
>> the
>> >>> primary storage. A 12G filesystem has a qcow2 size of 220GB.
>> >>>
>> >>> virsh snapshot-list --parent i-3-14-VM
>> >>> Name Creation Time State Parent
>> >>> ------------------------------------------------------------
>> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130125025313
>> 2013-01-25
>> >>> 02:53:13 +0000 running
>> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130126025313
>> 2013-01-26
>> >>> 02:53:13 +0000 running
>> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130125025313
>> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130127025313
>> 2013-01-27
>> >>> 02:53:13 +0000 running
>> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130126025313
>> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130128025313
>> 2013-01-28
>> >>> 02:53:13 +0000 running
>> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130127025313
>> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130129025313
>> 2013-01-29
>> >>> 02:53:13 +0000 running
>> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130128025313
>> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130130025313
>> 2013-01-30
>> >>> 02:53:13 +0000 running
>> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130129025313
>> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130131025313
>> 2013-01-31
>> >>> 02:53:13 +0000 running
>> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130130025313
>> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130201025313
>> 2013-02-01
>> >>> 02:53:13 +0000 running
>> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130131025313
>> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130201223643
>> 2013-02-01
>> >>> 22:36:43 +0000 running
>> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130201025313
>> >>>
>> >>> The actual qcow2 file looks like a much bigger wreck:
>> >>>
>> >>> qemu-img info
>> /gluster/qcow2/images/7ec192c1-ecee-4f5e-8e3d-567f7878ceb1
>> >>> image: /gluster/qcow2/images/7ec192c1-ecee-4f5e-8e3d-567f7878ceb1
>> >>> file format: qcow2
>> >>> virtual size: 100G (107374182400 bytes)
>> >>> disk size: 211G
>> >>> cluster_size: 65536
>> >>> backing file:
>> /gluster/qcow2/images/cb151441-209c-4f43-a2a4-b390c9bb9768
>> >>> (actual path:
>> /gluster/qcow2/images/cb151441-209c-4f43-a2a4-b390c9bb9768)
>> >>> Snapshot list:
>> >>> ID TAG VM SIZE DATE VM
>> CLOCK
>> >>> 1 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20121230025415
>> >>> 1.0G 2012-12-30 02:54:16 785:39:01.680
>> >>> 2 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20121231025415
>> >>> 1.0G 2012-12-31 02:54:16 809:35:35.417
>> >>> 3 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130101025415
>> >>> 1.0G 2013-01-01 02:54:16 833:24:20.046
>> >>> 4 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130102025415
>> >>> 1.0G 2013-01-02 02:54:15 857:13:02.766
>> >>> 5 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130103025415
>> >>> 1.0G 2013-01-03 02:54:15 881:01:42.467
>> >>> 6 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130104025415
>> >>> 1.0G 2013-01-04 02:54:15 904:50:29.840
>> >>> 7 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130105025415
>> >>> 1.0G 2013-01-05 02:54:16 928:39:17.670
>> >>> 8 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130106025415
>> >>> 1.0G 2013-01-06 02:54:16 952:28:07.139
>> >>> 9 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130107025415
>> >>> 1.1G 2013-01-07 02:54:16 976:16:58.396
>> >>> 10 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130108025415
>> >>> 1.1G 2013-01-08 02:54:15 1000:05:45.624
>> >>> 11 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130109025415
>> >>> 1.1G 2013-01-09 02:54:15 1023:54:25.482
>> >>> 12 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130110025415
>> >>> 1.1G 2013-01-10 02:54:16 1047:43:17.503
>> >>> 13 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130111025415
>> >>> 1.1G 2013-01-11 02:54:16 1071:32:03.747
>> >>> 14 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130112025415
>> >>> 1.1G 2013-01-12 02:54:16 1095:20:52.220
>> >>> 15 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130113025415
>> >>> 1.1G 2013-01-13 02:54:16 1119:09:43.387
>> >>> 16 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130114025415
>> >>> 1.1G 2013-01-14 02:54:16 1142:58:32.657
>> >>> 17 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130115025415
>> >>> 1.1G 2013-01-15 02:54:16 1166:47:20.084
>> >>> 18 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130116025416
>> >>> 1.1G 2013-01-16 02:54:16 1190:36:06.729
>> >>> 19 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130117025416
>> >>> 1.1G 2013-01-17 02:54:16 1214:24:53.904
>> >>> 20 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130118025415
>> >>> 1.1G 2013-01-18 02:54:16 1238:13:35.640
>> >>> 21 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130119025415
>> >>> 1.1G 2013-01-19 02:54:16 1262:02:17.839
>> >>> 22 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130120025415
>> >>> 1.1G 2013-01-20 02:54:16 1285:51:01.114
>> >>> 23 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130121025415
>> >>> 1.1G 2013-01-21 02:54:16 1309:39:49.956
>> >>> 24 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130122025416
>> >>> 1.1G 2013-01-22 02:54:16 1333:28:39.703
>> >>> 25 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130123025415
>> >>> 1.1G 2013-01-23 02:54:16 1357:17:25.426
>> >>> 26 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130124025416
>> >>> 1.1G 2013-01-24 02:54:16 1381:06:04.731
>> >>> 27 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130125025313
>> >>> 517M 2013-01-25 02:53:13 04:46:52.307
>> >>> 28 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130126025313
>> >>> 678M 2013-01-26 02:53:13 28:44:35.134
>> >>> 29 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130127025313
>> >>> 918M 2013-01-27 02:53:13 52:42:00.249
>> >>> 30 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130128025313
>> >>> 1.1G 2013-01-28 02:53:13 76:38:55.428
>> >>> 31 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130129025313
>> >>> 1.2G 2013-01-29 02:53:13 100:35:27.978
>> >>> 32 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130130025313
>> >>> 1.2G 2013-01-30 02:53:13 124:31:55.179
>> >>> 33 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130131025313
>> >>> 1.3G 2013-01-31 02:53:13 148:28:17.744
>> >>> 34 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130201025313
>> >>> 1.3G 2013-02-01 02:53:13 172:24:33.630
>> >>> 35 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130201223643
>> >>> 1.3G 2013-02-01 22:36:43 192:04:17.498
>> >>>
>> >>> Suggestions on how I can clean up these extra snapshots?
>> >>>
>> >>> -Bryan
>> >>>
>> >>
>> >>
>> >
>>
>
>
Re: SystemVM offline causes extreme snapshot bloat
Posted by Bryan Whitehead <dr...@megahappy.net>.
This is on cloudstack 3.0.2. I had already removed all the snapshots in the
cloudstack UI. So from the cloudstack point of view there was zero
snapshots when in fact there was many more.
The libvirt guys in #virt IRC were able to help me clean up all the
internal snapshots on the qcow2 file via "virsh snapshot-delete [...]
[...]". Unfortunately I cannot recover the 200G of space the qcow2 file has
grown to without shutting down the VM since I'm running qemu/libvirt that
is too old (CentOS 6.3 + some patches).
According to libvirt there is 0 snapshots left but directly running
"qemu-img info [qcow2 file]" shows there are still ~10 snapshots. (look at
my original email - you can see the difference between "qemu-img info"
output and "virsh snapshot-list"
Anyhow, scheduled snapshots is working again just fine... including
auto-rotation of a limit of 3-backups. But I will need to more closely
monitor logs for systemvm storage problems... I can deal with a 220G qcow2
file. Since many internal snapshots have been deleted there is a lot of
"free" space in the qcow2 file - so hopefully it won't grow again if this
happens again.
On Tue, Feb 5, 2013 at 1:33 PM, Ahmad Emneina <ae...@gmail.com> wrote:
> Bryan* ... sorry
>
>
> On Tue, Feb 5, 2013 at 1:33 PM, Ahmad Emneina <ae...@gmail.com> wrote:
>
> > I'd also file a bug for this Brian. The process was failing, and not
> > recovering gracefully, by cleaning up after itself.
> >
> >
> > On Tue, Feb 5, 2013 at 1:18 PM, Ahmad Emneina <ae...@gmail.com>
> wrote:
> >
> >> I think in 4.0 and master this was broken. Marcus or Edison could
> >> probably comment to that. Can you try deleting the snapshots from the
> >> cloudstack UI? If that doesnt work, we can work to devise a method
> where we
> >> query the db, identify valid snapshots, then using virsh commands delete
> >> the failed snapshots.
> >>
> >>
> >> On Fri, Feb 1, 2013 at 4:02 PM, Bryan Whitehead <driver@megahappy.net
> >wrote:
> >>
> >>> For some reason my storageVM was in a state where things "worked" but
> >>> automated snapshots were not working.
> >>>
> >>> Basically the step where libvirt created a stapshot on the primary
> >>> storage
> >>> worked - but the step to have the snapshot parent copied to the
> secondary
> >>> storage was not working. I got automated snapshots working again by
> >>> rebooting the storage systemVM.
> >>>
> >>> However, when i look at libvirt I can see there are many snapshots on
> the
> >>> primary storage. A 12G filesystem has a qcow2 size of 220GB.
> >>>
> >>> virsh snapshot-list --parent i-3-14-VM
> >>> Name Creation Time State Parent
> >>> ------------------------------------------------------------
> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130125025313 2013-01-25
> >>> 02:53:13 +0000 running
> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130126025313 2013-01-26
> >>> 02:53:13 +0000 running
> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130125025313
> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130127025313 2013-01-27
> >>> 02:53:13 +0000 running
> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130126025313
> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130128025313 2013-01-28
> >>> 02:53:13 +0000 running
> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130127025313
> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130129025313 2013-01-29
> >>> 02:53:13 +0000 running
> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130128025313
> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130130025313 2013-01-30
> >>> 02:53:13 +0000 running
> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130129025313
> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130131025313 2013-01-31
> >>> 02:53:13 +0000 running
> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130130025313
> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130201025313 2013-02-01
> >>> 02:53:13 +0000 running
> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130131025313
> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130201223643 2013-02-01
> >>> 22:36:43 +0000 running
> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130201025313
> >>>
> >>> The actual qcow2 file looks like a much bigger wreck:
> >>>
> >>> qemu-img info
> /gluster/qcow2/images/7ec192c1-ecee-4f5e-8e3d-567f7878ceb1
> >>> image: /gluster/qcow2/images/7ec192c1-ecee-4f5e-8e3d-567f7878ceb1
> >>> file format: qcow2
> >>> virtual size: 100G (107374182400 bytes)
> >>> disk size: 211G
> >>> cluster_size: 65536
> >>> backing file:
> /gluster/qcow2/images/cb151441-209c-4f43-a2a4-b390c9bb9768
> >>> (actual path:
> /gluster/qcow2/images/cb151441-209c-4f43-a2a4-b390c9bb9768)
> >>> Snapshot list:
> >>> ID TAG VM SIZE DATE VM
> CLOCK
> >>> 1 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20121230025415
> >>> 1.0G 2012-12-30 02:54:16 785:39:01.680
> >>> 2 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20121231025415
> >>> 1.0G 2012-12-31 02:54:16 809:35:35.417
> >>> 3 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130101025415
> >>> 1.0G 2013-01-01 02:54:16 833:24:20.046
> >>> 4 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130102025415
> >>> 1.0G 2013-01-02 02:54:15 857:13:02.766
> >>> 5 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130103025415
> >>> 1.0G 2013-01-03 02:54:15 881:01:42.467
> >>> 6 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130104025415
> >>> 1.0G 2013-01-04 02:54:15 904:50:29.840
> >>> 7 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130105025415
> >>> 1.0G 2013-01-05 02:54:16 928:39:17.670
> >>> 8 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130106025415
> >>> 1.0G 2013-01-06 02:54:16 952:28:07.139
> >>> 9 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130107025415
> >>> 1.1G 2013-01-07 02:54:16 976:16:58.396
> >>> 10 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130108025415
> >>> 1.1G 2013-01-08 02:54:15 1000:05:45.624
> >>> 11 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130109025415
> >>> 1.1G 2013-01-09 02:54:15 1023:54:25.482
> >>> 12 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130110025415
> >>> 1.1G 2013-01-10 02:54:16 1047:43:17.503
> >>> 13 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130111025415
> >>> 1.1G 2013-01-11 02:54:16 1071:32:03.747
> >>> 14 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130112025415
> >>> 1.1G 2013-01-12 02:54:16 1095:20:52.220
> >>> 15 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130113025415
> >>> 1.1G 2013-01-13 02:54:16 1119:09:43.387
> >>> 16 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130114025415
> >>> 1.1G 2013-01-14 02:54:16 1142:58:32.657
> >>> 17 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130115025415
> >>> 1.1G 2013-01-15 02:54:16 1166:47:20.084
> >>> 18 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130116025416
> >>> 1.1G 2013-01-16 02:54:16 1190:36:06.729
> >>> 19 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130117025416
> >>> 1.1G 2013-01-17 02:54:16 1214:24:53.904
> >>> 20 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130118025415
> >>> 1.1G 2013-01-18 02:54:16 1238:13:35.640
> >>> 21 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130119025415
> >>> 1.1G 2013-01-19 02:54:16 1262:02:17.839
> >>> 22 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130120025415
> >>> 1.1G 2013-01-20 02:54:16 1285:51:01.114
> >>> 23 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130121025415
> >>> 1.1G 2013-01-21 02:54:16 1309:39:49.956
> >>> 24 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130122025416
> >>> 1.1G 2013-01-22 02:54:16 1333:28:39.703
> >>> 25 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130123025415
> >>> 1.1G 2013-01-23 02:54:16 1357:17:25.426
> >>> 26 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130124025416
> >>> 1.1G 2013-01-24 02:54:16 1381:06:04.731
> >>> 27 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130125025313
> >>> 517M 2013-01-25 02:53:13 04:46:52.307
> >>> 28 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130126025313
> >>> 678M 2013-01-26 02:53:13 28:44:35.134
> >>> 29 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130127025313
> >>> 918M 2013-01-27 02:53:13 52:42:00.249
> >>> 30 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130128025313
> >>> 1.1G 2013-01-28 02:53:13 76:38:55.428
> >>> 31 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130129025313
> >>> 1.2G 2013-01-29 02:53:13 100:35:27.978
> >>> 32 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130130025313
> >>> 1.2G 2013-01-30 02:53:13 124:31:55.179
> >>> 33 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130131025313
> >>> 1.3G 2013-01-31 02:53:13 148:28:17.744
> >>> 34 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130201025313
> >>> 1.3G 2013-02-01 02:53:13 172:24:33.630
> >>> 35 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130201223643
> >>> 1.3G 2013-02-01 22:36:43 192:04:17.498
> >>>
> >>> Suggestions on how I can clean up these extra snapshots?
> >>>
> >>> -Bryan
> >>>
> >>
> >>
> >
>
Re: SystemVM offline causes extreme snapshot bloat
Posted by Ahmad Emneina <ae...@gmail.com>.
Bryan* ... sorry
On Tue, Feb 5, 2013 at 1:33 PM, Ahmad Emneina <ae...@gmail.com> wrote:
> I'd also file a bug for this Brian. The process was failing, and not
> recovering gracefully, by cleaning up after itself.
>
>
> On Tue, Feb 5, 2013 at 1:18 PM, Ahmad Emneina <ae...@gmail.com> wrote:
>
>> I think in 4.0 and master this was broken. Marcus or Edison could
>> probably comment to that. Can you try deleting the snapshots from the
>> cloudstack UI? If that doesnt work, we can work to devise a method where we
>> query the db, identify valid snapshots, then using virsh commands delete
>> the failed snapshots.
>>
>>
>> On Fri, Feb 1, 2013 at 4:02 PM, Bryan Whitehead <dr...@megahappy.net>wrote:
>>
>>> For some reason my storageVM was in a state where things "worked" but
>>> automated snapshots were not working.
>>>
>>> Basically the step where libvirt created a stapshot on the primary
>>> storage
>>> worked - but the step to have the snapshot parent copied to the secondary
>>> storage was not working. I got automated snapshots working again by
>>> rebooting the storage systemVM.
>>>
>>> However, when i look at libvirt I can see there are many snapshots on the
>>> primary storage. A 12G filesystem has a qcow2 size of 220GB.
>>>
>>> virsh snapshot-list --parent i-3-14-VM
>>> Name Creation Time State Parent
>>> ------------------------------------------------------------
>>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130125025313 2013-01-25
>>> 02:53:13 +0000 running
>>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130126025313 2013-01-26
>>> 02:53:13 +0000 running
>>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130125025313
>>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130127025313 2013-01-27
>>> 02:53:13 +0000 running
>>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130126025313
>>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130128025313 2013-01-28
>>> 02:53:13 +0000 running
>>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130127025313
>>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130129025313 2013-01-29
>>> 02:53:13 +0000 running
>>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130128025313
>>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130130025313 2013-01-30
>>> 02:53:13 +0000 running
>>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130129025313
>>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130131025313 2013-01-31
>>> 02:53:13 +0000 running
>>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130130025313
>>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130201025313 2013-02-01
>>> 02:53:13 +0000 running
>>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130131025313
>>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130201223643 2013-02-01
>>> 22:36:43 +0000 running
>>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130201025313
>>>
>>> The actual qcow2 file looks like a much bigger wreck:
>>>
>>> qemu-img info /gluster/qcow2/images/7ec192c1-ecee-4f5e-8e3d-567f7878ceb1
>>> image: /gluster/qcow2/images/7ec192c1-ecee-4f5e-8e3d-567f7878ceb1
>>> file format: qcow2
>>> virtual size: 100G (107374182400 bytes)
>>> disk size: 211G
>>> cluster_size: 65536
>>> backing file: /gluster/qcow2/images/cb151441-209c-4f43-a2a4-b390c9bb9768
>>> (actual path: /gluster/qcow2/images/cb151441-209c-4f43-a2a4-b390c9bb9768)
>>> Snapshot list:
>>> ID TAG VM SIZE DATE VM CLOCK
>>> 1 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20121230025415
>>> 1.0G 2012-12-30 02:54:16 785:39:01.680
>>> 2 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20121231025415
>>> 1.0G 2012-12-31 02:54:16 809:35:35.417
>>> 3 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130101025415
>>> 1.0G 2013-01-01 02:54:16 833:24:20.046
>>> 4 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130102025415
>>> 1.0G 2013-01-02 02:54:15 857:13:02.766
>>> 5 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130103025415
>>> 1.0G 2013-01-03 02:54:15 881:01:42.467
>>> 6 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130104025415
>>> 1.0G 2013-01-04 02:54:15 904:50:29.840
>>> 7 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130105025415
>>> 1.0G 2013-01-05 02:54:16 928:39:17.670
>>> 8 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130106025415
>>> 1.0G 2013-01-06 02:54:16 952:28:07.139
>>> 9 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130107025415
>>> 1.1G 2013-01-07 02:54:16 976:16:58.396
>>> 10 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130108025415
>>> 1.1G 2013-01-08 02:54:15 1000:05:45.624
>>> 11 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130109025415
>>> 1.1G 2013-01-09 02:54:15 1023:54:25.482
>>> 12 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130110025415
>>> 1.1G 2013-01-10 02:54:16 1047:43:17.503
>>> 13 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130111025415
>>> 1.1G 2013-01-11 02:54:16 1071:32:03.747
>>> 14 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130112025415
>>> 1.1G 2013-01-12 02:54:16 1095:20:52.220
>>> 15 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130113025415
>>> 1.1G 2013-01-13 02:54:16 1119:09:43.387
>>> 16 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130114025415
>>> 1.1G 2013-01-14 02:54:16 1142:58:32.657
>>> 17 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130115025415
>>> 1.1G 2013-01-15 02:54:16 1166:47:20.084
>>> 18 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130116025416
>>> 1.1G 2013-01-16 02:54:16 1190:36:06.729
>>> 19 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130117025416
>>> 1.1G 2013-01-17 02:54:16 1214:24:53.904
>>> 20 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130118025415
>>> 1.1G 2013-01-18 02:54:16 1238:13:35.640
>>> 21 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130119025415
>>> 1.1G 2013-01-19 02:54:16 1262:02:17.839
>>> 22 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130120025415
>>> 1.1G 2013-01-20 02:54:16 1285:51:01.114
>>> 23 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130121025415
>>> 1.1G 2013-01-21 02:54:16 1309:39:49.956
>>> 24 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130122025416
>>> 1.1G 2013-01-22 02:54:16 1333:28:39.703
>>> 25 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130123025415
>>> 1.1G 2013-01-23 02:54:16 1357:17:25.426
>>> 26 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130124025416
>>> 1.1G 2013-01-24 02:54:16 1381:06:04.731
>>> 27 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130125025313
>>> 517M 2013-01-25 02:53:13 04:46:52.307
>>> 28 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130126025313
>>> 678M 2013-01-26 02:53:13 28:44:35.134
>>> 29 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130127025313
>>> 918M 2013-01-27 02:53:13 52:42:00.249
>>> 30 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130128025313
>>> 1.1G 2013-01-28 02:53:13 76:38:55.428
>>> 31 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130129025313
>>> 1.2G 2013-01-29 02:53:13 100:35:27.978
>>> 32 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130130025313
>>> 1.2G 2013-01-30 02:53:13 124:31:55.179
>>> 33 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130131025313
>>> 1.3G 2013-01-31 02:53:13 148:28:17.744
>>> 34 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130201025313
>>> 1.3G 2013-02-01 02:53:13 172:24:33.630
>>> 35 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130201223643
>>> 1.3G 2013-02-01 22:36:43 192:04:17.498
>>>
>>> Suggestions on how I can clean up these extra snapshots?
>>>
>>> -Bryan
>>>
>>
>>
>
Re: SystemVM offline causes extreme snapshot bloat
Posted by Ahmad Emneina <ae...@gmail.com>.
I'd also file a bug for this Brian. The process was failing, and not
recovering gracefully, by cleaning up after itself.
On Tue, Feb 5, 2013 at 1:18 PM, Ahmad Emneina <ae...@gmail.com> wrote:
> I think in 4.0 and master this was broken. Marcus or Edison could probably
> comment to that. Can you try deleting the snapshots from the cloudstack UI?
> If that doesnt work, we can work to devise a method where we query the db,
> identify valid snapshots, then using virsh commands delete the failed
> snapshots.
>
>
> On Fri, Feb 1, 2013 at 4:02 PM, Bryan Whitehead <dr...@megahappy.net>wrote:
>
>> For some reason my storageVM was in a state where things "worked" but
>> automated snapshots were not working.
>>
>> Basically the step where libvirt created a stapshot on the primary storage
>> worked - but the step to have the snapshot parent copied to the secondary
>> storage was not working. I got automated snapshots working again by
>> rebooting the storage systemVM.
>>
>> However, when i look at libvirt I can see there are many snapshots on the
>> primary storage. A 12G filesystem has a qcow2 size of 220GB.
>>
>> virsh snapshot-list --parent i-3-14-VM
>> Name Creation Time State Parent
>> ------------------------------------------------------------
>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130125025313 2013-01-25
>> 02:53:13 +0000 running
>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130126025313 2013-01-26
>> 02:53:13 +0000 running
>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130125025313
>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130127025313 2013-01-27
>> 02:53:13 +0000 running
>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130126025313
>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130128025313 2013-01-28
>> 02:53:13 +0000 running
>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130127025313
>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130129025313 2013-01-29
>> 02:53:13 +0000 running
>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130128025313
>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130130025313 2013-01-30
>> 02:53:13 +0000 running
>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130129025313
>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130131025313 2013-01-31
>> 02:53:13 +0000 running
>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130130025313
>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130201025313 2013-02-01
>> 02:53:13 +0000 running
>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130131025313
>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130201223643 2013-02-01
>> 22:36:43 +0000 running
>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130201025313
>>
>> The actual qcow2 file looks like a much bigger wreck:
>>
>> qemu-img info /gluster/qcow2/images/7ec192c1-ecee-4f5e-8e3d-567f7878ceb1
>> image: /gluster/qcow2/images/7ec192c1-ecee-4f5e-8e3d-567f7878ceb1
>> file format: qcow2
>> virtual size: 100G (107374182400 bytes)
>> disk size: 211G
>> cluster_size: 65536
>> backing file: /gluster/qcow2/images/cb151441-209c-4f43-a2a4-b390c9bb9768
>> (actual path: /gluster/qcow2/images/cb151441-209c-4f43-a2a4-b390c9bb9768)
>> Snapshot list:
>> ID TAG VM SIZE DATE VM CLOCK
>> 1 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20121230025415
>> 1.0G 2012-12-30 02:54:16 785:39:01.680
>> 2 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20121231025415
>> 1.0G 2012-12-31 02:54:16 809:35:35.417
>> 3 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130101025415
>> 1.0G 2013-01-01 02:54:16 833:24:20.046
>> 4 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130102025415
>> 1.0G 2013-01-02 02:54:15 857:13:02.766
>> 5 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130103025415
>> 1.0G 2013-01-03 02:54:15 881:01:42.467
>> 6 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130104025415
>> 1.0G 2013-01-04 02:54:15 904:50:29.840
>> 7 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130105025415
>> 1.0G 2013-01-05 02:54:16 928:39:17.670
>> 8 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130106025415
>> 1.0G 2013-01-06 02:54:16 952:28:07.139
>> 9 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130107025415
>> 1.1G 2013-01-07 02:54:16 976:16:58.396
>> 10 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130108025415
>> 1.1G 2013-01-08 02:54:15 1000:05:45.624
>> 11 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130109025415
>> 1.1G 2013-01-09 02:54:15 1023:54:25.482
>> 12 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130110025415
>> 1.1G 2013-01-10 02:54:16 1047:43:17.503
>> 13 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130111025415
>> 1.1G 2013-01-11 02:54:16 1071:32:03.747
>> 14 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130112025415
>> 1.1G 2013-01-12 02:54:16 1095:20:52.220
>> 15 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130113025415
>> 1.1G 2013-01-13 02:54:16 1119:09:43.387
>> 16 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130114025415
>> 1.1G 2013-01-14 02:54:16 1142:58:32.657
>> 17 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130115025415
>> 1.1G 2013-01-15 02:54:16 1166:47:20.084
>> 18 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130116025416
>> 1.1G 2013-01-16 02:54:16 1190:36:06.729
>> 19 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130117025416
>> 1.1G 2013-01-17 02:54:16 1214:24:53.904
>> 20 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130118025415
>> 1.1G 2013-01-18 02:54:16 1238:13:35.640
>> 21 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130119025415
>> 1.1G 2013-01-19 02:54:16 1262:02:17.839
>> 22 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130120025415
>> 1.1G 2013-01-20 02:54:16 1285:51:01.114
>> 23 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130121025415
>> 1.1G 2013-01-21 02:54:16 1309:39:49.956
>> 24 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130122025416
>> 1.1G 2013-01-22 02:54:16 1333:28:39.703
>> 25 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130123025415
>> 1.1G 2013-01-23 02:54:16 1357:17:25.426
>> 26 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130124025416
>> 1.1G 2013-01-24 02:54:16 1381:06:04.731
>> 27 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130125025313
>> 517M 2013-01-25 02:53:13 04:46:52.307
>> 28 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130126025313
>> 678M 2013-01-26 02:53:13 28:44:35.134
>> 29 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130127025313
>> 918M 2013-01-27 02:53:13 52:42:00.249
>> 30 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130128025313
>> 1.1G 2013-01-28 02:53:13 76:38:55.428
>> 31 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130129025313
>> 1.2G 2013-01-29 02:53:13 100:35:27.978
>> 32 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130130025313
>> 1.2G 2013-01-30 02:53:13 124:31:55.179
>> 33 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130131025313
>> 1.3G 2013-01-31 02:53:13 148:28:17.744
>> 34 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130201025313
>> 1.3G 2013-02-01 02:53:13 172:24:33.630
>> 35 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130201223643
>> 1.3G 2013-02-01 22:36:43 192:04:17.498
>>
>> Suggestions on how I can clean up these extra snapshots?
>>
>> -Bryan
>>
>
>
Re: SystemVM offline causes extreme snapshot bloat
Posted by Ahmad Emneina <ae...@gmail.com>.
I think in 4.0 and master this was broken. Marcus or Edison could probably
comment to that. Can you try deleting the snapshots from the cloudstack UI?
If that doesnt work, we can work to devise a method where we query the db,
identify valid snapshots, then using virsh commands delete the failed
snapshots.
On Fri, Feb 1, 2013 at 4:02 PM, Bryan Whitehead <dr...@megahappy.net>wrote:
> For some reason my storageVM was in a state where things "worked" but
> automated snapshots were not working.
>
> Basically the step where libvirt created a stapshot on the primary storage
> worked - but the step to have the snapshot parent copied to the secondary
> storage was not working. I got automated snapshots working again by
> rebooting the storage systemVM.
>
> However, when i look at libvirt I can see there are many snapshots on the
> primary storage. A 12G filesystem has a qcow2 size of 220GB.
>
> virsh snapshot-list --parent i-3-14-VM
> Name Creation Time State Parent
> ------------------------------------------------------------
> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130125025313 2013-01-25
> 02:53:13 +0000 running
> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130126025313 2013-01-26
> 02:53:13 +0000 running
> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130125025313
> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130127025313 2013-01-27
> 02:53:13 +0000 running
> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130126025313
> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130128025313 2013-01-28
> 02:53:13 +0000 running
> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130127025313
> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130129025313 2013-01-29
> 02:53:13 +0000 running
> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130128025313
> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130130025313 2013-01-30
> 02:53:13 +0000 running
> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130129025313
> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130131025313 2013-01-31
> 02:53:13 +0000 running
> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130130025313
> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130201025313 2013-02-01
> 02:53:13 +0000 running
> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130131025313
> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130201223643 2013-02-01
> 22:36:43 +0000 running
> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130201025313
>
> The actual qcow2 file looks like a much bigger wreck:
>
> qemu-img info /gluster/qcow2/images/7ec192c1-ecee-4f5e-8e3d-567f7878ceb1
> image: /gluster/qcow2/images/7ec192c1-ecee-4f5e-8e3d-567f7878ceb1
> file format: qcow2
> virtual size: 100G (107374182400 bytes)
> disk size: 211G
> cluster_size: 65536
> backing file: /gluster/qcow2/images/cb151441-209c-4f43-a2a4-b390c9bb9768
> (actual path: /gluster/qcow2/images/cb151441-209c-4f43-a2a4-b390c9bb9768)
> Snapshot list:
> ID TAG VM SIZE DATE VM CLOCK
> 1 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20121230025415
> 1.0G 2012-12-30 02:54:16 785:39:01.680
> 2 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20121231025415
> 1.0G 2012-12-31 02:54:16 809:35:35.417
> 3 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130101025415
> 1.0G 2013-01-01 02:54:16 833:24:20.046
> 4 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130102025415
> 1.0G 2013-01-02 02:54:15 857:13:02.766
> 5 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130103025415
> 1.0G 2013-01-03 02:54:15 881:01:42.467
> 6 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130104025415
> 1.0G 2013-01-04 02:54:15 904:50:29.840
> 7 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130105025415
> 1.0G 2013-01-05 02:54:16 928:39:17.670
> 8 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130106025415
> 1.0G 2013-01-06 02:54:16 952:28:07.139
> 9 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130107025415
> 1.1G 2013-01-07 02:54:16 976:16:58.396
> 10 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130108025415
> 1.1G 2013-01-08 02:54:15 1000:05:45.624
> 11 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130109025415
> 1.1G 2013-01-09 02:54:15 1023:54:25.482
> 12 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130110025415
> 1.1G 2013-01-10 02:54:16 1047:43:17.503
> 13 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130111025415
> 1.1G 2013-01-11 02:54:16 1071:32:03.747
> 14 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130112025415
> 1.1G 2013-01-12 02:54:16 1095:20:52.220
> 15 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130113025415
> 1.1G 2013-01-13 02:54:16 1119:09:43.387
> 16 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130114025415
> 1.1G 2013-01-14 02:54:16 1142:58:32.657
> 17 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130115025415
> 1.1G 2013-01-15 02:54:16 1166:47:20.084
> 18 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130116025416
> 1.1G 2013-01-16 02:54:16 1190:36:06.729
> 19 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130117025416
> 1.1G 2013-01-17 02:54:16 1214:24:53.904
> 20 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130118025415
> 1.1G 2013-01-18 02:54:16 1238:13:35.640
> 21 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130119025415
> 1.1G 2013-01-19 02:54:16 1262:02:17.839
> 22 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130120025415
> 1.1G 2013-01-20 02:54:16 1285:51:01.114
> 23 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130121025415
> 1.1G 2013-01-21 02:54:16 1309:39:49.956
> 24 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130122025416
> 1.1G 2013-01-22 02:54:16 1333:28:39.703
> 25 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130123025415
> 1.1G 2013-01-23 02:54:16 1357:17:25.426
> 26 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130124025416
> 1.1G 2013-01-24 02:54:16 1381:06:04.731
> 27 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130125025313
> 517M 2013-01-25 02:53:13 04:46:52.307
> 28 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130126025313
> 678M 2013-01-26 02:53:13 28:44:35.134
> 29 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130127025313
> 918M 2013-01-27 02:53:13 52:42:00.249
> 30 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130128025313
> 1.1G 2013-01-28 02:53:13 76:38:55.428
> 31 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130129025313
> 1.2G 2013-01-29 02:53:13 100:35:27.978
> 32 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130130025313
> 1.2G 2013-01-30 02:53:13 124:31:55.179
> 33 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130131025313
> 1.3G 2013-01-31 02:53:13 148:28:17.744
> 34 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130201025313
> 1.3G 2013-02-01 02:53:13 172:24:33.630
> 35 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130201223643
> 1.3G 2013-02-01 22:36:43 192:04:17.498
>
> Suggestions on how I can clean up these extra snapshots?
>
> -Bryan
>