You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by "Sami Rajala (FAPPS)" <sa...@fapps.com> on 2018/09/13 04:56:09 UTC
VM Snapshot not removed from primary SR
Hello
I have had weird issue since I updated CS from 4.9 to 4.10.
CS does not delete/remove temporary snapshot from primary SR and I have
had to removed it by manual time to time, before 30 snapshot limit has
reached.
+ doesn¹t CS not update secondary_storege count on snapshot_count table
It works fine on 4.9, but stop to work on 4.10 and I have wait possibility
to update 4.11.1.
Update to 4.11.1 has done now and looks like this not work still
I removed all snapshot and start over from clean table, snapshot policy
is: DAILY, keep 2, delta = 5
There are now 3 snapshot on primary SR and 3 on secondary SR
The Environment is Xen 7 + CS 4.11.1 + NFS storage
Is there some other parameter I should look and any other work around I
has to do to get it work ?
Any hits?
BR
-sami
VM snapshot Log for last round:
2018-09-13 06:02:15,983 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-83:ctx-cfb1cd95 job-28020) (logid:4c7a1c38) Executing
AsyncJobVO {id:28020, userId: 1, accountId: 7, instanceType: Snapshot,
instanceId: 2656, cmd:
org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd,
cmdInfo:
{"policyid":"17","ctxUserId":"1","volumeid":"238","ctxStartEventId":"1","id
":"2656","ctxAccountId":"7"}, cmdVersion: 0, status: IN_PROGRESS,
processStatus: 0, resultCode: 0, result: null, initMsid: 19873467853209,
completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
2018-09-13 06:02:15,991 DEBUG [c.c.u.AccountManagerImpl]
(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6) (logid:4c7a1c38)
Access to Acct[479c643e-9c84-41fb-9f0a-9bb999893a25-juha] granted to
Acct[479c643e-9c84-41fb-9f0a-9bb999893a25-juha] by DomainChecker
2018-09-13 06:02:16,045 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6) (logid:4c7a1c38)
Sync job-28021 execution on object VmWorkJobQueue.228
2018-09-13 06:02:17,115 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021) (logid:4c7a1c38)
Executing AsyncJobVO {id:28021, userId: 1, accountId: 7, instanceType:
null, instanceId: null, cmd: com.cloud.vm.VmWorkTakeVolumeSnapshot,
cmdInfo:
rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrVGFrZVZvbHVtZVNuYXBzaG90BL5gG4Li1c8CAAZ
aAAthc3luY0JhY2t1cFoACXF1aWVzY2VWbUwADGxvY2F0aW9uVHlwZXQAKUxjb20vY2xvdWQvc3
RvcmFnZS9TbmFwc2hvdCRMb2NhdGlvblR5cGU7TAAIcG9saWN5SWR0ABBMamF2YS9sYW5nL0xvb
mc7TAAKc25hcHNob3RJZHEAfgACTAAIdm9sdW1lSWRxAH4AAnhyABNjb20uY2xvdWQudm0uVm1X
b3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlkTAALaGFuZGxlck5hbWV
0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAAHAAAAAAAAAAEAAAAAAAAA5HQAFFZvbHVtZU
FwaVNlcnZpY2VJbXBsAABwc3IADmphdmEubGFuZy5Mb25nO4vkkMyPI98CAAFKAAV2YWx1ZXhyA
BBqYXZhLmxhbmcuTnVtYmVyhqyVHQuU4IsCAAB4cAAAAAAAAAARc3EAfgAHAAAAAAAACmBzcQB-
AAcAAAAAAAAA7g, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0,
resultCode: 0, result: null, initMsid: 19873467853209, completeMsid: null,
lastUpdated: null, lastPolled: null, created: Thu Sep 13 06:02:16 EEST
2018}
2018-09-13 06:02:17,116 DEBUG [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021) (logid:4c7a1c38)
Run VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot for VM 228, job
origin: 28020
2018-09-13 06:02:17,117 DEBUG [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
(logid:4c7a1c38) Execute VM work job:
com.cloud.vm.VmWorkTakeVolumeSnapshot{"volumeId":238,"policyId":17,"snapsho
tId":2656,"quiesceVm":false,"asyncBackup":false,"userId":1,"accountId":7,"v
mId":228,"handlerName":"VolumeApiServiceImpl"}
2018-09-13 06:02:17,197 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
(logid:4c7a1c38) getCommandHostDelegation: class
org.apache.cloudstack.storage.command.CreateObjectCommand
2018-09-13 06:02:17,197 DEBUG [c.c.h.XenServerGuru]
(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
(logid:4c7a1c38) We are returning the default host to execute commands
because the command is not of Copy type.
2018-09-13 06:02:17,199 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
(logid:4c7a1c38) Seq 128-4630826316843724436: Sending { Cmd , MgmtId:
19873467853209, via: 128(xenserver-15), Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org
.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"969c796d
-072e-4272-adc6-93a492c03ea3","volumeType":"ROOT","dataStore":{"org.apache.
cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"6568d30f-82b6-35a0-bfe5-
be17c1398ed8","id":2,"poolType":"NetworkFilesystem","host":"10.10.1.33","pa
th":"/pool3/primary3","port":2049,"url":"NetworkFilesystem://10.10.1.33/poo
l3/primary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a0-bfe5-be17c1398ed8","
isManaged":false}},"name":"ROOT-228","size":21474836480,"path":"3d2a9716-8f
a1-497e-8753-941117a0ef50","volumeId":238,"vmName":"i-7-228-VM","accountId"
:7,"format":"VHD","provisioningType":"THIN","id":238,"deviceId":0,"hypervis
orType":"XenServer"},"parentSnapshotPath":"c7b2824a-7402-4f66-b2e6-06ed4520
72ad","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"
uuid":"6568d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType":"NetworkFile
system","host":"10.10.1.33","path":"/pool3/primary3","port":2049,"url":"Net
workFilesystem://10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUID=6568d30f
-82b6-35a0-bfe5-be17c1398ed8","isManaged":false}},"vmName":"i-7-228-VM","na
me":"BGI-Maria_ROOT-228_20180913030215","hypervisorType":"XenServer","id":2
656,"quiescevm":false,"parents":["c7b2824a-7402-4f66-b2e6-06ed452072ad"],"p
hysicalSize":0}},"wait":0}}] }
2018-09-13 06:02:17,199 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
(logid:4c7a1c38) Seq 128-4630826316843724436: Executing: { Cmd , MgmtId:
19873467853209, via: 128(xenserver-15), Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org
.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"969c796d
-072e-4272-adc6-93a492c03ea3","volumeType":"ROOT","dataStore":{"org.apache.
cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"6568d30f-82b6-35a0-bfe5-
be17c1398ed8","id":2,"poolType":"NetworkFilesystem","host":"10.10.1.33","pa
th":"/pool3/primary3","port":2049,"url":"NetworkFilesystem://10.10.1.33/poo
l3/primary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a0-bfe5-be17c1398ed8","
isManaged":false}},"name":"ROOT-228","size":21474836480,"path":"3d2a9716-8f
a1-497e-8753-941117a0ef50","volumeId":238,"vmName":"i-7-228-VM","accountId"
:7,"format":"VHD","provisioningType":"THIN","id":238,"deviceId":0,"hypervis
orType":"XenServer"},"parentSnapshotPath":"c7b2824a-7402-4f66-b2e6-06ed4520
72ad","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"
uuid":"6568d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType":"NetworkFile
system","host":"10.10.1.33","path":"/pool3/primary3","port":2049,"url":"Net
workFilesystem://10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUID=6568d30f
-82b6-35a0-bfe5-be17c1398ed8","isManaged":false}},"vmName":"i-7-228-VM","na
me":"BGI-Maria_ROOT-228_20180913030215","hypervisorType":"XenServer","id":2
656,"quiescevm":false,"parents":["c7b2824a-7402-4f66-b2e6-06ed452072ad"],"p
hysicalSize":0}},"wait":0}}] }
2018-09-13 06:02:19,815 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-206:ctx-370f4cd9) (logid:4c7a1c38) Seq
128-4630826316843724436: Response Received:
2018-09-13 06:02:19,815 DEBUG [c.c.a.t.Request]
(DirectAgent-206:ctx-370f4cd9) (logid:4c7a1c38) Seq
128-4630826316843724436: Processing: { Ans: , MgmtId: 19873467853209,
via: 128(xenserver-15), Ver: v1, Flags: 10,
[{"org.apache.cloudstack.storage.command.CreateObjectAnswer":{"data":{"org.
apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"e91fb041-779d-4090-
b14c-4f57f4cda54b","id":0,"quiescevm":false,"physicalSize":0}},"result":tru
e,"wait":0}}] }
2018-09-13 06:02:19,815 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
(logid:4c7a1c38) Seq 128-4630826316843724436: Received: { Ans: , MgmtId:
19873467853209, via: 128(xenserver-15), Ver: v1, Flags: 10, {
CreateObjectAnswer } }
2018-09-13 06:02:19,934 DEBUG [o.a.c.s.m.AncientDataMotionStrategy]
(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
(logid:4c7a1c38) copyAsync inspecting src type SNAPSHOT copyAsync
inspecting dest type SNAPSHOT
2018-09-13 06:02:20,017 ERROR [o.a.c.s.e.DefaultEndPointSelector]
(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
(logid:4c7a1c38) IR24 select BACKUPSNAPSHOT from primary to secondary 2656
dest=2656
2018-09-13 06:02:20,023 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
(logid:4c7a1c38) getCommandHostDelegation: class
org.apache.cloudstack.storage.command.CopyCommand
2018-09-13 06:02:20,023 DEBUG [c.c.h.XenServerGuru]
(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
(logid:4c7a1c38) We are returning the default host to execute commands
because the source and destination objects are not NFS type.
2018-09-13 06:02:20,025 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
(logid:4c7a1c38) Seq 130-7345089517264579763: Sending { Cmd , MgmtId:
19873467853209, via: 130(xenserver-81), Ver: v1, Flags: 100111,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache
.cloudstack.storage.to.SnapshotObjectTO":{"path":"e91fb041-779d-4090-b14c-4
f57f4cda54b","volume":{"uuid":"969c796d-072e-4272-adc6-93a492c03ea3","volum
eType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataSto
reTO":{"uuid":"6568d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType":"Net
workFilesystem","host":"10.10.1.33","path":"/pool3/primary3","port":2049,"u
rl":"NetworkFilesystem://10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUID=
6568d30f-82b6-35a0-bfe5-be17c1398ed8","isManaged":false}},"name":"ROOT-228"
,"size":21474836480,"path":"3d2a9716-8fa1-497e-8753-941117a0ef50","volumeId
":238,"vmName":"i-7-228-VM","accountId":7,"format":"VHD","provisioningType"
:"THIN","id":238,"deviceId":0,"hypervisorType":"XenServer"},"parentSnapshot
Path":"c7b2824a-7402-4f66-b2e6-06ed452072ad","dataStore":{"org.apache.cloud
stack.storage.to.PrimaryDataStoreTO":{"uuid":"6568d30f-82b6-35a0-bfe5-be17c
1398ed8","id":2,"poolType":"NetworkFilesystem","host":"10.10.1.33","path":"
/pool3/primary3","port":2049,"url":"NetworkFilesystem://10.10.1.33/pool3/pr
imary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a0-bfe5-be17c1398ed8","isMan
aged":false}},"vmName":"i-7-228-VM","name":"BGI-Maria_ROOT-228_201809130302
15","hypervisorType":"XenServer","id":2656,"quiescevm":false,"parents":["c7
b2824a-7402-4f66-b2e6-06ed452072ad"],"physicalSize":0}},"destTO":{"org.apac
he.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/7/238","volum
e":{"uuid":"969c796d-072e-4272-adc6-93a492c03ea3","volumeType":"ROOT","data
Store":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"6568
d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType":"NetworkFilesystem","ho
st":"10.10.1.33","path":"/pool3/primary3","port":2049,"url":"NetworkFilesys
tem://10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a0-
bfe5-be17c1398ed8","isManaged":false}},"name":"ROOT-228","size":21474836480
,"path":"3d2a9716-8fa1-497e-8753-941117a0ef50","volumeId":238,"vmName":"i-7
-228-VM","accountId":7,"format":"VHD","provisioningType":"THIN","id":238,"d
eviceId":0,"hypervisorType":"XenServer"},"parentSnapshotPath":"snapshots/7/
238/0c8c6ee3-ce3d-4241-8509-5193d2cff491.vhd","dataStore":{"com.cloud.agent
.api.to.NfsTO":{"_url":"nfs://10.10.1.34/pool8/secondary8","_role":"Image"}
},"vmName":"i-7-228-VM","name":"BGI-Maria_ROOT-228_20180913030215","hypervi
sorType":"XenServer","id":2656,"quiescevm":false,"parents":["snapshots/7/23
8/93a42896-0da1-4b82-8985-d28464f1ac4a.vhd","snapshots/7/238/0c8c6ee3-ce3d-
4241-8509-5193d2cff491.vhd"],"physicalSize":0}},"executeInSequence":true,"o
ptions":{"fullSnapshot":"false"},"options2":{},"wait":28800}}] }
2018-09-13 06:02:20,025 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
(logid:4c7a1c38) Seq 130-7345089517264579763: Executing: { Cmd , MgmtId:
19873467853209, via: 130(xenserver-81), Ver: v1, Flags: 100111,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache
.cloudstack.storage.to.SnapshotObjectTO":{"path":"e91fb041-779d-4090-b14c-4
f57f4cda54b","volume":{"uuid":"969c796d-072e-4272-adc6-93a492c03ea3","volum
eType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataSto
reTO":{"uuid":"6568d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType":"Net
workFilesystem","host":"10.10.1.33","path":"/pool3/primary3","port":2049,"u
rl":"NetworkFilesystem://10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUID=
6568d30f-82b6-35a0-bfe5-be17c1398ed8","isManaged":false}},"name":"ROOT-228"
,"size":21474836480,"path":"3d2a9716-8fa1-497e-8753-941117a0ef50","volumeId
":238,"vmName":"i-7-228-VM","accountId":7,"format":"VHD","provisioningType"
:"THIN","id":238,"deviceId":0,"hypervisorType":"XenServer"},"parentSnapshot
Path":"c7b2824a-7402-4f66-b2e6-06ed452072ad","dataStore":{"org.apache.cloud
stack.storage.to.PrimaryDataStoreTO":{"uuid":"6568d30f-82b6-35a0-bfe5-be17c
1398ed8","id":2,"poolType":"NetworkFilesystem","host":"10.10.1.33","path":"
/pool3/primary3","port":2049,"url":"NetworkFilesystem://10.10.1.33/pool3/pr
imary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a0-bfe5-be17c1398ed8","isMan
aged":false}},"vmName":"i-7-228-VM","name":"BGI-Maria_ROOT-228_201809130302
15","hypervisorType":"XenServer","id":2656,"quiescevm":false,"parents":["c7
b2824a-7402-4f66-b2e6-06ed452072ad"],"physicalSize":0}},"destTO":{"org.apac
he.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/7/238","volum
e":{"uuid":"969c796d-072e-4272-adc6-93a492c03ea3","volumeType":"ROOT","data
Store":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"6568
d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType":"NetworkFilesystem","ho
st":"10.10.1.33","path":"/pool3/primary3","port":2049,"url":"NetworkFilesys
tem://10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a0-
bfe5-be17c1398ed8","isManaged":false}},"name":"ROOT-228","size":21474836480
,"path":"3d2a9716-8fa1-497e-8753-941117a0ef50","volumeId":238,"vmName":"i-7
-228-VM","accountId":7,"format":"VHD","provisioningType":"THIN","id":238,"d
eviceId":0,"hypervisorType":"XenServer"},"parentSnapshotPath":"snapshots/7/
238/0c8c6ee3-ce3d-4241-8509-5193d2cff491.vhd","dataStore":{"com.cloud.agent
.api.to.NfsTO":{"_url":"nfs://10.10.1.34/pool8/secondary8","_role":"Image"}
},"vmName":"i-7-228-VM","name":"BGI-Maria_ROOT-228_20180913030215","hypervi
sorType":"XenServer","id":2656,"quiescevm":false,"parents":["snapshots/7/23
8/93a42896-0da1-4b82-8985-d28464f1ac4a.vhd","snapshots/7/238/0c8c6ee3-ce3d-
4241-8509-5193d2cff491.vhd"],"physicalSize":0}},"executeInSequence":true,"o
ptions":{"fullSnapshot":"false"},"options2":{},"wait":28800}}] }
2018-09-13 06:04:28,895 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Host 10.10.1.81
OpaqueRef:f83331df-6634-63fd-6b95-44607a6069b8: Removing SR
2018-09-13 06:04:28,913 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Host 10.10.1.81
OpaqueRef:05380b03-4cb8-85a7-6a6a-a1595c188f60: Unplugging pbd
2018-09-13 06:04:30,881 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Host 10.10.1.81
OpaqueRef:f83331df-6634-63fd-6b95-44607a6069b8: Forgetting
2018-09-13 06:04:30,891 DEBUG [c.c.h.x.r.XenServerStorageProcessor]
(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Successfully destroyed
snapshot on volume: 3d2a9716-8fa1-497e-8753-941117a0ef50 execept this
current snapshot e91fb041-779d-4090-b14c-4f57f4cda54b
2018-09-13 06:04:30,891 INFO [c.c.h.x.r.XenServerStorageProcessor]
(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) New snapshot details:
SnapshotTO[datastore=null|volume=null|pathsnapshots/7/238/0560d536-c945-4a6
1-8c4c-dbaf29f29993.vhd]
2018-09-13 06:04:30,891 INFO [c.c.h.x.r.XenServerStorageProcessor]
(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) New snapshot physical
utilization: 2342932992
2018-09-13 06:04:30,891 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Seq
130-7345089517264579763: Response Received:
2018-09-13 06:04:30,891 DEBUG [c.c.a.t.Request]
(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Seq
130-7345089517264579763: Processing: { Ans: , MgmtId: 19873467853209,
via: 130(xenserver-81), Ver: v1, Flags: 110,
[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.ap
ache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/7/238/0560d
536-c945-4a61-8c4c-dbaf29f29993.vhd","parentSnapshotPath":"snapshots/7/238/
0c8c6ee3-ce3d-4241-8509-5193d2cff491.vhd","id":0,"quiescevm":false,"physica
lSize":2342932992}},"result":true,"wait":0}}] }
2018-09-13 06:04:30,891 DEBUG [c.c.a.m.AgentAttache]
(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Seq
130-7345089517264579763: No more commands found
2018-09-13 06:04:30,891 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
(logid:4c7a1c38) Seq 130-7345089517264579763: Received: { Ans: , MgmtId:
19873467853209, via: 130(xenserver-81), Ver: v1, Flags: 110, {
CopyCmdAnswer } }
2018-09-13 06:04:30,957 DEBUG [c.c.r.ResourceLimitManagerImpl]
(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
(logid:4c7a1c38) Updating resource Type = secondary_storage count for
Account = 7 Operation = decreasing Amount = 19131903488
2018-09-13 06:04:30,969 DEBUG [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
(logid:4c7a1c38) Done executing VM work job:
com.cloud.vm.VmWorkTakeVolumeSnapshot{"volumeId":238,"policyId":17,"snapsho
tId":2656,"quiesceVm":false,"asyncBackup":false,"userId":1,"accountId":7,"v
mId":228,"handlerName":"VolumeApiServiceImpl"}
2018-09-13 06:04:30,969 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
(logid:4c7a1c38) Complete async job-28021, jobStatus: SUCCEEDED,
resultCode: 0, result:
rO0ABXNyAA5qYXZhLmxhbmcuTG9uZzuL5JDMjyPfAgABSgAFdmFsdWV4cgAQamF2YS5sYW5nLk5
1bWJlcoaslR0LlOCLAgAAeHAAAAAAAAAKYA
2018-09-13 06:04:30,970 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
(logid:4c7a1c38) Publish async job-28021 complete on message bus
2018-09-13 06:04:30,970 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
(logid:4c7a1c38) Wake up jobs related to job-28021
2018-09-13 06:04:30,970 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
(logid:4c7a1c38) Update db status for job-28021
2018-09-13 06:04:30,971 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
(logid:4c7a1c38) Wake up jobs joined with job-28021 and disjoin all
subjobs created from job- 28021
2018-09-13 06:04:30,983 DEBUG [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021) (logid:4c7a1c38)
Done with run of VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot for VM
228, job origin: 28020
2018-09-13 06:04:30,983 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021) (logid:4c7a1c38)
Done executing com.cloud.vm.VmWorkTakeVolumeSnapshot for job-28021
2018-09-13 06:04:30,984 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021) (logid:4c7a1c38)
Remove job-28021 from job monitoring
2018-09-13 06:04:31,026 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6) (logid:4c7a1c38)
Complete async job-28020, jobStatus: SUCCEEDED, resultCode: 0, result:
org.apache.cloudstack.api.response.SnapshotResponse/snapshot/{"id":"3c8662b
2-c76f-4e3a-a928-f76dad820122","account":"juha","domainid":"d3c2420f-7503-4
560-a2da-d7e7149ce390","domain":"BGI","snapshottype":"DAILY","volumeid":"96
9c796d-072e-4272-adc6-93a492c03ea3","volumename":"ROOT-228","volumetype":"R
OOT","created":"2018-09-13T06:02:15+0300","name":"BGI-Maria_ROOT-228_201809
13030215","intervaltype":"DAILY","state":"BackedUp","physicalsize":23429329
92,"zoneid":"533c8f88-2637-4c0e-8210-e0a355b62d82","tags":[],"revertable":f
alse,"ostypeid":"da491b96-3748-11e5-a683-121326fde999","osdisplayname":"Cen
tOS 6.5 (64-bit)","virtualsize":21474836480}
2018-09-13 06:04:31,027 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6) (logid:4c7a1c38)
Publish async job-28020 complete on message bus
2018-09-13 06:04:31,027 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6) (logid:4c7a1c38)
Wake up jobs related to job-28020
2018-09-13 06:04:31,027 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6) (logid:4c7a1c38)
Update db status for job-28020
2018-09-13 06:04:31,028 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6) (logid:4c7a1c38)
Wake up jobs joined with job-28020 and disjoin all subjobs created from
job- 28020
2018-09-13 06:04:31,039 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-83:ctx-cfb1cd95 job-28020) (logid:4c7a1c38) Done
executing
org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for
job-28020
2018-09-13 06:04:31,039 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(API-Job-Executor-83:ctx-cfb1cd95 job-28020) (logid:4c7a1c38) Remove
job-28020 from job monitoring
Re: VM Snapshot not removed from primary SR
Posted by "Sami Rajala (FAPPS)" <sa...@fapps.com>.
Continue,
I created template from VM-s latest snapshot after 3 days ( one parent + 2
child snapshot) - without errors/warning
CS create VM from this template ok, but Xen could not start VM
so, I return back to situation where every snapshot is full and manual
removing snapshot from primary
Has anyone any advice?
BR
-sami
13.9.2018 7.56, "Sami Rajala (FAPPS)" <sa...@fapps.com>:
>Hello
>
>I have had weird issue since I updated CS from 4.9 to 4.10.
>CS does not delete/remove temporary snapshot from primary SR and I have
>had to removed it by manual time to time, before 30 snapshot limit has
>reached.
>+ doesn¹t CS not update secondary_storege count on snapshot_count table
>
>It works fine on 4.9, but stop to work on 4.10 and I have wait possibility
>to update 4.11.1.
>Update to 4.11.1 has done now and looks like this not work still
>
>I removed all snapshot and start over from clean table, snapshot policy
>is: DAILY, keep 2, delta = 5
>There are now 3 snapshot on primary SR and 3 on secondary SR
>
>The Environment is Xen 7 + CS 4.11.1 + NFS storage
>
>Is there some other parameter I should look and any other work around I
>has to do to get it work ?
>
>Any hits?
>
>BR
>-sami
>
>VM snapshot Log for last round:
>
>2018-09-13 06:02:15,983 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>(API-Job-Executor-83:ctx-cfb1cd95 job-28020) (logid:4c7a1c38) Executing
>AsyncJobVO {id:28020, userId: 1, accountId: 7, instanceType: Snapshot,
>instanceId: 2656, cmd:
>org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd,
>cmdInfo:
>{"policyid":"17","ctxUserId":"1","volumeid":"238","ctxStartEventId":"1","i
>d
>":"2656","ctxAccountId":"7"}, cmdVersion: 0, status: IN_PROGRESS,
>processStatus: 0, resultCode: 0, result: null, initMsid: 19873467853209,
>completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
>2018-09-13 06:02:15,991 DEBUG [c.c.u.AccountManagerImpl]
>(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6) (logid:4c7a1c38)
>Access to Acct[479c643e-9c84-41fb-9f0a-9bb999893a25-juha] granted to
>Acct[479c643e-9c84-41fb-9f0a-9bb999893a25-juha] by DomainChecker
>2018-09-13 06:02:16,045 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6) (logid:4c7a1c38)
>Sync job-28021 execution on object VmWorkJobQueue.228
>2018-09-13 06:02:17,115 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021) (logid:4c7a1c38)
>Executing AsyncJobVO {id:28021, userId: 1, accountId: 7, instanceType:
>null, instanceId: null, cmd: com.cloud.vm.VmWorkTakeVolumeSnapshot,
>cmdInfo:
>rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrVGFrZVZvbHVtZVNuYXBzaG90BL5gG4Li1c8CAA
>Z
>aAAthc3luY0JhY2t1cFoACXF1aWVzY2VWbUwADGxvY2F0aW9uVHlwZXQAKUxjb20vY2xvdWQvc
>3
>RvcmFnZS9TbmFwc2hvdCRMb2NhdGlvblR5cGU7TAAIcG9saWN5SWR0ABBMamF2YS9sYW5nL0xv
>b
>mc7TAAKc25hcHNob3RJZHEAfgACTAAIdm9sdW1lSWRxAH4AAnhyABNjb20uY2xvdWQudm0uVm1
>X
>b3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlkTAALaGFuZGxlck5hbW
>V
>0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAAHAAAAAAAAAAEAAAAAAAAA5HQAFFZvbHVtZ
>U
>FwaVNlcnZpY2VJbXBsAABwc3IADmphdmEubGFuZy5Mb25nO4vkkMyPI98CAAFKAAV2YWx1ZXhy
>A
>BBqYXZhLmxhbmcuTnVtYmVyhqyVHQuU4IsCAAB4cAAAAAAAAAARc3EAfgAHAAAAAAAACmBzcQB
>-
>AAcAAAAAAAAA7g, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0,
>resultCode: 0, result: null, initMsid: 19873467853209, completeMsid: null,
>lastUpdated: null, lastPolled: null, created: Thu Sep 13 06:02:16 EEST
>2018}
>2018-09-13 06:02:17,116 DEBUG [c.c.v.VmWorkJobDispatcher]
>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021) (logid:4c7a1c38)
>Run VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot for VM 228, job
>origin: 28020
>2018-09-13 06:02:17,117 DEBUG [c.c.v.VmWorkJobHandlerProxy]
>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>(logid:4c7a1c38) Execute VM work job:
>com.cloud.vm.VmWorkTakeVolumeSnapshot{"volumeId":238,"policyId":17,"snapsh
>o
>tId":2656,"quiesceVm":false,"asyncBackup":false,"userId":1,"accountId":7,"
>v
>mId":228,"handlerName":"VolumeApiServiceImpl"}
>2018-09-13 06:02:17,197 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>(logid:4c7a1c38) getCommandHostDelegation: class
>org.apache.cloudstack.storage.command.CreateObjectCommand
>2018-09-13 06:02:17,197 DEBUG [c.c.h.XenServerGuru]
>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>(logid:4c7a1c38) We are returning the default host to execute commands
>because the command is not of Copy type.
>2018-09-13 06:02:17,199 DEBUG [c.c.a.t.Request]
>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>(logid:4c7a1c38) Seq 128-4630826316843724436: Sending { Cmd , MgmtId:
>19873467853209, via: 128(xenserver-15), Ver: v1, Flags: 100011,
>[{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"or
>g
>.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"969c796
>d
>-072e-4272-adc6-93a492c03ea3","volumeType":"ROOT","dataStore":{"org.apache
>.
>cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"6568d30f-82b6-35a0-bfe5
>-
>be17c1398ed8","id":2,"poolType":"NetworkFilesystem","host":"10.10.1.33","p
>a
>th":"/pool3/primary3","port":2049,"url":"NetworkFilesystem://10.10.1.33/po
>o
>l3/primary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a0-bfe5-be17c1398ed8",
>"
>isManaged":false}},"name":"ROOT-228","size":21474836480,"path":"3d2a9716-8
>f
>a1-497e-8753-941117a0ef50","volumeId":238,"vmName":"i-7-228-VM","accountId
>"
>:7,"format":"VHD","provisioningType":"THIN","id":238,"deviceId":0,"hypervi
>s
>orType":"XenServer"},"parentSnapshotPath":"c7b2824a-7402-4f66-b2e6-06ed452
>0
>72ad","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{
>"
>uuid":"6568d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType":"NetworkFil
>e
>system","host":"10.10.1.33","path":"/pool3/primary3","port":2049,"url":"Ne
>t
>workFilesystem://10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUID=6568d30
>f
>-82b6-35a0-bfe5-be17c1398ed8","isManaged":false}},"vmName":"i-7-228-VM","n
>a
>me":"BGI-Maria_ROOT-228_20180913030215","hypervisorType":"XenServer","id":
>2
>656,"quiescevm":false,"parents":["c7b2824a-7402-4f66-b2e6-06ed452072ad"],"
>p
>hysicalSize":0}},"wait":0}}] }
>2018-09-13 06:02:17,199 DEBUG [c.c.a.t.Request]
>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>(logid:4c7a1c38) Seq 128-4630826316843724436: Executing: { Cmd , MgmtId:
>19873467853209, via: 128(xenserver-15), Ver: v1, Flags: 100011,
>[{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"or
>g
>.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"969c796
>d
>-072e-4272-adc6-93a492c03ea3","volumeType":"ROOT","dataStore":{"org.apache
>.
>cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"6568d30f-82b6-35a0-bfe5
>-
>be17c1398ed8","id":2,"poolType":"NetworkFilesystem","host":"10.10.1.33","p
>a
>th":"/pool3/primary3","port":2049,"url":"NetworkFilesystem://10.10.1.33/po
>o
>l3/primary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a0-bfe5-be17c1398ed8",
>"
>isManaged":false}},"name":"ROOT-228","size":21474836480,"path":"3d2a9716-8
>f
>a1-497e-8753-941117a0ef50","volumeId":238,"vmName":"i-7-228-VM","accountId
>"
>:7,"format":"VHD","provisioningType":"THIN","id":238,"deviceId":0,"hypervi
>s
>orType":"XenServer"},"parentSnapshotPath":"c7b2824a-7402-4f66-b2e6-06ed452
>0
>72ad","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{
>"
>uuid":"6568d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType":"NetworkFil
>e
>system","host":"10.10.1.33","path":"/pool3/primary3","port":2049,"url":"Ne
>t
>workFilesystem://10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUID=6568d30
>f
>-82b6-35a0-bfe5-be17c1398ed8","isManaged":false}},"vmName":"i-7-228-VM","n
>a
>me":"BGI-Maria_ROOT-228_20180913030215","hypervisorType":"XenServer","id":
>2
>656,"quiescevm":false,"parents":["c7b2824a-7402-4f66-b2e6-06ed452072ad"],"
>p
>hysicalSize":0}},"wait":0}}] }
>2018-09-13 06:02:19,815 DEBUG [c.c.a.m.DirectAgentAttache]
>(DirectAgent-206:ctx-370f4cd9) (logid:4c7a1c38) Seq
>128-4630826316843724436: Response Received:
>2018-09-13 06:02:19,815 DEBUG [c.c.a.t.Request]
>(DirectAgent-206:ctx-370f4cd9) (logid:4c7a1c38) Seq
>128-4630826316843724436: Processing: { Ans: , MgmtId: 19873467853209,
>via: 128(xenserver-15), Ver: v1, Flags: 10,
>[{"org.apache.cloudstack.storage.command.CreateObjectAnswer":{"data":{"org
>.
>apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"e91fb041-779d-4090
>-
>b14c-4f57f4cda54b","id":0,"quiescevm":false,"physicalSize":0}},"result":tr
>u
>e,"wait":0}}] }
>2018-09-13 06:02:19,815 DEBUG [c.c.a.t.Request]
>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>(logid:4c7a1c38) Seq 128-4630826316843724436: Received: { Ans: , MgmtId:
>19873467853209, via: 128(xenserver-15), Ver: v1, Flags: 10, {
>CreateObjectAnswer } }
>2018-09-13 06:02:19,934 DEBUG [o.a.c.s.m.AncientDataMotionStrategy]
>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>(logid:4c7a1c38) copyAsync inspecting src type SNAPSHOT copyAsync
>inspecting dest type SNAPSHOT
>2018-09-13 06:02:20,017 ERROR [o.a.c.s.e.DefaultEndPointSelector]
>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>(logid:4c7a1c38) IR24 select BACKUPSNAPSHOT from primary to secondary 2656
>dest=2656
>2018-09-13 06:02:20,023 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>(logid:4c7a1c38) getCommandHostDelegation: class
>org.apache.cloudstack.storage.command.CopyCommand
>2018-09-13 06:02:20,023 DEBUG [c.c.h.XenServerGuru]
>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>(logid:4c7a1c38) We are returning the default host to execute commands
>because the source and destination objects are not NFS type.
>2018-09-13 06:02:20,025 DEBUG [c.c.a.t.Request]
>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>(logid:4c7a1c38) Seq 130-7345089517264579763: Sending { Cmd , MgmtId:
>19873467853209, via: 130(xenserver-81), Ver: v1, Flags: 100111,
>[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apach
>e
>.cloudstack.storage.to.SnapshotObjectTO":{"path":"e91fb041-779d-4090-b14c-
>4
>f57f4cda54b","volume":{"uuid":"969c796d-072e-4272-adc6-93a492c03ea3","volu
>m
>eType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataSt
>o
>reTO":{"uuid":"6568d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType":"Ne
>t
>workFilesystem","host":"10.10.1.33","path":"/pool3/primary3","port":2049,"
>u
>rl":"NetworkFilesystem://10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUID
>=
>6568d30f-82b6-35a0-bfe5-be17c1398ed8","isManaged":false}},"name":"ROOT-228
>"
>,"size":21474836480,"path":"3d2a9716-8fa1-497e-8753-941117a0ef50","volumeI
>d
>":238,"vmName":"i-7-228-VM","accountId":7,"format":"VHD","provisioningType
>"
>:"THIN","id":238,"deviceId":0,"hypervisorType":"XenServer"},"parentSnapsho
>t
>Path":"c7b2824a-7402-4f66-b2e6-06ed452072ad","dataStore":{"org.apache.clou
>d
>stack.storage.to.PrimaryDataStoreTO":{"uuid":"6568d30f-82b6-35a0-bfe5-be17
>c
>1398ed8","id":2,"poolType":"NetworkFilesystem","host":"10.10.1.33","path":
>"
>/pool3/primary3","port":2049,"url":"NetworkFilesystem://10.10.1.33/pool3/p
>r
>imary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a0-bfe5-be17c1398ed8","isMa
>n
>aged":false}},"vmName":"i-7-228-VM","name":"BGI-Maria_ROOT-228_20180913030
>2
>15","hypervisorType":"XenServer","id":2656,"quiescevm":false,"parents":["c
>7
>b2824a-7402-4f66-b2e6-06ed452072ad"],"physicalSize":0}},"destTO":{"org.apa
>c
>he.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/7/238","volu
>m
>e":{"uuid":"969c796d-072e-4272-adc6-93a492c03ea3","volumeType":"ROOT","dat
>a
>Store":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"656
>8
>d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType":"NetworkFilesystem","h
>o
>st":"10.10.1.33","path":"/pool3/primary3","port":2049,"url":"NetworkFilesy
>s
>tem://10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a0
>-
>bfe5-be17c1398ed8","isManaged":false}},"name":"ROOT-228","size":2147483648
>0
>,"path":"3d2a9716-8fa1-497e-8753-941117a0ef50","volumeId":238,"vmName":"i-
>7
>-228-VM","accountId":7,"format":"VHD","provisioningType":"THIN","id":238,"
>d
>eviceId":0,"hypervisorType":"XenServer"},"parentSnapshotPath":"snapshots/7
>/
>238/0c8c6ee3-ce3d-4241-8509-5193d2cff491.vhd","dataStore":{"com.cloud.agen
>t
>.api.to.NfsTO":{"_url":"nfs://10.10.1.34/pool8/secondary8","_role":"Image"
>}
>},"vmName":"i-7-228-VM","name":"BGI-Maria_ROOT-228_20180913030215","hyperv
>i
>sorType":"XenServer","id":2656,"quiescevm":false,"parents":["snapshots/7/2
>3
>8/93a42896-0da1-4b82-8985-d28464f1ac4a.vhd","snapshots/7/238/0c8c6ee3-ce3d
>-
>4241-8509-5193d2cff491.vhd"],"physicalSize":0}},"executeInSequence":true,"
>o
>ptions":{"fullSnapshot":"false"},"options2":{},"wait":28800}}] }
>2018-09-13 06:02:20,025 DEBUG [c.c.a.t.Request]
>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>(logid:4c7a1c38) Seq 130-7345089517264579763: Executing: { Cmd , MgmtId:
>19873467853209, via: 130(xenserver-81), Ver: v1, Flags: 100111,
>[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apach
>e
>.cloudstack.storage.to.SnapshotObjectTO":{"path":"e91fb041-779d-4090-b14c-
>4
>f57f4cda54b","volume":{"uuid":"969c796d-072e-4272-adc6-93a492c03ea3","volu
>m
>eType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataSt
>o
>reTO":{"uuid":"6568d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType":"Ne
>t
>workFilesystem","host":"10.10.1.33","path":"/pool3/primary3","port":2049,"
>u
>rl":"NetworkFilesystem://10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUID
>=
>6568d30f-82b6-35a0-bfe5-be17c1398ed8","isManaged":false}},"name":"ROOT-228
>"
>,"size":21474836480,"path":"3d2a9716-8fa1-497e-8753-941117a0ef50","volumeI
>d
>":238,"vmName":"i-7-228-VM","accountId":7,"format":"VHD","provisioningType
>"
>:"THIN","id":238,"deviceId":0,"hypervisorType":"XenServer"},"parentSnapsho
>t
>Path":"c7b2824a-7402-4f66-b2e6-06ed452072ad","dataStore":{"org.apache.clou
>d
>stack.storage.to.PrimaryDataStoreTO":{"uuid":"6568d30f-82b6-35a0-bfe5-be17
>c
>1398ed8","id":2,"poolType":"NetworkFilesystem","host":"10.10.1.33","path":
>"
>/pool3/primary3","port":2049,"url":"NetworkFilesystem://10.10.1.33/pool3/p
>r
>imary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a0-bfe5-be17c1398ed8","isMa
>n
>aged":false}},"vmName":"i-7-228-VM","name":"BGI-Maria_ROOT-228_20180913030
>2
>15","hypervisorType":"XenServer","id":2656,"quiescevm":false,"parents":["c
>7
>b2824a-7402-4f66-b2e6-06ed452072ad"],"physicalSize":0}},"destTO":{"org.apa
>c
>he.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/7/238","volu
>m
>e":{"uuid":"969c796d-072e-4272-adc6-93a492c03ea3","volumeType":"ROOT","dat
>a
>Store":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"656
>8
>d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType":"NetworkFilesystem","h
>o
>st":"10.10.1.33","path":"/pool3/primary3","port":2049,"url":"NetworkFilesy
>s
>tem://10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a0
>-
>bfe5-be17c1398ed8","isManaged":false}},"name":"ROOT-228","size":2147483648
>0
>,"path":"3d2a9716-8fa1-497e-8753-941117a0ef50","volumeId":238,"vmName":"i-
>7
>-228-VM","accountId":7,"format":"VHD","provisioningType":"THIN","id":238,"
>d
>eviceId":0,"hypervisorType":"XenServer"},"parentSnapshotPath":"snapshots/7
>/
>238/0c8c6ee3-ce3d-4241-8509-5193d2cff491.vhd","dataStore":{"com.cloud.agen
>t
>.api.to.NfsTO":{"_url":"nfs://10.10.1.34/pool8/secondary8","_role":"Image"
>}
>},"vmName":"i-7-228-VM","name":"BGI-Maria_ROOT-228_20180913030215","hyperv
>i
>sorType":"XenServer","id":2656,"quiescevm":false,"parents":["snapshots/7/2
>3
>8/93a42896-0da1-4b82-8985-d28464f1ac4a.vhd","snapshots/7/238/0c8c6ee3-ce3d
>-
>4241-8509-5193d2cff491.vhd"],"physicalSize":0}},"executeInSequence":true,"
>o
>ptions":{"fullSnapshot":"false"},"options2":{},"wait":28800}}] }
>2018-09-13 06:04:28,895 DEBUG [c.c.h.x.r.CitrixResourceBase]
>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Host 10.10.1.81
>OpaqueRef:f83331df-6634-63fd-6b95-44607a6069b8: Removing SR
>2018-09-13 06:04:28,913 DEBUG [c.c.h.x.r.CitrixResourceBase]
>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Host 10.10.1.81
>OpaqueRef:05380b03-4cb8-85a7-6a6a-a1595c188f60: Unplugging pbd
>2018-09-13 06:04:30,881 DEBUG [c.c.h.x.r.CitrixResourceBase]
>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Host 10.10.1.81
>OpaqueRef:f83331df-6634-63fd-6b95-44607a6069b8: Forgetting
>2018-09-13 06:04:30,891 DEBUG [c.c.h.x.r.XenServerStorageProcessor]
>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Successfully destroyed
>snapshot on volume: 3d2a9716-8fa1-497e-8753-941117a0ef50 execept this
>current snapshot e91fb041-779d-4090-b14c-4f57f4cda54b
>2018-09-13 06:04:30,891 INFO [c.c.h.x.r.XenServerStorageProcessor]
>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) New snapshot details:
>SnapshotTO[datastore=null|volume=null|pathsnapshots/7/238/0560d536-c945-4a
>6
>1-8c4c-dbaf29f29993.vhd]
>2018-09-13 06:04:30,891 INFO [c.c.h.x.r.XenServerStorageProcessor]
>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) New snapshot physical
>utilization: 2342932992
>2018-09-13 06:04:30,891 DEBUG [c.c.a.m.DirectAgentAttache]
>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Seq
>130-7345089517264579763: Response Received:
>2018-09-13 06:04:30,891 DEBUG [c.c.a.t.Request]
>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Seq
>130-7345089517264579763: Processing: { Ans: , MgmtId: 19873467853209,
>via: 130(xenserver-81), Ver: v1, Flags: 110,
>[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.a
>p
>ache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/7/238/0560
>d
>536-c945-4a61-8c4c-dbaf29f29993.vhd","parentSnapshotPath":"snapshots/7/238
>/
>0c8c6ee3-ce3d-4241-8509-5193d2cff491.vhd","id":0,"quiescevm":false,"physic
>a
>lSize":2342932992}},"result":true,"wait":0}}] }
>2018-09-13 06:04:30,891 DEBUG [c.c.a.m.AgentAttache]
>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Seq
>130-7345089517264579763: No more commands found
>2018-09-13 06:04:30,891 DEBUG [c.c.a.t.Request]
>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>(logid:4c7a1c38) Seq 130-7345089517264579763: Received: { Ans: , MgmtId:
>19873467853209, via: 130(xenserver-81), Ver: v1, Flags: 110, {
>CopyCmdAnswer } }
>2018-09-13 06:04:30,957 DEBUG [c.c.r.ResourceLimitManagerImpl]
>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>(logid:4c7a1c38) Updating resource Type = secondary_storage count for
>Account = 7 Operation = decreasing Amount = 19131903488
>2018-09-13 06:04:30,969 DEBUG [c.c.v.VmWorkJobHandlerProxy]
>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>(logid:4c7a1c38) Done executing VM work job:
>com.cloud.vm.VmWorkTakeVolumeSnapshot{"volumeId":238,"policyId":17,"snapsh
>o
>tId":2656,"quiesceVm":false,"asyncBackup":false,"userId":1,"accountId":7,"
>v
>mId":228,"handlerName":"VolumeApiServiceImpl"}
>2018-09-13 06:04:30,969 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>(logid:4c7a1c38) Complete async job-28021, jobStatus: SUCCEEDED,
>resultCode: 0, result:
>rO0ABXNyAA5qYXZhLmxhbmcuTG9uZzuL5JDMjyPfAgABSgAFdmFsdWV4cgAQamF2YS5sYW5nLk
>5
>1bWJlcoaslR0LlOCLAgAAeHAAAAAAAAAKYA
>2018-09-13 06:04:30,970 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>(logid:4c7a1c38) Publish async job-28021 complete on message bus
>2018-09-13 06:04:30,970 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>(logid:4c7a1c38) Wake up jobs related to job-28021
>2018-09-13 06:04:30,970 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>(logid:4c7a1c38) Update db status for job-28021
>2018-09-13 06:04:30,971 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>(logid:4c7a1c38) Wake up jobs joined with job-28021 and disjoin all
>subjobs created from job- 28021
>2018-09-13 06:04:30,983 DEBUG [c.c.v.VmWorkJobDispatcher]
>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021) (logid:4c7a1c38)
>Done with run of VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot for VM
>228, job origin: 28020
>2018-09-13 06:04:30,983 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021) (logid:4c7a1c38)
>Done executing com.cloud.vm.VmWorkTakeVolumeSnapshot for job-28021
>2018-09-13 06:04:30,984 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021) (logid:4c7a1c38)
>Remove job-28021 from job monitoring
>2018-09-13 06:04:31,026 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6) (logid:4c7a1c38)
>Complete async job-28020, jobStatus: SUCCEEDED, resultCode: 0, result:
>org.apache.cloudstack.api.response.SnapshotResponse/snapshot/{"id":"3c8662
>b
>2-c76f-4e3a-a928-f76dad820122","account":"juha","domainid":"d3c2420f-7503-
>4
>560-a2da-d7e7149ce390","domain":"BGI","snapshottype":"DAILY","volumeid":"9
>6
>9c796d-072e-4272-adc6-93a492c03ea3","volumename":"ROOT-228","volumetype":"
>R
>OOT","created":"2018-09-13T06:02:15+0300","name":"BGI-Maria_ROOT-228_20180
>9
>13030215","intervaltype":"DAILY","state":"BackedUp","physicalsize":2342932
>9
>92,"zoneid":"533c8f88-2637-4c0e-8210-e0a355b62d82","tags":[],"revertable":
>f
>alse,"ostypeid":"da491b96-3748-11e5-a683-121326fde999","osdisplayname":"Ce
>n
>tOS 6.5 (64-bit)","virtualsize":21474836480}
>2018-09-13 06:04:31,027 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6) (logid:4c7a1c38)
>Publish async job-28020 complete on message bus
>2018-09-13 06:04:31,027 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6) (logid:4c7a1c38)
>Wake up jobs related to job-28020
>2018-09-13 06:04:31,027 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6) (logid:4c7a1c38)
>Update db status for job-28020
>2018-09-13 06:04:31,028 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6) (logid:4c7a1c38)
>Wake up jobs joined with job-28020 and disjoin all subjobs created from
>job- 28020
>2018-09-13 06:04:31,039 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>(API-Job-Executor-83:ctx-cfb1cd95 job-28020) (logid:4c7a1c38) Done
>executing
>org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for
>job-28020
>2018-09-13 06:04:31,039 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>(API-Job-Executor-83:ctx-cfb1cd95 job-28020) (logid:4c7a1c38) Remove
>job-28020 from job monitoring
>
Re: VM Snapshot not removed from primary SR
Posted by "Sami Rajala (FAPPS)" <sa...@fapps.com>.
Hi,
More logs for related issue:
for some reason backup fails with error ”BackupSnapshot Failed due to
Exception in backupsnapshot stage due to The uuid you supplied was invalid”
About the original issue, any advice what to do for getting it to works,
there were 25 snapshot on primary SR for each VM, I removed those by
manual
BR
-sami
2018-10-14 09:01:05,945 WARN [c.c.h.x.r.XenServerStorageProcessor]
(DirectAgent-288:ctx-98565a0a) (logid:3bb6a1d2) BackupSnapshot Failed due
to Exception in backupsnapshot stage due to The uuid you supplied was
invalid.
2018-10-14 09:01:06,563 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-288:ctx-98565a0a) (logid:3bb6a1d2) Seq
147-8476055973688108409: Response Received:
2018-10-14 09:01:06,564 DEBUG [c.c.a.t.Request]
(DirectAgent-288:ctx-98565a0a) (logid:3bb6a1d2) Seq
147-8476055973688108409: Processing: { Ans: , MgmtId: 19873467853209,
via: 147(xenserver-72), Ver: v1, Flags: 110,
[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"de
tails":"BackupSnapshot Failed due to Exception in backupsnapshot stage due
to The uuid you supplied was invalid.","wait":0}}] }
2018-10-14 09:01:06,564 DEBUG [c.c.a.m.AgentAttache]
(DirectAgent-288:ctx-98565a0a) (logid:3bb6a1d2) Seq
147-8476055973688108409: No more commands found
2018-10-14 09:01:06,564 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-71:ctx-93e0db9f job-29275/job-29276 ctx-a8d72f0c)
(logid:3bb6a1d2) Seq 147-8476055973688108409: Received: { Ans: , MgmtId:
19873467853209, via: 147(xenserver-72), Ver: v1, Flags: 110, {
CopyCmdAnswer } }
2018-10-14 09:01:06,593 DEBUG [c.c.s.s.SnapshotManagerImpl]
(Work-Job-Executor-71:ctx-93e0db9f job-29275/job-29276 ctx-a8d72f0c)
(logid:3bb6a1d2) Failed to create snapshotBackupSnapshot Failed due to
Exception in backupsnapshot stage due to The uuid you supplied was invalid.
2018-10-14 09:01:06,594 DEBUG [c.c.r.ResourceLimitManagerImpl]
(Work-Job-Executor-71:ctx-93e0db9f job-29275/job-29276 ctx-a8d72f0c)
(logid:3bb6a1d2) Updating resource Type = snapshot count for Account = 2
Operation = decreasing Amount = 1
2018-10-14 09:01:06,603 DEBUG [c.c.r.ResourceLimitManagerImpl]
(Work-Job-Executor-71:ctx-93e0db9f job-29275/job-29276 ctx-a8d72f0c)
(logid:3bb6a1d2) Updating resource Type = secondary_storage count for
Account = 2 Operation = decreasing Amount = 21474836480
2018-10-14 09:01:06,612 ERROR [o.a.c.s.v.VolumeServiceImpl]
(Work-Job-Executor-71:ctx-93e0db9f job-29275/job-29276 ctx-a8d72f0c)
(logid:3bb6a1d2) Take snapshot: 956 failed
2018-10-14 09:01:06,613 ERROR [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-71:ctx-93e0db9f job-29275/job-29276 ctx-a8d72f0c)
(logid:3bb6a1d2) Invocation exception, caused by:
com.cloud.utils.exception.CloudRuntimeException: BackupSnapshot Failed due
to Exception in backupsnapshot stage due to The uuid you supplied was
invalid.
10.10.2018 21.44, "Sami Rajala (FAPPS)" <sa...@fapps.com>:
>Hi,
>
>
>has job ²snapshot remove from primary² same job-ID or logid as snapshot
>process has?
>
>Here is latest snapshot log for logid:f2cbdadb
>
>2018-10-10 20:00:58,759 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>(API-Job-Executor-91:ctx-383c6cf4 job-29198) (logid:f2cbdadb) Executing
>AsyncJobVO {id:29198, userId: 1, accountId: 10, instanceType: Snapshot,
>instanceId: 180, cmd:
>org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd,
>cmdInfo:
>{"policyid":"53","ctxUserId":"1","volumeid":"924","ctxStartEventId":"1","i
>d
>":"180","ctxAccountId":"10"}, cmdVersion: 0, status: IN_PROGRESS,
>processStatus: 0, resultCode: 0, result: null, initMsid: 19873467853209,
>completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
>2018-10-10 20:00:58,766 DEBUG [c.c.u.AccountManagerImpl]
>(API-Job-Executor-91:ctx-383c6cf4 job-29198 ctx-b559e6db) (logid:f2cbdadb)
>Access to Acct[049bd243-386a-49e5-8215-70594e665951-sami] granted to
>Acct[049bd243-386a-49e5-8215-70594e665951-sami] by DomainChecker
>2018-10-10 20:00:58,822 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>(API-Job-Executor-91:ctx-383c6cf4 job-29198 ctx-b559e6db) (logid:f2cbdadb)
>Sync job-29199 execution on object VmWorkJobQueue.252
>2018-10-10 20:00:59,940 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199) (logid:f2cbdadb)
>Executing AsyncJobVO {id:29199, userId: 1, accountId: 10, instanceType:
>null, instanceId: null, cmd: com.cloud.vm.VmWorkTakeVolumeSnapshot,
>cmdInfo:
>rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrVGFrZVZvbHVtZVNuYXBzaG90BL5gG4Li1c8CAA
>Z
>aAAthc3luY0JhY2t1cFoACXF1aWVzY2VWbUwADGxvY2F0aW9uVHlwZXQAKUxjb20vY2xvdWQvc
>3
>RvcmFnZS9TbmFwc2hvdCRMb2NhdGlvblR5cGU7TAAIcG9saWN5SWR0ABBMamF2YS9sYW5nL0xv
>b
>mc7TAAKc25hcHNob3RJZHEAfgACTAAIdm9sdW1lSWRxAH4AAnhyABNjb20uY2xvdWQudm0uVm1
>X
>b3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlkTAALaGFuZGxlck5hbW
>V
>0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAAKAAAAAAAAAAEAAAAAAAAA_HQAFFZvbHVtZ
>U
>FwaVNlcnZpY2VJbXBsAABwc3IADmphdmEubGFuZy5Mb25nO4vkkMyPI98CAAFKAAV2YWx1ZXhy
>A
>BBqYXZhLmxhbmcuTnVtYmVyhqyVHQuU4IsCAAB4cAAAAAAAAAA1c3EAfgAHAAAAAAAAALRzcQB
>-
>AAcAAAAAAAADnA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0,
>resultCode: 0, result: null, initMsid: 19873467853209, completeMsid: null,
>lastUpdated: null, lastPolled: null, created: Wed Oct 10 20:00:58 EEST
>2018}
>2018-10-10 20:00:59,941 DEBUG [c.c.v.VmWorkJobDispatcher]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199) (logid:f2cbdadb)
>Run VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot for VM 252, job
>origin: 29198
>2018-10-10 20:00:59,942 DEBUG [c.c.v.VmWorkJobHandlerProxy]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
>(logid:f2cbdadb) Execute VM work job:
>com.cloud.vm.VmWorkTakeVolumeSnapshot{"volumeId":924,"policyId":53,"snapsh
>o
>tId":180,"quiesceVm":false,"asyncBackup":false,"userId":1,"accountId":10,"
>v
>mId":252,"handlerName":"VolumeApiServiceImpl"}
>2018-10-10 20:01:00,021 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
>(logid:f2cbdadb) getCommandHostDelegation: class
>org.apache.cloudstack.storage.command.CreateObjectCommand
>2018-10-10 20:01:00,021 DEBUG [c.c.h.XenServerGuru]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
>(logid:f2cbdadb) We are returning the default host to execute commands
>because the command is not of Copy type.
>2018-10-10 20:01:00,023 DEBUG [c.c.a.t.Request]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
>(logid:f2cbdadb) Seq 140-3328441599603612624: Sending { Cmd , MgmtId:
>19873467853209, via: 140(xenserver-69), Ver: v1, Flags: 100011,
>[{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"or
>g
>.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"284eadd
>7
>-c29a-423f-94eb-da4c548ad945","volumeType":"ROOT","dataStore":{"org.apache
>.
>cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8700cdda-f60b-35cc-93b7
>-
>3160058b481a","id":16,"poolType":"NetworkFilesystem","host":"10.10.1.33","
>p
>ath":"/pool6/primary6","port":2049,"url":"NetworkFilesystem://10.10.1.33/p
>o
>ol6/primary6/?ROLE=Primary&STOREUUID=8700cdda-f60b-35cc-93b7-3160058b481a"
>,
>"isManaged":false}},"name":"ROOT-252","size":21474836480,"path":"b81b00cc-
>6
>a68-42a1-94c0-7f4089b79973","volumeId":924,"vmName":"i-4-252-VM","accountI
>d
>":10,"format":"VHD","provisioningType":"THIN","id":924,"deviceId":0,"hyper
>v
>isorType":"XenServer"},"parentSnapshotPath":"41c73e2d-a4e0-4256-9bb7-1ee90
>7
>c877ff","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO"
>:
>{"uuid":"8700cdda-f60b-35cc-93b7-3160058b481a","id":16,"poolType":"Network
>F
>ilesystem","host":"10.10.1.33","path":"/pool6/primary6","port":2049,"url":
>"
>NetworkFilesystem://10.10.1.33/pool6/primary6/?ROLE=Primary&STOREUUID=8700
>c
>dda-f60b-35cc-93b7-3160058b481a","isManaged":false}},"vmName":"i-4-252-VM"
>,
>"name":"VM-server_ROOT-252_20181010170058","hypervisorType":"XenServer","i
>d
>":180,"quiescevm":false,"parents":["41c73e2d-a4e0-4256-9bb7-1ee907c877ff"]
>,
>"physicalSize":0}},"wait":0}}] }
>2018-10-10 20:01:00,023 DEBUG [c.c.a.t.Request]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
>(logid:f2cbdadb) Seq 140-3328441599603612624: Executing: { Cmd , MgmtId:
>19873467853209, via: 140(xenserver-69), Ver: v1, Flags: 100011,
>[{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"or
>g
>.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"284eadd
>7
>-c29a-423f-94eb-da4c548ad945","volumeType":"ROOT","dataStore":{"org.apache
>.
>cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8700cdda-f60b-35cc-93b7
>-
>3160058b481a","id":16,"poolType":"NetworkFilesystem","host":"10.10.1.33","
>p
>ath":"/pool6/primary6","port":2049,"url":"NetworkFilesystem://10.10.1.33/p
>o
>ol6/primary6/?ROLE=Primary&STOREUUID=8700cdda-f60b-35cc-93b7-3160058b481a"
>,
>"isManaged":false}},"name":"ROOT-252","size":21474836480,"path":"b81b00cc-
>6
>a68-42a1-94c0-7f4089b79973","volumeId":924,"vmName":"i-4-252-VM","accountI
>d
>":10,"format":"VHD","provisioningType":"THIN","id":924,"deviceId":0,"hyper
>v
>isorType":"XenServer"},"parentSnapshotPath":"41c73e2d-a4e0-4256-9bb7-1ee90
>7
>c877ff","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO"
>:
>{"uuid":"8700cdda-f60b-35cc-93b7-3160058b481a","id":16,"poolType":"Network
>F
>ilesystem","host":"10.10.1.33","path":"/pool6/primary6","port":2049,"url":
>"
>NetworkFilesystem://10.10.1.33/pool6/primary6/?ROLE=Primary&STOREUUID=8700
>c
>dda-f60b-35cc-93b7-3160058b481a","isManaged":false}},"vmName":"i-4-252-VM"
>,
>"name":"VM-server_ROOT-252_20181010170058","hypervisorType":"XenServer","i
>d
>":180,"quiescevm":false,"parents":["41c73e2d-a4e0-4256-9bb7-1ee907c877ff"]
>,
>"physicalSize":0}},"wait":0}}] }
>2018-10-10 20:01:01,813 DEBUG [c.c.a.m.DirectAgentAttache]
>(DirectAgent-471:ctx-77f75805) (logid:f2cbdadb) Seq
>140-3328441599603612624: Response Received:
>2018-10-10 20:01:01,813 DEBUG [c.c.a.t.Request]
>(DirectAgent-471:ctx-77f75805) (logid:f2cbdadb) Seq
>140-3328441599603612624: Processing: { Ans: , MgmtId: 19873467853209,
>via: 140(xenserver-69), Ver: v1, Flags: 10,
>[{"org.apache.cloudstack.storage.command.CreateObjectAnswer":{"data":{"org
>.
>apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"4ede3269-0d3b-4599
>-
>aa66-0a9a6e523201","id":0,"quiescevm":false,"physicalSize":0}},"result":tr
>u
>e,"wait":0}}] }
>2018-10-10 20:01:01,814 DEBUG [c.c.a.t.Request]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
>(logid:f2cbdadb) Seq 140-3328441599603612624: Received: { Ans: , MgmtId:
>19873467853209, via: 140(xenserver-69), Ver: v1, Flags: 10, {
>CreateObjectAnswer } }
>2018-10-10 20:01:01,941 DEBUG [o.a.c.s.m.AncientDataMotionStrategy]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
>(logid:f2cbdadb) copyAsync inspecting src type SNAPSHOT copyAsync
>inspecting dest type SNAPSHOT
>2018-10-10 20:01:02,008 ERROR [o.a.c.s.e.DefaultEndPointSelector]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
>(logid:f2cbdadb) IR24 select BACKUPSNAPSHOT from primary to secondary 180
>dest=180
>2018-10-10 20:01:02,015 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
>(logid:f2cbdadb) getCommandHostDelegation: class
>org.apache.cloudstack.storage.command.CopyCommand
>2018-10-10 20:01:02,015 DEBUG [c.c.h.XenServerGuru]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
>(logid:f2cbdadb) We are returning the default host to execute commands
>because the source and destination objects are not NFS type.
>2018-10-10 20:01:02,016 DEBUG [c.c.a.t.Request]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
>(logid:f2cbdadb) Seq 139-3695766444211015001: Sending { Cmd , MgmtId:
>19873467853209, via: 139(xenserver-67), Ver: v1, Flags: 100111,
>[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apach
>e
>.cloudstack.storage.to.SnapshotObjectTO":{"path":"4ede3269-0d3b-4599-aa66-
>0
>a9a6e523201","volume":{"uuid":"284eadd7-c29a-423f-94eb-da4c548ad945","volu
>m
>eType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataSt
>o
>reTO":{"uuid":"8700cdda-f60b-35cc-93b7-3160058b481a","id":16,"poolType":"N
>e
>tworkFilesystem","host":"10.10.1.33","path":"/pool6/primary6","port":2049,
>"
>url":"NetworkFilesystem://10.10.1.33/pool6/primary6/?ROLE=Primary&STOREUUI
>D
>=8700cdda-f60b-35cc-93b7-3160058b481a","isManaged":false}},"name":"ROOT-25
>2
>","size":21474836480,"path":"b81b00cc-6a68-42a1-94c0-7f4089b79973","volume
>I
>d":924,"vmName":"i-4-252-VM","accountId":10,"format":"VHD","provisioningTy
>p
>e":"THIN","id":924,"deviceId":0,"hypervisorType":"XenServer"},"parentSnaps
>h
>otPath":"41c73e2d-a4e0-4256-9bb7-1ee907c877ff","dataStore":{"org.apache.cl
>o
>udstack.storage.to.PrimaryDataStoreTO":{"uuid":"8700cdda-f60b-35cc-93b7-31
>6
>0058b481a","id":16,"poolType":"NetworkFilesystem","host":"10.10.1.33","pat
>h
>":"/pool6/primary6","port":2049,"url":"NetworkFilesystem://10.10.1.33/pool
>6
>/primary6/?ROLE=Primary&STOREUUID=8700cdda-f60b-35cc-93b7-3160058b481a","i
>s
>Managed":false}},"vmName":"i-4-252-VM","name":"VM-server_ROOT-252_20181010
>1
>70058","hypervisorType":"XenServer","id":180,"quiescevm":false,"parents":[
>"
>41c73e2d-a4e0-4256-9bb7-1ee907c877ff"],"physicalSize":0}},"destTO":{"org.a
>p
>ache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/10/924","v
>o
>lume":{"uuid":"284eadd7-c29a-423f-94eb-da4c548ad945","volumeType":"ROOT","
>d
>ataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"
>8
>700cdda-f60b-35cc-93b7-3160058b481a","id":16,"poolType":"NetworkFilesystem
>"
>,"host":"10.10.1.33","path":"/pool6/primary6","port":2049,"url":"NetworkFi
>l
>esystem://10.10.1.33/pool6/primary6/?ROLE=Primary&STOREUUID=8700cdda-f60b-
>3
>5cc-93b7-3160058b481a","isManaged":false}},"name":"ROOT-252","size":214748
>3
>6480,"path":"b81b00cc-6a68-42a1-94c0-7f4089b79973","volumeId":924,"vmName"
>:
>"i-4-252-VM","accountId":10,"format":"VHD","provisioningType":"THIN","id":
>9
>24,"deviceId":0,"hypervisorType":"XenServer"},"parentSnapshotPath":"snapsh
>o
>ts/10/924/40fc7082-9035-4762-ad8c-e4f8aa03d69c.vhd","dataStore":{"com.clou
>d
>.agent.api.to.NfsTO":{"_url":"nfs://10.10.1.33/pool4/secondary","_role":"I
>m
>age"}},"vmName":"i-4-252-VM","name":"VM-server_ROOT-252_20181010170058","h
>y
>pervisorType":"XenServer","id":180,"quiescevm":false,"parents":["snapshots
>/
>10/924/40fc7082-9035-4762-ad8c-e4f8aa03d69c.vhd"],"physicalSize":0}},"exec
>u
>teInSequence":true,"options":{"fullSnapshot":"false"},"options2":{},"wait"
>:
>28800}}] }
>2018-10-10 20:01:02,017 DEBUG [c.c.a.t.Request]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
>(logid:f2cbdadb) Seq 139-3695766444211015001: Executing: { Cmd , MgmtId:
>19873467853209, via: 139(xenserver-67), Ver: v1, Flags: 100111,
>[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apach
>e
>.cloudstack.storage.to.SnapshotObjectTO":{"path":"4ede3269-0d3b-4599-aa66-
>0
>a9a6e523201","volume":{"uuid":"284eadd7-c29a-423f-94eb-da4c548ad945","volu
>m
>eType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataSt
>o
>reTO":{"uuid":"8700cdda-f60b-35cc-93b7-3160058b481a","id":16,"poolType":"N
>e
>tworkFilesystem","host":"10.10.1.33","path":"/pool6/primary6","port":2049,
>"
>url":"NetworkFilesystem://10.10.1.33/pool6/primary6/?ROLE=Primary&STOREUUI
>D
>=8700cdda-f60b-35cc-93b7-3160058b481a","isManaged":false}},"name":"ROOT-25
>2
>","size":21474836480,"path":"b81b00cc-6a68-42a1-94c0-7f4089b79973","volume
>I
>d":924,"vmName":"i-4-252-VM","accountId":10,"format":"VHD","provisioningTy
>p
>e":"THIN","id":924,"deviceId":0,"hypervisorType":"XenServer"},"parentSnaps
>h
>otPath":"41c73e2d-a4e0-4256-9bb7-1ee907c877ff","dataStore":{"org.apache.cl
>o
>udstack.storage.to.PrimaryDataStoreTO":{"uuid":"8700cdda-f60b-35cc-93b7-31
>6
>0058b481a","id":16,"poolType":"NetworkFilesystem","host":"10.10.1.33","pat
>h
>":"/pool6/primary6","port":2049,"url":"NetworkFilesystem://10.10.1.33/pool
>6
>/primary6/?ROLE=Primary&STOREUUID=8700cdda-f60b-35cc-93b7-3160058b481a","i
>s
>Managed":false}},"vmName":"i-4-252-VM","name":"VM-server_ROOT-252_20181010
>1
>70058","hypervisorType":"XenServer","id":180,"quiescevm":false,"parents":[
>"
>41c73e2d-a4e0-4256-9bb7-1ee907c877ff"],"physicalSize":0}},"destTO":{"org.a
>p
>ache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/10/924","v
>o
>lume":{"uuid":"284eadd7-c29a-423f-94eb-da4c548ad945","volumeType":"ROOT","
>d
>ataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"
>8
>700cdda-f60b-35cc-93b7-3160058b481a","id":16,"poolType":"NetworkFilesystem
>"
>,"host":"10.10.1.33","path":"/pool6/primary6","port":2049,"url":"NetworkFi
>l
>esystem://10.10.1.33/pool6/primary6/?ROLE=Primary&STOREUUID=8700cdda-f60b-
>3
>5cc-93b7-3160058b481a","isManaged":false}},"name":"ROOT-252","size":214748
>3
>6480,"path":"b81b00cc-6a68-42a1-94c0-7f4089b79973","volumeId":924,"vmName"
>:
>"i-4-252-VM","accountId":10,"format":"VHD","provisioningType":"THIN","id":
>9
>24,"deviceId":0,"hypervisorType":"XenServer"},"parentSnapshotPath":"snapsh
>o
>ts/10/924/40fc7082-9035-4762-ad8c-e4f8aa03d69c.vhd","dataStore":{"com.clou
>d
>.agent.api.to.NfsTO":{"_url":"nfs://10.10.1.33/pool4/secondary","_role":"I
>m
>age"}},"vmName":"i-4-252-VM","name":"VM-server_ROOT-252_20181010170058","h
>y
>pervisorType":"XenServer","id":180,"quiescevm":false,"parents":["snapshots
>/
>10/924/40fc7082-9035-4762-ad8c-e4f8aa03d69c.vhd"],"physicalSize":0}},"exec
>u
>teInSequence":true,"options":{"fullSnapshot":"false"},"options2":{},"wait"
>:
>28800}}] }
>2018-10-10 20:01:36,587 DEBUG [c.c.h.x.r.CitrixResourceBase]
>(DirectAgent-149:ctx-5d298873) (logid:f2cbdadb) Host 10.10.1.67
>OpaqueRef:9bee59f7-483e-a69b-d455-6383b4fa39cf: Removing SR
>2018-10-10 20:01:36,601 DEBUG [c.c.h.x.r.CitrixResourceBase]
>(DirectAgent-149:ctx-5d298873) (logid:f2cbdadb) Host 10.10.1.67
>OpaqueRef:e14a95d1-fd13-7925-a017-aea16ef75fc3: Unplugging pbd
>2018-10-10 20:01:37,163 DEBUG [c.c.h.x.r.CitrixResourceBase]
>(DirectAgent-149:ctx-5d298873) (logid:f2cbdadb) Host 10.10.1.67
>OpaqueRef:9bee59f7-483e-a69b-d455-6383b4fa39cf: Forgetting
>2018-10-10 20:01:37,172 DEBUG [c.c.h.x.r.XenServerStorageProcessor]
>(DirectAgent-149:ctx-5d298873) (logid:f2cbdadb) Successfully destroyed
>snapshot on volume: b81b00cc-6a68-42a1-94c0-7f4089b79973 execept this
>current snapshot 4ede3269-0d3b-4599-aa66-0a9a6e523201
>2018-10-10 20:01:37,172 INFO [c.c.h.x.r.XenServerStorageProcessor]
>(DirectAgent-149:ctx-5d298873) (logid:f2cbdadb) New snapshot details:
>SnapshotTO[datastore=null|volume=null|pathsnapshots/10/924/782c2533-9e47-4
>f
>67-b965-ff40e6959c28.vhd]
>2018-10-10 20:01:37,172 INFO [c.c.h.x.r.XenServerStorageProcessor]
>(DirectAgent-149:ctx-5d298873) (logid:f2cbdadb) New snapshot physical
>utilization: 451809792
>2018-10-10 20:01:37,172 DEBUG [c.c.a.m.DirectAgentAttache]
>(DirectAgent-149:ctx-5d298873) (logid:f2cbdadb) Seq
>139-3695766444211015001: Response Received:
>2018-10-10 20:01:37,172 DEBUG [c.c.a.t.Request]
>(DirectAgent-149:ctx-5d298873) (logid:f2cbdadb) Seq
>139-3695766444211015001: Processing: { Ans: , MgmtId: 19873467853209,
>via: 139(xenserver-67), Ver: v1, Flags: 110,
>[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.a
>p
>ache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/10/924/782
>c
>2533-9e47-4f67-b965-ff40e6959c28.vhd","parentSnapshotPath":"snapshots/10/9
>2
>4/40fc7082-9035-4762-ad8c-e4f8aa03d69c.vhd","id":0,"quiescevm":false,"phys
>i
>calSize":451809792}},"result":true,"wait":0}}] }
>2018-10-10 20:01:37,173 DEBUG [c.c.a.t.Request]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
>(logid:f2cbdadb) Seq 139-3695766444211015001: Received: { Ans: , MgmtId:
>19873467853209, via: 139(xenserver-67), Ver: v1, Flags: 110, {
>CopyCmdAnswer } }
>2018-10-10 20:01:37,174 DEBUG [c.c.a.m.AgentAttache]
>(DirectAgent-149:ctx-5d298873) (logid:f2cbdadb) Seq
>139-3695766444211015001: No more commands found
>2018-10-10 20:01:37,224 DEBUG [c.c.s.s.SnapshotManagerImpl]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
>(logid:f2cbdadb) Max snaps: 2 exceeded for snapshot policy with Id: 53.
>Deleting oldest snapshot: 152
>2018-10-10 20:01:37,228 DEBUG [c.c.u.AccountManagerImpl]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
>(logid:f2cbdadb) Access to
>com.cloud.storage.SnapshotVO$$EnhancerByCGLIB$$a643c04@12127de8 granted to
>Acct[049bd243-386a-49e5-8215-70594e665951-sami] by DomainChecker
>2018-10-10 20:01:37,248 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
>(logid:f2cbdadb) delete snapshot chain for snapshot: 152
>2018-10-10 20:01:37,249 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
>(logid:f2cbdadb) Snapshot: 152 doesn't have children, so it's ok to delete
>it and its parents
>2018-10-10 20:01:37,287 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
>(logid:f2cbdadb) getCommandHostDelegation: class
>org.apache.cloudstack.storage.command.DeleteCommand
>2018-10-10 20:01:37,287 DEBUG [c.c.h.XenServerGuru]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
>(logid:f2cbdadb) We are returning the default host to execute commands
>because the command is not of Copy type.
>2018-10-10 20:01:37,288 DEBUG [c.c.a.t.Request]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
>(logid:f2cbdadb) Seq 161-629659522901785138: Sending { Cmd , MgmtId:
>19873467853209, via: 161(s-712-VM), Ver: v1, Flags: 100011,
>[{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apac
>h
>e.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/10/924/14645f
>6
>a-bcce-42f8-a7a2-d93cceb5f00a.vhd","volume":{"uuid":"284eadd7-c29a-423f-94
>e
>b-da4c548ad945","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.st
>o
>rage.to.PrimaryDataStoreTO":{"uuid":"8700cdda-f60b-35cc-93b7-3160058b481a"
>,
>"id":16,"poolType":"NetworkFilesystem","host":"10.10.1.33","path":"/pool6/
>p
>rimary6","port":2049,"url":"NetworkFilesystem://10.10.1.33/pool6/primary6/
>?
>ROLE=Primary&STOREUUID=8700cdda-f60b-35cc-93b7-3160058b481a","isManaged":f
>a
>lse}},"name":"ROOT-252","size":21474836480,"path":"b81b00cc-6a68-42a1-94c0
>-
>7f4089b79973","volumeId":924,"vmName":"i-4-252-VM","accountId":10,"format"
>:
>"VHD","provisioningType":"THIN","id":924,"deviceId":0,"hypervisorType":"Xe
>n
>Server"},"parentSnapshotPath":"snapshots/10/924/e98b594d-d2ae-4ba9-8507-04
>f
>811b265c8.vhd","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://
>1
>0.10.1.34/pool8/secondary8","_role":"Image"}},"vmName":"i-4-252-VM","name"
>:
>"VM-server_ROOT-252_20181008170058","hypervisorType":"XenServer","id":152,
>"
>quiescevm":false,"parents":["snapshots/10/924/fc51ac67-2bcd-4adf-b1a2-0e9b
>0
>ec1ce1b.vhd","snapshots/10/924/5684cebc-50bb-4837-8996-2490ba4ee441.vhd","
>s
>napshots/10/924/946cc6fb-cc2b-4d8a-888d-b8bcaf787171.vhd","snapshots/10/92
>4
>/e98b594d-d2ae-4ba9-8507-04f811b265c8.vhd"],"physicalSize":0}},"wait":0}}]
>}
>2018-10-10 20:01:37,368 DEBUG [c.c.a.t.Request]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
>(logid:f2cbdadb) Seq 161-629659522901785138: Received: { Ans: , MgmtId:
>19873467853209, via: 161(s-712-VM), Ver: v1, Flags: 10, { Answer } }
>2018-10-10 20:01:37,379 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
>(logid:f2cbdadb) Snapshot: 143 doesn't have children, so it's ok to delete
>it and its parents
>2018-10-10 20:01:37,413 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
>(logid:f2cbdadb) getCommandHostDelegation: class
>org.apache.cloudstack.storage.command.DeleteCommand
>2018-10-10 20:01:37,413 DEBUG [c.c.h.XenServerGuru]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
>(logid:f2cbdadb) We are returning the default host to execute commands
>because the command is not of Copy type.
>2018-10-10 20:01:37,415 DEBUG [c.c.a.t.Request]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
>(logid:f2cbdadb) Seq 161-629659522901785139: Sending { Cmd , MgmtId:
>19873467853209, via: 161(s-712-VM), Ver: v1, Flags: 100011,
>[{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apac
>h
>e.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/10/924/e98b59
>4
>d-d2ae-4ba9-8507-04f811b265c8.vhd","volume":{"uuid":"284eadd7-c29a-423f-94
>e
>b-da4c548ad945","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.st
>o
>rage.to.PrimaryDataStoreTO":{"uuid":"8700cdda-f60b-35cc-93b7-3160058b481a"
>,
>"id":16,"poolType":"NetworkFilesystem","host":"10.10.1.33","path":"/pool6/
>p
>rimary6","port":2049,"url":"NetworkFilesystem://10.10.1.33/pool6/primary6/
>?
>ROLE=Primary&STOREUUID=8700cdda-f60b-35cc-93b7-3160058b481a","isManaged":f
>a
>lse}},"name":"ROOT-252","size":21474836480,"path":"b81b00cc-6a68-42a1-94c0
>-
>7f4089b79973","volumeId":924,"vmName":"i-4-252-VM","accountId":10,"format"
>:
>"VHD","provisioningType":"THIN","id":924,"deviceId":0,"hypervisorType":"Xe
>n
>Server"},"parentSnapshotPath":"snapshots/10/924/946cc6fb-cc2b-4d8a-888d-b8
>b
>caf787171.vhd","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://
>1
>0.10.1.34/pool8/secondary8","_role":"Image"}},"vmName":"i-4-252-VM","name"
>:
>"VM-server_ROOT-252_20181007170057","hypervisorType":"XenServer","id":143,
>"
>quiescevm":false,"parents":["snapshots/10/924/fc51ac67-2bcd-4adf-b1a2-0e9b
>0
>ec1ce1b.vhd","snapshots/10/924/5684cebc-50bb-4837-8996-2490ba4ee441.vhd","
>s
>napshots/10/924/946cc6fb-cc2b-4d8a-888d-b8bcaf787171.vhd"],"physicalSize":
>0
>}},"wait":0}}] }
>2018-10-10 20:01:37,452 DEBUG [c.c.a.t.Request]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
>(logid:f2cbdadb) Seq 161-629659522901785139: Received: { Ans: , MgmtId:
>19873467853209, via: 161(s-712-VM), Ver: v1, Flags: 10, { Answer } }
>2018-10-10 20:01:37,462 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
>(logid:f2cbdadb) Snapshot: 134 doesn't have children, so it's ok to delete
>it and its parents
>2018-10-10 20:01:37,493 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
>(logid:f2cbdadb) getCommandHostDelegation: class
>org.apache.cloudstack.storage.command.DeleteCommand
>2018-10-10 20:01:37,493 DEBUG [c.c.h.XenServerGuru]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
>(logid:f2cbdadb) We are returning the default host to execute commands
>because the command is not of Copy type.
>2018-10-10 20:01:37,495 DEBUG [c.c.a.t.Request]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
>(logid:f2cbdadb) Seq 161-629659522901785140: Sending { Cmd , MgmtId:
>19873467853209, via: 161(s-712-VM), Ver: v1, Flags: 100011,
>[{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apac
>h
>e.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/10/924/946cc6
>f
>b-cc2b-4d8a-888d-b8bcaf787171.vhd","volume":{"uuid":"284eadd7-c29a-423f-94
>e
>b-da4c548ad945","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.st
>o
>rage.to.PrimaryDataStoreTO":{"uuid":"8700cdda-f60b-35cc-93b7-3160058b481a"
>,
>"id":16,"poolType":"NetworkFilesystem","host":"10.10.1.33","path":"/pool6/
>p
>rimary6","port":2049,"url":"NetworkFilesystem://10.10.1.33/pool6/primary6/
>?
>ROLE=Primary&STOREUUID=8700cdda-f60b-35cc-93b7-3160058b481a","isManaged":f
>a
>lse}},"name":"ROOT-252","size":21474836480,"path":"b81b00cc-6a68-42a1-94c0
>-
>7f4089b79973","volumeId":924,"vmName":"i-4-252-VM","accountId":10,"format"
>:
>"VHD","provisioningType":"THIN","id":924,"deviceId":0,"hypervisorType":"Xe
>n
>Server"},"parentSnapshotPath":"snapshots/10/924/5684cebc-50bb-4837-8996-24
>9
>0ba4ee441.vhd","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://
>1
>0.10.1.34/pool8/secondary8","_role":"Image"}},"vmName":"i-4-252-VM","name"
>:
>"VM-server_ROOT-252_20181006170057","hypervisorType":"XenServer","id":134,
>"
>quiescevm":false,"parents":["snapshots/10/924/fc51ac67-2bcd-4adf-b1a2-0e9b
>0
>ec1ce1b.vhd","snapshots/10/924/5684cebc-50bb-4837-8996-2490ba4ee441.vhd"],
>"
>physicalSize":0}},"wait":0}}] }
>2018-10-10 20:01:37,535 DEBUG [c.c.a.t.Request]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
>(logid:f2cbdadb) Seq 161-629659522901785140: Received: { Ans: , MgmtId:
>19873467853209, via: 161(s-712-VM), Ver: v1, Flags: 10, { Answer } }
>2018-10-10 20:01:37,546 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
>(logid:f2cbdadb) Snapshot: 125 doesn't have children, so it's ok to delete
>it and its parents
>2018-10-10 20:01:37,573 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
>(logid:f2cbdadb) getCommandHostDelegation: class
>org.apache.cloudstack.storage.command.DeleteCommand
>2018-10-10 20:01:37,573 DEBUG [c.c.h.XenServerGuru]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
>(logid:f2cbdadb) We are returning the default host to execute commands
>because the command is not of Copy type.
>2018-10-10 20:01:37,574 DEBUG [c.c.a.t.Request]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
>(logid:f2cbdadb) Seq 161-629659522901785141: Sending { Cmd , MgmtId:
>19873467853209, via: 161(s-712-VM), Ver: v1, Flags: 100011,
>[{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apac
>h
>e.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/10/924/5684ce
>b
>c-50bb-4837-8996-2490ba4ee441.vhd","volume":{"uuid":"284eadd7-c29a-423f-94
>e
>b-da4c548ad945","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.st
>o
>rage.to.PrimaryDataStoreTO":{"uuid":"8700cdda-f60b-35cc-93b7-3160058b481a"
>,
>"id":16,"poolType":"NetworkFilesystem","host":"10.10.1.33","path":"/pool6/
>p
>rimary6","port":2049,"url":"NetworkFilesystem://10.10.1.33/pool6/primary6/
>?
>ROLE=Primary&STOREUUID=8700cdda-f60b-35cc-93b7-3160058b481a","isManaged":f
>a
>lse}},"name":"ROOT-252","size":21474836480,"path":"b81b00cc-6a68-42a1-94c0
>-
>7f4089b79973","volumeId":924,"vmName":"i-4-252-VM","accountId":10,"format"
>:
>"VHD","provisioningType":"THIN","id":924,"deviceId":0,"hypervisorType":"Xe
>n
>Server"},"parentSnapshotPath":"snapshots/10/924/fc51ac67-2bcd-4adf-b1a2-0e
>9
>b0ec1ce1b.vhd","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://
>1
>0.10.1.34/pool8/secondary8","_role":"Image"}},"vmName":"i-4-252-VM","name"
>:
>"VM-server_ROOT-252_20181005170057","hypervisorType":"XenServer","id":125,
>"
>quiescevm":false,"parents":["snapshots/10/924/fc51ac67-2bcd-4adf-b1a2-0e9b
>0
>ec1ce1b.vhd"],"physicalSize":0}},"wait":0}}] }
>2018-10-10 20:01:37,667 DEBUG [c.c.a.t.Request]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
>(logid:f2cbdadb) Seq 161-629659522901785141: Received: { Ans: , MgmtId:
>19873467853209, via: 161(s-712-VM), Ver: v1, Flags: 10, { Answer } }
>2018-10-10 20:01:37,678 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
>(logid:f2cbdadb) Snapshot: 116 doesn't have children, so it's ok to delete
>it and its parents
>2018-10-10 20:01:37,700 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
>(logid:f2cbdadb) getCommandHostDelegation: class
>org.apache.cloudstack.storage.command.DeleteCommand
>2018-10-10 20:01:37,700 DEBUG [c.c.h.XenServerGuru]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
>(logid:f2cbdadb) We are returning the default host to execute commands
>because the command is not of Copy type.
>2018-10-10 20:01:37,702 DEBUG [c.c.a.t.Request]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
>(logid:f2cbdadb) Seq 161-629659522901785142: Sending { Cmd , MgmtId:
>19873467853209, via: 161(s-712-VM), Ver: v1, Flags: 100011,
>[{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apac
>h
>e.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/10/924/fc51ac
>6
>7-2bcd-4adf-b1a2-0e9b0ec1ce1b.vhd","volume":{"uuid":"284eadd7-c29a-423f-94
>e
>b-da4c548ad945","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.st
>o
>rage.to.PrimaryDataStoreTO":{"uuid":"8700cdda-f60b-35cc-93b7-3160058b481a"
>,
>"id":16,"poolType":"NetworkFilesystem","host":"10.10.1.33","path":"/pool6/
>p
>rimary6","port":2049,"url":"NetworkFilesystem://10.10.1.33/pool6/primary6/
>?
>ROLE=Primary&STOREUUID=8700cdda-f60b-35cc-93b7-3160058b481a","isManaged":f
>a
>lse}},"name":"ROOT-252","size":21474836480,"path":"b81b00cc-6a68-42a1-94c0
>-
>7f4089b79973","volumeId":924,"vmName":"i-4-252-VM","accountId":10,"format"
>:
>"VHD","provisioningType":"THIN","id":924,"deviceId":0,"hypervisorType":"Xe
>n
>Server"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.10.
>1
>.34/pool8/secondary8","_role":"Image"}},"vmName":"i-4-252-VM","name":"VM-s
>e
>rver_ROOT-252_20181004170057","hypervisorType":"XenServer","id":116,"quies
>c
>evm":false,"physicalSize":0}},"wait":0}}] }
>2018-10-10 20:01:45,260 DEBUG [c.c.a.t.Request]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
>(logid:f2cbdadb) Seq 161-629659522901785142: Received: { Ans: , MgmtId:
>19873467853209, via: 161(s-712-VM), Ver: v1, Flags: 10, { Answer } }
>2018-10-10 20:01:45,288 DEBUG [c.c.r.ResourceLimitManagerImpl]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
>(logid:f2cbdadb) Updating resource Type = snapshot count for Account = 10
>Operation = decreasing Amount = 1
>2018-10-10 20:01:45,300 DEBUG [c.c.r.ResourceLimitManagerImpl]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
>(logid:f2cbdadb) Updating resource Type = secondary_storage count for
>Account = 10 Operation = decreasing Amount = 434999808
>2018-10-10 20:01:45,332 DEBUG [c.c.r.ResourceLimitManagerImpl]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
>(logid:f2cbdadb) Updating resource Type = secondary_storage count for
>Account = 10 Operation = decreasing Amount = 21023026688
>2018-10-10 20:01:45,343 DEBUG [c.c.v.VmWorkJobHandlerProxy]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
>(logid:f2cbdadb) Done executing VM work job:
>com.cloud.vm.VmWorkTakeVolumeSnapshot{"volumeId":924,"policyId":53,"snapsh
>o
>tId":180,"quiesceVm":false,"asyncBackup":false,"userId":1,"accountId":10,"
>v
>mId":252,"handlerName":"VolumeApiServiceImpl"}
>2018-10-10 20:01:45,343 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
>(logid:f2cbdadb) Complete async job-29199, jobStatus: SUCCEEDED,
>resultCode: 0, result:
>rO0ABXNyAA5qYXZhLmxhbmcuTG9uZzuL5JDMjyPfAgABSgAFdmFsdWV4cgAQamF2YS5sYW5nLk
>5
>1bWJlcoaslR0LlOCLAgAAeHAAAAAAAAAAtA
>2018-10-10 20:01:45,344 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
>(logid:f2cbdadb) Publish async job-29199 complete on message bus
>2018-10-10 20:01:45,344 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
>(logid:f2cbdadb) Wake up jobs related to job-29199
>2018-10-10 20:01:45,344 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
>(logid:f2cbdadb) Update db status for job-29199
>2018-10-10 20:01:45,345 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
>(logid:f2cbdadb) Wake up jobs joined with job-29199 and disjoin all
>subjobs created from job- 29199
>2018-10-10 20:01:45,355 DEBUG [c.c.v.VmWorkJobDispatcher]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199) (logid:f2cbdadb)
>Done with run of VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot for VM
>252, job origin: 29198
>2018-10-10 20:01:45,355 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199) (logid:f2cbdadb)
>Done executing com.cloud.vm.VmWorkTakeVolumeSnapshot for job-29199
>2018-10-10 20:01:45,357 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199) (logid:f2cbdadb)
>Remove job-29199 from job monitoring
>2018-10-10 20:01:45,394 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>(API-Job-Executor-91:ctx-383c6cf4 job-29198 ctx-b559e6db)
>(logid:f2cbdadb)
>Complete async job-29198, jobStatus: SUCCEEDED, resultCode: 0, result:
>org.apache.cloudstack.api.response.SnapshotResponse/snapshot/{"id":"b87589
>b
>5-e1f0-4303-9cbb-870b2c06ace5","account":"sami","domainid":"2947e215-4889-
>4
>06d-b0a9-b8d82de1a276","domain":"VM-DOMAIN","snapshottype":"DAILY","volume
>i
>d":"284eadd7-c29a-423f-94eb-da4c548ad945","volumename":"ROOT-252","volumet
>y
>pe":"ROOT","created":"2018-10-10T20:00:58+0300","name":"VM-server_ROOT-252
>_
>20181010170058","intervaltype":"DAILY","state":"BackedUp","physicalsize":4
>5
>1809792,"zoneid":"533c8f88-2637-4c0e-8210-e0a355b62d82","tags":[],"reverta
>b
>le":false,"ostypeid":"da491b96-3748-11e5-a683-121326fde999","osdisplayname
>"
>:"CentOS 6.5 (64-bit)","virtualsize":21474836480}
>2018-10-10 20:01:45,394 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>(API-Job-Executor-91:ctx-383c6cf4 job-29198 ctx-b559e6db)
>(logid:f2cbdadb)
>Publish async job-29198 complete on message bus
>2018-10-10 20:01:45,394 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>(API-Job-Executor-91:ctx-383c6cf4 job-29198 ctx-b559e6db)
>(logid:f2cbdadb)
>Wake up jobs related to job-29198
>2018-10-10 20:01:45,394 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>(API-Job-Executor-91:ctx-383c6cf4 job-29198 ctx-b559e6db)
>(logid:f2cbdadb)
>Update db status for job-29198
>2018-10-10 20:01:45,395 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>(API-Job-Executor-91:ctx-383c6cf4 job-29198 ctx-b559e6db)
>(logid:f2cbdadb)
>Wake up jobs joined with job-29198 and disjoin all subjobs created from
>job- 29198
>2018-10-10 20:01:45,403 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>(API-Job-Executor-91:ctx-383c6cf4 job-29198) (logid:f2cbdadb) Done
>executing
>org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for
>job-29198
>2018-10-10 20:01:45,403 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>(API-Job-Executor-91:ctx-383c6cf4 job-29198) (logid:f2cbdadb) Remove
>job-29198 from job monitoring
>
>
>
>
>
>
>
>
>
>
>10.10.2018 19.18, "Andrei Mikhailovsky" <an...@arhont.com.INVALID>:
>
>>Hi
>>
>>I can confirm that I am also having this issue on 4.11.1.0. To be
>>honest,
>>this issue was always present for me as far as I remember.
>>
>>Kind of an urgent issue to fix to stop people running out of space.
>>
>>Cheers
>>
>>----- Original Message -----
>>> From: "Rafael Weingärtner" <ra...@gmail.com>
>>> To: "users" <us...@cloudstack.apache.org>
>>> Sent: Wednesday, 10 October, 2018 16:24:54
>>> Subject: Re: VM Snapshot not removed from primary SR
>>
>>> Well, I am almost sure I have seen a PR fixing something like you
>>>describe,
>>> I thought that it went into 4.11.1.0.However, only with a more in depth
>>> debugging I would be able to confirm your problem.
>>>
>>> Are you seeing any unexpected exception in your log files?
>>>
>>> On Wed, Oct 10, 2018 at 11:54 AM Sami Rajala (FAPPS)
>>><sa...@fapps.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> CS running on 4.11.1.0 and does not work.
>>>>
>>>> BR
>>>> -sami
>>>>
>>>>
>>>> 10.10.2018 16.49, "Rafael Weingärtner" <ra...@gmail.com>:
>>>>
>>>> >If I am not mistaken, this has already been fixed in 4.11.1.0
>>>> >
>>>> >On Wed, Oct 10, 2018 at 10:31 AM Sami Rajala <sa...@fapps.com>
>>>> >wrote:
>>>> >
>>>> >> Hi,
>>>> >>
>>>> >>
>>>> >> I have got zero advice for this.
>>>> >>
>>>> >> "I have had weird issue since I updated CS from 4.9 to 4.10.
>>>> >> CS does not delete/remove temporary snapshot from primary SR and I
>>>>have
>>>> >> had to removed it by manual time to time, before 30 snapshot limit
>>>>has
>>>> >> reached.
>>>> >> + doesn't CS not update secondary_storege count on snapshot_count
>>>>table²
>>>> >>
>>>> >>
>>>> >>
>>>> >>
>>>> >> Now I have a situation where Xenserver has 19 snapshot on primary
>>>>SR for
>>>> >> each VM + resource count for ²secondary_storege² in database
>>>>increase
>>>> >>all
>>>> >> the time per account.
>>>> >> ACS storage cleaner removed old snapshots from secondary when they
>>>> >>getting
>>>> >> old, but thats all, not decrease resource count for
>>>> ²secondary_storege²
>>>> >> with value of deleted snapshot ( or not remove snapshot from
>>>>primary SR
>>>> >> after snapshot has created and copied to secondary )
>>>> >>
>>>> >> I have still delete primary snapshot by manual + give a new value
>>>>for
>>>> >> ²secondary_storege² before they reach limit.
>>>> >>
>>>> >> has anyone got this kind of issue? any fix
>>>> >>
>>>> >> BR
>>>> >> -sami
>
Re: VM Snapshot not removed from primary SR
Posted by "Sami Rajala (FAPPS)" <sa...@fapps.com>.
Hi,
has job ²snapshot remove from primary² same job-ID or logid as snapshot
process has?
Here is latest snapshot log for logid:f2cbdadb
2018-10-10 20:00:58,759 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-91:ctx-383c6cf4 job-29198) (logid:f2cbdadb) Executing
AsyncJobVO {id:29198, userId: 1, accountId: 10, instanceType: Snapshot,
instanceId: 180, cmd:
org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd,
cmdInfo:
{"policyid":"53","ctxUserId":"1","volumeid":"924","ctxStartEventId":"1","id
":"180","ctxAccountId":"10"}, cmdVersion: 0, status: IN_PROGRESS,
processStatus: 0, resultCode: 0, result: null, initMsid: 19873467853209,
completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
2018-10-10 20:00:58,766 DEBUG [c.c.u.AccountManagerImpl]
(API-Job-Executor-91:ctx-383c6cf4 job-29198 ctx-b559e6db) (logid:f2cbdadb)
Access to Acct[049bd243-386a-49e5-8215-70594e665951-sami] granted to
Acct[049bd243-386a-49e5-8215-70594e665951-sami] by DomainChecker
2018-10-10 20:00:58,822 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-91:ctx-383c6cf4 job-29198 ctx-b559e6db) (logid:f2cbdadb)
Sync job-29199 execution on object VmWorkJobQueue.252
2018-10-10 20:00:59,940 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199) (logid:f2cbdadb)
Executing AsyncJobVO {id:29199, userId: 1, accountId: 10, instanceType:
null, instanceId: null, cmd: com.cloud.vm.VmWorkTakeVolumeSnapshot,
cmdInfo:
rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrVGFrZVZvbHVtZVNuYXBzaG90BL5gG4Li1c8CAAZ
aAAthc3luY0JhY2t1cFoACXF1aWVzY2VWbUwADGxvY2F0aW9uVHlwZXQAKUxjb20vY2xvdWQvc3
RvcmFnZS9TbmFwc2hvdCRMb2NhdGlvblR5cGU7TAAIcG9saWN5SWR0ABBMamF2YS9sYW5nL0xvb
mc7TAAKc25hcHNob3RJZHEAfgACTAAIdm9sdW1lSWRxAH4AAnhyABNjb20uY2xvdWQudm0uVm1X
b3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlkTAALaGFuZGxlck5hbWV
0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAAKAAAAAAAAAAEAAAAAAAAA_HQAFFZvbHVtZU
FwaVNlcnZpY2VJbXBsAABwc3IADmphdmEubGFuZy5Mb25nO4vkkMyPI98CAAFKAAV2YWx1ZXhyA
BBqYXZhLmxhbmcuTnVtYmVyhqyVHQuU4IsCAAB4cAAAAAAAAAA1c3EAfgAHAAAAAAAAALRzcQB-
AAcAAAAAAAADnA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0,
resultCode: 0, result: null, initMsid: 19873467853209, completeMsid: null,
lastUpdated: null, lastPolled: null, created: Wed Oct 10 20:00:58 EEST
2018}
2018-10-10 20:00:59,941 DEBUG [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199) (logid:f2cbdadb)
Run VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot for VM 252, job
origin: 29198
2018-10-10 20:00:59,942 DEBUG [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
(logid:f2cbdadb) Execute VM work job:
com.cloud.vm.VmWorkTakeVolumeSnapshot{"volumeId":924,"policyId":53,"snapsho
tId":180,"quiesceVm":false,"asyncBackup":false,"userId":1,"accountId":10,"v
mId":252,"handlerName":"VolumeApiServiceImpl"}
2018-10-10 20:01:00,021 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
(logid:f2cbdadb) getCommandHostDelegation: class
org.apache.cloudstack.storage.command.CreateObjectCommand
2018-10-10 20:01:00,021 DEBUG [c.c.h.XenServerGuru]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
(logid:f2cbdadb) We are returning the default host to execute commands
because the command is not of Copy type.
2018-10-10 20:01:00,023 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
(logid:f2cbdadb) Seq 140-3328441599603612624: Sending { Cmd , MgmtId:
19873467853209, via: 140(xenserver-69), Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org
.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"284eadd7
-c29a-423f-94eb-da4c548ad945","volumeType":"ROOT","dataStore":{"org.apache.
cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8700cdda-f60b-35cc-93b7-
3160058b481a","id":16,"poolType":"NetworkFilesystem","host":"10.10.1.33","p
ath":"/pool6/primary6","port":2049,"url":"NetworkFilesystem://10.10.1.33/po
ol6/primary6/?ROLE=Primary&STOREUUID=8700cdda-f60b-35cc-93b7-3160058b481a",
"isManaged":false}},"name":"ROOT-252","size":21474836480,"path":"b81b00cc-6
a68-42a1-94c0-7f4089b79973","volumeId":924,"vmName":"i-4-252-VM","accountId
":10,"format":"VHD","provisioningType":"THIN","id":924,"deviceId":0,"hyperv
isorType":"XenServer"},"parentSnapshotPath":"41c73e2d-a4e0-4256-9bb7-1ee907
c877ff","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":
{"uuid":"8700cdda-f60b-35cc-93b7-3160058b481a","id":16,"poolType":"NetworkF
ilesystem","host":"10.10.1.33","path":"/pool6/primary6","port":2049,"url":"
NetworkFilesystem://10.10.1.33/pool6/primary6/?ROLE=Primary&STOREUUID=8700c
dda-f60b-35cc-93b7-3160058b481a","isManaged":false}},"vmName":"i-4-252-VM",
"name":"VM-server_ROOT-252_20181010170058","hypervisorType":"XenServer","id
":180,"quiescevm":false,"parents":["41c73e2d-a4e0-4256-9bb7-1ee907c877ff"],
"physicalSize":0}},"wait":0}}] }
2018-10-10 20:01:00,023 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
(logid:f2cbdadb) Seq 140-3328441599603612624: Executing: { Cmd , MgmtId:
19873467853209, via: 140(xenserver-69), Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org
.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"284eadd7
-c29a-423f-94eb-da4c548ad945","volumeType":"ROOT","dataStore":{"org.apache.
cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8700cdda-f60b-35cc-93b7-
3160058b481a","id":16,"poolType":"NetworkFilesystem","host":"10.10.1.33","p
ath":"/pool6/primary6","port":2049,"url":"NetworkFilesystem://10.10.1.33/po
ol6/primary6/?ROLE=Primary&STOREUUID=8700cdda-f60b-35cc-93b7-3160058b481a",
"isManaged":false}},"name":"ROOT-252","size":21474836480,"path":"b81b00cc-6
a68-42a1-94c0-7f4089b79973","volumeId":924,"vmName":"i-4-252-VM","accountId
":10,"format":"VHD","provisioningType":"THIN","id":924,"deviceId":0,"hyperv
isorType":"XenServer"},"parentSnapshotPath":"41c73e2d-a4e0-4256-9bb7-1ee907
c877ff","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":
{"uuid":"8700cdda-f60b-35cc-93b7-3160058b481a","id":16,"poolType":"NetworkF
ilesystem","host":"10.10.1.33","path":"/pool6/primary6","port":2049,"url":"
NetworkFilesystem://10.10.1.33/pool6/primary6/?ROLE=Primary&STOREUUID=8700c
dda-f60b-35cc-93b7-3160058b481a","isManaged":false}},"vmName":"i-4-252-VM",
"name":"VM-server_ROOT-252_20181010170058","hypervisorType":"XenServer","id
":180,"quiescevm":false,"parents":["41c73e2d-a4e0-4256-9bb7-1ee907c877ff"],
"physicalSize":0}},"wait":0}}] }
2018-10-10 20:01:01,813 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-471:ctx-77f75805) (logid:f2cbdadb) Seq
140-3328441599603612624: Response Received:
2018-10-10 20:01:01,813 DEBUG [c.c.a.t.Request]
(DirectAgent-471:ctx-77f75805) (logid:f2cbdadb) Seq
140-3328441599603612624: Processing: { Ans: , MgmtId: 19873467853209,
via: 140(xenserver-69), Ver: v1, Flags: 10,
[{"org.apache.cloudstack.storage.command.CreateObjectAnswer":{"data":{"org.
apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"4ede3269-0d3b-4599-
aa66-0a9a6e523201","id":0,"quiescevm":false,"physicalSize":0}},"result":tru
e,"wait":0}}] }
2018-10-10 20:01:01,814 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
(logid:f2cbdadb) Seq 140-3328441599603612624: Received: { Ans: , MgmtId:
19873467853209, via: 140(xenserver-69), Ver: v1, Flags: 10, {
CreateObjectAnswer } }
2018-10-10 20:01:01,941 DEBUG [o.a.c.s.m.AncientDataMotionStrategy]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
(logid:f2cbdadb) copyAsync inspecting src type SNAPSHOT copyAsync
inspecting dest type SNAPSHOT
2018-10-10 20:01:02,008 ERROR [o.a.c.s.e.DefaultEndPointSelector]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
(logid:f2cbdadb) IR24 select BACKUPSNAPSHOT from primary to secondary 180
dest=180
2018-10-10 20:01:02,015 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
(logid:f2cbdadb) getCommandHostDelegation: class
org.apache.cloudstack.storage.command.CopyCommand
2018-10-10 20:01:02,015 DEBUG [c.c.h.XenServerGuru]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
(logid:f2cbdadb) We are returning the default host to execute commands
because the source and destination objects are not NFS type.
2018-10-10 20:01:02,016 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
(logid:f2cbdadb) Seq 139-3695766444211015001: Sending { Cmd , MgmtId:
19873467853209, via: 139(xenserver-67), Ver: v1, Flags: 100111,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache
.cloudstack.storage.to.SnapshotObjectTO":{"path":"4ede3269-0d3b-4599-aa66-0
a9a6e523201","volume":{"uuid":"284eadd7-c29a-423f-94eb-da4c548ad945","volum
eType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataSto
reTO":{"uuid":"8700cdda-f60b-35cc-93b7-3160058b481a","id":16,"poolType":"Ne
tworkFilesystem","host":"10.10.1.33","path":"/pool6/primary6","port":2049,"
url":"NetworkFilesystem://10.10.1.33/pool6/primary6/?ROLE=Primary&STOREUUID
=8700cdda-f60b-35cc-93b7-3160058b481a","isManaged":false}},"name":"ROOT-252
","size":21474836480,"path":"b81b00cc-6a68-42a1-94c0-7f4089b79973","volumeI
d":924,"vmName":"i-4-252-VM","accountId":10,"format":"VHD","provisioningTyp
e":"THIN","id":924,"deviceId":0,"hypervisorType":"XenServer"},"parentSnapsh
otPath":"41c73e2d-a4e0-4256-9bb7-1ee907c877ff","dataStore":{"org.apache.clo
udstack.storage.to.PrimaryDataStoreTO":{"uuid":"8700cdda-f60b-35cc-93b7-316
0058b481a","id":16,"poolType":"NetworkFilesystem","host":"10.10.1.33","path
":"/pool6/primary6","port":2049,"url":"NetworkFilesystem://10.10.1.33/pool6
/primary6/?ROLE=Primary&STOREUUID=8700cdda-f60b-35cc-93b7-3160058b481a","is
Managed":false}},"vmName":"i-4-252-VM","name":"VM-server_ROOT-252_201810101
70058","hypervisorType":"XenServer","id":180,"quiescevm":false,"parents":["
41c73e2d-a4e0-4256-9bb7-1ee907c877ff"],"physicalSize":0}},"destTO":{"org.ap
ache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/10/924","vo
lume":{"uuid":"284eadd7-c29a-423f-94eb-da4c548ad945","volumeType":"ROOT","d
ataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8
700cdda-f60b-35cc-93b7-3160058b481a","id":16,"poolType":"NetworkFilesystem"
,"host":"10.10.1.33","path":"/pool6/primary6","port":2049,"url":"NetworkFil
esystem://10.10.1.33/pool6/primary6/?ROLE=Primary&STOREUUID=8700cdda-f60b-3
5cc-93b7-3160058b481a","isManaged":false}},"name":"ROOT-252","size":2147483
6480,"path":"b81b00cc-6a68-42a1-94c0-7f4089b79973","volumeId":924,"vmName":
"i-4-252-VM","accountId":10,"format":"VHD","provisioningType":"THIN","id":9
24,"deviceId":0,"hypervisorType":"XenServer"},"parentSnapshotPath":"snapsho
ts/10/924/40fc7082-9035-4762-ad8c-e4f8aa03d69c.vhd","dataStore":{"com.cloud
.agent.api.to.NfsTO":{"_url":"nfs://10.10.1.33/pool4/secondary","_role":"Im
age"}},"vmName":"i-4-252-VM","name":"VM-server_ROOT-252_20181010170058","hy
pervisorType":"XenServer","id":180,"quiescevm":false,"parents":["snapshots/
10/924/40fc7082-9035-4762-ad8c-e4f8aa03d69c.vhd"],"physicalSize":0}},"execu
teInSequence":true,"options":{"fullSnapshot":"false"},"options2":{},"wait":
28800}}] }
2018-10-10 20:01:02,017 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
(logid:f2cbdadb) Seq 139-3695766444211015001: Executing: { Cmd , MgmtId:
19873467853209, via: 139(xenserver-67), Ver: v1, Flags: 100111,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache
.cloudstack.storage.to.SnapshotObjectTO":{"path":"4ede3269-0d3b-4599-aa66-0
a9a6e523201","volume":{"uuid":"284eadd7-c29a-423f-94eb-da4c548ad945","volum
eType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataSto
reTO":{"uuid":"8700cdda-f60b-35cc-93b7-3160058b481a","id":16,"poolType":"Ne
tworkFilesystem","host":"10.10.1.33","path":"/pool6/primary6","port":2049,"
url":"NetworkFilesystem://10.10.1.33/pool6/primary6/?ROLE=Primary&STOREUUID
=8700cdda-f60b-35cc-93b7-3160058b481a","isManaged":false}},"name":"ROOT-252
","size":21474836480,"path":"b81b00cc-6a68-42a1-94c0-7f4089b79973","volumeI
d":924,"vmName":"i-4-252-VM","accountId":10,"format":"VHD","provisioningTyp
e":"THIN","id":924,"deviceId":0,"hypervisorType":"XenServer"},"parentSnapsh
otPath":"41c73e2d-a4e0-4256-9bb7-1ee907c877ff","dataStore":{"org.apache.clo
udstack.storage.to.PrimaryDataStoreTO":{"uuid":"8700cdda-f60b-35cc-93b7-316
0058b481a","id":16,"poolType":"NetworkFilesystem","host":"10.10.1.33","path
":"/pool6/primary6","port":2049,"url":"NetworkFilesystem://10.10.1.33/pool6
/primary6/?ROLE=Primary&STOREUUID=8700cdda-f60b-35cc-93b7-3160058b481a","is
Managed":false}},"vmName":"i-4-252-VM","name":"VM-server_ROOT-252_201810101
70058","hypervisorType":"XenServer","id":180,"quiescevm":false,"parents":["
41c73e2d-a4e0-4256-9bb7-1ee907c877ff"],"physicalSize":0}},"destTO":{"org.ap
ache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/10/924","vo
lume":{"uuid":"284eadd7-c29a-423f-94eb-da4c548ad945","volumeType":"ROOT","d
ataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8
700cdda-f60b-35cc-93b7-3160058b481a","id":16,"poolType":"NetworkFilesystem"
,"host":"10.10.1.33","path":"/pool6/primary6","port":2049,"url":"NetworkFil
esystem://10.10.1.33/pool6/primary6/?ROLE=Primary&STOREUUID=8700cdda-f60b-3
5cc-93b7-3160058b481a","isManaged":false}},"name":"ROOT-252","size":2147483
6480,"path":"b81b00cc-6a68-42a1-94c0-7f4089b79973","volumeId":924,"vmName":
"i-4-252-VM","accountId":10,"format":"VHD","provisioningType":"THIN","id":9
24,"deviceId":0,"hypervisorType":"XenServer"},"parentSnapshotPath":"snapsho
ts/10/924/40fc7082-9035-4762-ad8c-e4f8aa03d69c.vhd","dataStore":{"com.cloud
.agent.api.to.NfsTO":{"_url":"nfs://10.10.1.33/pool4/secondary","_role":"Im
age"}},"vmName":"i-4-252-VM","name":"VM-server_ROOT-252_20181010170058","hy
pervisorType":"XenServer","id":180,"quiescevm":false,"parents":["snapshots/
10/924/40fc7082-9035-4762-ad8c-e4f8aa03d69c.vhd"],"physicalSize":0}},"execu
teInSequence":true,"options":{"fullSnapshot":"false"},"options2":{},"wait":
28800}}] }
2018-10-10 20:01:36,587 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-149:ctx-5d298873) (logid:f2cbdadb) Host 10.10.1.67
OpaqueRef:9bee59f7-483e-a69b-d455-6383b4fa39cf: Removing SR
2018-10-10 20:01:36,601 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-149:ctx-5d298873) (logid:f2cbdadb) Host 10.10.1.67
OpaqueRef:e14a95d1-fd13-7925-a017-aea16ef75fc3: Unplugging pbd
2018-10-10 20:01:37,163 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-149:ctx-5d298873) (logid:f2cbdadb) Host 10.10.1.67
OpaqueRef:9bee59f7-483e-a69b-d455-6383b4fa39cf: Forgetting
2018-10-10 20:01:37,172 DEBUG [c.c.h.x.r.XenServerStorageProcessor]
(DirectAgent-149:ctx-5d298873) (logid:f2cbdadb) Successfully destroyed
snapshot on volume: b81b00cc-6a68-42a1-94c0-7f4089b79973 execept this
current snapshot 4ede3269-0d3b-4599-aa66-0a9a6e523201
2018-10-10 20:01:37,172 INFO [c.c.h.x.r.XenServerStorageProcessor]
(DirectAgent-149:ctx-5d298873) (logid:f2cbdadb) New snapshot details:
SnapshotTO[datastore=null|volume=null|pathsnapshots/10/924/782c2533-9e47-4f
67-b965-ff40e6959c28.vhd]
2018-10-10 20:01:37,172 INFO [c.c.h.x.r.XenServerStorageProcessor]
(DirectAgent-149:ctx-5d298873) (logid:f2cbdadb) New snapshot physical
utilization: 451809792
2018-10-10 20:01:37,172 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-149:ctx-5d298873) (logid:f2cbdadb) Seq
139-3695766444211015001: Response Received:
2018-10-10 20:01:37,172 DEBUG [c.c.a.t.Request]
(DirectAgent-149:ctx-5d298873) (logid:f2cbdadb) Seq
139-3695766444211015001: Processing: { Ans: , MgmtId: 19873467853209,
via: 139(xenserver-67), Ver: v1, Flags: 110,
[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.ap
ache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/10/924/782c
2533-9e47-4f67-b965-ff40e6959c28.vhd","parentSnapshotPath":"snapshots/10/92
4/40fc7082-9035-4762-ad8c-e4f8aa03d69c.vhd","id":0,"quiescevm":false,"physi
calSize":451809792}},"result":true,"wait":0}}] }
2018-10-10 20:01:37,173 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
(logid:f2cbdadb) Seq 139-3695766444211015001: Received: { Ans: , MgmtId:
19873467853209, via: 139(xenserver-67), Ver: v1, Flags: 110, {
CopyCmdAnswer } }
2018-10-10 20:01:37,174 DEBUG [c.c.a.m.AgentAttache]
(DirectAgent-149:ctx-5d298873) (logid:f2cbdadb) Seq
139-3695766444211015001: No more commands found
2018-10-10 20:01:37,224 DEBUG [c.c.s.s.SnapshotManagerImpl]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
(logid:f2cbdadb) Max snaps: 2 exceeded for snapshot policy with Id: 53.
Deleting oldest snapshot: 152
2018-10-10 20:01:37,228 DEBUG [c.c.u.AccountManagerImpl]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
(logid:f2cbdadb) Access to
com.cloud.storage.SnapshotVO$$EnhancerByCGLIB$$a643c04@12127de8 granted to
Acct[049bd243-386a-49e5-8215-70594e665951-sami] by DomainChecker
2018-10-10 20:01:37,248 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
(logid:f2cbdadb) delete snapshot chain for snapshot: 152
2018-10-10 20:01:37,249 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
(logid:f2cbdadb) Snapshot: 152 doesn't have children, so it's ok to delete
it and its parents
2018-10-10 20:01:37,287 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
(logid:f2cbdadb) getCommandHostDelegation: class
org.apache.cloudstack.storage.command.DeleteCommand
2018-10-10 20:01:37,287 DEBUG [c.c.h.XenServerGuru]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
(logid:f2cbdadb) We are returning the default host to execute commands
because the command is not of Copy type.
2018-10-10 20:01:37,288 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
(logid:f2cbdadb) Seq 161-629659522901785138: Sending { Cmd , MgmtId:
19873467853209, via: 161(s-712-VM), Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apach
e.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/10/924/14645f6
a-bcce-42f8-a7a2-d93cceb5f00a.vhd","volume":{"uuid":"284eadd7-c29a-423f-94e
b-da4c548ad945","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.sto
rage.to.PrimaryDataStoreTO":{"uuid":"8700cdda-f60b-35cc-93b7-3160058b481a",
"id":16,"poolType":"NetworkFilesystem","host":"10.10.1.33","path":"/pool6/p
rimary6","port":2049,"url":"NetworkFilesystem://10.10.1.33/pool6/primary6/?
ROLE=Primary&STOREUUID=8700cdda-f60b-35cc-93b7-3160058b481a","isManaged":fa
lse}},"name":"ROOT-252","size":21474836480,"path":"b81b00cc-6a68-42a1-94c0-
7f4089b79973","volumeId":924,"vmName":"i-4-252-VM","accountId":10,"format":
"VHD","provisioningType":"THIN","id":924,"deviceId":0,"hypervisorType":"Xen
Server"},"parentSnapshotPath":"snapshots/10/924/e98b594d-d2ae-4ba9-8507-04f
811b265c8.vhd","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://1
0.10.1.34/pool8/secondary8","_role":"Image"}},"vmName":"i-4-252-VM","name":
"VM-server_ROOT-252_20181008170058","hypervisorType":"XenServer","id":152,"
quiescevm":false,"parents":["snapshots/10/924/fc51ac67-2bcd-4adf-b1a2-0e9b0
ec1ce1b.vhd","snapshots/10/924/5684cebc-50bb-4837-8996-2490ba4ee441.vhd","s
napshots/10/924/946cc6fb-cc2b-4d8a-888d-b8bcaf787171.vhd","snapshots/10/924
/e98b594d-d2ae-4ba9-8507-04f811b265c8.vhd"],"physicalSize":0}},"wait":0}}]
}
2018-10-10 20:01:37,368 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
(logid:f2cbdadb) Seq 161-629659522901785138: Received: { Ans: , MgmtId:
19873467853209, via: 161(s-712-VM), Ver: v1, Flags: 10, { Answer } }
2018-10-10 20:01:37,379 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
(logid:f2cbdadb) Snapshot: 143 doesn't have children, so it's ok to delete
it and its parents
2018-10-10 20:01:37,413 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
(logid:f2cbdadb) getCommandHostDelegation: class
org.apache.cloudstack.storage.command.DeleteCommand
2018-10-10 20:01:37,413 DEBUG [c.c.h.XenServerGuru]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
(logid:f2cbdadb) We are returning the default host to execute commands
because the command is not of Copy type.
2018-10-10 20:01:37,415 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
(logid:f2cbdadb) Seq 161-629659522901785139: Sending { Cmd , MgmtId:
19873467853209, via: 161(s-712-VM), Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apach
e.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/10/924/e98b594
d-d2ae-4ba9-8507-04f811b265c8.vhd","volume":{"uuid":"284eadd7-c29a-423f-94e
b-da4c548ad945","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.sto
rage.to.PrimaryDataStoreTO":{"uuid":"8700cdda-f60b-35cc-93b7-3160058b481a",
"id":16,"poolType":"NetworkFilesystem","host":"10.10.1.33","path":"/pool6/p
rimary6","port":2049,"url":"NetworkFilesystem://10.10.1.33/pool6/primary6/?
ROLE=Primary&STOREUUID=8700cdda-f60b-35cc-93b7-3160058b481a","isManaged":fa
lse}},"name":"ROOT-252","size":21474836480,"path":"b81b00cc-6a68-42a1-94c0-
7f4089b79973","volumeId":924,"vmName":"i-4-252-VM","accountId":10,"format":
"VHD","provisioningType":"THIN","id":924,"deviceId":0,"hypervisorType":"Xen
Server"},"parentSnapshotPath":"snapshots/10/924/946cc6fb-cc2b-4d8a-888d-b8b
caf787171.vhd","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://1
0.10.1.34/pool8/secondary8","_role":"Image"}},"vmName":"i-4-252-VM","name":
"VM-server_ROOT-252_20181007170057","hypervisorType":"XenServer","id":143,"
quiescevm":false,"parents":["snapshots/10/924/fc51ac67-2bcd-4adf-b1a2-0e9b0
ec1ce1b.vhd","snapshots/10/924/5684cebc-50bb-4837-8996-2490ba4ee441.vhd","s
napshots/10/924/946cc6fb-cc2b-4d8a-888d-b8bcaf787171.vhd"],"physicalSize":0
}},"wait":0}}] }
2018-10-10 20:01:37,452 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
(logid:f2cbdadb) Seq 161-629659522901785139: Received: { Ans: , MgmtId:
19873467853209, via: 161(s-712-VM), Ver: v1, Flags: 10, { Answer } }
2018-10-10 20:01:37,462 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
(logid:f2cbdadb) Snapshot: 134 doesn't have children, so it's ok to delete
it and its parents
2018-10-10 20:01:37,493 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
(logid:f2cbdadb) getCommandHostDelegation: class
org.apache.cloudstack.storage.command.DeleteCommand
2018-10-10 20:01:37,493 DEBUG [c.c.h.XenServerGuru]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
(logid:f2cbdadb) We are returning the default host to execute commands
because the command is not of Copy type.
2018-10-10 20:01:37,495 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
(logid:f2cbdadb) Seq 161-629659522901785140: Sending { Cmd , MgmtId:
19873467853209, via: 161(s-712-VM), Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apach
e.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/10/924/946cc6f
b-cc2b-4d8a-888d-b8bcaf787171.vhd","volume":{"uuid":"284eadd7-c29a-423f-94e
b-da4c548ad945","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.sto
rage.to.PrimaryDataStoreTO":{"uuid":"8700cdda-f60b-35cc-93b7-3160058b481a",
"id":16,"poolType":"NetworkFilesystem","host":"10.10.1.33","path":"/pool6/p
rimary6","port":2049,"url":"NetworkFilesystem://10.10.1.33/pool6/primary6/?
ROLE=Primary&STOREUUID=8700cdda-f60b-35cc-93b7-3160058b481a","isManaged":fa
lse}},"name":"ROOT-252","size":21474836480,"path":"b81b00cc-6a68-42a1-94c0-
7f4089b79973","volumeId":924,"vmName":"i-4-252-VM","accountId":10,"format":
"VHD","provisioningType":"THIN","id":924,"deviceId":0,"hypervisorType":"Xen
Server"},"parentSnapshotPath":"snapshots/10/924/5684cebc-50bb-4837-8996-249
0ba4ee441.vhd","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://1
0.10.1.34/pool8/secondary8","_role":"Image"}},"vmName":"i-4-252-VM","name":
"VM-server_ROOT-252_20181006170057","hypervisorType":"XenServer","id":134,"
quiescevm":false,"parents":["snapshots/10/924/fc51ac67-2bcd-4adf-b1a2-0e9b0
ec1ce1b.vhd","snapshots/10/924/5684cebc-50bb-4837-8996-2490ba4ee441.vhd"],"
physicalSize":0}},"wait":0}}] }
2018-10-10 20:01:37,535 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
(logid:f2cbdadb) Seq 161-629659522901785140: Received: { Ans: , MgmtId:
19873467853209, via: 161(s-712-VM), Ver: v1, Flags: 10, { Answer } }
2018-10-10 20:01:37,546 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
(logid:f2cbdadb) Snapshot: 125 doesn't have children, so it's ok to delete
it and its parents
2018-10-10 20:01:37,573 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
(logid:f2cbdadb) getCommandHostDelegation: class
org.apache.cloudstack.storage.command.DeleteCommand
2018-10-10 20:01:37,573 DEBUG [c.c.h.XenServerGuru]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
(logid:f2cbdadb) We are returning the default host to execute commands
because the command is not of Copy type.
2018-10-10 20:01:37,574 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
(logid:f2cbdadb) Seq 161-629659522901785141: Sending { Cmd , MgmtId:
19873467853209, via: 161(s-712-VM), Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apach
e.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/10/924/5684ceb
c-50bb-4837-8996-2490ba4ee441.vhd","volume":{"uuid":"284eadd7-c29a-423f-94e
b-da4c548ad945","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.sto
rage.to.PrimaryDataStoreTO":{"uuid":"8700cdda-f60b-35cc-93b7-3160058b481a",
"id":16,"poolType":"NetworkFilesystem","host":"10.10.1.33","path":"/pool6/p
rimary6","port":2049,"url":"NetworkFilesystem://10.10.1.33/pool6/primary6/?
ROLE=Primary&STOREUUID=8700cdda-f60b-35cc-93b7-3160058b481a","isManaged":fa
lse}},"name":"ROOT-252","size":21474836480,"path":"b81b00cc-6a68-42a1-94c0-
7f4089b79973","volumeId":924,"vmName":"i-4-252-VM","accountId":10,"format":
"VHD","provisioningType":"THIN","id":924,"deviceId":0,"hypervisorType":"Xen
Server"},"parentSnapshotPath":"snapshots/10/924/fc51ac67-2bcd-4adf-b1a2-0e9
b0ec1ce1b.vhd","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://1
0.10.1.34/pool8/secondary8","_role":"Image"}},"vmName":"i-4-252-VM","name":
"VM-server_ROOT-252_20181005170057","hypervisorType":"XenServer","id":125,"
quiescevm":false,"parents":["snapshots/10/924/fc51ac67-2bcd-4adf-b1a2-0e9b0
ec1ce1b.vhd"],"physicalSize":0}},"wait":0}}] }
2018-10-10 20:01:37,667 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
(logid:f2cbdadb) Seq 161-629659522901785141: Received: { Ans: , MgmtId:
19873467853209, via: 161(s-712-VM), Ver: v1, Flags: 10, { Answer } }
2018-10-10 20:01:37,678 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
(logid:f2cbdadb) Snapshot: 116 doesn't have children, so it's ok to delete
it and its parents
2018-10-10 20:01:37,700 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
(logid:f2cbdadb) getCommandHostDelegation: class
org.apache.cloudstack.storage.command.DeleteCommand
2018-10-10 20:01:37,700 DEBUG [c.c.h.XenServerGuru]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
(logid:f2cbdadb) We are returning the default host to execute commands
because the command is not of Copy type.
2018-10-10 20:01:37,702 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
(logid:f2cbdadb) Seq 161-629659522901785142: Sending { Cmd , MgmtId:
19873467853209, via: 161(s-712-VM), Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apach
e.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/10/924/fc51ac6
7-2bcd-4adf-b1a2-0e9b0ec1ce1b.vhd","volume":{"uuid":"284eadd7-c29a-423f-94e
b-da4c548ad945","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.sto
rage.to.PrimaryDataStoreTO":{"uuid":"8700cdda-f60b-35cc-93b7-3160058b481a",
"id":16,"poolType":"NetworkFilesystem","host":"10.10.1.33","path":"/pool6/p
rimary6","port":2049,"url":"NetworkFilesystem://10.10.1.33/pool6/primary6/?
ROLE=Primary&STOREUUID=8700cdda-f60b-35cc-93b7-3160058b481a","isManaged":fa
lse}},"name":"ROOT-252","size":21474836480,"path":"b81b00cc-6a68-42a1-94c0-
7f4089b79973","volumeId":924,"vmName":"i-4-252-VM","accountId":10,"format":
"VHD","provisioningType":"THIN","id":924,"deviceId":0,"hypervisorType":"Xen
Server"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.10.1
.34/pool8/secondary8","_role":"Image"}},"vmName":"i-4-252-VM","name":"VM-se
rver_ROOT-252_20181004170057","hypervisorType":"XenServer","id":116,"quiesc
evm":false,"physicalSize":0}},"wait":0}}] }
2018-10-10 20:01:45,260 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
(logid:f2cbdadb) Seq 161-629659522901785142: Received: { Ans: , MgmtId:
19873467853209, via: 161(s-712-VM), Ver: v1, Flags: 10, { Answer } }
2018-10-10 20:01:45,288 DEBUG [c.c.r.ResourceLimitManagerImpl]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
(logid:f2cbdadb) Updating resource Type = snapshot count for Account = 10
Operation = decreasing Amount = 1
2018-10-10 20:01:45,300 DEBUG [c.c.r.ResourceLimitManagerImpl]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
(logid:f2cbdadb) Updating resource Type = secondary_storage count for
Account = 10 Operation = decreasing Amount = 434999808
2018-10-10 20:01:45,332 DEBUG [c.c.r.ResourceLimitManagerImpl]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
(logid:f2cbdadb) Updating resource Type = secondary_storage count for
Account = 10 Operation = decreasing Amount = 21023026688
2018-10-10 20:01:45,343 DEBUG [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
(logid:f2cbdadb) Done executing VM work job:
com.cloud.vm.VmWorkTakeVolumeSnapshot{"volumeId":924,"policyId":53,"snapsho
tId":180,"quiesceVm":false,"asyncBackup":false,"userId":1,"accountId":10,"v
mId":252,"handlerName":"VolumeApiServiceImpl"}
2018-10-10 20:01:45,343 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
(logid:f2cbdadb) Complete async job-29199, jobStatus: SUCCEEDED,
resultCode: 0, result:
rO0ABXNyAA5qYXZhLmxhbmcuTG9uZzuL5JDMjyPfAgABSgAFdmFsdWV4cgAQamF2YS5sYW5nLk5
1bWJlcoaslR0LlOCLAgAAeHAAAAAAAAAAtA
2018-10-10 20:01:45,344 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
(logid:f2cbdadb) Publish async job-29199 complete on message bus
2018-10-10 20:01:45,344 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
(logid:f2cbdadb) Wake up jobs related to job-29199
2018-10-10 20:01:45,344 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
(logid:f2cbdadb) Update db status for job-29199
2018-10-10 20:01:45,345 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199 ctx-1f9eb415)
(logid:f2cbdadb) Wake up jobs joined with job-29199 and disjoin all
subjobs created from job- 29199
2018-10-10 20:01:45,355 DEBUG [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199) (logid:f2cbdadb)
Done with run of VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot for VM
252, job origin: 29198
2018-10-10 20:01:45,355 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199) (logid:f2cbdadb)
Done executing com.cloud.vm.VmWorkTakeVolumeSnapshot for job-29199
2018-10-10 20:01:45,357 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(Work-Job-Executor-36:ctx-138e84ea job-29198/job-29199) (logid:f2cbdadb)
Remove job-29199 from job monitoring
2018-10-10 20:01:45,394 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-91:ctx-383c6cf4 job-29198 ctx-b559e6db) (logid:f2cbdadb)
Complete async job-29198, jobStatus: SUCCEEDED, resultCode: 0, result:
org.apache.cloudstack.api.response.SnapshotResponse/snapshot/{"id":"b87589b
5-e1f0-4303-9cbb-870b2c06ace5","account":"sami","domainid":"2947e215-4889-4
06d-b0a9-b8d82de1a276","domain":"VM-DOMAIN","snapshottype":"DAILY","volumei
d":"284eadd7-c29a-423f-94eb-da4c548ad945","volumename":"ROOT-252","volumety
pe":"ROOT","created":"2018-10-10T20:00:58+0300","name":"VM-server_ROOT-252_
20181010170058","intervaltype":"DAILY","state":"BackedUp","physicalsize":45
1809792,"zoneid":"533c8f88-2637-4c0e-8210-e0a355b62d82","tags":[],"revertab
le":false,"ostypeid":"da491b96-3748-11e5-a683-121326fde999","osdisplayname"
:"CentOS 6.5 (64-bit)","virtualsize":21474836480}
2018-10-10 20:01:45,394 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-91:ctx-383c6cf4 job-29198 ctx-b559e6db) (logid:f2cbdadb)
Publish async job-29198 complete on message bus
2018-10-10 20:01:45,394 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-91:ctx-383c6cf4 job-29198 ctx-b559e6db) (logid:f2cbdadb)
Wake up jobs related to job-29198
2018-10-10 20:01:45,394 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-91:ctx-383c6cf4 job-29198 ctx-b559e6db) (logid:f2cbdadb)
Update db status for job-29198
2018-10-10 20:01:45,395 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-91:ctx-383c6cf4 job-29198 ctx-b559e6db) (logid:f2cbdadb)
Wake up jobs joined with job-29198 and disjoin all subjobs created from
job- 29198
2018-10-10 20:01:45,403 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-91:ctx-383c6cf4 job-29198) (logid:f2cbdadb) Done
executing
org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for
job-29198
2018-10-10 20:01:45,403 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(API-Job-Executor-91:ctx-383c6cf4 job-29198) (logid:f2cbdadb) Remove
job-29198 from job monitoring
10.10.2018 19.18, "Andrei Mikhailovsky" <an...@arhont.com.INVALID>:
>Hi
>
>I can confirm that I am also having this issue on 4.11.1.0. To be honest,
>this issue was always present for me as far as I remember.
>
>Kind of an urgent issue to fix to stop people running out of space.
>
>Cheers
>
>----- Original Message -----
>> From: "Rafael Weingärtner" <ra...@gmail.com>
>> To: "users" <us...@cloudstack.apache.org>
>> Sent: Wednesday, 10 October, 2018 16:24:54
>> Subject: Re: VM Snapshot not removed from primary SR
>
>> Well, I am almost sure I have seen a PR fixing something like you
>>describe,
>> I thought that it went into 4.11.1.0.However, only with a more in depth
>> debugging I would be able to confirm your problem.
>>
>> Are you seeing any unexpected exception in your log files?
>>
>> On Wed, Oct 10, 2018 at 11:54 AM Sami Rajala (FAPPS)
>><sa...@fapps.com>
>> wrote:
>>
>>> Hi,
>>>
>>> CS running on 4.11.1.0 and does not work.
>>>
>>> BR
>>> -sami
>>>
>>>
>>> 10.10.2018 16.49, "Rafael Weingärtner" <ra...@gmail.com>:
>>>
>>> >If I am not mistaken, this has already been fixed in 4.11.1.0
>>> >
>>> >On Wed, Oct 10, 2018 at 10:31 AM Sami Rajala <sa...@fapps.com>
>>> >wrote:
>>> >
>>> >> Hi,
>>> >>
>>> >>
>>> >> I have got zero advice for this.
>>> >>
>>> >> "I have had weird issue since I updated CS from 4.9 to 4.10.
>>> >> CS does not delete/remove temporary snapshot from primary SR and I
>>>have
>>> >> had to removed it by manual time to time, before 30 snapshot limit
>>>has
>>> >> reached.
>>> >> + doesn't CS not update secondary_storege count on snapshot_count
>>>table²
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> Now I have a situation where Xenserver has 19 snapshot on primary
>>>SR for
>>> >> each VM + resource count for ²secondary_storege² in database
>>>increase
>>> >>all
>>> >> the time per account.
>>> >> ACS storage cleaner removed old snapshots from secondary when they
>>> >>getting
>>> >> old, but thats all, not decrease resource count for
>>> ²secondary_storege²
>>> >> with value of deleted snapshot ( or not remove snapshot from
>>>primary SR
>>> >> after snapshot has created and copied to secondary )
>>> >>
>>> >> I have still delete primary snapshot by manual + give a new value
>>>for
>>> >> ²secondary_storege² before they reach limit.
>>> >>
>>> >> has anyone got this kind of issue? any fix
>>> >>
>>> >> BR
>>> >> -sami
Re: VM Snapshot not removed from primary SR
Posted by Andrei Mikhailovsky <an...@arhont.com.INVALID>.
Hi
I can confirm that I am also having this issue on 4.11.1.0. To be honest, this issue was always present for me as far as I remember.
Kind of an urgent issue to fix to stop people running out of space.
Cheers
----- Original Message -----
> From: "Rafael Weingärtner" <ra...@gmail.com>
> To: "users" <us...@cloudstack.apache.org>
> Sent: Wednesday, 10 October, 2018 16:24:54
> Subject: Re: VM Snapshot not removed from primary SR
> Well, I am almost sure I have seen a PR fixing something like you describe,
> I thought that it went into 4.11.1.0.However, only with a more in depth
> debugging I would be able to confirm your problem.
>
> Are you seeing any unexpected exception in your log files?
>
> On Wed, Oct 10, 2018 at 11:54 AM Sami Rajala (FAPPS) <sa...@fapps.com>
> wrote:
>
>> Hi,
>>
>> CS running on 4.11.1.0 and does not work.
>>
>> BR
>> -sami
>>
>>
>> 10.10.2018 16.49, "Rafael Weingärtner" <ra...@gmail.com>:
>>
>> >If I am not mistaken, this has already been fixed in 4.11.1.0
>> >
>> >On Wed, Oct 10, 2018 at 10:31 AM Sami Rajala <sa...@fapps.com>
>> >wrote:
>> >
>> >> Hi,
>> >>
>> >>
>> >> I have got zero advice for this.
>> >>
>> >> "I have had weird issue since I updated CS from 4.9 to 4.10.
>> >> CS does not delete/remove temporary snapshot from primary SR and I have
>> >> had to removed it by manual time to time, before 30 snapshot limit has
>> >> reached.
>> >> + doesn't CS not update secondary_storege count on snapshot_count table”
>> >>
>> >>
>> >>
>> >>
>> >> Now I have a situation where Xenserver has 19 snapshot on primary SR for
>> >> each VM + resource count for ”secondary_storege” in database increase
>> >>all
>> >> the time per account.
>> >> ACS storage cleaner removed old snapshots from secondary when they
>> >>getting
>> >> old, but thats all, not decrease resource count for ”secondary_storege”
>> >> with value of deleted snapshot ( or not remove snapshot from primary SR
>> >> after snapshot has created and copied to secondary )
>> >>
>> >> I have still delete primary snapshot by manual + give a new value for
>> >> ”secondary_storege” before they reach limit.
>> >>
>> >> has anyone got this kind of issue? any fix
>> >>
>> >> BR
>> >> -sami
>> >>
>> >>
>> >>
>> >> 14.9.2018 12.26, "Sami Rajala (FAPPS)" <sa...@fapps.com>:
>> >>
>> >> >Continue,
>> >> >
>> >> >I created template from VM-s latest snapshot after 3 days ( one parent
>> >>+
>> >> >2
>> >> >child snapshot) - without errors/warning
>> >> >CS create VM from this template ok, but Xen could not start VM
>> >> >
>> >> >so, I return back to situation where every snapshot is full and manual
>> >> >removing snapshot from primary
>> >> >
>> >> >Has anyone any advice?
>> >> >
>> >> >BR
>> >> >-sami
>> >> >
>> >> >
>> >> >
>> >> >13.9.2018 7.56, "Sami Rajala (FAPPS)" <sa...@fapps.com>:
>> >> >
>> >> >>Hello
>> >> >>
>> >> >>I have had weird issue since I updated CS from 4.9 to 4.10.
>> >> >>CS does not delete/remove temporary snapshot from primary SR and I
>> >>have
>> >> >>had to removed it by manual time to time, before 30 snapshot limit has
>> >> >>reached.
>> >> >>+ doesn¹t CS not update secondary_storege count on snapshot_count
>> >>table
>> >> >>
>> >> >>It works fine on 4.9, but stop to work on 4.10 and I have wait
>> >> >>possibility
>> >> >>to update 4.11.1.
>> >> >>Update to 4.11.1 has done now and looks like this not work still
>> >> >>
>> >> >>I removed all snapshot and start over from clean table, snapshot
>> >>policy
>> >> >>is: DAILY, keep 2, delta = 5
>> >> >>There are now 3 snapshot on primary SR and 3 on secondary SR
>> >> >>
>> >> >>The Environment is Xen 7 + CS 4.11.1 + NFS storage
>> >> >>
>> >> >>Is there some other parameter I should look and any other work around
>> >>I
>> >> >>has to do to get it work ?
>> >> >>
>> >> >>Any hits?
>> >> >>
>> >> >>BR
>> >> >>-sami
>> >> >>
>> >> >>VM snapshot Log for last round:
>> >> >>
>> >> >>2018-09-13 06:02:15,983 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> >> >>(API-Job-Executor-83:ctx-cfb1cd95 job-28020) (logid:4c7a1c38)
>> >>Executing
>> >> >>AsyncJobVO {id:28020, userId: 1, accountId: 7, instanceType: Snapshot,
>> >> >>instanceId: 2656, cmd:
>> >> >>org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd,
>> >> >>cmdInfo:
>> >>
>> >>>>{"policyid":"17","ctxUserId":"1","volumeid":"238","ctxStartEventId":"1"
>> >>>>,"
>> >> >>i
>> >> >>d
>> >> >>":"2656","ctxAccountId":"7"}, cmdVersion: 0, status: IN_PROGRESS,
>> >> >>processStatus: 0, resultCode: 0, result: null, initMsid:
>> >>19873467853209,
>> >> >>completeMsid: null, lastUpdated: null, lastPolled: null, created:
>> >>null}
>> >> >>2018-09-13 06:02:15,991 DEBUG [c.c.u.AccountManagerImpl]
>> >> >>(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6)
>> >> >>(logid:4c7a1c38)
>> >> >>Access to Acct[479c643e-9c84-41fb-9f0a-9bb999893a25-juha] granted to
>> >> >>Acct[479c643e-9c84-41fb-9f0a-9bb999893a25-juha] by DomainChecker
>> >> >>2018-09-13 06:02:16,045 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> >> >>(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6)
>> >> >>(logid:4c7a1c38)
>> >> >>Sync job-28021 execution on object VmWorkJobQueue.228
>> >> >>2018-09-13 06:02:17,115 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021)
>> >>(logid:4c7a1c38)
>> >> >>Executing AsyncJobVO {id:28021, userId: 1, accountId: 7, instanceType:
>> >> >>null, instanceId: null, cmd: com.cloud.vm.VmWorkTakeVolumeSnapshot,
>> >> >>cmdInfo:
>> >>
>> >>>>rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrVGFrZVZvbHVtZVNuYXBzaG90BL5gG4Li1c8
>> >>>>CA
>> >> >>A
>> >> >>Z
>> >>
>> >>>>aAAthc3luY0JhY2t1cFoACXF1aWVzY2VWbUwADGxvY2F0aW9uVHlwZXQAKUxjb20vY2xvdW
>> >>>>Qv
>> >> >>c
>> >> >>3
>> >>
>> >>>>RvcmFnZS9TbmFwc2hvdCRMb2NhdGlvblR5cGU7TAAIcG9saWN5SWR0ABBMamF2YS9sYW5nL
>> >>>>0x
>> >> >>v
>> >> >>b
>> >>
>> >>>>mc7TAAKc25hcHNob3RJZHEAfgACTAAIdm9sdW1lSWRxAH4AAnhyABNjb20uY2xvdWQudm0u
>> >>>>Vm
>> >> >>1
>> >> >>X
>> >>
>> >>>>b3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlkTAALaGFuZGxlck5
>> >>>>hb
>> >> >>W
>> >> >>V
>> >>
>> >>>>0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAAHAAAAAAAAAAEAAAAAAAAA5HQAFFZvbH
>> >>>>Vt
>> >> >>Z
>> >> >>U
>> >>
>> >>>>FwaVNlcnZpY2VJbXBsAABwc3IADmphdmEubGFuZy5Mb25nO4vkkMyPI98CAAFKAAV2YWx1Z
>> >>>>Xh
>> >> >>y
>> >> >>A
>> >>
>> >>>>BBqYXZhLmxhbmcuTnVtYmVyhqyVHQuU4IsCAAB4cAAAAAAAAAARc3EAfgAHAAAAAAAACmBz
>> >>>>cQ
>> >> >>B
>> >> >>-
>> >> >>AAcAAAAAAAAA7g, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0,
>> >> >>resultCode: 0, result: null, initMsid: 19873467853209, completeMsid:
>> >> >>null,
>> >> >>lastUpdated: null, lastPolled: null, created: Thu Sep 13 06:02:16 EEST
>> >> >>2018}
>> >> >>2018-09-13 06:02:17,116 DEBUG [c.c.v.VmWorkJobDispatcher]
>> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021)
>> >>(logid:4c7a1c38)
>> >> >>Run VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot for VM 228, job
>> >> >>origin: 28020
>> >> >>2018-09-13 06:02:17,117 DEBUG [c.c.v.VmWorkJobHandlerProxy]
>> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>> >> >>(logid:4c7a1c38) Execute VM work job:
>> >>
>> >>>>com.cloud.vm.VmWorkTakeVolumeSnapshot{"volumeId":238,"policyId":17,"sna
>> >>>>ps
>> >> >>h
>> >> >>o
>> >>
>> >>>>tId":2656,"quiesceVm":false,"asyncBackup":false,"userId":1,"accountId":
>> >>>>7,
>> >> >>"
>> >> >>v
>> >> >>mId":228,"handlerName":"VolumeApiServiceImpl"}
>> >> >>2018-09-13 06:02:17,197 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
>> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>> >> >>(logid:4c7a1c38) getCommandHostDelegation: class
>> >> >>org.apache.cloudstack.storage.command.CreateObjectCommand
>> >> >>2018-09-13 06:02:17,197 DEBUG [c.c.h.XenServerGuru]
>> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>> >> >>(logid:4c7a1c38) We are returning the default host to execute commands
>> >> >>because the command is not of Copy type.
>> >> >>2018-09-13 06:02:17,199 DEBUG [c.c.a.t.Request]
>> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>> >> >>(logid:4c7a1c38) Seq 128-4630826316843724436: Sending { Cmd , MgmtId:
>> >> >>19873467853209, via: 128(xenserver-15), Ver: v1, Flags: 100011,
>> >>
>> >>>>[{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{
>> >>>>"o
>> >> >>r
>> >> >>g
>> >>
>> >>>>.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"969c
>> >>>>79
>> >> >>6
>> >> >>d
>> >>
>> >>>>-072e-4272-adc6-93a492c03ea3","volumeType":"ROOT","dataStore":{"org.apa
>> >>>>ch
>> >> >>e
>> >> >>.
>> >> >>cloudstack.storage.to
>> >> .PrimaryDataStoreTO":{"uuid":"6568d30f-82b6-35a0-bfe
>> >> >>5
>> >> >>-
>> >>
>> >>>>be17c1398ed8","id":2,"poolType":"NetworkFilesystem","host":"10.10.1.33"
>> >>>>,"
>> >> >>p
>> >> >>a
>> >> >>th":"/pool3/primary3","port":2049,"url":"NetworkFilesystem://
>> >> 10.10.1.33/p
>> >> >>o
>> >> >>o
>> >>
>> >>>>l3/primary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a0-bfe5-be17c1398ed
>> >>>>8"
>> >> >>,
>> >> >>"
>> >>
>> >>>>isManaged":false}},"name":"ROOT-228","size":21474836480,"path":"3d2a971
>> >>>>6-
>> >> >>8
>> >> >>f
>> >>
>> >>>>a1-497e-8753-941117a0ef50","volumeId":238,"vmName":"i-7-228-VM","accoun
>> >>>>tI
>> >> >>d
>> >> >>"
>> >>
>> >>>>:7,"format":"VHD","provisioningType":"THIN","id":238,"deviceId":0,"hype
>> >>>>rv
>> >> >>i
>> >> >>s
>> >>
>> >>>>orType":"XenServer"},"parentSnapshotPath":"c7b2824a-7402-4f66-b2e6-06ed
>> >>>>45
>> >> >>2
>> >> >>0
>> >> >>72ad","dataStore":{"org.apache.cloudstack.storage.to
>> >> .PrimaryDataStoreTO":
>> >> >>{
>> >> >>"
>> >>
>> >>>>uuid":"6568d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType":"Network
>> >>>>Fi
>> >> >>l
>> >> >>e
>> >>
>> >>>>system","host":"10.10.1.33","path":"/pool3/primary3","port":2049,"url":
>> >>>>"N
>> >> >>e
>> >> >>t
>> >> >>workFilesystem://
>> >> 10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUID=6568d3
>> >> >>0
>> >> >>f
>> >>
>> >>>>-82b6-35a0-bfe5-be17c1398ed8","isManaged":false}},"vmName":"i-7-228-VM"
>> >>>>,"
>> >> >>n
>> >> >>a
>> >>
>> >>>>me":"BGI-Maria_ROOT-228_20180913030215","hypervisorType":"XenServer","i
>> >>>>d"
>> >> >>:
>> >> >>2
>> >>
>> >>>>656,"quiescevm":false,"parents":["c7b2824a-7402-4f66-b2e6-06ed452072ad"
>> >>>>],
>> >> >>"
>> >> >>p
>> >> >>hysicalSize":0}},"wait":0}}] }
>> >> >>2018-09-13 06:02:17,199 DEBUG [c.c.a.t.Request]
>> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>> >> >>(logid:4c7a1c38) Seq 128-4630826316843724436: Executing: { Cmd ,
>> >>MgmtId:
>> >> >>19873467853209, via: 128(xenserver-15), Ver: v1, Flags: 100011,
>> >>
>> >>>>[{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{
>> >>>>"o
>> >> >>r
>> >> >>g
>> >>
>> >>>>.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"969c
>> >>>>79
>> >> >>6
>> >> >>d
>> >>
>> >>>>-072e-4272-adc6-93a492c03ea3","volumeType":"ROOT","dataStore":{"org.apa
>> >>>>ch
>> >> >>e
>> >> >>.
>> >> >>cloudstack.storage.to
>> >> .PrimaryDataStoreTO":{"uuid":"6568d30f-82b6-35a0-bfe
>> >> >>5
>> >> >>-
>> >>
>> >>>>be17c1398ed8","id":2,"poolType":"NetworkFilesystem","host":"10.10.1.33"
>> >>>>,"
>> >> >>p
>> >> >>a
>> >> >>th":"/pool3/primary3","port":2049,"url":"NetworkFilesystem://
>> >> 10.10.1.33/p
>> >> >>o
>> >> >>o
>> >>
>> >>>>l3/primary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a0-bfe5-be17c1398ed
>> >>>>8"
>> >> >>,
>> >> >>"
>> >>
>> >>>>isManaged":false}},"name":"ROOT-228","size":21474836480,"path":"3d2a971
>> >>>>6-
>> >> >>8
>> >> >>f
>> >>
>> >>>>a1-497e-8753-941117a0ef50","volumeId":238,"vmName":"i-7-228-VM","accoun
>> >>>>tI
>> >> >>d
>> >> >>"
>> >>
>> >>>>:7,"format":"VHD","provisioningType":"THIN","id":238,"deviceId":0,"hype
>> >>>>rv
>> >> >>i
>> >> >>s
>> >>
>> >>>>orType":"XenServer"},"parentSnapshotPath":"c7b2824a-7402-4f66-b2e6-06ed
>> >>>>45
>> >> >>2
>> >> >>0
>> >> >>72ad","dataStore":{"org.apache.cloudstack.storage.to
>> >> .PrimaryDataStoreTO":
>> >> >>{
>> >> >>"
>> >>
>> >>>>uuid":"6568d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType":"Network
>> >>>>Fi
>> >> >>l
>> >> >>e
>> >>
>> >>>>system","host":"10.10.1.33","path":"/pool3/primary3","port":2049,"url":
>> >>>>"N
>> >> >>e
>> >> >>t
>> >> >>workFilesystem://
>> >> 10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUID=6568d3
>> >> >>0
>> >> >>f
>> >>
>> >>>>-82b6-35a0-bfe5-be17c1398ed8","isManaged":false}},"vmName":"i-7-228-VM"
>> >>>>,"
>> >> >>n
>> >> >>a
>> >>
>> >>>>me":"BGI-Maria_ROOT-228_20180913030215","hypervisorType":"XenServer","i
>> >>>>d"
>> >> >>:
>> >> >>2
>> >>
>> >>>>656,"quiescevm":false,"parents":["c7b2824a-7402-4f66-b2e6-06ed452072ad"
>> >>>>],
>> >> >>"
>> >> >>p
>> >> >>hysicalSize":0}},"wait":0}}] }
>> >> >>2018-09-13 06:02:19,815 DEBUG [c.c.a.m.DirectAgentAttache]
>> >> >>(DirectAgent-206:ctx-370f4cd9) (logid:4c7a1c38) Seq
>> >> >>128-4630826316843724436: Response Received:
>> >> >>2018-09-13 06:02:19,815 DEBUG [c.c.a.t.Request]
>> >> >>(DirectAgent-206:ctx-370f4cd9) (logid:4c7a1c38) Seq
>> >> >>128-4630826316843724436: Processing: { Ans: , MgmtId: 19873467853209,
>> >> >>via: 128(xenserver-15), Ver: v1, Flags: 10,
>> >>
>> >>>>[{"org.apache.cloudstack.storage.command.CreateObjectAnswer":{"data":{"
>> >>>>or
>> >> >>g
>> >> >>.
>> >>
>> >>>>apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"e91fb041-779d-4
>> >>>>09
>> >> >>0
>> >> >>-
>> >>
>> >>>>b14c-4f57f4cda54b","id":0,"quiescevm":false,"physicalSize":0}},"result"
>> >>>>:t
>> >> >>r
>> >> >>u
>> >> >>e,"wait":0}}] }
>> >> >>2018-09-13 06:02:19,815 DEBUG [c.c.a.t.Request]
>> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>> >> >>(logid:4c7a1c38) Seq 128-4630826316843724436: Received: { Ans: ,
>> >>MgmtId:
>> >> >>19873467853209, via: 128(xenserver-15), Ver: v1, Flags: 10, {
>> >> >>CreateObjectAnswer } }
>> >> >>2018-09-13 06:02:19,934 DEBUG [o.a.c.s.m.AncientDataMotionStrategy]
>> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>> >> >>(logid:4c7a1c38) copyAsync inspecting src type SNAPSHOT copyAsync
>> >> >>inspecting dest type SNAPSHOT
>> >> >>2018-09-13 06:02:20,017 ERROR [o.a.c.s.e.DefaultEndPointSelector]
>> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>> >> >>(logid:4c7a1c38) IR24 select BACKUPSNAPSHOT from primary to secondary
>> >> >>2656
>> >> >>dest=2656
>> >> >>2018-09-13 06:02:20,023 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
>> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>> >> >>(logid:4c7a1c38) getCommandHostDelegation: class
>> >> >>org.apache.cloudstack.storage.command.CopyCommand
>> >> >>2018-09-13 06:02:20,023 DEBUG [c.c.h.XenServerGuru]
>> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>> >> >>(logid:4c7a1c38) We are returning the default host to execute commands
>> >> >>because the source and destination objects are not NFS type.
>> >> >>2018-09-13 06:02:20,025 DEBUG [c.c.a.t.Request]
>> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>> >> >>(logid:4c7a1c38) Seq 130-7345089517264579763: Sending { Cmd , MgmtId:
>> >> >>19873467853209, via: 130(xenserver-81), Ver: v1, Flags: 100111,
>> >>
>> >>>>[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.ap
>> >>>>ac
>> >> >>h
>> >> >>e
>> >> >>.cloudstack.storage.to
>> >> .SnapshotObjectTO":{"path":"e91fb041-779d-4090-b14c
>> >> >>-
>> >> >>4
>> >>
>> >>>>f57f4cda54b","volume":{"uuid":"969c796d-072e-4272-adc6-93a492c03ea3","v
>> >>>>ol
>> >> >>u
>> >> >>m
>> >>
>> >>>>eType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDat
>> >>>>aS
>> >> >>t
>> >> >>o
>> >>
>> >>>>reTO":{"uuid":"6568d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType":
>> >>>>"N
>> >> >>e
>> >> >>t
>> >>
>> >>>>workFilesystem","host":"10.10.1.33","path":"/pool3/primary3","port":204
>> >>>>9,
>> >> >>"
>> >> >>u
>> >> >>rl":"NetworkFilesystem://
>> >> 10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUI
>> >> >>D
>> >> >>=
>> >>
>> >>>>6568d30f-82b6-35a0-bfe5-be17c1398ed8","isManaged":false}},"name":"ROOT-
>> >>>>22
>> >> >>8
>> >> >>"
>> >>
>> >>>>,"size":21474836480,"path":"3d2a9716-8fa1-497e-8753-941117a0ef50","volu
>> >>>>me
>> >> >>I
>> >> >>d
>> >>
>> >>>>":238,"vmName":"i-7-228-VM","accountId":7,"format":"VHD","provisioningT
>> >>>>yp
>> >> >>e
>> >> >>"
>> >>
>> >>>>:"THIN","id":238,"deviceId":0,"hypervisorType":"XenServer"},"parentSnap
>> >>>>sh
>> >> >>o
>> >> >>t
>> >>
>> >>>>Path":"c7b2824a-7402-4f66-b2e6-06ed452072ad","dataStore":{"org.apache.c
>> >>>>lo
>> >> >>u
>> >> >>d
>> >> >>stack.storage.to
>> >> .PrimaryDataStoreTO":{"uuid":"6568d30f-82b6-35a0-bfe5-be1
>> >> >>7
>> >> >>c
>> >>
>> >>>>1398ed8","id":2,"poolType":"NetworkFilesystem","host":"10.10.1.33","pat
>> >>>>h"
>> >> >>:
>> >> >>"
>> >> >>/pool3/primary3","port":2049,"url":"NetworkFilesystem://
>> >> 10.10.1.33/pool3/
>> >> >>p
>> >> >>r
>> >>
>> >>>>imary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a0-bfe5-be17c1398ed8","i
>> >>>>sM
>> >> >>a
>> >> >>n
>> >>
>> >>>>aged":false}},"vmName":"i-7-228-VM","name":"BGI-Maria_ROOT-228_20180913
>> >>>>03
>> >> >>0
>> >> >>2
>> >>
>> >>>>15","hypervisorType":"XenServer","id":2656,"quiescevm":false,"parents":
>> >>>>["
>> >> >>c
>> >> >>7
>> >>
>> >>>>b2824a-7402-4f66-b2e6-06ed452072ad"],"physicalSize":0}},"destTO":{"org.
>> >>>>ap
>> >> >>a
>> >> >>c
>> >> >>he.cloudstack.storage.to
>> >> .SnapshotObjectTO":{"path":"snapshots/7/238","vol
>> >> >>u
>> >> >>m
>> >>
>> >>>>e":{"uuid":"969c796d-072e-4272-adc6-93a492c03ea3","volumeType":"ROOT","
>> >>>>da
>> >> >>t
>> >> >>a
>> >> >>Store":{"org.apache.cloudstack.storage.to
>> >> .PrimaryDataStoreTO":{"uuid":"65
>> >> >>6
>> >> >>8
>> >>
>> >>>>d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType":"NetworkFilesystem"
>> >>>>,"
>> >> >>h
>> >> >>o
>> >>
>> >>>>st":"10.10.1.33","path":"/pool3/primary3","port":2049,"url":"NetworkFil
>> >>>>es
>> >> >>y
>> >> >>s
>> >> >>tem://
>> >> 10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a
>> >> >>0
>> >> >>-
>> >>
>> >>>>bfe5-be17c1398ed8","isManaged":false}},"name":"ROOT-228","size":2147483
>> >>>>64
>> >> >>8
>> >> >>0
>> >>
>> >>>>,"path":"3d2a9716-8fa1-497e-8753-941117a0ef50","volumeId":238,"vmName":
>> >>>>"i
>> >> >>-
>> >> >>7
>> >>
>> >>>>-228-VM","accountId":7,"format":"VHD","provisioningType":"THIN","id":23
>> >>>>8,
>> >> >>"
>> >> >>d
>> >>
>> >>>>eviceId":0,"hypervisorType":"XenServer"},"parentSnapshotPath":"snapshot
>> >>>>s/
>> >> >>7
>> >> >>/
>> >>
>> >>>>238/0c8c6ee3-ce3d-4241-8509-5193d2cff491.vhd","dataStore":{"com.cloud.a
>> >>>>ge
>> >> >>n
>> >> >>t
>> >> >>.api.to.NfsTO":{"_url":"nfs://10.10.1.34/pool8/secondary8
>> >> ","_role":"Image
>> >> >>"
>> >> >>}
>> >>
>> >>>>},"vmName":"i-7-228-VM","name":"BGI-Maria_ROOT-228_20180913030215","hyp
>> >>>>er
>> >> >>v
>> >> >>i
>> >>
>> >>>>sorType":"XenServer","id":2656,"quiescevm":false,"parents":["snapshots/
>> >>>>7/
>> >> >>2
>> >> >>3
>> >>
>> >>>>8/93a42896-0da1-4b82-8985-d28464f1ac4a.vhd","snapshots/7/238/0c8c6ee3-c
>> >>>>e3
>> >> >>d
>> >> >>-
>> >>
>> >>>>4241-8509-5193d2cff491.vhd"],"physicalSize":0}},"executeInSequence":tru
>> >>>>e,
>> >> >>"
>> >> >>o
>> >> >>ptions":{"fullSnapshot":"false"},"options2":{},"wait":28800}}] }
>> >> >>2018-09-13 06:02:20,025 DEBUG [c.c.a.t.Request]
>> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>> >> >>(logid:4c7a1c38) Seq 130-7345089517264579763: Executing: { Cmd ,
>> >>MgmtId:
>> >> >>19873467853209, via: 130(xenserver-81), Ver: v1, Flags: 100111,
>> >>
>> >>>>[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.ap
>> >>>>ac
>> >> >>h
>> >> >>e
>> >> >>.cloudstack.storage.to
>> >> .SnapshotObjectTO":{"path":"e91fb041-779d-4090-b14c
>> >> >>-
>> >> >>4
>> >>
>> >>>>f57f4cda54b","volume":{"uuid":"969c796d-072e-4272-adc6-93a492c03ea3","v
>> >>>>ol
>> >> >>u
>> >> >>m
>> >>
>> >>>>eType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDat
>> >>>>aS
>> >> >>t
>> >> >>o
>> >>
>> >>>>reTO":{"uuid":"6568d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType":
>> >>>>"N
>> >> >>e
>> >> >>t
>> >>
>> >>>>workFilesystem","host":"10.10.1.33","path":"/pool3/primary3","port":204
>> >>>>9,
>> >> >>"
>> >> >>u
>> >> >>rl":"NetworkFilesystem://
>> >> 10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUI
>> >> >>D
>> >> >>=
>> >>
>> >>>>6568d30f-82b6-35a0-bfe5-be17c1398ed8","isManaged":false}},"name":"ROOT-
>> >>>>22
>> >> >>8
>> >> >>"
>> >>
>> >>>>,"size":21474836480,"path":"3d2a9716-8fa1-497e-8753-941117a0ef50","volu
>> >>>>me
>> >> >>I
>> >> >>d
>> >>
>> >>>>":238,"vmName":"i-7-228-VM","accountId":7,"format":"VHD","provisioningT
>> >>>>yp
>> >> >>e
>> >> >>"
>> >>
>> >>>>:"THIN","id":238,"deviceId":0,"hypervisorType":"XenServer"},"parentSnap
>> >>>>sh
>> >> >>o
>> >> >>t
>> >>
>> >>>>Path":"c7b2824a-7402-4f66-b2e6-06ed452072ad","dataStore":{"org.apache.c
>> >>>>lo
>> >> >>u
>> >> >>d
>> >> >>stack.storage.to
>> >> .PrimaryDataStoreTO":{"uuid":"6568d30f-82b6-35a0-bfe5-be1
>> >> >>7
>> >> >>c
>> >>
>> >>>>1398ed8","id":2,"poolType":"NetworkFilesystem","host":"10.10.1.33","pat
>> >>>>h"
>> >> >>:
>> >> >>"
>> >> >>/pool3/primary3","port":2049,"url":"NetworkFilesystem://
>> >> 10.10.1.33/pool3/
>> >> >>p
>> >> >>r
>> >>
>> >>>>imary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a0-bfe5-be17c1398ed8","i
>> >>>>sM
>> >> >>a
>> >> >>n
>> >>
>> >>>>aged":false}},"vmName":"i-7-228-VM","name":"BGI-Maria_ROOT-228_20180913
>> >>>>03
>> >> >>0
>> >> >>2
>> >>
>> >>>>15","hypervisorType":"XenServer","id":2656,"quiescevm":false,"parents":
>> >>>>["
>> >> >>c
>> >> >>7
>> >>
>> >>>>b2824a-7402-4f66-b2e6-06ed452072ad"],"physicalSize":0}},"destTO":{"org.
>> >>>>ap
>> >> >>a
>> >> >>c
>> >> >>he.cloudstack.storage.to
>> >> .SnapshotObjectTO":{"path":"snapshots/7/238","vol
>> >> >>u
>> >> >>m
>> >>
>> >>>>e":{"uuid":"969c796d-072e-4272-adc6-93a492c03ea3","volumeType":"ROOT","
>> >>>>da
>> >> >>t
>> >> >>a
>> >> >>Store":{"org.apache.cloudstack.storage.to
>> >> .PrimaryDataStoreTO":{"uuid":"65
>> >> >>6
>> >> >>8
>> >>
>> >>>>d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType":"NetworkFilesystem"
>> >>>>,"
>> >> >>h
>> >> >>o
>> >>
>> >>>>st":"10.10.1.33","path":"/pool3/primary3","port":2049,"url":"NetworkFil
>> >>>>es
>> >> >>y
>> >> >>s
>> >> >>tem://
>> >> 10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a
>> >> >>0
>> >> >>-
>> >>
>> >>>>bfe5-be17c1398ed8","isManaged":false}},"name":"ROOT-228","size":2147483
>> >>>>64
>> >> >>8
>> >> >>0
>> >>
>> >>>>,"path":"3d2a9716-8fa1-497e-8753-941117a0ef50","volumeId":238,"vmName":
>> >>>>"i
>> >> >>-
>> >> >>7
>> >>
>> >>>>-228-VM","accountId":7,"format":"VHD","provisioningType":"THIN","id":23
>> >>>>8,
>> >> >>"
>> >> >>d
>> >>
>> >>>>eviceId":0,"hypervisorType":"XenServer"},"parentSnapshotPath":"snapshot
>> >>>>s/
>> >> >>7
>> >> >>/
>> >>
>> >>>>238/0c8c6ee3-ce3d-4241-8509-5193d2cff491.vhd","dataStore":{"com.cloud.a
>> >>>>ge
>> >> >>n
>> >> >>t
>> >> >>.api.to.NfsTO":{"_url":"nfs://10.10.1.34/pool8/secondary8
>> >> ","_role":"Image
>> >> >>"
>> >> >>}
>> >>
>> >>>>},"vmName":"i-7-228-VM","name":"BGI-Maria_ROOT-228_20180913030215","hyp
>> >>>>er
>> >> >>v
>> >> >>i
>> >>
>> >>>>sorType":"XenServer","id":2656,"quiescevm":false,"parents":["snapshots/
>> >>>>7/
>> >> >>2
>> >> >>3
>> >>
>> >>>>8/93a42896-0da1-4b82-8985-d28464f1ac4a.vhd","snapshots/7/238/0c8c6ee3-c
>> >>>>e3
>> >> >>d
>> >> >>-
>> >>
>> >>>>4241-8509-5193d2cff491.vhd"],"physicalSize":0}},"executeInSequence":tru
>> >>>>e,
>> >> >>"
>> >> >>o
>> >> >>ptions":{"fullSnapshot":"false"},"options2":{},"wait":28800}}] }
>> >> >>2018-09-13 06:04:28,895 DEBUG [c.c.h.x.r.CitrixResourceBase]
>> >> >>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Host 10.10.1.81
>> >> >>OpaqueRef:f83331df-6634-63fd-6b95-44607a6069b8: Removing SR
>> >> >>2018-09-13 06:04:28,913 DEBUG [c.c.h.x.r.CitrixResourceBase]
>> >> >>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Host 10.10.1.81
>> >> >>OpaqueRef:05380b03-4cb8-85a7-6a6a-a1595c188f60: Unplugging pbd
>> >> >>2018-09-13 06:04:30,881 DEBUG [c.c.h.x.r.CitrixResourceBase]
>> >> >>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Host 10.10.1.81
>> >> >>OpaqueRef:f83331df-6634-63fd-6b95-44607a6069b8: Forgetting
>> >> >>2018-09-13 06:04:30,891 DEBUG [c.c.h.x.r.XenServerStorageProcessor]
>> >> >>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Successfully destroyed
>> >> >>snapshot on volume: 3d2a9716-8fa1-497e-8753-941117a0ef50 execept this
>> >> >>current snapshot e91fb041-779d-4090-b14c-4f57f4cda54b
>> >> >>2018-09-13 06:04:30,891 INFO [c.c.h.x.r.XenServerStorageProcessor]
>> >> >>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) New snapshot details:
>> >>
>> >>>>SnapshotTO[datastore=null|volume=null|pathsnapshots/7/238/0560d536-c945
>> >>>>-4
>> >> >>a
>> >> >>6
>> >> >>1-8c4c-dbaf29f29993.vhd]
>> >> >>2018-09-13 06:04:30,891 INFO [c.c.h.x.r.XenServerStorageProcessor]
>> >> >>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) New snapshot physical
>> >> >>utilization: 2342932992
>> >> >>2018-09-13 06:04:30,891 DEBUG [c.c.a.m.DirectAgentAttache]
>> >> >>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Seq
>> >> >>130-7345089517264579763: Response Received:
>> >> >>2018-09-13 06:04:30,891 DEBUG [c.c.a.t.Request]
>> >> >>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Seq
>> >> >>130-7345089517264579763: Processing: { Ans: , MgmtId: 19873467853209,
>> >> >>via: 130(xenserver-81), Ver: v1, Flags: 110,
>> >>
>> >>>>[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"or
>> >>>>g.
>> >> >>a
>> >> >>p
>> >> >>ache.cloudstack.storage.to
>> >> .SnapshotObjectTO":{"path":"snapshots/7/238/056
>> >> >>0
>> >> >>d
>> >>
>> >>>>536-c945-4a61-8c4c-dbaf29f29993.vhd","parentSnapshotPath":"snapshots/7/
>> >>>>23
>> >> >>8
>> >> >>/
>> >>
>> >>>>0c8c6ee3-ce3d-4241-8509-5193d2cff491.vhd","id":0,"quiescevm":false,"phy
>> >>>>si
>> >> >>c
>> >> >>a
>> >> >>lSize":2342932992}},"result":true,"wait":0}}] }
>> >> >>2018-09-13 06:04:30,891 DEBUG [c.c.a.m.AgentAttache]
>> >> >>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Seq
>> >> >>130-7345089517264579763: No more commands found
>> >> >>2018-09-13 06:04:30,891 DEBUG [c.c.a.t.Request]
>> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>> >> >>(logid:4c7a1c38) Seq 130-7345089517264579763: Received: { Ans: ,
>> >>MgmtId:
>> >> >>19873467853209, via: 130(xenserver-81), Ver: v1, Flags: 110, {
>> >> >>CopyCmdAnswer } }
>> >> >>2018-09-13 06:04:30,957 DEBUG [c.c.r.ResourceLimitManagerImpl]
>> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>> >> >>(logid:4c7a1c38) Updating resource Type = secondary_storage count for
>> >> >>Account = 7 Operation = decreasing Amount = 19131903488
>> >> >>2018-09-13 06:04:30,969 DEBUG [c.c.v.VmWorkJobHandlerProxy]
>> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>> >> >>(logid:4c7a1c38) Done executing VM work job:
>> >>
>> >>>>com.cloud.vm.VmWorkTakeVolumeSnapshot{"volumeId":238,"policyId":17,"sna
>> >>>>ps
>> >> >>h
>> >> >>o
>> >>
>> >>>>tId":2656,"quiesceVm":false,"asyncBackup":false,"userId":1,"accountId":
>> >>>>7,
>> >> >>"
>> >> >>v
>> >> >>mId":228,"handlerName":"VolumeApiServiceImpl"}
>> >> >>2018-09-13 06:04:30,969 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>> >> >>(logid:4c7a1c38) Complete async job-28021, jobStatus: SUCCEEDED,
>> >> >>resultCode: 0, result:
>> >>
>> >>>>rO0ABXNyAA5qYXZhLmxhbmcuTG9uZzuL5JDMjyPfAgABSgAFdmFsdWV4cgAQamF2YS5sYW5
>> >>>>nL
>> >> >>k
>> >> >>5
>> >> >>1bWJlcoaslR0LlOCLAgAAeHAAAAAAAAAKYA
>> >> >>2018-09-13 06:04:30,970 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>> >> >>(logid:4c7a1c38) Publish async job-28021 complete on message bus
>> >> >>2018-09-13 06:04:30,970 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>> >> >>(logid:4c7a1c38) Wake up jobs related to job-28021
>> >> >>2018-09-13 06:04:30,970 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>> >> >>(logid:4c7a1c38) Update db status for job-28021
>> >> >>2018-09-13 06:04:30,971 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>> >> >>(logid:4c7a1c38) Wake up jobs joined with job-28021 and disjoin all
>> >> >>subjobs created from job- 28021
>> >> >>2018-09-13 06:04:30,983 DEBUG [c.c.v.VmWorkJobDispatcher]
>> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021)
>> >>(logid:4c7a1c38)
>> >> >>Done with run of VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot
>> >>for
>> >> >>VM
>> >> >>228, job origin: 28020
>> >> >>2018-09-13 06:04:30,983 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021)
>> >>(logid:4c7a1c38)
>> >> >>Done executing com.cloud.vm.VmWorkTakeVolumeSnapshot for job-28021
>> >> >>2018-09-13 06:04:30,984 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021)
>> >>(logid:4c7a1c38)
>> >> >>Remove job-28021 from job monitoring
>> >> >>2018-09-13 06:04:31,026 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> >> >>(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6)
>> >> >>(logid:4c7a1c38)
>> >> >>Complete async job-28020, jobStatus: SUCCEEDED, resultCode: 0, result:
>> >>
>> >>>>org.apache.cloudstack.api.response.SnapshotResponse/snapshot/{"id":"3c8
>> >>>>66
>> >> >>2
>> >> >>b
>> >>
>> >>>>2-c76f-4e3a-a928-f76dad820122","account":"juha","domainid":"d3c2420f-75
>> >>>>03
>> >> >>-
>> >> >>4
>> >>
>> >>>>560-a2da-d7e7149ce390","domain":"BGI","snapshottype":"DAILY","volumeid"
>> >>>>:"
>> >> >>9
>> >> >>6
>> >>
>> >>>>9c796d-072e-4272-adc6-93a492c03ea3","volumename":"ROOT-228","volumetype
>> >>>>":
>> >> >>"
>> >> >>R
>> >>
>> >>>>OOT","created":"2018-09-13T06:02:15+0300","name":"BGI-Maria_ROOT-228_20
>> >>>>18
>> >> >>0
>> >> >>9
>> >>
>> >>>>13030215","intervaltype":"DAILY","state":"BackedUp","physicalsize":2342
>> >>>>93
>> >> >>2
>> >> >>9
>> >>
>> >>>>92,"zoneid":"533c8f88-2637-4c0e-8210-e0a355b62d82","tags":[],"revertabl
>> >>>>e"
>> >> >>:
>> >> >>f
>> >>
>> >>>>alse,"ostypeid":"da491b96-3748-11e5-a683-121326fde999","osdisplayname":
>> >>>>"C
>> >> >>e
>> >> >>n
>> >> >>tOS 6.5 (64-bit)","virtualsize":21474836480}
>> >> >>2018-09-13 06:04:31,027 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> >> >>(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6)
>> >> >>(logid:4c7a1c38)
>> >> >>Publish async job-28020 complete on message bus
>> >> >>2018-09-13 06:04:31,027 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> >> >>(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6)
>> >> >>(logid:4c7a1c38)
>> >> >>Wake up jobs related to job-28020
>> >> >>2018-09-13 06:04:31,027 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> >> >>(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6)
>> >> >>(logid:4c7a1c38)
>> >> >>Update db status for job-28020
>> >> >>2018-09-13 06:04:31,028 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> >> >>(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6)
>> >> >>(logid:4c7a1c38)
>> >> >>Wake up jobs joined with job-28020 and disjoin all subjobs created
>> >>from
>> >> >>job- 28020
>> >> >>2018-09-13 06:04:31,039 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> >> >>(API-Job-Executor-83:ctx-cfb1cd95 job-28020) (logid:4c7a1c38) Done
>> >> >>executing
>> >> >>org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for
>> >> >>job-28020
>> >> >>2018-09-13 06:04:31,039 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>> >> >>(API-Job-Executor-83:ctx-cfb1cd95 job-28020) (logid:4c7a1c38) Remove
>> >> >>job-28020 from job monitoring
>> >>
>> >>
>> >>
>> >
>> >--
>> >Rafael Weingärtner
>>
>>
>>
>
> --
> Rafael Weingärtner
Re: VM Snapshot not removed from primary SR
Posted by Rafael Weingärtner <ra...@gmail.com>.
Well, I am almost sure I have seen a PR fixing something like you describe,
I thought that it went into 4.11.1.0.However, only with a more in depth
debugging I would be able to confirm your problem.
Are you seeing any unexpected exception in your log files?
On Wed, Oct 10, 2018 at 11:54 AM Sami Rajala (FAPPS) <sa...@fapps.com>
wrote:
> Hi,
>
> CS running on 4.11.1.0 and does not work.
>
> BR
> -sami
>
>
> 10.10.2018 16.49, "Rafael Weingärtner" <ra...@gmail.com>:
>
> >If I am not mistaken, this has already been fixed in 4.11.1.0
> >
> >On Wed, Oct 10, 2018 at 10:31 AM Sami Rajala <sa...@fapps.com>
> >wrote:
> >
> >> Hi,
> >>
> >>
> >> I have got zero advice for this.
> >>
> >> "I have had weird issue since I updated CS from 4.9 to 4.10.
> >> CS does not delete/remove temporary snapshot from primary SR and I have
> >> had to removed it by manual time to time, before 30 snapshot limit has
> >> reached.
> >> + doesn't CS not update secondary_storege count on snapshot_count table”
> >>
> >>
> >>
> >>
> >> Now I have a situation where Xenserver has 19 snapshot on primary SR for
> >> each VM + resource count for ”secondary_storege” in database increase
> >>all
> >> the time per account.
> >> ACS storage cleaner removed old snapshots from secondary when they
> >>getting
> >> old, but thats all, not decrease resource count for ”secondary_storege”
> >> with value of deleted snapshot ( or not remove snapshot from primary SR
> >> after snapshot has created and copied to secondary )
> >>
> >> I have still delete primary snapshot by manual + give a new value for
> >> ”secondary_storege” before they reach limit.
> >>
> >> has anyone got this kind of issue? any fix
> >>
> >> BR
> >> -sami
> >>
> >>
> >>
> >> 14.9.2018 12.26, "Sami Rajala (FAPPS)" <sa...@fapps.com>:
> >>
> >> >Continue,
> >> >
> >> >I created template from VM-s latest snapshot after 3 days ( one parent
> >>+
> >> >2
> >> >child snapshot) - without errors/warning
> >> >CS create VM from this template ok, but Xen could not start VM
> >> >
> >> >so, I return back to situation where every snapshot is full and manual
> >> >removing snapshot from primary
> >> >
> >> >Has anyone any advice?
> >> >
> >> >BR
> >> >-sami
> >> >
> >> >
> >> >
> >> >13.9.2018 7.56, "Sami Rajala (FAPPS)" <sa...@fapps.com>:
> >> >
> >> >>Hello
> >> >>
> >> >>I have had weird issue since I updated CS from 4.9 to 4.10.
> >> >>CS does not delete/remove temporary snapshot from primary SR and I
> >>have
> >> >>had to removed it by manual time to time, before 30 snapshot limit has
> >> >>reached.
> >> >>+ doesn¹t CS not update secondary_storege count on snapshot_count
> >>table
> >> >>
> >> >>It works fine on 4.9, but stop to work on 4.10 and I have wait
> >> >>possibility
> >> >>to update 4.11.1.
> >> >>Update to 4.11.1 has done now and looks like this not work still
> >> >>
> >> >>I removed all snapshot and start over from clean table, snapshot
> >>policy
> >> >>is: DAILY, keep 2, delta = 5
> >> >>There are now 3 snapshot on primary SR and 3 on secondary SR
> >> >>
> >> >>The Environment is Xen 7 + CS 4.11.1 + NFS storage
> >> >>
> >> >>Is there some other parameter I should look and any other work around
> >>I
> >> >>has to do to get it work ?
> >> >>
> >> >>Any hits?
> >> >>
> >> >>BR
> >> >>-sami
> >> >>
> >> >>VM snapshot Log for last round:
> >> >>
> >> >>2018-09-13 06:02:15,983 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> >>(API-Job-Executor-83:ctx-cfb1cd95 job-28020) (logid:4c7a1c38)
> >>Executing
> >> >>AsyncJobVO {id:28020, userId: 1, accountId: 7, instanceType: Snapshot,
> >> >>instanceId: 2656, cmd:
> >> >>org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd,
> >> >>cmdInfo:
> >>
> >>>>{"policyid":"17","ctxUserId":"1","volumeid":"238","ctxStartEventId":"1"
> >>>>,"
> >> >>i
> >> >>d
> >> >>":"2656","ctxAccountId":"7"}, cmdVersion: 0, status: IN_PROGRESS,
> >> >>processStatus: 0, resultCode: 0, result: null, initMsid:
> >>19873467853209,
> >> >>completeMsid: null, lastUpdated: null, lastPolled: null, created:
> >>null}
> >> >>2018-09-13 06:02:15,991 DEBUG [c.c.u.AccountManagerImpl]
> >> >>(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6)
> >> >>(logid:4c7a1c38)
> >> >>Access to Acct[479c643e-9c84-41fb-9f0a-9bb999893a25-juha] granted to
> >> >>Acct[479c643e-9c84-41fb-9f0a-9bb999893a25-juha] by DomainChecker
> >> >>2018-09-13 06:02:16,045 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> >>(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6)
> >> >>(logid:4c7a1c38)
> >> >>Sync job-28021 execution on object VmWorkJobQueue.228
> >> >>2018-09-13 06:02:17,115 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021)
> >>(logid:4c7a1c38)
> >> >>Executing AsyncJobVO {id:28021, userId: 1, accountId: 7, instanceType:
> >> >>null, instanceId: null, cmd: com.cloud.vm.VmWorkTakeVolumeSnapshot,
> >> >>cmdInfo:
> >>
> >>>>rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrVGFrZVZvbHVtZVNuYXBzaG90BL5gG4Li1c8
> >>>>CA
> >> >>A
> >> >>Z
> >>
> >>>>aAAthc3luY0JhY2t1cFoACXF1aWVzY2VWbUwADGxvY2F0aW9uVHlwZXQAKUxjb20vY2xvdW
> >>>>Qv
> >> >>c
> >> >>3
> >>
> >>>>RvcmFnZS9TbmFwc2hvdCRMb2NhdGlvblR5cGU7TAAIcG9saWN5SWR0ABBMamF2YS9sYW5nL
> >>>>0x
> >> >>v
> >> >>b
> >>
> >>>>mc7TAAKc25hcHNob3RJZHEAfgACTAAIdm9sdW1lSWRxAH4AAnhyABNjb20uY2xvdWQudm0u
> >>>>Vm
> >> >>1
> >> >>X
> >>
> >>>>b3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlkTAALaGFuZGxlck5
> >>>>hb
> >> >>W
> >> >>V
> >>
> >>>>0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAAHAAAAAAAAAAEAAAAAAAAA5HQAFFZvbH
> >>>>Vt
> >> >>Z
> >> >>U
> >>
> >>>>FwaVNlcnZpY2VJbXBsAABwc3IADmphdmEubGFuZy5Mb25nO4vkkMyPI98CAAFKAAV2YWx1Z
> >>>>Xh
> >> >>y
> >> >>A
> >>
> >>>>BBqYXZhLmxhbmcuTnVtYmVyhqyVHQuU4IsCAAB4cAAAAAAAAAARc3EAfgAHAAAAAAAACmBz
> >>>>cQ
> >> >>B
> >> >>-
> >> >>AAcAAAAAAAAA7g, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0,
> >> >>resultCode: 0, result: null, initMsid: 19873467853209, completeMsid:
> >> >>null,
> >> >>lastUpdated: null, lastPolled: null, created: Thu Sep 13 06:02:16 EEST
> >> >>2018}
> >> >>2018-09-13 06:02:17,116 DEBUG [c.c.v.VmWorkJobDispatcher]
> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021)
> >>(logid:4c7a1c38)
> >> >>Run VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot for VM 228, job
> >> >>origin: 28020
> >> >>2018-09-13 06:02:17,117 DEBUG [c.c.v.VmWorkJobHandlerProxy]
> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
> >> >>(logid:4c7a1c38) Execute VM work job:
> >>
> >>>>com.cloud.vm.VmWorkTakeVolumeSnapshot{"volumeId":238,"policyId":17,"sna
> >>>>ps
> >> >>h
> >> >>o
> >>
> >>>>tId":2656,"quiesceVm":false,"asyncBackup":false,"userId":1,"accountId":
> >>>>7,
> >> >>"
> >> >>v
> >> >>mId":228,"handlerName":"VolumeApiServiceImpl"}
> >> >>2018-09-13 06:02:17,197 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
> >> >>(logid:4c7a1c38) getCommandHostDelegation: class
> >> >>org.apache.cloudstack.storage.command.CreateObjectCommand
> >> >>2018-09-13 06:02:17,197 DEBUG [c.c.h.XenServerGuru]
> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
> >> >>(logid:4c7a1c38) We are returning the default host to execute commands
> >> >>because the command is not of Copy type.
> >> >>2018-09-13 06:02:17,199 DEBUG [c.c.a.t.Request]
> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
> >> >>(logid:4c7a1c38) Seq 128-4630826316843724436: Sending { Cmd , MgmtId:
> >> >>19873467853209, via: 128(xenserver-15), Ver: v1, Flags: 100011,
> >>
> >>>>[{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{
> >>>>"o
> >> >>r
> >> >>g
> >>
> >>>>.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"969c
> >>>>79
> >> >>6
> >> >>d
> >>
> >>>>-072e-4272-adc6-93a492c03ea3","volumeType":"ROOT","dataStore":{"org.apa
> >>>>ch
> >> >>e
> >> >>.
> >> >>cloudstack.storage.to
> >> .PrimaryDataStoreTO":{"uuid":"6568d30f-82b6-35a0-bfe
> >> >>5
> >> >>-
> >>
> >>>>be17c1398ed8","id":2,"poolType":"NetworkFilesystem","host":"10.10.1.33"
> >>>>,"
> >> >>p
> >> >>a
> >> >>th":"/pool3/primary3","port":2049,"url":"NetworkFilesystem://
> >> 10.10.1.33/p
> >> >>o
> >> >>o
> >>
> >>>>l3/primary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a0-bfe5-be17c1398ed
> >>>>8"
> >> >>,
> >> >>"
> >>
> >>>>isManaged":false}},"name":"ROOT-228","size":21474836480,"path":"3d2a971
> >>>>6-
> >> >>8
> >> >>f
> >>
> >>>>a1-497e-8753-941117a0ef50","volumeId":238,"vmName":"i-7-228-VM","accoun
> >>>>tI
> >> >>d
> >> >>"
> >>
> >>>>:7,"format":"VHD","provisioningType":"THIN","id":238,"deviceId":0,"hype
> >>>>rv
> >> >>i
> >> >>s
> >>
> >>>>orType":"XenServer"},"parentSnapshotPath":"c7b2824a-7402-4f66-b2e6-06ed
> >>>>45
> >> >>2
> >> >>0
> >> >>72ad","dataStore":{"org.apache.cloudstack.storage.to
> >> .PrimaryDataStoreTO":
> >> >>{
> >> >>"
> >>
> >>>>uuid":"6568d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType":"Network
> >>>>Fi
> >> >>l
> >> >>e
> >>
> >>>>system","host":"10.10.1.33","path":"/pool3/primary3","port":2049,"url":
> >>>>"N
> >> >>e
> >> >>t
> >> >>workFilesystem://
> >> 10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUID=6568d3
> >> >>0
> >> >>f
> >>
> >>>>-82b6-35a0-bfe5-be17c1398ed8","isManaged":false}},"vmName":"i-7-228-VM"
> >>>>,"
> >> >>n
> >> >>a
> >>
> >>>>me":"BGI-Maria_ROOT-228_20180913030215","hypervisorType":"XenServer","i
> >>>>d"
> >> >>:
> >> >>2
> >>
> >>>>656,"quiescevm":false,"parents":["c7b2824a-7402-4f66-b2e6-06ed452072ad"
> >>>>],
> >> >>"
> >> >>p
> >> >>hysicalSize":0}},"wait":0}}] }
> >> >>2018-09-13 06:02:17,199 DEBUG [c.c.a.t.Request]
> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
> >> >>(logid:4c7a1c38) Seq 128-4630826316843724436: Executing: { Cmd ,
> >>MgmtId:
> >> >>19873467853209, via: 128(xenserver-15), Ver: v1, Flags: 100011,
> >>
> >>>>[{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{
> >>>>"o
> >> >>r
> >> >>g
> >>
> >>>>.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"969c
> >>>>79
> >> >>6
> >> >>d
> >>
> >>>>-072e-4272-adc6-93a492c03ea3","volumeType":"ROOT","dataStore":{"org.apa
> >>>>ch
> >> >>e
> >> >>.
> >> >>cloudstack.storage.to
> >> .PrimaryDataStoreTO":{"uuid":"6568d30f-82b6-35a0-bfe
> >> >>5
> >> >>-
> >>
> >>>>be17c1398ed8","id":2,"poolType":"NetworkFilesystem","host":"10.10.1.33"
> >>>>,"
> >> >>p
> >> >>a
> >> >>th":"/pool3/primary3","port":2049,"url":"NetworkFilesystem://
> >> 10.10.1.33/p
> >> >>o
> >> >>o
> >>
> >>>>l3/primary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a0-bfe5-be17c1398ed
> >>>>8"
> >> >>,
> >> >>"
> >>
> >>>>isManaged":false}},"name":"ROOT-228","size":21474836480,"path":"3d2a971
> >>>>6-
> >> >>8
> >> >>f
> >>
> >>>>a1-497e-8753-941117a0ef50","volumeId":238,"vmName":"i-7-228-VM","accoun
> >>>>tI
> >> >>d
> >> >>"
> >>
> >>>>:7,"format":"VHD","provisioningType":"THIN","id":238,"deviceId":0,"hype
> >>>>rv
> >> >>i
> >> >>s
> >>
> >>>>orType":"XenServer"},"parentSnapshotPath":"c7b2824a-7402-4f66-b2e6-06ed
> >>>>45
> >> >>2
> >> >>0
> >> >>72ad","dataStore":{"org.apache.cloudstack.storage.to
> >> .PrimaryDataStoreTO":
> >> >>{
> >> >>"
> >>
> >>>>uuid":"6568d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType":"Network
> >>>>Fi
> >> >>l
> >> >>e
> >>
> >>>>system","host":"10.10.1.33","path":"/pool3/primary3","port":2049,"url":
> >>>>"N
> >> >>e
> >> >>t
> >> >>workFilesystem://
> >> 10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUID=6568d3
> >> >>0
> >> >>f
> >>
> >>>>-82b6-35a0-bfe5-be17c1398ed8","isManaged":false}},"vmName":"i-7-228-VM"
> >>>>,"
> >> >>n
> >> >>a
> >>
> >>>>me":"BGI-Maria_ROOT-228_20180913030215","hypervisorType":"XenServer","i
> >>>>d"
> >> >>:
> >> >>2
> >>
> >>>>656,"quiescevm":false,"parents":["c7b2824a-7402-4f66-b2e6-06ed452072ad"
> >>>>],
> >> >>"
> >> >>p
> >> >>hysicalSize":0}},"wait":0}}] }
> >> >>2018-09-13 06:02:19,815 DEBUG [c.c.a.m.DirectAgentAttache]
> >> >>(DirectAgent-206:ctx-370f4cd9) (logid:4c7a1c38) Seq
> >> >>128-4630826316843724436: Response Received:
> >> >>2018-09-13 06:02:19,815 DEBUG [c.c.a.t.Request]
> >> >>(DirectAgent-206:ctx-370f4cd9) (logid:4c7a1c38) Seq
> >> >>128-4630826316843724436: Processing: { Ans: , MgmtId: 19873467853209,
> >> >>via: 128(xenserver-15), Ver: v1, Flags: 10,
> >>
> >>>>[{"org.apache.cloudstack.storage.command.CreateObjectAnswer":{"data":{"
> >>>>or
> >> >>g
> >> >>.
> >>
> >>>>apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"e91fb041-779d-4
> >>>>09
> >> >>0
> >> >>-
> >>
> >>>>b14c-4f57f4cda54b","id":0,"quiescevm":false,"physicalSize":0}},"result"
> >>>>:t
> >> >>r
> >> >>u
> >> >>e,"wait":0}}] }
> >> >>2018-09-13 06:02:19,815 DEBUG [c.c.a.t.Request]
> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
> >> >>(logid:4c7a1c38) Seq 128-4630826316843724436: Received: { Ans: ,
> >>MgmtId:
> >> >>19873467853209, via: 128(xenserver-15), Ver: v1, Flags: 10, {
> >> >>CreateObjectAnswer } }
> >> >>2018-09-13 06:02:19,934 DEBUG [o.a.c.s.m.AncientDataMotionStrategy]
> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
> >> >>(logid:4c7a1c38) copyAsync inspecting src type SNAPSHOT copyAsync
> >> >>inspecting dest type SNAPSHOT
> >> >>2018-09-13 06:02:20,017 ERROR [o.a.c.s.e.DefaultEndPointSelector]
> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
> >> >>(logid:4c7a1c38) IR24 select BACKUPSNAPSHOT from primary to secondary
> >> >>2656
> >> >>dest=2656
> >> >>2018-09-13 06:02:20,023 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
> >> >>(logid:4c7a1c38) getCommandHostDelegation: class
> >> >>org.apache.cloudstack.storage.command.CopyCommand
> >> >>2018-09-13 06:02:20,023 DEBUG [c.c.h.XenServerGuru]
> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
> >> >>(logid:4c7a1c38) We are returning the default host to execute commands
> >> >>because the source and destination objects are not NFS type.
> >> >>2018-09-13 06:02:20,025 DEBUG [c.c.a.t.Request]
> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
> >> >>(logid:4c7a1c38) Seq 130-7345089517264579763: Sending { Cmd , MgmtId:
> >> >>19873467853209, via: 130(xenserver-81), Ver: v1, Flags: 100111,
> >>
> >>>>[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.ap
> >>>>ac
> >> >>h
> >> >>e
> >> >>.cloudstack.storage.to
> >> .SnapshotObjectTO":{"path":"e91fb041-779d-4090-b14c
> >> >>-
> >> >>4
> >>
> >>>>f57f4cda54b","volume":{"uuid":"969c796d-072e-4272-adc6-93a492c03ea3","v
> >>>>ol
> >> >>u
> >> >>m
> >>
> >>>>eType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDat
> >>>>aS
> >> >>t
> >> >>o
> >>
> >>>>reTO":{"uuid":"6568d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType":
> >>>>"N
> >> >>e
> >> >>t
> >>
> >>>>workFilesystem","host":"10.10.1.33","path":"/pool3/primary3","port":204
> >>>>9,
> >> >>"
> >> >>u
> >> >>rl":"NetworkFilesystem://
> >> 10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUI
> >> >>D
> >> >>=
> >>
> >>>>6568d30f-82b6-35a0-bfe5-be17c1398ed8","isManaged":false}},"name":"ROOT-
> >>>>22
> >> >>8
> >> >>"
> >>
> >>>>,"size":21474836480,"path":"3d2a9716-8fa1-497e-8753-941117a0ef50","volu
> >>>>me
> >> >>I
> >> >>d
> >>
> >>>>":238,"vmName":"i-7-228-VM","accountId":7,"format":"VHD","provisioningT
> >>>>yp
> >> >>e
> >> >>"
> >>
> >>>>:"THIN","id":238,"deviceId":0,"hypervisorType":"XenServer"},"parentSnap
> >>>>sh
> >> >>o
> >> >>t
> >>
> >>>>Path":"c7b2824a-7402-4f66-b2e6-06ed452072ad","dataStore":{"org.apache.c
> >>>>lo
> >> >>u
> >> >>d
> >> >>stack.storage.to
> >> .PrimaryDataStoreTO":{"uuid":"6568d30f-82b6-35a0-bfe5-be1
> >> >>7
> >> >>c
> >>
> >>>>1398ed8","id":2,"poolType":"NetworkFilesystem","host":"10.10.1.33","pat
> >>>>h"
> >> >>:
> >> >>"
> >> >>/pool3/primary3","port":2049,"url":"NetworkFilesystem://
> >> 10.10.1.33/pool3/
> >> >>p
> >> >>r
> >>
> >>>>imary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a0-bfe5-be17c1398ed8","i
> >>>>sM
> >> >>a
> >> >>n
> >>
> >>>>aged":false}},"vmName":"i-7-228-VM","name":"BGI-Maria_ROOT-228_20180913
> >>>>03
> >> >>0
> >> >>2
> >>
> >>>>15","hypervisorType":"XenServer","id":2656,"quiescevm":false,"parents":
> >>>>["
> >> >>c
> >> >>7
> >>
> >>>>b2824a-7402-4f66-b2e6-06ed452072ad"],"physicalSize":0}},"destTO":{"org.
> >>>>ap
> >> >>a
> >> >>c
> >> >>he.cloudstack.storage.to
> >> .SnapshotObjectTO":{"path":"snapshots/7/238","vol
> >> >>u
> >> >>m
> >>
> >>>>e":{"uuid":"969c796d-072e-4272-adc6-93a492c03ea3","volumeType":"ROOT","
> >>>>da
> >> >>t
> >> >>a
> >> >>Store":{"org.apache.cloudstack.storage.to
> >> .PrimaryDataStoreTO":{"uuid":"65
> >> >>6
> >> >>8
> >>
> >>>>d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType":"NetworkFilesystem"
> >>>>,"
> >> >>h
> >> >>o
> >>
> >>>>st":"10.10.1.33","path":"/pool3/primary3","port":2049,"url":"NetworkFil
> >>>>es
> >> >>y
> >> >>s
> >> >>tem://
> >> 10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a
> >> >>0
> >> >>-
> >>
> >>>>bfe5-be17c1398ed8","isManaged":false}},"name":"ROOT-228","size":2147483
> >>>>64
> >> >>8
> >> >>0
> >>
> >>>>,"path":"3d2a9716-8fa1-497e-8753-941117a0ef50","volumeId":238,"vmName":
> >>>>"i
> >> >>-
> >> >>7
> >>
> >>>>-228-VM","accountId":7,"format":"VHD","provisioningType":"THIN","id":23
> >>>>8,
> >> >>"
> >> >>d
> >>
> >>>>eviceId":0,"hypervisorType":"XenServer"},"parentSnapshotPath":"snapshot
> >>>>s/
> >> >>7
> >> >>/
> >>
> >>>>238/0c8c6ee3-ce3d-4241-8509-5193d2cff491.vhd","dataStore":{"com.cloud.a
> >>>>ge
> >> >>n
> >> >>t
> >> >>.api.to.NfsTO":{"_url":"nfs://10.10.1.34/pool8/secondary8
> >> ","_role":"Image
> >> >>"
> >> >>}
> >>
> >>>>},"vmName":"i-7-228-VM","name":"BGI-Maria_ROOT-228_20180913030215","hyp
> >>>>er
> >> >>v
> >> >>i
> >>
> >>>>sorType":"XenServer","id":2656,"quiescevm":false,"parents":["snapshots/
> >>>>7/
> >> >>2
> >> >>3
> >>
> >>>>8/93a42896-0da1-4b82-8985-d28464f1ac4a.vhd","snapshots/7/238/0c8c6ee3-c
> >>>>e3
> >> >>d
> >> >>-
> >>
> >>>>4241-8509-5193d2cff491.vhd"],"physicalSize":0}},"executeInSequence":tru
> >>>>e,
> >> >>"
> >> >>o
> >> >>ptions":{"fullSnapshot":"false"},"options2":{},"wait":28800}}] }
> >> >>2018-09-13 06:02:20,025 DEBUG [c.c.a.t.Request]
> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
> >> >>(logid:4c7a1c38) Seq 130-7345089517264579763: Executing: { Cmd ,
> >>MgmtId:
> >> >>19873467853209, via: 130(xenserver-81), Ver: v1, Flags: 100111,
> >>
> >>>>[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.ap
> >>>>ac
> >> >>h
> >> >>e
> >> >>.cloudstack.storage.to
> >> .SnapshotObjectTO":{"path":"e91fb041-779d-4090-b14c
> >> >>-
> >> >>4
> >>
> >>>>f57f4cda54b","volume":{"uuid":"969c796d-072e-4272-adc6-93a492c03ea3","v
> >>>>ol
> >> >>u
> >> >>m
> >>
> >>>>eType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDat
> >>>>aS
> >> >>t
> >> >>o
> >>
> >>>>reTO":{"uuid":"6568d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType":
> >>>>"N
> >> >>e
> >> >>t
> >>
> >>>>workFilesystem","host":"10.10.1.33","path":"/pool3/primary3","port":204
> >>>>9,
> >> >>"
> >> >>u
> >> >>rl":"NetworkFilesystem://
> >> 10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUI
> >> >>D
> >> >>=
> >>
> >>>>6568d30f-82b6-35a0-bfe5-be17c1398ed8","isManaged":false}},"name":"ROOT-
> >>>>22
> >> >>8
> >> >>"
> >>
> >>>>,"size":21474836480,"path":"3d2a9716-8fa1-497e-8753-941117a0ef50","volu
> >>>>me
> >> >>I
> >> >>d
> >>
> >>>>":238,"vmName":"i-7-228-VM","accountId":7,"format":"VHD","provisioningT
> >>>>yp
> >> >>e
> >> >>"
> >>
> >>>>:"THIN","id":238,"deviceId":0,"hypervisorType":"XenServer"},"parentSnap
> >>>>sh
> >> >>o
> >> >>t
> >>
> >>>>Path":"c7b2824a-7402-4f66-b2e6-06ed452072ad","dataStore":{"org.apache.c
> >>>>lo
> >> >>u
> >> >>d
> >> >>stack.storage.to
> >> .PrimaryDataStoreTO":{"uuid":"6568d30f-82b6-35a0-bfe5-be1
> >> >>7
> >> >>c
> >>
> >>>>1398ed8","id":2,"poolType":"NetworkFilesystem","host":"10.10.1.33","pat
> >>>>h"
> >> >>:
> >> >>"
> >> >>/pool3/primary3","port":2049,"url":"NetworkFilesystem://
> >> 10.10.1.33/pool3/
> >> >>p
> >> >>r
> >>
> >>>>imary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a0-bfe5-be17c1398ed8","i
> >>>>sM
> >> >>a
> >> >>n
> >>
> >>>>aged":false}},"vmName":"i-7-228-VM","name":"BGI-Maria_ROOT-228_20180913
> >>>>03
> >> >>0
> >> >>2
> >>
> >>>>15","hypervisorType":"XenServer","id":2656,"quiescevm":false,"parents":
> >>>>["
> >> >>c
> >> >>7
> >>
> >>>>b2824a-7402-4f66-b2e6-06ed452072ad"],"physicalSize":0}},"destTO":{"org.
> >>>>ap
> >> >>a
> >> >>c
> >> >>he.cloudstack.storage.to
> >> .SnapshotObjectTO":{"path":"snapshots/7/238","vol
> >> >>u
> >> >>m
> >>
> >>>>e":{"uuid":"969c796d-072e-4272-adc6-93a492c03ea3","volumeType":"ROOT","
> >>>>da
> >> >>t
> >> >>a
> >> >>Store":{"org.apache.cloudstack.storage.to
> >> .PrimaryDataStoreTO":{"uuid":"65
> >> >>6
> >> >>8
> >>
> >>>>d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType":"NetworkFilesystem"
> >>>>,"
> >> >>h
> >> >>o
> >>
> >>>>st":"10.10.1.33","path":"/pool3/primary3","port":2049,"url":"NetworkFil
> >>>>es
> >> >>y
> >> >>s
> >> >>tem://
> >> 10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a
> >> >>0
> >> >>-
> >>
> >>>>bfe5-be17c1398ed8","isManaged":false}},"name":"ROOT-228","size":2147483
> >>>>64
> >> >>8
> >> >>0
> >>
> >>>>,"path":"3d2a9716-8fa1-497e-8753-941117a0ef50","volumeId":238,"vmName":
> >>>>"i
> >> >>-
> >> >>7
> >>
> >>>>-228-VM","accountId":7,"format":"VHD","provisioningType":"THIN","id":23
> >>>>8,
> >> >>"
> >> >>d
> >>
> >>>>eviceId":0,"hypervisorType":"XenServer"},"parentSnapshotPath":"snapshot
> >>>>s/
> >> >>7
> >> >>/
> >>
> >>>>238/0c8c6ee3-ce3d-4241-8509-5193d2cff491.vhd","dataStore":{"com.cloud.a
> >>>>ge
> >> >>n
> >> >>t
> >> >>.api.to.NfsTO":{"_url":"nfs://10.10.1.34/pool8/secondary8
> >> ","_role":"Image
> >> >>"
> >> >>}
> >>
> >>>>},"vmName":"i-7-228-VM","name":"BGI-Maria_ROOT-228_20180913030215","hyp
> >>>>er
> >> >>v
> >> >>i
> >>
> >>>>sorType":"XenServer","id":2656,"quiescevm":false,"parents":["snapshots/
> >>>>7/
> >> >>2
> >> >>3
> >>
> >>>>8/93a42896-0da1-4b82-8985-d28464f1ac4a.vhd","snapshots/7/238/0c8c6ee3-c
> >>>>e3
> >> >>d
> >> >>-
> >>
> >>>>4241-8509-5193d2cff491.vhd"],"physicalSize":0}},"executeInSequence":tru
> >>>>e,
> >> >>"
> >> >>o
> >> >>ptions":{"fullSnapshot":"false"},"options2":{},"wait":28800}}] }
> >> >>2018-09-13 06:04:28,895 DEBUG [c.c.h.x.r.CitrixResourceBase]
> >> >>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Host 10.10.1.81
> >> >>OpaqueRef:f83331df-6634-63fd-6b95-44607a6069b8: Removing SR
> >> >>2018-09-13 06:04:28,913 DEBUG [c.c.h.x.r.CitrixResourceBase]
> >> >>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Host 10.10.1.81
> >> >>OpaqueRef:05380b03-4cb8-85a7-6a6a-a1595c188f60: Unplugging pbd
> >> >>2018-09-13 06:04:30,881 DEBUG [c.c.h.x.r.CitrixResourceBase]
> >> >>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Host 10.10.1.81
> >> >>OpaqueRef:f83331df-6634-63fd-6b95-44607a6069b8: Forgetting
> >> >>2018-09-13 06:04:30,891 DEBUG [c.c.h.x.r.XenServerStorageProcessor]
> >> >>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Successfully destroyed
> >> >>snapshot on volume: 3d2a9716-8fa1-497e-8753-941117a0ef50 execept this
> >> >>current snapshot e91fb041-779d-4090-b14c-4f57f4cda54b
> >> >>2018-09-13 06:04:30,891 INFO [c.c.h.x.r.XenServerStorageProcessor]
> >> >>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) New snapshot details:
> >>
> >>>>SnapshotTO[datastore=null|volume=null|pathsnapshots/7/238/0560d536-c945
> >>>>-4
> >> >>a
> >> >>6
> >> >>1-8c4c-dbaf29f29993.vhd]
> >> >>2018-09-13 06:04:30,891 INFO [c.c.h.x.r.XenServerStorageProcessor]
> >> >>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) New snapshot physical
> >> >>utilization: 2342932992
> >> >>2018-09-13 06:04:30,891 DEBUG [c.c.a.m.DirectAgentAttache]
> >> >>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Seq
> >> >>130-7345089517264579763: Response Received:
> >> >>2018-09-13 06:04:30,891 DEBUG [c.c.a.t.Request]
> >> >>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Seq
> >> >>130-7345089517264579763: Processing: { Ans: , MgmtId: 19873467853209,
> >> >>via: 130(xenserver-81), Ver: v1, Flags: 110,
> >>
> >>>>[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"or
> >>>>g.
> >> >>a
> >> >>p
> >> >>ache.cloudstack.storage.to
> >> .SnapshotObjectTO":{"path":"snapshots/7/238/056
> >> >>0
> >> >>d
> >>
> >>>>536-c945-4a61-8c4c-dbaf29f29993.vhd","parentSnapshotPath":"snapshots/7/
> >>>>23
> >> >>8
> >> >>/
> >>
> >>>>0c8c6ee3-ce3d-4241-8509-5193d2cff491.vhd","id":0,"quiescevm":false,"phy
> >>>>si
> >> >>c
> >> >>a
> >> >>lSize":2342932992}},"result":true,"wait":0}}] }
> >> >>2018-09-13 06:04:30,891 DEBUG [c.c.a.m.AgentAttache]
> >> >>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Seq
> >> >>130-7345089517264579763: No more commands found
> >> >>2018-09-13 06:04:30,891 DEBUG [c.c.a.t.Request]
> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
> >> >>(logid:4c7a1c38) Seq 130-7345089517264579763: Received: { Ans: ,
> >>MgmtId:
> >> >>19873467853209, via: 130(xenserver-81), Ver: v1, Flags: 110, {
> >> >>CopyCmdAnswer } }
> >> >>2018-09-13 06:04:30,957 DEBUG [c.c.r.ResourceLimitManagerImpl]
> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
> >> >>(logid:4c7a1c38) Updating resource Type = secondary_storage count for
> >> >>Account = 7 Operation = decreasing Amount = 19131903488
> >> >>2018-09-13 06:04:30,969 DEBUG [c.c.v.VmWorkJobHandlerProxy]
> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
> >> >>(logid:4c7a1c38) Done executing VM work job:
> >>
> >>>>com.cloud.vm.VmWorkTakeVolumeSnapshot{"volumeId":238,"policyId":17,"sna
> >>>>ps
> >> >>h
> >> >>o
> >>
> >>>>tId":2656,"quiesceVm":false,"asyncBackup":false,"userId":1,"accountId":
> >>>>7,
> >> >>"
> >> >>v
> >> >>mId":228,"handlerName":"VolumeApiServiceImpl"}
> >> >>2018-09-13 06:04:30,969 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
> >> >>(logid:4c7a1c38) Complete async job-28021, jobStatus: SUCCEEDED,
> >> >>resultCode: 0, result:
> >>
> >>>>rO0ABXNyAA5qYXZhLmxhbmcuTG9uZzuL5JDMjyPfAgABSgAFdmFsdWV4cgAQamF2YS5sYW5
> >>>>nL
> >> >>k
> >> >>5
> >> >>1bWJlcoaslR0LlOCLAgAAeHAAAAAAAAAKYA
> >> >>2018-09-13 06:04:30,970 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
> >> >>(logid:4c7a1c38) Publish async job-28021 complete on message bus
> >> >>2018-09-13 06:04:30,970 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
> >> >>(logid:4c7a1c38) Wake up jobs related to job-28021
> >> >>2018-09-13 06:04:30,970 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
> >> >>(logid:4c7a1c38) Update db status for job-28021
> >> >>2018-09-13 06:04:30,971 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
> >> >>(logid:4c7a1c38) Wake up jobs joined with job-28021 and disjoin all
> >> >>subjobs created from job- 28021
> >> >>2018-09-13 06:04:30,983 DEBUG [c.c.v.VmWorkJobDispatcher]
> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021)
> >>(logid:4c7a1c38)
> >> >>Done with run of VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot
> >>for
> >> >>VM
> >> >>228, job origin: 28020
> >> >>2018-09-13 06:04:30,983 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021)
> >>(logid:4c7a1c38)
> >> >>Done executing com.cloud.vm.VmWorkTakeVolumeSnapshot for job-28021
> >> >>2018-09-13 06:04:30,984 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021)
> >>(logid:4c7a1c38)
> >> >>Remove job-28021 from job monitoring
> >> >>2018-09-13 06:04:31,026 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> >>(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6)
> >> >>(logid:4c7a1c38)
> >> >>Complete async job-28020, jobStatus: SUCCEEDED, resultCode: 0, result:
> >>
> >>>>org.apache.cloudstack.api.response.SnapshotResponse/snapshot/{"id":"3c8
> >>>>66
> >> >>2
> >> >>b
> >>
> >>>>2-c76f-4e3a-a928-f76dad820122","account":"juha","domainid":"d3c2420f-75
> >>>>03
> >> >>-
> >> >>4
> >>
> >>>>560-a2da-d7e7149ce390","domain":"BGI","snapshottype":"DAILY","volumeid"
> >>>>:"
> >> >>9
> >> >>6
> >>
> >>>>9c796d-072e-4272-adc6-93a492c03ea3","volumename":"ROOT-228","volumetype
> >>>>":
> >> >>"
> >> >>R
> >>
> >>>>OOT","created":"2018-09-13T06:02:15+0300","name":"BGI-Maria_ROOT-228_20
> >>>>18
> >> >>0
> >> >>9
> >>
> >>>>13030215","intervaltype":"DAILY","state":"BackedUp","physicalsize":2342
> >>>>93
> >> >>2
> >> >>9
> >>
> >>>>92,"zoneid":"533c8f88-2637-4c0e-8210-e0a355b62d82","tags":[],"revertabl
> >>>>e"
> >> >>:
> >> >>f
> >>
> >>>>alse,"ostypeid":"da491b96-3748-11e5-a683-121326fde999","osdisplayname":
> >>>>"C
> >> >>e
> >> >>n
> >> >>tOS 6.5 (64-bit)","virtualsize":21474836480}
> >> >>2018-09-13 06:04:31,027 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> >>(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6)
> >> >>(logid:4c7a1c38)
> >> >>Publish async job-28020 complete on message bus
> >> >>2018-09-13 06:04:31,027 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> >>(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6)
> >> >>(logid:4c7a1c38)
> >> >>Wake up jobs related to job-28020
> >> >>2018-09-13 06:04:31,027 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> >>(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6)
> >> >>(logid:4c7a1c38)
> >> >>Update db status for job-28020
> >> >>2018-09-13 06:04:31,028 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> >>(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6)
> >> >>(logid:4c7a1c38)
> >> >>Wake up jobs joined with job-28020 and disjoin all subjobs created
> >>from
> >> >>job- 28020
> >> >>2018-09-13 06:04:31,039 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> >>(API-Job-Executor-83:ctx-cfb1cd95 job-28020) (logid:4c7a1c38) Done
> >> >>executing
> >> >>org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for
> >> >>job-28020
> >> >>2018-09-13 06:04:31,039 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> >> >>(API-Job-Executor-83:ctx-cfb1cd95 job-28020) (logid:4c7a1c38) Remove
> >> >>job-28020 from job monitoring
> >>
> >>
> >>
> >
> >--
> >Rafael Weingärtner
>
>
>
--
Rafael Weingärtner
Re: VM Snapshot not removed from primary SR
Posted by "Sami Rajala (FAPPS)" <sa...@fapps.com>.
Hi,
CS running on 4.11.1.0 and does not work.
BR
-sami
10.10.2018 16.49, "Rafael Weingärtner" <ra...@gmail.com>:
>If I am not mistaken, this has already been fixed in 4.11.1.0
>
>On Wed, Oct 10, 2018 at 10:31 AM Sami Rajala <sa...@fapps.com>
>wrote:
>
>> Hi,
>>
>>
>> I have got zero advice for this.
>>
>> "I have had weird issue since I updated CS from 4.9 to 4.10.
>> CS does not delete/remove temporary snapshot from primary SR and I have
>> had to removed it by manual time to time, before 30 snapshot limit has
>> reached.
>> + doesn't CS not update secondary_storege count on snapshot_count table”
>>
>>
>>
>>
>> Now I have a situation where Xenserver has 19 snapshot on primary SR for
>> each VM + resource count for ”secondary_storege” in database increase
>>all
>> the time per account.
>> ACS storage cleaner removed old snapshots from secondary when they
>>getting
>> old, but thats all, not decrease resource count for ”secondary_storege”
>> with value of deleted snapshot ( or not remove snapshot from primary SR
>> after snapshot has created and copied to secondary )
>>
>> I have still delete primary snapshot by manual + give a new value for
>> ”secondary_storege” before they reach limit.
>>
>> has anyone got this kind of issue? any fix
>>
>> BR
>> -sami
>>
>>
>>
>> 14.9.2018 12.26, "Sami Rajala (FAPPS)" <sa...@fapps.com>:
>>
>> >Continue,
>> >
>> >I created template from VM-s latest snapshot after 3 days ( one parent
>>+
>> >2
>> >child snapshot) - without errors/warning
>> >CS create VM from this template ok, but Xen could not start VM
>> >
>> >so, I return back to situation where every snapshot is full and manual
>> >removing snapshot from primary
>> >
>> >Has anyone any advice?
>> >
>> >BR
>> >-sami
>> >
>> >
>> >
>> >13.9.2018 7.56, "Sami Rajala (FAPPS)" <sa...@fapps.com>:
>> >
>> >>Hello
>> >>
>> >>I have had weird issue since I updated CS from 4.9 to 4.10.
>> >>CS does not delete/remove temporary snapshot from primary SR and I
>>have
>> >>had to removed it by manual time to time, before 30 snapshot limit has
>> >>reached.
>> >>+ doesn¹t CS not update secondary_storege count on snapshot_count
>>table
>> >>
>> >>It works fine on 4.9, but stop to work on 4.10 and I have wait
>> >>possibility
>> >>to update 4.11.1.
>> >>Update to 4.11.1 has done now and looks like this not work still
>> >>
>> >>I removed all snapshot and start over from clean table, snapshot
>>policy
>> >>is: DAILY, keep 2, delta = 5
>> >>There are now 3 snapshot on primary SR and 3 on secondary SR
>> >>
>> >>The Environment is Xen 7 + CS 4.11.1 + NFS storage
>> >>
>> >>Is there some other parameter I should look and any other work around
>>I
>> >>has to do to get it work ?
>> >>
>> >>Any hits?
>> >>
>> >>BR
>> >>-sami
>> >>
>> >>VM snapshot Log for last round:
>> >>
>> >>2018-09-13 06:02:15,983 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> >>(API-Job-Executor-83:ctx-cfb1cd95 job-28020) (logid:4c7a1c38)
>>Executing
>> >>AsyncJobVO {id:28020, userId: 1, accountId: 7, instanceType: Snapshot,
>> >>instanceId: 2656, cmd:
>> >>org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd,
>> >>cmdInfo:
>>
>>>>{"policyid":"17","ctxUserId":"1","volumeid":"238","ctxStartEventId":"1"
>>>>,"
>> >>i
>> >>d
>> >>":"2656","ctxAccountId":"7"}, cmdVersion: 0, status: IN_PROGRESS,
>> >>processStatus: 0, resultCode: 0, result: null, initMsid:
>>19873467853209,
>> >>completeMsid: null, lastUpdated: null, lastPolled: null, created:
>>null}
>> >>2018-09-13 06:02:15,991 DEBUG [c.c.u.AccountManagerImpl]
>> >>(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6)
>> >>(logid:4c7a1c38)
>> >>Access to Acct[479c643e-9c84-41fb-9f0a-9bb999893a25-juha] granted to
>> >>Acct[479c643e-9c84-41fb-9f0a-9bb999893a25-juha] by DomainChecker
>> >>2018-09-13 06:02:16,045 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> >>(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6)
>> >>(logid:4c7a1c38)
>> >>Sync job-28021 execution on object VmWorkJobQueue.228
>> >>2018-09-13 06:02:17,115 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021)
>>(logid:4c7a1c38)
>> >>Executing AsyncJobVO {id:28021, userId: 1, accountId: 7, instanceType:
>> >>null, instanceId: null, cmd: com.cloud.vm.VmWorkTakeVolumeSnapshot,
>> >>cmdInfo:
>>
>>>>rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrVGFrZVZvbHVtZVNuYXBzaG90BL5gG4Li1c8
>>>>CA
>> >>A
>> >>Z
>>
>>>>aAAthc3luY0JhY2t1cFoACXF1aWVzY2VWbUwADGxvY2F0aW9uVHlwZXQAKUxjb20vY2xvdW
>>>>Qv
>> >>c
>> >>3
>>
>>>>RvcmFnZS9TbmFwc2hvdCRMb2NhdGlvblR5cGU7TAAIcG9saWN5SWR0ABBMamF2YS9sYW5nL
>>>>0x
>> >>v
>> >>b
>>
>>>>mc7TAAKc25hcHNob3RJZHEAfgACTAAIdm9sdW1lSWRxAH4AAnhyABNjb20uY2xvdWQudm0u
>>>>Vm
>> >>1
>> >>X
>>
>>>>b3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlkTAALaGFuZGxlck5
>>>>hb
>> >>W
>> >>V
>>
>>>>0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAAHAAAAAAAAAAEAAAAAAAAA5HQAFFZvbH
>>>>Vt
>> >>Z
>> >>U
>>
>>>>FwaVNlcnZpY2VJbXBsAABwc3IADmphdmEubGFuZy5Mb25nO4vkkMyPI98CAAFKAAV2YWx1Z
>>>>Xh
>> >>y
>> >>A
>>
>>>>BBqYXZhLmxhbmcuTnVtYmVyhqyVHQuU4IsCAAB4cAAAAAAAAAARc3EAfgAHAAAAAAAACmBz
>>>>cQ
>> >>B
>> >>-
>> >>AAcAAAAAAAAA7g, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0,
>> >>resultCode: 0, result: null, initMsid: 19873467853209, completeMsid:
>> >>null,
>> >>lastUpdated: null, lastPolled: null, created: Thu Sep 13 06:02:16 EEST
>> >>2018}
>> >>2018-09-13 06:02:17,116 DEBUG [c.c.v.VmWorkJobDispatcher]
>> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021)
>>(logid:4c7a1c38)
>> >>Run VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot for VM 228, job
>> >>origin: 28020
>> >>2018-09-13 06:02:17,117 DEBUG [c.c.v.VmWorkJobHandlerProxy]
>> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>> >>(logid:4c7a1c38) Execute VM work job:
>>
>>>>com.cloud.vm.VmWorkTakeVolumeSnapshot{"volumeId":238,"policyId":17,"sna
>>>>ps
>> >>h
>> >>o
>>
>>>>tId":2656,"quiesceVm":false,"asyncBackup":false,"userId":1,"accountId":
>>>>7,
>> >>"
>> >>v
>> >>mId":228,"handlerName":"VolumeApiServiceImpl"}
>> >>2018-09-13 06:02:17,197 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
>> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>> >>(logid:4c7a1c38) getCommandHostDelegation: class
>> >>org.apache.cloudstack.storage.command.CreateObjectCommand
>> >>2018-09-13 06:02:17,197 DEBUG [c.c.h.XenServerGuru]
>> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>> >>(logid:4c7a1c38) We are returning the default host to execute commands
>> >>because the command is not of Copy type.
>> >>2018-09-13 06:02:17,199 DEBUG [c.c.a.t.Request]
>> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>> >>(logid:4c7a1c38) Seq 128-4630826316843724436: Sending { Cmd , MgmtId:
>> >>19873467853209, via: 128(xenserver-15), Ver: v1, Flags: 100011,
>>
>>>>[{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{
>>>>"o
>> >>r
>> >>g
>>
>>>>.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"969c
>>>>79
>> >>6
>> >>d
>>
>>>>-072e-4272-adc6-93a492c03ea3","volumeType":"ROOT","dataStore":{"org.apa
>>>>ch
>> >>e
>> >>.
>> >>cloudstack.storage.to
>> .PrimaryDataStoreTO":{"uuid":"6568d30f-82b6-35a0-bfe
>> >>5
>> >>-
>>
>>>>be17c1398ed8","id":2,"poolType":"NetworkFilesystem","host":"10.10.1.33"
>>>>,"
>> >>p
>> >>a
>> >>th":"/pool3/primary3","port":2049,"url":"NetworkFilesystem://
>> 10.10.1.33/p
>> >>o
>> >>o
>>
>>>>l3/primary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a0-bfe5-be17c1398ed
>>>>8"
>> >>,
>> >>"
>>
>>>>isManaged":false}},"name":"ROOT-228","size":21474836480,"path":"3d2a971
>>>>6-
>> >>8
>> >>f
>>
>>>>a1-497e-8753-941117a0ef50","volumeId":238,"vmName":"i-7-228-VM","accoun
>>>>tI
>> >>d
>> >>"
>>
>>>>:7,"format":"VHD","provisioningType":"THIN","id":238,"deviceId":0,"hype
>>>>rv
>> >>i
>> >>s
>>
>>>>orType":"XenServer"},"parentSnapshotPath":"c7b2824a-7402-4f66-b2e6-06ed
>>>>45
>> >>2
>> >>0
>> >>72ad","dataStore":{"org.apache.cloudstack.storage.to
>> .PrimaryDataStoreTO":
>> >>{
>> >>"
>>
>>>>uuid":"6568d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType":"Network
>>>>Fi
>> >>l
>> >>e
>>
>>>>system","host":"10.10.1.33","path":"/pool3/primary3","port":2049,"url":
>>>>"N
>> >>e
>> >>t
>> >>workFilesystem://
>> 10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUID=6568d3
>> >>0
>> >>f
>>
>>>>-82b6-35a0-bfe5-be17c1398ed8","isManaged":false}},"vmName":"i-7-228-VM"
>>>>,"
>> >>n
>> >>a
>>
>>>>me":"BGI-Maria_ROOT-228_20180913030215","hypervisorType":"XenServer","i
>>>>d"
>> >>:
>> >>2
>>
>>>>656,"quiescevm":false,"parents":["c7b2824a-7402-4f66-b2e6-06ed452072ad"
>>>>],
>> >>"
>> >>p
>> >>hysicalSize":0}},"wait":0}}] }
>> >>2018-09-13 06:02:17,199 DEBUG [c.c.a.t.Request]
>> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>> >>(logid:4c7a1c38) Seq 128-4630826316843724436: Executing: { Cmd ,
>>MgmtId:
>> >>19873467853209, via: 128(xenserver-15), Ver: v1, Flags: 100011,
>>
>>>>[{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{
>>>>"o
>> >>r
>> >>g
>>
>>>>.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"969c
>>>>79
>> >>6
>> >>d
>>
>>>>-072e-4272-adc6-93a492c03ea3","volumeType":"ROOT","dataStore":{"org.apa
>>>>ch
>> >>e
>> >>.
>> >>cloudstack.storage.to
>> .PrimaryDataStoreTO":{"uuid":"6568d30f-82b6-35a0-bfe
>> >>5
>> >>-
>>
>>>>be17c1398ed8","id":2,"poolType":"NetworkFilesystem","host":"10.10.1.33"
>>>>,"
>> >>p
>> >>a
>> >>th":"/pool3/primary3","port":2049,"url":"NetworkFilesystem://
>> 10.10.1.33/p
>> >>o
>> >>o
>>
>>>>l3/primary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a0-bfe5-be17c1398ed
>>>>8"
>> >>,
>> >>"
>>
>>>>isManaged":false}},"name":"ROOT-228","size":21474836480,"path":"3d2a971
>>>>6-
>> >>8
>> >>f
>>
>>>>a1-497e-8753-941117a0ef50","volumeId":238,"vmName":"i-7-228-VM","accoun
>>>>tI
>> >>d
>> >>"
>>
>>>>:7,"format":"VHD","provisioningType":"THIN","id":238,"deviceId":0,"hype
>>>>rv
>> >>i
>> >>s
>>
>>>>orType":"XenServer"},"parentSnapshotPath":"c7b2824a-7402-4f66-b2e6-06ed
>>>>45
>> >>2
>> >>0
>> >>72ad","dataStore":{"org.apache.cloudstack.storage.to
>> .PrimaryDataStoreTO":
>> >>{
>> >>"
>>
>>>>uuid":"6568d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType":"Network
>>>>Fi
>> >>l
>> >>e
>>
>>>>system","host":"10.10.1.33","path":"/pool3/primary3","port":2049,"url":
>>>>"N
>> >>e
>> >>t
>> >>workFilesystem://
>> 10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUID=6568d3
>> >>0
>> >>f
>>
>>>>-82b6-35a0-bfe5-be17c1398ed8","isManaged":false}},"vmName":"i-7-228-VM"
>>>>,"
>> >>n
>> >>a
>>
>>>>me":"BGI-Maria_ROOT-228_20180913030215","hypervisorType":"XenServer","i
>>>>d"
>> >>:
>> >>2
>>
>>>>656,"quiescevm":false,"parents":["c7b2824a-7402-4f66-b2e6-06ed452072ad"
>>>>],
>> >>"
>> >>p
>> >>hysicalSize":0}},"wait":0}}] }
>> >>2018-09-13 06:02:19,815 DEBUG [c.c.a.m.DirectAgentAttache]
>> >>(DirectAgent-206:ctx-370f4cd9) (logid:4c7a1c38) Seq
>> >>128-4630826316843724436: Response Received:
>> >>2018-09-13 06:02:19,815 DEBUG [c.c.a.t.Request]
>> >>(DirectAgent-206:ctx-370f4cd9) (logid:4c7a1c38) Seq
>> >>128-4630826316843724436: Processing: { Ans: , MgmtId: 19873467853209,
>> >>via: 128(xenserver-15), Ver: v1, Flags: 10,
>>
>>>>[{"org.apache.cloudstack.storage.command.CreateObjectAnswer":{"data":{"
>>>>or
>> >>g
>> >>.
>>
>>>>apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"e91fb041-779d-4
>>>>09
>> >>0
>> >>-
>>
>>>>b14c-4f57f4cda54b","id":0,"quiescevm":false,"physicalSize":0}},"result"
>>>>:t
>> >>r
>> >>u
>> >>e,"wait":0}}] }
>> >>2018-09-13 06:02:19,815 DEBUG [c.c.a.t.Request]
>> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>> >>(logid:4c7a1c38) Seq 128-4630826316843724436: Received: { Ans: ,
>>MgmtId:
>> >>19873467853209, via: 128(xenserver-15), Ver: v1, Flags: 10, {
>> >>CreateObjectAnswer } }
>> >>2018-09-13 06:02:19,934 DEBUG [o.a.c.s.m.AncientDataMotionStrategy]
>> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>> >>(logid:4c7a1c38) copyAsync inspecting src type SNAPSHOT copyAsync
>> >>inspecting dest type SNAPSHOT
>> >>2018-09-13 06:02:20,017 ERROR [o.a.c.s.e.DefaultEndPointSelector]
>> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>> >>(logid:4c7a1c38) IR24 select BACKUPSNAPSHOT from primary to secondary
>> >>2656
>> >>dest=2656
>> >>2018-09-13 06:02:20,023 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
>> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>> >>(logid:4c7a1c38) getCommandHostDelegation: class
>> >>org.apache.cloudstack.storage.command.CopyCommand
>> >>2018-09-13 06:02:20,023 DEBUG [c.c.h.XenServerGuru]
>> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>> >>(logid:4c7a1c38) We are returning the default host to execute commands
>> >>because the source and destination objects are not NFS type.
>> >>2018-09-13 06:02:20,025 DEBUG [c.c.a.t.Request]
>> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>> >>(logid:4c7a1c38) Seq 130-7345089517264579763: Sending { Cmd , MgmtId:
>> >>19873467853209, via: 130(xenserver-81), Ver: v1, Flags: 100111,
>>
>>>>[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.ap
>>>>ac
>> >>h
>> >>e
>> >>.cloudstack.storage.to
>> .SnapshotObjectTO":{"path":"e91fb041-779d-4090-b14c
>> >>-
>> >>4
>>
>>>>f57f4cda54b","volume":{"uuid":"969c796d-072e-4272-adc6-93a492c03ea3","v
>>>>ol
>> >>u
>> >>m
>>
>>>>eType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDat
>>>>aS
>> >>t
>> >>o
>>
>>>>reTO":{"uuid":"6568d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType":
>>>>"N
>> >>e
>> >>t
>>
>>>>workFilesystem","host":"10.10.1.33","path":"/pool3/primary3","port":204
>>>>9,
>> >>"
>> >>u
>> >>rl":"NetworkFilesystem://
>> 10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUI
>> >>D
>> >>=
>>
>>>>6568d30f-82b6-35a0-bfe5-be17c1398ed8","isManaged":false}},"name":"ROOT-
>>>>22
>> >>8
>> >>"
>>
>>>>,"size":21474836480,"path":"3d2a9716-8fa1-497e-8753-941117a0ef50","volu
>>>>me
>> >>I
>> >>d
>>
>>>>":238,"vmName":"i-7-228-VM","accountId":7,"format":"VHD","provisioningT
>>>>yp
>> >>e
>> >>"
>>
>>>>:"THIN","id":238,"deviceId":0,"hypervisorType":"XenServer"},"parentSnap
>>>>sh
>> >>o
>> >>t
>>
>>>>Path":"c7b2824a-7402-4f66-b2e6-06ed452072ad","dataStore":{"org.apache.c
>>>>lo
>> >>u
>> >>d
>> >>stack.storage.to
>> .PrimaryDataStoreTO":{"uuid":"6568d30f-82b6-35a0-bfe5-be1
>> >>7
>> >>c
>>
>>>>1398ed8","id":2,"poolType":"NetworkFilesystem","host":"10.10.1.33","pat
>>>>h"
>> >>:
>> >>"
>> >>/pool3/primary3","port":2049,"url":"NetworkFilesystem://
>> 10.10.1.33/pool3/
>> >>p
>> >>r
>>
>>>>imary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a0-bfe5-be17c1398ed8","i
>>>>sM
>> >>a
>> >>n
>>
>>>>aged":false}},"vmName":"i-7-228-VM","name":"BGI-Maria_ROOT-228_20180913
>>>>03
>> >>0
>> >>2
>>
>>>>15","hypervisorType":"XenServer","id":2656,"quiescevm":false,"parents":
>>>>["
>> >>c
>> >>7
>>
>>>>b2824a-7402-4f66-b2e6-06ed452072ad"],"physicalSize":0}},"destTO":{"org.
>>>>ap
>> >>a
>> >>c
>> >>he.cloudstack.storage.to
>> .SnapshotObjectTO":{"path":"snapshots/7/238","vol
>> >>u
>> >>m
>>
>>>>e":{"uuid":"969c796d-072e-4272-adc6-93a492c03ea3","volumeType":"ROOT","
>>>>da
>> >>t
>> >>a
>> >>Store":{"org.apache.cloudstack.storage.to
>> .PrimaryDataStoreTO":{"uuid":"65
>> >>6
>> >>8
>>
>>>>d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType":"NetworkFilesystem"
>>>>,"
>> >>h
>> >>o
>>
>>>>st":"10.10.1.33","path":"/pool3/primary3","port":2049,"url":"NetworkFil
>>>>es
>> >>y
>> >>s
>> >>tem://
>> 10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a
>> >>0
>> >>-
>>
>>>>bfe5-be17c1398ed8","isManaged":false}},"name":"ROOT-228","size":2147483
>>>>64
>> >>8
>> >>0
>>
>>>>,"path":"3d2a9716-8fa1-497e-8753-941117a0ef50","volumeId":238,"vmName":
>>>>"i
>> >>-
>> >>7
>>
>>>>-228-VM","accountId":7,"format":"VHD","provisioningType":"THIN","id":23
>>>>8,
>> >>"
>> >>d
>>
>>>>eviceId":0,"hypervisorType":"XenServer"},"parentSnapshotPath":"snapshot
>>>>s/
>> >>7
>> >>/
>>
>>>>238/0c8c6ee3-ce3d-4241-8509-5193d2cff491.vhd","dataStore":{"com.cloud.a
>>>>ge
>> >>n
>> >>t
>> >>.api.to.NfsTO":{"_url":"nfs://10.10.1.34/pool8/secondary8
>> ","_role":"Image
>> >>"
>> >>}
>>
>>>>},"vmName":"i-7-228-VM","name":"BGI-Maria_ROOT-228_20180913030215","hyp
>>>>er
>> >>v
>> >>i
>>
>>>>sorType":"XenServer","id":2656,"quiescevm":false,"parents":["snapshots/
>>>>7/
>> >>2
>> >>3
>>
>>>>8/93a42896-0da1-4b82-8985-d28464f1ac4a.vhd","snapshots/7/238/0c8c6ee3-c
>>>>e3
>> >>d
>> >>-
>>
>>>>4241-8509-5193d2cff491.vhd"],"physicalSize":0}},"executeInSequence":tru
>>>>e,
>> >>"
>> >>o
>> >>ptions":{"fullSnapshot":"false"},"options2":{},"wait":28800}}] }
>> >>2018-09-13 06:02:20,025 DEBUG [c.c.a.t.Request]
>> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>> >>(logid:4c7a1c38) Seq 130-7345089517264579763: Executing: { Cmd ,
>>MgmtId:
>> >>19873467853209, via: 130(xenserver-81), Ver: v1, Flags: 100111,
>>
>>>>[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.ap
>>>>ac
>> >>h
>> >>e
>> >>.cloudstack.storage.to
>> .SnapshotObjectTO":{"path":"e91fb041-779d-4090-b14c
>> >>-
>> >>4
>>
>>>>f57f4cda54b","volume":{"uuid":"969c796d-072e-4272-adc6-93a492c03ea3","v
>>>>ol
>> >>u
>> >>m
>>
>>>>eType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDat
>>>>aS
>> >>t
>> >>o
>>
>>>>reTO":{"uuid":"6568d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType":
>>>>"N
>> >>e
>> >>t
>>
>>>>workFilesystem","host":"10.10.1.33","path":"/pool3/primary3","port":204
>>>>9,
>> >>"
>> >>u
>> >>rl":"NetworkFilesystem://
>> 10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUI
>> >>D
>> >>=
>>
>>>>6568d30f-82b6-35a0-bfe5-be17c1398ed8","isManaged":false}},"name":"ROOT-
>>>>22
>> >>8
>> >>"
>>
>>>>,"size":21474836480,"path":"3d2a9716-8fa1-497e-8753-941117a0ef50","volu
>>>>me
>> >>I
>> >>d
>>
>>>>":238,"vmName":"i-7-228-VM","accountId":7,"format":"VHD","provisioningT
>>>>yp
>> >>e
>> >>"
>>
>>>>:"THIN","id":238,"deviceId":0,"hypervisorType":"XenServer"},"parentSnap
>>>>sh
>> >>o
>> >>t
>>
>>>>Path":"c7b2824a-7402-4f66-b2e6-06ed452072ad","dataStore":{"org.apache.c
>>>>lo
>> >>u
>> >>d
>> >>stack.storage.to
>> .PrimaryDataStoreTO":{"uuid":"6568d30f-82b6-35a0-bfe5-be1
>> >>7
>> >>c
>>
>>>>1398ed8","id":2,"poolType":"NetworkFilesystem","host":"10.10.1.33","pat
>>>>h"
>> >>:
>> >>"
>> >>/pool3/primary3","port":2049,"url":"NetworkFilesystem://
>> 10.10.1.33/pool3/
>> >>p
>> >>r
>>
>>>>imary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a0-bfe5-be17c1398ed8","i
>>>>sM
>> >>a
>> >>n
>>
>>>>aged":false}},"vmName":"i-7-228-VM","name":"BGI-Maria_ROOT-228_20180913
>>>>03
>> >>0
>> >>2
>>
>>>>15","hypervisorType":"XenServer","id":2656,"quiescevm":false,"parents":
>>>>["
>> >>c
>> >>7
>>
>>>>b2824a-7402-4f66-b2e6-06ed452072ad"],"physicalSize":0}},"destTO":{"org.
>>>>ap
>> >>a
>> >>c
>> >>he.cloudstack.storage.to
>> .SnapshotObjectTO":{"path":"snapshots/7/238","vol
>> >>u
>> >>m
>>
>>>>e":{"uuid":"969c796d-072e-4272-adc6-93a492c03ea3","volumeType":"ROOT","
>>>>da
>> >>t
>> >>a
>> >>Store":{"org.apache.cloudstack.storage.to
>> .PrimaryDataStoreTO":{"uuid":"65
>> >>6
>> >>8
>>
>>>>d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType":"NetworkFilesystem"
>>>>,"
>> >>h
>> >>o
>>
>>>>st":"10.10.1.33","path":"/pool3/primary3","port":2049,"url":"NetworkFil
>>>>es
>> >>y
>> >>s
>> >>tem://
>> 10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a
>> >>0
>> >>-
>>
>>>>bfe5-be17c1398ed8","isManaged":false}},"name":"ROOT-228","size":2147483
>>>>64
>> >>8
>> >>0
>>
>>>>,"path":"3d2a9716-8fa1-497e-8753-941117a0ef50","volumeId":238,"vmName":
>>>>"i
>> >>-
>> >>7
>>
>>>>-228-VM","accountId":7,"format":"VHD","provisioningType":"THIN","id":23
>>>>8,
>> >>"
>> >>d
>>
>>>>eviceId":0,"hypervisorType":"XenServer"},"parentSnapshotPath":"snapshot
>>>>s/
>> >>7
>> >>/
>>
>>>>238/0c8c6ee3-ce3d-4241-8509-5193d2cff491.vhd","dataStore":{"com.cloud.a
>>>>ge
>> >>n
>> >>t
>> >>.api.to.NfsTO":{"_url":"nfs://10.10.1.34/pool8/secondary8
>> ","_role":"Image
>> >>"
>> >>}
>>
>>>>},"vmName":"i-7-228-VM","name":"BGI-Maria_ROOT-228_20180913030215","hyp
>>>>er
>> >>v
>> >>i
>>
>>>>sorType":"XenServer","id":2656,"quiescevm":false,"parents":["snapshots/
>>>>7/
>> >>2
>> >>3
>>
>>>>8/93a42896-0da1-4b82-8985-d28464f1ac4a.vhd","snapshots/7/238/0c8c6ee3-c
>>>>e3
>> >>d
>> >>-
>>
>>>>4241-8509-5193d2cff491.vhd"],"physicalSize":0}},"executeInSequence":tru
>>>>e,
>> >>"
>> >>o
>> >>ptions":{"fullSnapshot":"false"},"options2":{},"wait":28800}}] }
>> >>2018-09-13 06:04:28,895 DEBUG [c.c.h.x.r.CitrixResourceBase]
>> >>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Host 10.10.1.81
>> >>OpaqueRef:f83331df-6634-63fd-6b95-44607a6069b8: Removing SR
>> >>2018-09-13 06:04:28,913 DEBUG [c.c.h.x.r.CitrixResourceBase]
>> >>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Host 10.10.1.81
>> >>OpaqueRef:05380b03-4cb8-85a7-6a6a-a1595c188f60: Unplugging pbd
>> >>2018-09-13 06:04:30,881 DEBUG [c.c.h.x.r.CitrixResourceBase]
>> >>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Host 10.10.1.81
>> >>OpaqueRef:f83331df-6634-63fd-6b95-44607a6069b8: Forgetting
>> >>2018-09-13 06:04:30,891 DEBUG [c.c.h.x.r.XenServerStorageProcessor]
>> >>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Successfully destroyed
>> >>snapshot on volume: 3d2a9716-8fa1-497e-8753-941117a0ef50 execept this
>> >>current snapshot e91fb041-779d-4090-b14c-4f57f4cda54b
>> >>2018-09-13 06:04:30,891 INFO [c.c.h.x.r.XenServerStorageProcessor]
>> >>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) New snapshot details:
>>
>>>>SnapshotTO[datastore=null|volume=null|pathsnapshots/7/238/0560d536-c945
>>>>-4
>> >>a
>> >>6
>> >>1-8c4c-dbaf29f29993.vhd]
>> >>2018-09-13 06:04:30,891 INFO [c.c.h.x.r.XenServerStorageProcessor]
>> >>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) New snapshot physical
>> >>utilization: 2342932992
>> >>2018-09-13 06:04:30,891 DEBUG [c.c.a.m.DirectAgentAttache]
>> >>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Seq
>> >>130-7345089517264579763: Response Received:
>> >>2018-09-13 06:04:30,891 DEBUG [c.c.a.t.Request]
>> >>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Seq
>> >>130-7345089517264579763: Processing: { Ans: , MgmtId: 19873467853209,
>> >>via: 130(xenserver-81), Ver: v1, Flags: 110,
>>
>>>>[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"or
>>>>g.
>> >>a
>> >>p
>> >>ache.cloudstack.storage.to
>> .SnapshotObjectTO":{"path":"snapshots/7/238/056
>> >>0
>> >>d
>>
>>>>536-c945-4a61-8c4c-dbaf29f29993.vhd","parentSnapshotPath":"snapshots/7/
>>>>23
>> >>8
>> >>/
>>
>>>>0c8c6ee3-ce3d-4241-8509-5193d2cff491.vhd","id":0,"quiescevm":false,"phy
>>>>si
>> >>c
>> >>a
>> >>lSize":2342932992}},"result":true,"wait":0}}] }
>> >>2018-09-13 06:04:30,891 DEBUG [c.c.a.m.AgentAttache]
>> >>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Seq
>> >>130-7345089517264579763: No more commands found
>> >>2018-09-13 06:04:30,891 DEBUG [c.c.a.t.Request]
>> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>> >>(logid:4c7a1c38) Seq 130-7345089517264579763: Received: { Ans: ,
>>MgmtId:
>> >>19873467853209, via: 130(xenserver-81), Ver: v1, Flags: 110, {
>> >>CopyCmdAnswer } }
>> >>2018-09-13 06:04:30,957 DEBUG [c.c.r.ResourceLimitManagerImpl]
>> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>> >>(logid:4c7a1c38) Updating resource Type = secondary_storage count for
>> >>Account = 7 Operation = decreasing Amount = 19131903488
>> >>2018-09-13 06:04:30,969 DEBUG [c.c.v.VmWorkJobHandlerProxy]
>> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>> >>(logid:4c7a1c38) Done executing VM work job:
>>
>>>>com.cloud.vm.VmWorkTakeVolumeSnapshot{"volumeId":238,"policyId":17,"sna
>>>>ps
>> >>h
>> >>o
>>
>>>>tId":2656,"quiesceVm":false,"asyncBackup":false,"userId":1,"accountId":
>>>>7,
>> >>"
>> >>v
>> >>mId":228,"handlerName":"VolumeApiServiceImpl"}
>> >>2018-09-13 06:04:30,969 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>> >>(logid:4c7a1c38) Complete async job-28021, jobStatus: SUCCEEDED,
>> >>resultCode: 0, result:
>>
>>>>rO0ABXNyAA5qYXZhLmxhbmcuTG9uZzuL5JDMjyPfAgABSgAFdmFsdWV4cgAQamF2YS5sYW5
>>>>nL
>> >>k
>> >>5
>> >>1bWJlcoaslR0LlOCLAgAAeHAAAAAAAAAKYA
>> >>2018-09-13 06:04:30,970 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>> >>(logid:4c7a1c38) Publish async job-28021 complete on message bus
>> >>2018-09-13 06:04:30,970 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>> >>(logid:4c7a1c38) Wake up jobs related to job-28021
>> >>2018-09-13 06:04:30,970 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>> >>(logid:4c7a1c38) Update db status for job-28021
>> >>2018-09-13 06:04:30,971 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>> >>(logid:4c7a1c38) Wake up jobs joined with job-28021 and disjoin all
>> >>subjobs created from job- 28021
>> >>2018-09-13 06:04:30,983 DEBUG [c.c.v.VmWorkJobDispatcher]
>> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021)
>>(logid:4c7a1c38)
>> >>Done with run of VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot
>>for
>> >>VM
>> >>228, job origin: 28020
>> >>2018-09-13 06:04:30,983 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021)
>>(logid:4c7a1c38)
>> >>Done executing com.cloud.vm.VmWorkTakeVolumeSnapshot for job-28021
>> >>2018-09-13 06:04:30,984 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021)
>>(logid:4c7a1c38)
>> >>Remove job-28021 from job monitoring
>> >>2018-09-13 06:04:31,026 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> >>(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6)
>> >>(logid:4c7a1c38)
>> >>Complete async job-28020, jobStatus: SUCCEEDED, resultCode: 0, result:
>>
>>>>org.apache.cloudstack.api.response.SnapshotResponse/snapshot/{"id":"3c8
>>>>66
>> >>2
>> >>b
>>
>>>>2-c76f-4e3a-a928-f76dad820122","account":"juha","domainid":"d3c2420f-75
>>>>03
>> >>-
>> >>4
>>
>>>>560-a2da-d7e7149ce390","domain":"BGI","snapshottype":"DAILY","volumeid"
>>>>:"
>> >>9
>> >>6
>>
>>>>9c796d-072e-4272-adc6-93a492c03ea3","volumename":"ROOT-228","volumetype
>>>>":
>> >>"
>> >>R
>>
>>>>OOT","created":"2018-09-13T06:02:15+0300","name":"BGI-Maria_ROOT-228_20
>>>>18
>> >>0
>> >>9
>>
>>>>13030215","intervaltype":"DAILY","state":"BackedUp","physicalsize":2342
>>>>93
>> >>2
>> >>9
>>
>>>>92,"zoneid":"533c8f88-2637-4c0e-8210-e0a355b62d82","tags":[],"revertabl
>>>>e"
>> >>:
>> >>f
>>
>>>>alse,"ostypeid":"da491b96-3748-11e5-a683-121326fde999","osdisplayname":
>>>>"C
>> >>e
>> >>n
>> >>tOS 6.5 (64-bit)","virtualsize":21474836480}
>> >>2018-09-13 06:04:31,027 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> >>(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6)
>> >>(logid:4c7a1c38)
>> >>Publish async job-28020 complete on message bus
>> >>2018-09-13 06:04:31,027 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> >>(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6)
>> >>(logid:4c7a1c38)
>> >>Wake up jobs related to job-28020
>> >>2018-09-13 06:04:31,027 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> >>(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6)
>> >>(logid:4c7a1c38)
>> >>Update db status for job-28020
>> >>2018-09-13 06:04:31,028 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> >>(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6)
>> >>(logid:4c7a1c38)
>> >>Wake up jobs joined with job-28020 and disjoin all subjobs created
>>from
>> >>job- 28020
>> >>2018-09-13 06:04:31,039 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> >>(API-Job-Executor-83:ctx-cfb1cd95 job-28020) (logid:4c7a1c38) Done
>> >>executing
>> >>org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for
>> >>job-28020
>> >>2018-09-13 06:04:31,039 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>> >>(API-Job-Executor-83:ctx-cfb1cd95 job-28020) (logid:4c7a1c38) Remove
>> >>job-28020 from job monitoring
>>
>>
>>
>
>--
>Rafael Weingärtner
Re: VM Snapshot not removed from primary SR
Posted by Rafael Weingärtner <ra...@gmail.com>.
If I am not mistaken, this has already been fixed in 4.11.1.0
On Wed, Oct 10, 2018 at 10:31 AM Sami Rajala <sa...@fapps.com> wrote:
> Hi,
>
>
> I have got zero advice for this.
>
> "I have had weird issue since I updated CS from 4.9 to 4.10.
> CS does not delete/remove temporary snapshot from primary SR and I have
> had to removed it by manual time to time, before 30 snapshot limit has
> reached.
> + doesn't CS not update secondary_storege count on snapshot_count table”
>
>
>
>
> Now I have a situation where Xenserver has 19 snapshot on primary SR for
> each VM + resource count for ”secondary_storege” in database increase all
> the time per account.
> ACS storage cleaner removed old snapshots from secondary when they getting
> old, but thats all, not decrease resource count for ”secondary_storege”
> with value of deleted snapshot ( or not remove snapshot from primary SR
> after snapshot has created and copied to secondary )
>
> I have still delete primary snapshot by manual + give a new value for
> ”secondary_storege” before they reach limit.
>
> has anyone got this kind of issue? any fix
>
> BR
> -sami
>
>
>
> 14.9.2018 12.26, "Sami Rajala (FAPPS)" <sa...@fapps.com>:
>
> >Continue,
> >
> >I created template from VM-s latest snapshot after 3 days ( one parent +
> >2
> >child snapshot) - without errors/warning
> >CS create VM from this template ok, but Xen could not start VM
> >
> >so, I return back to situation where every snapshot is full and manual
> >removing snapshot from primary
> >
> >Has anyone any advice?
> >
> >BR
> >-sami
> >
> >
> >
> >13.9.2018 7.56, "Sami Rajala (FAPPS)" <sa...@fapps.com>:
> >
> >>Hello
> >>
> >>I have had weird issue since I updated CS from 4.9 to 4.10.
> >>CS does not delete/remove temporary snapshot from primary SR and I have
> >>had to removed it by manual time to time, before 30 snapshot limit has
> >>reached.
> >>+ doesn¹t CS not update secondary_storege count on snapshot_count table
> >>
> >>It works fine on 4.9, but stop to work on 4.10 and I have wait
> >>possibility
> >>to update 4.11.1.
> >>Update to 4.11.1 has done now and looks like this not work still
> >>
> >>I removed all snapshot and start over from clean table, snapshot policy
> >>is: DAILY, keep 2, delta = 5
> >>There are now 3 snapshot on primary SR and 3 on secondary SR
> >>
> >>The Environment is Xen 7 + CS 4.11.1 + NFS storage
> >>
> >>Is there some other parameter I should look and any other work around I
> >>has to do to get it work ?
> >>
> >>Any hits?
> >>
> >>BR
> >>-sami
> >>
> >>VM snapshot Log for last round:
> >>
> >>2018-09-13 06:02:15,983 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>(API-Job-Executor-83:ctx-cfb1cd95 job-28020) (logid:4c7a1c38) Executing
> >>AsyncJobVO {id:28020, userId: 1, accountId: 7, instanceType: Snapshot,
> >>instanceId: 2656, cmd:
> >>org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd,
> >>cmdInfo:
> >>{"policyid":"17","ctxUserId":"1","volumeid":"238","ctxStartEventId":"1","
> >>i
> >>d
> >>":"2656","ctxAccountId":"7"}, cmdVersion: 0, status: IN_PROGRESS,
> >>processStatus: 0, resultCode: 0, result: null, initMsid: 19873467853209,
> >>completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
> >>2018-09-13 06:02:15,991 DEBUG [c.c.u.AccountManagerImpl]
> >>(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6)
> >>(logid:4c7a1c38)
> >>Access to Acct[479c643e-9c84-41fb-9f0a-9bb999893a25-juha] granted to
> >>Acct[479c643e-9c84-41fb-9f0a-9bb999893a25-juha] by DomainChecker
> >>2018-09-13 06:02:16,045 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6)
> >>(logid:4c7a1c38)
> >>Sync job-28021 execution on object VmWorkJobQueue.228
> >>2018-09-13 06:02:17,115 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021) (logid:4c7a1c38)
> >>Executing AsyncJobVO {id:28021, userId: 1, accountId: 7, instanceType:
> >>null, instanceId: null, cmd: com.cloud.vm.VmWorkTakeVolumeSnapshot,
> >>cmdInfo:
> >>rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrVGFrZVZvbHVtZVNuYXBzaG90BL5gG4Li1c8CA
> >>A
> >>Z
> >>aAAthc3luY0JhY2t1cFoACXF1aWVzY2VWbUwADGxvY2F0aW9uVHlwZXQAKUxjb20vY2xvdWQv
> >>c
> >>3
> >>RvcmFnZS9TbmFwc2hvdCRMb2NhdGlvblR5cGU7TAAIcG9saWN5SWR0ABBMamF2YS9sYW5nL0x
> >>v
> >>b
> >>mc7TAAKc25hcHNob3RJZHEAfgACTAAIdm9sdW1lSWRxAH4AAnhyABNjb20uY2xvdWQudm0uVm
> >>1
> >>X
> >>b3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlkTAALaGFuZGxlck5hb
> >>W
> >>V
> >>0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAAHAAAAAAAAAAEAAAAAAAAA5HQAFFZvbHVt
> >>Z
> >>U
> >>FwaVNlcnZpY2VJbXBsAABwc3IADmphdmEubGFuZy5Mb25nO4vkkMyPI98CAAFKAAV2YWx1ZXh
> >>y
> >>A
> >>BBqYXZhLmxhbmcuTnVtYmVyhqyVHQuU4IsCAAB4cAAAAAAAAAARc3EAfgAHAAAAAAAACmBzcQ
> >>B
> >>-
> >>AAcAAAAAAAAA7g, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0,
> >>resultCode: 0, result: null, initMsid: 19873467853209, completeMsid:
> >>null,
> >>lastUpdated: null, lastPolled: null, created: Thu Sep 13 06:02:16 EEST
> >>2018}
> >>2018-09-13 06:02:17,116 DEBUG [c.c.v.VmWorkJobDispatcher]
> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021) (logid:4c7a1c38)
> >>Run VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot for VM 228, job
> >>origin: 28020
> >>2018-09-13 06:02:17,117 DEBUG [c.c.v.VmWorkJobHandlerProxy]
> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
> >>(logid:4c7a1c38) Execute VM work job:
> >>com.cloud.vm.VmWorkTakeVolumeSnapshot{"volumeId":238,"policyId":17,"snaps
> >>h
> >>o
> >>tId":2656,"quiesceVm":false,"asyncBackup":false,"userId":1,"accountId":7,
> >>"
> >>v
> >>mId":228,"handlerName":"VolumeApiServiceImpl"}
> >>2018-09-13 06:02:17,197 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
> >>(logid:4c7a1c38) getCommandHostDelegation: class
> >>org.apache.cloudstack.storage.command.CreateObjectCommand
> >>2018-09-13 06:02:17,197 DEBUG [c.c.h.XenServerGuru]
> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
> >>(logid:4c7a1c38) We are returning the default host to execute commands
> >>because the command is not of Copy type.
> >>2018-09-13 06:02:17,199 DEBUG [c.c.a.t.Request]
> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
> >>(logid:4c7a1c38) Seq 128-4630826316843724436: Sending { Cmd , MgmtId:
> >>19873467853209, via: 128(xenserver-15), Ver: v1, Flags: 100011,
> >>[{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"o
> >>r
> >>g
> >>.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"969c79
> >>6
> >>d
> >>-072e-4272-adc6-93a492c03ea3","volumeType":"ROOT","dataStore":{"org.apach
> >>e
> >>.
> >>cloudstack.storage.to
> .PrimaryDataStoreTO":{"uuid":"6568d30f-82b6-35a0-bfe
> >>5
> >>-
> >>be17c1398ed8","id":2,"poolType":"NetworkFilesystem","host":"10.10.1.33","
> >>p
> >>a
> >>th":"/pool3/primary3","port":2049,"url":"NetworkFilesystem://
> 10.10.1.33/p
> >>o
> >>o
> >>l3/primary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a0-bfe5-be17c1398ed8"
> >>,
> >>"
> >>isManaged":false}},"name":"ROOT-228","size":21474836480,"path":"3d2a9716-
> >>8
> >>f
> >>a1-497e-8753-941117a0ef50","volumeId":238,"vmName":"i-7-228-VM","accountI
> >>d
> >>"
> >>:7,"format":"VHD","provisioningType":"THIN","id":238,"deviceId":0,"hyperv
> >>i
> >>s
> >>orType":"XenServer"},"parentSnapshotPath":"c7b2824a-7402-4f66-b2e6-06ed45
> >>2
> >>0
> >>72ad","dataStore":{"org.apache.cloudstack.storage.to
> .PrimaryDataStoreTO":
> >>{
> >>"
> >>uuid":"6568d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType":"NetworkFi
> >>l
> >>e
> >>system","host":"10.10.1.33","path":"/pool3/primary3","port":2049,"url":"N
> >>e
> >>t
> >>workFilesystem://
> 10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUID=6568d3
> >>0
> >>f
> >>-82b6-35a0-bfe5-be17c1398ed8","isManaged":false}},"vmName":"i-7-228-VM","
> >>n
> >>a
> >>me":"BGI-Maria_ROOT-228_20180913030215","hypervisorType":"XenServer","id"
> >>:
> >>2
> >>656,"quiescevm":false,"parents":["c7b2824a-7402-4f66-b2e6-06ed452072ad"],
> >>"
> >>p
> >>hysicalSize":0}},"wait":0}}] }
> >>2018-09-13 06:02:17,199 DEBUG [c.c.a.t.Request]
> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
> >>(logid:4c7a1c38) Seq 128-4630826316843724436: Executing: { Cmd , MgmtId:
> >>19873467853209, via: 128(xenserver-15), Ver: v1, Flags: 100011,
> >>[{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"o
> >>r
> >>g
> >>.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"969c79
> >>6
> >>d
> >>-072e-4272-adc6-93a492c03ea3","volumeType":"ROOT","dataStore":{"org.apach
> >>e
> >>.
> >>cloudstack.storage.to
> .PrimaryDataStoreTO":{"uuid":"6568d30f-82b6-35a0-bfe
> >>5
> >>-
> >>be17c1398ed8","id":2,"poolType":"NetworkFilesystem","host":"10.10.1.33","
> >>p
> >>a
> >>th":"/pool3/primary3","port":2049,"url":"NetworkFilesystem://
> 10.10.1.33/p
> >>o
> >>o
> >>l3/primary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a0-bfe5-be17c1398ed8"
> >>,
> >>"
> >>isManaged":false}},"name":"ROOT-228","size":21474836480,"path":"3d2a9716-
> >>8
> >>f
> >>a1-497e-8753-941117a0ef50","volumeId":238,"vmName":"i-7-228-VM","accountI
> >>d
> >>"
> >>:7,"format":"VHD","provisioningType":"THIN","id":238,"deviceId":0,"hyperv
> >>i
> >>s
> >>orType":"XenServer"},"parentSnapshotPath":"c7b2824a-7402-4f66-b2e6-06ed45
> >>2
> >>0
> >>72ad","dataStore":{"org.apache.cloudstack.storage.to
> .PrimaryDataStoreTO":
> >>{
> >>"
> >>uuid":"6568d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType":"NetworkFi
> >>l
> >>e
> >>system","host":"10.10.1.33","path":"/pool3/primary3","port":2049,"url":"N
> >>e
> >>t
> >>workFilesystem://
> 10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUID=6568d3
> >>0
> >>f
> >>-82b6-35a0-bfe5-be17c1398ed8","isManaged":false}},"vmName":"i-7-228-VM","
> >>n
> >>a
> >>me":"BGI-Maria_ROOT-228_20180913030215","hypervisorType":"XenServer","id"
> >>:
> >>2
> >>656,"quiescevm":false,"parents":["c7b2824a-7402-4f66-b2e6-06ed452072ad"],
> >>"
> >>p
> >>hysicalSize":0}},"wait":0}}] }
> >>2018-09-13 06:02:19,815 DEBUG [c.c.a.m.DirectAgentAttache]
> >>(DirectAgent-206:ctx-370f4cd9) (logid:4c7a1c38) Seq
> >>128-4630826316843724436: Response Received:
> >>2018-09-13 06:02:19,815 DEBUG [c.c.a.t.Request]
> >>(DirectAgent-206:ctx-370f4cd9) (logid:4c7a1c38) Seq
> >>128-4630826316843724436: Processing: { Ans: , MgmtId: 19873467853209,
> >>via: 128(xenserver-15), Ver: v1, Flags: 10,
> >>[{"org.apache.cloudstack.storage.command.CreateObjectAnswer":{"data":{"or
> >>g
> >>.
> >>apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"e91fb041-779d-409
> >>0
> >>-
> >>b14c-4f57f4cda54b","id":0,"quiescevm":false,"physicalSize":0}},"result":t
> >>r
> >>u
> >>e,"wait":0}}] }
> >>2018-09-13 06:02:19,815 DEBUG [c.c.a.t.Request]
> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
> >>(logid:4c7a1c38) Seq 128-4630826316843724436: Received: { Ans: , MgmtId:
> >>19873467853209, via: 128(xenserver-15), Ver: v1, Flags: 10, {
> >>CreateObjectAnswer } }
> >>2018-09-13 06:02:19,934 DEBUG [o.a.c.s.m.AncientDataMotionStrategy]
> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
> >>(logid:4c7a1c38) copyAsync inspecting src type SNAPSHOT copyAsync
> >>inspecting dest type SNAPSHOT
> >>2018-09-13 06:02:20,017 ERROR [o.a.c.s.e.DefaultEndPointSelector]
> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
> >>(logid:4c7a1c38) IR24 select BACKUPSNAPSHOT from primary to secondary
> >>2656
> >>dest=2656
> >>2018-09-13 06:02:20,023 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
> >>(logid:4c7a1c38) getCommandHostDelegation: class
> >>org.apache.cloudstack.storage.command.CopyCommand
> >>2018-09-13 06:02:20,023 DEBUG [c.c.h.XenServerGuru]
> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
> >>(logid:4c7a1c38) We are returning the default host to execute commands
> >>because the source and destination objects are not NFS type.
> >>2018-09-13 06:02:20,025 DEBUG [c.c.a.t.Request]
> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
> >>(logid:4c7a1c38) Seq 130-7345089517264579763: Sending { Cmd , MgmtId:
> >>19873467853209, via: 130(xenserver-81), Ver: v1, Flags: 100111,
> >>[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apac
> >>h
> >>e
> >>.cloudstack.storage.to
> .SnapshotObjectTO":{"path":"e91fb041-779d-4090-b14c
> >>-
> >>4
> >>f57f4cda54b","volume":{"uuid":"969c796d-072e-4272-adc6-93a492c03ea3","vol
> >>u
> >>m
> >>eType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataS
> >>t
> >>o
> >>reTO":{"uuid":"6568d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType":"N
> >>e
> >>t
> >>workFilesystem","host":"10.10.1.33","path":"/pool3/primary3","port":2049,
> >>"
> >>u
> >>rl":"NetworkFilesystem://
> 10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUI
> >>D
> >>=
> >>6568d30f-82b6-35a0-bfe5-be17c1398ed8","isManaged":false}},"name":"ROOT-22
> >>8
> >>"
> >>,"size":21474836480,"path":"3d2a9716-8fa1-497e-8753-941117a0ef50","volume
> >>I
> >>d
> >>":238,"vmName":"i-7-228-VM","accountId":7,"format":"VHD","provisioningTyp
> >>e
> >>"
> >>:"THIN","id":238,"deviceId":0,"hypervisorType":"XenServer"},"parentSnapsh
> >>o
> >>t
> >>Path":"c7b2824a-7402-4f66-b2e6-06ed452072ad","dataStore":{"org.apache.clo
> >>u
> >>d
> >>stack.storage.to
> .PrimaryDataStoreTO":{"uuid":"6568d30f-82b6-35a0-bfe5-be1
> >>7
> >>c
> >>1398ed8","id":2,"poolType":"NetworkFilesystem","host":"10.10.1.33","path"
> >>:
> >>"
> >>/pool3/primary3","port":2049,"url":"NetworkFilesystem://
> 10.10.1.33/pool3/
> >>p
> >>r
> >>imary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a0-bfe5-be17c1398ed8","isM
> >>a
> >>n
> >>aged":false}},"vmName":"i-7-228-VM","name":"BGI-Maria_ROOT-228_2018091303
> >>0
> >>2
> >>15","hypervisorType":"XenServer","id":2656,"quiescevm":false,"parents":["
> >>c
> >>7
> >>b2824a-7402-4f66-b2e6-06ed452072ad"],"physicalSize":0}},"destTO":{"org.ap
> >>a
> >>c
> >>he.cloudstack.storage.to
> .SnapshotObjectTO":{"path":"snapshots/7/238","vol
> >>u
> >>m
> >>e":{"uuid":"969c796d-072e-4272-adc6-93a492c03ea3","volumeType":"ROOT","da
> >>t
> >>a
> >>Store":{"org.apache.cloudstack.storage.to
> .PrimaryDataStoreTO":{"uuid":"65
> >>6
> >>8
> >>d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType":"NetworkFilesystem","
> >>h
> >>o
> >>st":"10.10.1.33","path":"/pool3/primary3","port":2049,"url":"NetworkFiles
> >>y
> >>s
> >>tem://
> 10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a
> >>0
> >>-
> >>bfe5-be17c1398ed8","isManaged":false}},"name":"ROOT-228","size":214748364
> >>8
> >>0
> >>,"path":"3d2a9716-8fa1-497e-8753-941117a0ef50","volumeId":238,"vmName":"i
> >>-
> >>7
> >>-228-VM","accountId":7,"format":"VHD","provisioningType":"THIN","id":238,
> >>"
> >>d
> >>eviceId":0,"hypervisorType":"XenServer"},"parentSnapshotPath":"snapshots/
> >>7
> >>/
> >>238/0c8c6ee3-ce3d-4241-8509-5193d2cff491.vhd","dataStore":{"com.cloud.age
> >>n
> >>t
> >>.api.to.NfsTO":{"_url":"nfs://10.10.1.34/pool8/secondary8
> ","_role":"Image
> >>"
> >>}
> >>},"vmName":"i-7-228-VM","name":"BGI-Maria_ROOT-228_20180913030215","hyper
> >>v
> >>i
> >>sorType":"XenServer","id":2656,"quiescevm":false,"parents":["snapshots/7/
> >>2
> >>3
> >>8/93a42896-0da1-4b82-8985-d28464f1ac4a.vhd","snapshots/7/238/0c8c6ee3-ce3
> >>d
> >>-
> >>4241-8509-5193d2cff491.vhd"],"physicalSize":0}},"executeInSequence":true,
> >>"
> >>o
> >>ptions":{"fullSnapshot":"false"},"options2":{},"wait":28800}}] }
> >>2018-09-13 06:02:20,025 DEBUG [c.c.a.t.Request]
> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
> >>(logid:4c7a1c38) Seq 130-7345089517264579763: Executing: { Cmd , MgmtId:
> >>19873467853209, via: 130(xenserver-81), Ver: v1, Flags: 100111,
> >>[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apac
> >>h
> >>e
> >>.cloudstack.storage.to
> .SnapshotObjectTO":{"path":"e91fb041-779d-4090-b14c
> >>-
> >>4
> >>f57f4cda54b","volume":{"uuid":"969c796d-072e-4272-adc6-93a492c03ea3","vol
> >>u
> >>m
> >>eType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataS
> >>t
> >>o
> >>reTO":{"uuid":"6568d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType":"N
> >>e
> >>t
> >>workFilesystem","host":"10.10.1.33","path":"/pool3/primary3","port":2049,
> >>"
> >>u
> >>rl":"NetworkFilesystem://
> 10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUI
> >>D
> >>=
> >>6568d30f-82b6-35a0-bfe5-be17c1398ed8","isManaged":false}},"name":"ROOT-22
> >>8
> >>"
> >>,"size":21474836480,"path":"3d2a9716-8fa1-497e-8753-941117a0ef50","volume
> >>I
> >>d
> >>":238,"vmName":"i-7-228-VM","accountId":7,"format":"VHD","provisioningTyp
> >>e
> >>"
> >>:"THIN","id":238,"deviceId":0,"hypervisorType":"XenServer"},"parentSnapsh
> >>o
> >>t
> >>Path":"c7b2824a-7402-4f66-b2e6-06ed452072ad","dataStore":{"org.apache.clo
> >>u
> >>d
> >>stack.storage.to
> .PrimaryDataStoreTO":{"uuid":"6568d30f-82b6-35a0-bfe5-be1
> >>7
> >>c
> >>1398ed8","id":2,"poolType":"NetworkFilesystem","host":"10.10.1.33","path"
> >>:
> >>"
> >>/pool3/primary3","port":2049,"url":"NetworkFilesystem://
> 10.10.1.33/pool3/
> >>p
> >>r
> >>imary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a0-bfe5-be17c1398ed8","isM
> >>a
> >>n
> >>aged":false}},"vmName":"i-7-228-VM","name":"BGI-Maria_ROOT-228_2018091303
> >>0
> >>2
> >>15","hypervisorType":"XenServer","id":2656,"quiescevm":false,"parents":["
> >>c
> >>7
> >>b2824a-7402-4f66-b2e6-06ed452072ad"],"physicalSize":0}},"destTO":{"org.ap
> >>a
> >>c
> >>he.cloudstack.storage.to
> .SnapshotObjectTO":{"path":"snapshots/7/238","vol
> >>u
> >>m
> >>e":{"uuid":"969c796d-072e-4272-adc6-93a492c03ea3","volumeType":"ROOT","da
> >>t
> >>a
> >>Store":{"org.apache.cloudstack.storage.to
> .PrimaryDataStoreTO":{"uuid":"65
> >>6
> >>8
> >>d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType":"NetworkFilesystem","
> >>h
> >>o
> >>st":"10.10.1.33","path":"/pool3/primary3","port":2049,"url":"NetworkFiles
> >>y
> >>s
> >>tem://
> 10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a
> >>0
> >>-
> >>bfe5-be17c1398ed8","isManaged":false}},"name":"ROOT-228","size":214748364
> >>8
> >>0
> >>,"path":"3d2a9716-8fa1-497e-8753-941117a0ef50","volumeId":238,"vmName":"i
> >>-
> >>7
> >>-228-VM","accountId":7,"format":"VHD","provisioningType":"THIN","id":238,
> >>"
> >>d
> >>eviceId":0,"hypervisorType":"XenServer"},"parentSnapshotPath":"snapshots/
> >>7
> >>/
> >>238/0c8c6ee3-ce3d-4241-8509-5193d2cff491.vhd","dataStore":{"com.cloud.age
> >>n
> >>t
> >>.api.to.NfsTO":{"_url":"nfs://10.10.1.34/pool8/secondary8
> ","_role":"Image
> >>"
> >>}
> >>},"vmName":"i-7-228-VM","name":"BGI-Maria_ROOT-228_20180913030215","hyper
> >>v
> >>i
> >>sorType":"XenServer","id":2656,"quiescevm":false,"parents":["snapshots/7/
> >>2
> >>3
> >>8/93a42896-0da1-4b82-8985-d28464f1ac4a.vhd","snapshots/7/238/0c8c6ee3-ce3
> >>d
> >>-
> >>4241-8509-5193d2cff491.vhd"],"physicalSize":0}},"executeInSequence":true,
> >>"
> >>o
> >>ptions":{"fullSnapshot":"false"},"options2":{},"wait":28800}}] }
> >>2018-09-13 06:04:28,895 DEBUG [c.c.h.x.r.CitrixResourceBase]
> >>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Host 10.10.1.81
> >>OpaqueRef:f83331df-6634-63fd-6b95-44607a6069b8: Removing SR
> >>2018-09-13 06:04:28,913 DEBUG [c.c.h.x.r.CitrixResourceBase]
> >>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Host 10.10.1.81
> >>OpaqueRef:05380b03-4cb8-85a7-6a6a-a1595c188f60: Unplugging pbd
> >>2018-09-13 06:04:30,881 DEBUG [c.c.h.x.r.CitrixResourceBase]
> >>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Host 10.10.1.81
> >>OpaqueRef:f83331df-6634-63fd-6b95-44607a6069b8: Forgetting
> >>2018-09-13 06:04:30,891 DEBUG [c.c.h.x.r.XenServerStorageProcessor]
> >>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Successfully destroyed
> >>snapshot on volume: 3d2a9716-8fa1-497e-8753-941117a0ef50 execept this
> >>current snapshot e91fb041-779d-4090-b14c-4f57f4cda54b
> >>2018-09-13 06:04:30,891 INFO [c.c.h.x.r.XenServerStorageProcessor]
> >>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) New snapshot details:
> >>SnapshotTO[datastore=null|volume=null|pathsnapshots/7/238/0560d536-c945-4
> >>a
> >>6
> >>1-8c4c-dbaf29f29993.vhd]
> >>2018-09-13 06:04:30,891 INFO [c.c.h.x.r.XenServerStorageProcessor]
> >>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) New snapshot physical
> >>utilization: 2342932992
> >>2018-09-13 06:04:30,891 DEBUG [c.c.a.m.DirectAgentAttache]
> >>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Seq
> >>130-7345089517264579763: Response Received:
> >>2018-09-13 06:04:30,891 DEBUG [c.c.a.t.Request]
> >>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Seq
> >>130-7345089517264579763: Processing: { Ans: , MgmtId: 19873467853209,
> >>via: 130(xenserver-81), Ver: v1, Flags: 110,
> >>[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.
> >>a
> >>p
> >>ache.cloudstack.storage.to
> .SnapshotObjectTO":{"path":"snapshots/7/238/056
> >>0
> >>d
> >>536-c945-4a61-8c4c-dbaf29f29993.vhd","parentSnapshotPath":"snapshots/7/23
> >>8
> >>/
> >>0c8c6ee3-ce3d-4241-8509-5193d2cff491.vhd","id":0,"quiescevm":false,"physi
> >>c
> >>a
> >>lSize":2342932992}},"result":true,"wait":0}}] }
> >>2018-09-13 06:04:30,891 DEBUG [c.c.a.m.AgentAttache]
> >>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Seq
> >>130-7345089517264579763: No more commands found
> >>2018-09-13 06:04:30,891 DEBUG [c.c.a.t.Request]
> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
> >>(logid:4c7a1c38) Seq 130-7345089517264579763: Received: { Ans: , MgmtId:
> >>19873467853209, via: 130(xenserver-81), Ver: v1, Flags: 110, {
> >>CopyCmdAnswer } }
> >>2018-09-13 06:04:30,957 DEBUG [c.c.r.ResourceLimitManagerImpl]
> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
> >>(logid:4c7a1c38) Updating resource Type = secondary_storage count for
> >>Account = 7 Operation = decreasing Amount = 19131903488
> >>2018-09-13 06:04:30,969 DEBUG [c.c.v.VmWorkJobHandlerProxy]
> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
> >>(logid:4c7a1c38) Done executing VM work job:
> >>com.cloud.vm.VmWorkTakeVolumeSnapshot{"volumeId":238,"policyId":17,"snaps
> >>h
> >>o
> >>tId":2656,"quiesceVm":false,"asyncBackup":false,"userId":1,"accountId":7,
> >>"
> >>v
> >>mId":228,"handlerName":"VolumeApiServiceImpl"}
> >>2018-09-13 06:04:30,969 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
> >>(logid:4c7a1c38) Complete async job-28021, jobStatus: SUCCEEDED,
> >>resultCode: 0, result:
> >>rO0ABXNyAA5qYXZhLmxhbmcuTG9uZzuL5JDMjyPfAgABSgAFdmFsdWV4cgAQamF2YS5sYW5nL
> >>k
> >>5
> >>1bWJlcoaslR0LlOCLAgAAeHAAAAAAAAAKYA
> >>2018-09-13 06:04:30,970 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
> >>(logid:4c7a1c38) Publish async job-28021 complete on message bus
> >>2018-09-13 06:04:30,970 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
> >>(logid:4c7a1c38) Wake up jobs related to job-28021
> >>2018-09-13 06:04:30,970 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
> >>(logid:4c7a1c38) Update db status for job-28021
> >>2018-09-13 06:04:30,971 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
> >>(logid:4c7a1c38) Wake up jobs joined with job-28021 and disjoin all
> >>subjobs created from job- 28021
> >>2018-09-13 06:04:30,983 DEBUG [c.c.v.VmWorkJobDispatcher]
> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021) (logid:4c7a1c38)
> >>Done with run of VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot for
> >>VM
> >>228, job origin: 28020
> >>2018-09-13 06:04:30,983 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021) (logid:4c7a1c38)
> >>Done executing com.cloud.vm.VmWorkTakeVolumeSnapshot for job-28021
> >>2018-09-13 06:04:30,984 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021) (logid:4c7a1c38)
> >>Remove job-28021 from job monitoring
> >>2018-09-13 06:04:31,026 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6)
> >>(logid:4c7a1c38)
> >>Complete async job-28020, jobStatus: SUCCEEDED, resultCode: 0, result:
> >>org.apache.cloudstack.api.response.SnapshotResponse/snapshot/{"id":"3c866
> >>2
> >>b
> >>2-c76f-4e3a-a928-f76dad820122","account":"juha","domainid":"d3c2420f-7503
> >>-
> >>4
> >>560-a2da-d7e7149ce390","domain":"BGI","snapshottype":"DAILY","volumeid":"
> >>9
> >>6
> >>9c796d-072e-4272-adc6-93a492c03ea3","volumename":"ROOT-228","volumetype":
> >>"
> >>R
> >>OOT","created":"2018-09-13T06:02:15+0300","name":"BGI-Maria_ROOT-228_2018
> >>0
> >>9
> >>13030215","intervaltype":"DAILY","state":"BackedUp","physicalsize":234293
> >>2
> >>9
> >>92,"zoneid":"533c8f88-2637-4c0e-8210-e0a355b62d82","tags":[],"revertable"
> >>:
> >>f
> >>alse,"ostypeid":"da491b96-3748-11e5-a683-121326fde999","osdisplayname":"C
> >>e
> >>n
> >>tOS 6.5 (64-bit)","virtualsize":21474836480}
> >>2018-09-13 06:04:31,027 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6)
> >>(logid:4c7a1c38)
> >>Publish async job-28020 complete on message bus
> >>2018-09-13 06:04:31,027 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6)
> >>(logid:4c7a1c38)
> >>Wake up jobs related to job-28020
> >>2018-09-13 06:04:31,027 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6)
> >>(logid:4c7a1c38)
> >>Update db status for job-28020
> >>2018-09-13 06:04:31,028 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6)
> >>(logid:4c7a1c38)
> >>Wake up jobs joined with job-28020 and disjoin all subjobs created from
> >>job- 28020
> >>2018-09-13 06:04:31,039 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>(API-Job-Executor-83:ctx-cfb1cd95 job-28020) (logid:4c7a1c38) Done
> >>executing
> >>org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for
> >>job-28020
> >>2018-09-13 06:04:31,039 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> >>(API-Job-Executor-83:ctx-cfb1cd95 job-28020) (logid:4c7a1c38) Remove
> >>job-28020 from job monitoring
>
>
>
--
Rafael Weingärtner
Re: VM Snapshot not removed from primary SR
Posted by Sami Rajala <sa...@fapps.com>.
Hi,
I have got zero advice for this.
"I have had weird issue since I updated CS from 4.9 to 4.10.
CS does not delete/remove temporary snapshot from primary SR and I have
had to removed it by manual time to time, before 30 snapshot limit has
reached.
+ doesn't CS not update secondary_storege count on snapshot_count table”
Now I have a situation where Xenserver has 19 snapshot on primary SR for
each VM + resource count for ”secondary_storege” in database increase all
the time per account.
ACS storage cleaner removed old snapshots from secondary when they getting
old, but thats all, not decrease resource count for ”secondary_storege”
with value of deleted snapshot ( or not remove snapshot from primary SR
after snapshot has created and copied to secondary )
I have still delete primary snapshot by manual + give a new value for
”secondary_storege” before they reach limit.
has anyone got this kind of issue? any fix
BR
-sami
14.9.2018 12.26, "Sami Rajala (FAPPS)" <sa...@fapps.com>:
>Continue,
>
>I created template from VM-s latest snapshot after 3 days ( one parent +
>2
>child snapshot) - without errors/warning
>CS create VM from this template ok, but Xen could not start VM
>
>so, I return back to situation where every snapshot is full and manual
>removing snapshot from primary
>
>Has anyone any advice?
>
>BR
>-sami
>
>
>
>13.9.2018 7.56, "Sami Rajala (FAPPS)" <sa...@fapps.com>:
>
>>Hello
>>
>>I have had weird issue since I updated CS from 4.9 to 4.10.
>>CS does not delete/remove temporary snapshot from primary SR and I have
>>had to removed it by manual time to time, before 30 snapshot limit has
>>reached.
>>+ doesn¹t CS not update secondary_storege count on snapshot_count table
>>
>>It works fine on 4.9, but stop to work on 4.10 and I have wait
>>possibility
>>to update 4.11.1.
>>Update to 4.11.1 has done now and looks like this not work still
>>
>>I removed all snapshot and start over from clean table, snapshot policy
>>is: DAILY, keep 2, delta = 5
>>There are now 3 snapshot on primary SR and 3 on secondary SR
>>
>>The Environment is Xen 7 + CS 4.11.1 + NFS storage
>>
>>Is there some other parameter I should look and any other work around I
>>has to do to get it work ?
>>
>>Any hits?
>>
>>BR
>>-sami
>>
>>VM snapshot Log for last round:
>>
>>2018-09-13 06:02:15,983 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>(API-Job-Executor-83:ctx-cfb1cd95 job-28020) (logid:4c7a1c38) Executing
>>AsyncJobVO {id:28020, userId: 1, accountId: 7, instanceType: Snapshot,
>>instanceId: 2656, cmd:
>>org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd,
>>cmdInfo:
>>{"policyid":"17","ctxUserId":"1","volumeid":"238","ctxStartEventId":"1","
>>i
>>d
>>":"2656","ctxAccountId":"7"}, cmdVersion: 0, status: IN_PROGRESS,
>>processStatus: 0, resultCode: 0, result: null, initMsid: 19873467853209,
>>completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
>>2018-09-13 06:02:15,991 DEBUG [c.c.u.AccountManagerImpl]
>>(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6)
>>(logid:4c7a1c38)
>>Access to Acct[479c643e-9c84-41fb-9f0a-9bb999893a25-juha] granted to
>>Acct[479c643e-9c84-41fb-9f0a-9bb999893a25-juha] by DomainChecker
>>2018-09-13 06:02:16,045 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6)
>>(logid:4c7a1c38)
>>Sync job-28021 execution on object VmWorkJobQueue.228
>>2018-09-13 06:02:17,115 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021) (logid:4c7a1c38)
>>Executing AsyncJobVO {id:28021, userId: 1, accountId: 7, instanceType:
>>null, instanceId: null, cmd: com.cloud.vm.VmWorkTakeVolumeSnapshot,
>>cmdInfo:
>>rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrVGFrZVZvbHVtZVNuYXBzaG90BL5gG4Li1c8CA
>>A
>>Z
>>aAAthc3luY0JhY2t1cFoACXF1aWVzY2VWbUwADGxvY2F0aW9uVHlwZXQAKUxjb20vY2xvdWQv
>>c
>>3
>>RvcmFnZS9TbmFwc2hvdCRMb2NhdGlvblR5cGU7TAAIcG9saWN5SWR0ABBMamF2YS9sYW5nL0x
>>v
>>b
>>mc7TAAKc25hcHNob3RJZHEAfgACTAAIdm9sdW1lSWRxAH4AAnhyABNjb20uY2xvdWQudm0uVm
>>1
>>X
>>b3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlkTAALaGFuZGxlck5hb
>>W
>>V
>>0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAAHAAAAAAAAAAEAAAAAAAAA5HQAFFZvbHVt
>>Z
>>U
>>FwaVNlcnZpY2VJbXBsAABwc3IADmphdmEubGFuZy5Mb25nO4vkkMyPI98CAAFKAAV2YWx1ZXh
>>y
>>A
>>BBqYXZhLmxhbmcuTnVtYmVyhqyVHQuU4IsCAAB4cAAAAAAAAAARc3EAfgAHAAAAAAAACmBzcQ
>>B
>>-
>>AAcAAAAAAAAA7g, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0,
>>resultCode: 0, result: null, initMsid: 19873467853209, completeMsid:
>>null,
>>lastUpdated: null, lastPolled: null, created: Thu Sep 13 06:02:16 EEST
>>2018}
>>2018-09-13 06:02:17,116 DEBUG [c.c.v.VmWorkJobDispatcher]
>>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021) (logid:4c7a1c38)
>>Run VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot for VM 228, job
>>origin: 28020
>>2018-09-13 06:02:17,117 DEBUG [c.c.v.VmWorkJobHandlerProxy]
>>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>>(logid:4c7a1c38) Execute VM work job:
>>com.cloud.vm.VmWorkTakeVolumeSnapshot{"volumeId":238,"policyId":17,"snaps
>>h
>>o
>>tId":2656,"quiesceVm":false,"asyncBackup":false,"userId":1,"accountId":7,
>>"
>>v
>>mId":228,"handlerName":"VolumeApiServiceImpl"}
>>2018-09-13 06:02:17,197 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
>>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>>(logid:4c7a1c38) getCommandHostDelegation: class
>>org.apache.cloudstack.storage.command.CreateObjectCommand
>>2018-09-13 06:02:17,197 DEBUG [c.c.h.XenServerGuru]
>>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>>(logid:4c7a1c38) We are returning the default host to execute commands
>>because the command is not of Copy type.
>>2018-09-13 06:02:17,199 DEBUG [c.c.a.t.Request]
>>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>>(logid:4c7a1c38) Seq 128-4630826316843724436: Sending { Cmd , MgmtId:
>>19873467853209, via: 128(xenserver-15), Ver: v1, Flags: 100011,
>>[{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"o
>>r
>>g
>>.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"969c79
>>6
>>d
>>-072e-4272-adc6-93a492c03ea3","volumeType":"ROOT","dataStore":{"org.apach
>>e
>>.
>>cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"6568d30f-82b6-35a0-bfe
>>5
>>-
>>be17c1398ed8","id":2,"poolType":"NetworkFilesystem","host":"10.10.1.33","
>>p
>>a
>>th":"/pool3/primary3","port":2049,"url":"NetworkFilesystem://10.10.1.33/p
>>o
>>o
>>l3/primary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a0-bfe5-be17c1398ed8"
>>,
>>"
>>isManaged":false}},"name":"ROOT-228","size":21474836480,"path":"3d2a9716-
>>8
>>f
>>a1-497e-8753-941117a0ef50","volumeId":238,"vmName":"i-7-228-VM","accountI
>>d
>>"
>>:7,"format":"VHD","provisioningType":"THIN","id":238,"deviceId":0,"hyperv
>>i
>>s
>>orType":"XenServer"},"parentSnapshotPath":"c7b2824a-7402-4f66-b2e6-06ed45
>>2
>>0
>>72ad","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":
>>{
>>"
>>uuid":"6568d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType":"NetworkFi
>>l
>>e
>>system","host":"10.10.1.33","path":"/pool3/primary3","port":2049,"url":"N
>>e
>>t
>>workFilesystem://10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUID=6568d3
>>0
>>f
>>-82b6-35a0-bfe5-be17c1398ed8","isManaged":false}},"vmName":"i-7-228-VM","
>>n
>>a
>>me":"BGI-Maria_ROOT-228_20180913030215","hypervisorType":"XenServer","id"
>>:
>>2
>>656,"quiescevm":false,"parents":["c7b2824a-7402-4f66-b2e6-06ed452072ad"],
>>"
>>p
>>hysicalSize":0}},"wait":0}}] }
>>2018-09-13 06:02:17,199 DEBUG [c.c.a.t.Request]
>>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>>(logid:4c7a1c38) Seq 128-4630826316843724436: Executing: { Cmd , MgmtId:
>>19873467853209, via: 128(xenserver-15), Ver: v1, Flags: 100011,
>>[{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"o
>>r
>>g
>>.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"969c79
>>6
>>d
>>-072e-4272-adc6-93a492c03ea3","volumeType":"ROOT","dataStore":{"org.apach
>>e
>>.
>>cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"6568d30f-82b6-35a0-bfe
>>5
>>-
>>be17c1398ed8","id":2,"poolType":"NetworkFilesystem","host":"10.10.1.33","
>>p
>>a
>>th":"/pool3/primary3","port":2049,"url":"NetworkFilesystem://10.10.1.33/p
>>o
>>o
>>l3/primary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a0-bfe5-be17c1398ed8"
>>,
>>"
>>isManaged":false}},"name":"ROOT-228","size":21474836480,"path":"3d2a9716-
>>8
>>f
>>a1-497e-8753-941117a0ef50","volumeId":238,"vmName":"i-7-228-VM","accountI
>>d
>>"
>>:7,"format":"VHD","provisioningType":"THIN","id":238,"deviceId":0,"hyperv
>>i
>>s
>>orType":"XenServer"},"parentSnapshotPath":"c7b2824a-7402-4f66-b2e6-06ed45
>>2
>>0
>>72ad","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":
>>{
>>"
>>uuid":"6568d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType":"NetworkFi
>>l
>>e
>>system","host":"10.10.1.33","path":"/pool3/primary3","port":2049,"url":"N
>>e
>>t
>>workFilesystem://10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUID=6568d3
>>0
>>f
>>-82b6-35a0-bfe5-be17c1398ed8","isManaged":false}},"vmName":"i-7-228-VM","
>>n
>>a
>>me":"BGI-Maria_ROOT-228_20180913030215","hypervisorType":"XenServer","id"
>>:
>>2
>>656,"quiescevm":false,"parents":["c7b2824a-7402-4f66-b2e6-06ed452072ad"],
>>"
>>p
>>hysicalSize":0}},"wait":0}}] }
>>2018-09-13 06:02:19,815 DEBUG [c.c.a.m.DirectAgentAttache]
>>(DirectAgent-206:ctx-370f4cd9) (logid:4c7a1c38) Seq
>>128-4630826316843724436: Response Received:
>>2018-09-13 06:02:19,815 DEBUG [c.c.a.t.Request]
>>(DirectAgent-206:ctx-370f4cd9) (logid:4c7a1c38) Seq
>>128-4630826316843724436: Processing: { Ans: , MgmtId: 19873467853209,
>>via: 128(xenserver-15), Ver: v1, Flags: 10,
>>[{"org.apache.cloudstack.storage.command.CreateObjectAnswer":{"data":{"or
>>g
>>.
>>apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"e91fb041-779d-409
>>0
>>-
>>b14c-4f57f4cda54b","id":0,"quiescevm":false,"physicalSize":0}},"result":t
>>r
>>u
>>e,"wait":0}}] }
>>2018-09-13 06:02:19,815 DEBUG [c.c.a.t.Request]
>>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>>(logid:4c7a1c38) Seq 128-4630826316843724436: Received: { Ans: , MgmtId:
>>19873467853209, via: 128(xenserver-15), Ver: v1, Flags: 10, {
>>CreateObjectAnswer } }
>>2018-09-13 06:02:19,934 DEBUG [o.a.c.s.m.AncientDataMotionStrategy]
>>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>>(logid:4c7a1c38) copyAsync inspecting src type SNAPSHOT copyAsync
>>inspecting dest type SNAPSHOT
>>2018-09-13 06:02:20,017 ERROR [o.a.c.s.e.DefaultEndPointSelector]
>>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>>(logid:4c7a1c38) IR24 select BACKUPSNAPSHOT from primary to secondary
>>2656
>>dest=2656
>>2018-09-13 06:02:20,023 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
>>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>>(logid:4c7a1c38) getCommandHostDelegation: class
>>org.apache.cloudstack.storage.command.CopyCommand
>>2018-09-13 06:02:20,023 DEBUG [c.c.h.XenServerGuru]
>>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>>(logid:4c7a1c38) We are returning the default host to execute commands
>>because the source and destination objects are not NFS type.
>>2018-09-13 06:02:20,025 DEBUG [c.c.a.t.Request]
>>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>>(logid:4c7a1c38) Seq 130-7345089517264579763: Sending { Cmd , MgmtId:
>>19873467853209, via: 130(xenserver-81), Ver: v1, Flags: 100111,
>>[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apac
>>h
>>e
>>.cloudstack.storage.to.SnapshotObjectTO":{"path":"e91fb041-779d-4090-b14c
>>-
>>4
>>f57f4cda54b","volume":{"uuid":"969c796d-072e-4272-adc6-93a492c03ea3","vol
>>u
>>m
>>eType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataS
>>t
>>o
>>reTO":{"uuid":"6568d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType":"N
>>e
>>t
>>workFilesystem","host":"10.10.1.33","path":"/pool3/primary3","port":2049,
>>"
>>u
>>rl":"NetworkFilesystem://10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUI
>>D
>>=
>>6568d30f-82b6-35a0-bfe5-be17c1398ed8","isManaged":false}},"name":"ROOT-22
>>8
>>"
>>,"size":21474836480,"path":"3d2a9716-8fa1-497e-8753-941117a0ef50","volume
>>I
>>d
>>":238,"vmName":"i-7-228-VM","accountId":7,"format":"VHD","provisioningTyp
>>e
>>"
>>:"THIN","id":238,"deviceId":0,"hypervisorType":"XenServer"},"parentSnapsh
>>o
>>t
>>Path":"c7b2824a-7402-4f66-b2e6-06ed452072ad","dataStore":{"org.apache.clo
>>u
>>d
>>stack.storage.to.PrimaryDataStoreTO":{"uuid":"6568d30f-82b6-35a0-bfe5-be1
>>7
>>c
>>1398ed8","id":2,"poolType":"NetworkFilesystem","host":"10.10.1.33","path"
>>:
>>"
>>/pool3/primary3","port":2049,"url":"NetworkFilesystem://10.10.1.33/pool3/
>>p
>>r
>>imary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a0-bfe5-be17c1398ed8","isM
>>a
>>n
>>aged":false}},"vmName":"i-7-228-VM","name":"BGI-Maria_ROOT-228_2018091303
>>0
>>2
>>15","hypervisorType":"XenServer","id":2656,"quiescevm":false,"parents":["
>>c
>>7
>>b2824a-7402-4f66-b2e6-06ed452072ad"],"physicalSize":0}},"destTO":{"org.ap
>>a
>>c
>>he.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/7/238","vol
>>u
>>m
>>e":{"uuid":"969c796d-072e-4272-adc6-93a492c03ea3","volumeType":"ROOT","da
>>t
>>a
>>Store":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"65
>>6
>>8
>>d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType":"NetworkFilesystem","
>>h
>>o
>>st":"10.10.1.33","path":"/pool3/primary3","port":2049,"url":"NetworkFiles
>>y
>>s
>>tem://10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a
>>0
>>-
>>bfe5-be17c1398ed8","isManaged":false}},"name":"ROOT-228","size":214748364
>>8
>>0
>>,"path":"3d2a9716-8fa1-497e-8753-941117a0ef50","volumeId":238,"vmName":"i
>>-
>>7
>>-228-VM","accountId":7,"format":"VHD","provisioningType":"THIN","id":238,
>>"
>>d
>>eviceId":0,"hypervisorType":"XenServer"},"parentSnapshotPath":"snapshots/
>>7
>>/
>>238/0c8c6ee3-ce3d-4241-8509-5193d2cff491.vhd","dataStore":{"com.cloud.age
>>n
>>t
>>.api.to.NfsTO":{"_url":"nfs://10.10.1.34/pool8/secondary8","_role":"Image
>>"
>>}
>>},"vmName":"i-7-228-VM","name":"BGI-Maria_ROOT-228_20180913030215","hyper
>>v
>>i
>>sorType":"XenServer","id":2656,"quiescevm":false,"parents":["snapshots/7/
>>2
>>3
>>8/93a42896-0da1-4b82-8985-d28464f1ac4a.vhd","snapshots/7/238/0c8c6ee3-ce3
>>d
>>-
>>4241-8509-5193d2cff491.vhd"],"physicalSize":0}},"executeInSequence":true,
>>"
>>o
>>ptions":{"fullSnapshot":"false"},"options2":{},"wait":28800}}] }
>>2018-09-13 06:02:20,025 DEBUG [c.c.a.t.Request]
>>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>>(logid:4c7a1c38) Seq 130-7345089517264579763: Executing: { Cmd , MgmtId:
>>19873467853209, via: 130(xenserver-81), Ver: v1, Flags: 100111,
>>[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apac
>>h
>>e
>>.cloudstack.storage.to.SnapshotObjectTO":{"path":"e91fb041-779d-4090-b14c
>>-
>>4
>>f57f4cda54b","volume":{"uuid":"969c796d-072e-4272-adc6-93a492c03ea3","vol
>>u
>>m
>>eType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataS
>>t
>>o
>>reTO":{"uuid":"6568d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType":"N
>>e
>>t
>>workFilesystem","host":"10.10.1.33","path":"/pool3/primary3","port":2049,
>>"
>>u
>>rl":"NetworkFilesystem://10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUI
>>D
>>=
>>6568d30f-82b6-35a0-bfe5-be17c1398ed8","isManaged":false}},"name":"ROOT-22
>>8
>>"
>>,"size":21474836480,"path":"3d2a9716-8fa1-497e-8753-941117a0ef50","volume
>>I
>>d
>>":238,"vmName":"i-7-228-VM","accountId":7,"format":"VHD","provisioningTyp
>>e
>>"
>>:"THIN","id":238,"deviceId":0,"hypervisorType":"XenServer"},"parentSnapsh
>>o
>>t
>>Path":"c7b2824a-7402-4f66-b2e6-06ed452072ad","dataStore":{"org.apache.clo
>>u
>>d
>>stack.storage.to.PrimaryDataStoreTO":{"uuid":"6568d30f-82b6-35a0-bfe5-be1
>>7
>>c
>>1398ed8","id":2,"poolType":"NetworkFilesystem","host":"10.10.1.33","path"
>>:
>>"
>>/pool3/primary3","port":2049,"url":"NetworkFilesystem://10.10.1.33/pool3/
>>p
>>r
>>imary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a0-bfe5-be17c1398ed8","isM
>>a
>>n
>>aged":false}},"vmName":"i-7-228-VM","name":"BGI-Maria_ROOT-228_2018091303
>>0
>>2
>>15","hypervisorType":"XenServer","id":2656,"quiescevm":false,"parents":["
>>c
>>7
>>b2824a-7402-4f66-b2e6-06ed452072ad"],"physicalSize":0}},"destTO":{"org.ap
>>a
>>c
>>he.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/7/238","vol
>>u
>>m
>>e":{"uuid":"969c796d-072e-4272-adc6-93a492c03ea3","volumeType":"ROOT","da
>>t
>>a
>>Store":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"65
>>6
>>8
>>d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType":"NetworkFilesystem","
>>h
>>o
>>st":"10.10.1.33","path":"/pool3/primary3","port":2049,"url":"NetworkFiles
>>y
>>s
>>tem://10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a
>>0
>>-
>>bfe5-be17c1398ed8","isManaged":false}},"name":"ROOT-228","size":214748364
>>8
>>0
>>,"path":"3d2a9716-8fa1-497e-8753-941117a0ef50","volumeId":238,"vmName":"i
>>-
>>7
>>-228-VM","accountId":7,"format":"VHD","provisioningType":"THIN","id":238,
>>"
>>d
>>eviceId":0,"hypervisorType":"XenServer"},"parentSnapshotPath":"snapshots/
>>7
>>/
>>238/0c8c6ee3-ce3d-4241-8509-5193d2cff491.vhd","dataStore":{"com.cloud.age
>>n
>>t
>>.api.to.NfsTO":{"_url":"nfs://10.10.1.34/pool8/secondary8","_role":"Image
>>"
>>}
>>},"vmName":"i-7-228-VM","name":"BGI-Maria_ROOT-228_20180913030215","hyper
>>v
>>i
>>sorType":"XenServer","id":2656,"quiescevm":false,"parents":["snapshots/7/
>>2
>>3
>>8/93a42896-0da1-4b82-8985-d28464f1ac4a.vhd","snapshots/7/238/0c8c6ee3-ce3
>>d
>>-
>>4241-8509-5193d2cff491.vhd"],"physicalSize":0}},"executeInSequence":true,
>>"
>>o
>>ptions":{"fullSnapshot":"false"},"options2":{},"wait":28800}}] }
>>2018-09-13 06:04:28,895 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Host 10.10.1.81
>>OpaqueRef:f83331df-6634-63fd-6b95-44607a6069b8: Removing SR
>>2018-09-13 06:04:28,913 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Host 10.10.1.81
>>OpaqueRef:05380b03-4cb8-85a7-6a6a-a1595c188f60: Unplugging pbd
>>2018-09-13 06:04:30,881 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Host 10.10.1.81
>>OpaqueRef:f83331df-6634-63fd-6b95-44607a6069b8: Forgetting
>>2018-09-13 06:04:30,891 DEBUG [c.c.h.x.r.XenServerStorageProcessor]
>>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Successfully destroyed
>>snapshot on volume: 3d2a9716-8fa1-497e-8753-941117a0ef50 execept this
>>current snapshot e91fb041-779d-4090-b14c-4f57f4cda54b
>>2018-09-13 06:04:30,891 INFO [c.c.h.x.r.XenServerStorageProcessor]
>>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) New snapshot details:
>>SnapshotTO[datastore=null|volume=null|pathsnapshots/7/238/0560d536-c945-4
>>a
>>6
>>1-8c4c-dbaf29f29993.vhd]
>>2018-09-13 06:04:30,891 INFO [c.c.h.x.r.XenServerStorageProcessor]
>>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) New snapshot physical
>>utilization: 2342932992
>>2018-09-13 06:04:30,891 DEBUG [c.c.a.m.DirectAgentAttache]
>>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Seq
>>130-7345089517264579763: Response Received:
>>2018-09-13 06:04:30,891 DEBUG [c.c.a.t.Request]
>>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Seq
>>130-7345089517264579763: Processing: { Ans: , MgmtId: 19873467853209,
>>via: 130(xenserver-81), Ver: v1, Flags: 110,
>>[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.
>>a
>>p
>>ache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/7/238/056
>>0
>>d
>>536-c945-4a61-8c4c-dbaf29f29993.vhd","parentSnapshotPath":"snapshots/7/23
>>8
>>/
>>0c8c6ee3-ce3d-4241-8509-5193d2cff491.vhd","id":0,"quiescevm":false,"physi
>>c
>>a
>>lSize":2342932992}},"result":true,"wait":0}}] }
>>2018-09-13 06:04:30,891 DEBUG [c.c.a.m.AgentAttache]
>>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Seq
>>130-7345089517264579763: No more commands found
>>2018-09-13 06:04:30,891 DEBUG [c.c.a.t.Request]
>>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>>(logid:4c7a1c38) Seq 130-7345089517264579763: Received: { Ans: , MgmtId:
>>19873467853209, via: 130(xenserver-81), Ver: v1, Flags: 110, {
>>CopyCmdAnswer } }
>>2018-09-13 06:04:30,957 DEBUG [c.c.r.ResourceLimitManagerImpl]
>>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>>(logid:4c7a1c38) Updating resource Type = secondary_storage count for
>>Account = 7 Operation = decreasing Amount = 19131903488
>>2018-09-13 06:04:30,969 DEBUG [c.c.v.VmWorkJobHandlerProxy]
>>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>>(logid:4c7a1c38) Done executing VM work job:
>>com.cloud.vm.VmWorkTakeVolumeSnapshot{"volumeId":238,"policyId":17,"snaps
>>h
>>o
>>tId":2656,"quiesceVm":false,"asyncBackup":false,"userId":1,"accountId":7,
>>"
>>v
>>mId":228,"handlerName":"VolumeApiServiceImpl"}
>>2018-09-13 06:04:30,969 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>>(logid:4c7a1c38) Complete async job-28021, jobStatus: SUCCEEDED,
>>resultCode: 0, result:
>>rO0ABXNyAA5qYXZhLmxhbmcuTG9uZzuL5JDMjyPfAgABSgAFdmFsdWV4cgAQamF2YS5sYW5nL
>>k
>>5
>>1bWJlcoaslR0LlOCLAgAAeHAAAAAAAAAKYA
>>2018-09-13 06:04:30,970 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>>(logid:4c7a1c38) Publish async job-28021 complete on message bus
>>2018-09-13 06:04:30,970 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>>(logid:4c7a1c38) Wake up jobs related to job-28021
>>2018-09-13 06:04:30,970 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>>(logid:4c7a1c38) Update db status for job-28021
>>2018-09-13 06:04:30,971 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
>>(logid:4c7a1c38) Wake up jobs joined with job-28021 and disjoin all
>>subjobs created from job- 28021
>>2018-09-13 06:04:30,983 DEBUG [c.c.v.VmWorkJobDispatcher]
>>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021) (logid:4c7a1c38)
>>Done with run of VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot for
>>VM
>>228, job origin: 28020
>>2018-09-13 06:04:30,983 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021) (logid:4c7a1c38)
>>Done executing com.cloud.vm.VmWorkTakeVolumeSnapshot for job-28021
>>2018-09-13 06:04:30,984 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021) (logid:4c7a1c38)
>>Remove job-28021 from job monitoring
>>2018-09-13 06:04:31,026 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6)
>>(logid:4c7a1c38)
>>Complete async job-28020, jobStatus: SUCCEEDED, resultCode: 0, result:
>>org.apache.cloudstack.api.response.SnapshotResponse/snapshot/{"id":"3c866
>>2
>>b
>>2-c76f-4e3a-a928-f76dad820122","account":"juha","domainid":"d3c2420f-7503
>>-
>>4
>>560-a2da-d7e7149ce390","domain":"BGI","snapshottype":"DAILY","volumeid":"
>>9
>>6
>>9c796d-072e-4272-adc6-93a492c03ea3","volumename":"ROOT-228","volumetype":
>>"
>>R
>>OOT","created":"2018-09-13T06:02:15+0300","name":"BGI-Maria_ROOT-228_2018
>>0
>>9
>>13030215","intervaltype":"DAILY","state":"BackedUp","physicalsize":234293
>>2
>>9
>>92,"zoneid":"533c8f88-2637-4c0e-8210-e0a355b62d82","tags":[],"revertable"
>>:
>>f
>>alse,"ostypeid":"da491b96-3748-11e5-a683-121326fde999","osdisplayname":"C
>>e
>>n
>>tOS 6.5 (64-bit)","virtualsize":21474836480}
>>2018-09-13 06:04:31,027 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6)
>>(logid:4c7a1c38)
>>Publish async job-28020 complete on message bus
>>2018-09-13 06:04:31,027 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6)
>>(logid:4c7a1c38)
>>Wake up jobs related to job-28020
>>2018-09-13 06:04:31,027 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6)
>>(logid:4c7a1c38)
>>Update db status for job-28020
>>2018-09-13 06:04:31,028 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6)
>>(logid:4c7a1c38)
>>Wake up jobs joined with job-28020 and disjoin all subjobs created from
>>job- 28020
>>2018-09-13 06:04:31,039 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>(API-Job-Executor-83:ctx-cfb1cd95 job-28020) (logid:4c7a1c38) Done
>>executing
>>org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for
>>job-28020
>>2018-09-13 06:04:31,039 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>(API-Job-Executor-83:ctx-cfb1cd95 job-28020) (logid:4c7a1c38) Remove
>>job-28020 from job monitoring