You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Piotr Pisz <pi...@piszki.pl> on 2020/12/07 15:23:15 UTC
RE: snapshot is not working
Hello,
Andrija, Rakesh, can you elaborate and explain this: “it's a known issue and that's why the "path" is missing” ?
I have this bug all the time, snapshots are done correctly on Ceph / RBD but we can't make new templates from them.
Each attempt to copy a volume to Secondary Storage ends with this message (XML error: missing storage pool source).
Let's save ourselves by loading templates from external URLs, no problem here.
Best regards,
Piotr
From: Andrija Panic <an...@gmail.com>
Sent: Wednesday, November 18, 2020 1:08 AM
To: users <us...@cloudstack.apache.org>; Piotr Pisz <pi...@piszki.pl>
Subject: Re: snapshot is not working
that actually makes it broken with the default behaviour of copying the snapshots to Secondary Storage (NOT copying it to Sec Stor is supported/implemented ONLY for KVM/Ceph combo)
You might want to help, by raising a bug in the GitHub... it's supposed to work
Best.
On Mon, 9 Nov 2020 at 18:51, Piotr Pisz <piotr@piszki.pl <ma...@piszki.pl> > wrote:
Thank You Rakesh!
After set snapshot.to.secondary to FALSE snapshot on RBD work 😊
Best regards,
Piotr
-----Original Message-----
From: Rakesh v <www.rakeshv.com@gmail.com <ma...@gmail.com> >
Sent: Monday, November 9, 2020 5:34 PM
To: users@cloudstack.apache.org <ma...@cloudstack.apache.org>
Subject: Re: snapshot is not working
Is the global setting which contains the string "snaphot.to.secondary" is false? If so it's a known issue and that's why the "path" is missing
Sent from my iPhone
> On 09-Nov-2020, at 1:22 PM, Piotr Pisz <piotr@piszki.pl <ma...@piszki.pl> > wrote:
>
> Hello,
>
>
>
> I need help, snapshots are not working (see below), this is a new CS
> installation on Ceph RBD. Besides, everything works fine, could you
> tell us what might be wrong?
>
>
>
> Take snapshot: 544 failed
>
> com.cloud.utils.exception.CloudRuntimeException:
> com.cloud.utils.exception.CloudRuntimeException:
> org.libvirt.LibvirtException: XML error: missing storage pool source
> path
>
>
>
> ???
>
>
>
> Best regards,
>
> Piotr
>
>
>
>
>
> 2020-11-09 12:56:46,525 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (API-Job-Executor-35:ctx-326b32af job-1894) (logid:836f528d) Add
> job-1894 into job monitoring
>
> 2020-11-09 12:56:46,528 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (qtp1497845528-1209:ctx-a9fac85e ctx-7d1c63e1) (logid:a5ddc58d) submit
> async job-1894, details: AsyncJobVO {id:1894, userId: 2, accountId: 2,
> instanceType: Snapshot, instanceId: 13, cmd:
> org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd,
> cmdInfo:
> {"quiescevm":"false","httpmethod":"GET","ctxAccountId":"2","uuid":"82f
> 3faf3-e09c-4eaf-886c-d81bad998b64","cmdEventType":"SNAPSHOT.CREATE","a
> syncBackup":"false","response":"json","ctxUserId":"2","volumeId":"9945
> a23b-a376-4f85-b378-9f7da8017dac","name":"test","ctxStartEventId":"427
> 0","id":"13","ctxDetails":"{\"interface
> com.cloud.storage.Volume\":\"9945a23b-a376-4f85-b378-9f7da8017dac\",\"
> interface
> com.cloud.storage.Snapshot\":\"82f3faf3-e09c-4eaf-886c-d81bad998b64\"}
> ","_":"1604922798590"},
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> result: null, initMsid: 176206389407942, completeMsid: null, lastUpdated:
> null, lastPolled: null, created: null, removed: null}
>
> 2020-11-09 12:56:46,528 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-35:ctx-326b32af job-1894) (logid:db479fbc) Executing
> AsyncJobVO {id:1894, userId: 2, accountId: 2, instanceType: Snapshot,
> instanceId: 13, cmd:
> org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd,
> cmdInfo:
> {"quiescevm":"false","httpmethod":"GET","ctxAccountId":"2","uuid":"82f
> 3faf3-e09c-4eaf-886c-d81bad998b64","cmdEventType":"SNAPSHOT.CREATE","a
> syncBackup":"false","response":"json","ctxUserId":"2","volumeId":"9945
> a23b-a376-4f85-b378-9f7da8017dac","name":"test","ctxStartEventId":"427
> 0","id":"13","ctxDetails":"{\"interface
> com.cloud.storage.Volume\":\"9945a23b-a376-4f85-b378-9f7da8017dac\",\"
> interface
> com.cloud.storage.Snapshot\":\"82f3faf3-e09c-4eaf-886c-d81bad998b64\"}
> ","_":"1604922798590"},
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> result: null, initMsid: 176206389407942, completeMsid: null, lastUpdated:
> null, lastPolled: null, created: null, removed: null}
>
> 2020-11-09 12:56:46,528 DEBUG [c.c.a.ApiServlet]
> (qtp1497845528-1209:ctx-a9fac85e ctx-7d1c63e1) (logid:a5ddc58d)
> ===END===
> 10.89.11.31 -- GET
> command=createSnapshot&response=json&volumeId=9945a23b-a376-4f85-b378-
> 9f7da8017dac&quiescevm=false&asyncBackup=false&name=test&_=16049227985
> 90
>
> 2020-11-09 12:56:46,549 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-35:ctx-326b32af job-1894 ctx-f8b08221)
> (logid:db479fbc) Sync job-1897 execution on object VmWorkJobQueue.183
>
> 2020-11-09 12:56:46,850 DEBUG [c.c.a.m.ClusteredAgentAttache]
> (AgentManager-Handler-13:null) (logid:) Seq 1-8981303556883622697:
> Routing from 176206389408022
>
> 2020-11-09 12:56:46,851 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentManager-Handler-6:null) (logid:) Seq 1-8981303556883622697:
> MgmtId
> 176206389408022: Resp: Routing to peer
>
> 2020-11-09 12:56:47,086 DEBUG [c.c.a.m.ClusteredAgentAttache]
> (AgentManager-Handler-3:null) (logid:) Seq 43-2584503236157253953:
> Routing from 176206389408022
>
> 2020-11-09 12:56:47,126 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentManager-Handler-14:null) (logid:) Seq 43-2584503236157253953:
> MgmtId
> 176206389408022: Resp: Routing to peer
>
> 2020-11-09 12:56:47,131 DEBUG [c.c.a.m.ClusteredAgentAttache]
> (AgentManager-Handler-15:null) (logid:) Seq 46-3634967849241411607:
> Routing from 176206389408022
>
> 2020-11-09 12:56:47,133 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentManager-Handler-10:null) (logid:) Seq 46-3634967849241411607:
> MgmtId
> 176206389408022: Resp: Routing to peer
>
> 2020-11-09 12:56:47,181 DEBUG [c.c.s.StatsCollector]
> (StatsCollector-6:ctx-100fc604) (logid:fde56d09) StorageCollector is
> running...
>
> 2020-11-09 12:56:47,184 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
> (StatsCollector-6:ctx-100fc604) (logid:fde56d09) getCommandHostDelegation:
> class com.cloud.agent.api.GetStorageStatsCommand
>
> 2020-11-09 12:56:47,184 DEBUG [c.c.h.XenServerGuru]
> (StatsCollector-6:ctx-100fc604) (logid:fde56d09) We are returning the
> default host to execute commands because the command is not of Copy type.
>
> 2020-11-09 12:56:47,230 DEBUG [c.c.a.t.Request]
> (StatsCollector-6:ctx-100fc604) (logid:fde56d09) Seq
> 58-4654188739910703033: Received: { Ans: , MgmtId: 176206389407942, via:
> 58(s-10-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
>
> 2020-11-09 12:56:47,231 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
> (StatsCollector-6:ctx-100fc604) (logid:fde56d09) getCommandHostDelegation:
> class com.cloud.agent.api.GetStorageStatsCommand
>
> 2020-11-09 12:56:47,231 DEBUG [c.c.h.XenServerGuru]
> (StatsCollector-6:ctx-100fc604) (logid:fde56d09) We are returning the
> default host to execute commands because the command is not of Copy type.
>
> 2020-11-09 12:56:47,232 DEBUG [c.c.a.m.ClusteredAgentAttache]
> (StatsCollector-6:ctx-100fc604) (logid:fde56d09) Seq
> 37-1383449510532885592: Forwarding null to 176206389407982
>
> 2020-11-09 12:56:47,406 DEBUG [c.c.a.ApiServlet]
> (qtp1497845528-1280:ctx-b087af82) (logid:3a8983a6) ===START===
> 10.89.11.31 -- GET
> command=listInfrastructure&response=json&_=1604922838197
>
> 2020-11-09 12:56:47,410 DEBUG [c.c.a.ApiServer]
> (qtp1497845528-1280:ctx-b087af82 ctx-052d26fb) (logid:3a8983a6) CIDRs
> from which account 'Acct[c3dd8977-12ba-11eb-9854-a0423f44350c-admin]'
> is allowed to perform API calls: 0.0.0.0/0,::/0 <http://0.0.0.0/0,::/0>
>
> 2020-11-09 12:56:47,416 DEBUG [c.c.a.ApiServlet]
> (qtp1497845528-1280:ctx-b087af82 ctx-052d26fb) (logid:3a8983a6)
> ===END===
> 10.89.11.31 -- GET
> command=listInfrastructure&response=json&_=1604922838197
>
> 2020-11-09 12:56:47,634 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (AsyncJobMgr-Heartbeat-1:ctx-ccb57103) (logid:56d3b696) Execute
> sync-queue
> item: SyncQueueItemVO {id:841, queueId: 601, contentType: AsyncJob,
> contentId: 1897, lastProcessMsid: 176206389407942, lastprocessNumber:
> 5,
> lastProcessTime: Mon Nov 09 12:56:47 CET 2020, created: Mon Nov 09
> 12:56:46 CET 2020}
>
> 2020-11-09 12:56:47,635 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (AsyncJobMgr-Heartbeat-1:ctx-ccb57103) (logid:56d3b696) Schedule
> queued
> job-1897
>
> 2020-11-09 12:56:47,637 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897) (logid:1f589139)
> Add
> job-1897 into job monitoring
>
> 2020-11-09 12:56:47,640 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897) (logid:db479fbc)
> Executing AsyncJobVO {id:1897, userId: 2, accountId: 2, instanceType:
> null, instanceId: null, cmd: com.cloud.vm.VmWorkTakeVolumeSnapshot,
> cmdInfo:
> rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrVGFrZVZvbHVtZVNuYXBzaG90BL5gG4Li1c
> 8CAAZaAAthc3luY0JhY2t1cFoACXF1aWVzY2VWbUwADGxvY2F0aW9uVHlwZXQAKUxjb20v
> Y2xvdWQvc3RvcmFnZS9TbmFwc2hvdCRMb2NhdGlvblR5cGU7TAAIcG9saWN5SWR0ABBMam
> F2YS9sYW5nL0xvbmc7TAAKc25hcHNob3RJZHEAfgACTAAIdm9sdW1lSWRxAH4AAnhyABNj
> b20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAA
> R2bUlkTAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAACAAAA
> AAAAAAIAAAAAAAAAt3QAFFZvbHVtZUFwaVNlcnZpY2VJbXBsAABwc3IADmphdmEubGFuZy
> 5Mb25nO4vkkMyPI98CAAFKAAV2YWx1ZXhyABBqYXZhLmxhbmcuTnVtYmVyhqyVHQuU4IsC
> AAB4cAAAAAAAAAAAc3EAfgAHAAAAAAAAAA1zcQB-AAcAAAAAAAACIA,
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> result: null, initMsid: 176206389407942, completeMsid: null, lastUpdated:
> null, lastPolled: null, created: Mon Nov 09 12:56:46 CET 2020, removed:
> null}
>
> 2020-11-09 12:56:47,640 DEBUG [c.c.v.VmWorkJobDispatcher]
> (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897) (logid:db479fbc)
> Run VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot for VM 183, job origin:
> 1894
>
> 2020-11-09 12:56:47,642 DEBUG [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897 ctx-5fb6d0f8)
> (logid:db479fbc) Execute VM work job:
> com.cloud.vm.VmWorkTakeVolumeSnapshot{"volumeId":544,"policyId":0,"sna
> pshotId":13,"quiesceVm":false,"asyncBackup":false,"userId":2,"accountI
> d":2,"vmId":183,"handlerName":"VolumeApiServiceImpl"}
>
> 2020-11-09 12:56:47,675 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
> (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897 ctx-5fb6d0f8)
> (logid:db479fbc) getCommandHostDelegation: class
> org.apache.cloudstack.storage.command.CreateObjectCommand
>
> 2020-11-09 12:56:47,675 DEBUG [c.c.h.XenServerGuru]
> (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897 ctx-5fb6d0f8)
> (logid:db479fbc) We are returning the default host to execute commands
> because the command is not of Copy type.
>
> 2020-11-09 12:56:47,676 DEBUG [c.c.a.m.ClusteredAgentAttache]
> (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897 ctx-5fb6d0f8)
> (logid:db479fbc) Seq 40-1407656358530002013: Forwarding Seq
> 40-1407656358530002013: { Cmd , MgmtId: 176206389407942, via:
> 40(srv-oz-r08-n08.cenagis.local), Ver: v1, Flags: 100011,
> [{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":
> {"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid"
> :"9945a23b-a376-4f85-b378-9f7da8017dac","volumeType":"ROOT","dataStore
> ":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"22d1
> 051a-86dc-3613-961d-f909245f5477","id":4,"poolType":"RBD","host":"rbd.
> cenagis.local","path":"rbd","port":6789,"url":"RBD://rbd.cenagis.local
> /rbd/?ROLE=Primary&STOREUUID=22d1051a-86dc-3613-961d-f909245f5477","is
> Managed":false}},"name":"ROOT-183","size":107374182400,"path":"9945a23
> b-a376-4f85-b378-9f7da8017dac","volumeId":544,"vmName":"i-2-183-VM","a
> ccountId":2,"format":"RAW","provisioningType":"THIN","id":544,"deviceI
> d":0,"cacheMode":"WRITEBACK","hypervisorType":"KVM","directDownload":f
> alse},"dataStore":{"org.apache.cloudstack.storage.to <http://apache.cloudstack.storage.to> .PrimaryDataStoreT
> O":{"uuid":"22d1051a-86dc-3613-961d-f909245f5477","id":4,"poolType":"R
> BD","host":"rbd.cenagis.local","path":"rbd","port":6789,"url":"RBD://r
> bd.cenagis.local/rbd/?ROLE=Primary&STOREUUID=22d1051a-86dc-3613-961d-f
> 909245f5477","isManaged":false}},"vmName":"i-2-183-VM","name":"test","
> hypervisorType":"KVM","id":13,"quiescevm":false,"physicalSize":0}},"wa
> it":0}}]
> } to 176206389407982
>
> 2020-11-09 12:56:47,976 DEBUG [c.c.a.m.ClusteredAgentAttache]
> (AgentManager-Handler-1:null) (logid:) Seq 1-8603282663161211720:
> Routing from 176206389407982
>
> 2020-11-09 12:56:47,982 DEBUG [c.c.a.m.ClusteredAgentAttache]
> (AgentManager-Handler-4:null) (logid:) Seq 1-8981303556883622698:
> Routing from 176206389408022
>
> 2020-11-09 12:56:48,043 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> (consoleproxy-1:ctx-79a33e59) (logid:4c5fdc64) Zone 1 is ready to
> launch console proxy
>
> 2020-11-09 12:56:48,150 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
> (secstorage-1:ctx-c8a78ea1) (logid:473dea1d) Zone 1 is ready to launch
> secondary storage VM
>
> 2020-11-09 12:56:48,190 DEBUG [c.c.a.t.Request]
> (StatsCollector-5:ctx-e6db6758) (logid:451965d6) Seq
> 1-8552617167353293650: Received: { Ans: , MgmtId: 176206389407942, via:
> 1(srv-oz-r01-n01.cenagis.local), Ver: v1, Flags: 10, {
> GetVmStatsAnswer } }
>
> 2020-11-09 12:56:48,190 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentManager-Handler-5:null) (logid:) Seq 1-8603282663161211720:
> MgmtId
> 176206389407982: Resp: Routing to peer
>
> 2020-11-09 12:56:48,198 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentManager-Handler-11:null) (logid:) Seq 1-8981303556883622698:
> MgmtId
> 176206389408022: Resp: Routing to peer
>
> 2020-11-09 12:56:48,212 DEBUG [c.c.a.t.Request]
> (AgentManager-Handler-12:null) (logid:) Seq 40-1407656358530002013:
> Processing: { Ans: , MgmtId: 176206389407942, via: 40, Ver: v1, Flags:
> 10,
> [{"org.apache.cloudstack.storage.command.CreateObjectAnswer":{"data":{
> "org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"rbd/9945a
> 23b-a376-4f85-b378-9f7da8017dac/f957d3a8-702a-45aa-9e42-5222c223ef19",
> "id":0,"quiescevm":false,"physicalSize":0}},"result":true,"wait":0}}]
> }
>
> 2020-11-09 12:56:48,212 DEBUG [c.c.a.t.Request]
> (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897 ctx-5fb6d0f8)
> (logid:db479fbc) Seq 40-1407656358530002013: Received: { Ans: , MgmtId:
> 176206389407942, via: 40(srv-oz-r08-n08.cenagis.local), Ver: v1, Flags:
> 10, { CreateObjectAnswer } }
>
> 2020-11-09 12:56:48,226 DEBUG [c.c.a.m.ClusteredAgentAttache]
> (StatsCollector-5:ctx-e6db6758) (logid:451965d6) Seq
> 22-6063533948300963066: Forwarding null to 176206389407982
>
> 2020-11-09 12:56:48,248 DEBUG [o.a.c.s.m.AncientDataMotionStrategy]
> (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897 ctx-5fb6d0f8)
> (logid:db479fbc) copyAsync inspecting src type SNAPSHOT copyAsync
> inspecting dest type SNAPSHOT
>
> 2020-11-09 12:56:48,249 DEBUG [c.c.a.ApiServlet]
> (qtp1497845528-1209:ctx-4e81cf12) (logid:2f3c90fe) ===START===
> 10.89.11.31 -- GET
> command=listHosts&response=json&type=routing&listAll=true&page=1&pages
> ize=20&_=1604922838198
>
> 2020-11-09 12:56:48,253 DEBUG [c.c.a.ApiServer]
> (qtp1497845528-1209:ctx-4e81cf12 ctx-0b68411a) (logid:2f3c90fe) CIDRs
> from which account 'Acct[c3dd8977-12ba-11eb-9854-a0423f44350c-admin]'
> is allowed to perform API calls: 0.0.0.0/0,::/0 <http://0.0.0.0/0,::/0>
>
> 2020-11-09 12:56:48,254 WARN [c.c.a.d.ParamGenericValidationWorker]
> (qtp1497845528-1209:ctx-4e81cf12 ctx-0b68411a) (logid:2f3c90fe)
> Received unknown parameters for command listHosts. Unknown parameters
> : listall
>
> 2020-11-09 12:56:48,254 DEBUG [c.c.a.q.QueryManagerImpl]
> (qtp1497845528-1209:ctx-4e81cf12 ctx-0b68411a) (logid:2f3c90fe)
> >>>Searching for hosts>>>
>
> 2020-11-09 12:56:48,259 DEBUG [c.c.a.q.QueryManagerImpl]
> (qtp1497845528-1209:ctx-4e81cf12 ctx-0b68411a) (logid:2f3c90fe)
> >>>Generating Response>>>
>
> 2020-11-09 12:56:48,293 DEBUG [c.c.a.ApiServlet]
> (qtp1497845528-1209:ctx-4e81cf12 ctx-0b68411a) (logid:2f3c90fe)
> ===END===
> 10.89.11.31 -- GET
> command=listHosts&response=json&type=routing&listAll=true&page=1&pages
> ize=20&_=1604922838198
>
> 2020-11-09 12:56:48,298 ERROR [o.a.c.s.e.DefaultEndPointSelector]
> (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897 ctx-5fb6d0f8)
> (logid:db479fbc) IR24 select BACKUPSNAPSHOT from primary to secondary
> 13
> dest=13
>
> 2020-11-09 12:56:48,302 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
> (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897 ctx-5fb6d0f8)
> (logid:db479fbc) getCommandHostDelegation: class
> org.apache.cloudstack.storage.command.CopyCommand
>
> 2020-11-09 12:56:48,303 DEBUG [c.c.a.m.ClusteredAgentAttache]
> (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897 ctx-5fb6d0f8)
> (logid:db479fbc) Seq 49-6537256334104996965: Forwarding Seq
> 49-6537256334104996965: { Cmd , MgmtId: 176206389407942, via:
> 49(srv-oz-r11-n11.cenagis.local), Ver: v1, Flags: 100011,
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.a
> pache.cloudstack.storage.to <http://pache.cloudstack.storage.to> .SnapshotObjectTO":{"path":"rbd/9945a23b-a3
> 76-4f85-b378-9f7da8017dac/f957d3a8-702a-45aa-9e42-5222c223ef19","volum
> e":{"uuid":"9945a23b-a376-4f85-b378-9f7da8017dac","volumeType":"ROOT",
> "dataStore":{"org.apache.cloudstack.storage.to <http://cloudstack.storage.to> .PrimaryDataStoreTO":{"u
> uid":"22d1051a-86dc-3613-961d-f909245f5477","id":4,"poolType":"RBD","h
> ost":"rbd.cenagis.local","path":"rbd","port":6789,"url":"RBD://rbd.cen
> agis.local/rbd/?ROLE=Primary&STOREUUID=22d1051a-86dc-3613-961d-f909245
> f5477","isManaged":false}},"name":"ROOT-183","size":107374182400,"path
> ":"9945a23b-a376-4f85-b378-9f7da8017dac","volumeId":544,"vmName":"i-2-
> 183-VM","accountId":2,"format":"RAW","provisioningType":"THIN","id":54
> 4,"deviceId":0,"cacheMode":"WRITEBACK","hypervisorType":"KVM","directD
> ownload":false},"dataStore":{"org.apache.cloudstack.storage.to.Primary
> DataStoreTO":{"uuid":"22d1051a-86dc-3613-961d-f909245f5477","id":4,"po
> olType":"RBD","host":"rbd.cenagis.local","path":"rbd","port":6789,"url
> ":"RBD://rbd.cenagis.local/rbd/?ROLE=Primary&STOREUUID=22d1051a-86dc-3
> 613-961d-f909245f5477","isManaged":false}},"vmName":"i-2-183-VM","name
> ":"test","hypervisorType":"KVM","id":13,"quiescevm":false,"physicalSiz
> e":0}},"destTO":{"org.apache.cloudstack.storage.to <http://cloudstack.storage.to> .SnapshotObjectTO":{
> "path":"snapshots/2/544","volume":{"uuid":"9945a23b-a376-4f85-b378-9f7
> da8017dac","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.sto
> rage.to.PrimaryDataStoreTO":{"uuid":"22d1051a-86dc-3613-961d-f909245f5
> 477","id":4,"poolType":"RBD","host":"rbd.cenagis.local","path":"rbd","
> port":6789,"url":"RBD://rbd.cenagis.local/rbd/?ROLE=Primary&STOREUUID=
> 22d1051a-86dc-3613-961d-f909245f5477","isManaged":false}},"name":"ROOT
> -183","size":107374182400,"path":"9945a23b-a376-4f85-b378-9f7da8017dac
> ","volumeId":544,"vmName":"i-2-183-VM","accountId":2,"format":"RAW","p
> rovisioningType":"THIN","id":544,"deviceId":0,"cacheMode":"WRITEBACK",
> "hypervisorType":"KVM","directDownload":false},"dataStore":{"com.cloud
> .agent.api.to.NfsTO":{"_url":"nfs://sys2.cenagis.local/","_role":"Imag
> e"}},"vmName":"i-2-183-VM","name":"test","hypervisorType":"KVM","id":1
> 3,"quiescevm":false,"physicalSize":0}},"executeInSequence":false,"opti
> ons":{"fullSnapshot":"true"},"options2":{},"wait":216000}}]
> } to 176206389407982
>
> 2020-11-09 12:56:48,364 DEBUG [c.c.a.t.Request]
> (AgentManager-Handler-7:null) (logid:) Seq 49-6537256334104996965:
> Processing: { Ans: , MgmtId: 176206389407942, via: 49, Ver: v1, Flags:
> 10,
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException:
> org.libvirt.LibvirtException: XML error: missing storage pool source
> path","wait":0}}] }
>
> 2020-11-09 12:56:48,364 DEBUG [c.c.a.t.Request]
> (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897 ctx-5fb6d0f8)
> (logid:db479fbc) Seq 49-6537256334104996965: Received: { Ans: , MgmtId:
> 176206389407942, via: 49(srv-oz-r11-n11.cenagis.local), Ver: v1, Flags:
> 10, { CopyCmdAnswer } }
>
> 2020-11-09 12:56:48,373 DEBUG [c.c.s.s.SnapshotManagerImpl]
> (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897 ctx-5fb6d0f8)
> (logid:db479fbc) Failed to create
> snapshotcom.cloud.utils.exception.CloudRuntimeException:
> org.libvirt.LibvirtException: XML error: missing storage pool source
> path
>
> 2020-11-09 12:56:48,373 DEBUG [c.c.r.ResourceLimitManagerImpl]
> (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897 ctx-5fb6d0f8)
> (logid:db479fbc) Updating resource Type = snapshot count for Account =
> 2 Operation = decreasing Amount = 1
>
> 2020-11-09 12:56:48,375 DEBUG [c.c.r.ResourceLimitManagerImpl]
> (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897 ctx-5fb6d0f8)
> (logid:db479fbc) Updating resource Type = secondary_storage count for
> Account = 2 Operation = decreasing Amount = 107374182400
>
> 2020-11-09 12:56:48,377 ERROR [o.a.c.s.v.VolumeServiceImpl]
> (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897 ctx-5fb6d0f8)
> (logid:db479fbc) Take snapshot: 544 failed
>
> com.cloud.utils.exception.CloudRuntimeException:
> com.cloud.utils.exception.CloudRuntimeException:
> org.libvirt.LibvirtException: XML error: missing storage pool source
> path
>
> at
> org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnaps
> hot(SnapshotServiceImpl.java:301)
>
> at
> org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.backupS
> napshot(DefaultSnapshotStrategy.java:171)
>
> at
> com.cloud.storage.snapshot.SnapshotManagerImpl.backupSnapshotToSeconda
> ry(SnapshotManagerImpl.java:1213)
>
> at
> com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotMa
> nagerImpl.java:1164)
>
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
>
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeM
> ethodAccessorImpl.java:62)
>
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Del
> egatingMethodAccessorImpl.java:43)
>
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflectio
> n(AopUtils.java:344)
>
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoi
> npoint(ReflectiveMethodInvocation.java:198)
>
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(R
> eflectiveMethodInvocation.java:163)
>
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke
> (ExposeInvocationInterceptor.java:95)
>
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(R
> eflectiveMethodInvocation.java:186)
>
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamic
> AopProxy.java:212)
>
> at com.sun.proxy.$Proxy205.takeSnapshot(Unknown Source)
>
> at
> org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(Vo
> lumeServiceImpl.java:2073)
>
> at
> com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(V
> olumeApiServiceImpl.java:2541)
>
> at
> com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(V
> olumeApiServiceImpl.java:3465)
>
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
>
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeM
> ethodAccessorImpl.java:62)
>
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Del
> egatingMethodAccessorImpl.java:43)
>
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>
> at
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerPro
> xy.java:107)
>
> at
> com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServic
> eImpl.java:3471)
>
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
>
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeM
> ethodAccessorImpl.java:62)
>
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Del
> egatingMethodAccessorImpl.java:43)
>
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflectio
> n(AopUtils.java:344)
>
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoi
> npoint(ReflectiveMethodInvocation.java:198)
>
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(R
> eflectiveMethodInvocation.java:163)
>
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke
> (ExposeInvocationInterceptor.java:95)
>
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(R
> eflectiveMethodInvocation.java:186)
>
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamic
> AopProxy.java:212)
>
> at com.sun.proxy.$Proxy212.handleVmWorkJob(Unknown Source)
>
> at
> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInC
> ontext(AsyncJobManagerImpl.java:603)
>
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(Man
> agedContextRunnable.java:49)
>
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.cal
> l(DefaultManagedContext.java:56)
>
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callW
> ithContext(DefaultManagedContext.java:103)
>
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWi
> thContext(DefaultManagedContext.java:53)
>
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Manag
> edContextRunnable.java:46)
>
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(As
> yncJobManagerImpl.java:551)
>
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executor
> s.java:515)
>
> at
> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> Executor.java:1128)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> lExecutor.java:628)
>
> at java.base/java.lang.Thread.run(Thread.java:834)
>
> 2020-11-09 12:56:48,378 ERROR [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897 ctx-5fb6d0f8)
> (logid:db479fbc) Invocation exception, caused by:
> com.cloud.utils.exception.CloudRuntimeException:
> com.cloud.utils.exception.CloudRuntimeException:
> org.libvirt.LibvirtException: XML error: missing storage pool source
> path
>
> 2020-11-09 12:56:48,378 INFO [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897 ctx-5fb6d0f8)
> (logid:db479fbc) Rethrow exception
> com.cloud.utils.exception.CloudRuntimeException:
> com.cloud.utils.exception.CloudRuntimeException:
> org.libvirt.LibvirtException: XML error: missing storage pool source
> path
>
> 2020-11-09 12:56:48,378 DEBUG [c.c.v.VmWorkJobDispatcher]
> (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897) (logid:db479fbc)
> Done with run of VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot
> for VM 183, job origin: 1894
>
> 2020-11-09 12:56:48,378 ERROR [c.c.v.VmWorkJobDispatcher]
> (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897) (logid:db479fbc)
> Unable to complete AsyncJobVO {id:1897, userId: 2, accountId: 2,
> instanceType: null, instanceId: null, cmd:
> com.cloud.vm.VmWorkTakeVolumeSnapshot, cmdInfo:
> rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrVGFrZVZvbHVtZVNuYXBzaG90BL5gG4Li1c
> 8CAAZaAAthc3luY0JhY2t1cFoACXF1aWVzY2VWbUwADGxvY2F0aW9uVHlwZXQAKUxjb20v
> Y2xvdWQvc3RvcmFnZS9TbmFwc2hvdCRMb2NhdGlvblR5cGU7TAAIcG9saWN5SWR0ABBMam
> F2YS9sYW5nL0xvbmc7TAAKc25hcHNob3RJZHEAfgACTAAIdm9sdW1lSWRxAH4AAnhyABNj
> b20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAA
> R2bUlkTAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAACAAAA
> AAAAAAIAAAAAAAAAt3QAFFZvbHVtZUFwaVNlcnZpY2VJbXBsAABwc3IADmphdmEubGFuZy
> 5Mb25nO4vkkMyPI98CAAFKAAV2YWx1ZXhyABBqYXZhLmxhbmcuTnVtYmVyhqyVHQuU4IsC
> AAB4cAAAAAAAAAAAc3EAfgAHAAAAAAAAAA1zcQB-AAcAAAAAAAACIA,
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> result: null, initMsid: 176206389407942, completeMsid: null, lastUpdated:
> null, lastPolled: null, created: Mon Nov 09 12:56:46 CET 2020, removed:
> null}, job origin:1894
>
> com.cloud.utils.exception.CloudRuntimeException:
> com.cloud.utils.exception.CloudRuntimeException:
> org.libvirt.LibvirtException: XML error: missing storage pool source
> path
>
> at
> org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnaps
> hot(SnapshotServiceImpl.java:301)
>
> at
> org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.backupS
> napshot(DefaultSnapshotStrategy.java:171)
>
> at
> com.cloud.storage.snapshot.SnapshotManagerImpl.backupSnapshotToSeconda
> ry(SnapshotManagerImpl.java:1213)
>
> at
> com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotMa
> nagerImpl.java:1164)
>
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
>
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeM
> ethodAccessorImpl.java:62)
>
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Del
> egatingMethodAccessorImpl.java:43)
>
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflectio
> n(AopUtils.java:344)
>
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoi
> npoint(ReflectiveMethodInvocation.java:198)
>
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(R
> eflectiveMethodInvocation.java:163)
>
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke
> (ExposeInvocationInterceptor.java:95)
>
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(R
> eflectiveMethodInvocation.java:186)
>
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamic
> AopProxy.java:212)
>
> at com.sun.proxy.$Proxy205.takeSnapshot(Unknown Source)
>
> at
> org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(Vo
> lumeServiceImpl.java:2073)
>
> at
> com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(V
> olumeApiServiceImpl.java:2541)
>
> at
> com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(V
> olumeApiServiceImpl.java:3465)
>
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
>
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeM
> ethodAccessorImpl.java:62)
>
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Del
> egatingMethodAccessorImpl.java:43)
>
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>
> at
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerPro
> xy.java:107)
>
> at
> com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServic
> eImpl.java:3471)
>
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
>
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeM
> ethodAccessorImpl.java:62)
>
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Del
> egatingMethodAccessorImpl.java:43)
>
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflectio
> n(AopUtils.java:344)
>
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoi
> npoint(ReflectiveMethodInvocation.java:198)
>
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(R
> eflectiveMethodInvocation.java:163)
>
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke
> (ExposeInvocationInterceptor.java:95)
>
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(R
> eflectiveMethodInvocation.java:186)
>
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamic
> AopProxy.java:212)
>
> at com.sun.proxy.$Proxy212.handleVmWorkJob(Unknown Source)
>
> at
> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInC
> ontext(AsyncJobManagerImpl.java:603)
>
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(Man
> agedContextRunnable.java:49)
>
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.cal
> l(DefaultManagedContext.java:56)
>
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callW
> ithContext(DefaultManagedContext.java:103)
>
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWi
> thContext(DefaultManagedContext.java:53)
>
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Manag
> edContextRunnable.java:46)
>
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(As
> yncJobManagerImpl.java:551)
>
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executor
> s.java:515)
>
> at
> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> Executor.java:1128)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> lExecutor.java:628)
>
> at java.base/java.lang.Thread.run(Thread.java:834)
>
> 2020-11-09 12:56:48,380 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897) (logid:db479fbc)
> Complete async job-1897, jobStatus: FAILED, resultCode: 0, result:
> rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdG
> lvbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNl
> cHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cg
> ATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5n
> L1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3
> RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNz
> ZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0AHpjb20uY2xvdWQudX
> RpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvbjogb3JnLmxpYnZpcnQuTGli
> dmlydEV4Y2VwdGlvbjogWE1MIGVycm9yOiBtaXNzaW5nIHN0b3JhZ2UgcG9vbCBzb3VyY2
> UgcGF0aHVyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhw
> AAAAMHNyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIACEIABmZvcm
> 1hdEkACmxpbmVOdW1iZXJMAA9jbGFzc0xvYWRlck5hbWVxAH4ABUwADmRlY2xhcmluZ0Ns
> YXNzcQB-AAVMAAhmaWxlTmFtZXEAfgAFTAAKbWV0aG9kTmFtZXEAfgAFTAAKbW9kdWxlTm
> FtZXEAfgAFTAANbW9kdWxlVmVyc2lvbnEAfgAFeHABAAABLXQAA2FwcHQAOm9yZy5hcGFj
> aGUuY2xvdWRzdGFjay5zdG9yYWdlLnNuYXBzaG90LlNuYXBzaG90U2VydmljZUltcGx0AB
> hTbmFwc2hvdFNlcnZpY2VJbXBsLmphdmF0AA5iYWNrdXBTbmFwc2hvdHBwc3EAfgAMAQAA
> AKtxAH4ADnQAPm9yZy5hcGFjaGUuY2xvdWRzdGFjay5zdG9yYWdlLnNuYXBzaG90LkRlZm
> F1bHRTbmFwc2hvdFN0cmF0ZWd5dAAcRGVmYXVsdFNuYXBzaG90U3RyYXRlZ3kuamF2YXEA
> fgARcHBzcQB-AAwBAAAEvXEAfgAOdAAuY29tLmNsb3VkLnN0b3JhZ2Uuc25hcHNob3QuU2
> 5hcHNob3RNYW5hZ2VySW1wbHQAGFNuYXBzaG90TWFuYWdlckltcGwuamF2YXQAGWJhY2t1
> cFNuYXBzaG90VG9TZWNvbmRhcnlwcHNxAH4ADAEAAASMcQB-AA5xAH4AFnEAfgAXdAAMdG
> FrZVNuYXBzaG90cHBzcQB-AAwC_____nB0AC1qZGsuaW50ZXJuYWwucmVmbGVjdC5OYXRp
> dmVNZXRob2RBY2Nlc3NvckltcGx0AB1OYXRpdmVNZXRob2RBY2Nlc3NvckltcGwuamF2YX
> QAB2ludm9rZTB0AAlqYXZhLmJhc2V0AAYxMS4wLjhzcQB-AAwCAAAAPnBxAH4AHHEAfgAd
> dAAGaW52b2tlcQB-AB9xAH4AIHNxAH4ADAIAAAArcHQAMWpkay5pbnRlcm5hbC5yZWZsZW
> N0LkRlbGVnYXRpbmdNZXRob2RBY2Nlc3NvckltcGx0ACFEZWxlZ2F0aW5nTWV0aG9kQWNj
> ZXNzb3JJbXBsLmphdmFxAH4AInEAfgAfcQB-ACBzcQB-AAwCAAACNnB0ABhqYXZhLmxhbm
> cucmVmbGVjdC5NZXRob2R0AAtNZXRob2QuamF2YXEAfgAicQB-AB9xAH4AIHNxAH4ADAEA
> AAFYcQB-AA50AChvcmcuc3ByaW5nZnJhbWV3b3JrLmFvcC5zdXBwb3J0LkFvcFV0aWxzdA
> ANQW9wVXRpbHMuamF2YXQAHmludm9rZUpvaW5wb2ludFVzaW5nUmVmbGVjdGlvbnBwc3EA
> fgAMAQAAAMZxAH4ADnQAPG9yZy5zcHJpbmdmcmFtZXdvcmsuYW9wLmZyYW1ld29yay5SZW
> ZsZWN0aXZlTWV0aG9kSW52b2NhdGlvbnQAH1JlZmxlY3RpdmVNZXRob2RJbnZvY2F0aW9u
> LmphdmF0AA9pbnZva2VKb2lucG9pbnRwcHNxAH4ADAEAAACjcQB-AA5xAH4ALnEAfgAvdA
> AHcHJvY2VlZHBwc3EAfgAMAQAAAF9xAH4ADnQAP29yZy5zcHJpbmdmcmFtZXdvcmsuYW9w
> LmludGVyY2VwdG9yLkV4cG9zZUludm9jYXRpb25JbnRlcmNlcHRvcnQAIEV4cG9zZUludm
> 9jYXRpb25JbnRlcmNlcHRvci5qYXZhcQB-ACJwcHNxAH4ADAEAAAC6cQB-AA5xAH4ALnEA
> fgAvcQB-ADJwcHNxAH4ADAEAAADUcQB-AA50ADRvcmcuc3ByaW5nZnJhbWV3b3JrLmFvcC
> 5mcmFtZXdvcmsuSmRrRHluYW1pY0FvcFByb3h5dAAXSmRrRHluYW1pY0FvcFByb3h5Lmph
> dmFxAH4AInBwc3EAfgAMAP____9wdAAXY29tLnN1bi5wcm94eS4kUHJveHkyMDVwcQB-AB
> pwcHNxAH4ADAEAAAgZcQB-AA50ADZvcmcuYXBhY2hlLmNsb3Vkc3RhY2suc3RvcmFnZS52
> b2x1bWUuVm9sdW1lU2VydmljZUltcGx0ABZWb2x1bWVTZXJ2aWNlSW1wbC5qYXZhcQB-AB
> pwcHNxAH4ADAEAAAntcQB-AA50ACZjb20uY2xvdWQuc3RvcmFnZS5Wb2x1bWVBcGlTZXJ2
> aWNlSW1wbHQAGVZvbHVtZUFwaVNlcnZpY2VJbXBsLmphdmF0AB1vcmNoZXN0cmF0ZVRha2
> VWb2x1bWVTbmFwc2hvdHBwc3EAfgAMAQAADYlxAH4ADnEAfgBAcQB-AEFxAH4AQnBwc3EA
> fgAMAv____5wcQB-ABxxAH4AHXEAfgAecQB-AB9xAH4AIHNxAH4ADAIAAAA-cHEAfgAccQ
> B-AB1xAH4AInEAfgAfcQB-ACBzcQB-AAwCAAAAK3BxAH4AJHEAfgAlcQB-ACJxAH4AH3EA
> fgAgc3EAfgAMAgAAAjZwcQB-ACdxAH4AKHEAfgAicQB-AB9xAH4AIHNxAH4ADAEAAABrcQ
> B-AA50ACJjb20uY2xvdWQudm0uVm1Xb3JrSm9iSGFuZGxlclByb3h5dAAaVm1Xb3JrSm9i
> SGFuZGxlclByb3h5LmphdmF0AA9oYW5kbGVWbVdvcmtKb2JwcHNxAH4ADAEAAA2PcQB-AA
> 5xAH4AQHEAfgBBcQB-AEtwcHNxAH4ADAL____-cHEAfgAccQB-AB1xAH4AHnEAfgAfcQB-
> ACBzcQB-AAwCAAAAPnBxAH4AHHEAfgAdcQB-ACJxAH4AH3EAfgAgc3EAfgAMAgAAACtwcQ
> B-ACRxAH4AJXEAfgAicQB-AB9xAH4AIHNxAH4ADAIAAAI2cHEAfgAncQB-AChxAH4AInEA
> fgAfcQB-ACBzcQB-AAwBAAABWHEAfgAOcQB-ACpxAH4AK3EAfgAscHBzcQB-AAwBAAAAxn
> EAfgAOcQB-AC5xAH4AL3EAfgAwcHBzcQB-AAwBAAAAo3EAfgAOcQB-AC5xAH4AL3EAfgAy
> cHBzcQB-AAwBAAAAX3EAfgAOcQB-ADRxAH4ANXEAfgAicHBzcQB-AAwBAAAAunEAfgAOcQ
> B-AC5xAH4AL3EAfgAycHBzcQB-AAwBAAAA1HEAfgAOcQB-ADhxAH4AOXEAfgAicHBzcQB-
> AAwA_____3B0ABdjb20uc3VuLnByb3h5LiRQcm94eTIxMnBxAH4AS3Bwc3EAfgAMAQAAAG
> ZxAH4ADnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1Xb3JrSm9i
> RGlzcGF0Y2hlci5qYXZhdAAGcnVuSm9icHBzcQB-AAwBAAACW3EAfgAOdAA_b3JnLmFwYW
> NoZS5jbG91ZHN0YWNrLmZyYW1ld29yay5qb2JzLmltcGwuQXN5bmNKb2JNYW5hZ2VySW1w
> bCQ1dAAYQXN5bmNKb2JNYW5hZ2VySW1wbC5qYXZhdAAMcnVuSW5Db250ZXh0cHBzcQB-AA
> wBAAAAMXEAfgAOdAA-b3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5N
> YW5hZ2VkQ29udGV4dFJ1bm5hYmxlJDF0ABtNYW5hZ2VkQ29udGV4dFJ1bm5hYmxlLmphdm
> F0AANydW5wcHNxAH4ADAEAAAA4cQB-AA50AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFu
> YWdlZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TW
> FuYWdlZENvbnRleHQuamF2YXQABGNhbGxwcHNxAH4ADAEAAABncQB-AA50AEBvcmcuYXBh
> Y2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb2
> 50ZXh0cQB-AGd0AA9jYWxsV2l0aENvbnRleHRwcHNxAH4ADAEAAAA1cQB-AA5xAH4AanEA
> fgBndAAOcnVuV2l0aENvbnRleHRwcHNxAH4ADAEAAAAucQB-AA50ADxvcmcuYXBhY2hlLm
> Nsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxAH4A
> Y3EAfgBkcHBzcQB-AAwBAAACJ3EAfgAOcQB-AF5xAH4AX3EAfgBkcHBzcQB-AAwCAAACA3
> B0AC5qYXZhLnV0aWwuY29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAO
> RXhlY3V0b3JzLmphdmFxAH4AaHEAfgAfcQB-ACBzcQB-AAwCAAABCHB0AB9qYXZhLnV0aW
> wuY29uY3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFzay5qYXZhcQB-AGRxAH4AH3EA
> fgAgc3EAfgAMAgAABGhwdAAnamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZW
> N1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5Xb3JrZXJxAH4AH3EAfgAg
> c3EAfgAMAgAAAnRwdAAuamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG
> 9yJFdvcmtlcnEAfgB5cQB-AGRxAH4AH3EAfgAgc3EAfgAMAgAAA0JwdAAQamF2YS5sYW5n
> LlRocmVhZHQAC1RocmVhZC5qYXZhcQB-AGRxAH4AH3EAfgAgc3IAH2phdmEudXRpbC5Db2
> xsZWN0aW9ucyRFbXB0eUxpc3R6uBe0PKee3gIAAHhweAAAEJp3CAAAAAAAAAAAeA
>
> 2020-11-09 12:56:48,380 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897) (logid:db479fbc)
> Publish async job-1897 complete on message bus
>
> 2020-11-09 12:56:48,380 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897) (logid:db479fbc)
> Wake up jobs related to job-1897
>
> 2020-11-09 12:56:48,380 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897) (logid:db479fbc)
> Update db status for job-1897
>
> 2020-11-09 12:56:48,381 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897) (logid:db479fbc)
> Wake up jobs joined with job-1897 and disjoin all subjobs created from
> job- 1897
>
> 2020-11-09 12:56:48,385 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897) (logid:db479fbc)
> Done executing com.cloud.vm.VmWorkTakeVolumeSnapshot for job-1897
>
> 2020-11-09 12:56:48,385 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897) (logid:db479fbc)
> Remove job-1897 from job monitoring
>
> 2020-11-09 12:56:48,392 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-35:ctx-326b32af job-1894) (logid:db479fbc) Complete
> async job-1894, jobStatus: FAILED, resultCode: 530, result:
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":
> [],"errorcode":530,"errortext":"Failed
> to create snapshot due to an internal error creating snapshot for
> volume 9945a23b-a376-4f85-b378-9f7da8017dac"}
>
> 2020-11-09 12:56:48,392 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-35:ctx-326b32af job-1894) (logid:db479fbc) Publish
> async
> job-1894 complete on message bus
>
--
Andrija Panić
Re: snapshot is not working
Posted by Gabriel Bräscher <ga...@gmail.com>.
Hello,
I am right now working on a fix for this.
Please take a look at issue:
https://github.com/apache/cloudstack/issues/4498
Until then, the recommendation for those who want to take RBD snapshots is
to set global settings *snapshot.backup.to.secondary* to *false.*
Regards,
Gabriel.
Em seg., 7 de dez. de 2020 às 12:23, Piotr Pisz <pi...@piszki.pl> escreveu:
> Hello,
>
>
>
> Andrija, Rakesh, can you elaborate and explain this: “it's a known issue
> and that's why the "path" is missing” ?
>
> I have this bug all the time, snapshots are done correctly on Ceph / RBD
> but we can't make new templates from them.
>
> Each attempt to copy a volume to Secondary Storage ends with this message
> (XML error: missing storage pool source).
>
> Let's save ourselves by loading templates from external URLs, no problem
> here.
>
>
>
> Best regards,
>
> Piotr
>
>
>
>
>
>
>
> From: Andrija Panic <an...@gmail.com>
> Sent: Wednesday, November 18, 2020 1:08 AM
> To: users <us...@cloudstack.apache.org>; Piotr Pisz <pi...@piszki.pl>
> Subject: Re: snapshot is not working
>
>
>
> that actually makes it broken with the default behaviour of copying the
> snapshots to Secondary Storage (NOT copying it to Sec Stor is
> supported/implemented ONLY for KVM/Ceph combo)
>
>
>
> You might want to help, by raising a bug in the GitHub... it's supposed to
> work
>
>
>
> Best.
>
>
>
> On Mon, 9 Nov 2020 at 18:51, Piotr Pisz <piotr@piszki.pl <mailto:
> piotr@piszki.pl> > wrote:
>
> Thank You Rakesh!
>
> After set snapshot.to.secondary to FALSE snapshot on RBD work 😊
>
> Best regards,
> Piotr
>
>
>
> -----Original Message-----
> From: Rakesh v <www.rakeshv.com@gmail.com <mailto:
> www.rakeshv.com@gmail.com> >
> Sent: Monday, November 9, 2020 5:34 PM
> To: users@cloudstack.apache.org <ma...@cloudstack.apache.org>
> Subject: Re: snapshot is not working
>
> Is the global setting which contains the string "snaphot.to.secondary" is
> false? If so it's a known issue and that's why the "path" is missing
>
> Sent from my iPhone
>
> > On 09-Nov-2020, at 1:22 PM, Piotr Pisz <piotr@piszki.pl <mailto:
> piotr@piszki.pl> > wrote:
> >
> > Hello,
> >
> >
> >
> > I need help, snapshots are not working (see below), this is a new CS
> > installation on Ceph RBD. Besides, everything works fine, could you
> > tell us what might be wrong?
> >
> >
> >
> > Take snapshot: 544 failed
> >
> > com.cloud.utils.exception.CloudRuntimeException:
> > com.cloud.utils.exception.CloudRuntimeException:
> > org.libvirt.LibvirtException: XML error: missing storage pool source
> > path
> >
> >
> >
> > ???
> >
> >
> >
> > Best regards,
> >
> > Piotr
> >
> >
> >
> >
> >
> > 2020-11-09 12:56:46,525 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> > (API-Job-Executor-35:ctx-326b32af job-1894) (logid:836f528d) Add
> > job-1894 into job monitoring
> >
> > 2020-11-09 12:56:46,528 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (qtp1497845528-1209:ctx-a9fac85e ctx-7d1c63e1) (logid:a5ddc58d) submit
> > async job-1894, details: AsyncJobVO {id:1894, userId: 2, accountId: 2,
> > instanceType: Snapshot, instanceId: 13, cmd:
> > org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd,
> > cmdInfo:
> > {"quiescevm":"false","httpmethod":"GET","ctxAccountId":"2","uuid":"82f
> > 3faf3-e09c-4eaf-886c-d81bad998b64","cmdEventType":"SNAPSHOT.CREATE","a
> > syncBackup":"false","response":"json","ctxUserId":"2","volumeId":"9945
> > a23b-a376-4f85-b378-9f7da8017dac","name":"test","ctxStartEventId":"427
> > 0","id":"13","ctxDetails":"{\"interface
> > com.cloud.storage.Volume\":\"9945a23b-a376-4f85-b378-9f7da8017dac\",\"
> > interface
> > com.cloud.storage.Snapshot\":\"82f3faf3-e09c-4eaf-886c-d81bad998b64\"}
> > ","_":"1604922798590"},
> > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> > result: null, initMsid: 176206389407942, completeMsid: null,
> lastUpdated:
> > null, lastPolled: null, created: null, removed: null}
> >
> > 2020-11-09 12:56:46,528 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (API-Job-Executor-35:ctx-326b32af job-1894) (logid:db479fbc) Executing
> > AsyncJobVO {id:1894, userId: 2, accountId: 2, instanceType: Snapshot,
> > instanceId: 13, cmd:
> > org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd,
> > cmdInfo:
> > {"quiescevm":"false","httpmethod":"GET","ctxAccountId":"2","uuid":"82f
> > 3faf3-e09c-4eaf-886c-d81bad998b64","cmdEventType":"SNAPSHOT.CREATE","a
> > syncBackup":"false","response":"json","ctxUserId":"2","volumeId":"9945
> > a23b-a376-4f85-b378-9f7da8017dac","name":"test","ctxStartEventId":"427
> > 0","id":"13","ctxDetails":"{\"interface
> > com.cloud.storage.Volume\":\"9945a23b-a376-4f85-b378-9f7da8017dac\",\"
> > interface
> > com.cloud.storage.Snapshot\":\"82f3faf3-e09c-4eaf-886c-d81bad998b64\"}
> > ","_":"1604922798590"},
> > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> > result: null, initMsid: 176206389407942, completeMsid: null,
> lastUpdated:
> > null, lastPolled: null, created: null, removed: null}
> >
> > 2020-11-09 12:56:46,528 DEBUG [c.c.a.ApiServlet]
> > (qtp1497845528-1209:ctx-a9fac85e ctx-7d1c63e1) (logid:a5ddc58d)
> > ===END===
> > 10.89.11.31 -- GET
> > command=createSnapshot&response=json&volumeId=9945a23b-a376-4f85-b378-
> > 9f7da8017dac&quiescevm=false&asyncBackup=false&name=test&_=16049227985
> > 90
> >
> > 2020-11-09 12:56:46,549 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (API-Job-Executor-35:ctx-326b32af job-1894 ctx-f8b08221)
> > (logid:db479fbc) Sync job-1897 execution on object VmWorkJobQueue.183
> >
> > 2020-11-09 12:56:46,850 DEBUG [c.c.a.m.ClusteredAgentAttache]
> > (AgentManager-Handler-13:null) (logid:) Seq 1-8981303556883622697:
> > Routing from 176206389408022
> >
> > 2020-11-09 12:56:46,851 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> > (AgentManager-Handler-6:null) (logid:) Seq 1-8981303556883622697:
> > MgmtId
> > 176206389408022: Resp: Routing to peer
> >
> > 2020-11-09 12:56:47,086 DEBUG [c.c.a.m.ClusteredAgentAttache]
> > (AgentManager-Handler-3:null) (logid:) Seq 43-2584503236157253953:
> > Routing from 176206389408022
> >
> > 2020-11-09 12:56:47,126 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> > (AgentManager-Handler-14:null) (logid:) Seq 43-2584503236157253953:
> > MgmtId
> > 176206389408022: Resp: Routing to peer
> >
> > 2020-11-09 12:56:47,131 DEBUG [c.c.a.m.ClusteredAgentAttache]
> > (AgentManager-Handler-15:null) (logid:) Seq 46-3634967849241411607:
> > Routing from 176206389408022
> >
> > 2020-11-09 12:56:47,133 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> > (AgentManager-Handler-10:null) (logid:) Seq 46-3634967849241411607:
> > MgmtId
> > 176206389408022: Resp: Routing to peer
> >
> > 2020-11-09 12:56:47,181 DEBUG [c.c.s.StatsCollector]
> > (StatsCollector-6:ctx-100fc604) (logid:fde56d09) StorageCollector is
> > running...
> >
> > 2020-11-09 12:56:47,184 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
> > (StatsCollector-6:ctx-100fc604) (logid:fde56d09)
> getCommandHostDelegation:
> > class com.cloud.agent.api.GetStorageStatsCommand
> >
> > 2020-11-09 12:56:47,184 DEBUG [c.c.h.XenServerGuru]
> > (StatsCollector-6:ctx-100fc604) (logid:fde56d09) We are returning the
> > default host to execute commands because the command is not of Copy type.
> >
> > 2020-11-09 12:56:47,230 DEBUG [c.c.a.t.Request]
> > (StatsCollector-6:ctx-100fc604) (logid:fde56d09) Seq
> > 58-4654188739910703033: Received: { Ans: , MgmtId: 176206389407942,
> via:
> > 58(s-10-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
> >
> > 2020-11-09 12:56:47,231 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
> > (StatsCollector-6:ctx-100fc604) (logid:fde56d09)
> getCommandHostDelegation:
> > class com.cloud.agent.api.GetStorageStatsCommand
> >
> > 2020-11-09 12:56:47,231 DEBUG [c.c.h.XenServerGuru]
> > (StatsCollector-6:ctx-100fc604) (logid:fde56d09) We are returning the
> > default host to execute commands because the command is not of Copy type.
> >
> > 2020-11-09 12:56:47,232 DEBUG [c.c.a.m.ClusteredAgentAttache]
> > (StatsCollector-6:ctx-100fc604) (logid:fde56d09) Seq
> > 37-1383449510532885592: Forwarding null to 176206389407982
> >
> > 2020-11-09 12:56:47,406 DEBUG [c.c.a.ApiServlet]
> > (qtp1497845528-1280:ctx-b087af82) (logid:3a8983a6) ===START===
> > 10.89.11.31 -- GET
> > command=listInfrastructure&response=json&_=1604922838197
> >
> > 2020-11-09 12:56:47,410 DEBUG [c.c.a.ApiServer]
> > (qtp1497845528-1280:ctx-b087af82 ctx-052d26fb) (logid:3a8983a6) CIDRs
> > from which account 'Acct[c3dd8977-12ba-11eb-9854-a0423f44350c-admin]'
> > is allowed to perform API calls: 0.0.0.0/0,::/0 <http://0.0.0.0/0,::/0>
> >
> > 2020-11-09 12:56:47,416 DEBUG [c.c.a.ApiServlet]
> > (qtp1497845528-1280:ctx-b087af82 ctx-052d26fb) (logid:3a8983a6)
> > ===END===
> > 10.89.11.31 -- GET
> > command=listInfrastructure&response=json&_=1604922838197
> >
> > 2020-11-09 12:56:47,634 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (AsyncJobMgr-Heartbeat-1:ctx-ccb57103) (logid:56d3b696) Execute
> > sync-queue
> > item: SyncQueueItemVO {id:841, queueId: 601, contentType: AsyncJob,
> > contentId: 1897, lastProcessMsid: 176206389407942, lastprocessNumber:
> > 5,
> > lastProcessTime: Mon Nov 09 12:56:47 CET 2020, created: Mon Nov 09
> > 12:56:46 CET 2020}
> >
> > 2020-11-09 12:56:47,635 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (AsyncJobMgr-Heartbeat-1:ctx-ccb57103) (logid:56d3b696) Schedule
> > queued
> > job-1897
> >
> > 2020-11-09 12:56:47,637 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897) (logid:1f589139)
> > Add
> > job-1897 into job monitoring
> >
> > 2020-11-09 12:56:47,640 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897) (logid:db479fbc)
> > Executing AsyncJobVO {id:1897, userId: 2, accountId: 2, instanceType:
> > null, instanceId: null, cmd: com.cloud.vm.VmWorkTakeVolumeSnapshot,
> > cmdInfo:
> > rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrVGFrZVZvbHVtZVNuYXBzaG90BL5gG4Li1c
> > 8CAAZaAAthc3luY0JhY2t1cFoACXF1aWVzY2VWbUwADGxvY2F0aW9uVHlwZXQAKUxjb20v
> > Y2xvdWQvc3RvcmFnZS9TbmFwc2hvdCRMb2NhdGlvblR5cGU7TAAIcG9saWN5SWR0ABBMam
> > F2YS9sYW5nL0xvbmc7TAAKc25hcHNob3RJZHEAfgACTAAIdm9sdW1lSWRxAH4AAnhyABNj
> > b20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAA
> > R2bUlkTAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAACAAAA
> > AAAAAAIAAAAAAAAAt3QAFFZvbHVtZUFwaVNlcnZpY2VJbXBsAABwc3IADmphdmEubGFuZy
> > 5Mb25nO4vkkMyPI98CAAFKAAV2YWx1ZXhyABBqYXZhLmxhbmcuTnVtYmVyhqyVHQuU4IsC
> > AAB4cAAAAAAAAAAAc3EAfgAHAAAAAAAAAA1zcQB-AAcAAAAAAAACIA,
> > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> > result: null, initMsid: 176206389407942, completeMsid: null,
> lastUpdated:
> > null, lastPolled: null, created: Mon Nov 09 12:56:46 CET 2020, removed:
> > null}
> >
> > 2020-11-09 12:56:47,640 DEBUG [c.c.v.VmWorkJobDispatcher]
> > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897) (logid:db479fbc)
> > Run VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot for VM 183, job
> origin:
> > 1894
> >
> > 2020-11-09 12:56:47,642 DEBUG [c.c.v.VmWorkJobHandlerProxy]
> > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897 ctx-5fb6d0f8)
> > (logid:db479fbc) Execute VM work job:
> > com.cloud.vm.VmWorkTakeVolumeSnapshot{"volumeId":544,"policyId":0,"sna
> > pshotId":13,"quiesceVm":false,"asyncBackup":false,"userId":2,"accountI
> > d":2,"vmId":183,"handlerName":"VolumeApiServiceImpl"}
> >
> > 2020-11-09 12:56:47,675 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
> > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897 ctx-5fb6d0f8)
> > (logid:db479fbc) getCommandHostDelegation: class
> > org.apache.cloudstack.storage.command.CreateObjectCommand
> >
> > 2020-11-09 12:56:47,675 DEBUG [c.c.h.XenServerGuru]
> > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897 ctx-5fb6d0f8)
> > (logid:db479fbc) We are returning the default host to execute commands
> > because the command is not of Copy type.
> >
> > 2020-11-09 12:56:47,676 DEBUG [c.c.a.m.ClusteredAgentAttache]
> > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897 ctx-5fb6d0f8)
> > (logid:db479fbc) Seq 40-1407656358530002013: Forwarding Seq
> > 40-1407656358530002013: { Cmd , MgmtId: 176206389407942, via:
> > 40(srv-oz-r08-n08.cenagis.local), Ver: v1, Flags: 100011,
> > [{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":
> > {"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid"
> > :"9945a23b-a376-4f85-b378-9f7da8017dac","volumeType":"ROOT","dataStore
> > ":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"22d1
> > 051a-86dc-3613-961d-f909245f5477","id":4,"poolType":"RBD","host":"rbd.
> > cenagis.local","path":"rbd","port":6789,"url":"RBD://rbd.cenagis.local
> > /rbd/?ROLE=Primary&STOREUUID=22d1051a-86dc-3613-961d-f909245f5477","is
> > Managed":false}},"name":"ROOT-183","size":107374182400,"path":"9945a23
> > b-a376-4f85-b378-9f7da8017dac","volumeId":544,"vmName":"i-2-183-VM","a
> > ccountId":2,"format":"RAW","provisioningType":"THIN","id":544,"deviceI
> > d":0,"cacheMode":"WRITEBACK","hypervisorType":"KVM","directDownload":f
> > alse},"dataStore":{"org.apache.cloudstack.storage.to <
> http://apache.cloudstack.storage.to> .PrimaryDataStoreT
> > O":{"uuid":"22d1051a-86dc-3613-961d-f909245f5477","id":4,"poolType":"R
> > BD","host":"rbd.cenagis.local","path":"rbd","port":6789,"url":"RBD://r
> > bd.cenagis.local/rbd/?ROLE=Primary&STOREUUID=22d1051a-86dc-3613-961d-f
> > 909245f5477","isManaged":false}},"vmName":"i-2-183-VM","name":"test","
> > hypervisorType":"KVM","id":13,"quiescevm":false,"physicalSize":0}},"wa
> > it":0}}]
> > } to 176206389407982
> >
> > 2020-11-09 12:56:47,976 DEBUG [c.c.a.m.ClusteredAgentAttache]
> > (AgentManager-Handler-1:null) (logid:) Seq 1-8603282663161211720:
> > Routing from 176206389407982
> >
> > 2020-11-09 12:56:47,982 DEBUG [c.c.a.m.ClusteredAgentAttache]
> > (AgentManager-Handler-4:null) (logid:) Seq 1-8981303556883622698:
> > Routing from 176206389408022
> >
> > 2020-11-09 12:56:48,043 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> > (consoleproxy-1:ctx-79a33e59) (logid:4c5fdc64) Zone 1 is ready to
> > launch console proxy
> >
> > 2020-11-09 12:56:48,150 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
> > (secstorage-1:ctx-c8a78ea1) (logid:473dea1d) Zone 1 is ready to launch
> > secondary storage VM
> >
> > 2020-11-09 12:56:48,190 DEBUG [c.c.a.t.Request]
> > (StatsCollector-5:ctx-e6db6758) (logid:451965d6) Seq
> > 1-8552617167353293650: Received: { Ans: , MgmtId: 176206389407942, via:
> > 1(srv-oz-r01-n01.cenagis.local), Ver: v1, Flags: 10, {
> > GetVmStatsAnswer } }
> >
> > 2020-11-09 12:56:48,190 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> > (AgentManager-Handler-5:null) (logid:) Seq 1-8603282663161211720:
> > MgmtId
> > 176206389407982: Resp: Routing to peer
> >
> > 2020-11-09 12:56:48,198 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> > (AgentManager-Handler-11:null) (logid:) Seq 1-8981303556883622698:
> > MgmtId
> > 176206389408022: Resp: Routing to peer
> >
> > 2020-11-09 12:56:48,212 DEBUG [c.c.a.t.Request]
> > (AgentManager-Handler-12:null) (logid:) Seq 40-1407656358530002013:
> > Processing: { Ans: , MgmtId: 176206389407942, via: 40, Ver: v1, Flags:
> > 10,
> > [{"org.apache.cloudstack.storage.command.CreateObjectAnswer":{"data":{
> > "org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"rbd/9945a
> > 23b-a376-4f85-b378-9f7da8017dac/f957d3a8-702a-45aa-9e42-5222c223ef19",
> > "id":0,"quiescevm":false,"physicalSize":0}},"result":true,"wait":0}}]
> > }
> >
> > 2020-11-09 12:56:48,212 DEBUG [c.c.a.t.Request]
> > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897 ctx-5fb6d0f8)
> > (logid:db479fbc) Seq 40-1407656358530002013: Received: { Ans: , MgmtId:
> > 176206389407942, via: 40(srv-oz-r08-n08.cenagis.local), Ver: v1, Flags:
> > 10, { CreateObjectAnswer } }
> >
> > 2020-11-09 12:56:48,226 DEBUG [c.c.a.m.ClusteredAgentAttache]
> > (StatsCollector-5:ctx-e6db6758) (logid:451965d6) Seq
> > 22-6063533948300963066: Forwarding null to 176206389407982
> >
> > 2020-11-09 12:56:48,248 DEBUG [o.a.c.s.m.AncientDataMotionStrategy]
> > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897 ctx-5fb6d0f8)
> > (logid:db479fbc) copyAsync inspecting src type SNAPSHOT copyAsync
> > inspecting dest type SNAPSHOT
> >
> > 2020-11-09 12:56:48,249 DEBUG [c.c.a.ApiServlet]
> > (qtp1497845528-1209:ctx-4e81cf12) (logid:2f3c90fe) ===START===
> > 10.89.11.31 -- GET
> > command=listHosts&response=json&type=routing&listAll=true&page=1&pages
> > ize=20&_=1604922838198
> >
> > 2020-11-09 12:56:48,253 DEBUG [c.c.a.ApiServer]
> > (qtp1497845528-1209:ctx-4e81cf12 ctx-0b68411a) (logid:2f3c90fe) CIDRs
> > from which account 'Acct[c3dd8977-12ba-11eb-9854-a0423f44350c-admin]'
> > is allowed to perform API calls: 0.0.0.0/0,::/0 <http://0.0.0.0/0,::/0>
> >
> > 2020-11-09 12:56:48,254 WARN [c.c.a.d.ParamGenericValidationWorker]
> > (qtp1497845528-1209:ctx-4e81cf12 ctx-0b68411a) (logid:2f3c90fe)
> > Received unknown parameters for command listHosts. Unknown parameters
> > : listall
> >
> > 2020-11-09 12:56:48,254 DEBUG [c.c.a.q.QueryManagerImpl]
> > (qtp1497845528-1209:ctx-4e81cf12 ctx-0b68411a) (logid:2f3c90fe)
> > >>>Searching for hosts>>>
> >
> > 2020-11-09 12:56:48,259 DEBUG [c.c.a.q.QueryManagerImpl]
> > (qtp1497845528-1209:ctx-4e81cf12 ctx-0b68411a) (logid:2f3c90fe)
> > >>>Generating Response>>>
> >
> > 2020-11-09 12:56:48,293 DEBUG [c.c.a.ApiServlet]
> > (qtp1497845528-1209:ctx-4e81cf12 ctx-0b68411a) (logid:2f3c90fe)
> > ===END===
> > 10.89.11.31 -- GET
> > command=listHosts&response=json&type=routing&listAll=true&page=1&pages
> > ize=20&_=1604922838198
> >
> > 2020-11-09 12:56:48,298 ERROR [o.a.c.s.e.DefaultEndPointSelector]
> > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897 ctx-5fb6d0f8)
> > (logid:db479fbc) IR24 select BACKUPSNAPSHOT from primary to secondary
> > 13
> > dest=13
> >
> > 2020-11-09 12:56:48,302 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
> > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897 ctx-5fb6d0f8)
> > (logid:db479fbc) getCommandHostDelegation: class
> > org.apache.cloudstack.storage.command.CopyCommand
> >
> > 2020-11-09 12:56:48,303 DEBUG [c.c.a.m.ClusteredAgentAttache]
> > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897 ctx-5fb6d0f8)
> > (logid:db479fbc) Seq 49-6537256334104996965: Forwarding Seq
> > 49-6537256334104996965: { Cmd , MgmtId: 176206389407942, via:
> > 49(srv-oz-r11-n11.cenagis.local), Ver: v1, Flags: 100011,
> > [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.a
> > pache.cloudstack.storage.to <http://pache.cloudstack.storage.to>
> .SnapshotObjectTO":{"path":"rbd/9945a23b-a3
> > 76-4f85-b378-9f7da8017dac/f957d3a8-702a-45aa-9e42-5222c223ef19","volum
> > e":{"uuid":"9945a23b-a376-4f85-b378-9f7da8017dac","volumeType":"ROOT",
> > "dataStore":{"org.apache.cloudstack.storage.to <
> http://cloudstack.storage.to> .PrimaryDataStoreTO":{"u
> > uid":"22d1051a-86dc-3613-961d-f909245f5477","id":4,"poolType":"RBD","h
> > ost":"rbd.cenagis.local","path":"rbd","port":6789,"url":"RBD://rbd.cen
> > agis.local/rbd/?ROLE=Primary&STOREUUID=22d1051a-86dc-3613-961d-f909245
> > f5477","isManaged":false}},"name":"ROOT-183","size":107374182400,"path
> > ":"9945a23b-a376-4f85-b378-9f7da8017dac","volumeId":544,"vmName":"i-2-
> > 183-VM","accountId":2,"format":"RAW","provisioningType":"THIN","id":54
> > 4,"deviceId":0,"cacheMode":"WRITEBACK","hypervisorType":"KVM","directD
> > ownload":false},"dataStore":{"org.apache.cloudstack.storage.to.Primary
> > DataStoreTO":{"uuid":"22d1051a-86dc-3613-961d-f909245f5477","id":4,"po
> > olType":"RBD","host":"rbd.cenagis.local","path":"rbd","port":6789,"url
> > ":"RBD://rbd.cenagis.local/rbd/?ROLE=Primary&STOREUUID=22d1051a-86dc-3
> > 613-961d-f909245f5477","isManaged":false}},"vmName":"i-2-183-VM","name
> > ":"test","hypervisorType":"KVM","id":13,"quiescevm":false,"physicalSiz
> > e":0}},"destTO":{"org.apache.cloudstack.storage.to <
> http://cloudstack.storage.to> .SnapshotObjectTO":{
> > "path":"snapshots/2/544","volume":{"uuid":"9945a23b-a376-4f85-b378-9f7
> > da8017dac","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.sto
> > rage.to.PrimaryDataStoreTO":{"uuid":"22d1051a-86dc-3613-961d-f909245f5
> > 477","id":4,"poolType":"RBD","host":"rbd.cenagis.local","path":"rbd","
> > port":6789,"url":"RBD://rbd.cenagis.local/rbd/?ROLE=Primary&STOREUUID=
> > 22d1051a-86dc-3613-961d-f909245f5477","isManaged":false}},"name":"ROOT
> > -183","size":107374182400,"path":"9945a23b-a376-4f85-b378-9f7da8017dac
> > ","volumeId":544,"vmName":"i-2-183-VM","accountId":2,"format":"RAW","p
> > rovisioningType":"THIN","id":544,"deviceId":0,"cacheMode":"WRITEBACK",
> > "hypervisorType":"KVM","directDownload":false},"dataStore":{"com.cloud
> > .agent.api.to.NfsTO":{"_url":"nfs://sys2.cenagis.local/","_role":"Imag
> > e"}},"vmName":"i-2-183-VM","name":"test","hypervisorType":"KVM","id":1
> > 3,"quiescevm":false,"physicalSize":0}},"executeInSequence":false,"opti
> > ons":{"fullSnapshot":"true"},"options2":{},"wait":216000}}]
> > } to 176206389407982
> >
> > 2020-11-09 12:56:48,364 DEBUG [c.c.a.t.Request]
> > (AgentManager-Handler-7:null) (logid:) Seq 49-6537256334104996965:
> > Processing: { Ans: , MgmtId: 176206389407942, via: 49, Ver: v1, Flags:
> > 10,
> >
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException:
>
> > org.libvirt.LibvirtException: XML error: missing storage pool source
> > path","wait":0}}] }
> >
> > 2020-11-09 12:56:48,364 DEBUG [c.c.a.t.Request]
> > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897 ctx-5fb6d0f8)
> > (logid:db479fbc) Seq 49-6537256334104996965: Received: { Ans: , MgmtId:
> > 176206389407942, via: 49(srv-oz-r11-n11.cenagis.local), Ver: v1, Flags:
> > 10, { CopyCmdAnswer } }
> >
> > 2020-11-09 12:56:48,373 DEBUG [c.c.s.s.SnapshotManagerImpl]
> > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897 ctx-5fb6d0f8)
> > (logid:db479fbc) Failed to create
> > snapshotcom.cloud.utils.exception.CloudRuntimeException:
> > org.libvirt.LibvirtException: XML error: missing storage pool source
> > path
> >
> > 2020-11-09 12:56:48,373 DEBUG [c.c.r.ResourceLimitManagerImpl]
> > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897 ctx-5fb6d0f8)
> > (logid:db479fbc) Updating resource Type = snapshot count for Account =
> > 2 Operation = decreasing Amount = 1
> >
> > 2020-11-09 12:56:48,375 DEBUG [c.c.r.ResourceLimitManagerImpl]
> > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897 ctx-5fb6d0f8)
> > (logid:db479fbc) Updating resource Type = secondary_storage count for
> > Account = 2 Operation = decreasing Amount = 107374182400
> >
> > 2020-11-09 12:56:48,377 ERROR [o.a.c.s.v.VolumeServiceImpl]
> > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897 ctx-5fb6d0f8)
> > (logid:db479fbc) Take snapshot: 544 failed
> >
> > com.cloud.utils.exception.CloudRuntimeException:
> > com.cloud.utils.exception.CloudRuntimeException:
> > org.libvirt.LibvirtException: XML error: missing storage pool source
> > path
> >
> > at
> > org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnaps
> > hot(SnapshotServiceImpl.java:301)
> >
> > at
> > org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.backupS
> > napshot(DefaultSnapshotStrategy.java:171)
> >
> > at
> > com.cloud.storage.snapshot.SnapshotManagerImpl.backupSnapshotToSeconda
> > ry(SnapshotManagerImpl.java:1213)
> >
> > at
> > com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotMa
> > nagerImpl.java:1164)
> >
> > at
> > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> > Method)
> >
> > at
> > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeM
> > ethodAccessorImpl.java:62)
> >
> > at
> > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Del
> > egatingMethodAccessorImpl.java:43)
> >
> > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> >
> > at
> > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflectio
> > n(AopUtils.java:344)
> >
> > at
> > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoi
> > npoint(ReflectiveMethodInvocation.java:198)
> >
> > at
> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(R
> > eflectiveMethodInvocation.java:163)
> >
> > at
> > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke
> > (ExposeInvocationInterceptor.java:95)
> >
> > at
> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(R
> > eflectiveMethodInvocation.java:186)
> >
> > at
> > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamic
> > AopProxy.java:212)
> >
> > at com.sun.proxy.$Proxy205.takeSnapshot(Unknown Source)
> >
> > at
> > org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(Vo
> > lumeServiceImpl.java:2073)
> >
> > at
> > com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(V
> > olumeApiServiceImpl.java:2541)
> >
> > at
> > com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(V
> > olumeApiServiceImpl.java:3465)
> >
> > at
> > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> > Method)
> >
> > at
> > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeM
> > ethodAccessorImpl.java:62)
> >
> > at
> > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Del
> > egatingMethodAccessorImpl.java:43)
> >
> > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> >
> > at
> > com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerPro
> > xy.java:107)
> >
> > at
> > com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServic
> > eImpl.java:3471)
> >
> > at
> > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> > Method)
> >
> > at
> > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeM
> > ethodAccessorImpl.java:62)
> >
> > at
> > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Del
> > egatingMethodAccessorImpl.java:43)
> >
> > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> >
> > at
> > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflectio
> > n(AopUtils.java:344)
> >
> > at
> > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoi
> > npoint(ReflectiveMethodInvocation.java:198)
> >
> > at
> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(R
> > eflectiveMethodInvocation.java:163)
> >
> > at
> > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke
> > (ExposeInvocationInterceptor.java:95)
> >
> > at
> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(R
> > eflectiveMethodInvocation.java:186)
> >
> > at
> > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamic
> > AopProxy.java:212)
> >
> > at com.sun.proxy.$Proxy212.handleVmWorkJob(Unknown Source)
> >
> > at
> > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> >
> > at
> > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInC
> > ontext(AsyncJobManagerImpl.java:603)
> >
> > at
> > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(Man
> > agedContextRunnable.java:49)
> >
> > at
> > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.cal
> > l(DefaultManagedContext.java:56)
> >
> > at
> > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callW
> > ithContext(DefaultManagedContext.java:103)
> >
> > at
> > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWi
> > thContext(DefaultManagedContext.java:53)
> >
> > at
> > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Manag
> > edContextRunnable.java:46)
> >
> > at
> > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(As
> > yncJobManagerImpl.java:551)
> >
> > at
> > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executor
> > s.java:515)
> >
> > at
> > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> >
> > at
> > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> > Executor.java:1128)
> >
> > at
> > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> > lExecutor.java:628)
> >
> > at java.base/java.lang.Thread.run(Thread.java:834)
> >
> > 2020-11-09 12:56:48,378 ERROR [c.c.v.VmWorkJobHandlerProxy]
> > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897 ctx-5fb6d0f8)
> > (logid:db479fbc) Invocation exception, caused by:
> > com.cloud.utils.exception.CloudRuntimeException:
> > com.cloud.utils.exception.CloudRuntimeException:
> > org.libvirt.LibvirtException: XML error: missing storage pool source
> > path
> >
> > 2020-11-09 12:56:48,378 INFO [c.c.v.VmWorkJobHandlerProxy]
> > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897 ctx-5fb6d0f8)
> > (logid:db479fbc) Rethrow exception
> > com.cloud.utils.exception.CloudRuntimeException:
> > com.cloud.utils.exception.CloudRuntimeException:
> > org.libvirt.LibvirtException: XML error: missing storage pool source
> > path
> >
> > 2020-11-09 12:56:48,378 DEBUG [c.c.v.VmWorkJobDispatcher]
> > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897) (logid:db479fbc)
> > Done with run of VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot
> > for VM 183, job origin: 1894
> >
> > 2020-11-09 12:56:48,378 ERROR [c.c.v.VmWorkJobDispatcher]
> > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897) (logid:db479fbc)
> > Unable to complete AsyncJobVO {id:1897, userId: 2, accountId: 2,
> > instanceType: null, instanceId: null, cmd:
> > com.cloud.vm.VmWorkTakeVolumeSnapshot, cmdInfo:
> > rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrVGFrZVZvbHVtZVNuYXBzaG90BL5gG4Li1c
> > 8CAAZaAAthc3luY0JhY2t1cFoACXF1aWVzY2VWbUwADGxvY2F0aW9uVHlwZXQAKUxjb20v
> > Y2xvdWQvc3RvcmFnZS9TbmFwc2hvdCRMb2NhdGlvblR5cGU7TAAIcG9saWN5SWR0ABBMam
> > F2YS9sYW5nL0xvbmc7TAAKc25hcHNob3RJZHEAfgACTAAIdm9sdW1lSWRxAH4AAnhyABNj
> > b20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAA
> > R2bUlkTAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAACAAAA
> > AAAAAAIAAAAAAAAAt3QAFFZvbHVtZUFwaVNlcnZpY2VJbXBsAABwc3IADmphdmEubGFuZy
> > 5Mb25nO4vkkMyPI98CAAFKAAV2YWx1ZXhyABBqYXZhLmxhbmcuTnVtYmVyhqyVHQuU4IsC
> > AAB4cAAAAAAAAAAAc3EAfgAHAAAAAAAAAA1zcQB-AAcAAAAAAAACIA,
> > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> > result: null, initMsid: 176206389407942, completeMsid: null,
> lastUpdated:
> > null, lastPolled: null, created: Mon Nov 09 12:56:46 CET 2020, removed:
> > null}, job origin:1894
> >
> > com.cloud.utils.exception.CloudRuntimeException:
> > com.cloud.utils.exception.CloudRuntimeException:
> > org.libvirt.LibvirtException: XML error: missing storage pool source
> > path
> >
> > at
> > org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnaps
> > hot(SnapshotServiceImpl.java:301)
> >
> > at
> > org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.backupS
> > napshot(DefaultSnapshotStrategy.java:171)
> >
> > at
> > com.cloud.storage.snapshot.SnapshotManagerImpl.backupSnapshotToSeconda
> > ry(SnapshotManagerImpl.java:1213)
> >
> > at
> > com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotMa
> > nagerImpl.java:1164)
> >
> > at
> > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> > Method)
> >
> > at
> > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeM
> > ethodAccessorImpl.java:62)
> >
> > at
> > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Del
> > egatingMethodAccessorImpl.java:43)
> >
> > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> >
> > at
> > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflectio
> > n(AopUtils.java:344)
> >
> > at
> > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoi
> > npoint(ReflectiveMethodInvocation.java:198)
> >
> > at
> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(R
> > eflectiveMethodInvocation.java:163)
> >
> > at
> > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke
> > (ExposeInvocationInterceptor.java:95)
> >
> > at
> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(R
> > eflectiveMethodInvocation.java:186)
> >
> > at
> > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamic
> > AopProxy.java:212)
> >
> > at com.sun.proxy.$Proxy205.takeSnapshot(Unknown Source)
> >
> > at
> > org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(Vo
> > lumeServiceImpl.java:2073)
> >
> > at
> > com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(V
> > olumeApiServiceImpl.java:2541)
> >
> > at
> > com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(V
> > olumeApiServiceImpl.java:3465)
> >
> > at
> > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> > Method)
> >
> > at
> > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeM
> > ethodAccessorImpl.java:62)
> >
> > at
> > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Del
> > egatingMethodAccessorImpl.java:43)
> >
> > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> >
> > at
> > com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerPro
> > xy.java:107)
> >
> > at
> > com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServic
> > eImpl.java:3471)
> >
> > at
> > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> > Method)
> >
> > at
> > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeM
> > ethodAccessorImpl.java:62)
> >
> > at
> > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Del
> > egatingMethodAccessorImpl.java:43)
> >
> > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> >
> > at
> > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflectio
> > n(AopUtils.java:344)
> >
> > at
> > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoi
> > npoint(ReflectiveMethodInvocation.java:198)
> >
> > at
> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(R
> > eflectiveMethodInvocation.java:163)
> >
> > at
> > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke
> > (ExposeInvocationInterceptor.java:95)
> >
> > at
> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(R
> > eflectiveMethodInvocation.java:186)
> >
> > at
> > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamic
> > AopProxy.java:212)
> >
> > at com.sun.proxy.$Proxy212.handleVmWorkJob(Unknown Source)
> >
> > at
> > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> >
> > at
> > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInC
> > ontext(AsyncJobManagerImpl.java:603)
> >
> > at
> > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(Man
> > agedContextRunnable.java:49)
> >
> > at
> > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.cal
> > l(DefaultManagedContext.java:56)
> >
> > at
> > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callW
> > ithContext(DefaultManagedContext.java:103)
> >
> > at
> > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWi
> > thContext(DefaultManagedContext.java:53)
> >
> > at
> > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Manag
> > edContextRunnable.java:46)
> >
> > at
> > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(As
> > yncJobManagerImpl.java:551)
> >
> > at
> > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executor
> > s.java:515)
> >
> > at
> > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> >
> > at
> > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> > Executor.java:1128)
> >
> > at
> > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> > lExecutor.java:628)
> >
> > at java.base/java.lang.Thread.run(Thread.java:834)
> >
> > 2020-11-09 12:56:48,380 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897) (logid:db479fbc)
> > Complete async job-1897, jobStatus: FAILED, resultCode: 0, result:
> > rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdG
> > lvbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNl
> > cHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cg
> > ATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5n
> > L1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3
> > RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNz
> > ZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0AHpjb20uY2xvdWQudX
> > RpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvbjogb3JnLmxpYnZpcnQuTGli
> > dmlydEV4Y2VwdGlvbjogWE1MIGVycm9yOiBtaXNzaW5nIHN0b3JhZ2UgcG9vbCBzb3VyY2
> > UgcGF0aHVyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhw
> > AAAAMHNyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIACEIABmZvcm
> > 1hdEkACmxpbmVOdW1iZXJMAA9jbGFzc0xvYWRlck5hbWVxAH4ABUwADmRlY2xhcmluZ0Ns
> > YXNzcQB-AAVMAAhmaWxlTmFtZXEAfgAFTAAKbWV0aG9kTmFtZXEAfgAFTAAKbW9kdWxlTm
> > FtZXEAfgAFTAANbW9kdWxlVmVyc2lvbnEAfgAFeHABAAABLXQAA2FwcHQAOm9yZy5hcGFj
> > aGUuY2xvdWRzdGFjay5zdG9yYWdlLnNuYXBzaG90LlNuYXBzaG90U2VydmljZUltcGx0AB
> > hTbmFwc2hvdFNlcnZpY2VJbXBsLmphdmF0AA5iYWNrdXBTbmFwc2hvdHBwc3EAfgAMAQAA
> > AKtxAH4ADnQAPm9yZy5hcGFjaGUuY2xvdWRzdGFjay5zdG9yYWdlLnNuYXBzaG90LkRlZm
> > F1bHRTbmFwc2hvdFN0cmF0ZWd5dAAcRGVmYXVsdFNuYXBzaG90U3RyYXRlZ3kuamF2YXEA
> > fgARcHBzcQB-AAwBAAAEvXEAfgAOdAAuY29tLmNsb3VkLnN0b3JhZ2Uuc25hcHNob3QuU2
> > 5hcHNob3RNYW5hZ2VySW1wbHQAGFNuYXBzaG90TWFuYWdlckltcGwuamF2YXQAGWJhY2t1
> > cFNuYXBzaG90VG9TZWNvbmRhcnlwcHNxAH4ADAEAAASMcQB-AA5xAH4AFnEAfgAXdAAMdG
> > FrZVNuYXBzaG90cHBzcQB-AAwC_____nB0AC1qZGsuaW50ZXJuYWwucmVmbGVjdC5OYXRp
> > dmVNZXRob2RBY2Nlc3NvckltcGx0AB1OYXRpdmVNZXRob2RBY2Nlc3NvckltcGwuamF2YX
> > QAB2ludm9rZTB0AAlqYXZhLmJhc2V0AAYxMS4wLjhzcQB-AAwCAAAAPnBxAH4AHHEAfgAd
> > dAAGaW52b2tlcQB-AB9xAH4AIHNxAH4ADAIAAAArcHQAMWpkay5pbnRlcm5hbC5yZWZsZW
> > N0LkRlbGVnYXRpbmdNZXRob2RBY2Nlc3NvckltcGx0ACFEZWxlZ2F0aW5nTWV0aG9kQWNj
> > ZXNzb3JJbXBsLmphdmFxAH4AInEAfgAfcQB-ACBzcQB-AAwCAAACNnB0ABhqYXZhLmxhbm
> > cucmVmbGVjdC5NZXRob2R0AAtNZXRob2QuamF2YXEAfgAicQB-AB9xAH4AIHNxAH4ADAEA
> > AAFYcQB-AA50AChvcmcuc3ByaW5nZnJhbWV3b3JrLmFvcC5zdXBwb3J0LkFvcFV0aWxzdA
> > ANQW9wVXRpbHMuamF2YXQAHmludm9rZUpvaW5wb2ludFVzaW5nUmVmbGVjdGlvbnBwc3EA
> > fgAMAQAAAMZxAH4ADnQAPG9yZy5zcHJpbmdmcmFtZXdvcmsuYW9wLmZyYW1ld29yay5SZW
> > ZsZWN0aXZlTWV0aG9kSW52b2NhdGlvbnQAH1JlZmxlY3RpdmVNZXRob2RJbnZvY2F0aW9u
> > LmphdmF0AA9pbnZva2VKb2lucG9pbnRwcHNxAH4ADAEAAACjcQB-AA5xAH4ALnEAfgAvdA
> > AHcHJvY2VlZHBwc3EAfgAMAQAAAF9xAH4ADnQAP29yZy5zcHJpbmdmcmFtZXdvcmsuYW9w
> > LmludGVyY2VwdG9yLkV4cG9zZUludm9jYXRpb25JbnRlcmNlcHRvcnQAIEV4cG9zZUludm
> > 9jYXRpb25JbnRlcmNlcHRvci5qYXZhcQB-ACJwcHNxAH4ADAEAAAC6cQB-AA5xAH4ALnEA
> > fgAvcQB-ADJwcHNxAH4ADAEAAADUcQB-AA50ADRvcmcuc3ByaW5nZnJhbWV3b3JrLmFvcC
> > 5mcmFtZXdvcmsuSmRrRHluYW1pY0FvcFByb3h5dAAXSmRrRHluYW1pY0FvcFByb3h5Lmph
> > dmFxAH4AInBwc3EAfgAMAP____9wdAAXY29tLnN1bi5wcm94eS4kUHJveHkyMDVwcQB-AB
> > pwcHNxAH4ADAEAAAgZcQB-AA50ADZvcmcuYXBhY2hlLmNsb3Vkc3RhY2suc3RvcmFnZS52
> > b2x1bWUuVm9sdW1lU2VydmljZUltcGx0ABZWb2x1bWVTZXJ2aWNlSW1wbC5qYXZhcQB-AB
> > pwcHNxAH4ADAEAAAntcQB-AA50ACZjb20uY2xvdWQuc3RvcmFnZS5Wb2x1bWVBcGlTZXJ2
> > aWNlSW1wbHQAGVZvbHVtZUFwaVNlcnZpY2VJbXBsLmphdmF0AB1vcmNoZXN0cmF0ZVRha2
> > VWb2x1bWVTbmFwc2hvdHBwc3EAfgAMAQAADYlxAH4ADnEAfgBAcQB-AEFxAH4AQnBwc3EA
> > fgAMAv____5wcQB-ABxxAH4AHXEAfgAecQB-AB9xAH4AIHNxAH4ADAIAAAA-cHEAfgAccQ
> > B-AB1xAH4AInEAfgAfcQB-ACBzcQB-AAwCAAAAK3BxAH4AJHEAfgAlcQB-ACJxAH4AH3EA
> > fgAgc3EAfgAMAgAAAjZwcQB-ACdxAH4AKHEAfgAicQB-AB9xAH4AIHNxAH4ADAEAAABrcQ
> > B-AA50ACJjb20uY2xvdWQudm0uVm1Xb3JrSm9iSGFuZGxlclByb3h5dAAaVm1Xb3JrSm9i
> > SGFuZGxlclByb3h5LmphdmF0AA9oYW5kbGVWbVdvcmtKb2JwcHNxAH4ADAEAAA2PcQB-AA
> > 5xAH4AQHEAfgBBcQB-AEtwcHNxAH4ADAL____-cHEAfgAccQB-AB1xAH4AHnEAfgAfcQB-
> > ACBzcQB-AAwCAAAAPnBxAH4AHHEAfgAdcQB-ACJxAH4AH3EAfgAgc3EAfgAMAgAAACtwcQ
> > B-ACRxAH4AJXEAfgAicQB-AB9xAH4AIHNxAH4ADAIAAAI2cHEAfgAncQB-AChxAH4AInEA
> > fgAfcQB-ACBzcQB-AAwBAAABWHEAfgAOcQB-ACpxAH4AK3EAfgAscHBzcQB-AAwBAAAAxn
> > EAfgAOcQB-AC5xAH4AL3EAfgAwcHBzcQB-AAwBAAAAo3EAfgAOcQB-AC5xAH4AL3EAfgAy
> > cHBzcQB-AAwBAAAAX3EAfgAOcQB-ADRxAH4ANXEAfgAicHBzcQB-AAwBAAAAunEAfgAOcQ
> > B-AC5xAH4AL3EAfgAycHBzcQB-AAwBAAAA1HEAfgAOcQB-ADhxAH4AOXEAfgAicHBzcQB-
> > AAwA_____3B0ABdjb20uc3VuLnByb3h5LiRQcm94eTIxMnBxAH4AS3Bwc3EAfgAMAQAAAG
> > ZxAH4ADnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1Xb3JrSm9i
> > RGlzcGF0Y2hlci5qYXZhdAAGcnVuSm9icHBzcQB-AAwBAAACW3EAfgAOdAA_b3JnLmFwYW
> > NoZS5jbG91ZHN0YWNrLmZyYW1ld29yay5qb2JzLmltcGwuQXN5bmNKb2JNYW5hZ2VySW1w
> > bCQ1dAAYQXN5bmNKb2JNYW5hZ2VySW1wbC5qYXZhdAAMcnVuSW5Db250ZXh0cHBzcQB-AA
> > wBAAAAMXEAfgAOdAA-b3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5N
> > YW5hZ2VkQ29udGV4dFJ1bm5hYmxlJDF0ABtNYW5hZ2VkQ29udGV4dFJ1bm5hYmxlLmphdm
> > F0AANydW5wcHNxAH4ADAEAAAA4cQB-AA50AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFu
> > YWdlZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TW
> > FuYWdlZENvbnRleHQuamF2YXQABGNhbGxwcHNxAH4ADAEAAABncQB-AA50AEBvcmcuYXBh
> > Y2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb2
> > 50ZXh0cQB-AGd0AA9jYWxsV2l0aENvbnRleHRwcHNxAH4ADAEAAAA1cQB-AA5xAH4AanEA
> > fgBndAAOcnVuV2l0aENvbnRleHRwcHNxAH4ADAEAAAAucQB-AA50ADxvcmcuYXBhY2hlLm
> > Nsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxAH4A
> > Y3EAfgBkcHBzcQB-AAwBAAACJ3EAfgAOcQB-AF5xAH4AX3EAfgBkcHBzcQB-AAwCAAACA3
> > B0AC5qYXZhLnV0aWwuY29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAO
> > RXhlY3V0b3JzLmphdmFxAH4AaHEAfgAfcQB-ACBzcQB-AAwCAAABCHB0AB9qYXZhLnV0aW
> > wuY29uY3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFzay5qYXZhcQB-AGRxAH4AH3EA
> > fgAgc3EAfgAMAgAABGhwdAAnamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZW
> > N1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5Xb3JrZXJxAH4AH3EAfgAg
> > c3EAfgAMAgAAAnRwdAAuamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG
> > 9yJFdvcmtlcnEAfgB5cQB-AGRxAH4AH3EAfgAgc3EAfgAMAgAAA0JwdAAQamF2YS5sYW5n
> > LlRocmVhZHQAC1RocmVhZC5qYXZhcQB-AGRxAH4AH3EAfgAgc3IAH2phdmEudXRpbC5Db2
> > xsZWN0aW9ucyRFbXB0eUxpc3R6uBe0PKee3gIAAHhweAAAEJp3CAAAAAAAAAAAeA
> >
> > 2020-11-09 12:56:48,380 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897) (logid:db479fbc)
> > Publish async job-1897 complete on message bus
> >
> > 2020-11-09 12:56:48,380 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897) (logid:db479fbc)
> > Wake up jobs related to job-1897
> >
> > 2020-11-09 12:56:48,380 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897) (logid:db479fbc)
> > Update db status for job-1897
> >
> > 2020-11-09 12:56:48,381 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897) (logid:db479fbc)
> > Wake up jobs joined with job-1897 and disjoin all subjobs created from
> > job- 1897
> >
> > 2020-11-09 12:56:48,385 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897) (logid:db479fbc)
> > Done executing com.cloud.vm.VmWorkTakeVolumeSnapshot for job-1897
> >
> > 2020-11-09 12:56:48,385 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897) (logid:db479fbc)
> > Remove job-1897 from job monitoring
> >
> > 2020-11-09 12:56:48,392 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (API-Job-Executor-35:ctx-326b32af job-1894) (logid:db479fbc) Complete
> > async job-1894, jobStatus: FAILED, resultCode: 530, result:
> > org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":
> > [],"errorcode":530,"errortext":"Failed
> > to create snapshot due to an internal error creating snapshot for
> > volume 9945a23b-a376-4f85-b378-9f7da8017dac"}
> >
> > 2020-11-09 12:56:48,392 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (API-Job-Executor-35:ctx-326b32af job-1894) (logid:db479fbc) Publish
> > async
> > job-1894 complete on message bus
> >
>
>
>
>
>
>
> --
>
>
>
> Andrija Panić
>
>