You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Charlie Holeowsky <ch...@gmail.com> on 2020/02/03 08:57:42 UTC
Re: new primary storage
Hi Daan and users,
Yes, I resolved the issue but it's only a work-around, I have found that
the problem is reproducible but I do not understand if I am the only one to
have noticed it and if there is another more elegant and definitive
solution.
Il giorno ven 31 gen 2020 alle ore 13:31 Daan Hoogland <
daan.hoogland@gmail.com> ha scritto:
> sorry to not have any focus on this Charlie,
> Do I read correctly that you resolved your issue?
>
> On Tue, Jan 28, 2020 at 2:46 PM Charlie Holeowsky <
> charlie.holeowsky@gmail.com> wrote:
>
>> In this period I have performed some testsand I found a workarount for
>> the metrics problem.
>> I created a test environment in the laboratory with the main
>> characteristics equal to that of production (acs 4.11.2.0, all Ubuntu 16.04
>> OS, KVM, NFS as shared storage and advanced network). Then I added a ubuntu
>> 18.04 new primary storage.
>>
>> I create a new VM in the new storage server and and after a while the
>> metrics appeared as on the first storage, so the storage is working.
>>
>> I destroyed this VM, I create a new one on the first (old) storage and
>> then I migrate it on the new storage.
>>
>> After migrating the disk from the first primary storage to the second one
>> i have encountered the same problem (same error on agent.log
>> com.cloud.utils.exception.CloudRuntimeException: Can't find
>> volume:5a521eb0...) and the volume metrics and metrics didn't appear or
>> update.
>>
>> In this test environment I created a symbolic link in the filesystem of
>> the storage server taking the name of the disk just migrated (the one with
>> the name path) and the uuid name (the one that appear in error message).
>>
>>
>> Here is an example to better explain me.
>>
>> I took the list of volumes where path is different from uuid which
>> returns the data of the migrated volumes:
>>
>> mysql> select id,uuid,path from volumes where uuid!=path;
>>
>> +----+--------------------------------------+--------------------------------------+
>> | id | uuid | path
>> |
>>
>> +----+--------------------------------------+--------------------------------------+
>> | 10 | 5a521eb0-266f-4353-b4b2-1d63a483e5b5 |
>> 165b92ba-68f1-4172-be35-bbe1d032cb7c |
>> | 12 | acb3bb29-9bac-4a2a-aefa-3c6ac1c2846b |
>> 56aa3961-dbc2-4f98-9246-a7497eef3214 |
>>
>> +----+--------------------------------------+--------------------------------------+
>>
>> In the storage server I make the symbolic links (ln -s <path> <uuid>):
>>
>> # ln
>> -s 5a521eb0-266f-4353-b4b2-1d63a483e5b5 165b92ba-68f1-4172-be35-bbe1d032cb7c
>> # ln -s acb3bb29-9bac-4a2a-aefa-3c6ac1c2846b
>> 56aa3961-dbc2-4f98-9246-a7497eef3214
>>
>> After doing this, I waited some time and then I found that metrics were
>> updated and the message in agent.log no longer appeared.
>>
>>
>> Il giorno gio 23 gen 2020 alle ore 17:53 Charlie Holeowsky <
>> charlie.holeowsky@gmail.com> ha scritto:
>>
>>> I still don't understand
>>> why com.cloud.hypervisor.kvm.storage.LibvirtStoragePool don't find the
>>> volume d93d3c0a-3859-4473-951d-9b5c5912c76 that exists as file
>>> 39148fe1-842b-433a-8a7f-85e90f316e04...
>>>
>>> It's the only anomaly I have found. Where can I look again?
>>>
>>> Il giorno lun 20 gen 2020 alle ore 16:27 Daan Hoogland <
>>> daan.hoogland@gmail.com> ha scritto:
>>>
>>>> but the record you send earlier also says that is should be looking for
>>>> 39148fe1-842b-433a-8a7f-85e90f316e04, in the path field. the message might
>>>> be just that, a message.
>>>>
>>>> On Mon, Jan 20, 2020 at 3:35 PM Charlie Holeowsky <
>>>> charlie.holeowsky@gmail.com> wrote:
>>>>
>>>>> I think that's the problem because in the logs that I have forwarded
>>>>> it reads:
>>>>> Can't find volume: d93d3c0a-3859-4473-951d-9b5c5912c767
>>>>>
>>>>> This is the volume ID of migrated file but it do not exist on primary
>>>>> storage (new or old one) but it exist
>>>>> as 39148fe1-842b-433a-8a7f-85e90f316e04.
>>>>>
>>>>> Il giorno lun 20 gen 2020 alle ore 12:35 Daan Hoogland <
>>>>> daan.hoogland@gmail.com> ha scritto:
>>>>>
>>>>>> also, can you see the primary storage being mounted?
>>>>>>
>>>>>>
>>>>>> On Mon, Jan 20, 2020 at 12:33 PM Daan Hoogland <
>>>>>> daan.hoogland@gmail.com> wrote:
>>>>>>
>>>>>>> Why do you think that Charlie? Is it in the logs like that somewhere?
>>>>>>>
>>>>>>> On Mon, Jan 20, 2020 at 9:52 AM Charlie Holeowsky <
>>>>>>> charlie.holeowsky@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi Daan,
>>>>>>>> in fact I find the volume file
>>>>>>>> (39148fe1-842b-433a-8a7f-85e90f316e04) in the repositry id = 3 (the new
>>>>>>>> one) but it seems to me that the cloudstack system goes looking for the
>>>>>>>> volume with its "old" name (path) that doesn't exist...
>>>>>>>>
>>>>>>>> Il giorno sab 18 gen 2020 alle ore 21:41 Daan Hoogland <
>>>>>>>> daan.hoogland@gmail.com> ha scritto:
>>>>>>>>
>>>>>>>>> Charlie,
>>>>>>>>> forgive my not replying in a timely manner. This might happen if
>>>>>>>>> the disk was migrated. In this case probably from primary storage with id 1
>>>>>>>>> to the one with id 3. the second record (pool_id 1) is removed, so you can
>>>>>>>>> ignore that one. The first seems legit. You should be able to find that
>>>>>>>>> disks on your primary storage with id 3.
>>>>>>>>> hope this helps.
>>>>>>>>>
>>>>>>>>> On Thu, Jan 16, 2020 at 2:07 PM Charlie Holeowsky <
>>>>>>>>> charlie.holeowsky@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Daan and users,
>>>>>>>>>> to better explain I show you the two records related to the disk
>>>>>>>>>> that generates the error message.
>>>>>>>>>>
>>>>>>>>>> In the first query there are the data of the disk currently in
>>>>>>>>>> use which has the "uuid" equal to the name searched by the procedure
>>>>>>>>>> com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.getPhysicalDiskand a
>>>>>>>>>> has a different "path" field.
>>>>>>>>>>
>>>>>>>>>> In the second one we note that the "path" field is equal to the
>>>>>>>>>> "uuid" of the volume in use but has null "uuid" (and state=Expunged).
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> mysql> select * from volumes where
>>>>>>>>>> uuid='d93d3c0a-3859-4473-951d-9b5c5912c767';
>>>>>>>>>>
>>>>>>>>>> +-----+------------+-----------+---------+--------------+-------------+-----------+----------+--------------------------------------+-------------+---------------------+--------------------------------------+--------+----------------+------------+---------+-------------+-----------+------------------+-------------+----------------------------+-------------+---------------------+----------+---------------------+---------+-------+------------+--------------+-----------+------------------------+--------+----------------+--------+----------+----------+---------------+-------------------+
>>>>>>>>>> | id | account_id | domain_id | pool_id | last_pool_id |
>>>>>>>>>> instance_id | device_id | name | uuid |
>>>>>>>>>> size | folder | path |
>>>>>>>>>> pod_id | data_center_id | iscsi_name | host_ip | volume_type | pool_type |
>>>>>>>>>> disk_offering_id | template_id | first_snapshot_backup_uuid | recreatable |
>>>>>>>>>> created | attached | updated | removed | state |
>>>>>>>>>> chain_info | update_count | disk_type | vm_snapshot_chain_size | iso_id |
>>>>>>>>>> display_volume | format | min_iops | max_iops | hv_ss_reserve |
>>>>>>>>>> provisioning_type |
>>>>>>>>>>
>>>>>>>>>> +-----+------------+-----------+---------+--------------+-------------+-----------+----------+--------------------------------------+-------------+---------------------+--------------------------------------+--------+----------------+------------+---------+-------------+-----------+------------------+-------------+----------------------------+-------------+---------------------+----------+---------------------+---------+-------+------------+--------------+-----------+------------------------+--------+----------------+--------+----------+----------+---------------+-------------------+
>>>>>>>>>> | 213 | 2 | 1 | 3 | 1 |
>>>>>>>>>> 148 | 1 | DATA-148 | d93d3c0a-3859-4473-951d-9b5c5912c767 |
>>>>>>>>>> 53687091200 | /srv/primary | 39148fe1-842b-433a-8a7f-85e90f316e04 | NULL
>>>>>>>>>> | 1 | NULL | NULL | DATADISK | NULL |
>>>>>>>>>> 34 | NULL | NULL | 0 |
>>>>>>>>>> 2019-11-26 10:41:46 | NULL | 2019-11-26 10:41:50 | NULL | Ready |
>>>>>>>>>> NULL | 2 | NULL | NULL | NULL |
>>>>>>>>>> 1 | QCOW2 | NULL | NULL | NULL | thin
>>>>>>>>>> |
>>>>>>>>>>
>>>>>>>>>> +-----+------------+-----------+---------+--------------+-------------+-----------+----------+--------------------------------------+-------------+---------------------+--------------------------------------+--------+----------------+------------+---------+-------------+-----------+------------------+-------------+----------------------------+-------------+---------------------+----------+---------------------+---------+-------+------------+--------------+-----------+------------------------+--------+----------------+--------+----------+----------+---------------+-------------------+
>>>>>>>>>> 1 row in set (0.00 sec)
>>>>>>>>>>
>>>>>>>>>> mysql> select * from volumes where
>>>>>>>>>> path='d93d3c0a-3859-4473-951d-9b5c5912c767';
>>>>>>>>>>
>>>>>>>>>> +-----+------------+-----------+---------+--------------+-------------+-----------+----------+------+-------------+--------+--------------------------------------+--------+----------------+------------+---------+-------------+-----------+------------------+-------------+----------------------------+-------------+---------------------+----------+---------------------+---------------------+----------+------------+--------------+-----------+------------------------+--------+----------------+--------+----------+----------+---------------+-------------------+
>>>>>>>>>> | id | account_id | domain_id | pool_id | last_pool_id |
>>>>>>>>>> instance_id | device_id | name | uuid | size | folder | path
>>>>>>>>>> | pod_id | data_center_id | iscsi_name |
>>>>>>>>>> host_ip | volume_type | pool_type | disk_offering_id | template_id |
>>>>>>>>>> first_snapshot_backup_uuid | recreatable | created | attached |
>>>>>>>>>> updated | removed | state | chain_info |
>>>>>>>>>> update_count | disk_type | vm_snapshot_chain_size | iso_id | display_volume
>>>>>>>>>> | format | min_iops | max_iops | hv_ss_reserve | provisioning_type |
>>>>>>>>>>
>>>>>>>>>> +-----+------------+-----------+---------+--------------+-------------+-----------+----------+------+-------------+--------+--------------------------------------+--------+----------------+------------+---------+-------------+-----------+------------------+-------------+----------------------------+-------------+---------------------+----------+---------------------+---------------------+----------+------------+--------------+-----------+------------------------+--------+----------------+--------+----------+----------+---------------+-------------------+
>>>>>>>>>> | 212 | 2 | 1 | 1 | NULL |
>>>>>>>>>> 148 | 1 | DATA-148 | NULL | 53687091200 | NULL |
>>>>>>>>>> d93d3c0a-3859-4473-951d-9b5c5912c767 | NULL | 1 | NULL
>>>>>>>>>> | NULL | DATADISK | NULL | 34 | NULL | NULL
>>>>>>>>>> | 0 | 2019-11-26 10:38:23 | NULL |
>>>>>>>>>> 2019-11-26 10:41:50 | 2019-11-26 10:41:50 | Expunged | NULL |
>>>>>>>>>> 8 | NULL | NULL | NULL | 1 | QCOW2
>>>>>>>>>> | NULL | NULL | NULL | thin |
>>>>>>>>>>
>>>>>>>>>> +-----+------------+-----------+---------+--------------+-------------+-----------+----------+------+-------------+--------+--------------------------------------+--------+----------------+------------+---------+-------------+-----------+------------------+-------------+----------------------------+-------------+---------------------+----------+---------------------+---------------------+----------+------------+--------------+-----------+------------------------+--------+----------------+--------+----------+----------+---------------+-------------------+
>>>>>>>>>> 1 row in set (0.00 sec)
>>>>>>>>>>
>>>>>>>>>> Il giorno mar 14 gen 2020 alle ore 15:05 Daan Hoogland <
>>>>>>>>>> daan.hoogland@gmail.com> ha scritto:
>>>>>>>>>>
>>>>>>>>>>> So Charlie,
>>>>>>>>>>> d93d3c0a-3859-4473-951d-9b5c5912c767 is actually a valid disk?
>>>>>>>>>>> does it exist on the backend nfs?
>>>>>>>>>>> and the pool 9af0d1c6-85f2-3c55-94af-6ac17cb4024c does it exist
>>>>>>>>>>> both in cloudstack and on the backend?
>>>>>>>>>>>
>>>>>>>>>>> if both are answered with yes, you probably have a permissions
>>>>>>>>>>> issue, which might be in the network.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Jan 14, 2020 at 10:21 AM Charlie Holeowsky <
>>>>>>>>>>> charlie.holeowsky@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Daan and users,
>>>>>>>>>>>> the infrastructure is based on the Linux environment. The
>>>>>>>>>>>> management server, hosts and storage are all Ubuntu 16.04 except the new
>>>>>>>>>>>> storage server which is an Ubuntu 18.04. The hypervisor used is Qemu-kvm
>>>>>>>>>>>> with NFS to share the storage.
>>>>>>>>>>>>
>>>>>>>>>>>> We tried to add another primary storage and creating a VM that
>>>>>>>>>>>> would use it we found no problems, the statistics update and no error
>>>>>>>>>>>> messages appear.
>>>>>>>>>>>>
>>>>>>>>>>>> Gere is an excerpt of the logs of the most complete agent:
>>>>>>>>>>>>
>>>>>>>>>>>> 2020-01-14 09:01:45,749 INFO
>>>>>>>>>>>> [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null)
>>>>>>>>>>>> (logid:c3851d3a) Trying to fetch storage pool
>>>>>>>>>>>> 171e90f4-511e-3b10-9310-b9eec0094be6 from libvirt
>>>>>>>>>>>> 2020-01-14 09:01:45,752 INFO
>>>>>>>>>>>> [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null)
>>>>>>>>>>>> (logid:c3851d3a) Asking libvirt to refresh storage pool
>>>>>>>>>>>> 171e90f4-511e-3b10-9310-b9eec0094be6
>>>>>>>>>>>> 2020-01-14 09:01:46,641 INFO
>>>>>>>>>>>> [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null)
>>>>>>>>>>>> (logid:c3851d3a) Trying to fetch storage pool
>>>>>>>>>>>> 9af0d1c6-85f2-3c55-94af-6ac17cb4024c from libvirt
>>>>>>>>>>>> 2020-01-14 09:01:46,643 INFO
>>>>>>>>>>>> [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null)
>>>>>>>>>>>> (logid:c3851d3a) Asking libvirt to refresh storage pool
>>>>>>>>>>>> 9af0d1c6-85f2-3c55-94af-6ac17cb4024c
>>>>>>>>>>>> 2020-01-14 09:05:51,529 INFO
>>>>>>>>>>>> [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null)
>>>>>>>>>>>> (logid:2765ff88) Trying to fetch storage pool
>>>>>>>>>>>> 9af0d1c6-85f2-3c55-94af-6ac17cb4024c from libvirt
>>>>>>>>>>>> 2020-01-14 09:05:51,532 INFO
>>>>>>>>>>>> [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null)
>>>>>>>>>>>> (logid:2765ff88) Asking libvirt to refresh storage pool
>>>>>>>>>>>> 9af0d1c6-85f2-3c55-94af-6ac17cb4024c
>>>>>>>>>>>> 2020-01-14 09:10:47,286 INFO
>>>>>>>>>>>> [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null)
>>>>>>>>>>>> (logid:6d27b740) Trying to fetch storage pool
>>>>>>>>>>>> 9af0d1c6-85f2-3c55-94af-6ac17cb4024c from libvirt
>>>>>>>>>>>> 2020-01-14 09:10:47,419 WARN [cloud.agent.Agent]
>>>>>>>>>>>> (agentRequest-Handler-3:null) (logid:6d27b740) Caught:
>>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Can't find
>>>>>>>>>>>> volume:d93d3c0a-3859-4473-951d-9b5c5912c767
>>>>>>>>>>>> at
>>>>>>>>>>>> com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.getPhysicalDisk(LibvirtStoragePool.java:149)
>>>>>>>>>>>> at
>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.getVolumeStat(LibvirtGetVolumeStatsCommandWrapper.java:63)
>>>>>>>>>>>> at
>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:52)
>>>>>>>>>>>> at
>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:40)
>>>>>>>>>>>> at
>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
>>>>>>>>>>>> at
>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1450)
>>>>>>>>>>>> at com.cloud.agent.Agent.processRequest(Agent.java:645)
>>>>>>>>>>>> at
>>>>>>>>>>>> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1083)
>>>>>>>>>>>> at com.cloud.utils.nio.Task.call(Task.java:83)
>>>>>>>>>>>> at com.cloud.utils.nio.Task.call(Task.java:29)
>>>>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>>>>>>>> 2020-01-14 09:20:48,390 INFO
>>>>>>>>>>>> [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null)
>>>>>>>>>>>> (logid:ec72387b) Trying to fetch storage pool
>>>>>>>>>>>> 9af0d1c6-85f2-3c55-94af-6ac17cb4024c from libvirt
>>>>>>>>>>>> 2020-01-14 09:20:48,536 WARN [cloud.agent.Agent]
>>>>>>>>>>>> (agentRequest-Handler-4:null) (logid:ec72387b) Caught:
>>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Can't find
>>>>>>>>>>>> volume:d93d3c0a-3859-4473-951d-9b5c5912c767
>>>>>>>>>>>> at
>>>>>>>>>>>> com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.getPhysicalDisk(LibvirtStoragePool.java:149)
>>>>>>>>>>>> at
>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.getVolumeStat(LibvirtGetVolumeStatsCommandWrapper.java:63)
>>>>>>>>>>>> at
>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:52)
>>>>>>>>>>>> at
>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:40)
>>>>>>>>>>>> at
>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
>>>>>>>>>>>> at
>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1450)
>>>>>>>>>>>> at com.cloud.agent.Agent.processRequest(Agent.java:645)
>>>>>>>>>>>> at
>>>>>>>>>>>> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1083)
>>>>>>>>>>>> at com.cloud.utils.nio.Task.call(Task.java:83)
>>>>>>>>>>>> at com.cloud.utils.nio.Task.call(Task.java:29)
>>>>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>>>>>>>> 2020-01-14 09:25:15,259 INFO
>>>>>>>>>>>> [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null)
>>>>>>>>>>>> (logid:1a7e082e) Trying to fetch storage pool
>>>>>>>>>>>> 9af0d1c6-85f2-3c55-94af-6ac17cb4024c from libvirt
>>>>>>>>>>>> 2020-01-14 09:25:15,261 INFO
>>>>>>>>>>>> [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null)
>>>>>>>>>>>> (logid:1a7e082e) Asking libvirt to refresh storage pool
>>>>>>>>>>>> 9af0d1c6-85f2-3c55-94af-6ac17cb4024c
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> And here the management server log:
>>>>>>>>>>>>
>>>>>>>>>>>> 2020-01-14 09:21:27,105 DEBUG [c.c.a.t.Request]
>>>>>>>>>>>> (AgentManager-Handler-2:null) (logid:) Seq 15-705657766613619075:
>>>>>>>>>>>> Processing: { Ans: , MgmtId: 220777304233416, via: 15, Ver: v1, Flags: 10,
>>>>>>>>>>>> [{"com.cloud.agent.api.Answer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException:
>>>>>>>>>>>> Can't find volume:d93d3c0a-3859-4473-951d-9b5c5912c767\n\tat
>>>>>>>>>>>> com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.getPhysicalDisk(LibvirtStoragePool.java:149)\n\tat
>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.getVolumeStat(LibvirtGetVolumeStatsCommandWrapper.java:63)\n\tat
>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:52)\n\tat
>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:40)\n\tat
>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)\n\tat
>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1450)\n\tat
>>>>>>>>>>>> com.cloud.agent.Agent.processRequest(Agent.java:645)\n\tat
>>>>>>>>>>>> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1083)\n\tat
>>>>>>>>>>>> com.cloud.utils.nio.Task.call(Task.java:83)\n\tat
>>>>>>>>>>>> com.cloud.utils.nio.Task.call(Task.java:29)\n\tat
>>>>>>>>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat
>>>>>>>>>>>> java.lang.Thread.run(Thread.java:748)\n","wait":0}}] }
>>>>>>>>>>>> 2020-01-14 09:21:27,105 DEBUG [c.c.a.t.Request]
>>>>>>>>>>>> (StatsCollector-6:ctx-fd801d0a) (logid:ec72387b) Seq 15-705657766613619075:
>>>>>>>>>>>> Received: { Ans: , MgmtId: 220777304233416, via: 15(csdell017), Ver: v1,
>>>>>>>>>>>> Flags: 10, { Answer } }
>>>>>>>>>>>> 2020-01-14 09:21:27,105 DEBUG [c.c.a.m.AgentManagerImpl]
>>>>>>>>>>>> (StatsCollector-6:ctx-fd801d0a) (logid:ec72387b) Details from executing
>>>>>>>>>>>> class com.cloud.agent.api.GetVolumeStatsCommand:
>>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Can't find
>>>>>>>>>>>> volume:d93d3c0a-3859-4473-951d-9b5c5912c767
>>>>>>>>>>>> at
>>>>>>>>>>>> com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.getPhysicalDisk(LibvirtStoragePool.java:149)
>>>>>>>>>>>> at
>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.getVolumeStat(LibvirtGetVolumeStatsCommandWrapper.java:63)
>>>>>>>>>>>> at
>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:52)
>>>>>>>>>>>> at
>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:40)
>>>>>>>>>>>> at
>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
>>>>>>>>>>>> at
>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1450)
>>>>>>>>>>>> at com.cloud.agent.Agent.processRequest(Agent.java:645)
>>>>>>>>>>>> at
>>>>>>>>>>>> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1083)
>>>>>>>>>>>> at com.cloud.utils.nio.Task.call(Task.java:83)
>>>>>>>>>>>> at com.cloud.utils.nio.Task.call(Task.java:29)
>>>>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> on 09/01/20 12:58, Daan Hoogland wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Charlie, I think you'll have to explain a bit more about your environment
>>>>>>>>>>>> to get an answer. what type of storage is it? Where did you migrate the VM
>>>>>>>>>>>> from and to? What types() of hypervisors are you using? Though saying **the**
>>>>>>>>>>>> agent logs suggests KVM, you are still leaving people guessing a lot.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Daan
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Daan
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Daan
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Daan
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Daan
>>>>
>>>
>
> --
> Daan
>
Re: new primary storage
Posted by Daan Hoogland <da...@gmail.com>.
sounds good Charlie, I hope you get a chance to upgrade soon.
On Mon, Feb 17, 2020 at 10:55 AM Charlie Holeowsky <
charlie.holeowsky@gmail.com> wrote:
> Hi Daan and Users,
> I opened the issue on Github and found that this bug (#3181) was fixed on
> 4.13.0 release.
>
> Il giorno lun 3 feb 2020 alle ore 10:25 Daan Hoogland <
> daan.hoogland@gmail.com> ha scritto:
>
>> thanks Charlie, can you take your issue to a github issue on the project
>> please? thanks
>>
>> On Mon, Feb 3, 2020 at 9:57 AM Charlie Holeowsky <
>> charlie.holeowsky@gmail.com> wrote:
>>
>>> Hi Daan and users,
>>> Yes, I resolved the issue but it's only a work-around, I have found that
>>> the problem is reproducible but I do not understand if I am the only one to
>>> have noticed it and if there is another more elegant and definitive
>>> solution.
>>>
>>>
>>>
>>> Il giorno ven 31 gen 2020 alle ore 13:31 Daan Hoogland <
>>> daan.hoogland@gmail.com> ha scritto:
>>>
>>>> sorry to not have any focus on this Charlie,
>>>> Do I read correctly that you resolved your issue?
>>>>
>>>> On Tue, Jan 28, 2020 at 2:46 PM Charlie Holeowsky <
>>>> charlie.holeowsky@gmail.com> wrote:
>>>>
>>>>> In this period I have performed some testsand I found a workarount for
>>>>> the metrics problem.
>>>>> I created a test environment in the laboratory with the main
>>>>> characteristics equal to that of production (acs 4.11.2.0, all Ubuntu 16.04
>>>>> OS, KVM, NFS as shared storage and advanced network). Then I added a ubuntu
>>>>> 18.04 new primary storage.
>>>>>
>>>>> I create a new VM in the new storage server and and after a while the
>>>>> metrics appeared as on the first storage, so the storage is working.
>>>>>
>>>>> I destroyed this VM, I create a new one on the first (old) storage and
>>>>> then I migrate it on the new storage.
>>>>>
>>>>> After migrating the disk from the first primary storage to the second
>>>>> one i have encountered the same problem (same error on agent.log
>>>>> com.cloud.utils.exception.CloudRuntimeException: Can't find
>>>>> volume:5a521eb0...) and the volume metrics and metrics didn't appear or
>>>>> update.
>>>>>
>>>>> In this test environment I created a symbolic link in the filesystem
>>>>> of the storage server taking the name of the disk just migrated (the one
>>>>> with the name path) and the uuid name (the one that appear in error
>>>>> message).
>>>>>
>>>>>
>>>>> Here is an example to better explain me.
>>>>>
>>>>> I took the list of volumes where path is different from uuid which
>>>>> returns the data of the migrated volumes:
>>>>>
>>>>> mysql> select id,uuid,path from volumes where uuid!=path;
>>>>>
>>>>> +----+--------------------------------------+--------------------------------------+
>>>>> | id | uuid | path
>>>>> |
>>>>>
>>>>> +----+--------------------------------------+--------------------------------------+
>>>>> | 10 | 5a521eb0-266f-4353-b4b2-1d63a483e5b5 |
>>>>> 165b92ba-68f1-4172-be35-bbe1d032cb7c |
>>>>> | 12 | acb3bb29-9bac-4a2a-aefa-3c6ac1c2846b |
>>>>> 56aa3961-dbc2-4f98-9246-a7497eef3214 |
>>>>>
>>>>> +----+--------------------------------------+--------------------------------------+
>>>>>
>>>>> In the storage server I make the symbolic links (ln -s <path> <uuid>):
>>>>>
>>>>> # ln
>>>>> -s 5a521eb0-266f-4353-b4b2-1d63a483e5b5 165b92ba-68f1-4172-be35-bbe1d032cb7c
>>>>> # ln -s acb3bb29-9bac-4a2a-aefa-3c6ac1c2846b
>>>>> 56aa3961-dbc2-4f98-9246-a7497eef3214
>>>>>
>>>>> After doing this, I waited some time and then I found that metrics
>>>>> were updated and the message in agent.log no longer appeared.
>>>>>
>>>>>
>>>>> Il giorno gio 23 gen 2020 alle ore 17:53 Charlie Holeowsky <
>>>>> charlie.holeowsky@gmail.com> ha scritto:
>>>>>
>>>>>> I still don't understand
>>>>>> why com.cloud.hypervisor.kvm.storage.LibvirtStoragePool don't find the
>>>>>> volume d93d3c0a-3859-4473-951d-9b5c5912c76 that exists as file
>>>>>> 39148fe1-842b-433a-8a7f-85e90f316e04...
>>>>>>
>>>>>> It's the only anomaly I have found. Where can I look again?
>>>>>>
>>>>>> Il giorno lun 20 gen 2020 alle ore 16:27 Daan Hoogland <
>>>>>> daan.hoogland@gmail.com> ha scritto:
>>>>>>
>>>>>>> but the record you send earlier also says that is should be looking
>>>>>>> for 39148fe1-842b-433a-8a7f-85e90f316e04, in the path field. the message
>>>>>>> might be just that, a message.
>>>>>>>
>>>>>>> On Mon, Jan 20, 2020 at 3:35 PM Charlie Holeowsky <
>>>>>>> charlie.holeowsky@gmail.com> wrote:
>>>>>>>
>>>>>>>> I think that's the problem because in the logs that I have
>>>>>>>> forwarded it reads:
>>>>>>>> Can't find volume: d93d3c0a-3859-4473-951d-9b5c5912c767
>>>>>>>>
>>>>>>>> This is the volume ID of migrated file but it do not exist on
>>>>>>>> primary storage (new or old one) but it exist
>>>>>>>> as 39148fe1-842b-433a-8a7f-85e90f316e04.
>>>>>>>>
>>>>>>>> Il giorno lun 20 gen 2020 alle ore 12:35 Daan Hoogland <
>>>>>>>> daan.hoogland@gmail.com> ha scritto:
>>>>>>>>
>>>>>>>>> also, can you see the primary storage being mounted?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Jan 20, 2020 at 12:33 PM Daan Hoogland <
>>>>>>>>> daan.hoogland@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Why do you think that Charlie? Is it in the logs like that
>>>>>>>>>> somewhere?
>>>>>>>>>>
>>>>>>>>>> On Mon, Jan 20, 2020 at 9:52 AM Charlie Holeowsky <
>>>>>>>>>> charlie.holeowsky@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Daan,
>>>>>>>>>>> in fact I find the volume file
>>>>>>>>>>> (39148fe1-842b-433a-8a7f-85e90f316e04) in the repositry id = 3 (the new
>>>>>>>>>>> one) but it seems to me that the cloudstack system goes looking for the
>>>>>>>>>>> volume with its "old" name (path) that doesn't exist...
>>>>>>>>>>>
>>>>>>>>>>> Il giorno sab 18 gen 2020 alle ore 21:41 Daan Hoogland <
>>>>>>>>>>> daan.hoogland@gmail.com> ha scritto:
>>>>>>>>>>>
>>>>>>>>>>>> Charlie,
>>>>>>>>>>>> forgive my not replying in a timely manner. This might happen
>>>>>>>>>>>> if the disk was migrated. In this case probably from primary storage with
>>>>>>>>>>>> id 1 to the one with id 3. the second record (pool_id 1) is removed, so you
>>>>>>>>>>>> can ignore that one. The first seems legit. You should be able to find that
>>>>>>>>>>>> disks on your primary storage with id 3.
>>>>>>>>>>>> hope this helps.
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Jan 16, 2020 at 2:07 PM Charlie Holeowsky <
>>>>>>>>>>>> charlie.holeowsky@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Daan and users,
>>>>>>>>>>>>> to better explain I show you the two records related to the
>>>>>>>>>>>>> disk that generates the error message.
>>>>>>>>>>>>>
>>>>>>>>>>>>> In the first query there are the data of the disk currently in
>>>>>>>>>>>>> use which has the "uuid" equal to the name searched by the procedure
>>>>>>>>>>>>> com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.getPhysicalDiskand a
>>>>>>>>>>>>> has a different "path" field.
>>>>>>>>>>>>>
>>>>>>>>>>>>> In the second one we note that the "path" field is equal to
>>>>>>>>>>>>> the "uuid" of the volume in use but has null "uuid" (and state=Expunged).
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> mysql> select * from volumes where
>>>>>>>>>>>>> uuid='d93d3c0a-3859-4473-951d-9b5c5912c767';
>>>>>>>>>>>>>
>>>>>>>>>>>>> +-----+------------+-----------+---------+--------------+-------------+-----------+----------+--------------------------------------+-------------+---------------------+--------------------------------------+--------+----------------+------------+---------+-------------+-----------+------------------+-------------+----------------------------+-------------+---------------------+----------+---------------------+---------+-------+------------+--------------+-----------+------------------------+--------+----------------+--------+----------+----------+---------------+-------------------+
>>>>>>>>>>>>> | id | account_id | domain_id | pool_id | last_pool_id |
>>>>>>>>>>>>> instance_id | device_id | name | uuid |
>>>>>>>>>>>>> size | folder | path |
>>>>>>>>>>>>> pod_id | data_center_id | iscsi_name | host_ip | volume_type | pool_type |
>>>>>>>>>>>>> disk_offering_id | template_id | first_snapshot_backup_uuid | recreatable |
>>>>>>>>>>>>> created | attached | updated | removed | state |
>>>>>>>>>>>>> chain_info | update_count | disk_type | vm_snapshot_chain_size | iso_id |
>>>>>>>>>>>>> display_volume | format | min_iops | max_iops | hv_ss_reserve |
>>>>>>>>>>>>> provisioning_type |
>>>>>>>>>>>>>
>>>>>>>>>>>>> +-----+------------+-----------+---------+--------------+-------------+-----------+----------+--------------------------------------+-------------+---------------------+--------------------------------------+--------+----------------+------------+---------+-------------+-----------+------------------+-------------+----------------------------+-------------+---------------------+----------+---------------------+---------+-------+------------+--------------+-----------+------------------------+--------+----------------+--------+----------+----------+---------------+-------------------+
>>>>>>>>>>>>> | 213 | 2 | 1 | 3 | 1 |
>>>>>>>>>>>>> 148 | 1 | DATA-148 | d93d3c0a-3859-4473-951d-9b5c5912c767 |
>>>>>>>>>>>>> 53687091200 | /srv/primary | 39148fe1-842b-433a-8a7f-85e90f316e04 | NULL
>>>>>>>>>>>>> | 1 | NULL | NULL | DATADISK | NULL |
>>>>>>>>>>>>> 34 | NULL | NULL | 0 |
>>>>>>>>>>>>> 2019-11-26 10:41:46 | NULL | 2019-11-26 10:41:50 | NULL | Ready |
>>>>>>>>>>>>> NULL | 2 | NULL | NULL | NULL |
>>>>>>>>>>>>> 1 | QCOW2 | NULL | NULL | NULL | thin
>>>>>>>>>>>>> |
>>>>>>>>>>>>>
>>>>>>>>>>>>> +-----+------------+-----------+---------+--------------+-------------+-----------+----------+--------------------------------------+-------------+---------------------+--------------------------------------+--------+----------------+------------+---------+-------------+-----------+------------------+-------------+----------------------------+-------------+---------------------+----------+---------------------+---------+-------+------------+--------------+-----------+------------------------+--------+----------------+--------+----------+----------+---------------+-------------------+
>>>>>>>>>>>>> 1 row in set (0.00 sec)
>>>>>>>>>>>>>
>>>>>>>>>>>>> mysql> select * from volumes where
>>>>>>>>>>>>> path='d93d3c0a-3859-4473-951d-9b5c5912c767';
>>>>>>>>>>>>>
>>>>>>>>>>>>> +-----+------------+-----------+---------+--------------+-------------+-----------+----------+------+-------------+--------+--------------------------------------+--------+----------------+------------+---------+-------------+-----------+------------------+-------------+----------------------------+-------------+---------------------+----------+---------------------+---------------------+----------+------------+--------------+-----------+------------------------+--------+----------------+--------+----------+----------+---------------+-------------------+
>>>>>>>>>>>>> | id | account_id | domain_id | pool_id | last_pool_id |
>>>>>>>>>>>>> instance_id | device_id | name | uuid | size | folder | path
>>>>>>>>>>>>> | pod_id | data_center_id | iscsi_name |
>>>>>>>>>>>>> host_ip | volume_type | pool_type | disk_offering_id | template_id |
>>>>>>>>>>>>> first_snapshot_backup_uuid | recreatable | created | attached |
>>>>>>>>>>>>> updated | removed | state | chain_info |
>>>>>>>>>>>>> update_count | disk_type | vm_snapshot_chain_size | iso_id | display_volume
>>>>>>>>>>>>> | format | min_iops | max_iops | hv_ss_reserve | provisioning_type |
>>>>>>>>>>>>>
>>>>>>>>>>>>> +-----+------------+-----------+---------+--------------+-------------+-----------+----------+------+-------------+--------+--------------------------------------+--------+----------------+------------+---------+-------------+-----------+------------------+-------------+----------------------------+-------------+---------------------+----------+---------------------+---------------------+----------+------------+--------------+-----------+------------------------+--------+----------------+--------+----------+----------+---------------+-------------------+
>>>>>>>>>>>>> | 212 | 2 | 1 | 1 | NULL |
>>>>>>>>>>>>> 148 | 1 | DATA-148 | NULL | 53687091200 | NULL |
>>>>>>>>>>>>> d93d3c0a-3859-4473-951d-9b5c5912c767 | NULL | 1 | NULL
>>>>>>>>>>>>> | NULL | DATADISK | NULL | 34 | NULL | NULL
>>>>>>>>>>>>> | 0 | 2019-11-26 10:38:23 | NULL |
>>>>>>>>>>>>> 2019-11-26 10:41:50 | 2019-11-26 10:41:50 | Expunged | NULL |
>>>>>>>>>>>>> 8 | NULL | NULL | NULL | 1 | QCOW2
>>>>>>>>>>>>> | NULL | NULL | NULL | thin |
>>>>>>>>>>>>>
>>>>>>>>>>>>> +-----+------------+-----------+---------+--------------+-------------+-----------+----------+------+-------------+--------+--------------------------------------+--------+----------------+------------+---------+-------------+-----------+------------------+-------------+----------------------------+-------------+---------------------+----------+---------------------+---------------------+----------+------------+--------------+-----------+------------------------+--------+----------------+--------+----------+----------+---------------+-------------------+
>>>>>>>>>>>>> 1 row in set (0.00 sec)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Il giorno mar 14 gen 2020 alle ore 15:05 Daan Hoogland <
>>>>>>>>>>>>> daan.hoogland@gmail.com> ha scritto:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> So Charlie,
>>>>>>>>>>>>>> d93d3c0a-3859-4473-951d-9b5c5912c767 is actually a valid
>>>>>>>>>>>>>> disk? does it exist on the backend nfs?
>>>>>>>>>>>>>> and the pool 9af0d1c6-85f2-3c55-94af-6ac17cb4024c does it
>>>>>>>>>>>>>> exist both in cloudstack and on the backend?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> if both are answered with yes, you probably have a
>>>>>>>>>>>>>> permissions issue, which might be in the network.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Jan 14, 2020 at 10:21 AM Charlie Holeowsky <
>>>>>>>>>>>>>> charlie.holeowsky@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Daan and users,
>>>>>>>>>>>>>>> the infrastructure is based on the Linux environment. The
>>>>>>>>>>>>>>> management server, hosts and storage are all Ubuntu 16.04 except the new
>>>>>>>>>>>>>>> storage server which is an Ubuntu 18.04. The hypervisor used is Qemu-kvm
>>>>>>>>>>>>>>> with NFS to share the storage.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> We tried to add another primary storage and creating a VM
>>>>>>>>>>>>>>> that would use it we found no problems, the statistics update and no error
>>>>>>>>>>>>>>> messages appear.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Gere is an excerpt of the logs of the most complete agent:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2020-01-14 09:01:45,749 INFO
>>>>>>>>>>>>>>> [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null)
>>>>>>>>>>>>>>> (logid:c3851d3a) Trying to fetch storage pool
>>>>>>>>>>>>>>> 171e90f4-511e-3b10-9310-b9eec0094be6 from libvirt
>>>>>>>>>>>>>>> 2020-01-14 09:01:45,752 INFO
>>>>>>>>>>>>>>> [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null)
>>>>>>>>>>>>>>> (logid:c3851d3a) Asking libvirt to refresh storage pool
>>>>>>>>>>>>>>> 171e90f4-511e-3b10-9310-b9eec0094be6
>>>>>>>>>>>>>>> 2020-01-14 09:01:46,641 INFO
>>>>>>>>>>>>>>> [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null)
>>>>>>>>>>>>>>> (logid:c3851d3a) Trying to fetch storage pool
>>>>>>>>>>>>>>> 9af0d1c6-85f2-3c55-94af-6ac17cb4024c from libvirt
>>>>>>>>>>>>>>> 2020-01-14 09:01:46,643 INFO
>>>>>>>>>>>>>>> [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null)
>>>>>>>>>>>>>>> (logid:c3851d3a) Asking libvirt to refresh storage pool
>>>>>>>>>>>>>>> 9af0d1c6-85f2-3c55-94af-6ac17cb4024c
>>>>>>>>>>>>>>> 2020-01-14 09:05:51,529 INFO
>>>>>>>>>>>>>>> [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null)
>>>>>>>>>>>>>>> (logid:2765ff88) Trying to fetch storage pool
>>>>>>>>>>>>>>> 9af0d1c6-85f2-3c55-94af-6ac17cb4024c from libvirt
>>>>>>>>>>>>>>> 2020-01-14 09:05:51,532 INFO
>>>>>>>>>>>>>>> [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null)
>>>>>>>>>>>>>>> (logid:2765ff88) Asking libvirt to refresh storage pool
>>>>>>>>>>>>>>> 9af0d1c6-85f2-3c55-94af-6ac17cb4024c
>>>>>>>>>>>>>>> 2020-01-14 09:10:47,286 INFO
>>>>>>>>>>>>>>> [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null)
>>>>>>>>>>>>>>> (logid:6d27b740) Trying to fetch storage pool
>>>>>>>>>>>>>>> 9af0d1c6-85f2-3c55-94af-6ac17cb4024c from libvirt
>>>>>>>>>>>>>>> 2020-01-14 09:10:47,419 WARN [cloud.agent.Agent]
>>>>>>>>>>>>>>> (agentRequest-Handler-3:null) (logid:6d27b740) Caught:
>>>>>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Can't find
>>>>>>>>>>>>>>> volume:d93d3c0a-3859-4473-951d-9b5c5912c767
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.getPhysicalDisk(LibvirtStoragePool.java:149)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.getVolumeStat(LibvirtGetVolumeStatsCommandWrapper.java:63)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:52)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:40)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1450)
>>>>>>>>>>>>>>> at com.cloud.agent.Agent.processRequest(Agent.java:645)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1083)
>>>>>>>>>>>>>>> at com.cloud.utils.nio.Task.call(Task.java:83)
>>>>>>>>>>>>>>> at com.cloud.utils.nio.Task.call(Task.java:29)
>>>>>>>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>>>>>>>>>>> 2020-01-14 09:20:48,390 INFO
>>>>>>>>>>>>>>> [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null)
>>>>>>>>>>>>>>> (logid:ec72387b) Trying to fetch storage pool
>>>>>>>>>>>>>>> 9af0d1c6-85f2-3c55-94af-6ac17cb4024c from libvirt
>>>>>>>>>>>>>>> 2020-01-14 09:20:48,536 WARN [cloud.agent.Agent]
>>>>>>>>>>>>>>> (agentRequest-Handler-4:null) (logid:ec72387b) Caught:
>>>>>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Can't find
>>>>>>>>>>>>>>> volume:d93d3c0a-3859-4473-951d-9b5c5912c767
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.getPhysicalDisk(LibvirtStoragePool.java:149)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.getVolumeStat(LibvirtGetVolumeStatsCommandWrapper.java:63)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:52)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:40)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1450)
>>>>>>>>>>>>>>> at com.cloud.agent.Agent.processRequest(Agent.java:645)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1083)
>>>>>>>>>>>>>>> at com.cloud.utils.nio.Task.call(Task.java:83)
>>>>>>>>>>>>>>> at com.cloud.utils.nio.Task.call(Task.java:29)
>>>>>>>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>>>>>>>>>>> 2020-01-14 09:25:15,259 INFO
>>>>>>>>>>>>>>> [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null)
>>>>>>>>>>>>>>> (logid:1a7e082e) Trying to fetch storage pool
>>>>>>>>>>>>>>> 9af0d1c6-85f2-3c55-94af-6ac17cb4024c from libvirt
>>>>>>>>>>>>>>> 2020-01-14 09:25:15,261 INFO
>>>>>>>>>>>>>>> [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null)
>>>>>>>>>>>>>>> (logid:1a7e082e) Asking libvirt to refresh storage pool
>>>>>>>>>>>>>>> 9af0d1c6-85f2-3c55-94af-6ac17cb4024c
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> And here the management server log:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2020-01-14 09:21:27,105 DEBUG [c.c.a.t.Request]
>>>>>>>>>>>>>>> (AgentManager-Handler-2:null) (logid:) Seq 15-705657766613619075:
>>>>>>>>>>>>>>> Processing: { Ans: , MgmtId: 220777304233416, via: 15, Ver: v1, Flags: 10,
>>>>>>>>>>>>>>> [{"com.cloud.agent.api.Answer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException:
>>>>>>>>>>>>>>> Can't find volume:d93d3c0a-3859-4473-951d-9b5c5912c767\n\tat
>>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.getPhysicalDisk(LibvirtStoragePool.java:149)\n\tat
>>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.getVolumeStat(LibvirtGetVolumeStatsCommandWrapper.java:63)\n\tat
>>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:52)\n\tat
>>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:40)\n\tat
>>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)\n\tat
>>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1450)\n\tat
>>>>>>>>>>>>>>> com.cloud.agent.Agent.processRequest(Agent.java:645)\n\tat
>>>>>>>>>>>>>>> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1083)\n\tat
>>>>>>>>>>>>>>> com.cloud.utils.nio.Task.call(Task.java:83)\n\tat
>>>>>>>>>>>>>>> com.cloud.utils.nio.Task.call(Task.java:29)\n\tat
>>>>>>>>>>>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat
>>>>>>>>>>>>>>> java.lang.Thread.run(Thread.java:748)\n","wait":0}}] }
>>>>>>>>>>>>>>> 2020-01-14 09:21:27,105 DEBUG [c.c.a.t.Request]
>>>>>>>>>>>>>>> (StatsCollector-6:ctx-fd801d0a) (logid:ec72387b) Seq 15-705657766613619075:
>>>>>>>>>>>>>>> Received: { Ans: , MgmtId: 220777304233416, via: 15(csdell017), Ver: v1,
>>>>>>>>>>>>>>> Flags: 10, { Answer } }
>>>>>>>>>>>>>>> 2020-01-14 09:21:27,105 DEBUG [c.c.a.m.AgentManagerImpl]
>>>>>>>>>>>>>>> (StatsCollector-6:ctx-fd801d0a) (logid:ec72387b) Details from executing
>>>>>>>>>>>>>>> class com.cloud.agent.api.GetVolumeStatsCommand:
>>>>>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Can't find
>>>>>>>>>>>>>>> volume:d93d3c0a-3859-4473-951d-9b5c5912c767
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.getPhysicalDisk(LibvirtStoragePool.java:149)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.getVolumeStat(LibvirtGetVolumeStatsCommandWrapper.java:63)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:52)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:40)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1450)
>>>>>>>>>>>>>>> at com.cloud.agent.Agent.processRequest(Agent.java:645)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1083)
>>>>>>>>>>>>>>> at com.cloud.utils.nio.Task.call(Task.java:83)
>>>>>>>>>>>>>>> at com.cloud.utils.nio.Task.call(Task.java:29)
>>>>>>>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> on 09/01/20 12:58, Daan Hoogland wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Charlie, I think you'll have to explain a bit more about your environment
>>>>>>>>>>>>>>> to get an answer. what type of storage is it? Where did you migrate the VM
>>>>>>>>>>>>>>> from and to? What types() of hypervisors are you using? Though saying **the**
>>>>>>>>>>>>>>> agent logs suggests KVM, you are still leaving people guessing a lot.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Daan
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Daan
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Daan
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Daan
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Daan
>>>>>>>
>>>>>>
>>>>
>>>> --
>>>> Daan
>>>>
>>>
>>
>> --
>> Daan
>>
>
--
Daan
Re: new primary storage
Posted by Charlie Holeowsky <ch...@gmail.com>.
Hi Daan and Users,
I opened the issue on Github and found that this bug (#3181) was fixed on
4.13.0 release.
Il giorno lun 3 feb 2020 alle ore 10:25 Daan Hoogland <
daan.hoogland@gmail.com> ha scritto:
> thanks Charlie, can you take your issue to a github issue on the project
> please? thanks
>
> On Mon, Feb 3, 2020 at 9:57 AM Charlie Holeowsky <
> charlie.holeowsky@gmail.com> wrote:
>
>> Hi Daan and users,
>> Yes, I resolved the issue but it's only a work-around, I have found that
>> the problem is reproducible but I do not understand if I am the only one to
>> have noticed it and if there is another more elegant and definitive
>> solution.
>>
>>
>>
>> Il giorno ven 31 gen 2020 alle ore 13:31 Daan Hoogland <
>> daan.hoogland@gmail.com> ha scritto:
>>
>>> sorry to not have any focus on this Charlie,
>>> Do I read correctly that you resolved your issue?
>>>
>>> On Tue, Jan 28, 2020 at 2:46 PM Charlie Holeowsky <
>>> charlie.holeowsky@gmail.com> wrote:
>>>
>>>> In this period I have performed some testsand I found a workarount for
>>>> the metrics problem.
>>>> I created a test environment in the laboratory with the main
>>>> characteristics equal to that of production (acs 4.11.2.0, all Ubuntu 16.04
>>>> OS, KVM, NFS as shared storage and advanced network). Then I added a ubuntu
>>>> 18.04 new primary storage.
>>>>
>>>> I create a new VM in the new storage server and and after a while the
>>>> metrics appeared as on the first storage, so the storage is working.
>>>>
>>>> I destroyed this VM, I create a new one on the first (old) storage and
>>>> then I migrate it on the new storage.
>>>>
>>>> After migrating the disk from the first primary storage to the second
>>>> one i have encountered the same problem (same error on agent.log
>>>> com.cloud.utils.exception.CloudRuntimeException: Can't find
>>>> volume:5a521eb0...) and the volume metrics and metrics didn't appear or
>>>> update.
>>>>
>>>> In this test environment I created a symbolic link in the filesystem of
>>>> the storage server taking the name of the disk just migrated (the one with
>>>> the name path) and the uuid name (the one that appear in error message).
>>>>
>>>>
>>>> Here is an example to better explain me.
>>>>
>>>> I took the list of volumes where path is different from uuid which
>>>> returns the data of the migrated volumes:
>>>>
>>>> mysql> select id,uuid,path from volumes where uuid!=path;
>>>>
>>>> +----+--------------------------------------+--------------------------------------+
>>>> | id | uuid | path
>>>> |
>>>>
>>>> +----+--------------------------------------+--------------------------------------+
>>>> | 10 | 5a521eb0-266f-4353-b4b2-1d63a483e5b5 |
>>>> 165b92ba-68f1-4172-be35-bbe1d032cb7c |
>>>> | 12 | acb3bb29-9bac-4a2a-aefa-3c6ac1c2846b |
>>>> 56aa3961-dbc2-4f98-9246-a7497eef3214 |
>>>>
>>>> +----+--------------------------------------+--------------------------------------+
>>>>
>>>> In the storage server I make the symbolic links (ln -s <path> <uuid>):
>>>>
>>>> # ln
>>>> -s 5a521eb0-266f-4353-b4b2-1d63a483e5b5 165b92ba-68f1-4172-be35-bbe1d032cb7c
>>>> # ln -s acb3bb29-9bac-4a2a-aefa-3c6ac1c2846b
>>>> 56aa3961-dbc2-4f98-9246-a7497eef3214
>>>>
>>>> After doing this, I waited some time and then I found that metrics were
>>>> updated and the message in agent.log no longer appeared.
>>>>
>>>>
>>>> Il giorno gio 23 gen 2020 alle ore 17:53 Charlie Holeowsky <
>>>> charlie.holeowsky@gmail.com> ha scritto:
>>>>
>>>>> I still don't understand
>>>>> why com.cloud.hypervisor.kvm.storage.LibvirtStoragePool don't find the
>>>>> volume d93d3c0a-3859-4473-951d-9b5c5912c76 that exists as file
>>>>> 39148fe1-842b-433a-8a7f-85e90f316e04...
>>>>>
>>>>> It's the only anomaly I have found. Where can I look again?
>>>>>
>>>>> Il giorno lun 20 gen 2020 alle ore 16:27 Daan Hoogland <
>>>>> daan.hoogland@gmail.com> ha scritto:
>>>>>
>>>>>> but the record you send earlier also says that is should be looking
>>>>>> for 39148fe1-842b-433a-8a7f-85e90f316e04, in the path field. the message
>>>>>> might be just that, a message.
>>>>>>
>>>>>> On Mon, Jan 20, 2020 at 3:35 PM Charlie Holeowsky <
>>>>>> charlie.holeowsky@gmail.com> wrote:
>>>>>>
>>>>>>> I think that's the problem because in the logs that I have forwarded
>>>>>>> it reads:
>>>>>>> Can't find volume: d93d3c0a-3859-4473-951d-9b5c5912c767
>>>>>>>
>>>>>>> This is the volume ID of migrated file but it do not exist on
>>>>>>> primary storage (new or old one) but it exist
>>>>>>> as 39148fe1-842b-433a-8a7f-85e90f316e04.
>>>>>>>
>>>>>>> Il giorno lun 20 gen 2020 alle ore 12:35 Daan Hoogland <
>>>>>>> daan.hoogland@gmail.com> ha scritto:
>>>>>>>
>>>>>>>> also, can you see the primary storage being mounted?
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Jan 20, 2020 at 12:33 PM Daan Hoogland <
>>>>>>>> daan.hoogland@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Why do you think that Charlie? Is it in the logs like that
>>>>>>>>> somewhere?
>>>>>>>>>
>>>>>>>>> On Mon, Jan 20, 2020 at 9:52 AM Charlie Holeowsky <
>>>>>>>>> charlie.holeowsky@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Daan,
>>>>>>>>>> in fact I find the volume file
>>>>>>>>>> (39148fe1-842b-433a-8a7f-85e90f316e04) in the repositry id = 3 (the new
>>>>>>>>>> one) but it seems to me that the cloudstack system goes looking for the
>>>>>>>>>> volume with its "old" name (path) that doesn't exist...
>>>>>>>>>>
>>>>>>>>>> Il giorno sab 18 gen 2020 alle ore 21:41 Daan Hoogland <
>>>>>>>>>> daan.hoogland@gmail.com> ha scritto:
>>>>>>>>>>
>>>>>>>>>>> Charlie,
>>>>>>>>>>> forgive my not replying in a timely manner. This might happen if
>>>>>>>>>>> the disk was migrated. In this case probably from primary storage with id 1
>>>>>>>>>>> to the one with id 3. the second record (pool_id 1) is removed, so you can
>>>>>>>>>>> ignore that one. The first seems legit. You should be able to find that
>>>>>>>>>>> disks on your primary storage with id 3.
>>>>>>>>>>> hope this helps.
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Jan 16, 2020 at 2:07 PM Charlie Holeowsky <
>>>>>>>>>>> charlie.holeowsky@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Daan and users,
>>>>>>>>>>>> to better explain I show you the two records related to the
>>>>>>>>>>>> disk that generates the error message.
>>>>>>>>>>>>
>>>>>>>>>>>> In the first query there are the data of the disk currently in
>>>>>>>>>>>> use which has the "uuid" equal to the name searched by the procedure
>>>>>>>>>>>> com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.getPhysicalDiskand a
>>>>>>>>>>>> has a different "path" field.
>>>>>>>>>>>>
>>>>>>>>>>>> In the second one we note that the "path" field is equal to the
>>>>>>>>>>>> "uuid" of the volume in use but has null "uuid" (and state=Expunged).
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> mysql> select * from volumes where
>>>>>>>>>>>> uuid='d93d3c0a-3859-4473-951d-9b5c5912c767';
>>>>>>>>>>>>
>>>>>>>>>>>> +-----+------------+-----------+---------+--------------+-------------+-----------+----------+--------------------------------------+-------------+---------------------+--------------------------------------+--------+----------------+------------+---------+-------------+-----------+------------------+-------------+----------------------------+-------------+---------------------+----------+---------------------+---------+-------+------------+--------------+-----------+------------------------+--------+----------------+--------+----------+----------+---------------+-------------------+
>>>>>>>>>>>> | id | account_id | domain_id | pool_id | last_pool_id |
>>>>>>>>>>>> instance_id | device_id | name | uuid |
>>>>>>>>>>>> size | folder | path |
>>>>>>>>>>>> pod_id | data_center_id | iscsi_name | host_ip | volume_type | pool_type |
>>>>>>>>>>>> disk_offering_id | template_id | first_snapshot_backup_uuid | recreatable |
>>>>>>>>>>>> created | attached | updated | removed | state |
>>>>>>>>>>>> chain_info | update_count | disk_type | vm_snapshot_chain_size | iso_id |
>>>>>>>>>>>> display_volume | format | min_iops | max_iops | hv_ss_reserve |
>>>>>>>>>>>> provisioning_type |
>>>>>>>>>>>>
>>>>>>>>>>>> +-----+------------+-----------+---------+--------------+-------------+-----------+----------+--------------------------------------+-------------+---------------------+--------------------------------------+--------+----------------+------------+---------+-------------+-----------+------------------+-------------+----------------------------+-------------+---------------------+----------+---------------------+---------+-------+------------+--------------+-----------+------------------------+--------+----------------+--------+----------+----------+---------------+-------------------+
>>>>>>>>>>>> | 213 | 2 | 1 | 3 | 1 |
>>>>>>>>>>>> 148 | 1 | DATA-148 | d93d3c0a-3859-4473-951d-9b5c5912c767 |
>>>>>>>>>>>> 53687091200 | /srv/primary | 39148fe1-842b-433a-8a7f-85e90f316e04 | NULL
>>>>>>>>>>>> | 1 | NULL | NULL | DATADISK | NULL |
>>>>>>>>>>>> 34 | NULL | NULL | 0 |
>>>>>>>>>>>> 2019-11-26 10:41:46 | NULL | 2019-11-26 10:41:50 | NULL | Ready |
>>>>>>>>>>>> NULL | 2 | NULL | NULL | NULL |
>>>>>>>>>>>> 1 | QCOW2 | NULL | NULL | NULL | thin
>>>>>>>>>>>> |
>>>>>>>>>>>>
>>>>>>>>>>>> +-----+------------+-----------+---------+--------------+-------------+-----------+----------+--------------------------------------+-------------+---------------------+--------------------------------------+--------+----------------+------------+---------+-------------+-----------+------------------+-------------+----------------------------+-------------+---------------------+----------+---------------------+---------+-------+------------+--------------+-----------+------------------------+--------+----------------+--------+----------+----------+---------------+-------------------+
>>>>>>>>>>>> 1 row in set (0.00 sec)
>>>>>>>>>>>>
>>>>>>>>>>>> mysql> select * from volumes where
>>>>>>>>>>>> path='d93d3c0a-3859-4473-951d-9b5c5912c767';
>>>>>>>>>>>>
>>>>>>>>>>>> +-----+------------+-----------+---------+--------------+-------------+-----------+----------+------+-------------+--------+--------------------------------------+--------+----------------+------------+---------+-------------+-----------+------------------+-------------+----------------------------+-------------+---------------------+----------+---------------------+---------------------+----------+------------+--------------+-----------+------------------------+--------+----------------+--------+----------+----------+---------------+-------------------+
>>>>>>>>>>>> | id | account_id | domain_id | pool_id | last_pool_id |
>>>>>>>>>>>> instance_id | device_id | name | uuid | size | folder | path
>>>>>>>>>>>> | pod_id | data_center_id | iscsi_name |
>>>>>>>>>>>> host_ip | volume_type | pool_type | disk_offering_id | template_id |
>>>>>>>>>>>> first_snapshot_backup_uuid | recreatable | created | attached |
>>>>>>>>>>>> updated | removed | state | chain_info |
>>>>>>>>>>>> update_count | disk_type | vm_snapshot_chain_size | iso_id | display_volume
>>>>>>>>>>>> | format | min_iops | max_iops | hv_ss_reserve | provisioning_type |
>>>>>>>>>>>>
>>>>>>>>>>>> +-----+------------+-----------+---------+--------------+-------------+-----------+----------+------+-------------+--------+--------------------------------------+--------+----------------+------------+---------+-------------+-----------+------------------+-------------+----------------------------+-------------+---------------------+----------+---------------------+---------------------+----------+------------+--------------+-----------+------------------------+--------+----------------+--------+----------+----------+---------------+-------------------+
>>>>>>>>>>>> | 212 | 2 | 1 | 1 | NULL |
>>>>>>>>>>>> 148 | 1 | DATA-148 | NULL | 53687091200 | NULL |
>>>>>>>>>>>> d93d3c0a-3859-4473-951d-9b5c5912c767 | NULL | 1 | NULL
>>>>>>>>>>>> | NULL | DATADISK | NULL | 34 | NULL | NULL
>>>>>>>>>>>> | 0 | 2019-11-26 10:38:23 | NULL |
>>>>>>>>>>>> 2019-11-26 10:41:50 | 2019-11-26 10:41:50 | Expunged | NULL |
>>>>>>>>>>>> 8 | NULL | NULL | NULL | 1 | QCOW2
>>>>>>>>>>>> | NULL | NULL | NULL | thin |
>>>>>>>>>>>>
>>>>>>>>>>>> +-----+------------+-----------+---------+--------------+-------------+-----------+----------+------+-------------+--------+--------------------------------------+--------+----------------+------------+---------+-------------+-----------+------------------+-------------+----------------------------+-------------+---------------------+----------+---------------------+---------------------+----------+------------+--------------+-----------+------------------------+--------+----------------+--------+----------+----------+---------------+-------------------+
>>>>>>>>>>>> 1 row in set (0.00 sec)
>>>>>>>>>>>>
>>>>>>>>>>>> Il giorno mar 14 gen 2020 alle ore 15:05 Daan Hoogland <
>>>>>>>>>>>> daan.hoogland@gmail.com> ha scritto:
>>>>>>>>>>>>
>>>>>>>>>>>>> So Charlie,
>>>>>>>>>>>>> d93d3c0a-3859-4473-951d-9b5c5912c767 is actually a valid disk?
>>>>>>>>>>>>> does it exist on the backend nfs?
>>>>>>>>>>>>> and the pool 9af0d1c6-85f2-3c55-94af-6ac17cb4024c does it
>>>>>>>>>>>>> exist both in cloudstack and on the backend?
>>>>>>>>>>>>>
>>>>>>>>>>>>> if both are answered with yes, you probably have a permissions
>>>>>>>>>>>>> issue, which might be in the network.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Jan 14, 2020 at 10:21 AM Charlie Holeowsky <
>>>>>>>>>>>>> charlie.holeowsky@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Daan and users,
>>>>>>>>>>>>>> the infrastructure is based on the Linux environment. The
>>>>>>>>>>>>>> management server, hosts and storage are all Ubuntu 16.04 except the new
>>>>>>>>>>>>>> storage server which is an Ubuntu 18.04. The hypervisor used is Qemu-kvm
>>>>>>>>>>>>>> with NFS to share the storage.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We tried to add another primary storage and creating a VM
>>>>>>>>>>>>>> that would use it we found no problems, the statistics update and no error
>>>>>>>>>>>>>> messages appear.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Gere is an excerpt of the logs of the most complete agent:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2020-01-14 09:01:45,749 INFO
>>>>>>>>>>>>>> [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null)
>>>>>>>>>>>>>> (logid:c3851d3a) Trying to fetch storage pool
>>>>>>>>>>>>>> 171e90f4-511e-3b10-9310-b9eec0094be6 from libvirt
>>>>>>>>>>>>>> 2020-01-14 09:01:45,752 INFO
>>>>>>>>>>>>>> [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null)
>>>>>>>>>>>>>> (logid:c3851d3a) Asking libvirt to refresh storage pool
>>>>>>>>>>>>>> 171e90f4-511e-3b10-9310-b9eec0094be6
>>>>>>>>>>>>>> 2020-01-14 09:01:46,641 INFO
>>>>>>>>>>>>>> [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null)
>>>>>>>>>>>>>> (logid:c3851d3a) Trying to fetch storage pool
>>>>>>>>>>>>>> 9af0d1c6-85f2-3c55-94af-6ac17cb4024c from libvirt
>>>>>>>>>>>>>> 2020-01-14 09:01:46,643 INFO
>>>>>>>>>>>>>> [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null)
>>>>>>>>>>>>>> (logid:c3851d3a) Asking libvirt to refresh storage pool
>>>>>>>>>>>>>> 9af0d1c6-85f2-3c55-94af-6ac17cb4024c
>>>>>>>>>>>>>> 2020-01-14 09:05:51,529 INFO
>>>>>>>>>>>>>> [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null)
>>>>>>>>>>>>>> (logid:2765ff88) Trying to fetch storage pool
>>>>>>>>>>>>>> 9af0d1c6-85f2-3c55-94af-6ac17cb4024c from libvirt
>>>>>>>>>>>>>> 2020-01-14 09:05:51,532 INFO
>>>>>>>>>>>>>> [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null)
>>>>>>>>>>>>>> (logid:2765ff88) Asking libvirt to refresh storage pool
>>>>>>>>>>>>>> 9af0d1c6-85f2-3c55-94af-6ac17cb4024c
>>>>>>>>>>>>>> 2020-01-14 09:10:47,286 INFO
>>>>>>>>>>>>>> [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null)
>>>>>>>>>>>>>> (logid:6d27b740) Trying to fetch storage pool
>>>>>>>>>>>>>> 9af0d1c6-85f2-3c55-94af-6ac17cb4024c from libvirt
>>>>>>>>>>>>>> 2020-01-14 09:10:47,419 WARN [cloud.agent.Agent]
>>>>>>>>>>>>>> (agentRequest-Handler-3:null) (logid:6d27b740) Caught:
>>>>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Can't find
>>>>>>>>>>>>>> volume:d93d3c0a-3859-4473-951d-9b5c5912c767
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.getPhysicalDisk(LibvirtStoragePool.java:149)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.getVolumeStat(LibvirtGetVolumeStatsCommandWrapper.java:63)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:52)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:40)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1450)
>>>>>>>>>>>>>> at com.cloud.agent.Agent.processRequest(Agent.java:645)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1083)
>>>>>>>>>>>>>> at com.cloud.utils.nio.Task.call(Task.java:83)
>>>>>>>>>>>>>> at com.cloud.utils.nio.Task.call(Task.java:29)
>>>>>>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>>>>>>>>>> 2020-01-14 09:20:48,390 INFO
>>>>>>>>>>>>>> [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null)
>>>>>>>>>>>>>> (logid:ec72387b) Trying to fetch storage pool
>>>>>>>>>>>>>> 9af0d1c6-85f2-3c55-94af-6ac17cb4024c from libvirt
>>>>>>>>>>>>>> 2020-01-14 09:20:48,536 WARN [cloud.agent.Agent]
>>>>>>>>>>>>>> (agentRequest-Handler-4:null) (logid:ec72387b) Caught:
>>>>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Can't find
>>>>>>>>>>>>>> volume:d93d3c0a-3859-4473-951d-9b5c5912c767
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.getPhysicalDisk(LibvirtStoragePool.java:149)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.getVolumeStat(LibvirtGetVolumeStatsCommandWrapper.java:63)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:52)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:40)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1450)
>>>>>>>>>>>>>> at com.cloud.agent.Agent.processRequest(Agent.java:645)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1083)
>>>>>>>>>>>>>> at com.cloud.utils.nio.Task.call(Task.java:83)
>>>>>>>>>>>>>> at com.cloud.utils.nio.Task.call(Task.java:29)
>>>>>>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>>>>>>>>>> 2020-01-14 09:25:15,259 INFO
>>>>>>>>>>>>>> [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null)
>>>>>>>>>>>>>> (logid:1a7e082e) Trying to fetch storage pool
>>>>>>>>>>>>>> 9af0d1c6-85f2-3c55-94af-6ac17cb4024c from libvirt
>>>>>>>>>>>>>> 2020-01-14 09:25:15,261 INFO
>>>>>>>>>>>>>> [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null)
>>>>>>>>>>>>>> (logid:1a7e082e) Asking libvirt to refresh storage pool
>>>>>>>>>>>>>> 9af0d1c6-85f2-3c55-94af-6ac17cb4024c
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> And here the management server log:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2020-01-14 09:21:27,105 DEBUG [c.c.a.t.Request]
>>>>>>>>>>>>>> (AgentManager-Handler-2:null) (logid:) Seq 15-705657766613619075:
>>>>>>>>>>>>>> Processing: { Ans: , MgmtId: 220777304233416, via: 15, Ver: v1, Flags: 10,
>>>>>>>>>>>>>> [{"com.cloud.agent.api.Answer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException:
>>>>>>>>>>>>>> Can't find volume:d93d3c0a-3859-4473-951d-9b5c5912c767\n\tat
>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.getPhysicalDisk(LibvirtStoragePool.java:149)\n\tat
>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.getVolumeStat(LibvirtGetVolumeStatsCommandWrapper.java:63)\n\tat
>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:52)\n\tat
>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:40)\n\tat
>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)\n\tat
>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1450)\n\tat
>>>>>>>>>>>>>> com.cloud.agent.Agent.processRequest(Agent.java:645)\n\tat
>>>>>>>>>>>>>> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1083)\n\tat
>>>>>>>>>>>>>> com.cloud.utils.nio.Task.call(Task.java:83)\n\tat
>>>>>>>>>>>>>> com.cloud.utils.nio.Task.call(Task.java:29)\n\tat
>>>>>>>>>>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat
>>>>>>>>>>>>>> java.lang.Thread.run(Thread.java:748)\n","wait":0}}] }
>>>>>>>>>>>>>> 2020-01-14 09:21:27,105 DEBUG [c.c.a.t.Request]
>>>>>>>>>>>>>> (StatsCollector-6:ctx-fd801d0a) (logid:ec72387b) Seq 15-705657766613619075:
>>>>>>>>>>>>>> Received: { Ans: , MgmtId: 220777304233416, via: 15(csdell017), Ver: v1,
>>>>>>>>>>>>>> Flags: 10, { Answer } }
>>>>>>>>>>>>>> 2020-01-14 09:21:27,105 DEBUG [c.c.a.m.AgentManagerImpl]
>>>>>>>>>>>>>> (StatsCollector-6:ctx-fd801d0a) (logid:ec72387b) Details from executing
>>>>>>>>>>>>>> class com.cloud.agent.api.GetVolumeStatsCommand:
>>>>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Can't find
>>>>>>>>>>>>>> volume:d93d3c0a-3859-4473-951d-9b5c5912c767
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.getPhysicalDisk(LibvirtStoragePool.java:149)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.getVolumeStat(LibvirtGetVolumeStatsCommandWrapper.java:63)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:52)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:40)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1450)
>>>>>>>>>>>>>> at com.cloud.agent.Agent.processRequest(Agent.java:645)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1083)
>>>>>>>>>>>>>> at com.cloud.utils.nio.Task.call(Task.java:83)
>>>>>>>>>>>>>> at com.cloud.utils.nio.Task.call(Task.java:29)
>>>>>>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> on 09/01/20 12:58, Daan Hoogland wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Charlie, I think you'll have to explain a bit more about your environment
>>>>>>>>>>>>>> to get an answer. what type of storage is it? Where did you migrate the VM
>>>>>>>>>>>>>> from and to? What types() of hypervisors are you using? Though saying **the**
>>>>>>>>>>>>>> agent logs suggests KVM, you are still leaving people guessing a lot.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Daan
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Daan
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Daan
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Daan
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Daan
>>>>>>
>>>>>
>>>
>>> --
>>> Daan
>>>
>>
>
> --
> Daan
>
Re: new primary storage
Posted by Daan Hoogland <da...@gmail.com>.
thanks Charlie, can you take your issue to a github issue on the project
please? thanks
On Mon, Feb 3, 2020 at 9:57 AM Charlie Holeowsky <
charlie.holeowsky@gmail.com> wrote:
> Hi Daan and users,
> Yes, I resolved the issue but it's only a work-around, I have found that
> the problem is reproducible but I do not understand if I am the only one to
> have noticed it and if there is another more elegant and definitive
> solution.
>
>
>
> Il giorno ven 31 gen 2020 alle ore 13:31 Daan Hoogland <
> daan.hoogland@gmail.com> ha scritto:
>
>> sorry to not have any focus on this Charlie,
>> Do I read correctly that you resolved your issue?
>>
>> On Tue, Jan 28, 2020 at 2:46 PM Charlie Holeowsky <
>> charlie.holeowsky@gmail.com> wrote:
>>
>>> In this period I have performed some testsand I found a workarount for
>>> the metrics problem.
>>> I created a test environment in the laboratory with the main
>>> characteristics equal to that of production (acs 4.11.2.0, all Ubuntu 16.04
>>> OS, KVM, NFS as shared storage and advanced network). Then I added a ubuntu
>>> 18.04 new primary storage.
>>>
>>> I create a new VM in the new storage server and and after a while the
>>> metrics appeared as on the first storage, so the storage is working.
>>>
>>> I destroyed this VM, I create a new one on the first (old) storage and
>>> then I migrate it on the new storage.
>>>
>>> After migrating the disk from the first primary storage to the second
>>> one i have encountered the same problem (same error on agent.log
>>> com.cloud.utils.exception.CloudRuntimeException: Can't find
>>> volume:5a521eb0...) and the volume metrics and metrics didn't appear or
>>> update.
>>>
>>> In this test environment I created a symbolic link in the filesystem of
>>> the storage server taking the name of the disk just migrated (the one with
>>> the name path) and the uuid name (the one that appear in error message).
>>>
>>>
>>> Here is an example to better explain me.
>>>
>>> I took the list of volumes where path is different from uuid which
>>> returns the data of the migrated volumes:
>>>
>>> mysql> select id,uuid,path from volumes where uuid!=path;
>>>
>>> +----+--------------------------------------+--------------------------------------+
>>> | id | uuid | path
>>> |
>>>
>>> +----+--------------------------------------+--------------------------------------+
>>> | 10 | 5a521eb0-266f-4353-b4b2-1d63a483e5b5 |
>>> 165b92ba-68f1-4172-be35-bbe1d032cb7c |
>>> | 12 | acb3bb29-9bac-4a2a-aefa-3c6ac1c2846b |
>>> 56aa3961-dbc2-4f98-9246-a7497eef3214 |
>>>
>>> +----+--------------------------------------+--------------------------------------+
>>>
>>> In the storage server I make the symbolic links (ln -s <path> <uuid>):
>>>
>>> # ln
>>> -s 5a521eb0-266f-4353-b4b2-1d63a483e5b5 165b92ba-68f1-4172-be35-bbe1d032cb7c
>>> # ln -s acb3bb29-9bac-4a2a-aefa-3c6ac1c2846b
>>> 56aa3961-dbc2-4f98-9246-a7497eef3214
>>>
>>> After doing this, I waited some time and then I found that metrics were
>>> updated and the message in agent.log no longer appeared.
>>>
>>>
>>> Il giorno gio 23 gen 2020 alle ore 17:53 Charlie Holeowsky <
>>> charlie.holeowsky@gmail.com> ha scritto:
>>>
>>>> I still don't understand
>>>> why com.cloud.hypervisor.kvm.storage.LibvirtStoragePool don't find the
>>>> volume d93d3c0a-3859-4473-951d-9b5c5912c76 that exists as file
>>>> 39148fe1-842b-433a-8a7f-85e90f316e04...
>>>>
>>>> It's the only anomaly I have found. Where can I look again?
>>>>
>>>> Il giorno lun 20 gen 2020 alle ore 16:27 Daan Hoogland <
>>>> daan.hoogland@gmail.com> ha scritto:
>>>>
>>>>> but the record you send earlier also says that is should be looking
>>>>> for 39148fe1-842b-433a-8a7f-85e90f316e04, in the path field. the message
>>>>> might be just that, a message.
>>>>>
>>>>> On Mon, Jan 20, 2020 at 3:35 PM Charlie Holeowsky <
>>>>> charlie.holeowsky@gmail.com> wrote:
>>>>>
>>>>>> I think that's the problem because in the logs that I have forwarded
>>>>>> it reads:
>>>>>> Can't find volume: d93d3c0a-3859-4473-951d-9b5c5912c767
>>>>>>
>>>>>> This is the volume ID of migrated file but it do not exist on primary
>>>>>> storage (new or old one) but it exist
>>>>>> as 39148fe1-842b-433a-8a7f-85e90f316e04.
>>>>>>
>>>>>> Il giorno lun 20 gen 2020 alle ore 12:35 Daan Hoogland <
>>>>>> daan.hoogland@gmail.com> ha scritto:
>>>>>>
>>>>>>> also, can you see the primary storage being mounted?
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Jan 20, 2020 at 12:33 PM Daan Hoogland <
>>>>>>> daan.hoogland@gmail.com> wrote:
>>>>>>>
>>>>>>>> Why do you think that Charlie? Is it in the logs like that
>>>>>>>> somewhere?
>>>>>>>>
>>>>>>>> On Mon, Jan 20, 2020 at 9:52 AM Charlie Holeowsky <
>>>>>>>> charlie.holeowsky@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Daan,
>>>>>>>>> in fact I find the volume file
>>>>>>>>> (39148fe1-842b-433a-8a7f-85e90f316e04) in the repositry id = 3 (the new
>>>>>>>>> one) but it seems to me that the cloudstack system goes looking for the
>>>>>>>>> volume with its "old" name (path) that doesn't exist...
>>>>>>>>>
>>>>>>>>> Il giorno sab 18 gen 2020 alle ore 21:41 Daan Hoogland <
>>>>>>>>> daan.hoogland@gmail.com> ha scritto:
>>>>>>>>>
>>>>>>>>>> Charlie,
>>>>>>>>>> forgive my not replying in a timely manner. This might happen if
>>>>>>>>>> the disk was migrated. In this case probably from primary storage with id 1
>>>>>>>>>> to the one with id 3. the second record (pool_id 1) is removed, so you can
>>>>>>>>>> ignore that one. The first seems legit. You should be able to find that
>>>>>>>>>> disks on your primary storage with id 3.
>>>>>>>>>> hope this helps.
>>>>>>>>>>
>>>>>>>>>> On Thu, Jan 16, 2020 at 2:07 PM Charlie Holeowsky <
>>>>>>>>>> charlie.holeowsky@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Daan and users,
>>>>>>>>>>> to better explain I show you the two records related to the disk
>>>>>>>>>>> that generates the error message.
>>>>>>>>>>>
>>>>>>>>>>> In the first query there are the data of the disk currently in
>>>>>>>>>>> use which has the "uuid" equal to the name searched by the procedure
>>>>>>>>>>> com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.getPhysicalDiskand a
>>>>>>>>>>> has a different "path" field.
>>>>>>>>>>>
>>>>>>>>>>> In the second one we note that the "path" field is equal to the
>>>>>>>>>>> "uuid" of the volume in use but has null "uuid" (and state=Expunged).
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> mysql> select * from volumes where
>>>>>>>>>>> uuid='d93d3c0a-3859-4473-951d-9b5c5912c767';
>>>>>>>>>>>
>>>>>>>>>>> +-----+------------+-----------+---------+--------------+-------------+-----------+----------+--------------------------------------+-------------+---------------------+--------------------------------------+--------+----------------+------------+---------+-------------+-----------+------------------+-------------+----------------------------+-------------+---------------------+----------+---------------------+---------+-------+------------+--------------+-----------+------------------------+--------+----------------+--------+----------+----------+---------------+-------------------+
>>>>>>>>>>> | id | account_id | domain_id | pool_id | last_pool_id |
>>>>>>>>>>> instance_id | device_id | name | uuid |
>>>>>>>>>>> size | folder | path |
>>>>>>>>>>> pod_id | data_center_id | iscsi_name | host_ip | volume_type | pool_type |
>>>>>>>>>>> disk_offering_id | template_id | first_snapshot_backup_uuid | recreatable |
>>>>>>>>>>> created | attached | updated | removed | state |
>>>>>>>>>>> chain_info | update_count | disk_type | vm_snapshot_chain_size | iso_id |
>>>>>>>>>>> display_volume | format | min_iops | max_iops | hv_ss_reserve |
>>>>>>>>>>> provisioning_type |
>>>>>>>>>>>
>>>>>>>>>>> +-----+------------+-----------+---------+--------------+-------------+-----------+----------+--------------------------------------+-------------+---------------------+--------------------------------------+--------+----------------+------------+---------+-------------+-----------+------------------+-------------+----------------------------+-------------+---------------------+----------+---------------------+---------+-------+------------+--------------+-----------+------------------------+--------+----------------+--------+----------+----------+---------------+-------------------+
>>>>>>>>>>> | 213 | 2 | 1 | 3 | 1 |
>>>>>>>>>>> 148 | 1 | DATA-148 | d93d3c0a-3859-4473-951d-9b5c5912c767 |
>>>>>>>>>>> 53687091200 | /srv/primary | 39148fe1-842b-433a-8a7f-85e90f316e04 | NULL
>>>>>>>>>>> | 1 | NULL | NULL | DATADISK | NULL |
>>>>>>>>>>> 34 | NULL | NULL | 0 |
>>>>>>>>>>> 2019-11-26 10:41:46 | NULL | 2019-11-26 10:41:50 | NULL | Ready |
>>>>>>>>>>> NULL | 2 | NULL | NULL | NULL |
>>>>>>>>>>> 1 | QCOW2 | NULL | NULL | NULL | thin
>>>>>>>>>>> |
>>>>>>>>>>>
>>>>>>>>>>> +-----+------------+-----------+---------+--------------+-------------+-----------+----------+--------------------------------------+-------------+---------------------+--------------------------------------+--------+----------------+------------+---------+-------------+-----------+------------------+-------------+----------------------------+-------------+---------------------+----------+---------------------+---------+-------+------------+--------------+-----------+------------------------+--------+----------------+--------+----------+----------+---------------+-------------------+
>>>>>>>>>>> 1 row in set (0.00 sec)
>>>>>>>>>>>
>>>>>>>>>>> mysql> select * from volumes where
>>>>>>>>>>> path='d93d3c0a-3859-4473-951d-9b5c5912c767';
>>>>>>>>>>>
>>>>>>>>>>> +-----+------------+-----------+---------+--------------+-------------+-----------+----------+------+-------------+--------+--------------------------------------+--------+----------------+------------+---------+-------------+-----------+------------------+-------------+----------------------------+-------------+---------------------+----------+---------------------+---------------------+----------+------------+--------------+-----------+------------------------+--------+----------------+--------+----------+----------+---------------+-------------------+
>>>>>>>>>>> | id | account_id | domain_id | pool_id | last_pool_id |
>>>>>>>>>>> instance_id | device_id | name | uuid | size | folder | path
>>>>>>>>>>> | pod_id | data_center_id | iscsi_name |
>>>>>>>>>>> host_ip | volume_type | pool_type | disk_offering_id | template_id |
>>>>>>>>>>> first_snapshot_backup_uuid | recreatable | created | attached |
>>>>>>>>>>> updated | removed | state | chain_info |
>>>>>>>>>>> update_count | disk_type | vm_snapshot_chain_size | iso_id | display_volume
>>>>>>>>>>> | format | min_iops | max_iops | hv_ss_reserve | provisioning_type |
>>>>>>>>>>>
>>>>>>>>>>> +-----+------------+-----------+---------+--------------+-------------+-----------+----------+------+-------------+--------+--------------------------------------+--------+----------------+------------+---------+-------------+-----------+------------------+-------------+----------------------------+-------------+---------------------+----------+---------------------+---------------------+----------+------------+--------------+-----------+------------------------+--------+----------------+--------+----------+----------+---------------+-------------------+
>>>>>>>>>>> | 212 | 2 | 1 | 1 | NULL |
>>>>>>>>>>> 148 | 1 | DATA-148 | NULL | 53687091200 | NULL |
>>>>>>>>>>> d93d3c0a-3859-4473-951d-9b5c5912c767 | NULL | 1 | NULL
>>>>>>>>>>> | NULL | DATADISK | NULL | 34 | NULL | NULL
>>>>>>>>>>> | 0 | 2019-11-26 10:38:23 | NULL |
>>>>>>>>>>> 2019-11-26 10:41:50 | 2019-11-26 10:41:50 | Expunged | NULL |
>>>>>>>>>>> 8 | NULL | NULL | NULL | 1 | QCOW2
>>>>>>>>>>> | NULL | NULL | NULL | thin |
>>>>>>>>>>>
>>>>>>>>>>> +-----+------------+-----------+---------+--------------+-------------+-----------+----------+------+-------------+--------+--------------------------------------+--------+----------------+------------+---------+-------------+-----------+------------------+-------------+----------------------------+-------------+---------------------+----------+---------------------+---------------------+----------+------------+--------------+-----------+------------------------+--------+----------------+--------+----------+----------+---------------+-------------------+
>>>>>>>>>>> 1 row in set (0.00 sec)
>>>>>>>>>>>
>>>>>>>>>>> Il giorno mar 14 gen 2020 alle ore 15:05 Daan Hoogland <
>>>>>>>>>>> daan.hoogland@gmail.com> ha scritto:
>>>>>>>>>>>
>>>>>>>>>>>> So Charlie,
>>>>>>>>>>>> d93d3c0a-3859-4473-951d-9b5c5912c767 is actually a valid disk?
>>>>>>>>>>>> does it exist on the backend nfs?
>>>>>>>>>>>> and the pool 9af0d1c6-85f2-3c55-94af-6ac17cb4024c does it exist
>>>>>>>>>>>> both in cloudstack and on the backend?
>>>>>>>>>>>>
>>>>>>>>>>>> if both are answered with yes, you probably have a permissions
>>>>>>>>>>>> issue, which might be in the network.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Jan 14, 2020 at 10:21 AM Charlie Holeowsky <
>>>>>>>>>>>> charlie.holeowsky@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Daan and users,
>>>>>>>>>>>>> the infrastructure is based on the Linux environment. The
>>>>>>>>>>>>> management server, hosts and storage are all Ubuntu 16.04 except the new
>>>>>>>>>>>>> storage server which is an Ubuntu 18.04. The hypervisor used is Qemu-kvm
>>>>>>>>>>>>> with NFS to share the storage.
>>>>>>>>>>>>>
>>>>>>>>>>>>> We tried to add another primary storage and creating a VM that
>>>>>>>>>>>>> would use it we found no problems, the statistics update and no error
>>>>>>>>>>>>> messages appear.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Gere is an excerpt of the logs of the most complete agent:
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2020-01-14 09:01:45,749 INFO
>>>>>>>>>>>>> [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null)
>>>>>>>>>>>>> (logid:c3851d3a) Trying to fetch storage pool
>>>>>>>>>>>>> 171e90f4-511e-3b10-9310-b9eec0094be6 from libvirt
>>>>>>>>>>>>> 2020-01-14 09:01:45,752 INFO
>>>>>>>>>>>>> [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null)
>>>>>>>>>>>>> (logid:c3851d3a) Asking libvirt to refresh storage pool
>>>>>>>>>>>>> 171e90f4-511e-3b10-9310-b9eec0094be6
>>>>>>>>>>>>> 2020-01-14 09:01:46,641 INFO
>>>>>>>>>>>>> [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null)
>>>>>>>>>>>>> (logid:c3851d3a) Trying to fetch storage pool
>>>>>>>>>>>>> 9af0d1c6-85f2-3c55-94af-6ac17cb4024c from libvirt
>>>>>>>>>>>>> 2020-01-14 09:01:46,643 INFO
>>>>>>>>>>>>> [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null)
>>>>>>>>>>>>> (logid:c3851d3a) Asking libvirt to refresh storage pool
>>>>>>>>>>>>> 9af0d1c6-85f2-3c55-94af-6ac17cb4024c
>>>>>>>>>>>>> 2020-01-14 09:05:51,529 INFO
>>>>>>>>>>>>> [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null)
>>>>>>>>>>>>> (logid:2765ff88) Trying to fetch storage pool
>>>>>>>>>>>>> 9af0d1c6-85f2-3c55-94af-6ac17cb4024c from libvirt
>>>>>>>>>>>>> 2020-01-14 09:05:51,532 INFO
>>>>>>>>>>>>> [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null)
>>>>>>>>>>>>> (logid:2765ff88) Asking libvirt to refresh storage pool
>>>>>>>>>>>>> 9af0d1c6-85f2-3c55-94af-6ac17cb4024c
>>>>>>>>>>>>> 2020-01-14 09:10:47,286 INFO
>>>>>>>>>>>>> [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null)
>>>>>>>>>>>>> (logid:6d27b740) Trying to fetch storage pool
>>>>>>>>>>>>> 9af0d1c6-85f2-3c55-94af-6ac17cb4024c from libvirt
>>>>>>>>>>>>> 2020-01-14 09:10:47,419 WARN [cloud.agent.Agent]
>>>>>>>>>>>>> (agentRequest-Handler-3:null) (logid:6d27b740) Caught:
>>>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Can't find
>>>>>>>>>>>>> volume:d93d3c0a-3859-4473-951d-9b5c5912c767
>>>>>>>>>>>>> at
>>>>>>>>>>>>> com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.getPhysicalDisk(LibvirtStoragePool.java:149)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.getVolumeStat(LibvirtGetVolumeStatsCommandWrapper.java:63)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:52)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:40)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1450)
>>>>>>>>>>>>> at com.cloud.agent.Agent.processRequest(Agent.java:645)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1083)
>>>>>>>>>>>>> at com.cloud.utils.nio.Task.call(Task.java:83)
>>>>>>>>>>>>> at com.cloud.utils.nio.Task.call(Task.java:29)
>>>>>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>>>>>>>>> 2020-01-14 09:20:48,390 INFO
>>>>>>>>>>>>> [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null)
>>>>>>>>>>>>> (logid:ec72387b) Trying to fetch storage pool
>>>>>>>>>>>>> 9af0d1c6-85f2-3c55-94af-6ac17cb4024c from libvirt
>>>>>>>>>>>>> 2020-01-14 09:20:48,536 WARN [cloud.agent.Agent]
>>>>>>>>>>>>> (agentRequest-Handler-4:null) (logid:ec72387b) Caught:
>>>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Can't find
>>>>>>>>>>>>> volume:d93d3c0a-3859-4473-951d-9b5c5912c767
>>>>>>>>>>>>> at
>>>>>>>>>>>>> com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.getPhysicalDisk(LibvirtStoragePool.java:149)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.getVolumeStat(LibvirtGetVolumeStatsCommandWrapper.java:63)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:52)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:40)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1450)
>>>>>>>>>>>>> at com.cloud.agent.Agent.processRequest(Agent.java:645)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1083)
>>>>>>>>>>>>> at com.cloud.utils.nio.Task.call(Task.java:83)
>>>>>>>>>>>>> at com.cloud.utils.nio.Task.call(Task.java:29)
>>>>>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>>>>>>>>> 2020-01-14 09:25:15,259 INFO
>>>>>>>>>>>>> [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null)
>>>>>>>>>>>>> (logid:1a7e082e) Trying to fetch storage pool
>>>>>>>>>>>>> 9af0d1c6-85f2-3c55-94af-6ac17cb4024c from libvirt
>>>>>>>>>>>>> 2020-01-14 09:25:15,261 INFO
>>>>>>>>>>>>> [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null)
>>>>>>>>>>>>> (logid:1a7e082e) Asking libvirt to refresh storage pool
>>>>>>>>>>>>> 9af0d1c6-85f2-3c55-94af-6ac17cb4024c
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> And here the management server log:
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2020-01-14 09:21:27,105 DEBUG [c.c.a.t.Request]
>>>>>>>>>>>>> (AgentManager-Handler-2:null) (logid:) Seq 15-705657766613619075:
>>>>>>>>>>>>> Processing: { Ans: , MgmtId: 220777304233416, via: 15, Ver: v1, Flags: 10,
>>>>>>>>>>>>> [{"com.cloud.agent.api.Answer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException:
>>>>>>>>>>>>> Can't find volume:d93d3c0a-3859-4473-951d-9b5c5912c767\n\tat
>>>>>>>>>>>>> com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.getPhysicalDisk(LibvirtStoragePool.java:149)\n\tat
>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.getVolumeStat(LibvirtGetVolumeStatsCommandWrapper.java:63)\n\tat
>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:52)\n\tat
>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:40)\n\tat
>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)\n\tat
>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1450)\n\tat
>>>>>>>>>>>>> com.cloud.agent.Agent.processRequest(Agent.java:645)\n\tat
>>>>>>>>>>>>> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1083)\n\tat
>>>>>>>>>>>>> com.cloud.utils.nio.Task.call(Task.java:83)\n\tat
>>>>>>>>>>>>> com.cloud.utils.nio.Task.call(Task.java:29)\n\tat
>>>>>>>>>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat
>>>>>>>>>>>>> java.lang.Thread.run(Thread.java:748)\n","wait":0}}] }
>>>>>>>>>>>>> 2020-01-14 09:21:27,105 DEBUG [c.c.a.t.Request]
>>>>>>>>>>>>> (StatsCollector-6:ctx-fd801d0a) (logid:ec72387b) Seq 15-705657766613619075:
>>>>>>>>>>>>> Received: { Ans: , MgmtId: 220777304233416, via: 15(csdell017), Ver: v1,
>>>>>>>>>>>>> Flags: 10, { Answer } }
>>>>>>>>>>>>> 2020-01-14 09:21:27,105 DEBUG [c.c.a.m.AgentManagerImpl]
>>>>>>>>>>>>> (StatsCollector-6:ctx-fd801d0a) (logid:ec72387b) Details from executing
>>>>>>>>>>>>> class com.cloud.agent.api.GetVolumeStatsCommand:
>>>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Can't find
>>>>>>>>>>>>> volume:d93d3c0a-3859-4473-951d-9b5c5912c767
>>>>>>>>>>>>> at
>>>>>>>>>>>>> com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.getPhysicalDisk(LibvirtStoragePool.java:149)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.getVolumeStat(LibvirtGetVolumeStatsCommandWrapper.java:63)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:52)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:40)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1450)
>>>>>>>>>>>>> at com.cloud.agent.Agent.processRequest(Agent.java:645)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1083)
>>>>>>>>>>>>> at com.cloud.utils.nio.Task.call(Task.java:83)
>>>>>>>>>>>>> at com.cloud.utils.nio.Task.call(Task.java:29)
>>>>>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> on 09/01/20 12:58, Daan Hoogland wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Charlie, I think you'll have to explain a bit more about your environment
>>>>>>>>>>>>> to get an answer. what type of storage is it? Where did you migrate the VM
>>>>>>>>>>>>> from and to? What types() of hypervisors are you using? Though saying **the**
>>>>>>>>>>>>> agent logs suggests KVM, you are still leaving people guessing a lot.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Daan
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Daan
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Daan
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Daan
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Daan
>>>>>
>>>>
>>
>> --
>> Daan
>>
>
--
Daan