You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Charlie Holeowsky <ch...@gmail.com> on 2020/01/07 16:57:25 UTC
Re: new primary storage
Hi all,
on a host that use this new storage I found a problem that could be related
to the fact that the statistics are not being updated.
After migrating a newly created VM to the new storage server, a series of
messages like the following appear in the agent logs:
2020-01-07 17:30:45,868 WARN [cloud.agent.Agent]
(agentRequest-Handler-2:null) (logid:c023b7f8) Caught:
com.cloud.utils.exception.CloudRuntimeException: Can't find
volume:d93d3c0a-3859-4473-951d-9b5c5912c767
at
com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.getPhysicalDisk(LibvirtStoragePool.java:149)
at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.getVolumeStat(LibvirtGetVolumeStatsCommandWrapper.java:63)
Looking for the volume uuid "d93d3c0a-3859-4473-951d-9b5c5912c767" I found
that it belongs to the VM just migrated but has a different "path" column
value (normally I see that the values of the two tables coincide).
On the other hand, there is another record in the volumes table with uuid =
NULL and path = d93d3c0a-3859-4473-951d-9b5c5912c767 with state=Expunged.
Could this be the cause of the failure to update the volume data statistics?
Il giorno mar 17 dic 2019 alle ore 15:32 charlie Holeowsky <
charlie.holeowsky@gmail.com> ha scritto:
> Hi,
> any idea about about this problem?
>
>
> Il giorno mar 10 dic 2019 alle ore 17:23 charlie Holeowsky <
> charlie.holeowsky@gmail.com> ha scritto:
>
>> Hi users,
>> I recently added a new primary nfs storage to my cluster (cloudstack
>> 4.11.2 with kvm on the Ubuntu system).
>>
>> It works well but on the metrics of the VM storage volume the "physical
>> size" and "usage" columns are empty while in the rows of the VM of the
>> other primary storage I can see the right data.
>>
>> Where can i go to look to see where the problem is?
>>
>> Thanks
>> Charlie
>>
>
Re: new primary storage
Posted by Daan Hoogland <da...@gmail.com>.
Charlie, I think you'll have to explain a bit more about your environment
to get an answer. what type of storage is it? Where did you migrate the VM
from and to? What types() of hypervisors are you using? Though saying *the*
agent logs suggests KVM, you are still leaving people guessing a lot.
On Tue, Jan 7, 2020 at 6:54 PM Charlie Holeowsky <
charlie.holeowsky@gmail.com> wrote:
> Hi all,
> on a host that use this new storage I found a problem that could be related
> to the fact that the statistics are not being updated.
>
> After migrating a newly created VM to the new storage server, a series of
> messages like the following appear in the agent logs:
>
> 2020-01-07 17:30:45,868 WARN [cloud.agent.Agent]
> (agentRequest-Handler-2:null) (logid:c023b7f8) Caught:
> com.cloud.utils.exception.CloudRuntimeException: Can't find
> volume:d93d3c0a-3859-4473-951d-9b5c5912c767
> at
>
> com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.getPhysicalDisk(LibvirtStoragePool.java:149)
> at
>
> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.getVolumeStat(LibvirtGetVolumeStatsCommandWrapper.java:63)
>
> Looking for the volume uuid "d93d3c0a-3859-4473-951d-9b5c5912c767" I found
> that it belongs to the VM just migrated but has a different "path" column
> value (normally I see that the values of the two tables coincide).
>
> On the other hand, there is another record in the volumes table with uuid =
> NULL and path = d93d3c0a-3859-4473-951d-9b5c5912c767 with state=Expunged.
>
> Could this be the cause of the failure to update the volume data
> statistics?
>
> Il giorno mar 17 dic 2019 alle ore 15:32 charlie Holeowsky <
> charlie.holeowsky@gmail.com> ha scritto:
>
> > Hi,
> > any idea about about this problem?
> >
> >
> > Il giorno mar 10 dic 2019 alle ore 17:23 charlie Holeowsky <
> > charlie.holeowsky@gmail.com> ha scritto:
> >
> >> Hi users,
> >> I recently added a new primary nfs storage to my cluster (cloudstack
> >> 4.11.2 with kvm on the Ubuntu system).
> >>
> >> It works well but on the metrics of the VM storage volume the "physical
> >> size" and "usage" columns are empty while in the rows of the VM of the
> >> other primary storage I can see the right data.
> >>
> >> Where can i go to look to see where the problem is?
> >>
> >> Thanks
> >> Charlie
> >>
> >
>
--
Daan
Re: new primary storage
Posted by Andrija Panic <an...@gmail.com>.
There are some known issues in (4.13?) where KVM host is asked to report
statistics for other hypervisors storage pools, and that can cause silly
errors in the log - I would ignore those for now.
Better spin a **new** VM on that new storage pool (not migrate over
existing VM) and give it some time and see if your original problem (stats
missing) is fixed.
best
On Fri, 10 Jan 2020 at 17:24, Charlie Holeowsky <ch...@gmail.com>
wrote:
> Hi all,
> I keep getting the following error about a volume that has been migrated:
>
> 2020-01-10 11:21:28,701 DEBUG [c.c.a.t.Request]
> (AgentManager-Handler-2:null) (logid:) Seq 15-6725563093524421010:
> Processing: { Ans: , MgmtId: 220777304233416, via: 15, Ver: v1, Flags: 10,
>
> [{"com.cloud.agent.api.Answer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException:
> Can't find volume:d93d3c0a-3859-4473-951d-9b5c5912c767\n\tat
>
> com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.getPhysicalDisk(LibvirtStoragePool.java:149)\n\tat
>
> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.getVolumeStat(LibvirtGetVolumeStatsCommandWrapper.java:63)\n\tat
>
> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:52)\n\tat
>
> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:40)\n\tat
>
> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)\n\tat
>
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1450)\n\tat
> com.cloud.agent.Agent.processRequest(Agent.java:645)\n\tat
> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1083)\n\tat
> com.cloud.utils.nio.Task.call(Task.java:83)\n\tat
> com.cloud.utils.nio.Task.call(Task.java:29)\n\tat
> java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat
> java.lang.Thread.run(Thread.java:748)\n","wait":0}}] }
> 2020-01-10 11:21:28,701 DEBUG [c.c.a.m.AgentManagerImpl]
> (StatsCollector-1:ctx-9766c817) (logid:5715358a) Details from executing
> class com.cloud.agent.api.GetVolumeStatsCommand:
> com.cloud.utils.exception.CloudRuntimeException: Can't find
> volume:d93d3c0a-3859-4473-951d-9b5c5912c767
>
> what could happen if I create a symbolic link between the file currently in
> use (the actual path of volume in use) and what cloudstack is looking for
> (the d93d3c0a-3859-4473-951d-9b5c5912c767, the uuid of volume in use) so
> that it can collect the data of the file?
>
> Il giorno mar 7 gen 2020 alle ore 17:57 Charlie Holeowsky <
> charlie.holeowsky@gmail.com> ha scritto:
>
> > Hi all,
> > on a host that use this new storage I found a problem that could be
> > related to the fact that the statistics are not being updated.
> >
> > After migrating a newly created VM to the new storage server, a series of
> > messages like the following appear in the agent logs:
> >
> > 2020-01-07 17:30:45,868 WARN [cloud.agent.Agent]
> > (agentRequest-Handler-2:null) (logid:c023b7f8) Caught:
> > com.cloud.utils.exception.CloudRuntimeException: Can't find
> > volume:d93d3c0a-3859-4473-951d-9b5c5912c767
> > at
> >
> com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.getPhysicalDisk(LibvirtStoragePool.java:149)
> > at
> >
> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.getVolumeStat(LibvirtGetVolumeStatsCommandWrapper.java:63)
> >
> > Looking for the volume uuid "d93d3c0a-3859-4473-951d-9b5c5912c767" I
> found
> > that it belongs to the VM just migrated but has a different "path" column
> > value (normally I see that the values of the two tables coincide).
> >
> > On the other hand, there is another record in the volumes table with uuid
> > = NULL and path = d93d3c0a-3859-4473-951d-9b5c5912c767 with
> state=Expunged.
> >
> > Could this be the cause of the failure to update the volume data
> > statistics?
> >
> > Il giorno mar 17 dic 2019 alle ore 15:32 charlie Holeowsky <
> > charlie.holeowsky@gmail.com> ha scritto:
> >
> >> Hi,
> >> any idea about about this problem?
> >>
> >>
> >> Il giorno mar 10 dic 2019 alle ore 17:23 charlie Holeowsky <
> >> charlie.holeowsky@gmail.com> ha scritto:
> >>
> >>> Hi users,
> >>> I recently added a new primary nfs storage to my cluster (cloudstack
> >>> 4.11.2 with kvm on the Ubuntu system).
> >>>
> >>> It works well but on the metrics of the VM storage volume the "physical
> >>> size" and "usage" columns are empty while in the rows of the VM of the
> >>> other primary storage I can see the right data.
> >>>
> >>> Where can i go to look to see where the problem is?
> >>>
> >>> Thanks
> >>> Charlie
> >>>
> >>
>
--
Andrija Panić
Re: new primary storage
Posted by Charlie Holeowsky <ch...@gmail.com>.
Hi all,
I keep getting the following error about a volume that has been migrated:
2020-01-10 11:21:28,701 DEBUG [c.c.a.t.Request]
(AgentManager-Handler-2:null) (logid:) Seq 15-6725563093524421010:
Processing: { Ans: , MgmtId: 220777304233416, via: 15, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException:
Can't find volume:d93d3c0a-3859-4473-951d-9b5c5912c767\n\tat
com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.getPhysicalDisk(LibvirtStoragePool.java:149)\n\tat
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.getVolumeStat(LibvirtGetVolumeStatsCommandWrapper.java:63)\n\tat
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:52)\n\tat
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:40)\n\tat
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)\n\tat
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1450)\n\tat
com.cloud.agent.Agent.processRequest(Agent.java:645)\n\tat
com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1083)\n\tat
com.cloud.utils.nio.Task.call(Task.java:83)\n\tat
com.cloud.utils.nio.Task.call(Task.java:29)\n\tat
java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat
java.lang.Thread.run(Thread.java:748)\n","wait":0}}] }
2020-01-10 11:21:28,701 DEBUG [c.c.a.m.AgentManagerImpl]
(StatsCollector-1:ctx-9766c817) (logid:5715358a) Details from executing
class com.cloud.agent.api.GetVolumeStatsCommand:
com.cloud.utils.exception.CloudRuntimeException: Can't find
volume:d93d3c0a-3859-4473-951d-9b5c5912c767
what could happen if I create a symbolic link between the file currently in
use (the actual path of volume in use) and what cloudstack is looking for
(the d93d3c0a-3859-4473-951d-9b5c5912c767, the uuid of volume in use) so
that it can collect the data of the file?
Il giorno mar 7 gen 2020 alle ore 17:57 Charlie Holeowsky <
charlie.holeowsky@gmail.com> ha scritto:
> Hi all,
> on a host that use this new storage I found a problem that could be
> related to the fact that the statistics are not being updated.
>
> After migrating a newly created VM to the new storage server, a series of
> messages like the following appear in the agent logs:
>
> 2020-01-07 17:30:45,868 WARN [cloud.agent.Agent]
> (agentRequest-Handler-2:null) (logid:c023b7f8) Caught:
> com.cloud.utils.exception.CloudRuntimeException: Can't find
> volume:d93d3c0a-3859-4473-951d-9b5c5912c767
> at
> com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.getPhysicalDisk(LibvirtStoragePool.java:149)
> at
> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.getVolumeStat(LibvirtGetVolumeStatsCommandWrapper.java:63)
>
> Looking for the volume uuid "d93d3c0a-3859-4473-951d-9b5c5912c767" I found
> that it belongs to the VM just migrated but has a different "path" column
> value (normally I see that the values of the two tables coincide).
>
> On the other hand, there is another record in the volumes table with uuid
> = NULL and path = d93d3c0a-3859-4473-951d-9b5c5912c767 with state=Expunged.
>
> Could this be the cause of the failure to update the volume data
> statistics?
>
> Il giorno mar 17 dic 2019 alle ore 15:32 charlie Holeowsky <
> charlie.holeowsky@gmail.com> ha scritto:
>
>> Hi,
>> any idea about about this problem?
>>
>>
>> Il giorno mar 10 dic 2019 alle ore 17:23 charlie Holeowsky <
>> charlie.holeowsky@gmail.com> ha scritto:
>>
>>> Hi users,
>>> I recently added a new primary nfs storage to my cluster (cloudstack
>>> 4.11.2 with kvm on the Ubuntu system).
>>>
>>> It works well but on the metrics of the VM storage volume the "physical
>>> size" and "usage" columns are empty while in the rows of the VM of the
>>> other primary storage I can see the right data.
>>>
>>> Where can i go to look to see where the problem is?
>>>
>>> Thanks
>>> Charlie
>>>
>>