You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by "jdhirst (via GitHub)" <gi...@apache.org> on 2025/06/29 07:33:34 UTC
[I] KVM local storage pool gets removed from hosts [cloudstack]
jdhirst opened a new issue, #11104:
URL: https://github.com/apache/cloudstack/issues/11104
### problem
## Issue Details
After switching over to main for testing the CKS improvements, I ran into an issue where my compute nodes lose their local storage pools in libvirt. The pools exist in Cloudstack however the pools get undefined. This can be temporarily fixed by manually defining the pools with the same UUID via `virsh pool-define`, however they get removed after as well.
Both of the compute nodes have `local.storage.uuid` defined in their agent.properties file. Once this issue starts to happen, restarting the agent does not cause the pool to come back. However, if you change the UUID in `agent.properties`, a new pool gets created and will start without issue.
The affect of this is that after the pool is removed, existing instances continue to run, however new instances cannot be created until the pool is manually recreated.
Creating a new pool with a different UUID works for a time, but eventually also gets removed. I'm not exactly sure what is triggering the removal.
## Log Snippets
Both management server and agent encounter errors trying to find the pool:
```
2025-06-29 07:36:58,138 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-8a6c9ea0]) (logid:58ac492b) Details from executing class com.cloud.agent.api.GetVolumeStatsCommand: Can't get vm disk stats: Could not fetch storage pool 27ea62dc-ded2-4a25-9115-277c981f33fa from libvirt due to org.libvirt.LibvirtException: Storage pool not found: no storage pool with matching uuid '27ea62dc-ded2-4a25-9115-277c981f33fa'
2025-06-29 07:36:58,096 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-2:[]) (logid:58ac492b) Processing command: com.cloud.agent.api.GetVolumeStatsCommand
2025-06-29 07:36:58,096 DEBUG [kvm.resource.LibvirtConnection] (AgentRequest-Handler-2:[]) (logid:58ac492b) Looking for libvirtd connection at: qemu:///system
2025-06-29 07:36:58,096 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-2:[]) (logid:58ac492b) Trying to fetch storage pool 27ea62dc-ded2-4a25-9115-277c981f33fa from libvirt
2025-06-29 07:36:58,096 DEBUG [kvm.resource.LibvirtConnection] (AgentRequest-Handler-2:[]) (logid:58ac492b) Looking for libvirtd connection at: qemu:///system
2025-06-29 07:36:58,097 DEBUG [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-2:[]) (logid:58ac492b) Could not find storage pool 27ea62dc-ded2-4a25-9115-277c981f33fa in libvirt
```
The pool is being removed by the management server:
```
2025-06-29 03:36:47,496 DEBUG [c.c.s.StorageManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Removing pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} from host Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}
2025-06-29 03:36:47,511 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Seq 2-3926857400090361861: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.DeleteStoragePoolCommand":{"_pool":{"id":"13","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d","host":"10.25.0.3","path":"/var/lib/libvirt/images","port":"0","type":"Filesystem"},"_localPath":"/mnt//492af6d4-b5d1-3dc8-8a50-040101d28230","_removeDatastore":"false","wait":"0","bypassHostMaintenance":"false"}}] }
2025-06-29 03:36:47,563 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Seq 2-3926857400090361861: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { Answer } }
2025-06-29 03:36:47,576 INFO [o.a.c.s.d.p.DefaultHostListener] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Connection removed between storage pool: StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} and host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}
2025-06-29 03:36:47,606 DEBUG [c.c.s.StorageManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Found storage pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} of type Filesystem with overprovisioning factor 2
2025-06-29 03:36:47,608 DEBUG [c.c.s.StorageManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Total over provisioned capacity of the pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} is (306.79 GB) 329410707456
2025-06-29 03:36:47,629 DEBUG [c.c.s.StorageManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Successfully set Capacity - (306.79 GB) 329410707456 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"}, PodId 1
```
Agent's logs showing pool removal:
```
2025-06-29 03:36:47,233 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-1:[]) (logid:4c5c4abb) Request:Seq 1-4725683383995203589: { Cmd , MgmtId: 90520738888109, via: 1, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.DeleteStoragePoolCommand":{"_pool":{"id":"12","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa","host":"10.25.0.2","path":"/var/lib/libvirt/images","port":"0","type":"Filesystem"},"_localPath":"/mnt//36cfe92b-5de4-3baf-9813-796fbddeb8af","_removeDatastore":"false","wait":"0","bypassHostMaintenance":"false"}}] }
2025-06-29 03:36:47,233 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-1:[]) (logid:4c5c4abb) Processing command: com.cloud.agent.api.DeleteStoragePoolCommand
2025-06-29 03:36:47,233 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-1:[]) (logid:4c5c4abb) Attempting to remove storage pool 27ea62dc-ded2-4a25-9115-277c981f33fa from libvirt
2025-06-29 03:36:47,234 DEBUG [kvm.resource.LibvirtConnection] (AgentRequest-Handler-1:[]) (logid:4c5c4abb) Looking for libvirtd connection at: qemu:///system
2025-06-29 03:36:47,235 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-1:[]) (logid:4c5c4abb) Storage pool 27ea62dc-ded2-4a25-9115-277c981f33fa has no corresponding secret. Not removing any secret.
2025-06-29 03:36:47,236 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-1:[]) (logid:4c5c4abb) Storage pool 27ea62dc-ded2-4a25-9115-277c981f33fa was successfully removed from libvirt.
```
The pools still show as up in the UI:
And in the DB: (as you can see, I attempted to replace the pools with new ones to see what would happen)
|id|name|uuid|pool_type|port|data_center_id|pod_id|cluster_id|used_bytes|capacity_bytes|host_address|user_info|path|created|removed|update_time|status|storage_provider_name|scope|hypervisor|managed|capacity_iops|parent|used_iops|
|--|----|----|---------|----|--------------|------|----------|----------|--------------|------------|---------|----|-------|-------|-----------|------|---------------------|-----|----------|-------|-------------|------|---------|
|1|cs-compute1.cloud.redacted.net-local-0322a17b||Filesystem|0|1|1|1|424638771200|913383583744|10.25.0.2||/var/lib/libvirt/images|2024-09-17 12:14:34|2025-06-28 12:44:14|2025-06-27 12:40:43|Up|DefaultPrimary|HOST|KVM|0||0|120|
|2|cs-compute2.cloud.redacted.net-local-7aa8e196||Filesystem|0|1|1|1|114410532864|164705353728|10.25.0.3||/var/lib/libvirt/images|2024-09-17 12:17:05|2025-06-28 12:43:04|2025-06-28 09:09:47|Up|DefaultPrimary|HOST|KVM|0||0|60|
|4|nfs|10805e0a-2205-3158-993f-9350b08b9137|NetworkFilesystem|2049|1|1|1|36540069576704|61401196134400|nas.redacted.net||/volume1/cs-primary|2024-09-19 16:55:58||2025-06-29 06:34:38|Up|DefaultPrimary|CLUSTER|KVM|0||0||
|5|nfs-ssd|80a947f2-ab02-3d8c-b1c0-7ba964ae0e80|NetworkFilesystem|2049|1|1|1|358665551872|1418370088960|nas.redacted.net||/volume2/cs-primary-ssd|2024-09-28 10:31:47||2025-06-29 06:34:38|Up|DefaultPrimary|CLUSTER|KVM|0||0||
|12|cs-compute1.cloud.redacted.net-local-27ea62dc|27ea62dc-ded2-4a25-9115-277c981f33fa|Filesystem|0|1|1|1|424638771200|913383583744|10.25.0.2||/var/lib/libvirt/images|2025-06-28 13:15:34||2025-06-29 00:38:22|Up|DefaultPrimary|HOST|KVM|0||0|29|
|13|cs-compute2.cloud.redacted.net-local-d997caf8|d997caf8-e98c-498c-aac2-3016f6ae2f5d|Filesystem|0|1|1|1|114449989632|164705353728|10.25.0.3||/var/lib/libvirt/images|2025-06-28 13:21:47||2025-06-29 00:49:49|Up|DefaultPrimary|HOST|KVM|0||0|60|
The removed column is set to NULL for the two current local storage pools even **after** they have been removed from the hosts.
## Tracing
I attempted to do some tracing of this issue (see trace logs attached), however I am still unable to find what triggers the pool removal call in the first place.
## Log Files
### versions
I'm using the latest nightly build as of yesterday, however this issue has been there ever since I switched over to nightly.
Current build: 4.21.0.0-SNAPSHOT.20250628
OS: RHEL 8.10
The cluster is made up of one management node and two KVM compute nodes, all running the same build.
### The steps to reproduce the bug
1.
2.
3.
...
### What to do about it?
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KVM local storage pool gets removed from hosts [cloudstack]
Posted by "sureshanaparti (via GitHub)" <gi...@apache.org>.
sureshanaparti closed issue #11104: KVM local storage pool gets removed from hosts
URL: https://github.com/apache/cloudstack/issues/11104
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KVM local storage pool gets removed from hosts [cloudstack]
Posted by "kiranchavala (via GitHub)" <gi...@apache.org>.
kiranchavala commented on issue #11104:
URL: https://github.com/apache/cloudstack/issues/11104#issuecomment-3017906356
@jdhirst
Can you add both parameters (local.storage.uuid and local.storage.path ) in the agent.properities and cross check
https://docs.cloudstack.apache.org/en/4.20.1.0/adminguide/storage.html#manually-adding-local-storage-pool
https://docs.cloudstack.apache.org/en/4.20.1.0/adminguide/storage.html#adding-a-local-storage-pool-via-ui
There was recent fix
https://github.com/apache/cloudstack/pull/10231
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KVM local storage pool gets removed from hosts [cloudstack]
Posted by "jdhirst (via GitHub)" <gi...@apache.org>.
jdhirst commented on issue #11104:
URL: https://github.com/apache/cloudstack/issues/11104#issuecomment-3036017680
@harikrishna-patnala Tagging you since you wrote b17808bfba1d914b17d25ef82c0f372b295465c5, what do you think of this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KVM local storage pool gets removed from hosts [cloudstack]
Posted by "jdhirst (via GitHub)" <gi...@apache.org>.
jdhirst commented on issue #11104:
URL: https://github.com/apache/cloudstack/issues/11104#issuecomment-3036009385
@kiranchavala I was able to reproduce this issue on a fresh installation using a single RHEL 8 VM following the [quick installation guide](https://docs.cloudstack.apache.org/en/latest/quickinstallationguide/qig.html)
Everything was working fine until the agent service restarted and then the local pool vanished with the same behaviour I was seeing before.
This is the repo I used:
```
[cloudstack]
name=cloudstack
baseurl=https://download.cloudstack.org/testing/nightly/2025-07-02/el8/4.21/
enabled=1
gpgcheck=0
```
Logs:
agent:
[agent.err.log](https://github.com/user-attachments/files/21053648/agent.err.log)
[agent.log](https://github.com/user-attachments/files/21053643/agent.log)
[security_group.log](https://github.com/user-attachments/files/21053650/security_group.log)
[setup.log](https://github.com/user-attachments/files/21053651/setup.log)
[agent.out.log](https://github.com/user-attachments/files/21053654/agent.out.log)
management:
[access.log](https://github.com/user-attachments/files/21053671/access.log)
[apilog.log](https://github.com/user-attachments/files/21053680/apilog.log)
[management-server.err.log](https://github.com/user-attachments/files/21053703/management-server.err.log)
[management-server.out.log](https://github.com/user-attachments/files/21053686/management-server.out.log)
[management-server.log](https://github.com/user-attachments/files/21053683/management-server.log)
## debugging
I connected a remote debugger and started to poke through the code and found that this method is being called:
I upgraded to the latest build since I saw that there were some storage-related changes that came in from 4.20, however this didn't change anything.
I then proceeded to downgrade the management server to the latest build on the 4.20 branch (and nuked the db / re-created the zone) and ran the debugger again.
I was then not able to reproduce the issue, and the storage pool would always return even if I destroyed it from libvirt.
## possible root cause
I believe that the root cause of this issue is the change that was made to StoragePoolMonitor.java as part of b17808bfba1d914b17d25ef82c0f372b295465c5
https://github.com/apache/cloudstack/blob/b17808bfba1d914b17d25ef82c0f372b295465c5/server/src/main/java/com/cloud/storage/listener/StoragePoolMonitor.java#L159
This check for previously connected pools fails to take local pools into account and therefore always targets them for destruction:
```
List<StoragePoolHostVO> previouslyConnectedPools = new ArrayList<>();
previouslyConnectedPools.addAll(_storageManager.findStoragePoolsConnectedToHost(host.getId()));
```
Local pools are not available in the `pools` list as they are not returned in the pool list queries further up in the file so therefore they don't get removed from this previously connected pools list, and thus get destroyed.
This can be seen here:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KVM local storage pool gets removed from hosts [cloudstack]
Posted by "kiranchavala (via GitHub)" <gi...@apache.org>.
kiranchavala commented on issue #11104:
URL: https://github.com/apache/cloudstack/issues/11104#issuecomment-3018833703
Thanks @jdhirst
Will do a quick test on the latest nightly build packages
https://download.cloudstack.org/testing/nightly/latest/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KVM local storage pool gets removed from hosts [cloudstack]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #11104:
URL: https://github.com/apache/cloudstack/issues/11104#issuecomment-3016377711
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KVM local storage pool gets removed from hosts [cloudstack]
Posted by "jdhirst (via GitHub)" <gi...@apache.org>.
jdhirst commented on issue #11104:
URL: https://github.com/apache/cloudstack/issues/11104#issuecomment-3018017361
@kiranchavala I set the path and the same behaviour occurs. I manually re-defined the pool and upon agent restart, it vanishes again.
Here is my agent.properties
```
#Storage
#Mon Jun 30 08:51:10 CEST 2025
cluster=1
pod=1
host.reserved.mem.mb=1024
domr.scripts.dir=scripts/network/domr/kvm
vm.memballoon.disable=true
guest.cpu.mode=host-passthrough
guest.network.device=cloudbr0
libvirt.events.enabled=true
hypervisor.type=kvm
zone=1
public.network.device=cloudbr0
local.storage.uuid=27ea62dc-ded2-4a25-9115-277c981f33fa
host=172.16.20.40@static
workers=5
guest.cpu.features=vmx
LibvirtComputingResource.name=cs-compute1.cloud.redacted.net
resource=com.cloud.hypervisor.kvm.resource.LibvirtComputingResource
private.network.device=cloudbr0
keystore.passphrase=tpm5Qcq2sPJD9aqh
port=8250
LibvirtComputingResource.uuid=ddcf716a-cb61-4697-8501-9320c3f30a4b
guid=6a680d56-20e5-39e2-b48d-240eb5d552cc
LibvirtComputingResource.id=1
iscsi.session.cleanup.enabled=false
vm.migrate.wait=3600
local.storage.path=/var/lib/libvirt/images
```
Here is the local storage from the UI (not created via the UI however, automatically created via setting the local.storage.uuid value in agent properties)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KVM local storage pool gets removed from hosts [cloudstack]
Posted by "sureshanaparti (via GitHub)" <gi...@apache.org>.
sureshanaparti commented on issue #11104:
URL: https://github.com/apache/cloudstack/issues/11104#issuecomment-3127426271
Fixed in #11200
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KVM local storage pool gets removed from hosts [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on issue #11104:
URL: https://github.com/apache/cloudstack/issues/11104#issuecomment-3113003570
@jdhirst can you test #11200 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org