You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by "akrasnov-drv (via GitHub)" <gi...@apache.org> on 2025/03/25 17:14:00 UTC
[I] Wrong local primary storage is added on host maintenance/restart [cloudstack]
akrasnov-drv opened a new issue, #10616:
URL: https://github.com/apache/cloudstack/issues/10616
### problem
I enabled local storage for guest VMs.
When I place a host to maintenance, or if I just restart a host (without maintenance), CloudStack creates additional local storage instances with names hostname-local-random
In virsh I see just 1 pool corresponding to local storage but in primary storage in UI I see several.
If I try to start a VM on such a host, CS fails to deploy VM because it does not find local storage.
Another rather strange thing, is that if I put to maintenance wrong storage (intending to remove it), I see in virsh pool name changes to the one specified in disabled (still single local pool but with another name).
It happens also for empty pools and hosts without any VM.
### versions
CloudStack 4.20.0.0
Ubuntu 22.04.5 LTS
QEMU 1:6.2+dfsg-2ubuntu6.24
libvirt 8.0.0-1ubuntu7.10
local storage for guests is enabled
### The steps to reproduce the bug
1.Enable local storage for guests
2.Add host(s)
3.Ensure local primary storage was created for each host
4.Place host to maintenance / restart
5. After host is back check that there are 2 or more primary storage IDs are in primary storage list for the same host
6. Check that there is only 1 virsh pool corresponding to 1 of primary storage instances
7. Disable some of non-existing pools and see that virsh pool was renamed
...
### What to do about it?
To workaround this, after restart/maintenance I have to disable all existing local storage for a host, then enable 1 really existing, and then remove all others.
Cloudstack should not keep/add non-existing local storage. At least it should not affect VM deployment (CS should try to use the proper one)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Wrong local primary storage is added on host maintenance/restart [cloudstack]
Posted by "harikrishna-patnala (via GitHub)" <gi...@apache.org>.
harikrishna-patnala commented on issue #10616:
URL: https://github.com/apache/cloudstack/issues/10616#issuecomment-2830152675
@akrasnov-drv local.storage.uuid should not be empty and as far as I know this has to be entered manually by the operator.
when I checked the document also it says the same https://docs.cloudstack.apache.org/en/latest/adminguide/storage.html#manually-adding-local-storage-pool
In your case, may I ask you to do the following to make it a cleaner slate
1. remove all the local storages from CloudStack
2. make sure you have "local.storage.uuid" and "local.storage.path" in all the hosts
3. Restart cloudstack agent on all the KVM hosts
(you can quickly test the same steps on one of the hosts and repeat the same for other if it works)
I hope this helps.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Wrong local primary storage is added on host maintenance/restart [cloudstack]
Posted by "akrasnov-drv (via GitHub)" <gi...@apache.org>.
akrasnov-drv commented on issue #10616:
URL: https://github.com/apache/cloudstack/issues/10616#issuecomment-2758248794
I just checked and found no of this 2 in my `agent.properties`
I don't think I had path, but I believe I had uuid previously automatically added.
Must be it was removed when I removed additional stores from UI.
My cluster works well now, I mean local primary storage is functional and VMs are created, even though I have not storage related vars in `agent.properties`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Wrong local primary storage is added on host maintenance/restart [cloudstack]
Posted by "akrasnov-drv (via GitHub)" <gi...@apache.org>.
akrasnov-drv commented on issue #10616:
URL: https://github.com/apache/cloudstack/issues/10616#issuecomment-2821597205
No, it did not help, at least not fully.
I have a cluster with nodes and local storage working.
I added `local.storage.uuid=` to `/etc/cloudstack/agent/agent.properties` with uuid of existing local pool and restarted agents.
Then I restarted management and found doubled storage for a server. Existing virsh pool still matches `local.storage.uuid=`
So I decided to remove the extra one. But when I placed the extra one to maintenance, virsh pool was rename (if not replaced) and does not match `local.storage.uuid=` anymore.
Here is the relevant output, no other commands was executed in that shell session:
```
root@host:~# virsh pool-list
Name State Autostart
------------------------------------------------------------
4798c891-2d66-35c8-b0e3-706fedb51483 active no
e6b7e1ea-9cfb-409a-8daf-fb7bdc124d2e active no
root@host:~# grep storage /etc/cloudstack/agent/agent.properties
local.storage.uuid=e6b7e1ea-9cfb-409a-8daf-fb7bdc124d2e
#### here I placed local storage for 53529f92-5fea-4ad0-b908-f0d95630ff3d to maintenance
root@host:~# virsh pool-list
Name State Autostart
------------------------------------------------------------
4798c891-2d66-35c8-b0e3-706fedb51483 active no
53529f92-5fea-4ad0-b908-f0d95630ff3d active no
root@host:~# grep storage /etc/cloudstack/agent/agent.properties
local.storage.uuid=e6b7e1ea-9cfb-409a-8daf-fb7bdc124d2e
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Wrong local primary storage is added on host maintenance/restart [cloudstack]
Posted by "harikrishna-patnala (via GitHub)" <gi...@apache.org>.
harikrishna-patnala closed issue #10616: Wrong local primary storage is added on host maintenance/restart
URL: https://github.com/apache/cloudstack/issues/10616
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Wrong local primary storage is added on host maintenance/restart [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on issue #10616:
URL: https://github.com/apache/cloudstack/issues/10616#issuecomment-2823892090
reopening on request by user (see https://github.com/apache/cloudstack/issues/10763#issue-3011342361)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Wrong local primary storage is added on host maintenance/restart [cloudstack]
Posted by "akrasnov-drv (via GitHub)" <gi...@apache.org>.
akrasnov-drv commented on issue #10616:
URL: https://github.com/apache/cloudstack/issues/10616#issuecomment-2849263531
Hi,
- Removed local storage
- stopped cloudstack-agent service
- added local storage path (uuid already was configured)
```
local.storage.uuid=a21cb1ec-9148-40cc-a54b-0e9f9924a31a
local.storage.path=/var/lib/libvirt/images
```
- ensured that there is no local pool in virsh
- started cloudstack-agent
And now I really see that the pool is created with proper uuid from agent.properties.
The main 2 differences with what I did before are
- storage path
- no existing pool (previously I tried to save existing pool uuid to agent properties)
Looks like local.storage agent properties are must to have
Why not to add them on the 1st agent start when it re-writes agent.properties anyway?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Wrong local primary storage is added on host maintenance/restart [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on issue #10616:
URL: https://github.com/apache/cloudstack/issues/10616#issuecomment-2793280881
@akrasnov-drv , is there still an issue to address or can we close this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Wrong local primary storage is added on host maintenance/restart [cloudstack]
Posted by "akrasnov-drv (via GitHub)" <gi...@apache.org>.
akrasnov-drv commented on issue #10616:
URL: https://github.com/apache/cloudstack/issues/10616#issuecomment-3027077201
I just found that on fresh install agent.properties really contains
```
# MANDATORY: The UUID for the local storage pool. Multiple values can be entered and separated by commas.
# Generated with "uuidgen".
local.storage.uuid=
```
lines. I must be overlooked it, as after agent connection original config is purged to contain just defined strings.
Even though it would be nice to generate uuid automatically on the 1st start, if it was not defined by admin, with the above lines it's can't be considered as a bug.
Feel free to create improvement request, or just to close this issue.
Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Wrong local primary storage is added on host maintenance/restart [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on issue #10616:
URL: https://github.com/apache/cloudstack/issues/10616#issuecomment-3031158959
@akrasnov-drv can you define how you would want the user experience improved in this respect and open a new issue for it? closing now, but if you need more discussion we can convert the issue to a discussion as well..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Wrong local primary storage is added on host maintenance/restart [cloudstack]
Posted by "akrasnov-drv (via GitHub)" <gi...@apache.org>.
akrasnov-drv commented on issue #10616:
URL: https://github.com/apache/cloudstack/issues/10616#issuecomment-2821089354
Sorry missed the updates.
I believe I have not removed anything related to local.storage from agent properties, it just initially was not there.
I'm checking now if the issue is solved by fixing local.storage.uuid in agent properties. If yes, it can be considered as a workaround. But as agent creates local storage after it already up, and this uuid is kind of crucial, I'd expect agent to set it after local storage is created.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Wrong local primary storage is added on host maintenance/restart [cloudstack]
Posted by "akrasnov-drv (via GitHub)" <gi...@apache.org>.
akrasnov-drv commented on issue #10616:
URL: https://github.com/apache/cloudstack/issues/10616#issuecomment-3163089262
Looks like this parameter is must to have, otherwise we are getting issues.
I'd suggest to automatically set `local.storage.uuid` in agent config with the new pool value if this parameter is not set yet (on first start).
Alternatively agent may reject to start without this value set (as it happens without guid, though in my pov it's less desirable)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Wrong local primary storage is added on host maintenance/restart [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland closed issue #10616: Wrong local primary storage is added on host maintenance/restart
URL: https://github.com/apache/cloudstack/issues/10616
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Wrong local primary storage is added on host maintenance/restart [cloudstack]
Posted by "harikrishna-patnala (via GitHub)" <gi...@apache.org>.
harikrishna-patnala commented on issue #10616:
URL: https://github.com/apache/cloudstack/issues/10616#issuecomment-2812126571
@akrasnov-drv during my testing while fixing the cases of local storages in github.com/apache/cloudstack/pull/10231 I found that the values of "local.storage.uuid and local.storage.path" in /etc/cloudstack/agent/agent.properties on the KVM host are the source to create corresponding primary storages in CloudStack. This happens whenever the kvm agent is restarted.
In your case I believe since you have removed all the storages except the one you wanted, same is reflecting now in the agent.properties.
If in case you find any issue with your environment, please feel free to reopen this or create new issue with more details needed for investigation. Thanks for all the information.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Wrong local primary storage is added on host maintenance/restart [cloudstack]
Posted by "harikrishna-patnala (via GitHub)" <gi...@apache.org>.
harikrishna-patnala commented on issue #10616:
URL: https://github.com/apache/cloudstack/issues/10616#issuecomment-2753909496
@akrasnov-drv may I know what are the values of "local.storage.uuid and local.storage.path" in /etc/cloudstack/agent/agent.properties on the KVM host.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Wrong local primary storage is added on host maintenance/restart [cloudstack]
Posted by "akrasnov-drv (via GitHub)" <gi...@apache.org>.
akrasnov-drv commented on issue #10616:
URL: https://github.com/apache/cloudstack/issues/10616#issuecomment-2849270658
Without path it also works.
So uuid should be configured beforehand. If I add uuid of existing pool and restart agent, it does not work well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Wrong local primary storage is added on host maintenance/restart [cloudstack]
Posted by "akrasnov-drv (via GitHub)" <gi...@apache.org>.
akrasnov-drv commented on issue #10616:
URL: https://github.com/apache/cloudstack/issues/10616#issuecomment-2849275315
And, btw, you pointed to "Manually adding Local Storage Pool" in the documentation. I do not add it manually. Actually I did not touch it at all till I found that my VMs does not start because ofI pool uuid mismatch between virsh and local primary storage in CS.
I still think this is a bug.
Or `local.storage.uuid` should be added automatically to agent.properties (if it's must to have), or CS should properly manage each host's single local storage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org