You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by "Sparc0 (via GitHub)" <gi...@apache.org> on 2026/03/03 16:54:27 UTC
[I] GET certain paths like systemVM and networkofferings are very very slow [cloudstack]
Sparc0 opened a new issue, #12740:
URL: https://github.com/apache/cloudstack/issues/12740
### problem
**Background**
We have just recently deployed our first production Cloudstack environment.
At the moment we have 2 zones 1 pod 1 cluster, with 4 hosts each inside.
In total we have created 4 extra networkofferings over the default ones, 5 shared networks and 5 instances.
And we have 4 systemvm's.
So as you can understand its still a very small deployment still.
Our management server is a VM.
Running on a oVirt cluster.
VM is 4 cores and 8GB ram.
MariaDB and CS Mangement server are located on the same VM.
**Issue**
When we try to GET certain paths either if its using the webui or cloudmonkey script the load times are very long and our MariaDB goes to 100% for a while on one core.
Path where the issue is seen are:
GET systemVMs ~45 seconds
GET networkofferings ~50seconds
<img width="2339" height="141" alt="Image" src="https://github.com/user-attachments/assets/fbd19df6-aa69-4ad0-bb45-c13468df5953" />
Doing following select queries towards the MariaDB the response back is instant and no load seen on the DB.
```
select * from network_offerings;
select * from console_proxy;
select * from vm_instance;
select * from secondary_storage_vm;
```
**More Background**
During the setup we accidently imported each ldap user entry as a domain resulted in ~52k now inactive domains. I believe that we started seeing this issue after that was done. We have deleted each domain but they still show up when doing a select * from domain but as inactive.
I also remember that we had alot of systemvm entries during the first cluster because cloudstack had some network issue and it created alot of systemvm entries. But they have also been deleted.
I have enabled expunged.resources.purge.enabled
And i have cleaned up our event and alert tables.
### versions
ACS 4.22
### The steps to reproduce the bug
Not been able to reproduce the issue.
### What to do about it?
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] GET certain paths like systemVM and networkofferings are very very slow [cloudstack]
Posted by "Sparc0 (via GitHub)" <gi...@apache.org>.
Sparc0 commented on issue #12740:
URL: https://github.com/apache/cloudstack/issues/12740#issuecomment-4295856184
We took a nuclear option and did: `DELETE FROM cloud.domain WHERE removed IS NOT NULL;`
To remove all inactive domains.
This sorted out the issue above.
```
[root@cloudstack-mgmt ~]# time cmk list systemvms
......
real 0m0.231s
user 0m0.089s
sys 0m0.019s
[root@cloudstack-mgmt ~]# time cmk list networkofferings
........
real 0m0.272s
user 0m0.085s
sys 0m0.035s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] GET certain paths like systemVM and networkofferings are very very slow [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on issue #12740:
URL: https://github.com/apache/cloudstack/issues/12740#issuecomment-4334129424
@Sparc0 how many records were removed (approx.)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] GET certain paths like systemVM and networkofferings are very very slow [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on issue #12740:
URL: https://github.com/apache/cloudstack/issues/12740#issuecomment-4341465891
> Around ~52k domains was removed.
ok, something of a limit of the system it seems (needs validation) thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] GET certain paths like systemVM and networkofferings are very very slow [cloudstack]
Posted by "Sparc0 (via GitHub)" <gi...@apache.org>.
Sparc0 commented on issue #12740:
URL: https://github.com/apache/cloudstack/issues/12740#issuecomment-4006347990
I thought of the reproduce issue of like i was able to reproduce the issue on another management server.
But its not a one-time issue, always facing slowness when i try to access this two mentioned paths.
```
[root@cloudstack-mgmt ~]# time cmk list systemvms
......
real 0m32.227s
user 0m0.087s
sys 0m0.042s
[root@cloudstack-mgmt ~]# time cmk list networkofferings
........
real 1m4.528s
user 0m0.091s
sys 0m0.050s
```
Its even noticeable when visiting the dashboard page because the background call to get the systemvms count takes time, so after a while that number updates to 4.
<img width="762" height="382" alt="Image" src="https://github.com/user-attachments/assets/64e03edd-4647-4527-8207-d2d7ca724ed3" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] GET certain paths like systemVM and networkofferings are very very slow [cloudstack]
Posted by "Sparc0 (via GitHub)" <gi...@apache.org>.
Sparc0 commented on issue #12740:
URL: https://github.com/apache/cloudstack/issues/12740#issuecomment-4335918906
Around ~52k domains was removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] GET certain paths like systemVM and networkofferings are very very slow [cloudstack]
Posted by "shwstppr (via GitHub)" <gi...@apache.org>.
shwstppr commented on issue #12740:
URL: https://github.com/apache/cloudstack/issues/12740#issuecomment-4004484057
@Sparc0 You mention - "Not been able to reproduce the issue." so is it a one-time issue are you facing the slowness always?
If it is reproducible in your environment then can you please check the behaviour from API call done using cmk, something like, `time cmk list systemvms...`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org