You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by "poli1025 (via GitHub)" <gi...@apache.org> on 2025/07/03 21:26:36 UTC
[I] Degraded cloudstack agent [cloudstack]
poli1025 opened a new issue, #11141:
URL: https://github.com/apache/cloudstack/issues/11141
### problem
Over time, the CloudStack agent becomes increasingly slow when starting or stopping virtual machines. This performance degradation is especially noticeable during bulk VM operations, where the delays can become significant. The only effective workaround I've found so far is to restart the CloudStack agent service, which temporarily restores normal performance.
### versions
the version is 4.20.1
### The steps to reproduce the bug
1.
2.
3.
...
### What to do about it?
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland closed issue #11141: Degraded cloudstack agent
URL: https://github.com/apache/cloudstack/issues/11141
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "jgotteswinter (via GitHub)" <gi...@apache.org>.
jgotteswinter commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3502155756
@bhouse-nexthop i configured a daily cron which restarts the agent every night a few days ago. So far, it did not show about again.
@vgarcia-linube i also enabled debug on one of my compute nodes, with same result.
i was thinking about configuring jmx, maybe this could give more information?
`JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote=true"
JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.port=3344" # Choose an unused port, e.g., 3344
JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.ssl=false" # Enable SSL in production
JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.authenticate=false" # Add auth in production`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "jgotteswinter (via GitHub)" <gi...@apache.org>.
jgotteswinter commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3509756353
just as a note about my k8s statement. I dont think its k8s itself, the cloudstack csi driver (leaseweb one, in my case) is suspicious to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3576784402
> I tested libvirt (10.0.0 and 10.6.0) on two Ubuntu 24.04 nodes, and monitored some metrics every 10 seconds by commands
>
> ```
> pid=$(pgrep -f /usr/bin/java)
> echo "CPU: $(ps -p $pid -o %cpu --no-headers)%"
> echo "MEM: $(ps -p $pid -o %mem --no-headers)%"
> echo "FD: $(ls /proc/$pid/fd 2>/dev/null | wc -l)"
> echo "Threads: $(ps -L -p $pid -o tid --no-headers 2>/dev/null | wc -l)"
> echo "Conn: $(ss -tanp | grep "pid=$pid" | grep ESTAB | wc -l)"
> ```
>
> My observations are
>
> * There are very slight changes with memory, file descriptors, threads and connections of the java process, I think we can ignore.
> * The CPU usage of the java process increased gradually and slowly with libvirt 10.0.0
> * The CPU usage of the java process decreased gradually and became stable after some hours with libvirt 10.6.0
>
> Hostname libvirt version original CPU usage new CPU usage
> kvm1 10.0.0-2ubuntu8.9 0.9% 2.1% (after 12 hours), 2.8% (after 16 hours)
> kvm2 10.0.0-2ubuntu8.9 5.3% 5.9% (after 1 hour), 6.5% (after 2 hours)
> kvm2 10.6.0-1ubuntu3 4.9% 1.3% (after 1 hour), 0.6% (after 3 hours and 6 hours)
> kvm2 10.6.0-1ubuntu3.3 5.2% 1.2% (after 1 hour), 0.8% (after 2 hours)
> note:
>
> * no workloads inside VMs
> * no VMs actions
> * original CPU usage is fetched 5 mins after restarting agent
>
> I would suggest some of you test `libvirt 10.6.0-1ubuntu3.3` or `libvirt 10.6.0-1ubuntu3` on a non-production environment. I have uploaded the .deb packages to https://github.com/weizhouapache/ubuntu-libvirt-packages
Quick update:
- on kvm1 (with libvirt 10.0.0-2ubuntu8.9), CPU usage has increased to 7.5% after around 24 hours
- on kvm2 (with libvirt 10.6.0-1ubuntu3.3), CPU usage is still 0.6% after around 12 hours
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3570760617
so, it looks like everybody uses Ubuntu 24.04. Anybody uses other OS distros ?
I will try with libvirt 10.6.0 shipped with Ubuntu 24.10 release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3709306211
tnx for your feedback @bradh352 . This reduces the issue to an doc update for the compatability matrix I guess???
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "daviftorres (via GitHub)" <gi...@apache.org>.
daviftorres commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3705867716
> [@daviftorres](https://github.com/daviftorres) how is the bisection going?
>
> I'm thinking about just deploying 10.6 to our production environment then rolling back to whatever the proper solution ends up being. This is still hitting us pretty hard at our level of load ... a once a day restart just isn't enough.
Hey @bradh352, the bisection tests are still ongoing. Last time, likely because of the low workload in the non-prod environment, it took about 2–3 weeks before we saw CPU spikes significant enough to confidently determine whether there was an issue. I’m hoping to have solid metrics to share within the next week or so.
Regards,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "TadiosAbebe (via GitHub)" <gi...@apache.org>.
TadiosAbebe commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3569210929
@bradh352 , opening the console is also my first step when checking whether a compute host is experiencing issues. Over the past few weeks, I ran several tests across my five compute hosts (all using Ceph) to better understand the behavior.
### Initial Tests
These tests were performed a few days after restarting both cloudstack-agent and libvirtd.
| host | vm_create(in sec) | vm_stop(in sec) | vm_start(in sec) | vm_console(in sec) |
|---|---|---|---|---|
| host1 | 86 | 30 | 60 | 5|
| host2 | 37 | 18 | 28 | 4 |
| host3 | 54 | 9 | 33 | 1 |
| host4 | 49 | 3 | 19 | 0 |
| host5 | 12 | 3 | 19 | 0 |
### After Restarting libvirtd
I restarted libvirtd on all hosts and repeated the same tests.
| host | vm_create(in sec) | vm_stop(in sec) | vm_start(in sec) | vm_console(in sec) |
|---|---|---|---|---|
| host1 | 21 | 3 | 19 | 0|
| host2 | 22 | 3 | 19 | 0 |
| host3 | 25 | 3 | 19 | 0 |
| host4 | 21 | 7 | 19 | 0 |
| host5 | 22 | 3 | 18 | 0 |
### Same Tests Again (After ~2 Days)
| host | vm_create(in sec) | vm_stop(in sec) | vm_start(in sec) | vm_console(in sec) |
|---|---|---|---|---|
| host1 | 31 | 9 | 21 | 1|
| host2 | 28 | 3 | 18 | 0 |
| host3 | 28 | 6 | 23 | 1 |
| host4 | 30 | 3 | 19 | 0 |
| host5 | 26 | 3 | 18 | 0 |
These results may not be perfectly accurate due to different factors, but the performance differences are still notable especially the improvement immediately after restarting libvirtd.
My main concern with scheduling an automated restart of cloudstack-agent was, if it restarts in the middle of an operation, it may leave behind orphaned resources. I believe this is how I ended up with volumes stuck in Destroy state and a virtual router stuck in Expunging, which I’m still unable to fully remove.
It would be helpful to know whether restarting libvirtd is truly the reliable fix for you too. Even if that’s the case, I still don’t understand why this degradation happens over time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "daviftorres (via GitHub)" <gi...@apache.org>.
daviftorres commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3591800152
Hey @weizhouapache, you got it right. At least that is the conclusion I drew from what I saw (hopefully it does not mislead the investigation).
I will check which version of `libvirt` we are using and propose making changes based on your directions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3502208119
> we also have this kind of issue. After a few days the agent stops executing tasks. After restarting the agent it starts working again.
>
> * ACS currently 4.20.1.0, upgrade to 4.20.2.0 is in planning
> * Ubuntu 24.04
> * Primary Storage is provided by a Netapp NFS Share
> * 20-30 vms per host currently
>
> Is there any useful debugging data i can collect when this error occurs?
@jgotteswinter
can you share the number of storage pools on the hosts ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "bradh352 (via GitHub)" <gi...@apache.org>.
bradh352 commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3501869743
I wonder if this is the same issue I've been struggling with. That said, it seems at least in my case just restarting the agent isn't always enough, something gets 'hung up', like a lock, when the agent tries to connect to the management nodes which means sometimes I need to restart the management nodes as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "poli1025 (via GitHub)" <gi...@apache.org>.
poli1025 commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3129393152
We’ve identified that the issue is specifically related to the agent — there is no noticeable delay in the API calls.
We haven’t found any errors in the logs, but we are observing that the agent is progressively taking longer to perform the required actions.
The bulk operations are being executed via Terraform, such as the creation of 50 machines. The next day, when we destroy and recreate them, the process takes significantly more time. However, if we restart the agent, the performance returns to normal. Over time, the degradation happens again, and the creation time can eventually triple.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "bhouse-nexthop (via GitHub)" <gi...@apache.org>.
bhouse-nexthop commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3506636141
> > [@vgarcia-linube](https://github.com/vgarcia-linube) on the settings that helped you said:
> > > limit the number of threads of the agent and try to read the usage metrics in longer time spans (I think it defaults to 10 minutes or something like that, setting it to once every two hours mitigates it a bit, just enough so you don't have to restart the agent every few hours so it doesn't hog the kvm node cpu
> >
> >
> > Can you provide the exact setting names (and where, such as agent.properties or something in management config) and values you used that helped? I'd like to see if that helps me.
> > Also does preemptively restarting the agent before it gets too bad in something like cron.daily prevent the issue from occurring?
>
> could check `stats.interval` in global settings
@vgarcia-linube can you elaborate on which stats you changed? There are a lot of `stats.interval` settings in the global config, so I'm thinking you didnt' change all of those did you? "Usage metrics" is a bit generic to know what you are referring to. You did mention ones that default to 10 minute time spans, the only one I see that matches that is `volume.stats.interval`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "cpaelzer (via GitHub)" <gi...@apache.org>.
cpaelzer commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3636547048
> In any case, I open a bug report with Canonical to bring them awareness of the symptoms we are experience because it could be part of a bigger issue.
>
> https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/2133531
Thank you for that report @daviftorres
Unless the actual change totally surprises us I agree to @weizhouapache who said:
> It would be best if Canonical backports the fix (if we can find) to libvirt 10.0.0, test it and include it in the next libvirt minor version
But that means we need to find the fix that made it work in 10.6.
I guess since we lack a good hint to spot the change by going through commits we have to rely on bisecting.
For that we miss a simple reproducer, but that only means I'd need your participation and it is a bit slower.
In [the bug that @daviftorres reported](https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/2133531/comments/2) I provided PPA builds for 24.04 noble of libvirt 10.0, 10.2, 10.4 and 10.6 which we'd need tests for, to then slowly grind our way down.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "marekhorecny (via GitHub)" <gi...@apache.org>.
marekhorecny commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3704924364
Hello
I installed the new version of libvirt (libvirt 10.6.0-1ubuntu3.3 ) in the production environment at the end of the year, just before midnight. It's only been a short time since then, just 35 hours, but so far everything seems to be fine.
- Since installation, there have been no alerts about problems with failing checks on the router.
- Agents have CPU usage from 0.something% to 1.5%.
- No slowdowns or timeouts with VM operations (creation, migration, console, or whatever).
I'll check back in about a week or two and let you know the results. By then, the bug should show up if it still exists.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3713344568
> [@DaanHoogland](https://github.com/DaanHoogland) why tempted?
shear prejudice
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "poli1025 (via GitHub)" <gi...@apache.org>.
poli1025 commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3242473709
@weizhouapache The initial deployments and agents work well. However, after a few days we observe degradation: VM creation, power cycling, and console access start to slow down.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "daviftorres (via GitHub)" <gi...@apache.org>.
daviftorres commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3597623682
Hey @weizhouapache,
How did you managed to get `libvirt 10.6.0-1ubuntu3.3` on Ubuntu 24.04 LTS? Or you are using Ubuntu 24.10 or newer?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "daviftorres (via GitHub)" <gi...@apache.org>.
daviftorres commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3597919806
Thank you. I will test in on Non-Prod envs but I am afraid I cannot ever push it into Prod because we will be out of the LTS support. Unless there is absolutely no other choice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "jgotteswinter (via GitHub)" <gi...@apache.org>.
jgotteswinter commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3710798778
I guess you can install those on debian as well, its just a java service without lots of dependencies. I dont know how widespread debian is as platform for acs, this might need testing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "jgotteswinter (via GitHub)" <gi...@apache.org>.
jgotteswinter commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3682546139
I am sticking with the 24hr restart of the agent, for me the problem disappeared. Backports have to much disadvantages imho.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "bhouse-nexthop (via GitHub)" <gi...@apache.org>.
bhouse-nexthop commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3710789992
Cloudstack doesn't really advertise Debian as a first class citizen of a distro as it doesn't list community packages: https://cloudstack.apache.org/downloads/#community-packages
Interestingly, I am at least seeing bookworm (debian 12) packages here:
https://download.cloudstack.org/debian/dists/bookworm/
It would be good to get trixie in there and make them more official by referencing them in the download page. I probably would have gone Debian on a deployment if I thought it was supported at the same level as Ubuntu.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "TadiosAbebe (via GitHub)" <gi...@apache.org>.
TadiosAbebe commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3579480031
> But I had a test all-in-one ACS on ubuntu 24.04 with libvirt 10.0.0, but I couldn’t reproduce the issue I’m seeing in the production environment. I repeatedly ran your test script:
>
> ```
> for i in `seq 1 20`;do
> cmk deploy virtualmachine name=L2-wei-test-$i serviceofferingid=xxx zoneid=xxx templateid=xxx networkids=xxx & >/dev/null;
> sleep 2;
> done
> ```
>
> to generate load, and the results were consistently fast:
>
> ```
> mysql> select id,name,created,update_time,(update_time-created) from vm_instance where removed is null and name like "L2-wei%";
> +-----+----------------+---------------------+---------------------+-----------------------+
> | id | name | created | update_time | (update_time-created) |
> +-----+----------------+---------------------+---------------------+-----------------------+
> | 191 | L2-wei-test-1 | 2025-11-25 11:22:07 | 2025-11-25 11:22:14 | 7 |
> | 192 | L2-wei-test-2 | 2025-11-25 11:22:09 | 2025-11-25 11:22:16 | 7 |
> | 193 | L2-wei-test-3 | 2025-11-25 11:22:11 | 2025-11-25 11:22:17 | 6 |
> | 194 | L2-wei-test-4 | 2025-11-25 11:22:13 | 2025-11-25 11:22:22 | 9 |
> | 195 | L2-wei-test-5 | 2025-11-25 11:22:15 | 2025-11-25 11:22:20 | 5 |
> | 196 | L2-wei-test-6 | 2025-11-25 11:22:17 | 2025-11-25 11:22:23 | 6 |
> | 197 | L2-wei-test-7 | 2025-11-25 11:22:19 | 2025-11-25 11:22:26 | 7 |
> | 198 | L2-wei-test-8 | 2025-11-25 11:22:21 | 2025-11-25 11:22:27 | 6 |
> | 199 | L2-wei-test-9 | 2025-11-25 11:22:23 | 2025-11-25 11:22:29 | 6 |
> | 200 | L2-wei-test-10 | 2025-11-25 11:22:25 | 2025-11-25 11:22:31 | 6 |
> | 201 | L2-wei-test-11 | 2025-11-25 11:22:27 | 2025-11-25 11:22:34 | 7 |
> | 202 | L2-wei-test-12 | 2025-11-25 11:22:29 | 2025-11-25 11:22:36 | 7 |
> | 203 | L2-wei-test-13 | 2025-11-25 11:22:31 | 2025-11-25 11:22:38 | 7 |
> | 204 | L2-wei-test-14 | 2025-11-25 11:22:33 | 2025-11-25 11:22:41 | 8 |
> | 205 | L2-wei-test-15 | 2025-11-25 11:22:35 | 2025-11-25 11:22:42 | 7 |
> | 206 | L2-wei-test-16 | 2025-11-25 11:22:37 | 2025-11-25 11:22:45 | 8 |
> | 207 | L2-wei-test-17 | 2025-11-25 11:22:39 | 2025-11-25 11:22:48 | 9 |
> | 208 | L2-wei-test-18 | 2025-11-25 11:22:41 | 2025-11-25 11:22:49 | 8 |
> | 209 | L2-wei-test-19 | 2025-11-25 11:22:43 | 2025-11-25 11:22:51 | 8 |
> | 210 | L2-wei-test-20 | 2025-11-25 11:22:45 | 2025-11-25 11:22:55 | 10 |
> +-----+----------------+---------------------+---------------------+-----------------------+
> ```
>
> I'll try to create a full environment consisting ceph and multiple kvm host on a test environment to see if i can replicate the issue there and see if libvirt 10.6.0 fix it, later this week.
My update:
After yesterdays test i let the all-in-one ACS sit without restarting libvirtd and cloudstack-agent for a while now, initially the resource utilization of the java process was
```
CPU: 0.5%
MEM: 2.4%
FD: 253
Threads: 83
Conn: 1
```
After about 6 or 7 hours, got up to about CPU: 1.1% without any interaction on the host, just running the above 20 small cirros VMs previously launched. Then i tried to simulate some workload in a loop creating and destroying those 20 instances. The resource CPU utilization increased to about 4.5% after about 10 hours
```
CPU: 4.5%
MEM: 2.6%
FD: 253
Threads: 78
Conn: 1
```
One thing i noticed when looking into the java process in htop is, 5 or 6 java process in our production cluster have a high CPU time for example on one of our compute host about 4 process have a CPU time value of around 53:54:30. Is this normal?
If i restart libvirtd and the CPU time of the java process those process with high CPU time goes away and the max becomes 0:05:26
I'll replace the libvirt with 10.6.0 on the all-in-one instance and see what changes now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "daviftorres (via GitHub)" <gi...@apache.org>.
daviftorres commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3655892005
> > Hey folks,
> > I just wanted to share my findings since I am tracking CPU utilization of the Java process of the Agents every 15 seconds.
>
> [@daviftorres](https://github.com/daviftorres) did you say that the agents were in Up state, but did not connect to the management servers ?
>
> btw, I just checked the CPU usage of my ubuntu 24 hosts (each has 25-27 running vms)
>
> * kvm1 with `libvirt 10.0.0-2ubuntu8.9`: **72.7%**
> * kvm2 with `libvirt 10.6.0-1ubuntu3.3`: **0.4%**
>
> I truly advise you to test libvirt 10.6.0 [@daviftorres](https://github.com/daviftorres)
Hey @weizhouapache ,
Have you tried to run the bisections that @cpaelzer staged the PPAs?
I did as I mentioned above and I could not make it so manifest the symptoms. I even created a script to deploy and destroy 50 VMs every to minutes but the CPU increase is not expressive:
<img width="495" height="360" alt="Image" src="https://github.com/user-attachments/assets/1547ebe0-3085-49f7-92c1-2d845bab28bc" />
What operations did you use to induce some the bug?
Regards,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3240994641
I created 50 vm instances in parallel, the deployments and agents worked well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "vgarcia-linube (via GitHub)" <gi...@apache.org>.
vgarcia-linube commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3509705817
The ones that seemed to do the trick were:
In cloudstack agent config: /etc/cloudstack/agent/agent.properties: workers -> 3
In global config in Cloudstack itself: storage.stats.interval: 120000
The more workers configured in cloudstack-agent, the faster the error showed up. Our clusters handle about ~600 vms, currently a mix of standalone instances and kubernetes nodes, but this happened long before we had any kubernetes cluster running
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "TadiosAbebe (via GitHub)" <gi...@apache.org>.
TadiosAbebe commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3570006079
> thanks for the information [@bradh352](https://github.com/bradh352) [@TadiosAbebe](https://github.com/TadiosAbebe) [@jgotteswinter](https://github.com/jgotteswinter)
>
> I suspected it is due to that all libvirt operations use the same instance of `LibvirtConnection`. a libvirt operation might delay other libvirt operations. we could introduce a thread pool of `LibvirtConnection` if it supports concurrent operations.
>
> however, I do not know why the issue became very often in recent releases. can everyone share the below ?
>
> * global configurations (`cmk list configurations filter=id,name,value keyword=stats.interval`)
> * java version of kvm host (`java --version`)
> * OS distribution of kvm host (`cat /etc/os-release`)
> * libvirt and qemu version of kvm host (`virsh version`)
> * Approx. number of VMs on kvm host
```
() 🐱 > list configurations filter=id,name,value keyword=stats.interval
{
"configuration": [
{
"name": "autoscale.stats.interval",
"value": "60"
},
{
"name": "cluster.heartbeat.threshold",
"value": "150000"
},
{
"name": "database.server.stats.interval",
"value": "60"
},
{
"name": "database.server.stats.retention",
"value": "3"
},
{
"name": "direct.network.stats.interval",
"value": "86400"
},
{
"name": "external.network.stats.interval",
"value": "300"
},
{
"name": "host.stats.interval",
"value": "60000"
},
{
"name": "management.server.stats.interval",
"value": "60"
},
{
"name": "router.stats.interval",
"value": "300"
},
{
"name": "storage.stats.interval",
"value": "60000"
},
{
"name": "storpool.storage.stats.interval",
"value": "3600"
},
{
"name": "storpool.volumes.stats.interval",
"value": "3600"
},
{
"name": "vm.disk.stats.interval",
"value": "0"
},
{
"name": "vm.disk.stats.interval.min",
"value": "300"
},
{
"name": "vm.network.stats.interval",
"value": "0"
},
{
"name": "vm.network.stats.interval.min",
"value": "300"
},
{
"name": "vm.stats.interval",
"value": "60000"
},
{
"name": "volume.stats.interval",
"value": "600000"
}
],
"count": 18
}
```
```
root@host1:/home/zgadmin# java --version
openjdk 17.0.16 2025-07-15
OpenJDK Runtime Environment (build 17.0.16+8-Ubuntu-0ubuntu124.04.1)
OpenJDK 64-Bit Server VM (build 17.0.16+8-Ubuntu-0ubuntu124.04.1, mixed mode, sharing)
```
```
root@host1:/home/zgadmin# cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.2 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo
```
```
root@host1:/home/zgadmin# virsh version
Compiled against library: libvirt 10.0.0
Using library: libvirt 10.0.0
Using API: QEMU 10.0.0
Running hypervisor: QEMU 8.2.2
```
```
Approx. number of VMs per kVM host is in beteen 8 to 10 for me.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "bradh352 (via GitHub)" <gi...@apache.org>.
bradh352 commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3570459420
```
root@hv101:~# cmk list configurations filter=id,name,value keyword=stats.interval
{
"configuration": [
{
"name": "autoscale.stats.interval",
"value": "60"
},
{
"name": "cluster.heartbeat.threshold",
"value": "150000"
},
{
"name": "database.server.stats.interval",
"value": "60"
},
{
"name": "database.server.stats.retention",
"value": "3"
},
{
"name": "direct.network.stats.interval",
"value": "86400"
},
{
"name": "external.network.stats.interval",
"value": "300"
},
{
"name": "host.stats.interval",
"value": "60000"
},
{
"name": "management.server.stats.interval",
"value": "60"
},
{
"name": "router.stats.interval",
"value": "300"
},
{
"name": "storage.stats.interval",
"value": "600000"
},
{
"name": "storpool.storage.stats.interval",
"value": "3600"
},
{
"name": "storpool.volumes.stats.interval",
"value": "3600"
},
{
"name": "vm.disk.stats.interval",
"value": "0"
},
{
"name": "vm.disk.stats.interval.min",
"value": "300"
},
{
"name": "vm.network.stats.interval",
"value": "0"
},
{
"name": "vm.network.stats.interval.min",
"value": "300"
},
{
"name": "vm.stats.interval",
"value": "60000"
},
{
"name": "volume.stats.interval",
"value": "600000"
}
],
"count": 18
}
```
```
root@hv101:~# java --version
openjdk 17.0.16 2025-07-15
OpenJDK Runtime Environment (build 17.0.16+8-Ubuntu-0ubuntu124.04.1)
OpenJDK 64-Bit Server VM (build 17.0.16+8-Ubuntu-0ubuntu124.04.1, mixed mode, sharing)
```
```
root@hv101:~# cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.3 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo
```
```
root@hv101:~# virsh version
Compiled against library: libvirt 10.0.0
Using library: libvirt 10.0.0
Using API: QEMU 10.0.0
Running hypervisor: QEMU 8.2.2
```
VMs per host: 30-40
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3579913262
>
> One thing i noticed when looking into the java process in htop is, 5 or 6 java process in our production cluster have a high CPU time for example on one of our compute host about 4 process have a CPU time value of around 53:54:30. Is this normal?
>
cool, thanks for sharing @TadiosAbebe
I just checked my kvm host with libvirt 10.0.0 (running 36 hours without touch)
- the CPU usage of java process has increased to 30.6%
- It took 0.5 seconds to process GetVncPortCommand and return GetVncPortAnswer
- There is indeed a java thread which CPU time is 40 hours (41:00:93)
On the kvm host with libvirt 10.6.0 (running 24 hours without touch)
- the CPU usage of java process is still 0.5%
- It took 0.004 seconds to process GetVncPortCommand and return GetVncPortAnswer
- The most CPU time of java threads is 1 hour (1:08:12)
> If i restart libvirtd and the CPU time of the java process those process with high CPU time goes away and the max becomes 0:05:26
>
> I'll replace the libvirt with 10.6.0 on the all-in-one instance and see what changes now.
I believe libvirt 10.6.0 is a worthwhile version to test, even though its official packages target Ubuntu 24.10 rather than Ubuntu 24.04.
cc @poli1025 @shwstppr @marekhorecny @vgarcia-linube @jgotteswinter @bradh352 @SviridoffA @TadiosAbebe
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "daviftorres (via GitHub)" <gi...@apache.org>.
daviftorres commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3591819812
Confirmed, we are using the version of `libvirt` you suspected is faulty:
<img width="535" height="313" alt="Image" src="https://github.com/user-attachments/assets/a0e2bb1a-1ca2-4bd2-bd21-a818e38d91ec" />
I will create a CR to upgrade one of our Zones and keep you posted of the results.
Thank you for your good work!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "daviftorres (via GitHub)" <gi...@apache.org>.
daviftorres commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3590066014
Hey folks,
I just wanted to share my findings since I am tracking CPU utilization of the Java process of the Agents every 15 seconds.
<img width="1888" height="770" alt="Image" src="https://github.com/user-attachments/assets/f490f476-778d-4b07-a48c-319e3d6b5224" />
It confirms what you guys are reporting that the CPU utilization gradually grows and each drop is because I had to act and restart the service manually.
Today I realized that I could see the number of connected agents by toggling Metrics.
<img width="1763" height="430" alt="Image" src="https://github.com/user-attachments/assets/6faf986f-b68f-48cd-bfc6-585928e95690" />
For my surprise, it shows only 8 connected Agents. When, in fact I have 13 active KVM Hosts.
<img width="1345" height="837" alt="Image" src="https://github.com/user-attachments/assets/75561bef-53df-4edc-ae91-aeb101a6d3be" />
Then, after restarting the four Agents with highest CPU usage, I got them connect back to the MS.
<img width="1754" height="432" alt="Image" src="https://github.com/user-attachments/assets/490fd25f-6117-4edf-bea8-56e72ca4f9da" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "jgotteswinter (via GitHub)" <gi...@apache.org>.
jgotteswinter commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3585118126
The main issue with 10.6.0 is that it needs to be maintained as backport. This shouldnt be a big deal but it needs to be handled properly for production
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "daviftorres (via GitHub)" <gi...@apache.org>.
daviftorres commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3756550107
> hmm, the changeset between 10.4 and 10.6 isn't exactly small, don't know how easy it would be to identify the actual commit that fixes this: [libvirt/libvirt@v10.4.0...v10.6.0](https://github.com/libvirt/libvirt/compare/v10.4.0...v10.6.0)
So far, the plan is to dissect even more. I have increased my testing environment from 3 to 5 KVM hosts and I have 2 more on the way. This might speed up the process to identify when the change that fixed the bug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3569587859
thanks for the information @bradh352 @TadiosAbebe @jgotteswinter
I suspected it is due to that all libvirt operations use the same instance of `LibvirtConnection`. a libvirt operation might delay other libvirt operations. we could introduce a thread pool of `LibvirtConnection` if it supports concurrent operations.
however, I do not know why the issue became very often in recent releases. can everyone share the below ?
- global configurations (`cmk list configurations filter=id,name,value keyword=stats.interval`)
- java version of kvm host (`java --version`)
- OS distribution of kvm host (`cat /etc/os-release`)
- libvirt and qemu version of kvm host (`virsh version`)
- Approx. number of VMs on kvm host
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "SviridoffA (via GitHub)" <gi...@apache.org>.
SviridoffA commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3570693983
Hi Everyone!! I'm from sub-issue 😉
```
{
"configuration": [
{
"name": "autoscale.stats.interval",
"value": "60"
},
{
"name": "cluster.heartbeat.threshold",
"value": "150000"
},
{
"name": "database.server.stats.interval",
"value": "60"
},
{
"name": "database.server.stats.retention",
"value": "3"
},
{
"name": "direct.network.stats.interval",
"value": "86400"
},
{
"name": "external.network.stats.interval",
"value": "300"
},
{
"name": "host.stats.interval",
"value": "60000"
},
{
"name": "management.server.stats.interval",
"value": "60"
},
{
"name": "router.stats.interval",
"value": "300"
},
{
"name": "storage.stats.interval",
"value": "60000"
},
{
"name": "storpool.storage.stats.interval",
"value": "3600"
},
{
"name": "storpool.volumes.stats.interval",
"value": "3600"
},
{
"name": "vm.disk.stats.interval",
"value": "0"
},
{
"name": "vm.disk.stats.interval.min",
"value": "300"
},
{
"name": "vm.network.stats.interval",
"value": "0"
},
{
"name": "vm.network.stats.interval.min",
"value": "300"
},
{
"name": "vm.stats.interval",
"value": "60000"
},
{
"name": "volume.stats.interval",
"value": "600000"
}
],
"count": 18
```
```
openjdk 21.0.8 2025-07-15
OpenJDK Runtime Environment (build 21.0.8+9-Ubuntu-0ubuntu124.04.1)
OpenJDK 64-Bit Server VM (build 21.0.8+9-Ubuntu-0ubuntu124.04.1, mixed mode, sharing)
```
21 I think its because of Linstor, I'm usin Linstor as Primary (hyperconverged), on management I have 17.0.16
```
PRETTY_NAME="Ubuntu 24.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.3 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo
```
```
Compiled against library: libvirt 10.0.0
Using library: libvirt 10.0.0
Using API: QEMU 10.0.0
Running hypervisor: QEMU 8.2.2
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3570106110
I tested 50 vms deployment on ol9 and u24, it seems the issue is with ubuntu 24
```
for i in `seq 1 50`;do
cmk deploy virtualmachine name=L2-wei-test-$i serviceofferingid=xxx zoneid=xxx templateid=xxx networkids=xxx & >/dev/null;
sleep 2;
done
```
- ol9 works without issue
ol9, ACS 4.20.2.0
```
Compiled against library: libvirt 10.10.0
Using library: libvirt 10.10.0
Using API: QEMU 10.10.0
Running hypervisor: QEMU 9.1.0
```
vm creation time:
```
mysql> select id,name,created,update_time,(update_time-created) from vm_instance where removed is null and name like "L2-wei%";
+-----+----------------+---------------------+---------------------+-----------------------+
| id | name | created | update_time | (update_time-created) |
+-----+----------------+---------------------+---------------------+-----------------------+
| 496 | L2-wei-test-1 | 2025-11-24 09:56:59 | 2025-11-24 09:57:05 | 46 |
| 497 | L2-wei-test-2 | 2025-11-24 09:57:01 | 2025-11-24 09:57:07 | 6 |
| 498 | L2-wei-test-3 | 2025-11-24 09:57:03 | 2025-11-24 09:57:12 | 9 |
| 499 | L2-wei-test-4 | 2025-11-24 09:57:05 | 2025-11-24 09:57:10 | 5 |
| 500 | L2-wei-test-5 | 2025-11-24 09:57:07 | 2025-11-24 09:57:13 | 6 |
| 501 | L2-wei-test-6 | 2025-11-24 09:57:09 | 2025-11-24 09:57:14 | 5 |
| 502 | L2-wei-test-7 | 2025-11-24 09:57:11 | 2025-11-24 09:57:23 | 12 |
| 503 | L2-wei-test-8 | 2025-11-24 09:57:13 | 2025-11-24 09:57:21 | 8 |
| 504 | L2-wei-test-9 | 2025-11-24 09:57:15 | 2025-11-24 09:57:40 | 25 |
| 505 | L2-wei-test-10 | 2025-11-24 09:57:17 | 2025-11-24 09:57:29 | 12 |
| 506 | L2-wei-test-11 | 2025-11-24 09:57:19 | 2025-11-24 09:57:24 | 5 |
| 507 | L2-wei-test-12 | 2025-11-24 09:57:21 | 2025-11-24 09:57:45 | 24 |
| 508 | L2-wei-test-13 | 2025-11-24 09:57:23 | 2025-11-24 09:57:31 | 8 |
| 509 | L2-wei-test-14 | 2025-11-24 09:57:25 | 2025-11-24 09:57:39 | 14 |
| 510 | L2-wei-test-15 | 2025-11-24 09:57:27 | 2025-11-24 09:58:11 | 84 |
| 511 | L2-wei-test-16 | 2025-11-24 09:57:29 | 2025-11-24 09:57:35 | 6 |
| 512 | L2-wei-test-17 | 2025-11-24 09:57:31 | 2025-11-24 09:57:38 | 7 |
| 513 | L2-wei-test-18 | 2025-11-24 09:57:33 | 2025-11-24 09:57:40 | 7 |
| 514 | L2-wei-test-19 | 2025-11-24 09:57:35 | 2025-11-24 09:57:54 | 19 |
| 515 | L2-wei-test-20 | 2025-11-24 09:57:37 | 2025-11-24 09:57:46 | 9 |
| 516 | L2-wei-test-21 | 2025-11-24 09:57:39 | 2025-11-24 09:57:48 | 9 |
| 517 | L2-wei-test-22 | 2025-11-24 09:57:41 | 2025-11-24 09:57:53 | 12 |
| 518 | L2-wei-test-23 | 2025-11-24 09:57:43 | 2025-11-24 09:57:48 | 5 |
| 519 | L2-wei-test-24 | 2025-11-24 09:57:45 | 2025-11-24 09:58:14 | 69 |
| 520 | L2-wei-test-25 | 2025-11-24 09:57:47 | 2025-11-24 09:57:54 | 7 |
| 521 | L2-wei-test-26 | 2025-11-24 09:57:49 | 2025-11-24 09:58:14 | 65 |
| 522 | L2-wei-test-27 | 2025-11-24 09:57:51 | 2025-11-24 09:58:05 | 54 |
| 523 | L2-wei-test-28 | 2025-11-24 09:57:53 | 2025-11-24 09:58:02 | 49 |
| 524 | L2-wei-test-29 | 2025-11-24 09:57:55 | 2025-11-24 09:58:25 | 70 |
| 525 | L2-wei-test-30 | 2025-11-24 09:57:57 | 2025-11-24 09:58:03 | 46 |
| 526 | L2-wei-test-31 | 2025-11-24 09:57:59 | 2025-11-24 09:58:13 | 54 |
| 527 | L2-wei-test-32 | 2025-11-24 09:58:01 | 2025-11-24 09:58:07 | 6 |
| 528 | L2-wei-test-33 | 2025-11-24 09:58:03 | 2025-11-24 09:58:13 | 10 |
| 529 | L2-wei-test-34 | 2025-11-24 09:58:05 | 2025-11-24 09:58:31 | 26 |
| 530 | L2-wei-test-35 | 2025-11-24 09:58:07 | 2025-11-24 09:58:17 | 10 |
| 531 | L2-wei-test-36 | 2025-11-24 09:58:09 | 2025-11-24 09:58:24 | 15 |
| 532 | L2-wei-test-37 | 2025-11-24 09:58:11 | 2025-11-24 09:58:53 | 42 |
| 533 | L2-wei-test-38 | 2025-11-24 09:58:13 | 2025-11-24 09:58:37 | 24 |
| 534 | L2-wei-test-39 | 2025-11-24 09:58:15 | 2025-11-24 09:58:23 | 8 |
| 535 | L2-wei-test-40 | 2025-11-24 09:58:17 | 2025-11-24 09:58:42 | 25 |
| 536 | L2-wei-test-41 | 2025-11-24 09:58:19 | 2025-11-24 09:58:28 | 9 |
| 537 | L2-wei-test-42 | 2025-11-24 09:58:21 | 2025-11-24 09:58:32 | 11 |
| 538 | L2-wei-test-43 | 2025-11-24 09:58:23 | 2025-11-24 09:58:51 | 28 |
| 539 | L2-wei-test-44 | 2025-11-24 09:58:25 | 2025-11-24 09:58:32 | 7 |
| 540 | L2-wei-test-45 | 2025-11-24 09:58:27 | 2025-11-24 09:58:39 | 12 |
| 541 | L2-wei-test-46 | 2025-11-24 09:58:29 | 2025-11-24 09:58:53 | 24 |
| 542 | L2-wei-test-47 | 2025-11-24 09:58:31 | 2025-11-24 09:58:43 | 12 |
| 543 | L2-wei-test-48 | 2025-11-24 09:58:33 | 2025-11-24 09:58:54 | 21 |
| 544 | L2-wei-test-49 | 2025-11-24 09:58:35 | 2025-11-24 09:59:13 | 78 |
| 545 | L2-wei-test-50 | 2025-11-24 09:58:37 | 2025-11-24 09:58:46 | 9 |
+-----+----------------+---------------------+---------------------+-----------------------+
50 rows in set (0.00 sec)
```
- ubuntu 24:
```
Compiled against library: libvirt 10.0.0
Using library: libvirt 10.0.0
Using API: QEMU 10.0.0
Running hypervisor: QEMU 8.2.2
```
vm creation time:
```
mysql> select id,name,created,update_time,(update_time-created) from vm_instance where removed is null and name like "L2-wei%";
+----+----------------+---------------------+---------------------+-----------------------+
| id | name | created | update_time | (update_time-created) |
+----+----------------+---------------------+---------------------+-----------------------+
| 6 | L2-wei-test-1 | 2025-11-24 10:01:40 | 2025-11-24 10:06:12 | 472 |
| 7 | L2-wei-test-2 | 2025-11-24 10:01:42 | 2025-11-24 10:07:41 | 599 |
| 8 | L2-wei-test-3 | 2025-11-24 10:01:44 | 2025-11-24 10:28:54 | 2710 |
| 9 | L2-wei-test-4 | 2025-11-24 10:01:46 | 2025-11-24 10:16:49 | 1503 |
| 10 | L2-wei-test-5 | 2025-11-24 10:01:48 | 2025-11-24 10:17:09 | 1561 |
| 11 | L2-wei-test-6 | 2025-11-24 10:01:50 | 2025-11-24 10:01:50 | 0 |
| 12 | L2-wei-test-7 | 2025-11-24 10:01:52 | 2025-11-24 10:11:00 | 948 |
| 13 | L2-wei-test-8 | 2025-11-24 10:01:54 | 2025-11-24 10:22:26 | 2072 |
| 14 | L2-wei-test-9 | 2025-11-24 10:01:56 | 2025-11-24 10:14:17 | 1261 |
| 15 | L2-wei-test-10 | 2025-11-24 10:01:58 | 2025-11-24 10:10:10 | 852 |
| 16 | L2-wei-test-11 | 2025-11-24 10:02:00 | 2025-11-24 10:17:10 | 1510 |
| 17 | L2-wei-test-12 | 2025-11-24 10:02:02 | 2025-11-24 10:30:18 | 2816 |
| 18 | L2-wei-test-13 | 2025-11-24 10:02:04 | 2025-11-24 10:09:12 | 708 |
| 19 | L2-wei-test-14 | 2025-11-24 10:02:06 | 2025-11-24 10:17:14 | 1508 |
| 20 | L2-wei-test-15 | 2025-11-24 10:02:08 | 2025-11-24 10:12:20 | 1012 |
| 21 | L2-wei-test-16 | 2025-11-24 10:02:10 | 2025-11-24 10:29:40 | 2730 |
| 22 | L2-wei-test-17 | 2025-11-24 10:02:12 | 2025-11-24 10:08:56 | 644 |
| 23 | L2-wei-test-18 | 2025-11-24 10:02:14 | 2025-11-24 10:17:19 | 1505 |
| 24 | L2-wei-test-19 | 2025-11-24 10:02:16 | 2025-11-24 10:17:21 | 1505 |
| 25 | L2-wei-test-20 | 2025-11-24 10:02:18 | 2025-11-24 10:14:46 | 1228 |
| 26 | L2-wei-test-21 | 2025-11-24 10:02:20 | 2025-11-24 10:24:09 | 2189 |
| 27 | L2-wei-test-22 | 2025-11-24 10:02:22 | 2025-11-24 10:29:35 | 2713 |
| 28 | L2-wei-test-23 | 2025-11-24 10:02:24 | 2025-11-24 10:03:07 | 83 |
| 29 | L2-wei-test-24 | 2025-11-24 10:02:26 | 2025-11-24 10:24:04 | 2178 |
| 30 | L2-wei-test-25 | 2025-11-24 10:02:28 | 2025-11-24 10:25:35 | 2307 |
| 31 | L2-wei-test-26 | 2025-11-24 10:02:30 | 2025-11-24 10:03:17 | 87 |
| 32 | L2-wei-test-27 | 2025-11-24 10:02:32 | 2025-11-24 10:24:47 | 2215 |
| 33 | L2-wei-test-28 | 2025-11-24 10:02:34 | 2025-11-24 10:17:40 | 1506 |
| 34 | L2-wei-test-29 | 2025-11-24 10:02:36 | 2025-11-24 10:30:23 | 2787 |
| 35 | L2-wei-test-30 | 2025-11-24 10:02:38 | 2025-11-24 10:02:46 | 8 |
| 36 | L2-wei-test-31 | 2025-11-24 10:02:40 | 2025-11-24 10:30:32 | 2792 |
| 37 | L2-wei-test-32 | 2025-11-24 10:02:42 | 2025-11-24 10:30:19 | 2777 |
| 38 | L2-wei-test-33 | 2025-11-24 10:02:44 | 2025-11-24 10:18:11 | 1567 |
| 39 | L2-wei-test-34 | 2025-11-24 10:02:46 | 2025-11-24 10:03:37 | 91 |
| 40 | L2-wei-test-35 | 2025-11-24 10:02:48 | 2025-11-24 10:17:56 | 1508 |
| 41 | L2-wei-test-36 | 2025-11-24 10:02:50 | 2025-11-24 10:29:49 | 2699 |
| 42 | L2-wei-test-37 | 2025-11-24 10:02:52 | 2025-11-24 10:30:04 | 2752 |
| 43 | L2-wei-test-38 | 2025-11-24 10:02:54 | 2025-11-24 10:10:06 | 752 |
| 44 | L2-wei-test-39 | 2025-11-24 10:02:56 | 2025-11-24 10:18:07 | 1551 |
| 45 | L2-wei-test-40 | 2025-11-24 10:02:58 | 2025-11-24 10:11:43 | 885 |
| 46 | L2-wei-test-41 | 2025-11-24 10:03:00 | 2025-11-24 10:03:49 | 49 |
| 47 | L2-wei-test-42 | 2025-11-24 10:03:02 | 2025-11-24 10:30:35 | 2733 |
| 48 | L2-wei-test-43 | 2025-11-24 10:03:04 | 2025-11-24 10:30:01 | 2697 |
| 49 | L2-wei-test-44 | 2025-11-24 10:03:06 | 2025-11-24 10:10:55 | 749 |
| 50 | L2-wei-test-45 | 2025-11-24 10:03:08 | 2025-11-24 10:12:58 | 950 |
| 51 | L2-wei-test-46 | 2025-11-24 10:03:10 | 2025-11-24 10:14:49 | 1139 |
| 52 | L2-wei-test-47 | 2025-11-24 10:03:12 | 2025-11-24 10:30:35 | 2723 |
| 53 | L2-wei-test-48 | 2025-11-24 10:03:14 | 2025-11-24 10:30:28 | 2714 |
| 54 | L2-wei-test-49 | 2025-11-24 10:03:16 | 2025-11-24 10:29:43 | 2627 |
| 55 | L2-wei-test-50 | 2025-11-24 10:03:18 | 2025-11-24 10:26:58 | 2340 |
+----+----------------+---------------------+---------------------+-----------------------+
50 rows in set (0.00 sec)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "shwstppr (via GitHub)" <gi...@apache.org>.
shwstppr commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3569708444
thanks @jgotteswinter cc @weizhouapache
I did take a quick look at the thread dump you shared. I'm seeing some hot/suspected threads (Roughly 2.1% of one core on average over the entire uptime),
```
"AgentOutRequest-Handler-2" #88 prio=5 os_prio=0 cpu=20647388.02ms elapsed=949378.81s tid=0x00007cc18814b210 nid=0xe5e2d runnable [0x00007cc0131f0000]
java.lang.Thread.State: RUNNABLE
at com.sun.jna.Native.invokePointer(Native Method)
at com.sun.jna.Function.invokePointer(Function.java:497)
at com.sun.jna.Function.invoke(Function.java:441)
at com.sun.jna.Function.invoke(Function.java:361)
at com.sun.jna.Library$Handler.invoke(Library.java:265)
at jdk.proxy2.$Proxy17.virConnectGetHostname(jdk.proxy2/Unknown Source)
at org.libvirt.Connect.getHostName(Unknown Source)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.getHostVmStateReport(LibvirtComputingResource.java:4050)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.getHostVmStateReport(LibvirtComputingResource.java:4005)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.getCurrentStatus(LibvirtComputingResource.java:3703)
at com.cloud.agent.Agent.processOtherTask(Agent.java:984)
at com.cloud.agent.Agent$ServerHandler.doTask(Agent.java:1234)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.util.concurrent.FutureTask.run(java.base@17.0.16/FutureTask.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.16/ThreadPoolExecutor.java:1136)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.16/ThreadPoolExecutor.java:635)
at java.lang.Thread.run(java.base@17.0.16/Thread.java:840)
```
Could it be something related to libvirt 10.0.0? (Because I've a running env for many days with libvirt 8.0 and I don't see this there. Though it is also OL8 host). Maybe if others experiencing this issue can share the libvirt version to get some idea
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3591554460
> Hey folks,
>
> I just wanted to share my findings since I am tracking CPU utilization of the Java process of the Agents every 15 seconds.
>
@daviftorres
did you say that the agents were in Up state, but did not connect to the management servers ?
btw, I just checked the CPU usage of my ubuntu 24 hosts (each has 25-27 running vms)
- kvm1 with `libvirt 10.0.0-2ubuntu8.9`: **72.7%**
- kvm2 with `libvirt 10.6.0-1ubuntu3.3`: **0.4%**
I truly advise you to test libvirt 10.6.0 @daviftorres
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "jgotteswinter (via GitHub)" <gi...@apache.org>.
jgotteswinter commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3501905807
I also had situations where the agent was restarted, up and running (at least from what i could see in the logs everything was fine). But it took several minutes for the agent to reconnect to the management servers. Usually this happens instantly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "marekhorecny (via GitHub)" <gi...@apache.org>.
marekhorecny commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3758769631
After two weeks of operation in the production environment, I can confirm that the agent with libvirt-10.6.0-1ubuntu3.3 behaves correctly without any apparent problems. The CPU load for the agent process still ranges from 0.x% to 1.8%.
For now, we will leave libvirt-10.6.0-1ubuntu3.3 in production as a temporary solution. If the bug fix is backported, we will return (after testing) to the original libvirt from the Ubuntu distribution. If the bug is not fixed, we will remain on libvirt-10.6.0-1ubuntu3.3 until the major upgrade to Ubuntu 26.04 (depending on compatibility with CloudStack).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "bradh352 (via GitHub)" <gi...@apache.org>.
bradh352 commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3710510202
@DaanHoogland It would be good to isolate and get a patch upstream to Ubuntu as excluding Ubuntu 24.04 from a supported release seems extreme.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "bradh352 (via GitHub)" <gi...@apache.org>.
bradh352 commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3702052148
@daviftorres how is the bisection going?
I'm thinking about just deploying 10.6 to our production environment then rolling back to whatever the proper solution ends up being. This is still hitting us pretty hard at our level of load ... a once a day restart just isn't enough.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "vgarcia-linube (via GitHub)" <gi...@apache.org>.
vgarcia-linube commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3292786693
The problem seems to be a leak in the handler threads while checking storage usage. The more agent threads you configure in agent.properties and the less time you configure to retrieve volume usage metrics, the worse it gets, and the faster it happens.
On a fresh start of the agent, you get the 'Trying to fetch storage pool xxxx from libvirt' message whenever the usage service is getting updated metrics. Those requests are either leaking or not getting garbage collected or something like that in time. Those requests start to overlap with time, and you end up seeing the same request to the same primary storage tens or hundreds of times. The only way to recover from that is to restart the agent, limit the number of threads of the agent and try to read the usage metrics in longer time spans (I think it defaults to 10 minutes or something like that, setting it to once every two hours mitigates it a bit, just enough so you don't have to restart the agent every few hours so it doesn't hog the kvm node cpu).
Here's a log with redacted storage uuids so it's easier to see (take a look at the timestamps)
[storage-log-no-uuids.log](https://github.com/user-attachments/files/22346459/storage-log-no-uuids.log)
This happens at least since Cloudstack 4.19
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "TadiosAbebe (via GitHub)" <gi...@apache.org>.
TadiosAbebe commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3503739612
I also see this issue frequently, day to day vm operation including starting, stopping, getting console, migrating seems to be affected by it and this happens only on specific compute node while the others work without similar issue. For me most of the time i don't even need to restart the cloudstack-agent, just force reconnecting the host from cloudstack management ui resolves the issue. But i haven't found a way to properly replicate/recreate the issue from my side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "bhouse-nexthop (via GitHub)" <gi...@apache.org>.
bhouse-nexthop commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3501970847
@vgarcia-linube on the settings that helped you said:
> limit the number of threads of the agent and try to read the usage metrics in longer time spans (I think it defaults to 10 minutes or something like that, setting it to once every two hours mitigates it a bit, just enough so you don't have to restart the agent every few hours so it doesn't hog the kvm node cpu
Can you provide the exact setting names (and where, such as agent.properties or something in management config) and values you used that helped? I'd like to see if that helps me.
Also does preemptively restarting the agent before it gets too bad in something like cron.daily prevent the issue from occurring?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3502210901
> [@vgarcia-linube](https://github.com/vgarcia-linube) on the settings that helped you said:
>
> > limit the number of threads of the agent and try to read the usage metrics in longer time spans (I think it defaults to 10 minutes or something like that, setting it to once every two hours mitigates it a bit, just enough so you don't have to restart the agent every few hours so it doesn't hog the kvm node cpu
>
> Can you provide the exact setting names (and where, such as agent.properties or something in management config) and values you used that helped? I'd like to see if that helps me.
>
> Also does preemptively restarting the agent before it gets too bad in something like cron.daily prevent the issue from occurring?
could check `stats.interval` in global settings
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "jgotteswinter (via GitHub)" <gi...@apache.org>.
jgotteswinter commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3739205501
I can confirm that 11.3 also works perfectly fine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3585190130
> The main issue with 10.6.0 is that it needs to be maintained as backport. This shouldnt be a big deal but it needs to be handled properly for production. Before putting more effort into this, i would observe it for a little longer. In my case the agent stopped working after several days.
the agent stopped might also due to the high cpu usage.
in my testing env, the CPU usage of java process of kvm host with libvirt 10.0.0 has increased to 55.6% (from 30.6% yesterday), memory usage also increased from 4.4% to 8.7%
the CPU usage is still 0.5% on the kvm host with libvirt 10.6.0, memory usage is 4.3%
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "shwstppr (via GitHub)" <gi...@apache.org>.
shwstppr commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3569722509
Also, it maybe worth checking if increasing the `ping.interval` helps in any way
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "bradh352 (via GitHub)" <gi...@apache.org>.
bradh352 commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3570498243
Scanning through the release notes for libvirt 10.0 through 10.10, the only thing that stands out was in 10.8:
```
Fix various memleaks and overflows
Multiple memory leaks and overflows in corner cases were fixed based on upstream issues reported.
```
Doesn't have enough information to know if it might be related.
I also don't know about the @weizhouapache test, its seemed like issues happened immediately there so I'm not thinking its the same issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "marekhorecny (via GitHub)" <gi...@apache.org>.
marekhorecny commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3077481285
I have a similar problem, with other symptoms as well. For example, alerts in Cloudstack like "Health checks failed" for virtual routers on affected hosts. The agent is constantly consuming 100% CPU, even when there are no jobs or any current actions on that host.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "daviftorres (via GitHub)" <gi...@apache.org>.
daviftorres commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3682905770
Thank you, @bgo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "jgotteswinter (via GitHub)" <gi...@apache.org>.
jgotteswinter commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3732491636
I am testing Debian 13 since yesterday with automated high concurrent load tests around the instance and volume lifecycle, so far no signs of problems. We will probably move on to Debian.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "bhouse-nexthop (via GitHub)" <gi...@apache.org>.
bhouse-nexthop commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3739199576
hmm, the changeset between 10.4 and 10.6 isn't exactly small, don't know how easy it would be to identify the actual commit that fixes this: https://github.com/libvirt/libvirt/compare/v10.4.0...v10.6.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "daviftorres (via GitHub)" <gi...@apache.org>.
daviftorres commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3739149162
hey folks, I bring good news. I managed to identify where the big is and is not on `libvirtd` versions we are bisecting:
- v10.0 bug confirmed (unhealthy)
- v10.2 bug confirmed (unhealthy)
- v10.4 bug confirmed (unhealthy)
- v10.6 bug not present (healthy)
FYI, @cpaelzer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "daviftorres (via GitHub)" <gi...@apache.org>.
daviftorres commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3638913164
@cpaelzer ,
I dedicated a Dev environment for this tests:
- Host 10 runs `10.0.0-99ubuntu0.bisect4.3`
- Host 11 runs `10.2.0-0ubuntu0.bisect3.3`
- Host 12 runs `10.4.0-0ubuntu0.bisect2.2`
I will monitor closely the metrics and get back to you soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "TadiosAbebe (via GitHub)" <gi...@apache.org>.
TadiosAbebe commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3568368239
I am beginning to doubt whether this issue is related to the CloudStack agent. Over the past few weeks, I’ve been looking more closely at the problem. As I mentioned earlier, I encounter it fairly frequently, and it affects basic VM operations such as creating, starting, stopping, and accessing the console.
It turns out my earlier assumption was incorrect and force reconnecting the compute host does not consistently resolve the issue. This time, instead of reconnecting the host or restarting the cloudstack-agent service, I tried restarting the libvirtd service whenever i encounter the issue. I tested this several times, and in every case, simply restarting libvirtd (without restarting CloudStack agent) resolved the issue.
I haven't seen any noticeable differences in the libvirtd logs compared to a working host, so it’s difficult to pinpoint the root cause. Since this doesn’t appear to be a widely reported problem, I suspect it may be environmental. All my compute hosts are running Ubuntu Server 24.04, and the environment uses a hyperconverged Ceph setup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3656271762
> > > Hey folks,
> > > I just wanted to share my findings since I am tracking CPU utilization of the Java process of the Agents every 15 seconds.
> >
> >
> > [@daviftorres](https://github.com/daviftorres) did you say that the agents were in Up state, but did not connect to the management servers ?
> > btw, I just checked the CPU usage of my ubuntu 24 hosts (each has 25-27 running vms)
> >
> > * kvm1 with `libvirt 10.0.0-2ubuntu8.9`: **72.7%**
> > * kvm2 with `libvirt 10.6.0-1ubuntu3.3`: **0.4%**
> >
> > I truly advise you to test libvirt 10.6.0 [@daviftorres](https://github.com/daviftorres)
>
> Hey [@weizhouapache](https://github.com/weizhouapache) ,
>
> Have you tried to run the bisections that [@cpaelzer](https://github.com/cpaelzer) staged the PPAs?
>
> I did as I mentioned above and I could not make it so manifest the symptoms. I even created a script to deploy and destroy 50 VMs every to minutes but the CPU increase is not expressive:
>
> <img alt="Image" width="495" height="360" src="https://private-user-images.githubusercontent.com/90287660/526605910-1547ebe0-3085-49f7-92c1-2d845bab28bc.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NjU4MTI2ODgsIm5iZiI6MTc2NTgxMjM4OCwicGF0aCI6Ii85MDI4NzY2MC81MjY2MDU5MTAtMTU0N2ViZTAtMzA4NS00OWY3LTkyYzEtMmQ4NDViYWIyOGJjLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTEyMTUlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUxMjE1VDE1MjYyOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWRiOTIwMjhiMjIzNWI5MTJmOTgwNWZmNGRjNTkzYWJjMzA4NTg5ZGViZTFiMDBjOTAyNzdiNGQ0ZWJmYTBlZDgmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.byUsSZEUDvxwWb7Z9bqxQMjzUAZbnGjnumHqFwgP4Ro">
> What operations did you use to induce some the bug?
>
> Regards,
I installed the packages some days ago
- kvm1
<img width="480" height="115" alt="Image" src="https://github.com/user-attachments/assets/8ed183a4-e4b6-4cc6-95ae-62ec74363f05" />
<img width="480" height="450" alt="Image" src="https://github.com/user-attachments/assets/d5f3af84-1bf0-4037-a98e-38c1020b3600" />
- kvm2
<img width="600" height="125" alt="Image" src="https://github.com/user-attachments/assets/ee647048-b9a5-4665-a9e9-daf33964bb5e" />
<img width="560" height="460" alt="Image" src="https://github.com/user-attachments/assets/b99af35a-8427-489c-bc32-4f51d9e202b9" />
I will install 10.4.0 on kvm1 and 10.2.0 on kvm2, to confirm the issue is not related to the server.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "jgotteswinter (via GitHub)" <gi...@apache.org>.
jgotteswinter commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3569332676
I was able to collect a java thread dump while the error happened, no idea if this is helpful. Can anyone give a hint what kind of debug information would be helpful?
[agent-stuck.txt](https://github.com/user-attachments/files/23709143/agent-stuck.txt)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "bhouse-nexthop (via GitHub)" <gi...@apache.org>.
bhouse-nexthop commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3506628093
That's an interesting angle on this issue. I've got a test cluster, but it has more nodes (8 -- its an old supermicro microcloud 3U) and fewer vms, but I do NOT run k8s on that and its run for many weeks without seeing this issue (I've never seen it on this cluster).
But on another cluster that is much newer and more powerful but only has 3 nodes, is fairly heavily loaded with about 30 vms per node, but also runs k8s, I see this issue every 2 days ...
I hadn't thought to look at the k8s angle, but it is an additional difference over the load aspect. I wonder if I can reproduce by deploying a k8s cluster in the test environment.
Since someone else asked about the primary and secondary storage, I use Ceph RBD for primary and NFS (via Ganesha NFS on top of CephFS) for secondary storage ... its set up the same on both the test and prod clusters.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "vgarcia-linube (via GitHub)" <gi...@apache.org>.
vgarcia-linube commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3501937566
@jgotteswinter We've seen that too but it's usually pressure on other parts of the stack. For example, if the primary storage server is slow to respond, restarting the cloudstack-agent may need more time to initialize as it loops through the storage domains.
On this specific issue, we've reconfigured the agent to set its logs to debug but we didn't find any more clues than the logs we shared [here](https://github.com/apache/cloudstack/issues/11141#issuecomment-3292786693)
With about 500 instances in our clusters, we're seeing this event once every two days approximately, but time varies on node capability and number of machines concurrently running on that specific node
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "jgotteswinter (via GitHub)" <gi...@apache.org>.
jgotteswinter commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3502239280
@weizhouapache there is currently only one primary storage and one secondary
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "daviftorres (via GitHub)" <gi...@apache.org>.
daviftorres commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3678027038
Hey folks, here is some numbers. Over time under load I could finally see the CPU climbing.
<img width="502" height="343" alt="Image" src="https://github.com/user-attachments/assets/00dd30d6-3d27-41f6-bfd5-6556286c30d4" />
- Host 10 runs `10.0.0-99ubuntu0.bisect4.3`
- Host 11 runs `10.2.0-0ubuntu0.bisect3.3`
- Host 12 runs `10.4.0-0ubuntu0.bisect2.2`
Unfortunately, I do not have a fourth KVM host in this environment to test all versions. So I am upgrading Host 10 to `10.6.0-0ubuntu0.bisect1`, restart them all and report results in a week or so.
<img width="840" height="210" alt="Image" src="https://github.com/user-attachments/assets/dc82d5ff-7613-4832-8117-a2536ca20c29" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "bhouse-nexthop (via GitHub)" <gi...@apache.org>.
bhouse-nexthop commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3682734740
> I am sticking with the 24hr restart of the agent, for me the problem disappeared. Backports have to much disadvantages imho. The next LTS might fix this
I guess my load is higher than you, I find I need to restart it about 4x a day not to have an issue. I still haven't attempted libvirt 10.6 ... I was waiting for others to bisect a bit and report back.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "jgotteswinter (via GitHub)" <gi...@apache.org>.
jgotteswinter commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3710766336
@DaanHoogland why tempted? sorry for the offtopic question, i am trying to argue for debian somewhere else...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "TadiosAbebe (via GitHub)" <gi...@apache.org>.
TadiosAbebe commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3575316479
@weizhouapache, very interesting. I checked the resource utilization of the Java process on one of the hosts currently exhibiting the issue, and this was the output:
```
root@host3:/home/zgadmin# pid=$(pgrep -f /usr/bin/java)
echo "CPU: $(ps -p $pid -o %cpu --no-headers)%"
echo "MEM: $(ps -p $pid -o %mem --no-headers)%"
echo "FD: $(ls /proc/$pid/fd 2>/dev/null | wc -l)"
echo "Threads: $(ps -L -p $pid -o tid --no-headers 2>/dev/null | wc -l)"
echo "Conn: $(ss -tanp | grep "pid=$pid" | grep ESTAB | wc -l)"
CPU: 73.9%
MEM: 0.1%
FD: 253
Threads: 134
Conn: 1
```
But I had a test all-in-one ACS on ubuntu 24.04 with libvirt 10.0.0, but I couldn’t reproduce the issue I’m seeing in the production environment. I repeatedly ran your test script:
```
for i in `seq 1 20`;do
cmk deploy virtualmachine name=L2-wei-test-$i serviceofferingid=xxx zoneid=xxx templateid=xxx networkids=xxx & >/dev/null;
sleep 2;
done
```
to generate load, and the results were consistently fast:
```
mysql> select id,name,created,update_time,(update_time-created) from vm_instance where removed is null and name like "L2-wei%";
+-----+----------------+---------------------+---------------------+-----------------------+
| id | name | created | update_time | (update_time-created) |
+-----+----------------+---------------------+---------------------+-----------------------+
| 191 | L2-wei-test-1 | 2025-11-25 11:22:07 | 2025-11-25 11:22:14 | 7 |
| 192 | L2-wei-test-2 | 2025-11-25 11:22:09 | 2025-11-25 11:22:16 | 7 |
| 193 | L2-wei-test-3 | 2025-11-25 11:22:11 | 2025-11-25 11:22:17 | 6 |
| 194 | L2-wei-test-4 | 2025-11-25 11:22:13 | 2025-11-25 11:22:22 | 9 |
| 195 | L2-wei-test-5 | 2025-11-25 11:22:15 | 2025-11-25 11:22:20 | 5 |
| 196 | L2-wei-test-6 | 2025-11-25 11:22:17 | 2025-11-25 11:22:23 | 6 |
| 197 | L2-wei-test-7 | 2025-11-25 11:22:19 | 2025-11-25 11:22:26 | 7 |
| 198 | L2-wei-test-8 | 2025-11-25 11:22:21 | 2025-11-25 11:22:27 | 6 |
| 199 | L2-wei-test-9 | 2025-11-25 11:22:23 | 2025-11-25 11:22:29 | 6 |
| 200 | L2-wei-test-10 | 2025-11-25 11:22:25 | 2025-11-25 11:22:31 | 6 |
| 201 | L2-wei-test-11 | 2025-11-25 11:22:27 | 2025-11-25 11:22:34 | 7 |
| 202 | L2-wei-test-12 | 2025-11-25 11:22:29 | 2025-11-25 11:22:36 | 7 |
| 203 | L2-wei-test-13 | 2025-11-25 11:22:31 | 2025-11-25 11:22:38 | 7 |
| 204 | L2-wei-test-14 | 2025-11-25 11:22:33 | 2025-11-25 11:22:41 | 8 |
| 205 | L2-wei-test-15 | 2025-11-25 11:22:35 | 2025-11-25 11:22:42 | 7 |
| 206 | L2-wei-test-16 | 2025-11-25 11:22:37 | 2025-11-25 11:22:45 | 8 |
| 207 | L2-wei-test-17 | 2025-11-25 11:22:39 | 2025-11-25 11:22:48 | 9 |
| 208 | L2-wei-test-18 | 2025-11-25 11:22:41 | 2025-11-25 11:22:49 | 8 |
| 209 | L2-wei-test-19 | 2025-11-25 11:22:43 | 2025-11-25 11:22:51 | 8 |
| 210 | L2-wei-test-20 | 2025-11-25 11:22:45 | 2025-11-25 11:22:55 | 10 |
+-----+----------------+---------------------+---------------------+-----------------------+
```
I'll try to create a full environment consisting ceph and multiple kvm host on a test environment to see if i can replicate the issue there and see if libvirt 10.6.0 fix it, later this week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3574475001
I tested libvirt (10.0.0 and 10.6.0) on two Ubuntu 24.04 nodes, and monitored some metrics every 10 seconds by commands
```
pid=$(pgrep -f /usr/bin/java)
echo "CPU: $(ps -p $pid -o %cpu --no-headers)%"
echo "MEM: $(ps -p $pid -o %mem --no-headers)%"
echo "FD: $(ls /proc/$pid/fd 2>/dev/null | wc -l)"
echo "Threads: $(ps -L -p $pid -o tid --no-headers 2>/dev/null | wc -l)"
echo "Conn: $(ss -tanp | grep "pid=$pid" | grep ESTAB | wc -l)"
```
My observations are
- There are very slight changes with memory, file descriptors, threads and connections of the java process, I think we can ignore.
- The CPU usage of the java process increased gradually and slowly with libvirt 10.0.0
- The CPU usage of the java process decreased gradually and became stable after some hours
| Hostname | libvirt version | original CPU usage| new CPU usage|
|:-----------|:------------|------------:|------------- |
| kvm1 | 10.0.0-2ubuntu8.9 | 0.9% | 2.1% (after 12 hours)|
| kvm2 | 10.0.0-2ubuntu8.9 | 5.3% | 5.9% (after 1 hour), 6.5% (after 2 hours)|
| kvm2 | 10.6.0-1ubuntu3 | 4.9% | 1.3% (after 1 hour), 0.6% (after 3 hours and 6 hours)|
| kvm2 | 10.6.0-1ubuntu3.3 | 5.2% | 1.2% (after 1 hour), 0.8% (after 2 hours)||
note:
- no workloads inside VMs
- no VMs actions
- original CPU usage is fetched 5 mins after restarting agent
I would suggest some of you test `libvirt 10.6.0-1ubuntu3.3` or `libvirt 10.6.0-1ubuntu3` on a non-production environment. I have uploaded the .deb packages to https://github.com/weizhouapache/ubuntu-libvirt-packages
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3597867639
> Hey [@weizhouapache](https://github.com/weizhouapache),
>
> How did you managed to get `libvirt 10.6.0-1ubuntu3.3` on Ubuntu 24.04 LTS? Or you are using Ubuntu 24.10 or newer?
@daviftorres
I tested the libvirt 10.6.0 packages of Ubuntu 24.10
I have uploaded to https://github.com/weizhouapache/ubuntu-libvirt-packages
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3585170964
@TadiosAbebe
regarding the questions
> Why libvirt 10.6.0?
because the packages have been built (for Ubuntu 24.10) and ready for testing
> What would be the impact of manually upgrading libvirt from 10.0.0 to 10.6.0 down the line in production servers?
In my opinion, the risk is small.
libvirt is just a management tool, the VMs do not use it actually
> What would be the best resolution for this issue?
I think upgrading to libvirt 10.6.0 is acceptable solution, if it is tested for several days without issues.
It would be best if Canonical backports the fix (if we can find) to libvirt 10.0.0, test it and include it in the next libvirt minor version
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "TadiosAbebe (via GitHub)" <gi...@apache.org>.
TadiosAbebe commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3585103609
@weizhouapache , Update:
Java process CPU utilization with libvirt 10.6.0 installed on the AIO ACS seems to have the consistent value after 24 hours, During those hours an automated script has created and destroyed 3087 instances. the CPU utilization before starting the benchmark was 1.1% and after 24 the utilization was
```
CPU: 1.0%
MEM: 2.6%
FD: 253
Threads: 78
Conn: 1
```
Libvirt 10.6.0 seems to fix the issue on the test env. So i guess my follow up questions would be
1. Why libvirt 10.6.0?
2. What would be the impact of manually upgrading libvirt from 10.0.0 to 10.6.0 down the line in production servers?
3. What would be the best resolution for this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "shwstppr (via GitHub)" <gi...@apache.org>.
shwstppr commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3078336776
@poli1025 is degradation specific only for the agent or is there overall slowness in handling start-stop API calls? (So we can rule out any MS issue)
You mention bulk operations. Can you please give some insight on that?
In the agent logs, do you see any related logs, like some stuck task of a background job giving an error?
Would ut be possible for you provide logs and may be heap dump to analyze this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3033705993
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3298718305
thanks for the information @vgarcia-linube
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "jgotteswinter (via GitHub)" <gi...@apache.org>.
jgotteswinter commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3501597411
we also have this kind of issue. After a few days the agent stops executing tasks. After restarting the agent it starts working again.
- ACS currently 4.20.1.0, upgrade to 4.20.2.0 is in planning
- Ubuntu 24.04
- Primary Storage is provided by a Netapp NFS Share
- 20-30 vms per host currently
Is there any useful debugging data i can collect when this error occurs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "jgotteswinter (via GitHub)" <gi...@apache.org>.
jgotteswinter commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3506074019
@TadiosAbebe thats a interesting observation, i will try that.
Some of our instances run k8s with the leaseweb csi driver, we had deployments which triggered a lot of block volume actions. I think this frequently caused this problem. Meanwhile we got rid of these deployments, the problem still shows up but less frequent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "bradh352 (via GitHub)" <gi...@apache.org>.
bradh352 commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3568380154
@TadiosAbebe my environment matches yours .... ubuntu 24.04, hypercoverged ceph. And the reported behavior also matches.
That said, using the force reconnect never did work for me as a resolution. And when a host got in that state, just restarting the agents wouldn't work for me and I'm not exactly sure why. I never did fully isolate it. Sometimes I had to restart the management servers which was really concerning, I think I tried restarting libvirtd too.
I set up a cron job to restart the agent daily at 5 min intervals across hosts: https://github.com/bradh352/ansible-role-service-cloudstack/commit/fdb14d92f115a5845d95880a05f467b42a84aff1
This has helped considerably. That said, I just went through to "test" everything right now to see if everything was healthy, which entails trying to open the console on one vm per host, and I found a host in a bad state ... ugh. So restarting the agent daily is definitely not a resolution. Out of curiosity I decided to restart libvirtd, and that *did* fix it on the bad host.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "jgotteswinter (via GitHub)" <gi...@apache.org>.
jgotteswinter commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3569686800
`(xxx-dev-root) 🐱 > list configurations filter=id,name,value keyword=stats.interval
{
"configuration": [
{
"name": "autoscale.stats.interval",
"value": "60"
},
{
"name": "cluster.heartbeat.threshold",
"value": "150000"
},
{
"name": "database.server.stats.interval",
"value": "60"
},
{
"name": "database.server.stats.retention",
"value": "3"
},
{
"name": "direct.network.stats.interval",
"value": "86400"
},
{
"name": "external.network.stats.interval",
"value": "300"
},
{
"name": "host.stats.interval",
"value": "60000"
},
{
"name": "management.server.stats.interval",
"value": "60"
},
{
"name": "router.stats.interval",
"value": "300"
},
{
"name": "storage.stats.interval",
"value": "60000"
},
{
"name": "storpool.storage.stats.interval",
"value": "3600"
},
{
"name": "storpool.volumes.stats.interval",
"value": "3600"
},
{
"name": "vm.disk.stats.interval",
"value": "0"
},
{
"name": "vm.disk.stats.interval.min",
"value": "300"
},
{
"name": "vm.network.stats.interval",
"value": "0"
},
{
"name": "vm.network.stats.interval.min",
"value": "300"
},
{
"name": "vm.stats.interval",
"value": "60000"
},
{
"name": "volume.stats.interval",
"value": "600000"
}
],
"count": 18
}`
`xx@xxx:~# java --version
openjdk 17.0.16 2025-07-15
OpenJDK Runtime Environment (build 17.0.16+8-Ubuntu-0ubuntu124.04.1)
OpenJDK 64-Bit Server VM (build 17.0.16+8-Ubuntu-0ubuntu124.04.1, mixed mode, sharing)`
`xxx@xxx:~# cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.3 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo`
`xxx@xxx:~# virsh version
Compiled against library: libvirt 10.0.0
Using library: libvirt 10.0.0
Using API: QEMU 10.0.0
Running hypervisor: QEMU 8.2.2`
`xxx@xxx:~# ps auxf |grep qemu-system |wc -l
25`
This is our development environment, there is not so much activity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "jgotteswinter (via GitHub)" <gi...@apache.org>.
jgotteswinter commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3570530323
I can confirm that i have seen the same slow vm actions @weizhouapache had in his tests. This was never a permanent problem, and sometimes it disappeared without further action from me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3575344301
@TadiosAbebe
I forgot to mention that both my ol9 and ubuntu24 envs were not used for more than 15 days, when I tested yesterday.
I did some more testing after my yesterday's comment.
when I restarted libvirtd or cloudstack-agent, the vm deployments went very fast, same as you have tested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "jgotteswinter (via GitHub)" <gi...@apache.org>.
jgotteswinter commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3682859129
> > I am sticking with the 24hr restart of the agent, for me the problem disappeared. Backports have to much disadvantages imho. The next LTS might fix this
>
> I guess my load is higher than you, I find I need to restart it about 4x a day not to have an issue. I still haven't attempted libvirt 10.6 ... I was waiting for others to bisect a bit and report back.
i am running the 10.6 in a low load dev environment. No problems so far, but like i said... low load.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3710747087
@bradh352 , I meant to exclude the combination ubuntu-24/libvirt-10.4 , not the entire range of ubuntu distros (though I am tempted)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "bradh352 (via GitHub)" <gi...@apache.org>.
bradh352 commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3706456203
I've been running 10.6 for a couple of days now and I can at least confirm it has fixed the issue (so far!). I disabled the daily restart cronjob to make sure its not just "better".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Degraded cloudstack agent [cloudstack]
Posted by "jgotteswinter (via GitHub)" <gi...@apache.org>.
jgotteswinter commented on issue #11141:
URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3575135093
Thanks a lot @weizhouapache. I upgraded to libvirt 10.6.0-1ubuntu3.3 on one of our development hypervisors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org