You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by "DaanHoogland (via GitHub)" <gi...@apache.org> on 2026/07/22 10:32:12 UTC
[I] prometheus improvements [cloudstack]
DaanHoogland opened a new issue, #13667:
URL: https://github.com/apache/cloudstack/issues/13667
### problem
1. Give the exporter's HttpServer an explicit bounded executor (httpServer.setExecutor(Executors.newFixedThreadPool(2))) so one slow scrape can't serialize/queue all others.
2. Add a short TTL/in-flight guard around updateMetrics() (e.g., skip recompute if last run was < N seconds ago, or synchronize so concurrent scrapes share one in-progress computation) so scrape frequency can never multiply backend load.
3. Instrument: log/measure updateMetrics() wall-clock time so the reporter (and CI) can confirm which sub-metric collector is actually slow and verify the fix closes the growth.
additional comments:
1. Stale dynamic config (CONFIRMED) — capacity.calculate.workers is a runtime-dynamic setting, but the new shared executor only reads it once at first creation; live changes are silently ignored until a restart.
2. Swallowed capacity-recalculation abort (CONFIRMED per the extra verify pass) — shutdown racing an in-flight recalculation throws RejectedExecutionException, caught by the blanket catch(Throwable) in recalculateCapacity(), silently skipping storage/IP/VLAN updates for that cycle.
3. Unsynchronized race on _capacityExecutorService (CONFIRMED per the extra verify pass) — can leak a freshly-recreated pool that's never shut down again.
4. Shared fixed-size pool serializes previously-independent callers (PLAUSIBLE) — rolling-maintenance host-drain gating can now queue behind the hourly timer or API-triggered recalculations.
5. Pool no longer bounded to actual task count, so it can stay oversized/stale relative to fleet size (efficiency).
6. Bundling this executor-lifecycle rewrite into what the reported bug (#13586) only needed a one-line fix for (altitude/scope creep).
7. Inconsistent lazy-vs-eager thread-pool lifecycle pattern within the same class (reuse/convention).
8. Minor: the synchronized getter is called per-loop-iteration instead of hoisted once (efficiency).
### versions
The versions of ACS, hypervisors, storage, network etc..
### The steps to reproduce the bug
1.
2.
3.
...
### What to do about it?
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] prometheus improvements [cloudstack]
Posted by "RosiKyu (via GitHub)" <gi...@apache.org>.
RosiKyu commented on issue #13667:
URL: https://github.com/apache/cloudstack/issues/13667#issuecomment-5994108200
@DaanHoogland could you please confirm that this is addressed with: https://github.com/apache/cloudstack/pull/13696, instead of the originally linked closed PR: https://github.com/apache/cloudstack/pull/13678 - so we can close this issue? Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] prometheus improvements [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on issue #13667:
URL: https://github.com/apache/cloudstack/issues/13667#issuecomment-5996791275
@RosiKyu , #13650 needs to be merged for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] prometheus improvements [cloudstack]
Posted by "PrashantBhanage (via GitHub)" <gi...@apache.org>.
PrashantBhanage commented on issue #13667:
URL: https://github.com/apache/cloudstack/issues/13667#issuecomment-5058202731
Opened #13678 with the three changes from the issue, scoped to the prometheus exporter plugin only. PTAL when you get a chance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] prometheus improvements [cloudstack]
Posted by "PrashantBhanage (via GitHub)" <gi...@apache.org>.
PrashantBhanage commented on issue #13667:
URL: https://github.com/apache/cloudstack/issues/13667#issuecomment-5071716236
> [@PrashantBhanage](https://github.com/PrashantBhanage) , happy to assign you, but let’s make sure we don’t create code conflicts (see my question [#13678 (comment)](https://github.com/apache/cloudstack/pull/13678#issuecomment-5069034627))
@DaanHoogland I have switched the base branch of PR #13678 to your ghi13586-prometheusDrainage-20 branch, and GitHub confirms there are zero merge conflicts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] prometheus improvements [cloudstack]
Posted by "PrashantBhanage (via GitHub)" <gi...@apache.org>.
PrashantBhanage commented on issue #13667:
URL: https://github.com/apache/cloudstack/issues/13667#issuecomment-5057907300
@DaanHoogland , I'd like to work on this could you assign it to me?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] prometheus improvements [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on issue #13667:
URL: https://github.com/apache/cloudstack/issues/13667#issuecomment-5069059651
@PrashantBhanage , happy to assign you, but let’s make sure we don’t create code conflicts (see my question https://github.com/apache/cloudstack/pull/13678#issuecomment-5069034627)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org