You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@rocketmq.apache.org by "tju-yxq (via GitHub)" <gi...@apache.org> on 2026/08/27 19:33:19 UTC

[I] Native alert collection can lose its lease during a long empty pass [rocketmq-dashboard]

tju-yxq opened a new issue, #2641:
URL: https://github.com/apache/rocketmq-dashboard/issues/2641

   `CollectorScheduler` acquires the native-alert database lease once at the beginning of a collection pass. It only calls `tryAcquire()` again from `persist()` when a collector returned samples.
   
   A pass that takes longer than the lease duration while all collectors return empty lists therefore stops touching `rmq_alert_collection_lease`. After the row expires, another Studio replica can acquire the lease and start the same pass while the first replica is still contacting brokers. The first replica may also continue running until a later non-empty result notices that it no longer owns the lease.
   
   This is separate from #1776/#1781, which addressed the legacy collection executor's `DiscardOldestPolicy`; this report is about lease ownership for the native alert collector introduced by #2533.
   
   ### Reproduction
   
   1. Configure a short `studio.alerting.collection-lease-duration` (or use a collection pass that lasts longer than the default `PT1M`).
   2. Let a native collector spend longer than that duration returning no samples.
   3. While the first pass is still running, attempt to acquire the same lease from a second Studio replica.
   
   ### Expected behavior
   
   The active pass should renew its lease while it is running, including passes that produce no samples. If renewal fails, it should stop scheduling or persisting further work instead of continuing as the active collector.
   
   ### Current behavior
   
   There is no renewal while an empty pass is running, so the database lease can expire and ownership can overlap across replicas.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Native alert collection can lose its lease during a long empty pass [rocketmq-dashboard]

Posted by "RockteMQ-AI (via GitHub)" <gi...@apache.org>.
RockteMQ-AI commented on issue #2641:
URL: https://github.com/apache/rocketmq-dashboard/issues/2641#issuecomment-5444643082

   **Issue Evaluation**
   
   Category: `bug` | Status: **Confirmed**
   
   This is a well-documented concurrency bug in the native alert collection mechanism. The issue is clear:
   
   **Root Cause:** The lease management in the alert collection process does not properly handle long-running empty passes. When the collection process takes longer than the lease duration without processing any alerts, the lease expires and can be acquired by another instance, leading to duplicate processing or lost alerts.
   
   **Impact:** In distributed deployments, this can cause:
   - Duplicate alert processing
   - Lost alerts during lease transitions
   - Inconsistent alert state across instances
   
   **Severity:** High - affects data integrity in distributed alert collection scenarios.
   
   **Suggested Fix:** Implement lease renewal mechanism that extends the lease before expiration, even during empty passes. Consider using a heartbeat pattern to keep the lease alive while the collection process is active.
   
   An automated fix proposal can be generated. Reply `/approve` to proceed with PR generation.
   
   ---
   *Automated evaluation by github-manager*


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Native alert collection can lose its lease during a long empty pass [rocketmq-dashboard]

Posted by "tju-yxq (via GitHub)" <gi...@apache.org>.
tju-yxq commented on issue #2641:
URL: https://github.com/apache/rocketmq-dashboard/issues/2641#issuecomment-5444370896

   Implemented in PR #2642.
   
   The scheduler now renews the collection lease throughout an active pass, including empty passes, with a holder- and expiry-checked database update. If renewal fails, outstanding collectors are cancelled and later results are not persisted. The lease duration and renewal interval are configurable.
   
   Verification on commit `4a5a01a2`:
   - focused backend tests: 9 passed;
   - full `server` Maven suite: 1792 passed, 0 failures, 0 errors, 0 skipped;
   - `git diff --check`: passed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Native alert collection can lose its lease during a long empty pass [rocketmq-dashboard]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #2641:
URL: https://github.com/apache/rocketmq-dashboard/issues/2641#issuecomment-5533800164

   This issue has been automatically closed because there has been no activity for 7 days. If you believe this issue is still relevant, please reopen it with updated information.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Native alert collection can lose its lease during a long empty pass [rocketmq-dashboard]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #2641: Native alert collection can lose its lease during a long empty pass
URL: https://github.com/apache/rocketmq-dashboard/issues/2641


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org