You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@rocketmq.apache.org by "chennaji9 (via GitHub)" <gi...@apache.org> on 2026/10/15 01:40:34 UTC

[PR] fix(metrics): discard results from cancelled collection jobs [rocketmq-dashboard]

chennaji9 opened a new pull request, #6690:
URL: https://github.com/apache/rocketmq-dashboard/pull/6690

   ## Problem
   
   When a native-metrics collection job hits its per-instance deadline (or the whole pass exceeds its timeout), the scheduler cancels the job's Future and logs "was cancelled" — but a collector that absorbs the interrupt can keep running and still return samples. While any sibling job from the same pass is still running, the shared lease heartbeat is still held, so the late `persist()` for the cancelled instance passes the lease check: the snapshot repository stores samples that predate the cancellation decision, and the alert processor runs `processSuccessfulCollection` for an instance the scheduler had already given up on. `Future.cancel(true)` only sets the interrupt bit; it cannot stop a Callable's side effects.
   
   ## Root Cause
   
   Cancellation state lived only in the Future. Nothing marked the job itself as cancelled for the collection loops and the persist entry, which check only the pass-level heartbeat/lease.
   
   ## Fix
   
   Each `CollectionJob` now carries an `AtomicBoolean` cancellation flag. `job.cancel()` marks the flag first, then interrupts the worker; the job lambda, both collector loops, and the `persist` entry all check the flag, so a cancelled job's late results are discarded instead of persisted. The boundary is deliberately "once cancelled, no new persistence starts" — no attempt is made to roll back writes that already reached the database.
   
   ## Testing
   
   `cd server && mvn test -Dtest=CollectorSchedulerTest` — 15/15 pass, including the new `discardsLateResultsFromACancelledJobTest`: a slow collector swallows the interrupt and returns late while two younger jobs keep the pass and its heartbeat alive; the healthy instance still persists while the cancelled job's samples reach neither `saveAll` nor `processSuccessfulCollection`. Red side: the same test applied to the base commit fails at that assertion — the late samples are persisted there. Latched executors with no sleeps on the critical path; no RocketMQ/MySQL needed.
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(metrics): discard results from cancelled collection jobs [rocketmq-dashboard]

Posted by "lizhimins (via GitHub)" <gi...@apache.org>.
lizhimins merged PR #6690:
URL: https://github.com/apache/rocketmq-dashboard/pull/6690


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org