[I] [Bug] A failed broker runtime-stats read resolves the broker's active disk/JVM/send-queue alerts [rocketmq-dashboard]

["unbridled-41 (via GitHub)" <gi...@apache.org>]

unbridled-41 opened a new issue, #5478:
URL: https://github.com/apache/rocketmq-dashboard/issues/5478

   ### Studio Version
   branch: `rocketmq-studio` @ `6a68042f` (the revision verified; the branch is based on it)
   deployed as: built from source (unit test)
   
   ### Runtime Environment
   
   Not environment-specific: the metric collector, verified with unit tests.
   
   ### Connected RocketMQ Cluster
   
   An Apache instance where a broker stays in the NameServer topology while its runtime stats cannot be read (down, restarting, or a missing master address).
   
   ### Describe the Bug
   
   When a broker's runtime stats cannot be read at all, the collector records only `broker.availability` and returns, so `broker.disk.usage_ratio`, `broker.jvm.heap.usage_ratio` and `broker.send_queue.usage_ratio` produce no sample. The collection scope declares all five keys regardless, and the alert reconciler skips reconciliation only when an unavailable sample carries **the rule's own metric key** — so an active rule on one of the three metrics advances with `clear` and emits RESOLVED on the very pass that failed to read the value; it flaps back when the broker recovers.
   
   The parse-miss branches already record unavailable samples for these metrics (the rationale accepted in #4521); the three early-return/exception branches did not.
   
   ### Steps to Reproduce
   
   1. Have an enabled CLUSTER rule on e.g. `broker.disk.usage_ratio` FIRING for broker X.
   2. Make X unreachable while it is still listed in the NameServer topology, so `examineBrokerClusterInfo` succeeds and `fetchBrokerRuntimeStats` throws.
   3. Observe the next collection pass resolve the incident.
   
   ### What Did You Expect to See?
   
   A collection failure records an unavailable sample for each affected metric and leaves the active alert alone, as `docs/studio-native-alerting-design.md` describes and the parse-miss branches already do.
   
   ### What Did You See Instead?
   
   The active alert is resolved by the collection that failed.
   
   ### Evidence
   
   - `cluster/metrics/collectors/ApacheRocketMqClusterMetricsCollector.java` (the three failure branches), `cluster/metrics/CollectorScheduler.java` (the declared scope), `ops/alert/NativeAlertProcessor.java` (the coverage guard and `advance(clear)`), `ops/alert/AlertStateMachine.java` ("A regular metric must not clear an alert merely because collection failed").
   - `cd server && mvn -o test -Dtest=ApacheRocketMqClusterMetricsCollectorTest` — the added case fails before the change (no `broker.disk.usage_ratio` sample) and passes after (`Tests run: 5`); the wider `cluster.metrics.**` + `ops.alert.**` sweep runs 473 tests with 0 failures.
   
   ### Impact
   
   A false RESOLVED (and its notification) for an active infrastructure alert, plus a flap on recovery.
   
   ### Acceptance Criteria
   
   - The failure branches record an unavailable sample for each runtime metric with the same labels.
   - A regression test pins it.
   
   Related: #4519 (the same defect, fixed for the parse-miss branches by #4521).
   
   **Corresponding PR:** #5449
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug] A failed broker runtime-stats read resolves the broker's active disk/JVM/send-queue alerts [rocketmq-dashboard]

["lizhimins (via GitHub)" <gi...@apache.org>]

lizhimins closed issue #5478: [Bug] A failed broker runtime-stats read resolves the broker's active disk/JVM/send-queue alerts
URL: https://github.com/apache/rocketmq-dashboard/issues/5478


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug] A failed broker runtime-stats read resolves the broker's active disk/JVM/send-queue alerts [rocketmq-dashboard]

["lizhimins (via GitHub)" <gi...@apache.org>]

lizhimins commented on issue #5478:
URL: https://github.com/apache/rocketmq-dashboard/issues/5478#issuecomment-6078548984

   Resolved by #5449 (merged into `rocketmq-studio` as 9e2bb75b4).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org