You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "sadanand48 (via GitHub)" <gi...@apache.org> on 2026/09/28 08:44:42 UTC
[PR] HDDS-16625. Expose SCM transactionInfo metrics and add Grafana HA Catchup panel. [ozone]
sadanand48 opened a new pull request, #11346:
URL: https://github.com/apache/ozone/pull/11346
## What changes were proposed in this pull request?
In SCM HA troubleshooting, the existing dashboard shows applied index/lag but does not show the SCM DB flush state directly.
This makes it hard to correlate Ratis catch-up with DB durability progress on leader/followers.
This change adds SCM HA metrics for DB flush visibility and surfaces them in Grafana.
Changes
Expose new SCM HA metrics from SCM:
latest flushed transactionInfo index
last SCM DB flush time (epoch ms)
Wire metrics from SCM HA transaction buffer into SCMHAMetrics.
Add unit coverage for new metrics in SCM HA metrics test.
Add/extend Grafana dashboard:
Ozone - SCM HA Catchup
new panels:
SCM Flush Index (transactionInfo)
SCM Last Flush Time
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-16625
## How was this patch tested?
Manually checked the panel on a docker compose cluster
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16625. Expose SCM transactionInfo metrics and add Grafana HA Catchup panel. [ozone]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #11346:
URL: https://github.com/apache/ozone/pull/11346#discussion_r4157071563
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/SCMHADBTransactionBufferImpl.java:
##########
@@ -110,6 +110,26 @@ public SnapshotInfo getLatestSnapshot() {
return latestSnapshot.get();
}
+ @Override
+ public long getLatestSnapshotIndex() {
+ rwLock.readLock().lock();
+ try {
+ return latestSnapshot.get().getIndex();
Review Comment:
This is not necessarily the latest *flushed* index. `SCMStateMachine.takeSnapshot()` calls `setLatestSnapshot(...)` at line 359 before `transactionBuffer.flush()` at line 364, so a scrape in that interval—or after a failed flush—can expose an index that is not durable in the SCM DB. Track a separate flushed watermark initialized from persisted transaction info and advance it only after `commitBatchOperation` succeeds.
This issue also appears on line 124 of the same file.
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/SCMHAMetrics.java:
##########
@@ -64,13 +76,22 @@ public static void unRegister() {
public synchronized void getMetrics(MetricsCollector collector, boolean all) {
// Check current node state (1 leader, 0 follower)
int state = currNodeId.equals(leaderId) ? 1 : 0;
+ long latestSnapshotIndex = transactionBuffer == null ? -1
+ : transactionBuffer.getLatestSnapshotIndex();
+ long lastSnapshotTimeMs = transactionBuffer == null ? -1
+ : transactionBuffer.getLastSnapshotTimeMs();
Review Comment:
The index and timestamp are fetched under two separate read-lock acquisitions, so a flush can complete between these calls and emit an old index paired with the new flush time. Since the dashboard correlates these values as one DB durability state, expose/read one immutable flush-state snapshot under a single lock.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16625. Expose SCM transactionInfo metrics and add Grafana HA Catchup panel. [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #11346: HDDS-16625. Expose SCM transactionInfo metrics and add Grafana HA Catchup panel.
URL: https://github.com/apache/ozone/pull/11346
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16625. Expose SCM transactionInfo metrics and add Grafana HA Catchup panel. [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11346:
URL: https://github.com/apache/ozone/pull/11346#issuecomment-6354107159
This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16625. Expose SCM transactionInfo metrics and add Grafana HA Catchup panel. [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11346:
URL: https://github.com/apache/ozone/pull/11346#issuecomment-6511585723
Thank you for your contribution. This PR is being closed due to inactivity. Please contact a maintainer if you would like to reopen it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org