You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "jojochuang (via GitHub)" <gi...@apache.org> on 2026/04/02 22:13:24 UTC
[PR] HDDS-13133. Display Ratis state machine event timeline in OM web UI [ozone]
jojochuang opened a new pull request, #10034:
URL: https://github.com/apache/ozone/pull/10034
## What changes were proposed in this pull request?
HDDS-13133. Display Ratis state machine event timeline in OM web UI
Please describe your PR in detail:
* Add a new dashboard to OM web UI to show Ratis state machine events in chronological order.
* At every Ratis state machine notification callback, record the notification messages and keep the last 100 of them.
* A simple front end UI to show the state machine transitions.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-13133
## How was this patch tested?
Unit tests to ensure OM and SCM state machine events are recorded.
Visited OM and SCM web UI to check for the output.
<img width="888" height="386" alt="Screenshot 2026-04-02 at 2 50 23 PM" src="https://github.com/user-attachments/assets/5d6a9906-ffc3-496d-9a4d-8268d1ff7879" />
<img width="1011" height="339" alt="Screenshot 2026-04-02 at 3 12 54 PM" src="https://github.com/user-attachments/assets/db512d28-dbd3-47fa-9649-a4bbf37e5455" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13133. Display Ratis state machine event timeline in OM web UI [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on code in PR #10034:
URL: https://github.com/apache/ozone/pull/10034#discussion_r3046630426
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/SCMStateMachine.java:
##########
@@ -213,6 +215,10 @@ public void notifyNotLeader(Collection<TransactionContext> pendingEntries) {
return;
}
LOG.info("current leader SCM steps down.");
+ SCMMetrics metrics = StorageContainerManager.getMetrics();
+ if (metrics != null) {
Review Comment:
Seems that the return value is never null.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13133. Display Ratis state machine event timeline in OM web UI [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on code in PR #10034:
URL: https://github.com/apache/ozone/pull/10034#discussion_r3046630426
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/SCMStateMachine.java:
##########
@@ -213,6 +215,10 @@ public void notifyNotLeader(Collection<TransactionContext> pendingEntries) {
return;
}
LOG.info("current leader SCM steps down.");
+ SCMMetrics metrics = StorageContainerManager.getMetrics();
+ if (metrics != null) {
Review Comment:
Seems that the return value is never null.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13133. Display Ratis state machine event timeline in OM web UI [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang merged PR #10034:
URL: https://github.com/apache/ozone/pull/10034
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13133. Display Ratis state machine event timeline in OM web UI [ozone]
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on code in PR #10034:
URL: https://github.com/apache/ozone/pull/10034#discussion_r3406727873
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/placement/metrics/SCMMetrics.java:
##########
@@ -155,6 +171,22 @@ public void decrContainerStat(ContainerStat deltaStat) {
this.containerReportWriteCount.incr(-1 * deltaStat.getWriteCount().get());
}
+ public void addRatisEvent(String event) {
+ synchronized (ratisEvents) {
+ if (ratisEvents.size() >= maxRatisEvents) {
+ ratisEvents.removeFirst();
+ }
+ ratisEvents.add(Time.formatTime(Time.now()) + "|" + event);
+ }
+ }
+
+ @Metric("Ratis state machine events")
+ public String getRatisEvents() {
+ synchronized (ratisEvents) {
+ return String.join("\n", ratisEvents);
Review Comment:
Actually the streamed Ratis events can have arbitrary content, it's not just the hardcoded separator. It looks like this was never meant to be surfaced to Prometheus or JMX and I don't think we should be tracking Ratis events as metrics at all. We can keep the Linklist and query it from memory elsewhere. Filed HDDS-15552.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13133. Display Ratis state machine event timeline in OM web UI [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on code in PR #10034:
URL: https://github.com/apache/ozone/pull/10034#discussion_r3046630426
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/SCMStateMachine.java:
##########
@@ -213,6 +215,10 @@ public void notifyNotLeader(Collection<TransactionContext> pendingEntries) {
return;
}
LOG.info("current leader SCM steps down.");
+ SCMMetrics metrics = StorageContainerManager.getMetrics();
+ if (metrics != null) {
Review Comment:
S
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13133. Display Ratis state machine event timeline in OM web UI [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on code in PR #10034:
URL: https://github.com/apache/ozone/pull/10034#discussion_r3046665768
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/SCMStateMachine.java:
##########
@@ -213,6 +215,10 @@ public void notifyNotLeader(Collection<TransactionContext> pendingEntries) {
return;
}
LOG.info("current leader SCM steps down.");
+ SCMMetrics metrics = StorageContainerManager.getMetrics();
+ if (metrics != null) {
Review Comment:
Seems that the returned metrics is never null. Add a field and a helper method.
```java
private final SCMMetrics metrics = StorageContainerManager.getMetrics();
```
```java
private void addRatisEvent(String eventMessage) {
LOG.info(eventMessage);
metrics.addRatisEvent(eventMessage);
}
```
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/SCMStateMachine.java:
##########
@@ -297,10 +310,16 @@ public void notifyLeaderChanged(RaftGroupMemberId groupMemberId,
if (!groupMemberId.getPeerId().equals(newLeaderId)) {
LOG.info("leader changed, yet current SCM is still follower.");
+ if (metrics != null) {
+ metrics.addRatisEvent("Leader changed to " + newLeaderId + ", yet current SCM is still follower.");
Review Comment:
Let's include the id of this server. The log message and the
```java
addRatisEvent("Leader changed to " + newLeaderId
+ ", this server " + groupMemberId.getPeerId() + " remains a Follower.");
```
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/SCMStateMachine.java:
##########
@@ -394,11 +413,35 @@ public void notifyLeaderReady() {
scm.getScmContext().setLeaderReady();
scm.getSCMServiceManager().notifyStatusChanged();
scm.getFinalizationManager().onLeaderReady();
+ SCMMetrics metrics = StorageContainerManager.getMetrics();
+ if (metrics != null) {
+ metrics.addRatisEvent("Ready to serve requests as the leader");
+ }
}
@Override
public void notifyConfigurationChanged(long term, long index,
RaftProtos.RaftConfigurationProto newRaftConfiguration) {
+ SCMMetrics metrics = StorageContainerManager.getMetrics();
+ if (metrics != null) {
+ List<RaftProtos.RaftPeerProto> newPeers =
+ newRaftConfiguration.getPeersList();
+ List<RaftProtos.RaftPeerProto> newListeners =
+ newRaftConfiguration.getListenersList();
+ List<String> newPeerIds = new ArrayList<>();
+ List<String> newListenersIds = new ArrayList<>();
+ for (RaftProtos.RaftPeerProto raftPeerProto : newPeers) {
+ newPeerIds.add(RaftPeerId.valueOf(raftPeerProto.getId()).toString());
+ }
+ for (RaftProtos.RaftPeerProto raftListenerProto : newListeners) {
+ newListenersIds.add(RaftPeerId.valueOf(raftListenerProto.getId()).toString());
+ }
+ metrics.addRatisEvent(
+ "New peers " + newPeerIds +
+ (newListenersIds.isEmpty() ? "" : ", new listeners " + newListenersIds) +
+ " added at term index (" +
Review Comment:
getPeersList() returns the current peers in the conf. It include both the newly added peers and the existing peers (unless they are removed).
```java
addRatisEvent("notifyConfigurationChanged at " + TermIndex.valueOf(term, index)
+ ": " + TextFormat.shortDebugString(newRaftConfiguration));
```
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/placement/metrics/SCMMetrics.java:
##########
@@ -33,6 +36,9 @@ public class SCMMetrics {
public static final String SOURCE_NAME =
SCMMetrics.class.getSimpleName();
+ private final List<String> ratisEvents = new ArrayList<>();
+ private static final int MAX_RATIS_EVENTS = 100;
Review Comment:
Let's make it configurable?
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/placement/metrics/SCMMetrics.java:
##########
@@ -155,6 +161,22 @@ public void decrContainerStat(ContainerStat deltaStat) {
this.containerReportWriteCount.incr(-1 * deltaStat.getWriteCount().get());
}
+ public void addRatisEvent(String event) {
+ synchronized (ratisEvents) {
+ if (ratisEvents.size() >= MAX_RATIS_EVENTS) {
+ ratisEvents.remove(0);
+ }
+ ratisEvents.add(Time.formatTime(Time.now()) + "|" + event);
+ }
+ }
+
+ @Metric("Ratis state machine events")
+ public String getRatisEvents() {
+ synchronized (ratisEvents) {
+ return String.join("\n", ratisEvents);
Review Comment:
- Should the metrics publish only the new events?
- How often this method is involved by the metrics system? It could be inefficient if it keep creating the same, long string again and again.
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/SCMStateMachine.java:
##########
@@ -213,6 +215,10 @@ public void notifyNotLeader(Collection<TransactionContext> pendingEntries) {
return;
}
LOG.info("current leader SCM steps down.");
+ SCMMetrics metrics = StorageContainerManager.getMetrics();
+ if (metrics != null) {
+ metrics.addRatisEvent("current leader SCM steps down.");
Review Comment:
Let's include the id.
```java
addRatisEvent("This server " + getId() + " has stepped down from the Leader.");
```
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/placement/metrics/SCMMetrics.java:
##########
@@ -155,6 +161,22 @@ public void decrContainerStat(ContainerStat deltaStat) {
this.containerReportWriteCount.incr(-1 * deltaStat.getWriteCount().get());
}
+ public void addRatisEvent(String event) {
+ synchronized (ratisEvents) {
+ if (ratisEvents.size() >= MAX_RATIS_EVENTS) {
+ ratisEvents.remove(0);
Review Comment:
Use LinkedList and removeFirst() to avoid array copying.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13133. Display Ratis state machine event timeline in OM web UI [ozone]
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on code in PR #10034:
URL: https://github.com/apache/ozone/pull/10034#discussion_r3406654507
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/placement/metrics/SCMMetrics.java:
##########
@@ -155,6 +171,22 @@ public void decrContainerStat(ContainerStat deltaStat) {
this.containerReportWriteCount.incr(-1 * deltaStat.getWriteCount().get());
}
+ public void addRatisEvent(String event) {
+ synchronized (ratisEvents) {
+ if (ratisEvents.size() >= maxRatisEvents) {
+ ratisEvents.removeFirst();
+ }
+ ratisEvents.add(Time.formatTime(Time.now()) + "|" + event);
+ }
+ }
+
+ @Metric("Ratis state machine events")
+ public String getRatisEvents() {
+ synchronized (ratisEvents) {
+ return String.join("\n", ratisEvents);
Review Comment:
Newlines are invalid separators in Prometheus. This change prevents all SCM metrics from being ingested into Prometheus. @jojochuang we should:
- Convert this to a different separator like `|`
- Add a command to our acceptance tests to check that all nodes' Prometheus instances have registered metrics.
- Probably `GET http://localhost:9090/api/v1/targets` and ensure that `health=up` for each component.
<img width="1855" height="251" alt="Image" src="https://github.com/user-attachments/assets/12439069-f0ce-4460-9328-2e7239150ccf" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13133. Display Ratis state machine event timeline in OM web UI [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on PR #10034:
URL: https://github.com/apache/ozone/pull/10034#issuecomment-4353952123
All test passed. Merged the PR. Thanks @szetszwo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org