You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "sumitagrawl (via GitHub)" <gi...@apache.org> on 2026/01/26 20:01:56 UTC
[PR] HDDS-14070. set statemachine ready on election performed for follower [ozone]
sumitagrawl opened a new pull request, #9671:
URL: https://github.com/apache/ozone/pull/9671
## What changes were proposed in this pull request?
Exit of safemode status for follower when all rules are satisfied. This is as:
- For leader applyTransaction completion is implicit before being choosen as leader.
- For follower, exit safe mode rule as transaction sync and apply transaction keeps happening from leader node.
This change in reference to below point in JIRA HDDS-5263:
`But once the SCM Ratis server started it will replay logs from Transactioninfo last applied Index, so after that I see all pipelines are removed. (might be due to close pipeline)`
So this have no issue for leader node once choosen during leader election. And for follower node, no need follow same restriction as dynamically the pipeline can be closed on runtime and same can be created by leader continuously.
This check is present during startup to ensure SCM is writable and allow time to create pipeline as best effort and its not true always for the case of pipeline closure later on due to system unhealthy or other reason (System never moves back to Safemode later on as current behavior for any dynamic change later on).
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-14070
## How was this patch tested?
- impact of existing test case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14070. set statemachine ready on election performed for follower [ozone]
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on code in PR #9671:
URL: https://github.com/apache/ozone/pull/9671#discussion_r2756039571
##########
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/safemode/TestSCMSafeModeManager.java:
##########
@@ -177,6 +181,16 @@ public void testSafeModeExitRule() throws Exception {
}
ContainerManager containerManager = mock(ContainerManager.class);
when(containerManager.getContainers(ReplicationType.RATIS)).thenReturn(containers);
+
+ StorageContainerManager mockScmManager = mock(StorageContainerManager.class);
+ SCMHAManager mockScmhaManager = mock(SCMHAManager.class);
+ when(mockScmManager.getScmHAManager()).thenReturn(mockScmhaManager);
+ SCMRatisServer mockScmRatisServer = mock(SCMRatisServer.class);
+ when(mockScmhaManager.getRatisServer()).thenReturn(mockScmRatisServer);
+ SCMStateMachine mockScmStateMachine = mock(SCMStateMachine.class);
+ when(mockScmRatisServer.getSCMStateMachine()).thenReturn(mockScmStateMachine);
+ when((mockScmStateMachine.isRefreshedAfterLeaderReady())).thenReturn(true);
+ scmContext = new SCMContext.Builder().setSCM(mockScmManager).build();
Review Comment:
added integration test case also
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14070. set statemachine ready on election performed for follower [ozone]
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl merged PR #9671:
URL: https://github.com/apache/ozone/pull/9671
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14070. set statemachine ready on election performed for follower [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on code in PR #9671:
URL: https://github.com/apache/ozone/pull/9671#discussion_r2751673004
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/SCMStateMachine.java:
##########
@@ -285,6 +285,15 @@ public void notifyLeaderChanged(RaftGroupMemberId groupMemberId,
currentLeaderTerm.set(scm.getScmHAManager().getRatisServer().getDivision()
.getInfo().getCurrentTerm());
+ if (!refreshedAfterLeaderReady.get()) {
+ // refresh and validate safe mode rules if it can exit safe mode
+ // if being leader, all previous term transactions have been applied
+ // if other states, just refresh safe mode rules, and transaction keeps flushing from leader
+ // and does not depend on pending transactions.
+ refreshedAfterLeaderReady.set(true);
Review Comment:
This is not atomic. Use
```java
if (refreshedAfterLeaderReady.compareAndSet(false, true)) {
```
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/SCMStateMachine.java:
##########
@@ -355,12 +364,9 @@ public void notifyTermIndexUpdated(long term, long index) {
}
if (currentLeaderTerm.get() == term) {
- // Means all transactions before this term have been applied.
// This means after a restart, all pending transactions have been applied.
- // Perform
- // 1. Refresh Safemode rules state.
- // 2. Start DN Rpc server.
if (!refreshedAfterLeaderReady.get()) {
Review Comment:
Similarly, it needs `compareAndSet`.
BTW, why refreshedAfterLeaderReady won't be set to false after set to true?
##########
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/safemode/TestSCMSafeModeManager.java:
##########
@@ -177,6 +181,16 @@ public void testSafeModeExitRule() throws Exception {
}
ContainerManager containerManager = mock(ContainerManager.class);
when(containerManager.getContainers(ReplicationType.RATIS)).thenReturn(containers);
+
+ StorageContainerManager mockScmManager = mock(StorageContainerManager.class);
+ SCMHAManager mockScmhaManager = mock(SCMHAManager.class);
+ when(mockScmManager.getScmHAManager()).thenReturn(mockScmhaManager);
+ SCMRatisServer mockScmRatisServer = mock(SCMRatisServer.class);
+ when(mockScmhaManager.getRatisServer()).thenReturn(mockScmRatisServer);
+ SCMStateMachine mockScmStateMachine = mock(SCMStateMachine.class);
+ when(mockScmRatisServer.getSCMStateMachine()).thenReturn(mockScmStateMachine);
+ when((mockScmStateMachine.isRefreshedAfterLeaderReady())).thenReturn(true);
+ scmContext = new SCMContext.Builder().setSCM(mockScmManager).build();
Review Comment:
Please don't use mock. We need to a real cluster test which test SCM changing leader multiple times.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14070. set statemachine ready on election performed for follower [ozone]
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on code in PR #9671:
URL: https://github.com/apache/ozone/pull/9671#discussion_r2755856874
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/SCMStateMachine.java:
##########
@@ -355,12 +364,9 @@ public void notifyTermIndexUpdated(long term, long index) {
}
if (currentLeaderTerm.get() == term) {
- // Means all transactions before this term have been applied.
// This means after a restart, all pending transactions have been applied.
- // Perform
- // 1. Refresh Safemode rules state.
- // 2. Start DN Rpc server.
if (!refreshedAfterLeaderReady.get()) {
Review Comment:
We do not move back to **safemode** once it exit, even later rules are not satisfied later. that is the reason we are not setting to false.
This is the best effort on startup only to wait till system moves to healthy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14070. set statemachine ready on election performed for follower [ozone]
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on code in PR #9671:
URL: https://github.com/apache/ozone/pull/9671#discussion_r2756041632
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/SCMStateMachine.java:
##########
@@ -285,6 +285,15 @@ public void notifyLeaderChanged(RaftGroupMemberId groupMemberId,
currentLeaderTerm.set(scm.getScmHAManager().getRatisServer().getDivision()
.getInfo().getCurrentTerm());
+ if (!refreshedAfterLeaderReady.get()) {
+ // refresh and validate safe mode rules if it can exit safe mode
+ // if being leader, all previous term transactions have been applied
+ // if other states, just refresh safe mode rules, and transaction keeps flushing from leader
+ // and does not depend on pending transactions.
+ refreshedAfterLeaderReady.set(true);
Review Comment:
done
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/SCMStateMachine.java:
##########
@@ -355,12 +364,9 @@ public void notifyTermIndexUpdated(long term, long index) {
}
if (currentLeaderTerm.get() == term) {
- // Means all transactions before this term have been applied.
// This means after a restart, all pending transactions have been applied.
- // Perform
- // 1. Refresh Safemode rules state.
- // 2. Start DN Rpc server.
if (!refreshedAfterLeaderReady.get()) {
Review Comment:
updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org