You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "xichen01 (via GitHub)" <gi...@apache.org> on 2026/04/09 10:15:54 UTC
[PR] HDDS-14989. Delay follower SCM DN server start until Ratis log catch-up [ozone]
xichen01 opened a new pull request, #10059:
URL: https://github.com/apache/ozone/pull/10059
## What changes were proposed in this pull request?
### Context
Fixed a bug where reading the key could result in a NO_REPLICA_FOUND error during SCM restart and leader transfer.
For details:
https://issues.apache.org/jira/browse/HDDS-14989
Or can see `testFollowerCatchupAfterContainerClose` for detailed reproduction way
### Fix
- When the Follower SCM starts, it starts the DatanodeProtocolServer to receive FCR and ICR from the Datanode after catching up with the leader's committed log entries.
- Only allow the leader SCM to update the container via Ratis by executing `updateContainerState`.
## What is the link to the Apache JIRA'
https://issues.apache.org/jira/browse/HDDS-14989
## How was this patch tested?
nee test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14989. Delay follower SCM DN server start until Ratis log catch-up [ozone]
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on code in PR #10059:
URL: https://github.com/apache/ozone/pull/10059#discussion_r3247254446
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java:
##########
@@ -1580,8 +1573,10 @@ public void start() throws IOException {
}
getBlockProtocolServer().start();
- // start datanode protocol server
- getDatanodeProtocolServer().start();
+ // If HA is enabled, start datanode protocol server once leader is ready.
+ if (!scmStorageConfig.isSCMHAEnabled()) {
Review Comment:
This is not required, as this will delay exit of safemode rule at follower due to delay in DN registration.
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/SCMStateMachine.java:
##########
@@ -401,19 +366,100 @@ public void notifyTermIndexUpdated(long term, long index) {
if (transactionBuffer != null) {
transactionBuffer.updateLatestTrxInfo(TransactionInfo.valueOf(term, index));
}
+ }
- if (currentLeaderTerm.get() == term) {
- // This means after a restart, all pending transactions have been applied.
+ public boolean getIsStateMachineReady() {
+ return isStateMachineReady.get();
+ }
+
+ /**
+ * Start the DN protocol server and trigger safe mode re-evaluation.
+ *
+ * <p>In HA mode the DN server is deliberately not started during
+ * {@link org.apache.hadoop.hdds.scm.server.StorageContainerManager#start()}.
+ * Instead it is deferred until the SCM state machine has caught up with
+ * the leader's committed log entries, so that DN heartbeats are processed
+ * against the latest container/pipeline state rather than a stale snapshot.
+ *
+ * <p>The method is guarded by {@code isStateMachineReady} (CAS) to ensure
+ * the non-idempotent {@code DatanodeProtocolServer.start()} is invoked
+ * exactly once.
+ */
+ private void tryStartDNServerAndRefreshSafeMode() {
+ if (isStateMachineReady.get()) {
+ return;
+ }
+ if (scm.getScmContext().isLeader() || isFollowerCaughtUp()) {
if (isStateMachineReady.compareAndSet(false, true)) {
- // Refresh Safemode rules state if not already done.
+ scm.getDatanodeProtocolServer().start();
scm.getScmSafeModeManager().refreshAndValidate();
}
- currentLeaderTerm.set(-1L);
}
}
- public boolean getIsStateMachineReady() {
- return isStateMachineReady.get();
+ /**
+ * Check whether this follower's state machine has caught up with the
+ * leader's committed log entries.
+ * @return true if {@code lastAppliedIndex >= leaderCommitIndex}
+ */
+ private boolean isFollowerCaughtUp() {
Review Comment:
We can avoid much change to StateMachine based, as this will not resolve problem completely, due to nature that sync from leader may not be immediate, and leader keep trying sync and update
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14989. Delay follower SCM DN server start until Ratis log catch-up [ozone]
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on PR #10059:
URL: https://github.com/apache/ozone/pull/10059#issuecomment-4458791166
FYI.. related issue for Safemode Exit [HDDS-15238](https://issues.apache.org/jira/browse/HDDS-15238)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14989. Delay follower SCM DN server start until Ratis log catch-up [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10059:
URL: https://github.com/apache/ozone/pull/10059#issuecomment-4636589203
This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14989. Delay follower SCM DN server start until Ratis log catch-up [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10059:
URL: https://github.com/apache/ozone/pull/10059#issuecomment-4747250645
Thank you for your contribution. This PR is being closed due to inactivity. Please contact a maintainer if you would like to reopen it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14989. Delay follower SCM DN server start until Ratis log catch-up [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on PR #10059:
URL: https://github.com/apache/ozone/pull/10059#issuecomment-4439259964
@xichen01 , is this PR ready for review now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14989. Delay follower SCM DN server start until Ratis log catch-up [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on code in PR #10059:
URL: https://github.com/apache/ozone/pull/10059#discussion_r3121480790
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/SCMStateMachine.java:
##########
@@ -369,19 +366,100 @@ public void notifyTermIndexUpdated(long term, long index) {
if (transactionBuffer != null) {
transactionBuffer.updateLatestTrxInfo(TransactionInfo.valueOf(term, index));
}
+ }
+
+ public boolean getIsStateMachineReady() {
+ return isStateMachineReady.get();
+ }
- if (currentLeaderTerm.get() == term) {
- // This means after a restart, all pending transactions have been applied.
+ /**
+ * Start the DN protocol server and trigger safe mode re-evaluation.
+ *
+ * <p>In HA mode the DN server is deliberately not started during
+ * {@link org.apache.hadoop.hdds.scm.server.StorageContainerManager#start()}.
+ * Instead it is deferred until the SCM state machine has caught up with
+ * the leader's committed log entries, so that DN heartbeats are processed
+ * against the latest container/pipeline state rather than a stale snapshot.
+ *
+ * <p>The method is guarded by {@code isStateMachineReady} (CAS) to ensure
+ * the non-idempotent {@code DatanodeProtocolServer.start()} is invoked
+ * exactly once.
+ */
+ private void tryStartDNServerAndRefreshSafeMode() {
+ if (isStateMachineReady.get()) {
+ return;
+ }
+ if (scm.getScmContext().isLeader() || isFollowerCaughtUp()) {
if (isStateMachineReady.compareAndSet(false, true)) {
- // Refresh Safemode rules state if not already done.
+ scm.getDatanodeProtocolServer().start();
scm.getScmSafeModeManager().refreshAndValidate();
}
- currentLeaderTerm.set(-1L);
}
}
- public boolean getIsStateMachineReady() {
- return isStateMachineReady.get();
+ /**
+ * Check whether this follower's state machine has caught up with the
+ * leader's committed log entries.
+ * @return true if {@code lastAppliedIndex >= leaderCommitIndex}
+ */
+ private boolean isFollowerCaughtUp() {
+ try {
+ RaftServer.Division division = scm.getScmHAManager()
+ .getRatisServer().getDivision();
+ DivisionInfo divisionInfo = division.getInfo();
+ long lastAppliedIndex = divisionInfo.getLastAppliedIndex();
+
+ RaftPeerId leaderId = divisionInfo.getLeaderId();
+ if (leaderId != null) {
+ for (RaftProtos.CommitInfoProto info : division.getCommitInfos()) {
+ if (info.getServer().getId().equals(leaderId.toByteString())) {
+ long leaderCommit = info.getCommitIndex();
+ boolean caughtUp = lastAppliedIndex >= leaderCommit;
Review Comment:
Let's apply the fix where we need to take into account the metadata entry.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14989. Delay follower SCM DN server start until Ratis log catch-up [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #10059: HDDS-14989. Delay follower SCM DN server start until Ratis log catch-up
URL: https://github.com/apache/ozone/pull/10059
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org