You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2020/12/03 07:17:23 UTC
[GitHub] [ozone] GlenGeng opened a new pull request #1314: HDDS-3988: DN can distinguish SCMCommand from stale leader SCM
GlenGeng opened a new pull request #1314:
URL: https://github.com/apache/ozone/pull/1314
## What changes were proposed in this pull request?
As part of SCMCommand SCM will also send its current term, which will be used in Datanode to identify if the command was sent by the latest leader SCM.
Datanode will maintain the highest term that it has seen and compare it with the term that is received as part of SCMCommand.
If the term in the Datanode and SCMCommand are same, the command is added to the command queue for processing.
If the term in the Datanode is less than the term received in SCMCommand, Datanode will update its term and add the command to the command queue for processing.
If the term in the Datanode is greater than the term received in SCMCommand, Datanode will ignore the command.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-3988
## How was this patch tested?
CI
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] timmylicheng commented on a change in pull request #1314: HDDS-3988: DN can distinguish SCMCommand from stale leader SCM
Posted by GitBox <gi...@apache.org>.
timmylicheng commented on a change in pull request #1314:
URL: https://github.com/apache/ozone/pull/1314#discussion_r535130177
##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeManager.java
##########
@@ -656,8 +669,57 @@ public void setContainers(DatanodeDetails datanodeDetails,
// Since datanode commands are added through event queue, onMessage method
// should take care of adding commands to command queue.
// Refactor and remove all the usage of this method and delete this method.
+ /**
+ * Only leader SCM can send SCMCommand to datanode, and needs record its
+ * term in the command so that datanode can distinguish commands from stale
+ * leader SCM by comparing term.
+ *
+ * There are 7 SCMCommands:
Review comment:
Nit: Try not to list all the commands by name. It would be hard to maintain the correctness of the comments.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] timmylicheng merged pull request #1314: HDDS-3988: DN can distinguish SCMCommand from stale leader SCM
Posted by GitBox <gi...@apache.org>.
timmylicheng merged pull request #1314:
URL: https://github.com/apache/ozone/pull/1314
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] timmylicheng commented on pull request #1314: HDDS-3988: DN can distinguish SCMCommand from stale leader SCM
Posted by GitBox <gi...@apache.org>.
timmylicheng commented on pull request #1314:
URL: https://github.com/apache/ozone/pull/1314#issuecomment-739667406
+1. Merging
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] GlenGeng commented on a change in pull request #1314: HDDS-3988: DN can distinguish SCMCommand from stale leader SCM
Posted by GitBox <gi...@apache.org>.
GlenGeng commented on a change in pull request #1314:
URL: https://github.com/apache/ozone/pull/1314#discussion_r533923194
##########
File path: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/StateContext.java
##########
@@ -470,6 +484,65 @@ public void execute(ExecutorService service, long time, TimeUnit unit)
}
}
+ /**
+ * After startup, datanode needs detect latest leader SCM before handling
+ * any SCMCommand, so that it won't be disturbed by stale leader SCM.
+ *
+ * The rule is: after majority SCMs are in HEARTBEAT state and has
+ * heard from leader SCMs (commandQueue is not empty), datanode will init
+ * termOfLeaderSCM with the max term found in commandQueue.
+ *
+ * The init process also works for non-HA mode. In that case, term of all
+ * SCMCommands will be 0.
+ */
+ void initTermOfLeaderSCM() {
+ // only init once
+ if (termOfLeaderSCM.isPresent()) {
+ return;
+ }
+
+ AtomicInteger scmNum = new AtomicInteger(0);
+ AtomicInteger activeScmNum = new AtomicInteger(0);
+
+ getParent().getConnectionManager().getValues()
+ .forEach(endpoint -> {
+ if (endpoint.isPassive()) {
+ return;
+ }
+ scmNum.incrementAndGet();
+ if (endpoint.getState()
+ == EndpointStateMachine.EndPointStates.HEARTBEAT) {
+ activeScmNum.incrementAndGet();
+ }
+ });
+
+ // majority SCMs should be in HEARTBEAT state.
+ if (activeScmNum.get() < scmNum.get() / 2 + 1) {
+ return;
+ }
+
+ // if commandQueue is not empty, init termOfLeaderSCM
+ // with the largest term found in commandQueue
+ commandQueue.stream()
+ .mapToLong(SCMCommand::getTerm)
+ .max()
+ .ifPresent(term -> termOfLeaderSCM = Optional.of(term));
+ }
+
+ /**
+ * monotonically increase termOfLeaderSCM.
+ * Always record the latest term that has seen.
+ */
+ void updateTermOfLeaderSCM(SCMCommand<?> command) {
+ if (!termOfLeaderSCM.isPresent()) {
+ LOG.error("should init termOfLeaderSCM before update it.");
+ return;
+ }
+
+ termOfLeaderSCM = Optional.of(
Review comment:
Good Point.
Since this is executed in a single thread, let's do this first. We surely need to consider the thread safety in future change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] timmylicheng commented on pull request #1314: HDDS-3988: DN can distinguish SCMCommand from stale leader SCM
Posted by GitBox <gi...@apache.org>.
timmylicheng commented on pull request #1314:
URL: https://github.com/apache/ozone/pull/1314#issuecomment-737882241
LGTM overall. Left comments inline on a minor issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] GlenGeng commented on a change in pull request #1314: HDDS-3988: DN can distinguish SCMCommand from stale leader SCM
Posted by GitBox <gi...@apache.org>.
GlenGeng commented on a change in pull request #1314:
URL: https://github.com/apache/ozone/pull/1314#discussion_r535173512
##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeManager.java
##########
@@ -656,8 +669,57 @@ public void setContainers(DatanodeDetails datanodeDetails,
// Since datanode commands are added through event queue, onMessage method
// should take care of adding commands to command queue.
// Refactor and remove all the usage of this method and delete this method.
+ /**
+ * Only leader SCM can send SCMCommand to datanode, and needs record its
+ * term in the command so that datanode can distinguish commands from stale
+ * leader SCM by comparing term.
+ *
+ * There are 7 SCMCommands:
Review comment:
Good point. I've removed the comments for good.
As discussed in design of SCMContext, we will query and inject the term before firing event of DATANODE_COMMAND explicitly for all related places. I will revisit this code in dev of phase 2.0.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] GlenGeng commented on pull request #1314: HDDS-3988: DN can distinguish SCMCommand from stale leader SCM
Posted by GitBox <gi...@apache.org>.
GlenGeng commented on pull request #1314:
URL: https://github.com/apache/ozone/pull/1314#issuecomment-737903970
@timmylicheng Thanks for looking at this patch!
We surely need unit tests for this feature. I've file a new for this task https://issues.apache.org/jira/browse/HDDS-4544
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] GlenGeng closed pull request #1314: HDDS-3988: DN can distinguish SCMCommand from stale leader SCM
Posted by GitBox <gi...@apache.org>.
GlenGeng closed pull request #1314:
URL: https://github.com/apache/ozone/pull/1314
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] GlenGeng commented on a change in pull request #1314: HDDS-3988: DN can distinguish SCMCommand from stale leader SCM
Posted by GitBox <gi...@apache.org>.
GlenGeng commented on a change in pull request #1314:
URL: https://github.com/apache/ozone/pull/1314#discussion_r533922364
##########
File path: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/StateContext.java
##########
@@ -470,6 +484,65 @@ public void execute(ExecutorService service, long time, TimeUnit unit)
}
}
+ /**
+ * After startup, datanode needs detect latest leader SCM before handling
+ * any SCMCommand, so that it won't be disturbed by stale leader SCM.
+ *
+ * The rule is: after majority SCMs are in HEARTBEAT state and has
+ * heard from leader SCMs (commandQueue is not empty), datanode will init
+ * termOfLeaderSCM with the max term found in commandQueue.
+ *
+ * The init process also works for non-HA mode. In that case, term of all
+ * SCMCommands will be 0.
+ */
+ void initTermOfLeaderSCM() {
+ // only init once
+ if (termOfLeaderSCM.isPresent()) {
+ return;
+ }
+
+ AtomicInteger scmNum = new AtomicInteger(0);
+ AtomicInteger activeScmNum = new AtomicInteger(0);
+
+ getParent().getConnectionManager().getValues()
+ .forEach(endpoint -> {
+ if (endpoint.isPassive()) {
+ return;
+ }
+ scmNum.incrementAndGet();
+ if (endpoint.getState()
+ == EndpointStateMachine.EndPointStates.HEARTBEAT) {
+ activeScmNum.incrementAndGet();
+ }
+ });
+
+ // majority SCMs should be in HEARTBEAT state.
+ if (activeScmNum.get() < scmNum.get() / 2 + 1) {
+ return;
+ }
+
+ // if commandQueue is not empty, init termOfLeaderSCM
+ // with the largest term found in commandQueue
+ commandQueue.stream()
+ .mapToLong(SCMCommand::getTerm)
+ .max()
+ .ifPresent(term -> termOfLeaderSCM = Optional.of(term));
+ }
+
+ /**
+ * monotonically increase termOfLeaderSCM.
+ * Always record the latest term that has seen.
+ */
+ void updateTermOfLeaderSCM(SCMCommand<?> command) {
Review comment:
Sure
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] GlenGeng commented on a change in pull request #1314: HDDS-3988: DN can distinguish SCMCommand from stale leader SCM
Posted by GitBox <gi...@apache.org>.
GlenGeng commented on a change in pull request #1314:
URL: https://github.com/apache/ozone/pull/1314#discussion_r533922178
##########
File path: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/StateContext.java
##########
@@ -470,6 +484,65 @@ public void execute(ExecutorService service, long time, TimeUnit unit)
}
}
+ /**
+ * After startup, datanode needs detect latest leader SCM before handling
+ * any SCMCommand, so that it won't be disturbed by stale leader SCM.
+ *
+ * The rule is: after majority SCMs are in HEARTBEAT state and has
+ * heard from leader SCMs (commandQueue is not empty), datanode will init
+ * termOfLeaderSCM with the max term found in commandQueue.
+ *
+ * The init process also works for non-HA mode. In that case, term of all
+ * SCMCommands will be 0.
+ */
+ void initTermOfLeaderSCM() {
Review comment:
Sure
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] timmylicheng edited a comment on pull request #1314: HDDS-3988: DN can distinguish SCMCommand from stale leader SCM
Posted by GitBox <gi...@apache.org>.
timmylicheng edited a comment on pull request #1314:
URL: https://github.com/apache/ozone/pull/1314#issuecomment-737882241
LGTM overall. Left comments inline on a minor issue.
Also would like to see some level of tests on this feature.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] GlenGeng commented on a change in pull request #1314: HDDS-3988: DN can distinguish SCMCommand from stale leader SCM
Posted by GitBox <gi...@apache.org>.
GlenGeng commented on a change in pull request #1314:
URL: https://github.com/apache/ozone/pull/1314#discussion_r533924763
##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeManager.java
##########
@@ -130,6 +135,14 @@ public SCMNodeManager(OzoneConfiguration conf,
this.useHostname = conf.getBoolean(
DFSConfigKeysLegacy.DFS_DATANODE_USE_DN_HOSTNAME,
DFSConfigKeysLegacy.DFS_DATANODE_USE_DN_HOSTNAME_DEFAULT);
+ this.scmhaManager = scmhaManager;
+ }
+
+ public SCMNodeManager(OzoneConfiguration conf,
Review comment:
Quite a lot of unit tests rely on this old Ctor.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] GlenGeng commented on pull request #1314: HDDS-3988: DN can distinguish SCMCommand from stale leader SCM
Posted by GitBox <gi...@apache.org>.
GlenGeng commented on pull request #1314:
URL: https://github.com/apache/ozone/pull/1314#issuecomment-737765188
@ChenSammi @timmylicheng @nandakumar131 CI has been passed upon the latest HDDS-2823. Please revise this PR, Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [hadoop-ozone] GlenGeng commented on a change in pull request #1314: HDDS-3988: DN can distinguish SCMCommand from stale leader SCM
Posted by GitBox <gi...@apache.org>.
GlenGeng commented on a change in pull request #1314:
URL: https://github.com/apache/hadoop-ozone/pull/1314#discussion_r486282159
##########
File path: hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/ha/MockSCMHAManager.java
##########
@@ -78,8 +79,8 @@ public void start() throws IOException {
* {@inheritDoc}
*/
@Override
- public boolean isLeader() {
- return isLeader;
+ public Optional<Long> isLeader() {
Review comment:
Good point. We need add a lot of test cases, including UT and acceptance test, before merging 2823 back to master.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] amaliujia commented on a change in pull request #1314: HDDS-3988: DN can distinguish SCMCommand from stale leader SCM
Posted by GitBox <gi...@apache.org>.
amaliujia commented on a change in pull request #1314:
URL: https://github.com/apache/hadoop-ozone/pull/1314#discussion_r486495949
##########
File path: hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/ha/MockSCMHAManager.java
##########
@@ -78,8 +79,8 @@ public void start() throws IOException {
* {@inheritDoc}
*/
@Override
- public boolean isLeader() {
- return isLeader;
+ public Optional<Long> isLeader() {
Review comment:
Agreed. I have been thinking about it and we might be some testing infrastructure to simulate these complicated consensus cases. E.g. split-brian.
OM HA might have done something similar.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] timmylicheng commented on a change in pull request #1314: HDDS-3988: DN can distinguish SCMCommand from stale leader SCM
Posted by GitBox <gi...@apache.org>.
timmylicheng commented on a change in pull request #1314:
URL: https://github.com/apache/hadoop-ozone/pull/1314#discussion_r473924140
##########
File path: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/StateContext.java
##########
@@ -470,6 +484,65 @@ public void execute(ExecutorService service, long time, TimeUnit unit)
}
}
+ /**
+ * After startup, datanode needs detect latest leader SCM before handling
+ * any SCMCommand, so that it won't be disturbed by stale leader SCM.
+ *
+ * The rule is: after majority SCMs are in HEARTBEAT state and has
+ * heard from leader SCMs (commandQueue is not empty), datanode will init
+ * termOfLeaderSCM with the max term found in commandQueue.
+ *
+ * The init process also works for non-HA mode. In that case, term of all
+ * SCMCommands will be 0.
+ */
+ void initTermOfLeaderSCM() {
+ // only init once
+ if (termOfLeaderSCM.isPresent()) {
+ return;
+ }
+
+ AtomicInteger scmNum = new AtomicInteger(0);
+ AtomicInteger activeScmNum = new AtomicInteger(0);
+
+ getParent().getConnectionManager().getValues()
+ .forEach(endpoint -> {
+ if (endpoint.isPassive()) {
+ return;
+ }
+ scmNum.incrementAndGet();
+ if (endpoint.getState()
+ == EndpointStateMachine.EndPointStates.HEARTBEAT) {
+ activeScmNum.incrementAndGet();
+ }
+ });
+
+ // majority SCMs should be in HEARTBEAT state.
+ if (activeScmNum.get() < scmNum.get() / 2 + 1) {
+ return;
+ }
+
+ // if commandQueue is not empty, init termOfLeaderSCM
+ // with the largest term found in commandQueue
+ commandQueue.stream()
+ .mapToLong(SCMCommand::getTerm)
+ .max()
+ .ifPresent(term -> termOfLeaderSCM = Optional.of(term));
+ }
+
+ /**
+ * monotonically increase termOfLeaderSCM.
+ * Always record the latest term that has seen.
+ */
+ void updateTermOfLeaderSCM(SCMCommand<?> command) {
+ if (!termOfLeaderSCM.isPresent()) {
+ LOG.error("should init termOfLeaderSCM before update it.");
+ return;
+ }
+
+ termOfLeaderSCM = Optional.of(
Review comment:
Shall we use AtomicLong for termOfLeaderSCM?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] amaliujia commented on a change in pull request #1314: HDDS-3988: DN can distinguish SCMCommand from stale leader SCM
Posted by GitBox <gi...@apache.org>.
amaliujia commented on a change in pull request #1314:
URL: https://github.com/apache/hadoop-ozone/pull/1314#discussion_r485371150
##########
File path: hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/ha/MockSCMHAManager.java
##########
@@ -78,8 +79,8 @@ public void start() throws IOException {
* {@inheritDoc}
*/
@Override
- public boolean isLeader() {
- return isLeader;
+ public Optional<Long> isLeader() {
Review comment:
another challenge is how to test such change. ideally we can have a minicluster setup with configurable nodes so we can simulate the scenario of split brian.
##########
File path: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/StateContext.java
##########
@@ -478,7 +551,26 @@ public void execute(ExecutorService service, long time, TimeUnit unit)
public SCMCommand getNextCommand() {
lock.lock();
try {
- return commandQueue.poll();
+ initTermOfLeaderSCM();
+ if (!termOfLeaderSCM.isPresent()) {
+ return null; // not ready yet
+ }
+
+ while (true) {
+ SCMCommand<?> command = commandQueue.poll();
+ if (command == null) {
+ return null;
+ }
+
+ updateTermOfLeaderSCM(command);
+ if (command.getTerm() == termOfLeaderSCM.get()) {
Review comment:
I am confused on when `termOfLeaderSCM` is updated to the newest leader term. Is `termOfLeaderSCM` updated in during leader selection?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] amaliujia commented on pull request #1314: HDDS-3988: DN can distinguish SCMCommand from stale leader SCM
Posted by GitBox <gi...@apache.org>.
amaliujia commented on pull request #1314:
URL: https://github.com/apache/hadoop-ozone/pull/1314#issuecomment-689290907
cc @amaliujia
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] timmylicheng commented on a change in pull request #1314: HDDS-3988: DN can distinguish SCMCommand from stale leader SCM
Posted by GitBox <gi...@apache.org>.
timmylicheng commented on a change in pull request #1314:
URL: https://github.com/apache/hadoop-ozone/pull/1314#discussion_r468973335
##########
File path: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/StateContext.java
##########
@@ -470,6 +484,65 @@ public void execute(ExecutorService service, long time, TimeUnit unit)
}
}
+ /**
+ * After startup, datanode needs detect latest leader SCM before handling
+ * any SCMCommand, so that it won't be disturbed by stale leader SCM.
+ *
+ * The rule is: after majority SCMs are in HEARTBEAT state and has
+ * heard from leader SCMs (commandQueue is not empty), datanode will init
+ * termOfLeaderSCM with the max term found in commandQueue.
+ *
+ * The init process also works for non-HA mode. In that case, term of all
+ * SCMCommands will be 0.
+ */
+ void initTermOfLeaderSCM() {
+ // only init once
+ if (termOfLeaderSCM.isPresent()) {
+ return;
+ }
+
+ AtomicInteger scmNum = new AtomicInteger(0);
+ AtomicInteger activeScmNum = new AtomicInteger(0);
+
+ getParent().getConnectionManager().getValues()
+ .forEach(endpoint -> {
+ if (endpoint.isPassive()) {
+ return;
+ }
+ scmNum.incrementAndGet();
+ if (endpoint.getState()
+ == EndpointStateMachine.EndPointStates.HEARTBEAT) {
+ activeScmNum.incrementAndGet();
+ }
+ });
+
+ // majority SCMs should be in HEARTBEAT state.
+ if (activeScmNum.get() < scmNum.get() / 2 + 1) {
+ return;
+ }
+
+ // if commandQueue is not empty, init termOfLeaderSCM
+ // with the largest term found in commandQueue
+ commandQueue.stream()
+ .mapToLong(SCMCommand::getTerm)
+ .max()
+ .ifPresent(term -> termOfLeaderSCM = Optional.of(term));
+ }
+
+ /**
+ * monotonically increase termOfLeaderSCM.
+ * Always record the latest term that has seen.
+ */
+ void updateTermOfLeaderSCM(SCMCommand<?> command) {
Review comment:
Can be a private method
##########
File path: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/StateContext.java
##########
@@ -470,6 +484,65 @@ public void execute(ExecutorService service, long time, TimeUnit unit)
}
}
+ /**
+ * After startup, datanode needs detect latest leader SCM before handling
+ * any SCMCommand, so that it won't be disturbed by stale leader SCM.
+ *
+ * The rule is: after majority SCMs are in HEARTBEAT state and has
+ * heard from leader SCMs (commandQueue is not empty), datanode will init
+ * termOfLeaderSCM with the max term found in commandQueue.
+ *
+ * The init process also works for non-HA mode. In that case, term of all
+ * SCMCommands will be 0.
+ */
+ void initTermOfLeaderSCM() {
Review comment:
Can be a private method
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] timmylicheng commented on a change in pull request #1314: HDDS-3988: DN can distinguish SCMCommand from stale leader SCM
Posted by GitBox <gi...@apache.org>.
timmylicheng commented on a change in pull request #1314:
URL: https://github.com/apache/hadoop-ozone/pull/1314#discussion_r473935146
##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeManager.java
##########
@@ -130,6 +135,14 @@ public SCMNodeManager(OzoneConfiguration conf,
this.useHostname = conf.getBoolean(
DFSConfigKeysLegacy.DFS_DATANODE_USE_DN_HOSTNAME,
DFSConfigKeysLegacy.DFS_DATANODE_USE_DN_HOSTNAME_DEFAULT);
+ this.scmhaManager = scmhaManager;
+ }
+
+ public SCMNodeManager(OzoneConfiguration conf,
Review comment:
Any case SCMNodeManager would not require SCMHAManager?
Note that PipelineManager will always rely on SCMHAManager
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] GlenGeng commented on a change in pull request #1314: HDDS-3988: DN can distinguish SCMCommand from stale leader SCM
Posted by GitBox <gi...@apache.org>.
GlenGeng commented on a change in pull request #1314:
URL: https://github.com/apache/hadoop-ozone/pull/1314#discussion_r486281665
##########
File path: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/StateContext.java
##########
@@ -478,7 +551,26 @@ public void execute(ExecutorService service, long time, TimeUnit unit)
public SCMCommand getNextCommand() {
lock.lock();
try {
- return commandQueue.poll();
+ initTermOfLeaderSCM();
+ if (!termOfLeaderSCM.isPresent()) {
+ return null; // not ready yet
+ }
+
+ while (true) {
+ SCMCommand<?> command = commandQueue.poll();
+ if (command == null) {
+ return null;
+ }
+
+ updateTermOfLeaderSCM(command);
+ if (command.getTerm() == termOfLeaderSCM.get()) {
Review comment:
Do you mean whether `termOfLeaderSCM` is updated in leader election of SCM ? No, it won't. Datanode detects the latest SCM term by heartbeat with SCMs, whose interval is larger than 30s.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org