You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2020/08/11 11:24:37 UTC
[GitHub] [hadoop-ozone] GlenGeng opened a new pull request #1314: HDDS-3988: DN can distinguish SCMCommand from stale leader SCM
GlenGeng opened a new pull request #1314:
URL: https://github.com/apache/hadoop-ozone/pull/1314
## What changes were proposed in this pull request?
As part of SCMCommand SCM will also send its current term, which will be used in Datanode to identify if the command was sent by the latest leader SCM.
Datanode will maintain the highest term that it has seen and compare it with the term that is received as part of SCMCommand.
If the term in the Datanode and SCMCommand are same, the command is added to the command queue for processing.
If the term in the Datanode is less than the term received in SCMCommand, Datanode will update its term and add the command to the command queue for processing.
If the term in the Datanode is greater than the term received in SCMCommand, Datanode will ignore the command.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-3988
## How was this patch tested?
CI
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] amaliujia commented on a change in pull request #1314: HDDS-3988: DN can distinguish SCMCommand from stale leader SCM
Posted by GitBox <gi...@apache.org>.
amaliujia commented on a change in pull request #1314:
URL: https://github.com/apache/hadoop-ozone/pull/1314#discussion_r486495949
##########
File path: hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/ha/MockSCMHAManager.java
##########
@@ -78,8 +79,8 @@ public void start() throws IOException {
* {@inheritDoc}
*/
@Override
- public boolean isLeader() {
- return isLeader;
+ public Optional<Long> isLeader() {
Review comment:
Agreed. I have been thinking about it and we might be some testing infrastructure to simulate these complicated consensus cases. E.g. split-brian.
OM HA might have done something similar.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] timmylicheng commented on a change in pull request #1314: HDDS-3988: DN can distinguish SCMCommand from stale leader SCM
Posted by GitBox <gi...@apache.org>.
timmylicheng commented on a change in pull request #1314:
URL: https://github.com/apache/hadoop-ozone/pull/1314#discussion_r473924140
##########
File path: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/StateContext.java
##########
@@ -470,6 +484,65 @@ public void execute(ExecutorService service, long time, TimeUnit unit)
}
}
+ /**
+ * After startup, datanode needs detect latest leader SCM before handling
+ * any SCMCommand, so that it won't be disturbed by stale leader SCM.
+ *
+ * The rule is: after majority SCMs are in HEARTBEAT state and has
+ * heard from leader SCMs (commandQueue is not empty), datanode will init
+ * termOfLeaderSCM with the max term found in commandQueue.
+ *
+ * The init process also works for non-HA mode. In that case, term of all
+ * SCMCommands will be 0.
+ */
+ void initTermOfLeaderSCM() {
+ // only init once
+ if (termOfLeaderSCM.isPresent()) {
+ return;
+ }
+
+ AtomicInteger scmNum = new AtomicInteger(0);
+ AtomicInteger activeScmNum = new AtomicInteger(0);
+
+ getParent().getConnectionManager().getValues()
+ .forEach(endpoint -> {
+ if (endpoint.isPassive()) {
+ return;
+ }
+ scmNum.incrementAndGet();
+ if (endpoint.getState()
+ == EndpointStateMachine.EndPointStates.HEARTBEAT) {
+ activeScmNum.incrementAndGet();
+ }
+ });
+
+ // majority SCMs should be in HEARTBEAT state.
+ if (activeScmNum.get() < scmNum.get() / 2 + 1) {
+ return;
+ }
+
+ // if commandQueue is not empty, init termOfLeaderSCM
+ // with the largest term found in commandQueue
+ commandQueue.stream()
+ .mapToLong(SCMCommand::getTerm)
+ .max()
+ .ifPresent(term -> termOfLeaderSCM = Optional.of(term));
+ }
+
+ /**
+ * monotonically increase termOfLeaderSCM.
+ * Always record the latest term that has seen.
+ */
+ void updateTermOfLeaderSCM(SCMCommand<?> command) {
+ if (!termOfLeaderSCM.isPresent()) {
+ LOG.error("should init termOfLeaderSCM before update it.");
+ return;
+ }
+
+ termOfLeaderSCM = Optional.of(
Review comment:
Shall we use AtomicLong for termOfLeaderSCM?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] GlenGeng commented on a change in pull request #1314: HDDS-3988: DN can distinguish SCMCommand from stale leader SCM
Posted by GitBox <gi...@apache.org>.
GlenGeng commented on a change in pull request #1314:
URL: https://github.com/apache/hadoop-ozone/pull/1314#discussion_r486282159
##########
File path: hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/ha/MockSCMHAManager.java
##########
@@ -78,8 +79,8 @@ public void start() throws IOException {
* {@inheritDoc}
*/
@Override
- public boolean isLeader() {
- return isLeader;
+ public Optional<Long> isLeader() {
Review comment:
Good point. We need add a lot of test cases, including UT and acceptance test, before merging 2823 back to master.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] amaliujia commented on a change in pull request #1314: HDDS-3988: DN can distinguish SCMCommand from stale leader SCM
Posted by GitBox <gi...@apache.org>.
amaliujia commented on a change in pull request #1314:
URL: https://github.com/apache/hadoop-ozone/pull/1314#discussion_r485371150
##########
File path: hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/ha/MockSCMHAManager.java
##########
@@ -78,8 +79,8 @@ public void start() throws IOException {
* {@inheritDoc}
*/
@Override
- public boolean isLeader() {
- return isLeader;
+ public Optional<Long> isLeader() {
Review comment:
another challenge is how to test such change. ideally we can have a minicluster setup with configurable nodes so we can simulate the scenario of split brian.
##########
File path: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/StateContext.java
##########
@@ -478,7 +551,26 @@ public void execute(ExecutorService service, long time, TimeUnit unit)
public SCMCommand getNextCommand() {
lock.lock();
try {
- return commandQueue.poll();
+ initTermOfLeaderSCM();
+ if (!termOfLeaderSCM.isPresent()) {
+ return null; // not ready yet
+ }
+
+ while (true) {
+ SCMCommand<?> command = commandQueue.poll();
+ if (command == null) {
+ return null;
+ }
+
+ updateTermOfLeaderSCM(command);
+ if (command.getTerm() == termOfLeaderSCM.get()) {
Review comment:
I am confused on when `termOfLeaderSCM` is updated to the newest leader term. Is `termOfLeaderSCM` updated in during leader selection?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] amaliujia commented on pull request #1314: HDDS-3988: DN can distinguish SCMCommand from stale leader SCM
Posted by GitBox <gi...@apache.org>.
amaliujia commented on pull request #1314:
URL: https://github.com/apache/hadoop-ozone/pull/1314#issuecomment-689290907
cc @amaliujia
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] timmylicheng commented on a change in pull request #1314: HDDS-3988: DN can distinguish SCMCommand from stale leader SCM
Posted by GitBox <gi...@apache.org>.
timmylicheng commented on a change in pull request #1314:
URL: https://github.com/apache/hadoop-ozone/pull/1314#discussion_r468973335
##########
File path: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/StateContext.java
##########
@@ -470,6 +484,65 @@ public void execute(ExecutorService service, long time, TimeUnit unit)
}
}
+ /**
+ * After startup, datanode needs detect latest leader SCM before handling
+ * any SCMCommand, so that it won't be disturbed by stale leader SCM.
+ *
+ * The rule is: after majority SCMs are in HEARTBEAT state and has
+ * heard from leader SCMs (commandQueue is not empty), datanode will init
+ * termOfLeaderSCM with the max term found in commandQueue.
+ *
+ * The init process also works for non-HA mode. In that case, term of all
+ * SCMCommands will be 0.
+ */
+ void initTermOfLeaderSCM() {
+ // only init once
+ if (termOfLeaderSCM.isPresent()) {
+ return;
+ }
+
+ AtomicInteger scmNum = new AtomicInteger(0);
+ AtomicInteger activeScmNum = new AtomicInteger(0);
+
+ getParent().getConnectionManager().getValues()
+ .forEach(endpoint -> {
+ if (endpoint.isPassive()) {
+ return;
+ }
+ scmNum.incrementAndGet();
+ if (endpoint.getState()
+ == EndpointStateMachine.EndPointStates.HEARTBEAT) {
+ activeScmNum.incrementAndGet();
+ }
+ });
+
+ // majority SCMs should be in HEARTBEAT state.
+ if (activeScmNum.get() < scmNum.get() / 2 + 1) {
+ return;
+ }
+
+ // if commandQueue is not empty, init termOfLeaderSCM
+ // with the largest term found in commandQueue
+ commandQueue.stream()
+ .mapToLong(SCMCommand::getTerm)
+ .max()
+ .ifPresent(term -> termOfLeaderSCM = Optional.of(term));
+ }
+
+ /**
+ * monotonically increase termOfLeaderSCM.
+ * Always record the latest term that has seen.
+ */
+ void updateTermOfLeaderSCM(SCMCommand<?> command) {
Review comment:
Can be a private method
##########
File path: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/StateContext.java
##########
@@ -470,6 +484,65 @@ public void execute(ExecutorService service, long time, TimeUnit unit)
}
}
+ /**
+ * After startup, datanode needs detect latest leader SCM before handling
+ * any SCMCommand, so that it won't be disturbed by stale leader SCM.
+ *
+ * The rule is: after majority SCMs are in HEARTBEAT state and has
+ * heard from leader SCMs (commandQueue is not empty), datanode will init
+ * termOfLeaderSCM with the max term found in commandQueue.
+ *
+ * The init process also works for non-HA mode. In that case, term of all
+ * SCMCommands will be 0.
+ */
+ void initTermOfLeaderSCM() {
Review comment:
Can be a private method
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] timmylicheng commented on a change in pull request #1314: HDDS-3988: DN can distinguish SCMCommand from stale leader SCM
Posted by GitBox <gi...@apache.org>.
timmylicheng commented on a change in pull request #1314:
URL: https://github.com/apache/hadoop-ozone/pull/1314#discussion_r473935146
##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeManager.java
##########
@@ -130,6 +135,14 @@ public SCMNodeManager(OzoneConfiguration conf,
this.useHostname = conf.getBoolean(
DFSConfigKeysLegacy.DFS_DATANODE_USE_DN_HOSTNAME,
DFSConfigKeysLegacy.DFS_DATANODE_USE_DN_HOSTNAME_DEFAULT);
+ this.scmhaManager = scmhaManager;
+ }
+
+ public SCMNodeManager(OzoneConfiguration conf,
Review comment:
Any case SCMNodeManager would not require SCMHAManager?
Note that PipelineManager will always rely on SCMHAManager
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] GlenGeng commented on a change in pull request #1314: HDDS-3988: DN can distinguish SCMCommand from stale leader SCM
Posted by GitBox <gi...@apache.org>.
GlenGeng commented on a change in pull request #1314:
URL: https://github.com/apache/hadoop-ozone/pull/1314#discussion_r486281665
##########
File path: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/StateContext.java
##########
@@ -478,7 +551,26 @@ public void execute(ExecutorService service, long time, TimeUnit unit)
public SCMCommand getNextCommand() {
lock.lock();
try {
- return commandQueue.poll();
+ initTermOfLeaderSCM();
+ if (!termOfLeaderSCM.isPresent()) {
+ return null; // not ready yet
+ }
+
+ while (true) {
+ SCMCommand<?> command = commandQueue.poll();
+ if (command == null) {
+ return null;
+ }
+
+ updateTermOfLeaderSCM(command);
+ if (command.getTerm() == termOfLeaderSCM.get()) {
Review comment:
Do you mean whether `termOfLeaderSCM` is updated in leader election of SCM ? No, it won't. Datanode detects the latest SCM term by heartbeat with SCMs, whose interval is larger than 30s.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org