You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "kostacie (via GitHub)" <gi...@apache.org> on 2025/04/24 14:45:59 UTC
[PR] HDDS-12378. Change default hdds.scm.safemode.min.datanode to 3 [ozone]
kostacie opened a new pull request, #8331:
URL: https://github.com/apache/ozone/pull/8331
## What changes were proposed in this pull request?
This PR changes the value of `hdds.scm.safemode.min.datanode` to 3 in `HddsConfigKeys`.
Currently, the default value of `hdds.scm.safemode.min.datanode` is set to 1.
This means that an user can deploy an Ozone cluster (with default configuration) with just one datanode and the cluster will come out of safemode. After coming out of safemode the writes with three way replication will fail.
By changing the default value of `hdds.scm.safemode.min.datanode` from 1 to 3, we will make sure that the cluster deployed with default configuration will not fail during write once it's out of safemode.
The users who wants to run Ozone with just one datanode can manually update the config.
## What is the link to the Apache JIRA
[HDDS-12378](https://issues.apache.org/jira/browse/HDDS-12378)
## How was this patch tested?
CI:
https://github.com/kostacie/ozone/actions/runs/14641406750
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12378. Change default hdds.scm.safemode.min.datanode to 3 [ozone]
Posted by "kostacie (via GitHub)" <gi...@apache.org>.
kostacie commented on code in PR #8331:
URL: https://github.com/apache/ozone/pull/8331#discussion_r2060692346
##########
hadoop-hdds/docs/content/concept/StorageContainerManager.md:
##########
@@ -93,6 +93,6 @@ key | default | description
----|---------|------------
ozone.scm.container.size | 5GB | Default container size used by Ozone
ozone.scm.block.size | 256MB | The default size of a data block.
-hdds.scm.safemode.min.datanode | 1 | Minimum number of datanodes to start the real work.
+hdds.scm.safemode.min.datanode | 3 | Minimum number of datanodes to start the real work.
Review Comment:
Yes, sure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12378. Change default hdds.scm.safemode.min.datanode to 3 [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #8331:
URL: https://github.com/apache/ozone/pull/8331#issuecomment-2833189079
```
Basic Freon smoketest for one datanode | FAIL |
255 != 0
--
Single Node :: Smoketest for one datanode | FAIL |
1 test, 0 passed, 1 failed
--
ERROR: Test execution of ozonescripts/test.sh is FAILED!!!!
```
Need to set `hdds.scm.safemode.min.datanode=1` in `ozonescripts`:
https://github.com/apache/ozone/blob/cf1fb88740c4465f17070d9a61039d5c5a169683/hadoop-ozone/dist/src/main/compose/ozonescripts/docker-config#L26
Please also check unit test failures:
```
org.apache.hadoop.hdds.scm.safemode.TestHealthyPipelineSafeModeRule
org.apache.hadoop.hdds.scm.safemode.TestSCMSafeModeManager
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12378. Change default hdds.scm.safemode.min.datanode to 3 [ozone]
Posted by "nandakumar131 (via GitHub)" <gi...@apache.org>.
nandakumar131 commented on code in PR #8331:
URL: https://github.com/apache/ozone/pull/8331#discussion_r2068390762
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/safemode/DataNodeSafeModeRule.java:
##########
@@ -66,7 +69,17 @@ protected boolean validate() {
if (validateBasedOnReportProcessing()) {
return registeredDns >= requiredDns;
}
- return nodeManager.getNodes(NodeStatus.inServiceHealthy()).size() >= requiredDns;
+
+ int healthyCount = nodeManager.getNodes(NodeStatus.inServiceHealthy()).size();
+ int healthyReadOnlyCount = nodeManager.getNodes(NodeStatus.inServiceHealthyReadOnly()).size();
+ int staleCount = nodeManager.getNodes(NodeStatus.inServiceStale()).size();
+
+ if (healthyCount + healthyReadOnlyCount + staleCount == 1) {
+ LOG.warn("Only one Datanode is available in the cluster. " +
+ "Consider setting 'hdds.scm.safemode.min.datanode=1' in the configuration.");
+ }
Review Comment:
SCM doesn't remember Datanode list across SCM restart. So during SCM restart, initially the datanode list in SCM will be empty and at some point the first Datanode will register making `(healthyCount + healthyReadOnlyCount + staleCount == 1)` to true and printing the `warn` log message.
We will end up printing this `warn` log message during every SCM start-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12378. Change default hdds.scm.safemode.min.datanode to 3 [ozone]
Posted by "kostacie (via GitHub)" <gi...@apache.org>.
kostacie commented on code in PR #8331:
URL: https://github.com/apache/ozone/pull/8331#discussion_r2060711380
##########
hadoop-hdds/docs/content/concept/StorageContainerManager.md:
##########
@@ -93,6 +93,6 @@ key | default | description
----|---------|------------
ozone.scm.container.size | 5GB | Default container size used by Ozone
ozone.scm.block.size | 256MB | The default size of a data block.
-hdds.scm.safemode.min.datanode | 1 | Minimum number of datanodes to start the real work.
+hdds.scm.safemode.min.datanode | 3 | Minimum number of datanodes to start the real work.
ozone.scm.http-address | 0.0.0.0:9876 | HTTP address of the SCM server
-ozone.metadata.dirs | none | Directory to store persisted data (RocksDB).
\ No newline at end of file
+ozone.metadata.dirs | none | Directory to store persisted data (RocksDB).
Review Comment:
For some reason I can't revert that as it changes itself and adds an extra line.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12378. Change default hdds.scm.safemode.min.datanode to 3 [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #8331:
URL: https://github.com/apache/ozone/pull/8331#issuecomment-2839133316
@aryangupta1998 @myskov would you like to take another look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12378. Change default hdds.scm.safemode.min.datanode to 3 [ozone]
Posted by "myskov (via GitHub)" <gi...@apache.org>.
myskov commented on PR #8331:
URL: https://github.com/apache/ozone/pull/8331#issuecomment-2830345552
> then we should display a warn message to the user to update the "hdds.scm.safemode.min.datanode" to 1.
I'm not sure we should ever suggest user to set `hdds.scm.safemode.min.datanode` to 1. Even README guide suggest running 3 datanodes
```
cd compose/ozone
docker-compose up -d --scale datanode=3
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12378. Change default hdds.scm.safemode.min.datanode to 3 [ozone]
Posted by "nandakumar131 (via GitHub)" <gi...@apache.org>.
nandakumar131 commented on code in PR #8331:
URL: https://github.com/apache/ozone/pull/8331#discussion_r2068390762
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/safemode/DataNodeSafeModeRule.java:
##########
@@ -66,7 +69,17 @@ protected boolean validate() {
if (validateBasedOnReportProcessing()) {
return registeredDns >= requiredDns;
}
- return nodeManager.getNodes(NodeStatus.inServiceHealthy()).size() >= requiredDns;
+
+ int healthyCount = nodeManager.getNodes(NodeStatus.inServiceHealthy()).size();
+ int healthyReadOnlyCount = nodeManager.getNodes(NodeStatus.inServiceHealthyReadOnly()).size();
+ int staleCount = nodeManager.getNodes(NodeStatus.inServiceStale()).size();
+
+ if (healthyCount + healthyReadOnlyCount + staleCount == 1) {
+ LOG.warn("Only one Datanode is available in the cluster. " +
+ "Consider setting 'hdds.scm.safemode.min.datanode=1' in the configuration.");
+ }
Review Comment:
SCM doesn't remember Datanode list across SCM restart. So during SCM restart, initially the datanode list in SCM will be empty and at some point the first Datanode will register making `(healthyCount + healthyReadOnlyCount + staleCount == 1)` to true and print the `warn` log message.
We will end up printing this `warn` log message during every SCM start-up, which is incorrect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12378. Change default hdds.scm.safemode.min.datanode to 3 [ozone]
Posted by "aryangupta1998 (via GitHub)" <gi...@apache.org>.
aryangupta1998 commented on PR #8331:
URL: https://github.com/apache/ozone/pull/8331#issuecomment-2830359665
> I'm not sure we should ever suggest user to set hdds.scm.safemode.min.datanode to 1. Even README guide suggest running 3 datanodes
I meant to say we should only suggest when there's only one DN, because now we are setting the default value to 3 and if we have one DN, then SCM will not comeout of safemode until user sets hdds.scm.safemode.min.datanode to 1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12378. Change default hdds.scm.safemode.min.datanode to 3 [ozone]
Posted by "nandakumar131 (via GitHub)" <gi...@apache.org>.
nandakumar131 commented on code in PR #8331:
URL: https://github.com/apache/ozone/pull/8331#discussion_r2100793795
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/safemode/DataNodeSafeModeRule.java:
##########
@@ -66,7 +69,17 @@ protected boolean validate() {
if (validateBasedOnReportProcessing()) {
return registeredDns >= requiredDns;
}
- return nodeManager.getNodes(NodeStatus.inServiceHealthy()).size() >= requiredDns;
+
+ int healthyCount = nodeManager.getNodes(NodeStatus.inServiceHealthy()).size();
+ int healthyReadOnlyCount = nodeManager.getNodes(NodeStatus.inServiceHealthyReadOnly()).size();
+ int staleCount = nodeManager.getNodes(NodeStatus.inServiceStale()).size();
+
+ if (healthyCount + healthyReadOnlyCount + staleCount == 1) {
+ LOG.warn("Only one Datanode is available in the cluster. " +
+ "Consider setting 'hdds.scm.safemode.min.datanode=1' in the configuration.");
+ }
Review Comment:
We don't need any logging for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12378. Change default hdds.scm.safemode.min.datanode to 3 [ozone]
Posted by "myskov (via GitHub)" <gi...@apache.org>.
myskov commented on code in PR #8331:
URL: https://github.com/apache/ozone/pull/8331#discussion_r2060186972
##########
hadoop-hdds/docs/content/concept/StorageContainerManager.md:
##########
@@ -93,6 +93,6 @@ key | default | description
----|---------|------------
ozone.scm.container.size | 5GB | Default container size used by Ozone
ozone.scm.block.size | 256MB | The default size of a data block.
-hdds.scm.safemode.min.datanode | 1 | Minimum number of datanodes to start the real work.
+hdds.scm.safemode.min.datanode | 3 | Minimum number of datanodes to start the real work.
Review Comment:
There's also a Chinese version of the doc. Could you please make changes there too?
##########
hadoop-hdds/docs/content/concept/StorageContainerManager.md:
##########
@@ -93,6 +93,6 @@ key | default | description
----|---------|------------
ozone.scm.container.size | 5GB | Default container size used by Ozone
ozone.scm.block.size | 256MB | The default size of a data block.
-hdds.scm.safemode.min.datanode | 1 | Minimum number of datanodes to start the real work.
+hdds.scm.safemode.min.datanode | 3 | Minimum number of datanodes to start the real work.
ozone.scm.http-address | 0.0.0.0:9876 | HTTP address of the SCM server
-ozone.metadata.dirs | none | Directory to store persisted data (RocksDB).
\ No newline at end of file
+ozone.metadata.dirs | none | Directory to store persisted data (RocksDB).
Review Comment:
please revert irrelevant change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12378. Change default hdds.scm.safemode.min.datanode to 3 [ozone]
Posted by "nandakumar131 (via GitHub)" <gi...@apache.org>.
nandakumar131 merged PR #8331:
URL: https://github.com/apache/ozone/pull/8331
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12378. Change default hdds.scm.safemode.min.datanode to 3 [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #8331:
URL: https://github.com/apache/ozone/pull/8331#discussion_r2062363261
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/safemode/DataNodeSafeModeRule.java:
##########
@@ -27,6 +27,8 @@
import org.apache.hadoop.hdds.scm.server.SCMDatanodeProtocolServer.NodeRegistrationContainerReport;
import org.apache.hadoop.hdds.server.events.EventQueue;
import org.apache.hadoop.hdds.server.events.TypedEvent;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
Review Comment:
Please use `org.slf4j.Logger`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12378. Change default hdds.scm.safemode.min.datanode to 3 [ozone]
Posted by "kostacie (via GitHub)" <gi...@apache.org>.
kostacie commented on code in PR #8331:
URL: https://github.com/apache/ozone/pull/8331#discussion_r2073162332
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/safemode/DataNodeSafeModeRule.java:
##########
@@ -66,7 +69,17 @@ protected boolean validate() {
if (validateBasedOnReportProcessing()) {
return registeredDns >= requiredDns;
}
- return nodeManager.getNodes(NodeStatus.inServiceHealthy()).size() >= requiredDns;
+
+ int healthyCount = nodeManager.getNodes(NodeStatus.inServiceHealthy()).size();
+ int healthyReadOnlyCount = nodeManager.getNodes(NodeStatus.inServiceHealthyReadOnly()).size();
+ int staleCount = nodeManager.getNodes(NodeStatus.inServiceStale()).size();
+
+ if (healthyCount + healthyReadOnlyCount + staleCount == 1) {
+ LOG.warn("Only one Datanode is available in the cluster. " +
+ "Consider setting 'hdds.scm.safemode.min.datanode=1' in the configuration.");
+ }
Review Comment:
Thank you for the review @nandakumar131. Should we add a flag that would indicate that the log message has already been printed? Would it be a good solution?
Or would you recommend anything to avoid such problem?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12378. Change default hdds.scm.safemode.min.datanode to 3 [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #8331:
URL: https://github.com/apache/ozone/pull/8331#discussion_r2096963203
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/safemode/DataNodeSafeModeRule.java:
##########
@@ -66,7 +69,17 @@ protected boolean validate() {
if (validateBasedOnReportProcessing()) {
return registeredDns >= requiredDns;
}
- return nodeManager.getNodes(NodeStatus.inServiceHealthy()).size() >= requiredDns;
+
+ int healthyCount = nodeManager.getNodes(NodeStatus.inServiceHealthy()).size();
+ int healthyReadOnlyCount = nodeManager.getNodes(NodeStatus.inServiceHealthyReadOnly()).size();
+ int staleCount = nodeManager.getNodes(NodeStatus.inServiceStale()).size();
+
+ if (healthyCount + healthyReadOnlyCount + staleCount == 1) {
+ LOG.warn("Only one Datanode is available in the cluster. " +
+ "Consider setting 'hdds.scm.safemode.min.datanode=1' in the configuration.");
+ }
Review Comment:
I guess we can remove the warning, unless @nandakumar131 has some other suggestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12378. Change default hdds.scm.safemode.min.datanode to 3 [ozone]
Posted by "nandakumar131 (via GitHub)" <gi...@apache.org>.
nandakumar131 commented on PR #8331:
URL: https://github.com/apache/ozone/pull/8331#issuecomment-2898683938
Thanks @kostacie for the contribution.
Thanks to @aryangupta1998, @myskov & @adoroszlai for the reviews.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org