You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2021/09/29 10:00:13 UTC
[GitHub] [ozone] siddhantsangwan opened a new pull request #2695: HDDS-5797. Support setting Datanode Reserved Space in MiniOzoneCluster.
siddhantsangwan opened a new pull request #2695:
URL: https://github.com/apache/ozone/pull/2695
## What changes were proposed in this pull request?
Support setting `org.apache.hadoop.hdds.scm.ScmConfigKeys#HDDS_DATANODE_DIR_DU_RESERVED` for each volume in each Datanode in `MiniOzoneCluster` (sets the same reserved space for each volume).
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-5797
## How was this patch tested?
Passed `TestMiniOzoneCluster`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on a change in pull request #2695: HDDS-5797. Support setting Datanode Reserved Space in MiniOzoneCluster.
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on a change in pull request #2695:
URL: https://github.com/apache/ozone/pull/2695#discussion_r719159728
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneCluster.java
##########
@@ -315,6 +315,7 @@ void restartHddsDatanode(DatanodeDetails dn, boolean waitForDatanode)
protected Optional<String> omId = Optional.empty();
protected Boolean randomContainerPort = true;
+ protected Optional<String> datanodeReservedSpace = Optional.of("0B");
Review comment:
I think a better default value would be:
```suggestion
protected Optional<String> datanodeReservedSpace = Optional.empty();
```
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneCluster.java
##########
@@ -38,6 +31,13 @@
import org.apache.hadoop.security.authentication.client.AuthenticationException;
import org.apache.ozone.test.GenericTestUtils;
+import java.io.IOException;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.UUID;
+import java.util.concurrent.TimeoutException;
Review comment:
Nit: please avoid moving around imports unnecessarily.
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneClusterImpl.java
##########
@@ -818,6 +818,9 @@ protected String getSCMAddresses(List<StorageContainerManager> scms) {
Path dir = Paths.get(datanodeBaseDir, "data-" + j, "containers");
Files.createDirectories(dir);
dataDirs.add(dir.toString());
+ datanodeReservedSpace.ifPresent(
+ s -> dnConf.set(ScmConfigKeys.HDDS_DATANODE_DIR_DU_RESERVED,
+ dir + ":" + s));
Review comment:
Since the same config property is set for each volume, reserved space would only be set for the last one. It should be set once, outside the loop, similar to `HDDS_DATANODE_DIR_KEY`.
Can you please add an assertion in `TestMiniOzoneCluster#testMultipleDataDirs` to verify behavior?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] siddhantsangwan commented on pull request #2695: HDDS-5797. Support setting Datanode Reserved Space in MiniOzoneCluster.
Posted by GitBox <gi...@apache.org>.
siddhantsangwan commented on pull request #2695:
URL: https://github.com/apache/ozone/pull/2695#issuecomment-931552365
@adoroszlai Thanks for the review. I've updated the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] siddhantsangwan commented on a change in pull request #2695: HDDS-5797. Support setting Datanode Reserved Space in MiniOzoneCluster.
Posted by GitBox <gi...@apache.org>.
siddhantsangwan commented on a change in pull request #2695:
URL: https://github.com/apache/ozone/pull/2695#discussion_r719624864
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneClusterImpl.java
##########
@@ -818,6 +818,9 @@ protected String getSCMAddresses(List<StorageContainerManager> scms) {
Path dir = Paths.get(datanodeBaseDir, "data-" + j, "containers");
Files.createDirectories(dir);
dataDirs.add(dir.toString());
+ datanodeReservedSpace.ifPresent(
+ s -> dnConf.set(ScmConfigKeys.HDDS_DATANODE_DIR_DU_RESERVED,
+ dir + ":" + s));
Review comment:
For the unit test: Is there a better way than adding a `reservedInBytes` getter to `VolumeInfo` for testing, and then asserting its value for each volume?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on a change in pull request #2695: HDDS-5797. Support setting Datanode Reserved Space in MiniOzoneCluster.
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on a change in pull request #2695:
URL: https://github.com/apache/ozone/pull/2695#discussion_r719636562
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneClusterImpl.java
##########
@@ -818,6 +818,9 @@ protected String getSCMAddresses(List<StorageContainerManager> scms) {
Path dir = Paths.get(datanodeBaseDir, "data-" + j, "containers");
Files.createDirectories(dir);
dataDirs.add(dir.toString());
+ datanodeReservedSpace.ifPresent(
+ s -> dnConf.set(ScmConfigKeys.HDDS_DATANODE_DIR_DU_RESERVED,
+ dir + ":" + s));
Review comment:
> Is there a better way than adding a `reservedInBytes` getter to `VolumeInfo` for testing, and then asserting its value for each volume?
Sounds good enough to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on pull request #2695: HDDS-5797. Support setting Datanode Reserved Space in MiniOzoneCluster.
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on pull request #2695:
URL: https://github.com/apache/ozone/pull/2695#issuecomment-932456083
Thanks @siddhantsangwan for the patch, @JacksonYao287 for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai merged pull request #2695: HDDS-5797. Support setting Datanode Reserved Space in MiniOzoneCluster.
Posted by GitBox <gi...@apache.org>.
adoroszlai merged pull request #2695:
URL: https://github.com/apache/ozone/pull/2695
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org