You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2020/04/03 12:22:07 UTC
[GitHub] [hadoop-ozone] mukul1987 opened a new pull request #763: HDDS-3302.
Add ability in MiniOzoneCluster to create multiple storage directories.
mukul1987 opened a new pull request #763: HDDS-3302. Add ability in MiniOzoneCluster to create multiple storage directories.
URL: https://github.com/apache/hadoop-ozone/pull/763
## What changes were proposed in this pull request?
Add functionality in MiniOzoneCluster to create multiple datanode data directory per datanode.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-3302
## How was this patch tested?
Added new unit test in TestMiniOzoneCluster.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request
#763: HDDS-3302. Add ability in MiniOzoneCluster to create multiple storage
directories.
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on a change in pull request #763: HDDS-3302. Add ability in MiniOzoneCluster to create multiple storage directories.
URL: https://github.com/apache/hadoop-ozone/pull/763#discussion_r405343181
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestMiniOzoneCluster.java
##########
@@ -58,14 +58,12 @@
import org.junit.AfterClass;
import org.junit.Assert;
import org.junit.BeforeClass;
-import org.junit.Ignore;
import org.junit.Test;
import org.yaml.snakeyaml.Yaml;
/**
* Test cases for mini ozone cluster.
*/
-@Ignore
Review comment:
> It can be because a flakiness. In this case it would be great to have at least 3-5 green builds before the merge .
Running 20x, passed 5x so far:
https://github.com/adoroszlai/hadoop-ozone/runs/570025575
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek closed pull request #763: HDDS-3302. Add
ability in MiniOzoneCluster to create multiple storage directories.
Posted by GitBox <gi...@apache.org>.
elek closed pull request #763: HDDS-3302. Add ability in MiniOzoneCluster to create multiple storage directories.
URL: https://github.com/apache/hadoop-ozone/pull/763
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request
#763: HDDS-3302. Add ability in MiniOzoneCluster to create multiple storage
directories.
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on a change in pull request #763: HDDS-3302. Add ability in MiniOzoneCluster to create multiple storage directories.
URL: https://github.com/apache/hadoop-ozone/pull/763#discussion_r404715224
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestMiniOzoneCluster.java
##########
@@ -58,14 +58,12 @@
import org.junit.AfterClass;
import org.junit.Assert;
import org.junit.BeforeClass;
-import org.junit.Ignore;
import org.junit.Test;
import org.yaml.snakeyaml.Yaml;
/**
* Test cases for mini ozone cluster.
*/
-@Ignore
Review comment:
[Disabled on Dec 27](https://github.com/apache/hadoop-ozone/commit/9000323e3741a39cc8b645a3cb2acad7f13d2a88) on the `integration-test-cleanup` branch due to [a failed run](https://github.com/apache/hadoop-ozone/runs/365409169). Unfortunately logs are no longer available to check what was the problem.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on a change in pull request #763:
HDDS-3302. Add ability in MiniOzoneCluster to create multiple storage
directories.
Posted by GitBox <gi...@apache.org>.
elek commented on a change in pull request #763: HDDS-3302. Add ability in MiniOzoneCluster to create multiple storage directories.
URL: https://github.com/apache/hadoop-ozone/pull/763#discussion_r404911306
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestMiniOzoneCluster.java
##########
@@ -58,14 +58,12 @@
import org.junit.AfterClass;
import org.junit.Assert;
import org.junit.BeforeClass;
-import org.junit.Ignore;
import org.junit.Test;
import org.yaml.snakeyaml.Yaml;
/**
* Test cases for mini ozone cluster.
*/
-@Ignore
Review comment:
Thanks to check it. It can be because a flakiness. In this case it would be great to have at least 3-5 green builds before the merge .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request
#763: HDDS-3302. Add ability in MiniOzoneCluster to create multiple storage
directories.
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on a change in pull request #763: HDDS-3302. Add ability in MiniOzoneCluster to create multiple storage directories.
URL: https://github.com/apache/hadoop-ozone/pull/763#discussion_r405420197
##########
File path: hadoop-ozone/fault-injection-test/mini-chaos-tests/src/test/java/org/apache/hadoop/ozone/TestMiniChaosOzoneCluster.java
##########
@@ -90,6 +99,7 @@ public static void init() throws Exception {
.setNumOzoneManagers(numOzoneManagers)
.setFailureService(failureService)
.setOMServiceID(omServiceID)
+ .setNumDatanodesDirs(numDataVolumes)
Review comment:
Can we make the naming consistent?
* `numOfDatanodesDirs`
* `setNumDatanodesDirs` (implies `numDatanodesDirs` is being set, note no `Of`)
* `numDataVolumes`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request
#763: HDDS-3302. Add ability in MiniOzoneCluster to create multiple storage
directories.
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on a change in pull request #763: HDDS-3302. Add ability in MiniOzoneCluster to create multiple storage directories.
URL: https://github.com/apache/hadoop-ozone/pull/763#discussion_r405409669
##########
File path: hadoop-ozone/fault-injection-test/mini-chaos-tests/src/test/java/org/apache/hadoop/ozone/MiniOzoneChaosCluster.java
##########
@@ -360,6 +360,19 @@ protected void initializeConfiguration() throws IOException {
conf.setInt(OzoneConfigKeys.DFS_CONTAINER_RATIS_LOG_PURGE_GAP, 100);
}
+ /**
+ * Sets the number of HddsDatanodes data dirs to be started as part of
+ * MiniOzoneCluster.
Review comment:
This looks like partially leftover comment -- are data dirs really "started"?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request
#763: HDDS-3302. Add ability in MiniOzoneCluster to create multiple storage
directories.
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on a change in pull request #763: HDDS-3302. Add ability in MiniOzoneCluster to create multiple storage directories.
URL: https://github.com/apache/hadoop-ozone/pull/763#discussion_r405374399
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestMiniOzoneCluster.java
##########
@@ -58,14 +58,12 @@
import org.junit.AfterClass;
import org.junit.Assert;
import org.junit.BeforeClass;
-import org.junit.Ignore;
import org.junit.Test;
import org.yaml.snakeyaml.Yaml;
/**
* Test cases for mini ozone cluster.
*/
-@Ignore
Review comment:
20/20 passed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on a change in pull request #763:
HDDS-3302. Add ability in MiniOzoneCluster to create multiple storage
directories.
Posted by GitBox <gi...@apache.org>.
elek commented on a change in pull request #763: HDDS-3302. Add ability in MiniOzoneCluster to create multiple storage directories.
URL: https://github.com/apache/hadoop-ozone/pull/763#discussion_r404677222
##########
File path: hadoop-ozone/fault-injection-test/mini-chaos-tests/src/test/java/org/apache/hadoop/ozone/TestMiniChaosOzoneCluster.java
##########
@@ -123,6 +133,8 @@ public void run() {
cluster.startChaos(failureInterval, failureInterval, TimeUnit.SECONDS);
loadGenerator.startIO(numMinutes, TimeUnit.MINUTES);
} catch (Exception e) {
+ LOG.error("terminating with exception", e);
+ ExitUtil.terminate(-2, e);
Review comment:
Can you please help to understand the motivation behind this? I thought that if an exception is thrown the exit code will be non-zero anyway (If something is wrong with the error handling, GenericCli might already have the solution. As I remember it has some generic error handling logic).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] mukul1987 commented on a change in pull request
#763: HDDS-3302. Add ability in MiniOzoneCluster to create multiple storage
directories.
Posted by GitBox <gi...@apache.org>.
mukul1987 commented on a change in pull request #763: HDDS-3302. Add ability in MiniOzoneCluster to create multiple storage directories.
URL: https://github.com/apache/hadoop-ozone/pull/763#discussion_r405574401
##########
File path: hadoop-ozone/fault-injection-test/mini-chaos-tests/src/test/java/org/apache/hadoop/ozone/TestMiniChaosOzoneCluster.java
##########
@@ -123,6 +133,8 @@ public void run() {
cluster.startChaos(failureInterval, failureInterval, TimeUnit.SECONDS);
loadGenerator.startIO(numMinutes, TimeUnit.MINUTES);
} catch (Exception e) {
+ LOG.error("terminating with exception", e);
+ ExitUtil.terminate(-2, e);
Review comment:
done, moved to GenericCli.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request
#763: HDDS-3302. Add ability in MiniOzoneCluster to create multiple storage
directories.
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on a change in pull request #763: HDDS-3302. Add ability in MiniOzoneCluster to create multiple storage directories.
URL: https://github.com/apache/hadoop-ozone/pull/763#discussion_r405415828
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneCluster.java
##########
@@ -394,6 +395,19 @@ public Builder setNumDatanodes(int val) {
return this;
}
+ /**
+ * Sets the number of HddsDatanodes data dirs to be started as part of
+ * MiniOzoneCluster.
Review comment:
Same here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] mukul1987 commented on a change in pull request
#763: HDDS-3302. Add ability in MiniOzoneCluster to create multiple storage
directories.
Posted by GitBox <gi...@apache.org>.
mukul1987 commented on a change in pull request #763: HDDS-3302. Add ability in MiniOzoneCluster to create multiple storage directories.
URL: https://github.com/apache/hadoop-ozone/pull/763#discussion_r405574613
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestMiniOzoneCluster.java
##########
@@ -58,14 +58,12 @@
import org.junit.AfterClass;
import org.junit.Assert;
import org.junit.BeforeClass;
-import org.junit.Ignore;
import org.junit.Test;
import org.yaml.snakeyaml.Yaml;
/**
* Test cases for mini ozone cluster.
*/
-@Ignore
Review comment:
This works, so leaving this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on a change in pull request #763:
HDDS-3302. Add ability in MiniOzoneCluster to create multiple storage
directories.
Posted by GitBox <gi...@apache.org>.
elek commented on a change in pull request #763: HDDS-3302. Add ability in MiniOzoneCluster to create multiple storage directories.
URL: https://github.com/apache/hadoop-ozone/pull/763#discussion_r404678200
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestMiniOzoneCluster.java
##########
@@ -58,14 +58,12 @@
import org.junit.AfterClass;
import org.junit.Assert;
import org.junit.BeforeClass;
-import org.junit.Ignore;
import org.junit.Test;
import org.yaml.snakeyaml.Yaml;
/**
* Test cases for mini ozone cluster.
*/
-@Ignore
Review comment:
Why was it ignored and how is the problem fixed by this patch? Do we need to run multiple checks to prove that there is no intermittency any more?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request
#763: HDDS-3302. Add ability in MiniOzoneCluster to create multiple storage
directories.
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on a change in pull request #763: HDDS-3302. Add ability in MiniOzoneCluster to create multiple storage directories.
URL: https://github.com/apache/hadoop-ozone/pull/763#discussion_r405420589
##########
File path: hadoop-ozone/fault-injection-test/mini-chaos-tests/src/test/java/org/apache/hadoop/ozone/TestMiniChaosOzoneCluster.java
##########
@@ -69,6 +74,10 @@
description = "no of clients writing to OM")
private static int numClients = 3;
+ @Option(names = {"-v", "--numDataVolume"},
+ description = "no of datanode volumes")
Review comment:
```suggestion
description = "number of datanode volumes to create")
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org