You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2022/10/20 17:43:00 UTC
[jira] [Commented] (HDFS-16740) Mini cluster test flakiness
[ https://issues.apache.org/jira/browse/HDFS-16740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17621282#comment-17621282 ]
ASF GitHub Bot commented on HDFS-16740:
---------------------------------------
steveloughran commented on code in PR #4835:
URL: https://github.com/apache/hadoop/pull/4835#discussion_r1000928173
##########
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRollingUpgrade.java:
##########
@@ -83,37 +83,18 @@ public static void runCmd(DFSAdmin dfsadmin, boolean success,
}
@Rule
- public TemporaryFolder folder = new TemporaryFolder();
-
- /**
- * Create a default HDFS configuration which has test-specific data directories. This is
- * intended to protect against interactions between test runs that might corrupt results. Each
- * test run's data is automatically cleaned-up by JUnit.
- *
- * @return a default configuration with test-specific data directories
- */
- public Configuration getHdfsConfiguration() throws IOException {
- Configuration conf = new HdfsConfiguration();
Review Comment:
retain this, but just return the new config. allows for changes later
##########
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/MiniJournalCluster.java:
##########
@@ -64,7 +65,17 @@ public static class Builder {
public Builder(Configuration conf) {
this.conf = conf;
}
-
+
+ public Builder(Configuration conf, TemporaryFolder baseDir) {
Review Comment:
i don't want to add junit dependencies here. better to take a File ref and pass it in when used
##########
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java:
##########
@@ -240,6 +241,10 @@ public Builder(Configuration conf) {
}
}
+ public Builder(Configuration conf, TemporaryFolder baseDir) {
Review Comment:
i don't want to add junit dependencies here; we don't know where else it is used. and test dependencies don't get exported by maven. the code will need to be given baseDir.getRoot()
> Mini cluster test flakiness
> ---------------------------
>
> Key: HDFS-16740
> URL: https://issues.apache.org/jira/browse/HDFS-16740
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs, test
> Affects Versions: 3.4.0, 3.3.5
> Reporter: Steve Vaughan
> Assignee: Steve Vaughan
> Priority: Major
> Labels: pull-request-available
>
> Mini clusters used during HDFS unit tests are reporting test failures that do not appear to be directly related to submitted changes. The failures are the result of either interactions between tests run in parallel, or tests which share common disk space for tests. In all cases, the tests can be run individually serially without any errors. Addressing this issue will simplify future submissions by eliminating the confusion introduced by these unrelated test failures.
> We can apply lessons recently from TestRollingUpgrade, which was recently patched to unblock a recent submission. The fixes involved changing the HDFS configuration to use temporary disk space for each individual tests, and using try-with-resources to ensure that clusters were shutdown cleanly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org