You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2022/08/30 19:34:00 UTC
[jira] [Commented] (HDFS-16740) Mini cluster test flakiness
[ https://issues.apache.org/jira/browse/HDFS-16740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17598027#comment-17598027 ]
ASF GitHub Bot commented on HDFS-16740:
---------------------------------------
snmvaughan opened a new pull request, #4835:
URL: https://github.com/apache/hadoop/pull/4835
### Description of PR
Mini clusters used during HDFS unit tests are reporting test failures that do not appear to be directly related to submitted changes. The failures are the result of either interactions between tests run in parallel, or tests which share common disk space for tests. In all cases, the tests can be run individually serially without any errors. Addressing this issue will simplify future submissions by eliminating the confusion introduced by these unrelated test failures.
We can apply lessons recently from TestRollingUpgrade, which was recently patched to unblock a recent submission. The fixes involved changing the HDFS configuration to use temporary disk space for each individual tests, and using try-with-resources to ensure that clusters were shutdown cleanly.
Add a constructor to the mini cluster constructors and builders that takes a TemporaryFolder. This encourages the use of managed temporary folders that are unique to the test and automatically cleaned up. The field is named "baseDir" to communicate that this folder acts as the base directory for test data. Using the new constructor encourages the consistent use of the managed temporary folder.
Allow the same baseDir to be provided multiple times, simplifying the coding of cluster restarts. This simplifies these tests as the use of the same TemporaryFolder will always work (even for restarts)
This change also reduces the number of tests that make assumptions about the mini clusters using encapsulation. These changes avoid configurations that directly use HDFS_MINIDFS_BASEDIR.
### How was this patch tested?
Both Maven Surefire and an IDE were used to demonstrate that the tests operate as expected.
### For code changes:
- [X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
> Mini cluster test flakiness
> ---------------------------
>
> Key: HDFS-16740
> URL: https://issues.apache.org/jira/browse/HDFS-16740
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs, test
> Affects Versions: 3.4.0, 3.3.9
> Reporter: Steve Vaughan
> Assignee: Steve Vaughan
> Priority: Major
>
> Mini clusters used during HDFS unit tests are reporting test failures that do not appear to be directly related to submitted changes. The failures are the result of either interactions between tests run in parallel, or tests which share common disk space for tests. In all cases, the tests can be run individually serially without any errors. Addressing this issue will simplify future submissions by eliminating the confusion introduced by these unrelated test failures.
> We can apply lessons recently from TestRollingUpgrade, which was recently patched to unblock a recent submission. The fixes involved changing the HDFS configuration to use temporary disk space for each individual tests, and using try-with-resources to ensure that clusters were shutdown cleanly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org