You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2024/04/30 08:17:00 UTC
[jira] [Commented] (MAPREDUCE-7475) 2 tests are non-idempotent (passes in the first run but fails in repeated runs in the same JVM)
[ https://issues.apache.org/jira/browse/MAPREDUCE-7475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17842290#comment-17842290 ]
ASF GitHub Bot commented on MAPREDUCE-7475:
-------------------------------------------
kaiyaok2 opened a new pull request, #6785:
URL: https://github.com/apache/hadoop/pull/6785
### Description of PR
As described in https://issues.apache.org/jira/browse/MAPREDUCE-7475: 2 tests are not idempotent and fails upon repeated execution within the same JVM instance due to self-induced state pollution. Specifically, these tests try to make the directory `TEST_ROOT_DIR` and write to it. The tests do not clean up (remove) the directory after execution. Therefore, in the second execution, `TEST_ROOT_DIR` would already exist and the exception `Could not create test dir` would be thrown. The tests shall be fixed as unit tests shall be idempotent.
Below are the 2 non-idempotent tests:
* `org.apache.hadoop.mapred.TestOldCombinerGrouping.testCombiner`
* `org.apache.hadoop.mapreduce.TestNewCombinerGrouping.testCombiner`
##Sample Failure Message for `org.apache.hadoop.mapreduce.TestNewCombinerGrouping.testCombiner` (in the 2nd run of the test):
```
java.lang.RuntimeException: Could not create test dir: /home/kaiyaok2/NIOExperiments/github.com/apache/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/build/test/data/e4ee268d-2946-43a2-93b5-f2b4ac647279
at org.apache.hadoop.mapreduce.TestNewCombinerGrouping.testCombiner(TestNewCombinerGrouping.java:109)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
```
### How was this patch tested?
After the patch, rerunning the tests in the same JVM does not produce any exceptions.
### Code changes:
Check if the test directory already exists (and delete it if so) before calling `mkdirs()`.
> 2 tests are non-idempotent (passes in the first run but fails in repeated runs in the same JVM)
> -----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-7475
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7475
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Environment: Ubuntu 22.04, Java 17
> Reporter: Kaiyao Ke
> Priority: Major
> Original Estimate: 10m
> Remaining Estimate: 10m
>
> 2 tests are not idempotent and fails upon repeated execution within the same JVM instance due to self-induced state pollution. Specifically, these tests try to make the directory TEST_ROOT_DIR and write to it. The tests do not clean up (remove) the directory after execution. Therefore, in the second execution, TEST_ROOT_DIR would already exist and the exception `Could not create test dir` would be thrown. Below are the 2 non-idempotent tests:
> * org.apache.hadoop.mapred.TestOldCombinerGrouping.testCombiner
> * org.apache.hadoop.mapreduce.TestNewCombinerGrouping.testCombiner
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org