You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Nigel Daley (JIRA)" <ji...@apache.org> on 2007/01/20 02:25:29 UTC
[jira] Created: (HADOOP-912) TestMiniMRWithDFS fails sporadically
TestMiniMRWithDFS fails sporadically
------------------------------------
Key: HADOOP-912
URL: https://issues.apache.org/jira/browse/HADOOP-912
Project: Hadoop
Issue Type: Bug
Components: mapred
Affects Versions: 0.10.1
Reporter: Nigel Daley
Assigned To: Arun C Murthy
TestMiniMRWithDFS fails sporadically with the following error:
junit.framework.AssertionFailedError: Spurious directory task_0001_m_000008_0 found in C:\hudson\workspace\Hadoop-WindowsSmokeTest\trunk\build\test\mapred\local\50068_0
at org.apache.hadoop.mapred.TestMiniMRWithDFS.checkTaskDirectories(TestMiniMRWithDFS.java:128)
at org.apache.hadoop.mapred.TestMiniMRWithDFS.testWithDFS(TestMiniMRWithDFS.java:163)
This tests checks that all mapred.local.dirs have been cleaned up after running a mapred job on MiniMRCluster. Effectively, this tests waits for TaskTracker.isIdle() to return true before checking the dirs have been cleaned up. I believe that HADOOP-639 reordered the cleanup code so that isIdle() will return true before the directory cleanup is complete.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-912) TestMiniMRWithDFS fails sporadically
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12466389 ]
Arun C Murthy commented on HADOOP-912:
--------------------------------------
I suspect this is more of a drawback of the TestMiniMRWithDFS than the MR-framework itself...
MiniMrCluster.waitUntilIdle checks for any 'running' tasks on the TaskTracker... post HADOOP-840 the TaskTracker the cleanup actions are queued and are done offline, which the test-case isn't aware of ; which I suspect leads to a timing issue where the TaskTracker is rightly *idle* and the 'cleanup thread' hasn't kicked in yet...
Should we add a 'sleep' in MiniMRCluster.waitUntilIdle or perhaps another api to check if the job has finished and has been cleaned-up? Thoughts?
> TestMiniMRWithDFS fails sporadically
> ------------------------------------
>
> Key: HADOOP-912
> URL: https://issues.apache.org/jira/browse/HADOOP-912
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.10.1
> Reporter: Nigel Daley
> Assigned To: Arun C Murthy
>
> TestMiniMRWithDFS fails sporadically with the following error:
> junit.framework.AssertionFailedError: Spurious directory task_0001_m_000008_0 found in C:\hudson\workspace\Hadoop-WindowsSmokeTest\trunk\build\test\mapred\local\50068_0
> at org.apache.hadoop.mapred.TestMiniMRWithDFS.checkTaskDirectories(TestMiniMRWithDFS.java:128)
> at org.apache.hadoop.mapred.TestMiniMRWithDFS.testWithDFS(TestMiniMRWithDFS.java:163)
> This tests checks that all mapred.local.dirs have been cleaned up after running a mapred job on MiniMRCluster. Effectively, this tests waits for TaskTracker.isIdle() to return true before checking the dirs have been cleaned up. I believe that HADOOP-639 reordered the cleanup code so that isIdle() will return true before the directory cleanup is complete.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-912) TestMiniMRWithDFS fails sporadically
Posted by "Nigel Daley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12466641 ]
Nigel Daley commented on HADOOP-912:
------------------------------------
+1
The automated Hadoop QA process got tripped up on the javadoc warnings introduced by the HADOOP-908 patch. The process didn't catch those warnings because a new patch was added (and committed) after Hadoop QA had already tested the first patch. A clear deficiency in the automated patch test system is that it tests the latest patch on a JIRA *only* when the state goes from Open to Patch Available the *first* time.
> TestMiniMRWithDFS fails sporadically
> ------------------------------------
>
> Key: HADOOP-912
> URL: https://issues.apache.org/jira/browse/HADOOP-912
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.10.1
> Reporter: Nigel Daley
> Assigned To: Arun C Murthy
> Fix For: 0.11.0
>
> Attachments: HADOOP-912_20070122.patch
>
>
> TestMiniMRWithDFS fails sporadically with the following error:
> junit.framework.AssertionFailedError: Spurious directory task_0001_m_000008_0 found in C:\hudson\workspace\Hadoop-WindowsSmokeTest\trunk\build\test\mapred\local\50068_0
> at org.apache.hadoop.mapred.TestMiniMRWithDFS.checkTaskDirectories(TestMiniMRWithDFS.java:128)
> at org.apache.hadoop.mapred.TestMiniMRWithDFS.testWithDFS(TestMiniMRWithDFS.java:163)
> This tests checks that all mapred.local.dirs have been cleaned up after running a mapred job on MiniMRCluster. Effectively, this tests waits for TaskTracker.isIdle() to return true before checking the dirs have been cleaned up. I believe that HADOOP-639 reordered the cleanup code so that isIdle() will return true before the directory cleanup is complete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-912) TestMiniMRWithDFS fails sporadically
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12466511 ]
Owen O'Malley commented on HADOOP-912:
--------------------------------------
+1
> TestMiniMRWithDFS fails sporadically
> ------------------------------------
>
> Key: HADOOP-912
> URL: https://issues.apache.org/jira/browse/HADOOP-912
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.10.1
> Reporter: Nigel Daley
> Assigned To: Arun C Murthy
> Attachments: HADOOP-912_20070122.patch
>
>
> TestMiniMRWithDFS fails sporadically with the following error:
> junit.framework.AssertionFailedError: Spurious directory task_0001_m_000008_0 found in C:\hudson\workspace\Hadoop-WindowsSmokeTest\trunk\build\test\mapred\local\50068_0
> at org.apache.hadoop.mapred.TestMiniMRWithDFS.checkTaskDirectories(TestMiniMRWithDFS.java:128)
> at org.apache.hadoop.mapred.TestMiniMRWithDFS.testWithDFS(TestMiniMRWithDFS.java:163)
> This tests checks that all mapred.local.dirs have been cleaned up after running a mapred job on MiniMRCluster. Effectively, this tests waits for TaskTracker.isIdle() to return true before checking the dirs have been cleaned up. I believe that HADOOP-639 reordered the cleanup code so that isIdle() will return true before the directory cleanup is complete.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-912) TestMiniMRWithDFS fails sporadically
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated HADOOP-912:
--------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thanks, Arun!
> TestMiniMRWithDFS fails sporadically
> ------------------------------------
>
> Key: HADOOP-912
> URL: https://issues.apache.org/jira/browse/HADOOP-912
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.10.1
> Reporter: Nigel Daley
> Assigned To: Arun C Murthy
> Fix For: 0.11.0
>
> Attachments: HADOOP-912_20070122.patch
>
>
> TestMiniMRWithDFS fails sporadically with the following error:
> junit.framework.AssertionFailedError: Spurious directory task_0001_m_000008_0 found in C:\hudson\workspace\Hadoop-WindowsSmokeTest\trunk\build\test\mapred\local\50068_0
> at org.apache.hadoop.mapred.TestMiniMRWithDFS.checkTaskDirectories(TestMiniMRWithDFS.java:128)
> at org.apache.hadoop.mapred.TestMiniMRWithDFS.testWithDFS(TestMiniMRWithDFS.java:163)
> This tests checks that all mapred.local.dirs have been cleaned up after running a mapred job on MiniMRCluster. Effectively, this tests waits for TaskTracker.isIdle() to return true before checking the dirs have been cleaned up. I believe that HADOOP-639 reordered the cleanup code so that isIdle() will return true before the directory cleanup is complete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-912) TestMiniMRWithDFS fails sporadically
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated HADOOP-912:
---------------------------------
Fix Version/s: 0.11.0
> TestMiniMRWithDFS fails sporadically
> ------------------------------------
>
> Key: HADOOP-912
> URL: https://issues.apache.org/jira/browse/HADOOP-912
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.10.1
> Reporter: Nigel Daley
> Assigned To: Arun C Murthy
> Fix For: 0.11.0
>
> Attachments: HADOOP-912_20070122.patch
>
>
> TestMiniMRWithDFS fails sporadically with the following error:
> junit.framework.AssertionFailedError: Spurious directory task_0001_m_000008_0 found in C:\hudson\workspace\Hadoop-WindowsSmokeTest\trunk\build\test\mapred\local\50068_0
> at org.apache.hadoop.mapred.TestMiniMRWithDFS.checkTaskDirectories(TestMiniMRWithDFS.java:128)
> at org.apache.hadoop.mapred.TestMiniMRWithDFS.testWithDFS(TestMiniMRWithDFS.java:163)
> This tests checks that all mapred.local.dirs have been cleaned up after running a mapred job on MiniMRCluster. Effectively, this tests waits for TaskTracker.isIdle() to return true before checking the dirs have been cleaned up. I believe that HADOOP-639 reordered the cleanup code so that isIdle() will return true before the directory cleanup is complete.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-912) TestMiniMRWithDFS fails sporadically
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated HADOOP-912:
---------------------------------
Status: Open (was: Patch Available)
I'll take a second look, needs another fix I think...
> TestMiniMRWithDFS fails sporadically
> ------------------------------------
>
> Key: HADOOP-912
> URL: https://issues.apache.org/jira/browse/HADOOP-912
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.10.1
> Reporter: Nigel Daley
> Assigned To: Arun C Murthy
> Fix For: 0.11.0
>
> Attachments: HADOOP-912_20070122.patch
>
>
> TestMiniMRWithDFS fails sporadically with the following error:
> junit.framework.AssertionFailedError: Spurious directory task_0001_m_000008_0 found in C:\hudson\workspace\Hadoop-WindowsSmokeTest\trunk\build\test\mapred\local\50068_0
> at org.apache.hadoop.mapred.TestMiniMRWithDFS.checkTaskDirectories(TestMiniMRWithDFS.java:128)
> at org.apache.hadoop.mapred.TestMiniMRWithDFS.testWithDFS(TestMiniMRWithDFS.java:163)
> This tests checks that all mapred.local.dirs have been cleaned up after running a mapred job on MiniMRCluster. Effectively, this tests waits for TaskTracker.isIdle() to return true before checking the dirs have been cleaned up. I believe that HADOOP-639 reordered the cleanup code so that isIdle() will return true before the directory cleanup is complete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-912) TestMiniMRWithDFS fails sporadically
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12466629 ]
Arun C Murthy commented on HADOOP-912:
--------------------------------------
I'd wager the javadoc warning wasn't related to this patch, and seems like HADOOP-916 fixed it...
> TestMiniMRWithDFS fails sporadically
> ------------------------------------
>
> Key: HADOOP-912
> URL: https://issues.apache.org/jira/browse/HADOOP-912
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.10.1
> Reporter: Nigel Daley
> Assigned To: Arun C Murthy
> Fix For: 0.11.0
>
> Attachments: HADOOP-912_20070122.patch
>
>
> TestMiniMRWithDFS fails sporadically with the following error:
> junit.framework.AssertionFailedError: Spurious directory task_0001_m_000008_0 found in C:\hudson\workspace\Hadoop-WindowsSmokeTest\trunk\build\test\mapred\local\50068_0
> at org.apache.hadoop.mapred.TestMiniMRWithDFS.checkTaskDirectories(TestMiniMRWithDFS.java:128)
> at org.apache.hadoop.mapred.TestMiniMRWithDFS.testWithDFS(TestMiniMRWithDFS.java:163)
> This tests checks that all mapred.local.dirs have been cleaned up after running a mapred job on MiniMRCluster. Effectively, this tests waits for TaskTracker.isIdle() to return true before checking the dirs have been cleaned up. I believe that HADOOP-639 reordered the cleanup code so that isIdle() will return true before the directory cleanup is complete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-912) TestMiniMRWithDFS fails sporadically
Posted by "Nigel Daley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12466658 ]
Nigel Daley commented on HADOOP-912:
------------------------------------
-1
I'm still seeing the same failure (although much less often) with this patch.
> TestMiniMRWithDFS fails sporadically
> ------------------------------------
>
> Key: HADOOP-912
> URL: https://issues.apache.org/jira/browse/HADOOP-912
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.10.1
> Reporter: Nigel Daley
> Assigned To: Arun C Murthy
> Fix For: 0.11.0
>
> Attachments: HADOOP-912_20070122.patch
>
>
> TestMiniMRWithDFS fails sporadically with the following error:
> junit.framework.AssertionFailedError: Spurious directory task_0001_m_000008_0 found in C:\hudson\workspace\Hadoop-WindowsSmokeTest\trunk\build\test\mapred\local\50068_0
> at org.apache.hadoop.mapred.TestMiniMRWithDFS.checkTaskDirectories(TestMiniMRWithDFS.java:128)
> at org.apache.hadoop.mapred.TestMiniMRWithDFS.testWithDFS(TestMiniMRWithDFS.java:163)
> This tests checks that all mapred.local.dirs have been cleaned up after running a mapred job on MiniMRCluster. Effectively, this tests waits for TaskTracker.isIdle() to return true before checking the dirs have been cleaned up. I believe that HADOOP-639 reordered the cleanup code so that isIdle() will return true before the directory cleanup is complete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-912) TestMiniMRWithDFS fails sporadically
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12466518 ]
Hadoop QA commented on HADOOP-912:
----------------------------------
-1, because the javadoc command appears to have generated warning messages when testing the latest attachment (http://issues.apache.org/jira/secure/attachment/12349364/HADOOP-912_20070122.patch) against trunk revision r497927. Please note that this message is automatically generated and may represent a problem with the automation system and not the patch.
> TestMiniMRWithDFS fails sporadically
> ------------------------------------
>
> Key: HADOOP-912
> URL: https://issues.apache.org/jira/browse/HADOOP-912
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.10.1
> Reporter: Nigel Daley
> Assigned To: Arun C Murthy
> Fix For: 0.11.0
>
> Attachments: HADOOP-912_20070122.patch
>
>
> TestMiniMRWithDFS fails sporadically with the following error:
> junit.framework.AssertionFailedError: Spurious directory task_0001_m_000008_0 found in C:\hudson\workspace\Hadoop-WindowsSmokeTest\trunk\build\test\mapred\local\50068_0
> at org.apache.hadoop.mapred.TestMiniMRWithDFS.checkTaskDirectories(TestMiniMRWithDFS.java:128)
> at org.apache.hadoop.mapred.TestMiniMRWithDFS.testWithDFS(TestMiniMRWithDFS.java:163)
> This tests checks that all mapred.local.dirs have been cleaned up after running a mapred job on MiniMRCluster. Effectively, this tests waits for TaskTracker.isIdle() to return true before checking the dirs have been cleaned up. I believe that HADOOP-639 reordered the cleanup code so that isIdle() will return true before the directory cleanup is complete.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-912) TestMiniMRWithDFS fails sporadically
Posted by "Nigel Daley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12466891 ]
Nigel Daley commented on HADOOP-912:
------------------------------------
+1
The automated Hadoop QA process is getting tripped up on the javadoc
warnings introduced by the HADOOP-908 patch. The process didn't
catch those warnings because a new patch was added (and committed)
after Hadoop QA had already tested the first patch. A clear
deficiency in the automated patch test system is that it tests the
latest patch on a JIRA *only* when the state goes from Open to Patch
Available the *first* time.
> TestMiniMRWithDFS fails sporadically
> ------------------------------------
>
> Key: HADOOP-912
> URL: https://issues.apache.org/jira/browse/HADOOP-912
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.10.1
> Reporter: Nigel Daley
> Assigned To: Arun C Murthy
> Fix For: 0.11.0
>
> Attachments: HADOOP-912_20070122.patch
>
>
> TestMiniMRWithDFS fails sporadically with the following error:
> junit.framework.AssertionFailedError: Spurious directory task_0001_m_000008_0 found in C:\hudson\workspace\Hadoop-WindowsSmokeTest\trunk\build\test\mapred\local\50068_0
> at org.apache.hadoop.mapred.TestMiniMRWithDFS.checkTaskDirectories(TestMiniMRWithDFS.java:128)
> at org.apache.hadoop.mapred.TestMiniMRWithDFS.testWithDFS(TestMiniMRWithDFS.java:163)
> This tests checks that all mapred.local.dirs have been cleaned up after running a mapred job on MiniMRCluster. Effectively, this tests waits for TaskTracker.isIdle() to return true before checking the dirs have been cleaned up. I believe that HADOOP-639 reordered the cleanup code so that isIdle() will return true before the directory cleanup is complete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-912) TestMiniMRWithDFS fails sporadically
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated HADOOP-912:
---------------------------------
Attachment: HADOOP-912_20070122.patch
> TestMiniMRWithDFS fails sporadically
> ------------------------------------
>
> Key: HADOOP-912
> URL: https://issues.apache.org/jira/browse/HADOOP-912
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.10.1
> Reporter: Nigel Daley
> Assigned To: Arun C Murthy
> Attachments: HADOOP-912_20070122.patch
>
>
> TestMiniMRWithDFS fails sporadically with the following error:
> junit.framework.AssertionFailedError: Spurious directory task_0001_m_000008_0 found in C:\hudson\workspace\Hadoop-WindowsSmokeTest\trunk\build\test\mapred\local\50068_0
> at org.apache.hadoop.mapred.TestMiniMRWithDFS.checkTaskDirectories(TestMiniMRWithDFS.java:128)
> at org.apache.hadoop.mapred.TestMiniMRWithDFS.testWithDFS(TestMiniMRWithDFS.java:163)
> This tests checks that all mapred.local.dirs have been cleaned up after running a mapred job on MiniMRCluster. Effectively, this tests waits for TaskTracker.isIdle() to return true before checking the dirs have been cleaned up. I believe that HADOOP-639 reordered the cleanup code so that isIdle() will return true before the directory cleanup is complete.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-912) TestMiniMRWithDFS fails sporadically
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated HADOOP-912:
---------------------------------
Status: Patch Available (was: Open)
Thanks for the review Owen.
> TestMiniMRWithDFS fails sporadically
> ------------------------------------
>
> Key: HADOOP-912
> URL: https://issues.apache.org/jira/browse/HADOOP-912
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.10.1
> Reporter: Nigel Daley
> Assigned To: Arun C Murthy
> Attachments: HADOOP-912_20070122.patch
>
>
> TestMiniMRWithDFS fails sporadically with the following error:
> junit.framework.AssertionFailedError: Spurious directory task_0001_m_000008_0 found in C:\hudson\workspace\Hadoop-WindowsSmokeTest\trunk\build\test\mapred\local\50068_0
> at org.apache.hadoop.mapred.TestMiniMRWithDFS.checkTaskDirectories(TestMiniMRWithDFS.java:128)
> at org.apache.hadoop.mapred.TestMiniMRWithDFS.testWithDFS(TestMiniMRWithDFS.java:163)
> This tests checks that all mapred.local.dirs have been cleaned up after running a mapred job on MiniMRCluster. Effectively, this tests waits for TaskTracker.isIdle() to return true before checking the dirs have been cleaned up. I believe that HADOOP-639 reordered the cleanup code so that isIdle() will return true before the directory cleanup is complete.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-912) TestMiniMRWithDFS fails sporadically
Posted by "Nigel Daley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12466826 ]
Nigel Daley commented on HADOOP-912:
------------------------------------
+1
This morning, I can't reproduce the problem anymore w/ this patch. I vote to commit it.
> TestMiniMRWithDFS fails sporadically
> ------------------------------------
>
> Key: HADOOP-912
> URL: https://issues.apache.org/jira/browse/HADOOP-912
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.10.1
> Reporter: Nigel Daley
> Assigned To: Arun C Murthy
> Fix For: 0.11.0
>
> Attachments: HADOOP-912_20070122.patch
>
>
> TestMiniMRWithDFS fails sporadically with the following error:
> junit.framework.AssertionFailedError: Spurious directory task_0001_m_000008_0 found in C:\hudson\workspace\Hadoop-WindowsSmokeTest\trunk\build\test\mapred\local\50068_0
> at org.apache.hadoop.mapred.TestMiniMRWithDFS.checkTaskDirectories(TestMiniMRWithDFS.java:128)
> at org.apache.hadoop.mapred.TestMiniMRWithDFS.testWithDFS(TestMiniMRWithDFS.java:163)
> This tests checks that all mapred.local.dirs have been cleaned up after running a mapred job on MiniMRCluster. Effectively, this tests waits for TaskTracker.isIdle() to return true before checking the dirs have been cleaned up. I believe that HADOOP-639 reordered the cleanup code so that isIdle() will return true before the directory cleanup is complete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-912) TestMiniMRWithDFS fails sporadically
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated HADOOP-912:
---------------------------------
Status: Patch Available (was: Open)
Thanks for verifying Nigel... this patch fixes a real bug in TaskTracker.isIdle() which, at the very least, leads the TT to ask the JT for a new task too early. I vote we commit this and take up any further issues as a separate bug.
> TestMiniMRWithDFS fails sporadically
> ------------------------------------
>
> Key: HADOOP-912
> URL: https://issues.apache.org/jira/browse/HADOOP-912
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.10.1
> Reporter: Nigel Daley
> Assigned To: Arun C Murthy
> Fix For: 0.11.0
>
> Attachments: HADOOP-912_20070122.patch
>
>
> TestMiniMRWithDFS fails sporadically with the following error:
> junit.framework.AssertionFailedError: Spurious directory task_0001_m_000008_0 found in C:\hudson\workspace\Hadoop-WindowsSmokeTest\trunk\build\test\mapred\local\50068_0
> at org.apache.hadoop.mapred.TestMiniMRWithDFS.checkTaskDirectories(TestMiniMRWithDFS.java:128)
> at org.apache.hadoop.mapred.TestMiniMRWithDFS.testWithDFS(TestMiniMRWithDFS.java:163)
> This tests checks that all mapred.local.dirs have been cleaned up after running a mapred job on MiniMRCluster. Effectively, this tests waits for TaskTracker.isIdle() to return true before checking the dirs have been cleaned up. I believe that HADOOP-639 reordered the cleanup code so that isIdle() will return true before the directory cleanup is complete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-912) TestMiniMRWithDFS fails sporadically
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12466425 ]
Arun C Murthy commented on HADOOP-912:
--------------------------------------
I've fixed org.apache.hadoop.mapred.TaskTracker.isIdle() to check if there are tasks waiting to be cleaned-up; what do you guys think of this approach - this is 'correct' according to the existing documentation of TaskTracker.isIdle(); however it means that TaskTracker.offerService() now waits for cleanup to complete before asking the JobTracker for new tasks (which is what the comment in offerService() is explicit about too)...
In all, this is a fix for TaskTracker.isIdle() - thoughts?
> TestMiniMRWithDFS fails sporadically
> ------------------------------------
>
> Key: HADOOP-912
> URL: https://issues.apache.org/jira/browse/HADOOP-912
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.10.1
> Reporter: Nigel Daley
> Assigned To: Arun C Murthy
> Attachments: HADOOP-912_20070122.patch
>
>
> TestMiniMRWithDFS fails sporadically with the following error:
> junit.framework.AssertionFailedError: Spurious directory task_0001_m_000008_0 found in C:\hudson\workspace\Hadoop-WindowsSmokeTest\trunk\build\test\mapred\local\50068_0
> at org.apache.hadoop.mapred.TestMiniMRWithDFS.checkTaskDirectories(TestMiniMRWithDFS.java:128)
> at org.apache.hadoop.mapred.TestMiniMRWithDFS.testWithDFS(TestMiniMRWithDFS.java:163)
> This tests checks that all mapred.local.dirs have been cleaned up after running a mapred job on MiniMRCluster. Effectively, this tests waits for TaskTracker.isIdle() to return true before checking the dirs have been cleaned up. I believe that HADOOP-639 reordered the cleanup code so that isIdle() will return true before the directory cleanup is complete.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira