You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Hemanth Yamijala (JIRA)" <ji...@apache.org> on 2009/05/14 07:32:45 UTC
[jira] Created: (HADOOP-5828) Use absolute path for JobTracker's
mapred.local.dir in MiniMRCluster
Use absolute path for JobTracker's mapred.local.dir in MiniMRCluster
--------------------------------------------------------------------
Key: HADOOP-5828
URL: https://issues.apache.org/jira/browse/HADOOP-5828
Project: Hadoop Core
Issue Type: Bug
Components: test
Reporter: Hemanth Yamijala
Assignee: Hemanth Yamijala
MiniMRCluster's JobTracker sets the mapred.local.dir to a relative path: "build/test/mapred/local". While currently this does not cause any problem, if at some time in future, the JobTracker starts using LocalDirAllocator for localizing files, this could cause failures in contrib tests. This issue was faced when working with an internal patch that tried to introduce LocalDirAllocator into the JobTracker, and capacity scheduler test cases started failing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5828) Use absolute path for JobTracker's
mapred.local.dir in MiniMRCluster
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-5828:
-------------------------------------
Attachment: HADOOP-5828.20.patch
Patch for branch 0.20. Just changes the test directory structure.
> Use absolute path for JobTracker's mapred.local.dir in MiniMRCluster
> --------------------------------------------------------------------
>
> Key: HADOOP-5828
> URL: https://issues.apache.org/jira/browse/HADOOP-5828
> Project: Hadoop Core
> Issue Type: Bug
> Components: test
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Attachments: HADOOP-5828.20.patch, HADOOP-5828.patch
>
>
> MiniMRCluster's JobTracker sets the mapred.local.dir to a relative path: "build/test/mapred/local". While currently this does not cause any problem, if at some time in future, the JobTracker starts using LocalDirAllocator for localizing files, this could cause failures in contrib tests. This issue was faced when working with an internal patch that tried to introduce LocalDirAllocator into the JobTracker, and capacity scheduler test cases started failing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-5828) Use absolute path for JobTracker's
mapred.local.dir in MiniMRCluster
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala resolved HADOOP-5828.
--------------------------------------
Resolution: Fixed
Fix Version/s: 0.20.1
Hadoop Flags: [Reviewed]
I just committed this to trunk and branch 0.20.
> Use absolute path for JobTracker's mapred.local.dir in MiniMRCluster
> --------------------------------------------------------------------
>
> Key: HADOOP-5828
> URL: https://issues.apache.org/jira/browse/HADOOP-5828
> Project: Hadoop Core
> Issue Type: Bug
> Components: test
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Fix For: 0.20.1
>
> Attachments: HADOOP-5828.20.patch, HADOOP-5828.patch
>
>
> MiniMRCluster's JobTracker sets the mapred.local.dir to a relative path: "build/test/mapred/local". While currently this does not cause any problem, if at some time in future, the JobTracker starts using LocalDirAllocator for localizing files, this could cause failures in contrib tests. This issue was faced when working with an internal patch that tried to introduce LocalDirAllocator into the JobTracker, and capacity scheduler test cases started failing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5828) Use absolute path for JobTracker's
mapred.local.dir in MiniMRCluster
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-5828:
-------------------------------------
Attachment: HADOOP-5828.patch
This patch converts the relative path for the mapred.local.dir in MiniMRCluster to an absolute path.
> Use absolute path for JobTracker's mapred.local.dir in MiniMRCluster
> --------------------------------------------------------------------
>
> Key: HADOOP-5828
> URL: https://issues.apache.org/jira/browse/HADOOP-5828
> Project: Hadoop Core
> Issue Type: Bug
> Components: test
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Attachments: HADOOP-5828.patch
>
>
> MiniMRCluster's JobTracker sets the mapred.local.dir to a relative path: "build/test/mapred/local". While currently this does not cause any problem, if at some time in future, the JobTracker starts using LocalDirAllocator for localizing files, this could cause failures in contrib tests. This issue was faced when working with an internal patch that tried to introduce LocalDirAllocator into the JobTracker, and capacity scheduler test cases started failing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5828) Use absolute path for JobTracker's
mapred.local.dir in MiniMRCluster
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12709313#action_12709313 ]
Hemanth Yamijala commented on HADOOP-5828:
------------------------------------------
Results of test-patch:
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 5 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
[exec]
> Use absolute path for JobTracker's mapred.local.dir in MiniMRCluster
> --------------------------------------------------------------------
>
> Key: HADOOP-5828
> URL: https://issues.apache.org/jira/browse/HADOOP-5828
> Project: Hadoop Core
> Issue Type: Bug
> Components: test
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Attachments: HADOOP-5828.patch
>
>
> MiniMRCluster's JobTracker sets the mapred.local.dir to a relative path: "build/test/mapred/local". While currently this does not cause any problem, if at some time in future, the JobTracker starts using LocalDirAllocator for localizing files, this could cause failures in contrib tests. This issue was faced when working with an internal patch that tried to introduce LocalDirAllocator into the JobTracker, and capacity scheduler test cases started failing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5828) Use absolute path for JobTracker's
mapred.local.dir in MiniMRCluster
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12709272#action_12709272 ]
Hemanth Yamijala commented on HADOOP-5828:
------------------------------------------
The failure was in DiskChecker.checkDir(), which called File.isDirectory() on the relative mapred.local.dir path. Debugging this, we found the issue to be one documented in http://bugs.sun.com/bugdatabase/view_bug.do;:YfiG?bug_id=4483097. When relative paths are passed to java.io.* APIs, some of them resolve them to absolute paths by prepending the path with the value of the system property user.dir. Some of them resolve them relative to the current working directory.
For contrib tests, we set the user.dir to a value that will typically be different from the current working directory. This triggers inconsistencies in the resolution to absolute files and causes APIs like File.isDirectory() to fail.
The recommendation from Sun is to use absolute rather than relative paths for such cases. Hence, the issue.
> Use absolute path for JobTracker's mapred.local.dir in MiniMRCluster
> --------------------------------------------------------------------
>
> Key: HADOOP-5828
> URL: https://issues.apache.org/jira/browse/HADOOP-5828
> Project: Hadoop Core
> Issue Type: Bug
> Components: test
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
>
> MiniMRCluster's JobTracker sets the mapred.local.dir to a relative path: "build/test/mapred/local". While currently this does not cause any problem, if at some time in future, the JobTracker starts using LocalDirAllocator for localizing files, this could cause failures in contrib tests. This issue was faced when working with an internal patch that tried to introduce LocalDirAllocator into the JobTracker, and capacity scheduler test cases started failing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5828) Use absolute path for JobTracker's
mapred.local.dir in MiniMRCluster
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12709768#action_12709768 ]
Hemanth Yamijala commented on HADOOP-5828:
------------------------------------------
I ran tests locally. Got some failures, (like in streaming), but those tests are failing on trunk as well. Guess this is good to go.
> Use absolute path for JobTracker's mapred.local.dir in MiniMRCluster
> --------------------------------------------------------------------
>
> Key: HADOOP-5828
> URL: https://issues.apache.org/jira/browse/HADOOP-5828
> Project: Hadoop Core
> Issue Type: Bug
> Components: test
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Attachments: HADOOP-5828.patch
>
>
> MiniMRCluster's JobTracker sets the mapred.local.dir to a relative path: "build/test/mapred/local". While currently this does not cause any problem, if at some time in future, the JobTracker starts using LocalDirAllocator for localizing files, this could cause failures in contrib tests. This issue was faced when working with an internal patch that tried to introduce LocalDirAllocator into the JobTracker, and capacity scheduler test cases started failing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5828) Use absolute path for JobTracker's
mapred.local.dir in MiniMRCluster
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-5828:
-------------------------------------
Status: Open (was: Patch Available)
> Use absolute path for JobTracker's mapred.local.dir in MiniMRCluster
> --------------------------------------------------------------------
>
> Key: HADOOP-5828
> URL: https://issues.apache.org/jira/browse/HADOOP-5828
> Project: Hadoop Core
> Issue Type: Bug
> Components: test
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Attachments: HADOOP-5828.20.patch, HADOOP-5828.patch
>
>
> MiniMRCluster's JobTracker sets the mapred.local.dir to a relative path: "build/test/mapred/local". While currently this does not cause any problem, if at some time in future, the JobTracker starts using LocalDirAllocator for localizing files, this could cause failures in contrib tests. This issue was faced when working with an internal patch that tried to introduce LocalDirAllocator into the JobTracker, and capacity scheduler test cases started failing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5828) Use absolute path for JobTracker's
mapred.local.dir in MiniMRCluster
Posted by "Sreekanth Ramakrishnan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12709277#action_12709277 ]
Sreekanth Ramakrishnan commented on HADOOP-5828:
------------------------------------------------
+1 to the patch. The changes looks fine.
> Use absolute path for JobTracker's mapred.local.dir in MiniMRCluster
> --------------------------------------------------------------------
>
> Key: HADOOP-5828
> URL: https://issues.apache.org/jira/browse/HADOOP-5828
> Project: Hadoop Core
> Issue Type: Bug
> Components: test
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Attachments: HADOOP-5828.patch
>
>
> MiniMRCluster's JobTracker sets the mapred.local.dir to a relative path: "build/test/mapred/local". While currently this does not cause any problem, if at some time in future, the JobTracker starts using LocalDirAllocator for localizing files, this could cause failures in contrib tests. This issue was faced when working with an internal patch that tried to introduce LocalDirAllocator into the JobTracker, and capacity scheduler test cases started failing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5828) Use absolute path for JobTracker's
mapred.local.dir in MiniMRCluster
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-5828:
-------------------------------------
Status: Patch Available (was: Open)
Running through Hudson.
> Use absolute path for JobTracker's mapred.local.dir in MiniMRCluster
> --------------------------------------------------------------------
>
> Key: HADOOP-5828
> URL: https://issues.apache.org/jira/browse/HADOOP-5828
> Project: Hadoop Core
> Issue Type: Bug
> Components: test
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Attachments: HADOOP-5828.patch
>
>
> MiniMRCluster's JobTracker sets the mapred.local.dir to a relative path: "build/test/mapred/local". While currently this does not cause any problem, if at some time in future, the JobTracker starts using LocalDirAllocator for localizing files, this could cause failures in contrib tests. This issue was faced when working with an internal patch that tried to introduce LocalDirAllocator into the JobTracker, and capacity scheduler test cases started failing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.