You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Vinod Kumar Vavilapalli (Created) (JIRA)" <ji...@apache.org> on 2012/03/08 20:45:57 UTC
[jira] [Created] (MAPREDUCE-3988) mapreduce.job.local.dir doesn't
point to a single directory on a node.
mapreduce.job.local.dir doesn't point to a single directory on a node.
----------------------------------------------------------------------
Key: MAPREDUCE-3988
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3988
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Vinod Kumar Vavilapalli
After MAPREDUCE-3975, mapreduce.job.local.dir is set correctly for the tasks but it doesn't point to the same directory for all tasks running on the node.
It is a public API. Either we should point to a single directory or point it to all directories and change the documentation to say that it points to all dirs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3988) mapreduce.job.local.dir doesn't
point to a single directory on a node.
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13245599#comment-13245599 ]
Hudson commented on MAPREDUCE-3988:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #2057 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2057/])
MAPREDUCE-3988. mapreduce.job.local.dir doesn't point to a single directory on a node. (Eric Payne via bobby) (Revision 1309086)
Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309086
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRChildTask.java
> mapreduce.job.local.dir doesn't point to a single directory on a node.
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-3988
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3988
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Eric Payne
> Fix For: 0.23.3, 2.0.0
>
> Attachments: MAPREDUCE-3988-1.txt, MAPREDUCE-3988-2.txt, MAPREDUCE-3988-2.txt
>
>
> After MAPREDUCE-3975, mapreduce.job.local.dir is set correctly for the tasks but it doesn't point to the same directory for all tasks running on the node.
> It is a public API. Either we should point to a single directory or point it to all directories and change the documentation to say that it points to all dirs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3988) mapreduce.job.local.dir doesn't
point to a single directory on a node.
Posted by "Robert Joseph Evans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-3988:
-------------------------------------------
Resolution: Fixed
Fix Version/s: 2.0.0
0.23.3
Status: Resolved (was: Patch Available)
Thanks Eric. I checked this into trunk, branch-2, and branch-0.23
> mapreduce.job.local.dir doesn't point to a single directory on a node.
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-3988
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3988
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Eric Payne
> Fix For: 0.23.3, 2.0.0
>
> Attachments: MAPREDUCE-3988-1.txt, MAPREDUCE-3988-2.txt, MAPREDUCE-3988-2.txt
>
>
> After MAPREDUCE-3975, mapreduce.job.local.dir is set correctly for the tasks but it doesn't point to the same directory for all tasks running on the node.
> It is a public API. Either we should point to a single directory or point it to all directories and change the documentation to say that it points to all dirs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3988) mapreduce.job.local.dir doesn't
point to a single directory on a node.
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13245618#comment-13245618 ]
Hudson commented on MAPREDUCE-3988:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #1995 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1995/])
MAPREDUCE-3988. mapreduce.job.local.dir doesn't point to a single directory on a node. (Eric Payne via bobby) (Revision 1309086)
Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309086
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRChildTask.java
> mapreduce.job.local.dir doesn't point to a single directory on a node.
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-3988
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3988
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Eric Payne
> Fix For: 0.23.3, 2.0.0
>
> Attachments: MAPREDUCE-3988-1.txt, MAPREDUCE-3988-2.txt, MAPREDUCE-3988-2.txt
>
>
> After MAPREDUCE-3975, mapreduce.job.local.dir is set correctly for the tasks but it doesn't point to the same directory for all tasks running on the node.
> It is a public API. Either we should point to a single directory or point it to all directories and change the documentation to say that it points to all dirs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3988) mapreduce.job.local.dir doesn't
point to a single directory on a node.
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13228168#comment-13228168 ]
Hadoop QA commented on MAPREDUCE-3988:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12518139/MAPREDUCE-3988-1.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2042//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2042//console
This message is automatically generated.
> mapreduce.job.local.dir doesn't point to a single directory on a node.
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-3988
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3988
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Eric Payne
> Attachments: MAPREDUCE-3988-1.txt
>
>
> After MAPREDUCE-3975, mapreduce.job.local.dir is set correctly for the tasks but it doesn't point to the same directory for all tasks running on the node.
> It is a public API. Either we should point to a single directory or point it to all directories and change the documentation to say that it points to all dirs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3988) mapreduce.job.local.dir doesn't
point to a single directory on a node.
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13246226#comment-13246226 ]
Hudson commented on MAPREDUCE-3988:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk #1005 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1005/])
MAPREDUCE-3988. mapreduce.job.local.dir doesn't point to a single directory on a node. (Eric Payne via bobby) (Revision 1309086)
Result = FAILURE
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309086
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRChildTask.java
> mapreduce.job.local.dir doesn't point to a single directory on a node.
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-3988
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3988
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Eric Payne
> Fix For: 0.23.3, 2.0.0
>
> Attachments: MAPREDUCE-3988-1.txt, MAPREDUCE-3988-2.txt, MAPREDUCE-3988-2.txt
>
>
> After MAPREDUCE-3975, mapreduce.job.local.dir is set correctly for the tasks but it doesn't point to the same directory for all tasks running on the node.
> It is a public API. Either we should point to a single directory or point it to all directories and change the documentation to say that it points to all dirs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3988) mapreduce.job.local.dir doesn't
point to a single directory on a node.
Posted by "Eric Payne (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Payne updated MAPREDUCE-3988:
----------------------------------
Status: Patch Available (was: Open)
> mapreduce.job.local.dir doesn't point to a single directory on a node.
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-3988
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3988
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Eric Payne
> Attachments: MAPREDUCE-3988-1.txt, MAPREDUCE-3988-2.txt, MAPREDUCE-3988-2.txt
>
>
> After MAPREDUCE-3975, mapreduce.job.local.dir is set correctly for the tasks but it doesn't point to the same directory for all tasks running on the node.
> It is a public API. Either we should point to a single directory or point it to all directories and change the documentation to say that it points to all dirs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3988) mapreduce.job.local.dir doesn't
point to a single directory on a node.
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13244345#comment-13244345 ]
Hadoop QA commented on MAPREDUCE-3988:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12520987/MAPREDUCE-3988-2.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry
org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2126//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2126//console
This message is automatically generated.
> mapreduce.job.local.dir doesn't point to a single directory on a node.
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-3988
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3988
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Eric Payne
> Attachments: MAPREDUCE-3988-1.txt, MAPREDUCE-3988-2.txt, MAPREDUCE-3988-2.txt
>
>
> After MAPREDUCE-3975, mapreduce.job.local.dir is set correctly for the tasks but it doesn't point to the same directory for all tasks running on the node.
> It is a public API. Either we should point to a single directory or point it to all directories and change the documentation to say that it points to all dirs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3988) mapreduce.job.local.dir doesn't
point to a single directory on a node.
Posted by "Eric Payne (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Payne updated MAPREDUCE-3988:
----------------------------------
Status: Open (was: Patch Available)
> mapreduce.job.local.dir doesn't point to a single directory on a node.
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-3988
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3988
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Eric Payne
> Attachments: MAPREDUCE-3988-1.txt
>
>
> After MAPREDUCE-3975, mapreduce.job.local.dir is set correctly for the tasks but it doesn't point to the same directory for all tasks running on the node.
> It is a public API. Either we should point to a single directory or point it to all directories and change the documentation to say that it points to all dirs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3988) mapreduce.job.local.dir doesn't
point to a single directory on a node.
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13246208#comment-13246208 ]
Hudson commented on MAPREDUCE-3988:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #218 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/218/])
svn merge -c 1309086 from trunk. FIXES MAPREDUCE-3988. mapreduce.job.local.dir doesn't point to a single directory on a node. (Eric Payne via bobby) (Revision 1309088)
Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309088
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRChildTask.java
> mapreduce.job.local.dir doesn't point to a single directory on a node.
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-3988
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3988
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Eric Payne
> Fix For: 0.23.3, 2.0.0
>
> Attachments: MAPREDUCE-3988-1.txt, MAPREDUCE-3988-2.txt, MAPREDUCE-3988-2.txt
>
>
> After MAPREDUCE-3975, mapreduce.job.local.dir is set correctly for the tasks but it doesn't point to the same directory for all tasks running on the node.
> It is a public API. Either we should point to a single directory or point it to all directories and change the documentation to say that it points to all dirs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3988) mapreduce.job.local.dir doesn't
point to a single directory on a node.
Posted by "Robert Joseph Evans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13228390#comment-13228390 ]
Robert Joseph Evans commented on MAPREDUCE-3988:
------------------------------------------------
Looking at the patch I think it looks good to me, but I would like Vinod's opinion on it before checking it in, because the last time I jumped the gun and that is why we need this patch.
> mapreduce.job.local.dir doesn't point to a single directory on a node.
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-3988
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3988
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Eric Payne
> Attachments: MAPREDUCE-3988-1.txt
>
>
> After MAPREDUCE-3975, mapreduce.job.local.dir is set correctly for the tasks but it doesn't point to the same directory for all tasks running on the node.
> It is a public API. Either we should point to a single directory or point it to all directories and change the documentation to say that it points to all dirs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3988) mapreduce.job.local.dir doesn't
point to a single directory on a node.
Posted by "Eric Payne (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Payne reassigned MAPREDUCE-3988:
-------------------------------------
Assignee: Eric Payne
> mapreduce.job.local.dir doesn't point to a single directory on a node.
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-3988
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3988
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Eric Payne
>
> After MAPREDUCE-3975, mapreduce.job.local.dir is set correctly for the tasks but it doesn't point to the same directory for all tasks running on the node.
> It is a public API. Either we should point to a single directory or point it to all directories and change the documentation to say that it points to all dirs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3988) mapreduce.job.local.dir doesn't
point to a single directory on a node.
Posted by "Eric Payne (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Payne updated MAPREDUCE-3988:
----------------------------------
Attachment: MAPREDUCE-3988-2.txt
Upmerged patch to branch-0.23, branch-2 and trunk.
Same patch applies to all.
> mapreduce.job.local.dir doesn't point to a single directory on a node.
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-3988
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3988
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Eric Payne
> Attachments: MAPREDUCE-3988-1.txt, MAPREDUCE-3988-2.txt
>
>
> After MAPREDUCE-3975, mapreduce.job.local.dir is set correctly for the tasks but it doesn't point to the same directory for all tasks running on the node.
> It is a public API. Either we should point to a single directory or point it to all directories and change the documentation to say that it points to all dirs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3988) mapreduce.job.local.dir doesn't
point to a single directory on a node.
Posted by "Eric Payne (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Payne updated MAPREDUCE-3988:
----------------------------------
Status: Patch Available (was: Open)
> mapreduce.job.local.dir doesn't point to a single directory on a node.
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-3988
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3988
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Eric Payne
> Attachments: MAPREDUCE-3988-1.txt
>
>
> After MAPREDUCE-3975, mapreduce.job.local.dir is set correctly for the tasks but it doesn't point to the same directory for all tasks running on the node.
> It is a public API. Either we should point to a single directory or point it to all directories and change the documentation to say that it points to all dirs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3988) mapreduce.job.local.dir doesn't
point to a single directory on a node.
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13246270#comment-13246270 ]
Hudson commented on MAPREDUCE-3988:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #1040 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1040/])
MAPREDUCE-3988. mapreduce.job.local.dir doesn't point to a single directory on a node. (Eric Payne via bobby) (Revision 1309086)
Result = FAILURE
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309086
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRChildTask.java
> mapreduce.job.local.dir doesn't point to a single directory on a node.
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-3988
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3988
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Eric Payne
> Fix For: 0.23.3, 2.0.0
>
> Attachments: MAPREDUCE-3988-1.txt, MAPREDUCE-3988-2.txt, MAPREDUCE-3988-2.txt
>
>
> After MAPREDUCE-3975, mapreduce.job.local.dir is set correctly for the tasks but it doesn't point to the same directory for all tasks running on the node.
> It is a public API. Either we should point to a single directory or point it to all directories and change the documentation to say that it points to all dirs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3988) mapreduce.job.local.dir doesn't
point to a single directory on a node.
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13245605#comment-13245605 ]
Hudson commented on MAPREDUCE-3988:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #1982 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1982/])
MAPREDUCE-3988. mapreduce.job.local.dir doesn't point to a single directory on a node. (Eric Payne via bobby) (Revision 1309086)
Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309086
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRChildTask.java
> mapreduce.job.local.dir doesn't point to a single directory on a node.
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-3988
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3988
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Eric Payne
> Fix For: 0.23.3, 2.0.0
>
> Attachments: MAPREDUCE-3988-1.txt, MAPREDUCE-3988-2.txt, MAPREDUCE-3988-2.txt
>
>
> After MAPREDUCE-3975, mapreduce.job.local.dir is set correctly for the tasks but it doesn't point to the same directory for all tasks running on the node.
> It is a public API. Either we should point to a single directory or point it to all directories and change the documentation to say that it points to all dirs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3988) mapreduce.job.local.dir doesn't
point to a single directory on a node.
Posted by "Eric Payne (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Payne updated MAPREDUCE-3988:
----------------------------------
Attachment: MAPREDUCE-3988-2.txt
> mapreduce.job.local.dir doesn't point to a single directory on a node.
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-3988
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3988
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Eric Payne
> Attachments: MAPREDUCE-3988-1.txt, MAPREDUCE-3988-2.txt, MAPREDUCE-3988-2.txt
>
>
> After MAPREDUCE-3975, mapreduce.job.local.dir is set correctly for the tasks but it doesn't point to the same directory for all tasks running on the node.
> It is a public API. Either we should point to a single directory or point it to all directories and change the documentation to say that it points to all dirs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3988) mapreduce.job.local.dir doesn't
point to a single directory on a node.
Posted by "Eric Payne (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Payne updated MAPREDUCE-3988:
----------------------------------
Attachment: MAPREDUCE-3988-1.txt
This patch uses the LocalDirAllocator.getLocalPathToRead() API to first check for existence of a scratch directory before using the LocalDirAllocator.getLocalPathForWrite() API if the directory does not exist.
There are a couple of race conditions with this implementation because of the fact that each YarnChild task attempt is running in its own JVM. This patch addresses one of the race conditions.
The race addressed by this patch is when 2 or more tasks at the same time detect there is no mapreduce.job.local.dir and try to create it in the same directory at the same time. Whichever one loses the race will get a FileAlreadyExistsException, which is ignored.
There is still a race condition wherein 2 tasks running on the same node could both see that there is no mapreduce.job.local.dir and then try to create it in 2 different directories; for eg, one in /grid/0 and one in /grid/1.
We could go a step further in addressing the second race and try to hash the attempt ID and map it to the same directory, but that seems overly complicated.
> mapreduce.job.local.dir doesn't point to a single directory on a node.
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-3988
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3988
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Eric Payne
> Attachments: MAPREDUCE-3988-1.txt
>
>
> After MAPREDUCE-3975, mapreduce.job.local.dir is set correctly for the tasks but it doesn't point to the same directory for all tasks running on the node.
> It is a public API. Either we should point to a single directory or point it to all directories and change the documentation to say that it points to all dirs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira