You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Jonathan Eagles (Created) (JIRA)" <ji...@apache.org> on 2012/01/24 18:00:41 UTC
[jira] [Created] (MAPREDUCE-3716) java.io.File.createTempFile fails
in map/reduce tasks
java.io.File.createTempFile fails in map/reduce tasks
-----------------------------------------------------
Key: MAPREDUCE-3716
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3716
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.1
Reporter: Jonathan Eagles
Assignee: Jonathan Eagles
Priority: Blocker
container-launch.sh specifies java option java.io.tmpdir when executing the java process for the child container but fails to create the tmpdir. All uses of createTempFile and other commands relying on java.io.tmpdir will fail when called from child container jvms.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3716) java.io.File.createTempFile fails
in map/reduce tasks
Posted by "Jonathan Eagles (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Eagles updated MAPREDUCE-3716:
---------------------------------------
Status: Patch Available (was: Open)
> java.io.File.createTempFile fails in map/reduce tasks
> -----------------------------------------------------
>
> Key: MAPREDUCE-3716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Jonathan Eagles
> Assignee: Jonathan Eagles
> Priority: Blocker
> Attachments: MAPREDUCE-3716.patch, MAPREDUCE-3716.patch, TestRandomWriter.patch
>
>
> container-launch.sh specifies java option java.io.tmpdir when executing the java process for the child container but fails to create the tmpdir. All uses of createTempFile and other commands relying on java.io.tmpdir will fail when called from child container jvms.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3716) java.io.File.createTempFile
fails in map/reduce tasks
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196569#comment-13196569 ]
Hadoop QA commented on MAPREDUCE-3716:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12512489/MAPREDUCE-3716.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1715//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1715//console
This message is automatically generated.
> java.io.File.createTempFile fails in map/reduce tasks
> -----------------------------------------------------
>
> Key: MAPREDUCE-3716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Jonathan Eagles
> Assignee: Jonathan Eagles
> Priority: Blocker
> Attachments: MAPREDUCE-3716.patch, MAPREDUCE-3716.patch, TestRandomWriter.patch
>
>
> container-launch.sh specifies java option java.io.tmpdir when executing the java process for the child container but fails to create the tmpdir. All uses of createTempFile and other commands relying on java.io.tmpdir will fail when called from child container jvms.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3716) java.io.File.createTempFile fails
in map/reduce tasks
Posted by "Jonathan Eagles (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Eagles updated MAPREDUCE-3716:
---------------------------------------
Attachment: MAPREDUCE-3716.patch
> java.io.File.createTempFile fails in map/reduce tasks
> -----------------------------------------------------
>
> Key: MAPREDUCE-3716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Jonathan Eagles
> Assignee: Jonathan Eagles
> Priority: Blocker
> Attachments: MAPREDUCE-3716.patch, MAPREDUCE-3716.patch, TestRandomWriter.patch
>
>
> container-launch.sh specifies java option java.io.tmpdir when executing the java process for the child container but fails to create the tmpdir. All uses of createTempFile and other commands relying on java.io.tmpdir will fail when called from child container jvms.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3716) java.io.File.createTempFile
fails in map/reduce tasks
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196882#comment-13196882 ]
Hudson commented on MAPREDUCE-3716:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #155 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/155/])
MAPREDUCE-3716. Fixing YARN+MR to allow MR jobs to be able to use java.io.File.createTempFile to create temporary files as part of their tasks. Contributed by Jonathan Eagles.
svn merge --ignore-ancestry -c 1238136 ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238138
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRChildTask.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.h
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java/mapred-default.xml
> java.io.File.createTempFile fails in map/reduce tasks
> -----------------------------------------------------
>
> Key: MAPREDUCE-3716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Jonathan Eagles
> Assignee: Jonathan Eagles
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3716.patch, MAPREDUCE-3716.patch, TestRandomWriter.patch
>
>
> container-launch.sh specifies java option java.io.tmpdir when executing the java process for the child container but fails to create the tmpdir. All uses of createTempFile and other commands relying on java.io.tmpdir will fail when called from child container jvms.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3716) java.io.File.createTempFile
fails in map/reduce tasks
Posted by "Vinod Kumar Vavilapalli (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193577#comment-13193577 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-3716:
----------------------------------------------------
Patch looks fine overall.
For MR case, we already have "mapreduce.task.tmp.dir". Ideally we will want to support this, but that seems too much of a change to me as it needs to passed as part of each container-request. Not much of use anyways.
I guess we can live with "./tmp". In that case, we should
- definitely remove "mapreduce.task.tmp.dir" from mapped-default.xml
- declare this a YARN level constant and use this throughout - MapReduceChildJVM, DefaultContainerExecutor
- emit a warning in JobClient if someone configures it to be other than "./tmp".
Also, a test for this is useful, please see if you can port {{TestMiniMRChildTask}} to MiniMRYarnCluster.
> java.io.File.createTempFile fails in map/reduce tasks
> -----------------------------------------------------
>
> Key: MAPREDUCE-3716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Jonathan Eagles
> Assignee: Jonathan Eagles
> Priority: Blocker
> Attachments: MAPREDUCE-3716.patch, TestRandomWriter.patch
>
>
> container-launch.sh specifies java option java.io.tmpdir when executing the java process for the child container but fails to create the tmpdir. All uses of createTempFile and other commands relying on java.io.tmpdir will fail when called from child container jvms.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3716) java.io.File.createTempFile
fails in map/reduce tasks
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196870#comment-13196870 ]
Hudson commented on MAPREDUCE-3716:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk #942 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/942/])
MAPREDUCE-3716. Fixing YARN+MR to allow MR jobs to be able to use java.io.File.createTempFile to create temporary files as part of their tasks. Contributed by Jonathan Eagles.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238136
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRChildTask.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.h
* /hadoop/common/trunk/hadoop-mapreduce-project/src/java/mapred-default.xml
> java.io.File.createTempFile fails in map/reduce tasks
> -----------------------------------------------------
>
> Key: MAPREDUCE-3716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Jonathan Eagles
> Assignee: Jonathan Eagles
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3716.patch, MAPREDUCE-3716.patch, TestRandomWriter.patch
>
>
> container-launch.sh specifies java option java.io.tmpdir when executing the java process for the child container but fails to create the tmpdir. All uses of createTempFile and other commands relying on java.io.tmpdir will fail when called from child container jvms.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3716) java.io.File.createTempFile
fails in map/reduce tasks
Posted by "Jonathan Eagles (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193408#comment-13193408 ]
Jonathan Eagles commented on MAPREDUCE-3716:
--------------------------------------------
There are differences in 23+ about tmp dir location now being under the current working dir. Due to this, could not add back the tmp dir creation in MapReduceChildJVM.java. Opted to move the tmp dir creation to the DefaultContainerExecutor/container-executor with the rest of the directory creation. Had difficulty adding tests for this since DefaultContainerExecutor tests are disabled.
> java.io.File.createTempFile fails in map/reduce tasks
> -----------------------------------------------------
>
> Key: MAPREDUCE-3716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Jonathan Eagles
> Assignee: Jonathan Eagles
> Priority: Blocker
> Attachments: MAPREDUCE-3716.patch, TestRandomWriter.patch
>
>
> container-launch.sh specifies java option java.io.tmpdir when executing the java process for the child container but fails to create the tmpdir. All uses of createTempFile and other commands relying on java.io.tmpdir will fail when called from child container jvms.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3716) java.io.File.createTempFile fails
in map/reduce tasks
Posted by "Mahadev konar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-3716:
-------------------------------------
Status: Open (was: Patch Available)
@Jon,
Mind taking a look at the improvements Vinod suggested?
> java.io.File.createTempFile fails in map/reduce tasks
> -----------------------------------------------------
>
> Key: MAPREDUCE-3716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Jonathan Eagles
> Assignee: Jonathan Eagles
> Priority: Blocker
> Attachments: MAPREDUCE-3716.patch, TestRandomWriter.patch
>
>
> container-launch.sh specifies java option java.io.tmpdir when executing the java process for the child container but fails to create the tmpdir. All uses of createTempFile and other commands relying on java.io.tmpdir will fail when called from child container jvms.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3716) java.io.File.createTempFile
fails in map/reduce tasks
Posted by "Jonathan Eagles (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196517#comment-13196517 ]
Jonathan Eagles commented on MAPREDUCE-3716:
--------------------------------------------
bq. * definitely remove "mapreduce.task.tmp.dir" from mapped-default.xml
done
bq. * declare this a YARN level constant and use this throughout - MapReduceChildJVM, DefaultContainerExecutor
done
bq. * emit a warning in JobClient if someone configures it to be other than "./tmp".
done mapreduce.task.tmp.dir is marked a final var and produces a warning
bq. * Also, a test for this is useful, please see if you can port TestMiniMRChildTask to MiniMRYarnCluster.
done. ported the TestMiniMRChildTask#testTaskTempDir
> java.io.File.createTempFile fails in map/reduce tasks
> -----------------------------------------------------
>
> Key: MAPREDUCE-3716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Jonathan Eagles
> Assignee: Jonathan Eagles
> Priority: Blocker
> Attachments: MAPREDUCE-3716.patch, MAPREDUCE-3716.patch, TestRandomWriter.patch
>
>
> container-launch.sh specifies java option java.io.tmpdir when executing the java process for the child container but fails to create the tmpdir. All uses of createTempFile and other commands relying on java.io.tmpdir will fail when called from child container jvms.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3716) java.io.File.createTempFile fails
in map/reduce tasks
Posted by "Jonathan Eagles (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Eagles updated MAPREDUCE-3716:
---------------------------------------
Attachment: TestRandomWriter.patch
Adding a patch that illustrates the error.
Example:
hadoop jar hadoop-mapreduce-examples-0.23.1-SNAPSHOT.jar randomwriter -Dmapreduce.randomwriter.mapsperhost=1 /rw1
> java.io.File.createTempFile fails in map/reduce tasks
> -----------------------------------------------------
>
> Key: MAPREDUCE-3716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Jonathan Eagles
> Assignee: Jonathan Eagles
> Priority: Blocker
> Attachments: TestRandomWriter.patch
>
>
> container-launch.sh specifies java option java.io.tmpdir when executing the java process for the child container but fails to create the tmpdir. All uses of createTempFile and other commands relying on java.io.tmpdir will fail when called from child container jvms.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3716) java.io.File.createTempFile fails
in map/reduce tasks
Posted by "Jonathan Eagles (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Eagles updated MAPREDUCE-3716:
---------------------------------------
Status: Patch Available (was: Open)
> java.io.File.createTempFile fails in map/reduce tasks
> -----------------------------------------------------
>
> Key: MAPREDUCE-3716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Jonathan Eagles
> Assignee: Jonathan Eagles
> Priority: Blocker
> Attachments: MAPREDUCE-3716.patch, TestRandomWriter.patch
>
>
> container-launch.sh specifies java option java.io.tmpdir when executing the java process for the child container but fails to create the tmpdir. All uses of createTempFile and other commands relying on java.io.tmpdir will fail when called from child container jvms.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3716) java.io.File.createTempFile
fails in map/reduce tasks
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196658#comment-13196658 ]
Hudson commented on MAPREDUCE-3716:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #1691 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1691/])
MAPREDUCE-3716. Fixing YARN+MR to allow MR jobs to be able to use java.io.File.createTempFile to create temporary files as part of their tasks. Contributed by Jonathan Eagles.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238136
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRChildTask.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.h
* /hadoop/common/trunk/hadoop-mapreduce-project/src/java/mapred-default.xml
> java.io.File.createTempFile fails in map/reduce tasks
> -----------------------------------------------------
>
> Key: MAPREDUCE-3716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Jonathan Eagles
> Assignee: Jonathan Eagles
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3716.patch, MAPREDUCE-3716.patch, TestRandomWriter.patch
>
>
> container-launch.sh specifies java option java.io.tmpdir when executing the java process for the child container but fails to create the tmpdir. All uses of createTempFile and other commands relying on java.io.tmpdir will fail when called from child container jvms.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3716) java.io.File.createTempFile
fails in map/reduce tasks
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196660#comment-13196660 ]
Hudson commented on MAPREDUCE-3716:
-----------------------------------
Integrated in Hadoop-Common-0.23-Commit #448 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/448/])
MAPREDUCE-3716. Fixing YARN+MR to allow MR jobs to be able to use java.io.File.createTempFile to create temporary files as part of their tasks. Contributed by Jonathan Eagles.
svn merge --ignore-ancestry -c 1238136 ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238138
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRChildTask.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.h
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java/mapred-default.xml
> java.io.File.createTempFile fails in map/reduce tasks
> -----------------------------------------------------
>
> Key: MAPREDUCE-3716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Jonathan Eagles
> Assignee: Jonathan Eagles
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3716.patch, MAPREDUCE-3716.patch, TestRandomWriter.patch
>
>
> container-launch.sh specifies java option java.io.tmpdir when executing the java process for the child container but fails to create the tmpdir. All uses of createTempFile and other commands relying on java.io.tmpdir will fail when called from child container jvms.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3716) java.io.File.createTempFile
fails in map/reduce tasks
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196662#comment-13196662 ]
Hudson commented on MAPREDUCE-3716:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Commit #439 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/439/])
MAPREDUCE-3716. Fixing YARN+MR to allow MR jobs to be able to use java.io.File.createTempFile to create temporary files as part of their tasks. Contributed by Jonathan Eagles.
svn merge --ignore-ancestry -c 1238136 ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238138
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRChildTask.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.h
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java/mapred-default.xml
> java.io.File.createTempFile fails in map/reduce tasks
> -----------------------------------------------------
>
> Key: MAPREDUCE-3716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Jonathan Eagles
> Assignee: Jonathan Eagles
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3716.patch, MAPREDUCE-3716.patch, TestRandomWriter.patch
>
>
> container-launch.sh specifies java option java.io.tmpdir when executing the java process for the child container but fails to create the tmpdir. All uses of createTempFile and other commands relying on java.io.tmpdir will fail when called from child container jvms.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3716) java.io.File.createTempFile
fails in map/reduce tasks
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196928#comment-13196928 ]
Hudson commented on MAPREDUCE-3716:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Build #177 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/177/])
MAPREDUCE-3716. Fixing YARN+MR to allow MR jobs to be able to use java.io.File.createTempFile to create temporary files as part of their tasks. Contributed by Jonathan Eagles.
svn merge --ignore-ancestry -c 1238136 ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238138
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRChildTask.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.h
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java/mapred-default.xml
> java.io.File.createTempFile fails in map/reduce tasks
> -----------------------------------------------------
>
> Key: MAPREDUCE-3716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Jonathan Eagles
> Assignee: Jonathan Eagles
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3716.patch, MAPREDUCE-3716.patch, TestRandomWriter.patch
>
>
> container-launch.sh specifies java option java.io.tmpdir when executing the java process for the child container but fails to create the tmpdir. All uses of createTempFile and other commands relying on java.io.tmpdir will fail when called from child container jvms.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3716) java.io.File.createTempFile
fails in map/reduce tasks
Posted by "Jonathan Eagles (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194149#comment-13194149 ]
Jonathan Eagles commented on MAPREDUCE-3716:
--------------------------------------------
Good comments, Vinod. Work in progress.
> java.io.File.createTempFile fails in map/reduce tasks
> -----------------------------------------------------
>
> Key: MAPREDUCE-3716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Jonathan Eagles
> Assignee: Jonathan Eagles
> Priority: Blocker
> Attachments: MAPREDUCE-3716.patch, TestRandomWriter.patch
>
>
> container-launch.sh specifies java option java.io.tmpdir when executing the java process for the child container but fails to create the tmpdir. All uses of createTempFile and other commands relying on java.io.tmpdir will fail when called from child container jvms.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3716) java.io.File.createTempFile
fails in map/reduce tasks
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196661#comment-13196661 ]
Hudson commented on MAPREDUCE-3716:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #1619 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1619/])
MAPREDUCE-3716. Fixing YARN+MR to allow MR jobs to be able to use java.io.File.createTempFile to create temporary files as part of their tasks. Contributed by Jonathan Eagles.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238136
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRChildTask.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.h
* /hadoop/common/trunk/hadoop-mapreduce-project/src/java/mapred-default.xml
> java.io.File.createTempFile fails in map/reduce tasks
> -----------------------------------------------------
>
> Key: MAPREDUCE-3716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Jonathan Eagles
> Assignee: Jonathan Eagles
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3716.patch, MAPREDUCE-3716.patch, TestRandomWriter.patch
>
>
> container-launch.sh specifies java option java.io.tmpdir when executing the java process for the child container but fails to create the tmpdir. All uses of createTempFile and other commands relying on java.io.tmpdir will fail when called from child container jvms.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3716) java.io.File.createTempFile
fails in map/reduce tasks
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193425#comment-13193425 ]
Hadoop QA commented on MAPREDUCE-3716:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12511896/MAPREDUCE-3716.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.mapred.TestJobClientGetJob
org.apache.hadoop.mapred.TestMRWithDistributedCache
org.apache.hadoop.mapred.TestLocalModeWithNewApis
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1677//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1677//console
This message is automatically generated.
> java.io.File.createTempFile fails in map/reduce tasks
> -----------------------------------------------------
>
> Key: MAPREDUCE-3716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Jonathan Eagles
> Assignee: Jonathan Eagles
> Priority: Blocker
> Attachments: MAPREDUCE-3716.patch, TestRandomWriter.patch
>
>
> container-launch.sh specifies java option java.io.tmpdir when executing the java process for the child container but fails to create the tmpdir. All uses of createTempFile and other commands relying on java.io.tmpdir will fail when called from child container jvms.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3716) java.io.File.createTempFile fails
in map/reduce tasks
Posted by "Jonathan Eagles (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Eagles updated MAPREDUCE-3716:
---------------------------------------
Attachment: MAPREDUCE-3716.patch
> java.io.File.createTempFile fails in map/reduce tasks
> -----------------------------------------------------
>
> Key: MAPREDUCE-3716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Jonathan Eagles
> Assignee: Jonathan Eagles
> Priority: Blocker
> Attachments: MAPREDUCE-3716.patch, TestRandomWriter.patch
>
>
> container-launch.sh specifies java option java.io.tmpdir when executing the java process for the child container but fails to create the tmpdir. All uses of createTempFile and other commands relying on java.io.tmpdir will fail when called from child container jvms.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3716) java.io.File.createTempFile
fails in map/reduce tasks
Posted by "Jonathan Eagles (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193400#comment-13193400 ]
Jonathan Eagles commented on MAPREDUCE-3716:
--------------------------------------------
This issue exists both using the default container executor and the linux container executor. So both need to be tested with this change.
> java.io.File.createTempFile fails in map/reduce tasks
> -----------------------------------------------------
>
> Key: MAPREDUCE-3716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Jonathan Eagles
> Assignee: Jonathan Eagles
> Priority: Blocker
> Attachments: TestRandomWriter.patch
>
>
> container-launch.sh specifies java option java.io.tmpdir when executing the java process for the child container but fails to create the tmpdir. All uses of createTempFile and other commands relying on java.io.tmpdir will fail when called from child container jvms.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3716) java.io.File.createTempFile fails
in map/reduce tasks
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3716:
-----------------------------------------------
Resolution: Fixed
Fix Version/s: 0.23.1
Release Note: Fixing YARN+MR to allow MR jobs to be able to use java.io.File.createTempFile to create temporary files as part of their tasks.
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I just committed this to trunk and branch-0.23. Thanks Jonathan!
> java.io.File.createTempFile fails in map/reduce tasks
> -----------------------------------------------------
>
> Key: MAPREDUCE-3716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Jonathan Eagles
> Assignee: Jonathan Eagles
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3716.patch, MAPREDUCE-3716.patch, TestRandomWriter.patch
>
>
> container-launch.sh specifies java option java.io.tmpdir when executing the java process for the child container but fails to create the tmpdir. All uses of createTempFile and other commands relying on java.io.tmpdir will fail when called from child container jvms.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3716) java.io.File.createTempFile
fails in map/reduce tasks
Posted by "Jonathan Eagles (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196695#comment-13196695 ]
Jonathan Eagles commented on MAPREDUCE-3716:
--------------------------------------------
Thanks, Vinod!
> java.io.File.createTempFile fails in map/reduce tasks
> -----------------------------------------------------
>
> Key: MAPREDUCE-3716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Jonathan Eagles
> Assignee: Jonathan Eagles
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3716.patch, MAPREDUCE-3716.patch, TestRandomWriter.patch
>
>
> container-launch.sh specifies java option java.io.tmpdir when executing the java process for the child container but fails to create the tmpdir. All uses of createTempFile and other commands relying on java.io.tmpdir will fail when called from child container jvms.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3716) java.io.File.createTempFile
fails in map/reduce tasks
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196670#comment-13196670 ]
Hudson commented on MAPREDUCE-3716:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #1636 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1636/])
MAPREDUCE-3716. Fixing YARN+MR to allow MR jobs to be able to use java.io.File.createTempFile to create temporary files as part of their tasks. Contributed by Jonathan Eagles.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238136
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRChildTask.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.h
* /hadoop/common/trunk/hadoop-mapreduce-project/src/java/mapred-default.xml
> java.io.File.createTempFile fails in map/reduce tasks
> -----------------------------------------------------
>
> Key: MAPREDUCE-3716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Jonathan Eagles
> Assignee: Jonathan Eagles
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3716.patch, MAPREDUCE-3716.patch, TestRandomWriter.patch
>
>
> container-launch.sh specifies java option java.io.tmpdir when executing the java process for the child container but fails to create the tmpdir. All uses of createTempFile and other commands relying on java.io.tmpdir will fail when called from child container jvms.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3716) java.io.File.createTempFile
fails in map/reduce tasks
Posted by "Vinod Kumar Vavilapalli (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196649#comment-13196649 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-3716:
----------------------------------------------------
+1. This looks good.
It'll helpful if we enable the remaining tests in TestMiniMRChildTask. But I'll open a separate ticket for that.
> java.io.File.createTempFile fails in map/reduce tasks
> -----------------------------------------------------
>
> Key: MAPREDUCE-3716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Jonathan Eagles
> Assignee: Jonathan Eagles
> Priority: Blocker
> Attachments: MAPREDUCE-3716.patch, MAPREDUCE-3716.patch, TestRandomWriter.patch
>
>
> container-launch.sh specifies java option java.io.tmpdir when executing the java process for the child container but fails to create the tmpdir. All uses of createTempFile and other commands relying on java.io.tmpdir will fail when called from child container jvms.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3716) java.io.File.createTempFile
fails in map/reduce tasks
Posted by "Jonathan Eagles (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193511#comment-13193511 ]
Jonathan Eagles commented on MAPREDUCE-3716:
--------------------------------------------
Core test failures are from MAPREDUCE-3729
> java.io.File.createTempFile fails in map/reduce tasks
> -----------------------------------------------------
>
> Key: MAPREDUCE-3716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Jonathan Eagles
> Assignee: Jonathan Eagles
> Priority: Blocker
> Attachments: MAPREDUCE-3716.patch, TestRandomWriter.patch
>
>
> container-launch.sh specifies java option java.io.tmpdir when executing the java process for the child container but fails to create the tmpdir. All uses of createTempFile and other commands relying on java.io.tmpdir will fail when called from child container jvms.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3716) java.io.File.createTempFile
fails in map/reduce tasks
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196935#comment-13196935 ]
Hudson commented on MAPREDUCE-3716:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #975 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/975/])
MAPREDUCE-3716. Fixing YARN+MR to allow MR jobs to be able to use java.io.File.createTempFile to create temporary files as part of their tasks. Contributed by Jonathan Eagles.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238136
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRChildTask.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.h
* /hadoop/common/trunk/hadoop-mapreduce-project/src/java/mapred-default.xml
> java.io.File.createTempFile fails in map/reduce tasks
> -----------------------------------------------------
>
> Key: MAPREDUCE-3716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Jonathan Eagles
> Assignee: Jonathan Eagles
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3716.patch, MAPREDUCE-3716.patch, TestRandomWriter.patch
>
>
> container-launch.sh specifies java option java.io.tmpdir when executing the java process for the child container but fails to create the tmpdir. All uses of createTempFile and other commands relying on java.io.tmpdir will fail when called from child container jvms.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3716) java.io.File.createTempFile
fails in map/reduce tasks
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196671#comment-13196671 ]
Hudson commented on MAPREDUCE-3716:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Commit #463 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/463/])
MAPREDUCE-3716. Fixing YARN+MR to allow MR jobs to be able to use java.io.File.createTempFile to create temporary files as part of their tasks. Contributed by Jonathan Eagles.
svn merge --ignore-ancestry -c 1238136 ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238138
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRChildTask.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.h
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java/mapred-default.xml
> java.io.File.createTempFile fails in map/reduce tasks
> -----------------------------------------------------
>
> Key: MAPREDUCE-3716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Jonathan Eagles
> Assignee: Jonathan Eagles
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3716.patch, MAPREDUCE-3716.patch, TestRandomWriter.patch
>
>
> container-launch.sh specifies java option java.io.tmpdir when executing the java process for the child container but fails to create the tmpdir. All uses of createTempFile and other commands relying on java.io.tmpdir will fail when called from child container jvms.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira