You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Luke Lu (JIRA)" <ji...@apache.org> on 2011/08/25 21:18:29 UTC
[jira] [Created] (MAPREDUCE-2880) Fix classpath construction for
MRv2
Fix classpath construction for MRv2
-----------------------------------
Key: MAPREDUCE-2880
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2880
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Affects Versions: 0.23.0
Reporter: Luke Lu
Fix For: 0.23.0
MRConstants.java refers a hard-coded version of MR AM jar. The build config works around with a symlink. The deployment currently needs symlink workaround as well. We need to fix this so that we can actually launch arbitrary versions of AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2880) Fix classpath construction for
MRv2
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-2880:
-------------------------------------
Attachment: MAPREDUCE-2880.patch
Final version.
Vinod - I've kept 'admin' to be compatible with 0.20.2xx. They can be overridden by users by passing in -D<> etc.
> Fix classpath construction for MRv2
> -----------------------------------
>
> Key: MAPREDUCE-2880
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2880
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Luke Lu
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch
>
>
> MRConstants.java refers a hard-coded version of MR AM jar. The build config works around with a symlink. The deployment currently needs symlink workaround as well. We need to fix this so that we can actually launch arbitrary versions of AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2880) Fix classpath construction for
MRv2
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13109733#comment-13109733 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-2880:
----------------------------------------------------
Had one final long look through the latest patch. Looks good to me. +1.
Thanks for being on top of this!
> Fix classpath construction for MRv2
> -----------------------------------
>
> Key: MAPREDUCE-2880
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2880
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Luke Lu
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch
>
>
> MRConstants.java refers a hard-coded version of MR AM jar. The build config works around with a symlink. The deployment currently needs symlink workaround as well. We need to fix this so that we can actually launch arbitrary versions of AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2880) Fix classpath construction for
MRv2
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-2880:
-------------------------------------
Status: Open (was: Patch Available)
> Fix classpath construction for MRv2
> -----------------------------------
>
> Key: MAPREDUCE-2880
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2880
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Luke Lu
> Assignee: Arun C Murthy
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2880.patch, MAPREDUCE-2880.patch
>
>
> MRConstants.java refers a hard-coded version of MR AM jar. The build config works around with a symlink. The deployment currently needs symlink workaround as well. We need to fix this so that we can actually launch arbitrary versions of AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2880) Fix classpath construction for
MRv2
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-2880:
-------------------------------------
Status: Open (was: Patch Available)
> Fix classpath construction for MRv2
> -----------------------------------
>
> Key: MAPREDUCE-2880
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2880
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Luke Lu
> Assignee: Arun C Murthy
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2880.patch
>
>
> MRConstants.java refers a hard-coded version of MR AM jar. The build config works around with a symlink. The deployment currently needs symlink workaround as well. We need to fix this so that we can actually launch arbitrary versions of AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2880) Fix classpath construction for
MRv2
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-2880:
-------------------------------------
Status: Patch Available (was: Open)
Kicking hudson...
> Fix classpath construction for MRv2
> -----------------------------------
>
> Key: MAPREDUCE-2880
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2880
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Luke Lu
> Assignee: Arun C Murthy
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2880.patch
>
>
> MRConstants.java refers a hard-coded version of MR AM jar. The build config works around with a symlink. The deployment currently needs symlink workaround as well. We need to fix this so that we can actually launch arbitrary versions of AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2880) Fix classpath construction for
MRv2
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13109610#comment-13109610 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-2880:
----------------------------------------------------
I did some debugging with the latest patch and cornered the test failures to this: The symlinks that are supposed to be created in container-dir are instead getting created in hadoop-mapreduce-jobclient dir.
*Expected*: _hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobs/org.apache.hadoop.mapreduce.v2.TestMRJobs-localDir/usercache/vinodkv/appcache/application_1316619583041_0001/container_1316619583041_0001_01_000001/MRAppJar.jar_
*Found*: _hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/MRAppJar.jar_
This is because of the change in {{DefaultContainerExecutor.launchContainer()}} which changed the currentWorkingDirectory of the shell to be null from new {{File(containerWorkDir.toUri().getPath())}}. We need to restore that, this PWD makes sense for both tests and the real world in case of _DefaultContainerExecutor_. For _LinuxContainerExecutor_, null makes sense for the real-world but not tests, but we don't have tests for LCE, so we are fine.
> Fix classpath construction for MRv2
> -----------------------------------
>
> Key: MAPREDUCE-2880
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2880
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Luke Lu
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch
>
>
> MRConstants.java refers a hard-coded version of MR AM jar. The build config works around with a symlink. The deployment currently needs symlink workaround as well. We need to fix this so that we can actually launch arbitrary versions of AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2880) Fix classpath construction for
MRv2
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13112540#comment-13112540 ]
Hudson commented on MAPREDUCE-2880:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Build #22 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/22/])
MAPREDUCE-2880. svn merge -c r1173783 --ignore-ancestry ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1173792
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DefaultSpeculator.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/MRConstants.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/BackupStore.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Constants.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MRConstants.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MROutputFiles.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ResourceMgrDelegate.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
> Fix classpath construction for MRv2
> -----------------------------------
>
> Key: MAPREDUCE-2880
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2880
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Luke Lu
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch
>
>
> MRConstants.java refers a hard-coded version of MR AM jar. The build config works around with a symlink. The deployment currently needs symlink workaround as well. We need to fix this so that we can actually launch arbitrary versions of AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2880) Fix classpath construction for
MRv2
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-2880:
-------------------------------------
Attachment: MAPREDUCE-2880.patch
Update patch, works on a secure cluster.
Work left:
* Add whitelist of env for child, not all of NM's env
* Make JAVA_HOME configurable by users so they can use alternate JVMs
* Admin opts for java (e.g. -Djava.net.preferIPv4Stack=true)
* Merge java.library.path
* Handle LD_LIBRARY_PATH from user and admin
> Fix classpath construction for MRv2
> -----------------------------------
>
> Key: MAPREDUCE-2880
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2880
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Luke Lu
> Assignee: Arun C Murthy
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2880.patch, MAPREDUCE-2880.patch
>
>
> MRConstants.java refers a hard-coded version of MR AM jar. The build config works around with a symlink. The deployment currently needs symlink workaround as well. We need to fix this so that we can actually launch arbitrary versions of AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2880) Fix classpath construction for
MRv2
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13109738#comment-13109738 ]
Hadoop QA commented on MAPREDUCE-2880:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12495416/MAPREDUCE-2880.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 9 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/813//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/813//console
This message is automatically generated.
> Fix classpath construction for MRv2
> -----------------------------------
>
> Key: MAPREDUCE-2880
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2880
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Luke Lu
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch
>
>
> MRConstants.java refers a hard-coded version of MR AM jar. The build config works around with a symlink. The deployment currently needs symlink workaround as well. We need to fix this so that we can actually launch arbitrary versions of AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2880) Fix classpath construction for
MRv2
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13107062#comment-13107062 ]
Hadoop QA commented on MAPREDUCE-2880:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12494907/MAPREDUCE-2880.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/772//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/772//console
This message is automatically generated.
> Fix classpath construction for MRv2
> -----------------------------------
>
> Key: MAPREDUCE-2880
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2880
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Luke Lu
> Assignee: Arun C Murthy
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2880.patch, MAPREDUCE-2880.patch
>
>
> MRConstants.java refers a hard-coded version of MR AM jar. The build config works around with a symlink. The deployment currently needs symlink workaround as well. We need to fix this so that we can actually launch arbitrary versions of AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2880) Fix classpath construction for
MRv2
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-2880:
-----------------------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Never mind my comment about the merge. I did it myself.
I committed this to trunk and branch-0.23. Thanks Arun!
> Fix classpath construction for MRv2
> -----------------------------------
>
> Key: MAPREDUCE-2880
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2880
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Luke Lu
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch
>
>
> MRConstants.java refers a hard-coded version of MR AM jar. The build config works around with a symlink. The deployment currently needs symlink workaround as well. We need to fix this so that we can actually launch arbitrary versions of AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2880) Fix classpath construction for
MRv2
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-2880:
-------------------------------------
Status: Patch Available (was: Open)
> Fix classpath construction for MRv2
> -----------------------------------
>
> Key: MAPREDUCE-2880
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2880
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Luke Lu
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch
>
>
> MRConstants.java refers a hard-coded version of MR AM jar. The build config works around with a symlink. The deployment currently needs symlink workaround as well. We need to fix this so that we can actually launch arbitrary versions of AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2880) Fix classpath construction for
MRv2
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-2880:
-------------------------------------
Status: Patch Available (was: Open)
Still need to test... too late, kicking hudson.
> Fix classpath construction for MRv2
> -----------------------------------
>
> Key: MAPREDUCE-2880
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2880
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Luke Lu
> Assignee: Arun C Murthy
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch
>
>
> MRConstants.java refers a hard-coded version of MR AM jar. The build config works around with a symlink. The deployment currently needs symlink workaround as well. We need to fix this so that we can actually launch arbitrary versions of AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2880) Fix classpath construction for
MRv2
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13109621#comment-13109621 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-2880:
----------------------------------------------------
Also, like I said earlier, the new configuration for java.opts can be called admin-opts as they are not overridable by users. But OTOH, the config for user-env is just the default supplied by admins, and can be overridable by the users. We should rename this to something better. Also we should rename admin-shell depending on whether we mean it to be user-overridable or not.
> Fix classpath construction for MRv2
> -----------------------------------
>
> Key: MAPREDUCE-2880
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2880
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Luke Lu
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch
>
>
> MRConstants.java refers a hard-coded version of MR AM jar. The build config works around with a symlink. The deployment currently needs symlink workaround as well. We need to fix this so that we can actually launch arbitrary versions of AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work started] (MAPREDUCE-2880) Fix classpath construction
for MRv2
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on MAPREDUCE-2880 started by Arun C Murthy.
> Fix classpath construction for MRv2
> -----------------------------------
>
> Key: MAPREDUCE-2880
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2880
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Luke Lu
> Assignee: Arun C Murthy
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2880.patch, MAPREDUCE-2880.patch
>
>
> MRConstants.java refers a hard-coded version of MR AM jar. The build config works around with a symlink. The deployment currently needs symlink workaround as well. We need to fix this so that we can actually launch arbitrary versions of AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2880) Fix classpath construction for
MRv2
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13109756#comment-13109756 ]
Hudson commented on MAPREDUCE-2880:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #932 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/932/])
MAPREDUCE-2880. Improved classpath-construction for mapreduce AM and containers. Contributed by Arun C Murthy.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1173783
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DefaultSpeculator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/MRConstants.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/BackupStore.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Constants.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MRConstants.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MROutputFiles.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ResourceMgrDelegate.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
> Fix classpath construction for MRv2
> -----------------------------------
>
> Key: MAPREDUCE-2880
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2880
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Luke Lu
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch
>
>
> MRConstants.java refers a hard-coded version of MR AM jar. The build config works around with a symlink. The deployment currently needs symlink workaround as well. We need to fix this so that we can actually launch arbitrary versions of AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2880) Fix classpath construction for
MRv2
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-2880:
-------------------------------------
Status: Patch Available (was: In Progress)
Trying hudson again, too late for me! :)
> Fix classpath construction for MRv2
> -----------------------------------
>
> Key: MAPREDUCE-2880
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2880
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Luke Lu
> Assignee: Arun C Murthy
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2880.patch, MAPREDUCE-2880.patch
>
>
> MRConstants.java refers a hard-coded version of MR AM jar. The build config works around with a symlink. The deployment currently needs symlink workaround as well. We need to fix this so that we can actually launch arbitrary versions of AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2880) Fix classpath construction for
MRv2
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-2880:
-------------------------------------
Attachment: MAPREDUCE-2880.patch
Latest version of the patch, still testing it... not ready for primetime yet.
> Fix classpath construction for MRv2
> -----------------------------------
>
> Key: MAPREDUCE-2880
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2880
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Luke Lu
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch
>
>
> MRConstants.java refers a hard-coded version of MR AM jar. The build config works around with a symlink. The deployment currently needs symlink workaround as well. We need to fix this so that we can actually launch arbitrary versions of AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2880) Fix classpath construction for
MRv2
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13107407#comment-13107407 ]
Hadoop QA commented on MAPREDUCE-2880:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12494967/MAPREDUCE-2880.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainersMonitor
org.apache.hadoop.yarn.server.nodemanager.containermanager.TestContainerManager
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/777//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/777//console
This message is automatically generated.
> Fix classpath construction for MRv2
> -----------------------------------
>
> Key: MAPREDUCE-2880
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2880
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Luke Lu
> Assignee: Arun C Murthy
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch
>
>
> MRConstants.java refers a hard-coded version of MR AM jar. The build config works around with a symlink. The deployment currently needs symlink workaround as well. We need to fix this so that we can actually launch arbitrary versions of AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2880) Fix classpath construction for
MRv2
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13107462#comment-13107462 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-2880:
----------------------------------------------------
Did a first pass of review. Didn't find it as scary as it threatened to be :)
Mostly very minor comments:
ContainerLaunch:
- {{sanitizeEnv()}}: I was confused about LOGNAME with the usual LOG context. Perhaps put a comment saying it refs to LOGINNAME.
- Rename setEnvIfUnset -> setEnvIfNotset ?
- Shouldn't we set YARN_HOME, HADOOP_HDFS_HOME, HADOOP_COMMON_HOME and JAVA_HOME as part of a NM whitelist? A configuration or better a shell script which we can source into the child shell. The later will help us add items on demand.
Rename ApplicationConstants.APPLICATION_CLASSPATH to {DEFAULT|PREFIX}_APPLICATION_CLASSPATH ?
MRJobConfig: Rename MAPRED_MAP_ADMIN_JAVA_OPTS -> MAPRED_FRAMEWORK_MAP_JAVA_OPTS or something similar? Definitely doesn't look like an admin config to me. Similarly for other params?
MRApps:
- The linkName for all dist-cache files and MR_APP_JAR removed from CLASSPATH works because of the glob pattern?
- {{setMRFrameworkClasspath(Map<String, String>)}} can be private.
- _setEnvFromInputString()_: Refs to "tt" should be fixed.
Holistically looks good, but I'll just look once more afresh tomorrow.
Stating what is already known, we should test this thoroughly :) Mainly the distributed cache related tests. TestMRJobs, which also has a dist-cache test is a good start.
> Fix classpath construction for MRv2
> -----------------------------------
>
> Key: MAPREDUCE-2880
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2880
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Luke Lu
> Assignee: Arun C Murthy
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch
>
>
> MRConstants.java refers a hard-coded version of MR AM jar. The build config works around with a symlink. The deployment currently needs symlink workaround as well. We need to fix this so that we can actually launch arbitrary versions of AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2880) Fix classpath construction for
MRv2
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-2880:
-------------------------------------
Assignee: Arun C Murthy
I'll take this up. We need it to fix this asap, particularly for issues like MAPREDUCE-2801.
> Fix classpath construction for MRv2
> -----------------------------------
>
> Key: MAPREDUCE-2880
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2880
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Luke Lu
> Assignee: Arun C Murthy
> Fix For: 0.23.0
>
>
> MRConstants.java refers a hard-coded version of MR AM jar. The build config works around with a symlink. The deployment currently needs symlink workaround as well. We need to fix this so that we can actually launch arbitrary versions of AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2880) Fix classpath construction for
MRv2
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13109752#comment-13109752 ]
Hudson commented on MAPREDUCE-2880:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #1010 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1010/])
MAPREDUCE-2880. Improved classpath-construction for mapreduce AM and containers. Contributed by Arun C Murthy.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1173783
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DefaultSpeculator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/MRConstants.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/BackupStore.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Constants.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MRConstants.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MROutputFiles.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ResourceMgrDelegate.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
> Fix classpath construction for MRv2
> -----------------------------------
>
> Key: MAPREDUCE-2880
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2880
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Luke Lu
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch
>
>
> MRConstants.java refers a hard-coded version of MR AM jar. The build config works around with a symlink. The deployment currently needs symlink workaround as well. We need to fix this so that we can actually launch arbitrary versions of AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2880) Fix classpath construction for
MRv2
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13112560#comment-13112560 ]
Hudson commented on MAPREDUCE-2880:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #838 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/838/])
MAPREDUCE-2880. Improved classpath-construction for mapreduce AM and containers. Contributed by Arun C Murthy.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1173783
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DefaultSpeculator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/MRConstants.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/BackupStore.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Constants.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MRConstants.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MROutputFiles.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ResourceMgrDelegate.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
> Fix classpath construction for MRv2
> -----------------------------------
>
> Key: MAPREDUCE-2880
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2880
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Luke Lu
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch
>
>
> MRConstants.java refers a hard-coded version of MR AM jar. The build config works around with a symlink. The deployment currently needs symlink workaround as well. We need to fix this so that we can actually launch arbitrary versions of AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2880) Fix classpath construction for
MRv2
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-2880:
-------------------------------------
Attachment: MAPREDUCE-2880.patch
Simplified the heck out of the CLASSPATH construction by relying on HADOOP_COMMON_HOME, HADOOP_HDFS_HOME and HADOOP_YARN_HOME on the NodeManager.
These are needed for all apps anyway, plus overridable since DC artifacts are ahead in the classpath.
Still need to test further, TestMRApp works!
> Fix classpath construction for MRv2
> -----------------------------------
>
> Key: MAPREDUCE-2880
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2880
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Luke Lu
> Assignee: Arun C Murthy
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2880.patch
>
>
> MRConstants.java refers a hard-coded version of MR AM jar. The build config works around with a symlink. The deployment currently needs symlink workaround as well. We need to fix this so that we can actually launch arbitrary versions of AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2880) Fix classpath construction for
MRv2
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13109773#comment-13109773 ]
Hudson commented on MAPREDUCE-2880:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #950 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/950/])
MAPREDUCE-2880. Improved classpath-construction for mapreduce AM and containers. Contributed by Arun C Murthy.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1173783
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DefaultSpeculator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/MRConstants.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/BackupStore.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Constants.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MRConstants.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MROutputFiles.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ResourceMgrDelegate.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
> Fix classpath construction for MRv2
> -----------------------------------
>
> Key: MAPREDUCE-2880
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2880
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Luke Lu
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch
>
>
> MRConstants.java refers a hard-coded version of MR AM jar. The build config works around with a symlink. The deployment currently needs symlink workaround as well. We need to fix this so that we can actually launch arbitrary versions of AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2880) Fix classpath construction for
MRv2
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13112521#comment-13112521 ]
Hudson commented on MAPREDUCE-2880:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk #808 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/808/])
MAPREDUCE-2880. Improved classpath-construction for mapreduce AM and containers. Contributed by Arun C Murthy.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1173783
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DefaultSpeculator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/MRConstants.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/BackupStore.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Constants.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MRConstants.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MROutputFiles.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ResourceMgrDelegate.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
> Fix classpath construction for MRv2
> -----------------------------------
>
> Key: MAPREDUCE-2880
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2880
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Luke Lu
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch
>
>
> MRConstants.java refers a hard-coded version of MR AM jar. The build config works around with a symlink. The deployment currently needs symlink workaround as well. We need to fix this so that we can actually launch arbitrary versions of AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2880) Fix classpath construction for
MRv2
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13112583#comment-13112583 ]
Hudson commented on MAPREDUCE-2880:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #17 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/17/])
MAPREDUCE-2880. svn merge -c r1173783 --ignore-ancestry ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1173792
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DefaultSpeculator.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/MRConstants.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/BackupStore.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Constants.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MRConstants.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MROutputFiles.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ResourceMgrDelegate.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
> Fix classpath construction for MRv2
> -----------------------------------
>
> Key: MAPREDUCE-2880
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2880
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Luke Lu
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch
>
>
> MRConstants.java refers a hard-coded version of MR AM jar. The build config works around with a symlink. The deployment currently needs symlink workaround as well. We need to fix this so that we can actually launch arbitrary versions of AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2880) Fix classpath construction for
MRv2
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-2880:
-------------------------------------
Attachment: MAPREDUCE-2880.patch
Ok, this seems code-complete... includes white-list of env for containers, admin java_opts/env and user-defined java_home.
> Fix classpath construction for MRv2
> -----------------------------------
>
> Key: MAPREDUCE-2880
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2880
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Luke Lu
> Assignee: Arun C Murthy
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch
>
>
> MRConstants.java refers a hard-coded version of MR AM jar. The build config works around with a symlink. The deployment currently needs symlink workaround as well. We need to fix this so that we can actually launch arbitrary versions of AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2880) Fix classpath construction for
MRv2
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13109714#comment-13109714 ]
Arun C Murthy commented on MAPREDUCE-2880:
------------------------------------------
Thanks to Vinod for all his help in reviewing and debugging this!
> Fix classpath construction for MRv2
> -----------------------------------
>
> Key: MAPREDUCE-2880
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2880
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Luke Lu
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch
>
>
> MRConstants.java refers a hard-coded version of MR AM jar. The build config works around with a symlink. The deployment currently needs symlink workaround as well. We need to fix this so that we can actually launch arbitrary versions of AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2880) Fix classpath construction for
MRv2
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105949#comment-13105949 ]
Hadoop QA commented on MAPREDUCE-2880:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12494770/MAPREDUCE-2880.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/758//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/758//console
This message is automatically generated.
> Fix classpath construction for MRv2
> -----------------------------------
>
> Key: MAPREDUCE-2880
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2880
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Luke Lu
> Assignee: Arun C Murthy
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2880.patch
>
>
> MRConstants.java refers a hard-coded version of MR AM jar. The build config works around with a symlink. The deployment currently needs symlink workaround as well. We need to fix this so that we can actually launch arbitrary versions of AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2880) Fix classpath construction for
MRv2
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13109737#comment-13109737 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-2880:
----------------------------------------------------
Oh, also when you merge into branch-0.23, you will need to update MiniMRYarnCluster.HADOOP_MAPREDUCE_CLIENT_APP_JAR_NAME (I just hate this).
> Fix classpath construction for MRv2
> -----------------------------------
>
> Key: MAPREDUCE-2880
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2880
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Luke Lu
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch
>
>
> MRConstants.java refers a hard-coded version of MR AM jar. The build config works around with a symlink. The deployment currently needs symlink workaround as well. We need to fix this so that we can actually launch arbitrary versions of AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2880) Fix classpath construction for
MRv2
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-2880:
-------------------------------------
Priority: Blocker (was: Major)
> Fix classpath construction for MRv2
> -----------------------------------
>
> Key: MAPREDUCE-2880
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2880
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Luke Lu
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch
>
>
> MRConstants.java refers a hard-coded version of MR AM jar. The build config works around with a symlink. The deployment currently needs symlink workaround as well. We need to fix this so that we can actually launch arbitrary versions of AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2880) Fix classpath construction for
MRv2
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-2880:
-------------------------------------
Attachment: MAPREDUCE-2880.patch
Works! And tested on a secure cluster - thanks to Vinod for his help in debugging.
Still need to fix unit tests, but it's nice to see very simple classpaths etc. for containers. Also adds admin java-opts, admin env etc.
> Fix classpath construction for MRv2
> -----------------------------------
>
> Key: MAPREDUCE-2880
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2880
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Luke Lu
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch
>
>
> MRConstants.java refers a hard-coded version of MR AM jar. The build config works around with a symlink. The deployment currently needs symlink workaround as well. We need to fix this so that we can actually launch arbitrary versions of AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2880) Fix classpath construction for
MRv2
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-2880:
-------------------------------------
Status: Open (was: Patch Available)
> Fix classpath construction for MRv2
> -----------------------------------
>
> Key: MAPREDUCE-2880
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2880
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Luke Lu
> Assignee: Arun C Murthy
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch
>
>
> MRConstants.java refers a hard-coded version of MR AM jar. The build config works around with a symlink. The deployment currently needs symlink workaround as well. We need to fix this so that we can actually launch arbitrary versions of AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2880) Fix classpath construction for
MRv2
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13107791#comment-13107791 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-2880:
----------------------------------------------------
Arun, can you also see if you can incorporate MAPREDUCE-2998, please see the last few comments there. I think we can knock of all the synchronization related stuff too, as getting the initial classpath is far cheaper now.
> Fix classpath construction for MRv2
> -----------------------------------
>
> Key: MAPREDUCE-2880
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2880
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Luke Lu
> Assignee: Arun C Murthy
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2880.patch, MAPREDUCE-2880.patch, MAPREDUCE-2880.patch
>
>
> MRConstants.java refers a hard-coded version of MR AM jar. The build config works around with a symlink. The deployment currently needs symlink workaround as well. We need to fix this so that we can actually launch arbitrary versions of AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira