You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Robert Joseph Evans (JIRA)" <ji...@apache.org> on 2012/10/22 16:58:12 UTC
[jira] [Created] (MAPREDUCE-4740) only .jars can be added to the
Distributed Cache classpath
Robert Joseph Evans created MAPREDUCE-4740:
----------------------------------------------
Summary: only .jars can be added to the Distributed Cache classpath
Key: MAPREDUCE-4740
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4740
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.0.2-alpha, 0.23.3
Reporter: Robert Joseph Evans
Assignee: Robert Joseph Evans
Priority: Blocker
Koji tracked down this one.
{noformat}
$ cat mycat.sh
#!/bin/sh
cat >& /dev/null
$JAVA_HOME/bin/jinfo $PPID | grep java.class.path
export | grep CLASSPATH
ls -l
$ myfile=/user/me/myclasses.zip; yarn jar
hadoop-streaming.jar
-Dmapreduce.job.cache.archives=hdfs:///${myfile}
-Dmapreduce.job.classpath.archives=${myfile} -input in.txt -output out -reducer NONE -mapper mycat.sh -file mycat.sh
{noformat}
So, cachearchive like class.zip or class.tar.gz were never set as part of the
classpath even though they were properly set by DistributedCache.addArchiveToClassPath.
It looks like we are parsing the classpath out of the configs, but then throwing that away. It looks simple enough to add them in the correct place.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4740) only .jars can be added to the
Distributed Cache classpath
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13481687#comment-13481687 ]
Hadoop QA commented on MAPREDUCE-4740:
--------------------------------------
{color:green}+1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12550335/MR-4740.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2958//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2958//console
This message is automatically generated.
> only .jars can be added to the Distributed Cache classpath
> ----------------------------------------------------------
>
> Key: MAPREDUCE-4740
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4740
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.3, 2.0.2-alpha
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Attachments: MR-4740.txt, MR-4740.txt, MR-4740.txt, MR-4740.txt
>
>
> Koji tracked down this one.
> {noformat}
> $ cat mycat.sh
> #!/bin/sh
> cat >& /dev/null
> $JAVA_HOME/bin/jinfo $PPID | grep java.class.path
> export | grep CLASSPATH
> ls -l
> $ myfile=/user/me/myclasses.zip; yarn jar
> hadoop-streaming.jar
> -Dmapreduce.job.cache.archives=hdfs:///${myfile}
> -Dmapreduce.job.classpath.archives=${myfile} -input in.txt -output out -reducer NONE -mapper mycat.sh -file mycat.sh
> {noformat}
> So, cachearchive like class.zip or class.tar.gz were never set as part of the
> classpath even though they were properly set by DistributedCache.addArchiveToClassPath.
> It looks like we are parsing the classpath out of the configs, but then throwing that away. It looks simple enough to add them in the correct place.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4740) only .jars can be added to the
Distributed Cache classpath
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-4740:
-------------------------------------------
Attachment: MR-4740.txt
Attaching patch that should fix the issue. It will not work if someone adds an archive with a # in it, and also adds it to the classpath. I do not believe that this works in 1.0 either but i will test it on 1.0 to be sure.
> only .jars can be added to the Distributed Cache classpath
> ----------------------------------------------------------
>
> Key: MAPREDUCE-4740
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4740
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.3, 2.0.2-alpha
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Attachments: MR-4740.txt
>
>
> Koji tracked down this one.
> {noformat}
> $ cat mycat.sh
> #!/bin/sh
> cat >& /dev/null
> $JAVA_HOME/bin/jinfo $PPID | grep java.class.path
> export | grep CLASSPATH
> ls -l
> $ myfile=/user/me/myclasses.zip; yarn jar
> hadoop-streaming.jar
> -Dmapreduce.job.cache.archives=hdfs:///${myfile}
> -Dmapreduce.job.classpath.archives=${myfile} -input in.txt -output out -reducer NONE -mapper mycat.sh -file mycat.sh
> {noformat}
> So, cachearchive like class.zip or class.tar.gz were never set as part of the
> classpath even though they were properly set by DistributedCache.addArchiveToClassPath.
> It looks like we are parsing the classpath out of the configs, but then throwing that away. It looks simple enough to add them in the correct place.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4740) only .jars can be added to the
Distributed Cache classpath
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-4740:
-------------------------------------------
Status: Patch Available (was: Open)
> only .jars can be added to the Distributed Cache classpath
> ----------------------------------------------------------
>
> Key: MAPREDUCE-4740
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4740
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 2.0.2-alpha, 0.23.3
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Attachments: MR-4740.txt
>
>
> Koji tracked down this one.
> {noformat}
> $ cat mycat.sh
> #!/bin/sh
> cat >& /dev/null
> $JAVA_HOME/bin/jinfo $PPID | grep java.class.path
> export | grep CLASSPATH
> ls -l
> $ myfile=/user/me/myclasses.zip; yarn jar
> hadoop-streaming.jar
> -Dmapreduce.job.cache.archives=hdfs:///${myfile}
> -Dmapreduce.job.classpath.archives=${myfile} -input in.txt -output out -reducer NONE -mapper mycat.sh -file mycat.sh
> {noformat}
> So, cachearchive like class.zip or class.tar.gz were never set as part of the
> classpath even though they were properly set by DistributedCache.addArchiveToClassPath.
> It looks like we are parsing the classpath out of the configs, but then throwing that away. It looks simple enough to add them in the correct place.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4740) only .jars can be added to the
Distributed Cache classpath
Posted by "Jason Lowe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jason Lowe updated MAPREDUCE-4740:
----------------------------------
Resolution: Fixed
Fix Version/s: 0.23.5
2.0.3-alpha
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks, Bobby. I committed this to trunk, branch-2, and branch-0.23.
> only .jars can be added to the Distributed Cache classpath
> ----------------------------------------------------------
>
> Key: MAPREDUCE-4740
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4740
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.3, 2.0.2-alpha
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 2.0.3-alpha, 0.23.5
>
> Attachments: MR-4740.txt, MR-4740.txt, MR-4740.txt, MR-4740.txt
>
>
> Koji tracked down this one.
> {noformat}
> $ cat mycat.sh
> #!/bin/sh
> cat >& /dev/null
> $JAVA_HOME/bin/jinfo $PPID | grep java.class.path
> export | grep CLASSPATH
> ls -l
> $ myfile=/user/me/myclasses.zip; yarn jar
> hadoop-streaming.jar
> -Dmapreduce.job.cache.archives=hdfs:///${myfile}
> -Dmapreduce.job.classpath.archives=${myfile} -input in.txt -output out -reducer NONE -mapper mycat.sh -file mycat.sh
> {noformat}
> So, cachearchive like class.zip or class.tar.gz were never set as part of the
> classpath even though they were properly set by DistributedCache.addArchiveToClassPath.
> It looks like we are parsing the classpath out of the configs, but then throwing that away. It looks simple enough to add them in the correct place.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4740) only .jars can be added to the
Distributed Cache classpath
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13482290#comment-13482290 ]
Hudson commented on MAPREDUCE-4740:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #413 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/413/])
svn merge -c 1401054 FIXES: MAPREDUCE-4740. only .jars can be added to the Distributed Cache classpath. Contributed by Robert Joseph Evans (Revision 1401059)
Result = SUCCESS
jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1401059
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
> only .jars can be added to the Distributed Cache classpath
> ----------------------------------------------------------
>
> Key: MAPREDUCE-4740
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4740
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.3, 2.0.2-alpha
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 2.0.3-alpha, 0.23.5
>
> Attachments: MR-4740.txt, MR-4740.txt, MR-4740.txt, MR-4740.txt
>
>
> Koji tracked down this one.
> {noformat}
> $ cat mycat.sh
> #!/bin/sh
> cat >& /dev/null
> $JAVA_HOME/bin/jinfo $PPID | grep java.class.path
> export | grep CLASSPATH
> ls -l
> $ myfile=/user/me/myclasses.zip; yarn jar
> hadoop-streaming.jar
> -Dmapreduce.job.cache.archives=hdfs:///${myfile}
> -Dmapreduce.job.classpath.archives=${myfile} -input in.txt -output out -reducer NONE -mapper mycat.sh -file mycat.sh
> {noformat}
> So, cachearchive like class.zip or class.tar.gz were never set as part of the
> classpath even though they were properly set by DistributedCache.addArchiveToClassPath.
> It looks like we are parsing the classpath out of the configs, but then throwing that away. It looks simple enough to add them in the correct place.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4740) only .jars can be added to the
Distributed Cache classpath
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13481436#comment-13481436 ]
Hadoop QA commented on MAPREDUCE-4740:
--------------------------------------
{color:green}+1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12550298/MR-4740.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2955//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2955//console
This message is automatically generated.
> only .jars can be added to the Distributed Cache classpath
> ----------------------------------------------------------
>
> Key: MAPREDUCE-4740
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4740
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.3, 2.0.2-alpha
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Attachments: MR-4740.txt
>
>
> Koji tracked down this one.
> {noformat}
> $ cat mycat.sh
> #!/bin/sh
> cat >& /dev/null
> $JAVA_HOME/bin/jinfo $PPID | grep java.class.path
> export | grep CLASSPATH
> ls -l
> $ myfile=/user/me/myclasses.zip; yarn jar
> hadoop-streaming.jar
> -Dmapreduce.job.cache.archives=hdfs:///${myfile}
> -Dmapreduce.job.classpath.archives=${myfile} -input in.txt -output out -reducer NONE -mapper mycat.sh -file mycat.sh
> {noformat}
> So, cachearchive like class.zip or class.tar.gz were never set as part of the
> classpath even though they were properly set by DistributedCache.addArchiveToClassPath.
> It looks like we are parsing the classpath out of the configs, but then throwing that away. It looks simple enough to add them in the correct place.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4740) only .jars can be added to the
Distributed Cache classpath
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-4740:
-------------------------------------------
Attachment: MR-4740.txt
Updated version that allows for a cache archive to have a symlink name, and it to also be added to the class path. This does not update the API to set this that will still not work if there is a #symlink in the archive being added to the classpath, but internally if the confs are set manually it will work, and the unit test verifies it.
> only .jars can be added to the Distributed Cache classpath
> ----------------------------------------------------------
>
> Key: MAPREDUCE-4740
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4740
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.3, 2.0.2-alpha
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Attachments: MR-4740.txt, MR-4740.txt
>
>
> Koji tracked down this one.
> {noformat}
> $ cat mycat.sh
> #!/bin/sh
> cat >& /dev/null
> $JAVA_HOME/bin/jinfo $PPID | grep java.class.path
> export | grep CLASSPATH
> ls -l
> $ myfile=/user/me/myclasses.zip; yarn jar
> hadoop-streaming.jar
> -Dmapreduce.job.cache.archives=hdfs:///${myfile}
> -Dmapreduce.job.classpath.archives=${myfile} -input in.txt -output out -reducer NONE -mapper mycat.sh -file mycat.sh
> {noformat}
> So, cachearchive like class.zip or class.tar.gz were never set as part of the
> classpath even though they were properly set by DistributedCache.addArchiveToClassPath.
> It looks like we are parsing the classpath out of the configs, but then throwing that away. It looks simple enough to add them in the correct place.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4740) only .jars can be added to the
Distributed Cache classpath
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13481438#comment-13481438 ]
Robert Joseph Evans commented on MAPREDUCE-4740:
------------------------------------------------
OK so the # does work on 1.0.2 if you set the confs manually. I will update the patch accordingly.
> only .jars can be added to the Distributed Cache classpath
> ----------------------------------------------------------
>
> Key: MAPREDUCE-4740
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4740
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.3, 2.0.2-alpha
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Attachments: MR-4740.txt
>
>
> Koji tracked down this one.
> {noformat}
> $ cat mycat.sh
> #!/bin/sh
> cat >& /dev/null
> $JAVA_HOME/bin/jinfo $PPID | grep java.class.path
> export | grep CLASSPATH
> ls -l
> $ myfile=/user/me/myclasses.zip; yarn jar
> hadoop-streaming.jar
> -Dmapreduce.job.cache.archives=hdfs:///${myfile}
> -Dmapreduce.job.classpath.archives=${myfile} -input in.txt -output out -reducer NONE -mapper mycat.sh -file mycat.sh
> {noformat}
> So, cachearchive like class.zip or class.tar.gz were never set as part of the
> classpath even though they were properly set by DistributedCache.addArchiveToClassPath.
> It looks like we are parsing the classpath out of the configs, but then throwing that away. It looks simple enough to add them in the correct place.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4740) only .jars can be added to the
Distributed Cache classpath
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13482250#comment-13482250 ]
Hudson commented on MAPREDUCE-4740:
-----------------------------------
Integrated in Hadoop-Yarn-trunk #12 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/12/])
MAPREDUCE-4740. only .jars can be added to the Distributed Cache classpath. Contributed by Robert Joseph Evans (Revision 1401054)
Result = FAILURE
jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1401054
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
> only .jars can be added to the Distributed Cache classpath
> ----------------------------------------------------------
>
> Key: MAPREDUCE-4740
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4740
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.3, 2.0.2-alpha
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 2.0.3-alpha, 0.23.5
>
> Attachments: MR-4740.txt, MR-4740.txt, MR-4740.txt, MR-4740.txt
>
>
> Koji tracked down this one.
> {noformat}
> $ cat mycat.sh
> #!/bin/sh
> cat >& /dev/null
> $JAVA_HOME/bin/jinfo $PPID | grep java.class.path
> export | grep CLASSPATH
> ls -l
> $ myfile=/user/me/myclasses.zip; yarn jar
> hadoop-streaming.jar
> -Dmapreduce.job.cache.archives=hdfs:///${myfile}
> -Dmapreduce.job.classpath.archives=${myfile} -input in.txt -output out -reducer NONE -mapper mycat.sh -file mycat.sh
> {noformat}
> So, cachearchive like class.zip or class.tar.gz were never set as part of the
> classpath even though they were properly set by DistributedCache.addArchiveToClassPath.
> It looks like we are parsing the classpath out of the configs, but then throwing that away. It looks simple enough to add them in the correct place.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4740) only .jars can be added to the
Distributed Cache classpath
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-4740:
-------------------------------------------
Attachment: MR-4740.txt
I missed that files can be named .JAR too. So I made the test case insensitive.
> only .jars can be added to the Distributed Cache classpath
> ----------------------------------------------------------
>
> Key: MAPREDUCE-4740
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4740
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.3, 2.0.2-alpha
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Attachments: MR-4740.txt, MR-4740.txt, MR-4740.txt, MR-4740.txt
>
>
> Koji tracked down this one.
> {noformat}
> $ cat mycat.sh
> #!/bin/sh
> cat >& /dev/null
> $JAVA_HOME/bin/jinfo $PPID | grep java.class.path
> export | grep CLASSPATH
> ls -l
> $ myfile=/user/me/myclasses.zip; yarn jar
> hadoop-streaming.jar
> -Dmapreduce.job.cache.archives=hdfs:///${myfile}
> -Dmapreduce.job.classpath.archives=${myfile} -input in.txt -output out -reducer NONE -mapper mycat.sh -file mycat.sh
> {noformat}
> So, cachearchive like class.zip or class.tar.gz were never set as part of the
> classpath even though they were properly set by DistributedCache.addArchiveToClassPath.
> It looks like we are parsing the classpath out of the configs, but then throwing that away. It looks simple enough to add them in the correct place.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4740) only .jars can be added to the
Distributed Cache classpath
Posted by "Jason Lowe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13481710#comment-13481710 ]
Jason Lowe commented on MAPREDUCE-4740:
---------------------------------------
+1, lgtm. I'll commit this shortly.
> only .jars can be added to the Distributed Cache classpath
> ----------------------------------------------------------
>
> Key: MAPREDUCE-4740
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4740
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.3, 2.0.2-alpha
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Attachments: MR-4740.txt, MR-4740.txt, MR-4740.txt, MR-4740.txt
>
>
> Koji tracked down this one.
> {noformat}
> $ cat mycat.sh
> #!/bin/sh
> cat >& /dev/null
> $JAVA_HOME/bin/jinfo $PPID | grep java.class.path
> export | grep CLASSPATH
> ls -l
> $ myfile=/user/me/myclasses.zip; yarn jar
> hadoop-streaming.jar
> -Dmapreduce.job.cache.archives=hdfs:///${myfile}
> -Dmapreduce.job.classpath.archives=${myfile} -input in.txt -output out -reducer NONE -mapper mycat.sh -file mycat.sh
> {noformat}
> So, cachearchive like class.zip or class.tar.gz were never set as part of the
> classpath even though they were properly set by DistributedCache.addArchiveToClassPath.
> It looks like we are parsing the classpath out of the configs, but then throwing that away. It looks simple enough to add them in the correct place.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4740) only .jars can be added to the
Distributed Cache classpath
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13481732#comment-13481732 ]
Hudson commented on MAPREDUCE-4740:
-----------------------------------
Integrated in Hadoop-trunk-Commit #2908 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/2908/])
MAPREDUCE-4740. only .jars can be added to the Distributed Cache classpath. Contributed by Robert Joseph Evans (Revision 1401054)
Result = SUCCESS
jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1401054
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
> only .jars can be added to the Distributed Cache classpath
> ----------------------------------------------------------
>
> Key: MAPREDUCE-4740
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4740
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.3, 2.0.2-alpha
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Attachments: MR-4740.txt, MR-4740.txt, MR-4740.txt, MR-4740.txt
>
>
> Koji tracked down this one.
> {noformat}
> $ cat mycat.sh
> #!/bin/sh
> cat >& /dev/null
> $JAVA_HOME/bin/jinfo $PPID | grep java.class.path
> export | grep CLASSPATH
> ls -l
> $ myfile=/user/me/myclasses.zip; yarn jar
> hadoop-streaming.jar
> -Dmapreduce.job.cache.archives=hdfs:///${myfile}
> -Dmapreduce.job.classpath.archives=${myfile} -input in.txt -output out -reducer NONE -mapper mycat.sh -file mycat.sh
> {noformat}
> So, cachearchive like class.zip or class.tar.gz were never set as part of the
> classpath even though they were properly set by DistributedCache.addArchiveToClassPath.
> It looks like we are parsing the classpath out of the configs, but then throwing that away. It looks simple enough to add them in the correct place.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4740) only .jars can be added to the
Distributed Cache classpath
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13482335#comment-13482335 ]
Hudson commented on MAPREDUCE-4740:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #1234 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1234/])
MAPREDUCE-4740. only .jars can be added to the Distributed Cache classpath. Contributed by Robert Joseph Evans (Revision 1401054)
Result = SUCCESS
jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1401054
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
> only .jars can be added to the Distributed Cache classpath
> ----------------------------------------------------------
>
> Key: MAPREDUCE-4740
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4740
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.3, 2.0.2-alpha
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 2.0.3-alpha, 0.23.5
>
> Attachments: MR-4740.txt, MR-4740.txt, MR-4740.txt, MR-4740.txt
>
>
> Koji tracked down this one.
> {noformat}
> $ cat mycat.sh
> #!/bin/sh
> cat >& /dev/null
> $JAVA_HOME/bin/jinfo $PPID | grep java.class.path
> export | grep CLASSPATH
> ls -l
> $ myfile=/user/me/myclasses.zip; yarn jar
> hadoop-streaming.jar
> -Dmapreduce.job.cache.archives=hdfs:///${myfile}
> -Dmapreduce.job.classpath.archives=${myfile} -input in.txt -output out -reducer NONE -mapper mycat.sh -file mycat.sh
> {noformat}
> So, cachearchive like class.zip or class.tar.gz were never set as part of the
> classpath even though they were properly set by DistributedCache.addArchiveToClassPath.
> It looks like we are parsing the classpath out of the configs, but then throwing that away. It looks simple enough to add them in the correct place.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4740) only .jars can be added to the
Distributed Cache classpath
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-4740:
-------------------------------------------
Attachment: MR-4740.txt
Sorry looks like 0.23 and trunk are different enough in the MRApps test file that the patch did not apply cleanly. This new version is based off of trunk.
> only .jars can be added to the Distributed Cache classpath
> ----------------------------------------------------------
>
> Key: MAPREDUCE-4740
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4740
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.3, 2.0.2-alpha
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Attachments: MR-4740.txt, MR-4740.txt, MR-4740.txt
>
>
> Koji tracked down this one.
> {noformat}
> $ cat mycat.sh
> #!/bin/sh
> cat >& /dev/null
> $JAVA_HOME/bin/jinfo $PPID | grep java.class.path
> export | grep CLASSPATH
> ls -l
> $ myfile=/user/me/myclasses.zip; yarn jar
> hadoop-streaming.jar
> -Dmapreduce.job.cache.archives=hdfs:///${myfile}
> -Dmapreduce.job.classpath.archives=${myfile} -input in.txt -output out -reducer NONE -mapper mycat.sh -file mycat.sh
> {noformat}
> So, cachearchive like class.zip or class.tar.gz were never set as part of the
> classpath even though they were properly set by DistributedCache.addArchiveToClassPath.
> It looks like we are parsing the classpath out of the configs, but then throwing that away. It looks simple enough to add them in the correct place.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4740) only .jars can be added to the
Distributed Cache classpath
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13482300#comment-13482300 ]
Hudson commented on MAPREDUCE-4740:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk #1204 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1204/])
MAPREDUCE-4740. only .jars can be added to the Distributed Cache classpath. Contributed by Robert Joseph Evans (Revision 1401054)
Result = SUCCESS
jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1401054
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
> only .jars can be added to the Distributed Cache classpath
> ----------------------------------------------------------
>
> Key: MAPREDUCE-4740
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4740
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.3, 2.0.2-alpha
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 2.0.3-alpha, 0.23.5
>
> Attachments: MR-4740.txt, MR-4740.txt, MR-4740.txt, MR-4740.txt
>
>
> Koji tracked down this one.
> {noformat}
> $ cat mycat.sh
> #!/bin/sh
> cat >& /dev/null
> $JAVA_HOME/bin/jinfo $PPID | grep java.class.path
> export | grep CLASSPATH
> ls -l
> $ myfile=/user/me/myclasses.zip; yarn jar
> hadoop-streaming.jar
> -Dmapreduce.job.cache.archives=hdfs:///${myfile}
> -Dmapreduce.job.classpath.archives=${myfile} -input in.txt -output out -reducer NONE -mapper mycat.sh -file mycat.sh
> {noformat}
> So, cachearchive like class.zip or class.tar.gz were never set as part of the
> classpath even though they were properly set by DistributedCache.addArchiveToClassPath.
> It looks like we are parsing the classpath out of the configs, but then throwing that away. It looks simple enough to add them in the correct place.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4740) only .jars can be added to the
Distributed Cache classpath
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13481543#comment-13481543 ]
Hadoop QA commented on MAPREDUCE-4740:
--------------------------------------
{color:green}+1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12550314/MR-4740.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2957//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2957//console
This message is automatically generated.
> only .jars can be added to the Distributed Cache classpath
> ----------------------------------------------------------
>
> Key: MAPREDUCE-4740
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4740
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.3, 2.0.2-alpha
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Attachments: MR-4740.txt, MR-4740.txt, MR-4740.txt
>
>
> Koji tracked down this one.
> {noformat}
> $ cat mycat.sh
> #!/bin/sh
> cat >& /dev/null
> $JAVA_HOME/bin/jinfo $PPID | grep java.class.path
> export | grep CLASSPATH
> ls -l
> $ myfile=/user/me/myclasses.zip; yarn jar
> hadoop-streaming.jar
> -Dmapreduce.job.cache.archives=hdfs:///${myfile}
> -Dmapreduce.job.classpath.archives=${myfile} -input in.txt -output out -reducer NONE -mapper mycat.sh -file mycat.sh
> {noformat}
> So, cachearchive like class.zip or class.tar.gz were never set as part of the
> classpath even though they were properly set by DistributedCache.addArchiveToClassPath.
> It looks like we are parsing the classpath out of the configs, but then throwing that away. It looks simple enough to add them in the correct place.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4740) only .jars can be added to the
Distributed Cache classpath
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13481476#comment-13481476 ]
Hadoop QA commented on MAPREDUCE-4740:
--------------------------------------
{color:red}-1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12550309/MR-4740.txt
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2956//console
This message is automatically generated.
> only .jars can be added to the Distributed Cache classpath
> ----------------------------------------------------------
>
> Key: MAPREDUCE-4740
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4740
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.3, 2.0.2-alpha
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Attachments: MR-4740.txt, MR-4740.txt
>
>
> Koji tracked down this one.
> {noformat}
> $ cat mycat.sh
> #!/bin/sh
> cat >& /dev/null
> $JAVA_HOME/bin/jinfo $PPID | grep java.class.path
> export | grep CLASSPATH
> ls -l
> $ myfile=/user/me/myclasses.zip; yarn jar
> hadoop-streaming.jar
> -Dmapreduce.job.cache.archives=hdfs:///${myfile}
> -Dmapreduce.job.classpath.archives=${myfile} -input in.txt -output out -reducer NONE -mapper mycat.sh -file mycat.sh
> {noformat}
> So, cachearchive like class.zip or class.tar.gz were never set as part of the
> classpath even though they were properly set by DistributedCache.addArchiveToClassPath.
> It looks like we are parsing the classpath out of the configs, but then throwing that away. It looks simple enough to add them in the correct place.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira