You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "John George (Created) (JIRA)" <ji...@apache.org> on 2012/01/19 16:02:40 UTC
[jira] [Created] (MAPREDUCE-3696) MR job via oozie does not work on
hadoop 23
MR job via oozie does not work on hadoop 23
-------------------------------------------
Key: MAPREDUCE-3696
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3696
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 0.23.1
Reporter: John George
Assignee: John George
Priority: Critical
NM throws an error on submitting an MR job via oozie on the latest Hadoop 23.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3696) MR job via oozie does not work
on hadoop 23
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13198172#comment-13198172 ]
Hudson commented on MAPREDUCE-3696:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #1651 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1651/])
MAPREDUCE-3696. MR job via oozie does not work on hadoop 23. (John George via mahadev)
mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1239310
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
> MR job via oozie does not work on hadoop 23
> -------------------------------------------
>
> Key: MAPREDUCE-3696
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3696
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: John George
> Assignee: John George
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MR-3696.patch, MR-3696.patch, MR-3696.patch
>
>
> NM throws an error on submitting an MR job via oozie on the latest Hadoop 23.
> *Courtesy: Mona Chitnis (ooize)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3696) MR job via oozie does not work
on hadoop 23
Posted by "John George (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189144#comment-13189144 ]
John George commented on MAPREDUCE-3696:
----------------------------------------
Hadoop 23 installation on a <secure> cluster and on submitting an MR job via Oozie, we see the
following error in the NM container logs
FATAL [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
java.lang.NoSuchMethodError:
org.codehaus.jackson.JsonFactory.enable(Lorg/codehaus/jackson/JsonParser$Feature;)Lorg/codehaus/jackson/JsonFactory;
at org.apache.avro.Schema.<clinit>(Schema.java:82)
at org.apache.hadoop.mapreduce.jobhistory.AMStarted.<clinit>(AMStarted.java:9)
at org.apache.hadoop.mapreduce.jobhistory.AMStartedEvent.<init>(AMStartedEvent.java:36)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.start(MRAppMaster.java:783)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:960)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:956)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:943)
JsonFactory.class is packaged in jackson-core-asl.jar, and versions 1.2 and above contain the aforementioned enable()
method. We are packaging Oozie with jar version jackson-core-asl-1.7.1.jar and jackson-mapper-asl-1.7.1.jar. Is there
some linking taking place to a stale version 1.0.1 of the jackson jar?
An older JIRA HADOOP-7606, mentions explicit maven dependency for this version in the pom.xml. I tried including a
similar dependency explicitly for Oozie's POM while building, but no difference.
> MR job via oozie does not work on hadoop 23
> -------------------------------------------
>
> Key: MAPREDUCE-3696
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3696
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.23.1
> Reporter: John George
> Assignee: John George
> Priority: Critical
>
> NM throws an error on submitting an MR job via oozie on the latest Hadoop 23.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3696) MR job via oozie does not work
on hadoop 23
Posted by "Mahadev konar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193210#comment-13193210 ]
Mahadev konar commented on MAPREDUCE-3696:
------------------------------------------
@John,
Just to confirm, the property is backwards compatible with 1.0 right?
> MR job via oozie does not work on hadoop 23
> -------------------------------------------
>
> Key: MAPREDUCE-3696
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3696
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: John George
> Assignee: John George
> Priority: Blocker
> Attachments: MR-3696.patch
>
>
> NM throws an error on submitting an MR job via oozie on the latest Hadoop 23.
> *Courtesy: Mona Chitnis (ooize)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3696) MR job via oozie does not work
on hadoop 23
Posted by "John George (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13190164#comment-13190164 ]
John George commented on MAPREDUCE-3696:
----------------------------------------
@Mahadev -I see that the launch container script has the same order in classpath when I run a wordcount using hadoop script.
> MR job via oozie does not work on hadoop 23
> -------------------------------------------
>
> Key: MAPREDUCE-3696
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3696
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.23.1
> Reporter: John George
> Assignee: John George
> Priority: Critical
>
> NM throws an error on submitting an MR job via oozie on the latest Hadoop 23.
> *Courtesy: Mona Chitnis (ooize)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3696) MR job via oozie does not work
on hadoop 23
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13198173#comment-13198173 ]
Hudson commented on MAPREDUCE-3696:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Commit #477 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/477/])
MAPREDUCE-3696. MR job via oozie does not work on hadoop 23. (John George via mahadev) - Merging r1239310 from trunk.
mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1239312
Files :
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
> MR job via oozie does not work on hadoop 23
> -------------------------------------------
>
> Key: MAPREDUCE-3696
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3696
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: John George
> Assignee: John George
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MR-3696.patch, MR-3696.patch, MR-3696.patch
>
>
> NM throws an error on submitting an MR job via oozie on the latest Hadoop 23.
> *Courtesy: Mona Chitnis (ooize)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3696) MR job via oozie does not work
on hadoop 23
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13198758#comment-13198758 ]
Hudson commented on MAPREDUCE-3696:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #157 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/157/])
MAPREDUCE-3696. MR job via oozie does not work on hadoop 23. (John George via mahadev) - Merging r1239310 from trunk.
mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1239312
Files :
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
> MR job via oozie does not work on hadoop 23
> -------------------------------------------
>
> Key: MAPREDUCE-3696
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3696
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: John George
> Assignee: John George
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MR-3696.patch, MR-3696.patch, MR-3696.patch
>
>
> NM throws an error on submitting an MR job via oozie on the latest Hadoop 23.
> *Courtesy: Mona Chitnis (ooize)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3696) MR job via oozie does not work
on hadoop 23
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194131#comment-13194131 ]
Hadoop QA commented on MAPREDUCE-3696:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12511842/MR-3696.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1684//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1684//console
This message is automatically generated.
> MR job via oozie does not work on hadoop 23
> -------------------------------------------
>
> Key: MAPREDUCE-3696
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3696
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: John George
> Assignee: John George
> Priority: Blocker
> Attachments: MR-3696.patch, MR-3696.patch
>
>
> NM throws an error on submitting an MR job via oozie on the latest Hadoop 23.
> *Courtesy: Mona Chitnis (ooize)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3696) MR job via oozie does not work
on hadoop 23
Posted by "Mahadev konar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13198115#comment-13198115 ]
Mahadev konar commented on MAPREDUCE-3696:
------------------------------------------
Thanks John for testing it out. The patch looks good to me. Ill go ahead and commit it.
> MR job via oozie does not work on hadoop 23
> -------------------------------------------
>
> Key: MAPREDUCE-3696
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3696
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: John George
> Assignee: John George
> Priority: Blocker
> Attachments: MR-3696.patch, MR-3696.patch, MR-3696.patch
>
>
> NM throws an error on submitting an MR job via oozie on the latest Hadoop 23.
> *Courtesy: Mona Chitnis (ooize)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3696) MR job via oozie does not work on
hadoop 23
Posted by "John George (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John George updated MAPREDUCE-3696:
-----------------------------------
Description:
NM throws an error on submitting an MR job via oozie on the latest Hadoop 23.
*Courtesy: Mona Chitnis (ooize)
was:NM throws an error on submitting an MR job via oozie on the latest Hadoop 23.
> MR job via oozie does not work on hadoop 23
> -------------------------------------------
>
> Key: MAPREDUCE-3696
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3696
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.23.1
> Reporter: John George
> Assignee: John George
> Priority: Critical
>
> NM throws an error on submitting an MR job via oozie on the latest Hadoop 23.
> *Courtesy: Mona Chitnis (ooize)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3696) MR job via oozie does not work
on hadoop 23
Posted by "Mahadev konar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189292#comment-13189292 ]
Mahadev konar commented on MAPREDUCE-3696:
------------------------------------------
@John,
Am not sure am getting why this is related to Oozie. On the same setup are you able to run any other jobs? I think this might just be a deployment issue.
> MR job via oozie does not work on hadoop 23
> -------------------------------------------
>
> Key: MAPREDUCE-3696
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3696
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.23.1
> Reporter: John George
> Assignee: John George
> Priority: Critical
>
> NM throws an error on submitting an MR job via oozie on the latest Hadoop 23.
> *Courtesy: Mona Chitnis (ooize)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3696) MR job via oozie does not work on
hadoop 23
Posted by "John George (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John George updated MAPREDUCE-3696:
-----------------------------------
Attachment: MR-3696.patch
> MR job via oozie does not work on hadoop 23
> -------------------------------------------
>
> Key: MAPREDUCE-3696
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3696
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: John George
> Assignee: John George
> Priority: Blocker
> Attachments: MR-3696.patch
>
>
> NM throws an error on submitting an MR job via oozie on the latest Hadoop 23.
> *Courtesy: Mona Chitnis (ooize)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3696) MR job via oozie does not work
on hadoop 23
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13198747#comment-13198747 ]
Hudson commented on MAPREDUCE-3696:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk #944 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/944/])
MAPREDUCE-3696. MR job via oozie does not work on hadoop 23. (John George via mahadev)
mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1239310
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
> MR job via oozie does not work on hadoop 23
> -------------------------------------------
>
> Key: MAPREDUCE-3696
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3696
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: John George
> Assignee: John George
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MR-3696.patch, MR-3696.patch, MR-3696.patch
>
>
> NM throws an error on submitting an MR job via oozie on the latest Hadoop 23.
> *Courtesy: Mona Chitnis (ooize)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3696) MR job via oozie does not work
on hadoop 23
Posted by "Mahadev konar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196534#comment-13196534 ]
Mahadev konar commented on MAPREDUCE-3696:
------------------------------------------
@John,
Sorry to keep bringing up new issues on this but I think we need a change for fixing the task classpaths to do the same as well and obey the property (or maybe its already fixed and we just need to make sure it is).
> MR job via oozie does not work on hadoop 23
> -------------------------------------------
>
> Key: MAPREDUCE-3696
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3696
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: John George
> Assignee: John George
> Priority: Blocker
> Attachments: MR-3696.patch, MR-3696.patch, MR-3696.patch
>
>
> NM throws an error on submitting an MR job via oozie on the latest Hadoop 23.
> *Courtesy: Mona Chitnis (ooize)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3696) MR job via oozie does not work on
hadoop 23
Posted by "Mahadev konar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-3696:
-------------------------------------
Resolution: Fixed
Fix Version/s: 0.23.1
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I just committed this. Thanks John!
> MR job via oozie does not work on hadoop 23
> -------------------------------------------
>
> Key: MAPREDUCE-3696
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3696
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: John George
> Assignee: John George
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MR-3696.patch, MR-3696.patch, MR-3696.patch
>
>
> NM throws an error on submitting an MR job via oozie on the latest Hadoop 23.
> *Courtesy: Mona Chitnis (ooize)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3696) MR job via oozie does not work
on hadoop 23
Posted by "Alejandro Abdelnur (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13190136#comment-13190136 ]
Alejandro Abdelnur commented on MAPREDUCE-3696:
-----------------------------------------------
@John, don't know what version of Pig you are using, but 0.9.0 which is avail in the maven repo does not include 3rd party JARs. still i'm seeing that the Pig POM declares as dependency jackson 1.4.2 and those JARs are pulled into Oozie's sharelib. (Oozie trunk speaking)
> MR job via oozie does not work on hadoop 23
> -------------------------------------------
>
> Key: MAPREDUCE-3696
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3696
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.23.1
> Reporter: John George
> Assignee: John George
> Priority: Critical
>
> NM throws an error on submitting an MR job via oozie on the latest Hadoop 23.
> *Courtesy: Mona Chitnis (ooize)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3696) MR job via oozie does not work on
hadoop 23
Posted by "John George (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John George updated MAPREDUCE-3696:
-----------------------------------
Status: Patch Available (was: Open)
> MR job via oozie does not work on hadoop 23
> -------------------------------------------
>
> Key: MAPREDUCE-3696
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3696
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: John George
> Assignee: John George
> Priority: Blocker
> Attachments: MR-3696.patch
>
>
> NM throws an error on submitting an MR job via oozie on the latest Hadoop 23.
> *Courtesy: Mona Chitnis (ooize)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (MAPREDUCE-3696) MR job via oozie
does not work on hadoop 23
Posted by "Mahadev konar (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13190187#comment-13190187 ]
Mahadev konar edited comment on MAPREDUCE-3696 at 1/20/12 10:51 PM:
--------------------------------------------------------------------
John,
Just for reference, MAPREDUCE-1938 added this property to be able to add your jars before others. Seems like in 0.23 we are doing that by default. We should maintain backwards compatibility and make sure that we load users jar later than hadoop jars unless the property is set as in MAPREDUCE-1938.
was (Author: mahadev):
John,
Just for reference, MAPREDUCE-1938 added this property to be able to add your jars before others by default. Seems like in 0.23 we are doing that by default. We should maintain backwards compatibility and make sure that we load users jar later than hadoop jars unless the property is set as in MAPREDUCE-1938.
> MR job via oozie does not work on hadoop 23
> -------------------------------------------
>
> Key: MAPREDUCE-3696
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3696
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: John George
> Assignee: John George
> Priority: Critical
>
> NM throws an error on submitting an MR job via oozie on the latest Hadoop 23.
> *Courtesy: Mona Chitnis (ooize)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3696) MR job via oozie does not work
on hadoop 23
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13198140#comment-13198140 ]
Hudson commented on MAPREDUCE-3696:
-----------------------------------
Integrated in Hadoop-Common-0.23-Commit #463 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/463/])
MAPREDUCE-3696. MR job via oozie does not work on hadoop 23. (John George via mahadev) - Merging r1239310 from trunk.
mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1239312
Files :
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
> MR job via oozie does not work on hadoop 23
> -------------------------------------------
>
> Key: MAPREDUCE-3696
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3696
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: John George
> Assignee: John George
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MR-3696.patch, MR-3696.patch, MR-3696.patch
>
>
> NM throws an error on submitting an MR job via oozie on the latest Hadoop 23.
> *Courtesy: Mona Chitnis (ooize)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3696) MR job via oozie does not work
on hadoop 23
Posted by "Mahadev konar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194060#comment-13194060 ]
Mahadev konar commented on MAPREDUCE-3696:
------------------------------------------
Not sure why hudson didnt run this. Will trigger it manually.
> MR job via oozie does not work on hadoop 23
> -------------------------------------------
>
> Key: MAPREDUCE-3696
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3696
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: John George
> Assignee: John George
> Priority: Blocker
> Attachments: MR-3696.patch
>
>
> NM throws an error on submitting an MR job via oozie on the latest Hadoop 23.
> *Courtesy: Mona Chitnis (ooize)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3696) MR job via oozie does not work
on hadoop 23
Posted by "Mahadev konar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13190149#comment-13190149 ]
Mahadev konar commented on MAPREDUCE-3696:
------------------------------------------
John,
In 0.20.* series there is a specific jira to add your own jars before the MR framework jars. We'll have to dig into it to see if that works or not. If that works then it might just be using htat flag to put your jars in front of the others jars.
> MR job via oozie does not work on hadoop 23
> -------------------------------------------
>
> Key: MAPREDUCE-3696
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3696
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.23.1
> Reporter: John George
> Assignee: John George
> Priority: Critical
>
> NM throws an error on submitting an MR job via oozie on the latest Hadoop 23.
> *Courtesy: Mona Chitnis (ooize)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3696) MR job via oozie does not work
on hadoop 23
Posted by "Mahadev konar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189300#comment-13189300 ]
Mahadev konar commented on MAPREDUCE-3696:
------------------------------------------
@John,
I think I know what the issue might be. Let me run some experiments and post on the jira.
> MR job via oozie does not work on hadoop 23
> -------------------------------------------
>
> Key: MAPREDUCE-3696
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3696
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.23.1
> Reporter: John George
> Assignee: John George
> Priority: Critical
>
> NM throws an error on submitting an MR job via oozie on the latest Hadoop 23.
> *Courtesy: Mona Chitnis (ooize)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3696) MR job via oozie does not work
on hadoop 23
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13198145#comment-13198145 ]
Hudson commented on MAPREDUCE-3696:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #1706 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1706/])
MAPREDUCE-3696. MR job via oozie does not work on hadoop 23. (John George via mahadev)
mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1239310
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
> MR job via oozie does not work on hadoop 23
> -------------------------------------------
>
> Key: MAPREDUCE-3696
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3696
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: John George
> Assignee: John George
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MR-3696.patch, MR-3696.patch, MR-3696.patch
>
>
> NM throws an error on submitting an MR job via oozie on the latest Hadoop 23.
> *Courtesy: Mona Chitnis (ooize)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3696) MR job via oozie does not work
on hadoop 23
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13198819#comment-13198819 ]
Hudson commented on MAPREDUCE-3696:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #977 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/977/])
MAPREDUCE-3696. MR job via oozie does not work on hadoop 23. (John George via mahadev)
mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1239310
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
> MR job via oozie does not work on hadoop 23
> -------------------------------------------
>
> Key: MAPREDUCE-3696
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3696
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: John George
> Assignee: John George
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MR-3696.patch, MR-3696.patch, MR-3696.patch
>
>
> NM throws an error on submitting an MR job via oozie on the latest Hadoop 23.
> *Courtesy: Mona Chitnis (ooize)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3696) MR job via oozie does not work
on hadoop 23
Posted by "John George (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197380#comment-13197380 ]
John George commented on MAPREDUCE-3696:
----------------------------------------
Mahadev - verified that the classpath on both the AM and tasks containers are the same by looking at the launch_container.sh CLASSPATH after passing in the mapreduce.job.user.classpath.first=true.
> MR job via oozie does not work on hadoop 23
> -------------------------------------------
>
> Key: MAPREDUCE-3696
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3696
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: John George
> Assignee: John George
> Priority: Blocker
> Attachments: MR-3696.patch, MR-3696.patch, MR-3696.patch
>
>
> NM throws an error on submitting an MR job via oozie on the latest Hadoop 23.
> *Courtesy: Mona Chitnis (ooize)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3696) MR job via oozie does not work
on hadoop 23
Posted by "Mahadev konar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13190187#comment-13190187 ]
Mahadev konar commented on MAPREDUCE-3696:
------------------------------------------
John,
Just for reference, MAPREDUCE-1938 added this property to be able to add your jars before others by default. Seems like in 0.23 we are doing that by default. We should maintain backwards compatibility and make sure that we load users jar later than hadoop jars unless the property is set as in MAPREDUCE-1938.
> MR job via oozie does not work on hadoop 23
> -------------------------------------------
>
> Key: MAPREDUCE-3696
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3696
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: John George
> Assignee: John George
> Priority: Critical
>
> NM throws an error on submitting an MR job via oozie on the latest Hadoop 23.
> *Courtesy: Mona Chitnis (ooize)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3696) MR job via oozie does not work
on hadoop 23
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13198779#comment-13198779 ]
Hudson commented on MAPREDUCE-3696:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Build #179 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/179/])
MAPREDUCE-3696. MR job via oozie does not work on hadoop 23. (John George via mahadev) - Merging r1239310 from trunk.
mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1239312
Files :
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
> MR job via oozie does not work on hadoop 23
> -------------------------------------------
>
> Key: MAPREDUCE-3696
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3696
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: John George
> Assignee: John George
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MR-3696.patch, MR-3696.patch, MR-3696.patch
>
>
> NM throws an error on submitting an MR job via oozie on the latest Hadoop 23.
> *Courtesy: Mona Chitnis (ooize)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3696) MR job via oozie does not work
on hadoop 23
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13198133#comment-13198133 ]
Hudson commented on MAPREDUCE-3696:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #1635 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1635/])
MAPREDUCE-3696. MR job via oozie does not work on hadoop 23. (John George via mahadev)
mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1239310
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
> MR job via oozie does not work on hadoop 23
> -------------------------------------------
>
> Key: MAPREDUCE-3696
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3696
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: John George
> Assignee: John George
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MR-3696.patch, MR-3696.patch, MR-3696.patch
>
>
> NM throws an error on submitting an MR job via oozie on the latest Hadoop 23.
> *Courtesy: Mona Chitnis (ooize)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3696) MR job via oozie does not work
on hadoop 23
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13198150#comment-13198150 ]
Hudson commented on MAPREDUCE-3696:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Commit #453 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/453/])
MAPREDUCE-3696. MR job via oozie does not work on hadoop 23. (John George via mahadev) - Merging r1239310 from trunk.
mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1239312
Files :
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
> MR job via oozie does not work on hadoop 23
> -------------------------------------------
>
> Key: MAPREDUCE-3696
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3696
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: John George
> Assignee: John George
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MR-3696.patch, MR-3696.patch, MR-3696.patch
>
>
> NM throws an error on submitting an MR job via oozie on the latest Hadoop 23.
> *Courtesy: Mona Chitnis (ooize)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3696) MR job via oozie does not work on
hadoop 23
Posted by "John George (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John George updated MAPREDUCE-3696:
-----------------------------------
Attachment: MR-3696.patch
Thanks for looking at this Mahadev.
Attaching a patch with HADOOP_USER_CLASSPATH_FIRST changes as well.
> MR job via oozie does not work on hadoop 23
> -------------------------------------------
>
> Key: MAPREDUCE-3696
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3696
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: John George
> Assignee: John George
> Priority: Blocker
> Attachments: MR-3696.patch, MR-3696.patch, MR-3696.patch
>
>
> NM throws an error on submitting an MR job via oozie on the latest Hadoop 23.
> *Courtesy: Mona Chitnis (ooize)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3696) MR job via oozie does not work on
hadoop 23
Posted by "John George (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John George updated MAPREDUCE-3696:
-----------------------------------
Attachment: MR-3696.patch
> MR job via oozie does not work on hadoop 23
> -------------------------------------------
>
> Key: MAPREDUCE-3696
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3696
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: John George
> Assignee: John George
> Priority: Blocker
> Attachments: MR-3696.patch, MR-3696.patch
>
>
> NM throws an error on submitting an MR job via oozie on the latest Hadoop 23.
> *Courtesy: Mona Chitnis (ooize)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3696) MR job via oozie does not work
on hadoop 23
Posted by "John George (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189933#comment-13189933 ]
John George commented on MAPREDUCE-3696:
----------------------------------------
The issue is that oozie is including a pig.jar file that picks up an older version of jackson. But what confuses me is that the pig.jar gets picked up before all the hadoop jars.
Assuming that oozie does not force USER CLASSPATHs to be picked up first, I wonder if it is wrong for App Master to include that before the default hadoop jars.
The following is what I am seeing in the launch_container.sh script.
{quote}
export CLASSPATH="job.jar:$PWD/*:file:/gri
{quote}
and
{quote}
ln -sf "/grid/1/tmp/mapred-local/filecache/7267871525837532837/pig.jar" "pig.jar"
{quote}
which obviously means that pig.jar in $PWD is going to be picked up before *jackson-*.jar in the hadoop lib directory - which does not seem right.
> MR job via oozie does not work on hadoop 23
> -------------------------------------------
>
> Key: MAPREDUCE-3696
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3696
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.23.1
> Reporter: John George
> Assignee: John George
> Priority: Critical
>
> NM throws an error on submitting an MR job via oozie on the latest Hadoop 23.
> *Courtesy: Mona Chitnis (ooize)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3696) MR job via oozie does not work
on hadoop 23
Posted by "Mahadev konar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194301#comment-13194301 ]
Mahadev konar commented on MAPREDUCE-3696:
------------------------------------------
@john,
One more thing I forgot to mention. In 1.0, you'd have to set:
{code}
export HADOOP_USER_CLASSPATH_FIRST=true
{code}
for submit/client side to use your HADOOP_CLASS_PATH before frameworks. Does that work here as well? If not we'd have to fix that too before claiming victory on the classpath issue.
> MR job via oozie does not work on hadoop 23
> -------------------------------------------
>
> Key: MAPREDUCE-3696
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3696
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: John George
> Assignee: John George
> Priority: Blocker
> Attachments: MR-3696.patch, MR-3696.patch
>
>
> NM throws an error on submitting an MR job via oozie on the latest Hadoop 23.
> *Courtesy: Mona Chitnis (ooize)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3696) MR job via oozie does not work
on hadoop 23
Posted by "John George (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194088#comment-13194088 ]
John George commented on MAPREDUCE-3696:
----------------------------------------
@Mahadev - the previous patch was using USER_JOB_CLASSPATH_FIRST as compared to it MAPREDUCE_JOB_USER_CLASSPATH_FIRST in 1.0. Hence will upload a new patch to ensure that it is compatible with 1.0. Behavior wise, afai can tell, it is compatible.
> MR job via oozie does not work on hadoop 23
> -------------------------------------------
>
> Key: MAPREDUCE-3696
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3696
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: John George
> Assignee: John George
> Priority: Blocker
> Attachments: MR-3696.patch, MR-3696.patch
>
>
> NM throws an error on submitting an MR job via oozie on the latest Hadoop 23.
> *Courtesy: Mona Chitnis (ooize)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3696) MR job via oozie does not work
on hadoop 23
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194164#comment-13194164 ]
Hadoop QA commented on MAPREDUCE-3696:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12512019/MR-3696.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1686//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1686//console
This message is automatically generated.
> MR job via oozie does not work on hadoop 23
> -------------------------------------------
>
> Key: MAPREDUCE-3696
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3696
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: John George
> Assignee: John George
> Priority: Blocker
> Attachments: MR-3696.patch, MR-3696.patch
>
>
> NM throws an error on submitting an MR job via oozie on the latest Hadoop 23.
> *Courtesy: Mona Chitnis (ooize)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3696) MR job via oozie does not work on
hadoop 23
Posted by "Mahadev konar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-3696:
-------------------------------------
Component/s: mrv2
> MR job via oozie does not work on hadoop 23
> -------------------------------------------
>
> Key: MAPREDUCE-3696
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3696
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: John George
> Assignee: John George
> Priority: Critical
>
> NM throws an error on submitting an MR job via oozie on the latest Hadoop 23.
> *Courtesy: Mona Chitnis (ooize)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3696) MR job via oozie does not work
on hadoop 23
Posted by "John George (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13190144#comment-13190144 ]
John George commented on MAPREDUCE-3696:
----------------------------------------
So, Pig needs to upgrade its jackson dependency for oozie to run a pig job then I guess. Either way, shouldn't the classpath have hadoop jars before the apps current directory?
{quote}
export CLASSPATH="job.jar:$PWD/*:file:/gri
{quote}
> MR job via oozie does not work on hadoop 23
> -------------------------------------------
>
> Key: MAPREDUCE-3696
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3696
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.23.1
> Reporter: John George
> Assignee: John George
> Priority: Critical
>
> NM throws an error on submitting an MR job via oozie on the latest Hadoop 23.
> *Courtesy: Mona Chitnis (ooize)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3696) MR job via oozie does not work
on hadoop 23
Posted by "John George (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189296#comment-13189296 ]
John George commented on MAPREDUCE-3696:
----------------------------------------
Yes - I am able to run other jobs. I have a hunch as well that this might just be a deployment/setup issue, but still wanted to file a JIRA just-in-case.
> MR job via oozie does not work on hadoop 23
> -------------------------------------------
>
> Key: MAPREDUCE-3696
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3696
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.23.1
> Reporter: John George
> Assignee: John George
> Priority: Critical
>
> NM throws an error on submitting an MR job via oozie on the latest Hadoop 23.
> *Courtesy: Mona Chitnis (ooize)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3696) MR job via oozie does not work
on hadoop 23
Posted by "John George (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197378#comment-13197378 ]
John George commented on MAPREDUCE-3696:
----------------------------------------
Mahadev - verified that mapreduce.job.user.classpath.first gets propagated to both the AppM and the tasks by looking at the launch_container.sh script.
> MR job via oozie does not work on hadoop 23
> -------------------------------------------
>
> Key: MAPREDUCE-3696
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3696
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: John George
> Assignee: John George
> Priority: Blocker
> Attachments: MR-3696.patch, MR-3696.patch, MR-3696.patch
>
>
> NM throws an error on submitting an MR job via oozie on the latest Hadoop 23.
> *Courtesy: Mona Chitnis (ooize)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (MAPREDUCE-3696) MR job via oozie
does not work on hadoop 23
Posted by "Mahadev konar (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13190149#comment-13190149 ]
Mahadev konar edited comment on MAPREDUCE-3696 at 1/20/12 9:51 PM:
-------------------------------------------------------------------
John,
In 0.20.* series there is a specific property to add your own jars before the MR framework jars. We'll have to dig into it to see if that works or not. If that works then it might just be using htat flag to put your jars in front of the others jars.
was (Author: mahadev):
John,
In 0.20.* series there is a specific jira to add your own jars before the MR framework jars. We'll have to dig into it to see if that works or not. If that works then it might just be using htat flag to put your jars in front of the others jars.
> MR job via oozie does not work on hadoop 23
> -------------------------------------------
>
> Key: MAPREDUCE-3696
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3696
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.23.1
> Reporter: John George
> Assignee: John George
> Priority: Critical
>
> NM throws an error on submitting an MR job via oozie on the latest Hadoop 23.
> *Courtesy: Mona Chitnis (ooize)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira