You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Pratik Gadiya <pr...@persistent.com> on 2015/01/02 15:37:33 UTC
Apache Hadoop 2.2 - Job execution error
Hi All,
I am using Pivotal Distribution to deploy Hadoop cluster.
For the same, I am following link mentioned below
http://pivotalhd.docs.pivotal.io/doc/2100/webhelp/topics/ManuallyInstallingandUsingPivotalHD21Stack.html
Deployment Configuration :
1) phd1.xyz.com - NameNode, ResourceManager
2) phd2.xyz.com - DataNode, NodeManager
I have above mentioned services UP and Running and also able to access the HDFS file system but not able to execute jobs on cluster
Stderr file log :
Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster
In order to resolve the same problem, I added classpath property tag in yarn-site.xml but still I am not able to execute the job, resulting into same error as above.
yarn-site.xml :
<property>
<name>yarn.application.classpath</name>
<value>/usr/lib/gphd/hadoop/*,/usr/lib/gphd/hadoop/lib/*,/usr/lib/gphd/hadoop-hdfs/*,/usr/lib/gphd/hadoop-hdfs/lib/*,/usr/lib/gphd/hadoop-yarn/*,/usr/lib/gphd/hadoop-yarn/lib/*,/usr/lib/gphd/hadoop-mapreduce/*,/usr/lib/gphd/hadoop-mapreduce/lib/*,/etc/gphd/hadoop/conf/*</value>
</property>
[root@phd2 bin]# ls /usr/lib/gphd/hadoop/
bin hadoop-annotations.jar hadoop-common-2.2.0-gphd-3.1.0.0-tests.jar hadoop-nfs.jar NOTICE.txt
CHANGES.txt hadoop-auth-2.2.0-gphd-3.1.0.0.jar hadoop-common.jar lib README.txt
etc hadoop-auth.jar hadoop-common-tests.jar libexec sbin
hadoop-annotations-2.2.0-gphd-3.1.0.0.jar hadoop-common-2.2.0-gphd-3.1.0.0.jar hadoop-nfs-2.2.0-gphd-3.1.0.0.jar LICENSE.txt
[root@phd2 bin]# ls /usr/lib/gphd/hadoop/lib
activation-1.1.jar commons-httpclient-3.1.jar jasper-runtime-5.5.23.jar jetty-security-7.6.10.v20130312.jar native
asm-3.2.jar commons-io-2.1.jar javax.servlet-2.5.0.v201103041518.jar jetty-server-7.6.10.v20130312.jar netty-3.6.2.Final.jar
avro-1.7.4.jar commons-lang-2.5.jar jaxb-api-2.2.2.jar jetty-servlet-7.6.10.v20130312.jar paranamer-2.3.jar
commons-beanutils-1.7.0.jar commons-logging-1.1.1.jar jaxb-impl-2.2.3-1.jar jetty-util-7.6.10.v20130312.jar protobuf-java-2.5.0.jar
commons-beanutils-core-1.8.0.jar commons-math-2.1.jar jersey-core-1.9.jar jetty-webapp-7.6.10.v20130312.jar servlet-api-2.5.jar
commons-cli-1.2.jar commons-net-3.1.jar jersey-json-1.9.jar jetty-xml-7.6.10.v20130312.jar slf4j-api-1.7.5.jar
commons-codec-1.4.jar guava-11.0.2.jar jersey-server-1.9.jar jsch-0.1.42.jar slf4j-log4j12-1.7.5.jar
commons-collections-3.2.1.jar jackson-core-asl-1.8.8.jar jets3t-0.6.1.jar jsp-api-2.1.jar snappy-java-1.0.4.1.jar
commons-compress-1.4.1.jar jackson-jaxrs-1.8.8.jar jettison-1.1.jar jsr305-1.3.9.jar stax-api-1.0.1.jar
commons-configuration-1.6.jar jackson-mapper-asl-1.8.8.jar jetty-continuation-7.6.10.v20130312.jar junit-4.8.2.jar xmlenc-0.52.jar
commons-digester-1.8.jar jackson-xc-1.8.8.jar jetty-http-7.6.10.v20130312.jar log4j-1.2.17.jar xz-1.0.jar
commons-el-1.0.jar jasper-compiler-5.5.23.jar jetty-io-7.6.10.v20130312.jar mockito-all-1.8.5.jar zookeeper.jar
[root@phd2 bin]# ls /usr/lib/gphd/hadoop-yarn/lib
aopalliance-1.0.jar commons-io-2.1.jar jackson-core-asl-1.8.8.jar jersey-guice-1.9.jar netty-3.6.2.Final.jar xz-1.0.jar
asm-3.2.jar guice-3.0.jar jackson-mapper-asl-1.8.8.jar jersey-server-1.9.jar paranamer-2.3.jar
avro-1.7.4.jar guice-servlet-3.0.jar javax.inject-1.jar junit-4.10.jar protobuf-java-2.5.0.jar
commons-compress-1.4.1.jar hamcrest-core-1.1.jar jersey-core-1.9.jar log4j-1.2.17.jar snappy-java-1.0.4.1.jar
I am not sure which all environment varibles should be exported or described in the hadoop-env.sh or othere configuration files.
Note :
Pivotal version which I have installed supports Apache Hadoop 2.2
Remember:
While replying, please keep gmail id in loop
Let me know how can I resolve the error.
With Regards,
Pratik
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
Re: Apache Hadoop 2.2 - Job execution error
Posted by Ted Yu <yu...@gmail.com>.
~/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-app/2.2.0/hadoop-mapreduce-client-app-2.2.0.jar
| grep MRAppMaster
...
45581 Mon Oct 07 06:26:42 PDT 2013
org/apache/hadoop/mapreduce/v2/app/MRAppMaster.class
I don't see the above jar in your listing.
Can you find it and put it on the classpath ?
BTW hadoop 2.6.0 was released last year, please consider using the latest
release.
Cheers
On Fri, Jan 2, 2015 at 12:46 PM, Pratik Gadiya <pratik_gadiya@persistent.com
> wrote:
> I didn’t get much help from Pivotal.
>
>
>
> So, I thought to post it on user mailing list
>
>
>
> Thanks,
>
> Pratik
>
> *From:* Ted Yu [mailto:yuzhihong@gmail.com]
> *Sent:* Friday, January 02, 2015 8:20 PM
> *To:* user@hadoop.apache.org
> *Cc:* Pratik Gadiya; pratikgadiya1@gmail.com
> *Subject:* Re: Apache Hadoop 2.2 - Job execution error
>
>
>
> Have you asked this question on Pivotal's mailing list ?
>
>
>
> Cheers
>
>
> On Jan 2, 2015, at 6:37 AM, Pratik Gadiya <pr...@persistent.com>
> wrote:
>
> Hi All,
>
>
>
> I am using Pivotal Distribution to deploy Hadoop cluster.
>
>
>
> For the same, I am following link mentioned below
>
>
> http://pivotalhd.docs.pivotal.io/doc/2100/webhelp/topics/ManuallyInstallingandUsingPivotalHD21Stack.html
>
>
>
>
>
> *Deployment Configuration** : *
>
>
>
> 1) phd1.xyz.com - NameNode, ResourceManager
>
> 2) phd2.xyz.com - DataNode, NodeManager
>
>
>
> I have above mentioned services *UP* and *Running* and also able to
> access the HDFS file system but not able to execute jobs on cluster
>
>
>
> *Stderr file log* :
>
> *Error: Could not find or load main class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster*
>
>
>
> In order to resolve the same problem, I added classpath property tag in
> yarn-site.xml but still I am not able to execute the job, resulting into
> same error as above.
>
>
>
> *yarn-site.xml :*
>
> <property>
>
> <name>yarn.application.classpath</name>
>
>
> <value>/usr/lib/gphd/hadoop/*,/usr/lib/gphd/hadoop/lib/*,/usr/lib/gphd/hadoop-hdfs/*,/usr/lib/gphd/hadoop-hdfs/lib/*,/usr/lib/gphd/hadoop-yarn/*,/usr/lib/gphd/hadoop-yarn/lib/*,/usr/lib/gphd/hadoop-mapreduce/*,/usr/lib/gphd/hadoop-mapreduce/lib/*,/etc/gphd/hadoop/conf/*</value>
>
> </property>
>
>
>
>
>
> [root@phd2 bin]# ls /usr/lib/gphd/hadoop/
>
> bin
> hadoop-annotations.jar
> hadoop-common-2.2.0-gphd-3.1.0.0-tests.jar hadoop-nfs.jar NOTICE.txt
>
> CHANGES.txt
> hadoop-auth-2.2.0-gphd-3.1.0.0.jar
> hadoop-common.jar lib README.txt
>
> etc
> hadoop-auth.jar
> hadoop-common-tests.jar libexec sbin
>
> hadoop-annotations-2.2.0-gphd-3.1.0.0.jar
> hadoop-common-2.2.0-gphd-3.1.0.0.jar
> hadoop-nfs-2.2.0-gphd-3.1.0.0.jar LICENSE.txt
>
>
>
>
>
> [root@phd2 bin]# ls /usr/lib/gphd/hadoop/lib
>
> activation-1.1.jar commons-httpclient-3.1.jar
> jasper-runtime-5.5.23.jar
> jetty-security-7.6.10.v20130312.jar native
>
> asm-3.2.jar commons-io-2.1.jar
> javax.servlet-2.5.0.v201103041518.jar
> jetty-server-7.6.10.v20130312.jar netty-3.6.2.Final.jar
>
> avro-1.7.4.jar commons-lang-2.5.jar
> jaxb-api-2.2.2.jar
> jetty-servlet-7.6.10.v20130312.jar paranamer-2.3.jar
>
> commons-beanutils-1.7.0.jar commons-logging-1.1.1.jar
> jaxb-impl-2.2.3-1.jar
> jetty-util-7.6.10.v20130312.jar protobuf-java-2.5.0.jar
>
> commons-beanutils-core-1.8.0.jar commons-math-2.1.jar
> jersey-core-1.9.jar
> jetty-webapp-7.6.10.v20130312.jar servlet-api-2.5.jar
>
> commons-cli-1.2.jar commons-net-3.1.jar
> jersey-json-1.9.jar
> jetty-xml-7.6.10.v20130312.jar slf4j-api-1.7.5.jar
>
> commons-codec-1.4.jar guava-11.0.2.jar
> jersey-server-1.9.jar
> jsch-0.1.42.jar slf4j-log4j12-1.7.5.jar
>
> commons-collections-3.2.1.jar jackson-core-asl-1.8.8.jar
> jets3t-0.6.1.jar
> jsp-api-2.1.jar snappy-java-1.0.4.1.jar
>
> commons-compress-1.4.1.jar jackson-jaxrs-1.8.8.jar
> jettison-1.1.jar
> jsr305-1.3.9.jar stax-api-1.0.1.jar
>
> commons-configuration-1.6.jar jackson-mapper-asl-1.8.8.jar
> jetty-continuation-7.6.10.v20130312.jar
> junit-4.8.2.jar xmlenc-0.52.jar
>
> commons-digester-1.8.jar jackson-xc-1.8.8.jar
> jetty-http-7.6.10.v20130312.jar
> log4j-1.2.17.jar xz-1.0.jar
>
> commons-el-1.0.jar jasper-compiler-5.5.23.jar
> jetty-io-7.6.10.v20130312.jar
> mockito-all-1.8.5.jar zookeeper.jar
>
>
>
>
>
> [root@phd2 bin]# ls /usr/lib/gphd/hadoop-yarn/lib
>
> aopalliance-1.0.jar commons-io-2.1.jar
> jackson-core-asl-1.8.8.jar jersey-guice-1.9.jar
> netty-3.6.2.Final.jar xz-1.0.jar
>
> asm-3.2.jar guice-3.0.jar
> jackson-mapper-asl-1.8.8.jar jersey-server-1.9.jar paranamer-2.3.jar
>
> avro-1.7.4.jar guice-servlet-3.0.jar
> javax.inject-1.jar junit-4.10.jar protobuf-java-2.5.0.jar
>
> commons-compress-1.4.1.jar hamcrest-core-1.1.jar
> jersey-core-1.9.jar log4j-1.2.17.jar snappy-java-1.0.4.1.jar
>
>
>
>
>
> I am not sure which all environment varibles should be exported or
> described in the hadoop-env.sh or othere configuration files.
>
>
>
> *Note* :
>
> Pivotal version which I have installed supports *Apache Hadoop 2.2*
>
>
>
> *Remember:*
>
> While replying, please keep gmail id in loop
>
>
>
> Let me know how can I resolve the error.
>
>
>
>
>
> With Regards,
>
> Pratik
>
>
>
> DISCLAIMER ========== This e-mail may contain privileged and confidential
> information which is the property of Persistent Systems Ltd. It is intended
> only for the use of the individual or entity to which it is addressed. If
> you are not the intended recipient, you are not authorized to read, retain,
> copy, print, distribute or use this message. If you have received this
> communication in error, please notify the sender and delete all copies of
> this message. Persistent Systems Ltd. does not accept any liability for
> virus infected mails.
>
> DISCLAIMER ========== This e-mail may contain privileged and
> confidential information which is the property of Persistent Systems Ltd.
> It is intended only for the use of the individual or entity to which it is
> addressed. If you are not the intended recipient, you are not authorized to
> read, retain, copy, print, distribute or use this message. If you have
> received this communication in error, please notify the sender and delete
> all copies of this message. Persistent Systems Ltd. does not accept any
> liability for virus infected mails.
>
Re: Apache Hadoop 2.2 - Job execution error
Posted by Ted Yu <yu...@gmail.com>.
~/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-app/2.2.0/hadoop-mapreduce-client-app-2.2.0.jar
| grep MRAppMaster
...
45581 Mon Oct 07 06:26:42 PDT 2013
org/apache/hadoop/mapreduce/v2/app/MRAppMaster.class
I don't see the above jar in your listing.
Can you find it and put it on the classpath ?
BTW hadoop 2.6.0 was released last year, please consider using the latest
release.
Cheers
On Fri, Jan 2, 2015 at 12:46 PM, Pratik Gadiya <pratik_gadiya@persistent.com
> wrote:
> I didn’t get much help from Pivotal.
>
>
>
> So, I thought to post it on user mailing list
>
>
>
> Thanks,
>
> Pratik
>
> *From:* Ted Yu [mailto:yuzhihong@gmail.com]
> *Sent:* Friday, January 02, 2015 8:20 PM
> *To:* user@hadoop.apache.org
> *Cc:* Pratik Gadiya; pratikgadiya1@gmail.com
> *Subject:* Re: Apache Hadoop 2.2 - Job execution error
>
>
>
> Have you asked this question on Pivotal's mailing list ?
>
>
>
> Cheers
>
>
> On Jan 2, 2015, at 6:37 AM, Pratik Gadiya <pr...@persistent.com>
> wrote:
>
> Hi All,
>
>
>
> I am using Pivotal Distribution to deploy Hadoop cluster.
>
>
>
> For the same, I am following link mentioned below
>
>
> http://pivotalhd.docs.pivotal.io/doc/2100/webhelp/topics/ManuallyInstallingandUsingPivotalHD21Stack.html
>
>
>
>
>
> *Deployment Configuration** : *
>
>
>
> 1) phd1.xyz.com - NameNode, ResourceManager
>
> 2) phd2.xyz.com - DataNode, NodeManager
>
>
>
> I have above mentioned services *UP* and *Running* and also able to
> access the HDFS file system but not able to execute jobs on cluster
>
>
>
> *Stderr file log* :
>
> *Error: Could not find or load main class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster*
>
>
>
> In order to resolve the same problem, I added classpath property tag in
> yarn-site.xml but still I am not able to execute the job, resulting into
> same error as above.
>
>
>
> *yarn-site.xml :*
>
> <property>
>
> <name>yarn.application.classpath</name>
>
>
> <value>/usr/lib/gphd/hadoop/*,/usr/lib/gphd/hadoop/lib/*,/usr/lib/gphd/hadoop-hdfs/*,/usr/lib/gphd/hadoop-hdfs/lib/*,/usr/lib/gphd/hadoop-yarn/*,/usr/lib/gphd/hadoop-yarn/lib/*,/usr/lib/gphd/hadoop-mapreduce/*,/usr/lib/gphd/hadoop-mapreduce/lib/*,/etc/gphd/hadoop/conf/*</value>
>
> </property>
>
>
>
>
>
> [root@phd2 bin]# ls /usr/lib/gphd/hadoop/
>
> bin
> hadoop-annotations.jar
> hadoop-common-2.2.0-gphd-3.1.0.0-tests.jar hadoop-nfs.jar NOTICE.txt
>
> CHANGES.txt
> hadoop-auth-2.2.0-gphd-3.1.0.0.jar
> hadoop-common.jar lib README.txt
>
> etc
> hadoop-auth.jar
> hadoop-common-tests.jar libexec sbin
>
> hadoop-annotations-2.2.0-gphd-3.1.0.0.jar
> hadoop-common-2.2.0-gphd-3.1.0.0.jar
> hadoop-nfs-2.2.0-gphd-3.1.0.0.jar LICENSE.txt
>
>
>
>
>
> [root@phd2 bin]# ls /usr/lib/gphd/hadoop/lib
>
> activation-1.1.jar commons-httpclient-3.1.jar
> jasper-runtime-5.5.23.jar
> jetty-security-7.6.10.v20130312.jar native
>
> asm-3.2.jar commons-io-2.1.jar
> javax.servlet-2.5.0.v201103041518.jar
> jetty-server-7.6.10.v20130312.jar netty-3.6.2.Final.jar
>
> avro-1.7.4.jar commons-lang-2.5.jar
> jaxb-api-2.2.2.jar
> jetty-servlet-7.6.10.v20130312.jar paranamer-2.3.jar
>
> commons-beanutils-1.7.0.jar commons-logging-1.1.1.jar
> jaxb-impl-2.2.3-1.jar
> jetty-util-7.6.10.v20130312.jar protobuf-java-2.5.0.jar
>
> commons-beanutils-core-1.8.0.jar commons-math-2.1.jar
> jersey-core-1.9.jar
> jetty-webapp-7.6.10.v20130312.jar servlet-api-2.5.jar
>
> commons-cli-1.2.jar commons-net-3.1.jar
> jersey-json-1.9.jar
> jetty-xml-7.6.10.v20130312.jar slf4j-api-1.7.5.jar
>
> commons-codec-1.4.jar guava-11.0.2.jar
> jersey-server-1.9.jar
> jsch-0.1.42.jar slf4j-log4j12-1.7.5.jar
>
> commons-collections-3.2.1.jar jackson-core-asl-1.8.8.jar
> jets3t-0.6.1.jar
> jsp-api-2.1.jar snappy-java-1.0.4.1.jar
>
> commons-compress-1.4.1.jar jackson-jaxrs-1.8.8.jar
> jettison-1.1.jar
> jsr305-1.3.9.jar stax-api-1.0.1.jar
>
> commons-configuration-1.6.jar jackson-mapper-asl-1.8.8.jar
> jetty-continuation-7.6.10.v20130312.jar
> junit-4.8.2.jar xmlenc-0.52.jar
>
> commons-digester-1.8.jar jackson-xc-1.8.8.jar
> jetty-http-7.6.10.v20130312.jar
> log4j-1.2.17.jar xz-1.0.jar
>
> commons-el-1.0.jar jasper-compiler-5.5.23.jar
> jetty-io-7.6.10.v20130312.jar
> mockito-all-1.8.5.jar zookeeper.jar
>
>
>
>
>
> [root@phd2 bin]# ls /usr/lib/gphd/hadoop-yarn/lib
>
> aopalliance-1.0.jar commons-io-2.1.jar
> jackson-core-asl-1.8.8.jar jersey-guice-1.9.jar
> netty-3.6.2.Final.jar xz-1.0.jar
>
> asm-3.2.jar guice-3.0.jar
> jackson-mapper-asl-1.8.8.jar jersey-server-1.9.jar paranamer-2.3.jar
>
> avro-1.7.4.jar guice-servlet-3.0.jar
> javax.inject-1.jar junit-4.10.jar protobuf-java-2.5.0.jar
>
> commons-compress-1.4.1.jar hamcrest-core-1.1.jar
> jersey-core-1.9.jar log4j-1.2.17.jar snappy-java-1.0.4.1.jar
>
>
>
>
>
> I am not sure which all environment varibles should be exported or
> described in the hadoop-env.sh or othere configuration files.
>
>
>
> *Note* :
>
> Pivotal version which I have installed supports *Apache Hadoop 2.2*
>
>
>
> *Remember:*
>
> While replying, please keep gmail id in loop
>
>
>
> Let me know how can I resolve the error.
>
>
>
>
>
> With Regards,
>
> Pratik
>
>
>
> DISCLAIMER ========== This e-mail may contain privileged and confidential
> information which is the property of Persistent Systems Ltd. It is intended
> only for the use of the individual or entity to which it is addressed. If
> you are not the intended recipient, you are not authorized to read, retain,
> copy, print, distribute or use this message. If you have received this
> communication in error, please notify the sender and delete all copies of
> this message. Persistent Systems Ltd. does not accept any liability for
> virus infected mails.
>
> DISCLAIMER ========== This e-mail may contain privileged and
> confidential information which is the property of Persistent Systems Ltd.
> It is intended only for the use of the individual or entity to which it is
> addressed. If you are not the intended recipient, you are not authorized to
> read, retain, copy, print, distribute or use this message. If you have
> received this communication in error, please notify the sender and delete
> all copies of this message. Persistent Systems Ltd. does not accept any
> liability for virus infected mails.
>
Re: Apache Hadoop 2.2 - Job execution error
Posted by Ted Yu <yu...@gmail.com>.
~/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-app/2.2.0/hadoop-mapreduce-client-app-2.2.0.jar
| grep MRAppMaster
...
45581 Mon Oct 07 06:26:42 PDT 2013
org/apache/hadoop/mapreduce/v2/app/MRAppMaster.class
I don't see the above jar in your listing.
Can you find it and put it on the classpath ?
BTW hadoop 2.6.0 was released last year, please consider using the latest
release.
Cheers
On Fri, Jan 2, 2015 at 12:46 PM, Pratik Gadiya <pratik_gadiya@persistent.com
> wrote:
> I didn’t get much help from Pivotal.
>
>
>
> So, I thought to post it on user mailing list
>
>
>
> Thanks,
>
> Pratik
>
> *From:* Ted Yu [mailto:yuzhihong@gmail.com]
> *Sent:* Friday, January 02, 2015 8:20 PM
> *To:* user@hadoop.apache.org
> *Cc:* Pratik Gadiya; pratikgadiya1@gmail.com
> *Subject:* Re: Apache Hadoop 2.2 - Job execution error
>
>
>
> Have you asked this question on Pivotal's mailing list ?
>
>
>
> Cheers
>
>
> On Jan 2, 2015, at 6:37 AM, Pratik Gadiya <pr...@persistent.com>
> wrote:
>
> Hi All,
>
>
>
> I am using Pivotal Distribution to deploy Hadoop cluster.
>
>
>
> For the same, I am following link mentioned below
>
>
> http://pivotalhd.docs.pivotal.io/doc/2100/webhelp/topics/ManuallyInstallingandUsingPivotalHD21Stack.html
>
>
>
>
>
> *Deployment Configuration** : *
>
>
>
> 1) phd1.xyz.com - NameNode, ResourceManager
>
> 2) phd2.xyz.com - DataNode, NodeManager
>
>
>
> I have above mentioned services *UP* and *Running* and also able to
> access the HDFS file system but not able to execute jobs on cluster
>
>
>
> *Stderr file log* :
>
> *Error: Could not find or load main class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster*
>
>
>
> In order to resolve the same problem, I added classpath property tag in
> yarn-site.xml but still I am not able to execute the job, resulting into
> same error as above.
>
>
>
> *yarn-site.xml :*
>
> <property>
>
> <name>yarn.application.classpath</name>
>
>
> <value>/usr/lib/gphd/hadoop/*,/usr/lib/gphd/hadoop/lib/*,/usr/lib/gphd/hadoop-hdfs/*,/usr/lib/gphd/hadoop-hdfs/lib/*,/usr/lib/gphd/hadoop-yarn/*,/usr/lib/gphd/hadoop-yarn/lib/*,/usr/lib/gphd/hadoop-mapreduce/*,/usr/lib/gphd/hadoop-mapreduce/lib/*,/etc/gphd/hadoop/conf/*</value>
>
> </property>
>
>
>
>
>
> [root@phd2 bin]# ls /usr/lib/gphd/hadoop/
>
> bin
> hadoop-annotations.jar
> hadoop-common-2.2.0-gphd-3.1.0.0-tests.jar hadoop-nfs.jar NOTICE.txt
>
> CHANGES.txt
> hadoop-auth-2.2.0-gphd-3.1.0.0.jar
> hadoop-common.jar lib README.txt
>
> etc
> hadoop-auth.jar
> hadoop-common-tests.jar libexec sbin
>
> hadoop-annotations-2.2.0-gphd-3.1.0.0.jar
> hadoop-common-2.2.0-gphd-3.1.0.0.jar
> hadoop-nfs-2.2.0-gphd-3.1.0.0.jar LICENSE.txt
>
>
>
>
>
> [root@phd2 bin]# ls /usr/lib/gphd/hadoop/lib
>
> activation-1.1.jar commons-httpclient-3.1.jar
> jasper-runtime-5.5.23.jar
> jetty-security-7.6.10.v20130312.jar native
>
> asm-3.2.jar commons-io-2.1.jar
> javax.servlet-2.5.0.v201103041518.jar
> jetty-server-7.6.10.v20130312.jar netty-3.6.2.Final.jar
>
> avro-1.7.4.jar commons-lang-2.5.jar
> jaxb-api-2.2.2.jar
> jetty-servlet-7.6.10.v20130312.jar paranamer-2.3.jar
>
> commons-beanutils-1.7.0.jar commons-logging-1.1.1.jar
> jaxb-impl-2.2.3-1.jar
> jetty-util-7.6.10.v20130312.jar protobuf-java-2.5.0.jar
>
> commons-beanutils-core-1.8.0.jar commons-math-2.1.jar
> jersey-core-1.9.jar
> jetty-webapp-7.6.10.v20130312.jar servlet-api-2.5.jar
>
> commons-cli-1.2.jar commons-net-3.1.jar
> jersey-json-1.9.jar
> jetty-xml-7.6.10.v20130312.jar slf4j-api-1.7.5.jar
>
> commons-codec-1.4.jar guava-11.0.2.jar
> jersey-server-1.9.jar
> jsch-0.1.42.jar slf4j-log4j12-1.7.5.jar
>
> commons-collections-3.2.1.jar jackson-core-asl-1.8.8.jar
> jets3t-0.6.1.jar
> jsp-api-2.1.jar snappy-java-1.0.4.1.jar
>
> commons-compress-1.4.1.jar jackson-jaxrs-1.8.8.jar
> jettison-1.1.jar
> jsr305-1.3.9.jar stax-api-1.0.1.jar
>
> commons-configuration-1.6.jar jackson-mapper-asl-1.8.8.jar
> jetty-continuation-7.6.10.v20130312.jar
> junit-4.8.2.jar xmlenc-0.52.jar
>
> commons-digester-1.8.jar jackson-xc-1.8.8.jar
> jetty-http-7.6.10.v20130312.jar
> log4j-1.2.17.jar xz-1.0.jar
>
> commons-el-1.0.jar jasper-compiler-5.5.23.jar
> jetty-io-7.6.10.v20130312.jar
> mockito-all-1.8.5.jar zookeeper.jar
>
>
>
>
>
> [root@phd2 bin]# ls /usr/lib/gphd/hadoop-yarn/lib
>
> aopalliance-1.0.jar commons-io-2.1.jar
> jackson-core-asl-1.8.8.jar jersey-guice-1.9.jar
> netty-3.6.2.Final.jar xz-1.0.jar
>
> asm-3.2.jar guice-3.0.jar
> jackson-mapper-asl-1.8.8.jar jersey-server-1.9.jar paranamer-2.3.jar
>
> avro-1.7.4.jar guice-servlet-3.0.jar
> javax.inject-1.jar junit-4.10.jar protobuf-java-2.5.0.jar
>
> commons-compress-1.4.1.jar hamcrest-core-1.1.jar
> jersey-core-1.9.jar log4j-1.2.17.jar snappy-java-1.0.4.1.jar
>
>
>
>
>
> I am not sure which all environment varibles should be exported or
> described in the hadoop-env.sh or othere configuration files.
>
>
>
> *Note* :
>
> Pivotal version which I have installed supports *Apache Hadoop 2.2*
>
>
>
> *Remember:*
>
> While replying, please keep gmail id in loop
>
>
>
> Let me know how can I resolve the error.
>
>
>
>
>
> With Regards,
>
> Pratik
>
>
>
> DISCLAIMER ========== This e-mail may contain privileged and confidential
> information which is the property of Persistent Systems Ltd. It is intended
> only for the use of the individual or entity to which it is addressed. If
> you are not the intended recipient, you are not authorized to read, retain,
> copy, print, distribute or use this message. If you have received this
> communication in error, please notify the sender and delete all copies of
> this message. Persistent Systems Ltd. does not accept any liability for
> virus infected mails.
>
> DISCLAIMER ========== This e-mail may contain privileged and
> confidential information which is the property of Persistent Systems Ltd.
> It is intended only for the use of the individual or entity to which it is
> addressed. If you are not the intended recipient, you are not authorized to
> read, retain, copy, print, distribute or use this message. If you have
> received this communication in error, please notify the sender and delete
> all copies of this message. Persistent Systems Ltd. does not accept any
> liability for virus infected mails.
>
Re: Apache Hadoop 2.2 - Job execution error
Posted by Ted Yu <yu...@gmail.com>.
~/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-app/2.2.0/hadoop-mapreduce-client-app-2.2.0.jar
| grep MRAppMaster
...
45581 Mon Oct 07 06:26:42 PDT 2013
org/apache/hadoop/mapreduce/v2/app/MRAppMaster.class
I don't see the above jar in your listing.
Can you find it and put it on the classpath ?
BTW hadoop 2.6.0 was released last year, please consider using the latest
release.
Cheers
On Fri, Jan 2, 2015 at 12:46 PM, Pratik Gadiya <pratik_gadiya@persistent.com
> wrote:
> I didn’t get much help from Pivotal.
>
>
>
> So, I thought to post it on user mailing list
>
>
>
> Thanks,
>
> Pratik
>
> *From:* Ted Yu [mailto:yuzhihong@gmail.com]
> *Sent:* Friday, January 02, 2015 8:20 PM
> *To:* user@hadoop.apache.org
> *Cc:* Pratik Gadiya; pratikgadiya1@gmail.com
> *Subject:* Re: Apache Hadoop 2.2 - Job execution error
>
>
>
> Have you asked this question on Pivotal's mailing list ?
>
>
>
> Cheers
>
>
> On Jan 2, 2015, at 6:37 AM, Pratik Gadiya <pr...@persistent.com>
> wrote:
>
> Hi All,
>
>
>
> I am using Pivotal Distribution to deploy Hadoop cluster.
>
>
>
> For the same, I am following link mentioned below
>
>
> http://pivotalhd.docs.pivotal.io/doc/2100/webhelp/topics/ManuallyInstallingandUsingPivotalHD21Stack.html
>
>
>
>
>
> *Deployment Configuration** : *
>
>
>
> 1) phd1.xyz.com - NameNode, ResourceManager
>
> 2) phd2.xyz.com - DataNode, NodeManager
>
>
>
> I have above mentioned services *UP* and *Running* and also able to
> access the HDFS file system but not able to execute jobs on cluster
>
>
>
> *Stderr file log* :
>
> *Error: Could not find or load main class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster*
>
>
>
> In order to resolve the same problem, I added classpath property tag in
> yarn-site.xml but still I am not able to execute the job, resulting into
> same error as above.
>
>
>
> *yarn-site.xml :*
>
> <property>
>
> <name>yarn.application.classpath</name>
>
>
> <value>/usr/lib/gphd/hadoop/*,/usr/lib/gphd/hadoop/lib/*,/usr/lib/gphd/hadoop-hdfs/*,/usr/lib/gphd/hadoop-hdfs/lib/*,/usr/lib/gphd/hadoop-yarn/*,/usr/lib/gphd/hadoop-yarn/lib/*,/usr/lib/gphd/hadoop-mapreduce/*,/usr/lib/gphd/hadoop-mapreduce/lib/*,/etc/gphd/hadoop/conf/*</value>
>
> </property>
>
>
>
>
>
> [root@phd2 bin]# ls /usr/lib/gphd/hadoop/
>
> bin
> hadoop-annotations.jar
> hadoop-common-2.2.0-gphd-3.1.0.0-tests.jar hadoop-nfs.jar NOTICE.txt
>
> CHANGES.txt
> hadoop-auth-2.2.0-gphd-3.1.0.0.jar
> hadoop-common.jar lib README.txt
>
> etc
> hadoop-auth.jar
> hadoop-common-tests.jar libexec sbin
>
> hadoop-annotations-2.2.0-gphd-3.1.0.0.jar
> hadoop-common-2.2.0-gphd-3.1.0.0.jar
> hadoop-nfs-2.2.0-gphd-3.1.0.0.jar LICENSE.txt
>
>
>
>
>
> [root@phd2 bin]# ls /usr/lib/gphd/hadoop/lib
>
> activation-1.1.jar commons-httpclient-3.1.jar
> jasper-runtime-5.5.23.jar
> jetty-security-7.6.10.v20130312.jar native
>
> asm-3.2.jar commons-io-2.1.jar
> javax.servlet-2.5.0.v201103041518.jar
> jetty-server-7.6.10.v20130312.jar netty-3.6.2.Final.jar
>
> avro-1.7.4.jar commons-lang-2.5.jar
> jaxb-api-2.2.2.jar
> jetty-servlet-7.6.10.v20130312.jar paranamer-2.3.jar
>
> commons-beanutils-1.7.0.jar commons-logging-1.1.1.jar
> jaxb-impl-2.2.3-1.jar
> jetty-util-7.6.10.v20130312.jar protobuf-java-2.5.0.jar
>
> commons-beanutils-core-1.8.0.jar commons-math-2.1.jar
> jersey-core-1.9.jar
> jetty-webapp-7.6.10.v20130312.jar servlet-api-2.5.jar
>
> commons-cli-1.2.jar commons-net-3.1.jar
> jersey-json-1.9.jar
> jetty-xml-7.6.10.v20130312.jar slf4j-api-1.7.5.jar
>
> commons-codec-1.4.jar guava-11.0.2.jar
> jersey-server-1.9.jar
> jsch-0.1.42.jar slf4j-log4j12-1.7.5.jar
>
> commons-collections-3.2.1.jar jackson-core-asl-1.8.8.jar
> jets3t-0.6.1.jar
> jsp-api-2.1.jar snappy-java-1.0.4.1.jar
>
> commons-compress-1.4.1.jar jackson-jaxrs-1.8.8.jar
> jettison-1.1.jar
> jsr305-1.3.9.jar stax-api-1.0.1.jar
>
> commons-configuration-1.6.jar jackson-mapper-asl-1.8.8.jar
> jetty-continuation-7.6.10.v20130312.jar
> junit-4.8.2.jar xmlenc-0.52.jar
>
> commons-digester-1.8.jar jackson-xc-1.8.8.jar
> jetty-http-7.6.10.v20130312.jar
> log4j-1.2.17.jar xz-1.0.jar
>
> commons-el-1.0.jar jasper-compiler-5.5.23.jar
> jetty-io-7.6.10.v20130312.jar
> mockito-all-1.8.5.jar zookeeper.jar
>
>
>
>
>
> [root@phd2 bin]# ls /usr/lib/gphd/hadoop-yarn/lib
>
> aopalliance-1.0.jar commons-io-2.1.jar
> jackson-core-asl-1.8.8.jar jersey-guice-1.9.jar
> netty-3.6.2.Final.jar xz-1.0.jar
>
> asm-3.2.jar guice-3.0.jar
> jackson-mapper-asl-1.8.8.jar jersey-server-1.9.jar paranamer-2.3.jar
>
> avro-1.7.4.jar guice-servlet-3.0.jar
> javax.inject-1.jar junit-4.10.jar protobuf-java-2.5.0.jar
>
> commons-compress-1.4.1.jar hamcrest-core-1.1.jar
> jersey-core-1.9.jar log4j-1.2.17.jar snappy-java-1.0.4.1.jar
>
>
>
>
>
> I am not sure which all environment varibles should be exported or
> described in the hadoop-env.sh or othere configuration files.
>
>
>
> *Note* :
>
> Pivotal version which I have installed supports *Apache Hadoop 2.2*
>
>
>
> *Remember:*
>
> While replying, please keep gmail id in loop
>
>
>
> Let me know how can I resolve the error.
>
>
>
>
>
> With Regards,
>
> Pratik
>
>
>
> DISCLAIMER ========== This e-mail may contain privileged and confidential
> information which is the property of Persistent Systems Ltd. It is intended
> only for the use of the individual or entity to which it is addressed. If
> you are not the intended recipient, you are not authorized to read, retain,
> copy, print, distribute or use this message. If you have received this
> communication in error, please notify the sender and delete all copies of
> this message. Persistent Systems Ltd. does not accept any liability for
> virus infected mails.
>
> DISCLAIMER ========== This e-mail may contain privileged and
> confidential information which is the property of Persistent Systems Ltd.
> It is intended only for the use of the individual or entity to which it is
> addressed. If you are not the intended recipient, you are not authorized to
> read, retain, copy, print, distribute or use this message. If you have
> received this communication in error, please notify the sender and delete
> all copies of this message. Persistent Systems Ltd. does not accept any
> liability for virus infected mails.
>
RE: Apache Hadoop 2.2 - Job execution error
Posted by Pratik Gadiya <pr...@persistent.com>.
I didn’t get much help from Pivotal.
So, I thought to post it on user mailing list
Thanks,
Pratik
From: Ted Yu [mailto:yuzhihong@gmail.com]
Sent: Friday, January 02, 2015 8:20 PM
To: user@hadoop.apache.org
Cc: Pratik Gadiya; pratikgadiya1@gmail.com
Subject: Re: Apache Hadoop 2.2 - Job execution error
Have you asked this question on Pivotal's mailing list ?
Cheers
On Jan 2, 2015, at 6:37 AM, Pratik Gadiya <pr...@persistent.com>> wrote:
Hi All,
I am using Pivotal Distribution to deploy Hadoop cluster.
For the same, I am following link mentioned below
http://pivotalhd.docs.pivotal.io/doc/2100/webhelp/topics/ManuallyInstallingandUsingPivotalHD21Stack.html
Deployment Configuration :
1) phd1.xyz.com<http://phd1.xyz.com> - NameNode, ResourceManager
2) phd2.xyz.com<http://phd2.xyz.com> - DataNode, NodeManager
I have above mentioned services UP and Running and also able to access the HDFS file system but not able to execute jobs on cluster
Stderr file log :
Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster
In order to resolve the same problem, I added classpath property tag in yarn-site.xml but still I am not able to execute the job, resulting into same error as above.
yarn-site.xml :
<property>
<name>yarn.application.classpath</name>
<value>/usr/lib/gphd/hadoop/*,/usr/lib/gphd/hadoop/lib/*,/usr/lib/gphd/hadoop-hdfs/*,/usr/lib/gphd/hadoop-hdfs/lib/*,/usr/lib/gphd/hadoop-yarn/*,/usr/lib/gphd/hadoop-yarn/lib/*,/usr/lib/gphd/hadoop-mapreduce/*,/usr/lib/gphd/hadoop-mapreduce/lib/*,/etc/gphd/hadoop/conf/*</value>
</property>
[root@phd2 bin]# ls /usr/lib/gphd/hadoop/
bin hadoop-annotations.jar hadoop-common-2.2.0-gphd-3.1.0.0-tests.jar hadoop-nfs.jar NOTICE.txt
CHANGES.txt hadoop-auth-2.2.0-gphd-3.1.0.0.jar hadoop-common.jar lib README.txt
etc hadoop-auth.jar hadoop-common-tests.jar libexec sbin
hadoop-annotations-2.2.0-gphd-3.1.0.0.jar hadoop-common-2.2.0-gphd-3.1.0.0.jar hadoop-nfs-2.2.0-gphd-3.1.0.0.jar LICENSE.txt
[root@phd2 bin]# ls /usr/lib/gphd/hadoop/lib
activation-1.1.jar commons-httpclient-3.1.jar jasper-runtime-5.5.23.jar jetty-security-7.6.10.v20130312.jar native
asm-3.2.jar commons-io-2.1.jar javax.servlet-2.5.0.v201103041518.jar jetty-server-7.6.10.v20130312.jar netty-3.6.2.Final.jar
avro-1.7.4.jar commons-lang-2.5.jar jaxb-api-2.2.2.jar jetty-servlet-7.6.10.v20130312.jar paranamer-2.3.jar
commons-beanutils-1.7.0.jar commons-logging-1.1.1.jar jaxb-impl-2.2.3-1.jar jetty-util-7.6.10.v20130312.jar protobuf-java-2.5.0.jar
commons-beanutils-core-1.8.0.jar commons-math-2.1.jar jersey-core-1.9.jar jetty-webapp-7.6.10.v20130312.jar servlet-api-2.5.jar
commons-cli-1.2.jar commons-net-3.1.jar jersey-json-1.9.jar jetty-xml-7.6.10.v20130312.jar slf4j-api-1.7.5.jar
commons-codec-1.4.jar guava-11.0.2.jar jersey-server-1.9.jar jsch-0.1.42.jar slf4j-log4j12-1.7.5.jar
commons-collections-3.2.1.jar jackson-core-asl-1.8.8.jar jets3t-0.6.1.jar jsp-api-2.1.jar snappy-java-1.0.4.1.jar
commons-compress-1.4.1.jar jackson-jaxrs-1.8.8.jar jettison-1.1.jar jsr305-1.3.9.jar stax-api-1.0.1.jar
commons-configuration-1.6.jar jackson-mapper-asl-1.8.8.jar jetty-continuation-7.6.10.v20130312.jar junit-4.8.2.jar xmlenc-0.52.jar
commons-digester-1.8.jar jackson-xc-1.8.8.jar jetty-http-7.6.10.v20130312.jar log4j-1.2.17.jar xz-1.0.jar
commons-el-1.0.jar jasper-compiler-5.5.23.jar jetty-io-7.6.10.v20130312.jar mockito-all-1.8.5.jar zookeeper.jar
[root@phd2 bin]# ls /usr/lib/gphd/hadoop-yarn/lib
aopalliance-1.0.jar commons-io-2.1.jar jackson-core-asl-1.8.8.jar jersey-guice-1.9.jar netty-3.6.2.Final.jar xz-1.0.jar
asm-3.2.jar guice-3.0.jar jackson-mapper-asl-1.8.8.jar jersey-server-1.9.jar paranamer-2.3.jar
avro-1.7.4.jar guice-servlet-3.0.jar javax.inject-1.jar junit-4.10.jar protobuf-java-2.5.0.jar
commons-compress-1.4.1.jar hamcrest-core-1.1.jar jersey-core-1.9.jar log4j-1.2.17.jar snappy-java-1.0.4.1.jar
I am not sure which all environment varibles should be exported or described in the hadoop-env.sh or othere configuration files.
Note :
Pivotal version which I have installed supports Apache Hadoop 2.2
Remember:
While replying, please keep gmail id in loop
Let me know how can I resolve the error.
With Regards,
Pratik
DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
RE: Apache Hadoop 2.2 - Job execution error
Posted by Pratik Gadiya <pr...@persistent.com>.
I didn’t get much help from Pivotal.
So, I thought to post it on user mailing list
Thanks,
Pratik
From: Ted Yu [mailto:yuzhihong@gmail.com]
Sent: Friday, January 02, 2015 8:20 PM
To: user@hadoop.apache.org
Cc: Pratik Gadiya; pratikgadiya1@gmail.com
Subject: Re: Apache Hadoop 2.2 - Job execution error
Have you asked this question on Pivotal's mailing list ?
Cheers
On Jan 2, 2015, at 6:37 AM, Pratik Gadiya <pr...@persistent.com>> wrote:
Hi All,
I am using Pivotal Distribution to deploy Hadoop cluster.
For the same, I am following link mentioned below
http://pivotalhd.docs.pivotal.io/doc/2100/webhelp/topics/ManuallyInstallingandUsingPivotalHD21Stack.html
Deployment Configuration :
1) phd1.xyz.com<http://phd1.xyz.com> - NameNode, ResourceManager
2) phd2.xyz.com<http://phd2.xyz.com> - DataNode, NodeManager
I have above mentioned services UP and Running and also able to access the HDFS file system but not able to execute jobs on cluster
Stderr file log :
Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster
In order to resolve the same problem, I added classpath property tag in yarn-site.xml but still I am not able to execute the job, resulting into same error as above.
yarn-site.xml :
<property>
<name>yarn.application.classpath</name>
<value>/usr/lib/gphd/hadoop/*,/usr/lib/gphd/hadoop/lib/*,/usr/lib/gphd/hadoop-hdfs/*,/usr/lib/gphd/hadoop-hdfs/lib/*,/usr/lib/gphd/hadoop-yarn/*,/usr/lib/gphd/hadoop-yarn/lib/*,/usr/lib/gphd/hadoop-mapreduce/*,/usr/lib/gphd/hadoop-mapreduce/lib/*,/etc/gphd/hadoop/conf/*</value>
</property>
[root@phd2 bin]# ls /usr/lib/gphd/hadoop/
bin hadoop-annotations.jar hadoop-common-2.2.0-gphd-3.1.0.0-tests.jar hadoop-nfs.jar NOTICE.txt
CHANGES.txt hadoop-auth-2.2.0-gphd-3.1.0.0.jar hadoop-common.jar lib README.txt
etc hadoop-auth.jar hadoop-common-tests.jar libexec sbin
hadoop-annotations-2.2.0-gphd-3.1.0.0.jar hadoop-common-2.2.0-gphd-3.1.0.0.jar hadoop-nfs-2.2.0-gphd-3.1.0.0.jar LICENSE.txt
[root@phd2 bin]# ls /usr/lib/gphd/hadoop/lib
activation-1.1.jar commons-httpclient-3.1.jar jasper-runtime-5.5.23.jar jetty-security-7.6.10.v20130312.jar native
asm-3.2.jar commons-io-2.1.jar javax.servlet-2.5.0.v201103041518.jar jetty-server-7.6.10.v20130312.jar netty-3.6.2.Final.jar
avro-1.7.4.jar commons-lang-2.5.jar jaxb-api-2.2.2.jar jetty-servlet-7.6.10.v20130312.jar paranamer-2.3.jar
commons-beanutils-1.7.0.jar commons-logging-1.1.1.jar jaxb-impl-2.2.3-1.jar jetty-util-7.6.10.v20130312.jar protobuf-java-2.5.0.jar
commons-beanutils-core-1.8.0.jar commons-math-2.1.jar jersey-core-1.9.jar jetty-webapp-7.6.10.v20130312.jar servlet-api-2.5.jar
commons-cli-1.2.jar commons-net-3.1.jar jersey-json-1.9.jar jetty-xml-7.6.10.v20130312.jar slf4j-api-1.7.5.jar
commons-codec-1.4.jar guava-11.0.2.jar jersey-server-1.9.jar jsch-0.1.42.jar slf4j-log4j12-1.7.5.jar
commons-collections-3.2.1.jar jackson-core-asl-1.8.8.jar jets3t-0.6.1.jar jsp-api-2.1.jar snappy-java-1.0.4.1.jar
commons-compress-1.4.1.jar jackson-jaxrs-1.8.8.jar jettison-1.1.jar jsr305-1.3.9.jar stax-api-1.0.1.jar
commons-configuration-1.6.jar jackson-mapper-asl-1.8.8.jar jetty-continuation-7.6.10.v20130312.jar junit-4.8.2.jar xmlenc-0.52.jar
commons-digester-1.8.jar jackson-xc-1.8.8.jar jetty-http-7.6.10.v20130312.jar log4j-1.2.17.jar xz-1.0.jar
commons-el-1.0.jar jasper-compiler-5.5.23.jar jetty-io-7.6.10.v20130312.jar mockito-all-1.8.5.jar zookeeper.jar
[root@phd2 bin]# ls /usr/lib/gphd/hadoop-yarn/lib
aopalliance-1.0.jar commons-io-2.1.jar jackson-core-asl-1.8.8.jar jersey-guice-1.9.jar netty-3.6.2.Final.jar xz-1.0.jar
asm-3.2.jar guice-3.0.jar jackson-mapper-asl-1.8.8.jar jersey-server-1.9.jar paranamer-2.3.jar
avro-1.7.4.jar guice-servlet-3.0.jar javax.inject-1.jar junit-4.10.jar protobuf-java-2.5.0.jar
commons-compress-1.4.1.jar hamcrest-core-1.1.jar jersey-core-1.9.jar log4j-1.2.17.jar snappy-java-1.0.4.1.jar
I am not sure which all environment varibles should be exported or described in the hadoop-env.sh or othere configuration files.
Note :
Pivotal version which I have installed supports Apache Hadoop 2.2
Remember:
While replying, please keep gmail id in loop
Let me know how can I resolve the error.
With Regards,
Pratik
DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
RE: Apache Hadoop 2.2 - Job execution error
Posted by Pratik Gadiya <pr...@persistent.com>.
I didn’t get much help from Pivotal.
So, I thought to post it on user mailing list
Thanks,
Pratik
From: Ted Yu [mailto:yuzhihong@gmail.com]
Sent: Friday, January 02, 2015 8:20 PM
To: user@hadoop.apache.org
Cc: Pratik Gadiya; pratikgadiya1@gmail.com
Subject: Re: Apache Hadoop 2.2 - Job execution error
Have you asked this question on Pivotal's mailing list ?
Cheers
On Jan 2, 2015, at 6:37 AM, Pratik Gadiya <pr...@persistent.com>> wrote:
Hi All,
I am using Pivotal Distribution to deploy Hadoop cluster.
For the same, I am following link mentioned below
http://pivotalhd.docs.pivotal.io/doc/2100/webhelp/topics/ManuallyInstallingandUsingPivotalHD21Stack.html
Deployment Configuration :
1) phd1.xyz.com<http://phd1.xyz.com> - NameNode, ResourceManager
2) phd2.xyz.com<http://phd2.xyz.com> - DataNode, NodeManager
I have above mentioned services UP and Running and also able to access the HDFS file system but not able to execute jobs on cluster
Stderr file log :
Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster
In order to resolve the same problem, I added classpath property tag in yarn-site.xml but still I am not able to execute the job, resulting into same error as above.
yarn-site.xml :
<property>
<name>yarn.application.classpath</name>
<value>/usr/lib/gphd/hadoop/*,/usr/lib/gphd/hadoop/lib/*,/usr/lib/gphd/hadoop-hdfs/*,/usr/lib/gphd/hadoop-hdfs/lib/*,/usr/lib/gphd/hadoop-yarn/*,/usr/lib/gphd/hadoop-yarn/lib/*,/usr/lib/gphd/hadoop-mapreduce/*,/usr/lib/gphd/hadoop-mapreduce/lib/*,/etc/gphd/hadoop/conf/*</value>
</property>
[root@phd2 bin]# ls /usr/lib/gphd/hadoop/
bin hadoop-annotations.jar hadoop-common-2.2.0-gphd-3.1.0.0-tests.jar hadoop-nfs.jar NOTICE.txt
CHANGES.txt hadoop-auth-2.2.0-gphd-3.1.0.0.jar hadoop-common.jar lib README.txt
etc hadoop-auth.jar hadoop-common-tests.jar libexec sbin
hadoop-annotations-2.2.0-gphd-3.1.0.0.jar hadoop-common-2.2.0-gphd-3.1.0.0.jar hadoop-nfs-2.2.0-gphd-3.1.0.0.jar LICENSE.txt
[root@phd2 bin]# ls /usr/lib/gphd/hadoop/lib
activation-1.1.jar commons-httpclient-3.1.jar jasper-runtime-5.5.23.jar jetty-security-7.6.10.v20130312.jar native
asm-3.2.jar commons-io-2.1.jar javax.servlet-2.5.0.v201103041518.jar jetty-server-7.6.10.v20130312.jar netty-3.6.2.Final.jar
avro-1.7.4.jar commons-lang-2.5.jar jaxb-api-2.2.2.jar jetty-servlet-7.6.10.v20130312.jar paranamer-2.3.jar
commons-beanutils-1.7.0.jar commons-logging-1.1.1.jar jaxb-impl-2.2.3-1.jar jetty-util-7.6.10.v20130312.jar protobuf-java-2.5.0.jar
commons-beanutils-core-1.8.0.jar commons-math-2.1.jar jersey-core-1.9.jar jetty-webapp-7.6.10.v20130312.jar servlet-api-2.5.jar
commons-cli-1.2.jar commons-net-3.1.jar jersey-json-1.9.jar jetty-xml-7.6.10.v20130312.jar slf4j-api-1.7.5.jar
commons-codec-1.4.jar guava-11.0.2.jar jersey-server-1.9.jar jsch-0.1.42.jar slf4j-log4j12-1.7.5.jar
commons-collections-3.2.1.jar jackson-core-asl-1.8.8.jar jets3t-0.6.1.jar jsp-api-2.1.jar snappy-java-1.0.4.1.jar
commons-compress-1.4.1.jar jackson-jaxrs-1.8.8.jar jettison-1.1.jar jsr305-1.3.9.jar stax-api-1.0.1.jar
commons-configuration-1.6.jar jackson-mapper-asl-1.8.8.jar jetty-continuation-7.6.10.v20130312.jar junit-4.8.2.jar xmlenc-0.52.jar
commons-digester-1.8.jar jackson-xc-1.8.8.jar jetty-http-7.6.10.v20130312.jar log4j-1.2.17.jar xz-1.0.jar
commons-el-1.0.jar jasper-compiler-5.5.23.jar jetty-io-7.6.10.v20130312.jar mockito-all-1.8.5.jar zookeeper.jar
[root@phd2 bin]# ls /usr/lib/gphd/hadoop-yarn/lib
aopalliance-1.0.jar commons-io-2.1.jar jackson-core-asl-1.8.8.jar jersey-guice-1.9.jar netty-3.6.2.Final.jar xz-1.0.jar
asm-3.2.jar guice-3.0.jar jackson-mapper-asl-1.8.8.jar jersey-server-1.9.jar paranamer-2.3.jar
avro-1.7.4.jar guice-servlet-3.0.jar javax.inject-1.jar junit-4.10.jar protobuf-java-2.5.0.jar
commons-compress-1.4.1.jar hamcrest-core-1.1.jar jersey-core-1.9.jar log4j-1.2.17.jar snappy-java-1.0.4.1.jar
I am not sure which all environment varibles should be exported or described in the hadoop-env.sh or othere configuration files.
Note :
Pivotal version which I have installed supports Apache Hadoop 2.2
Remember:
While replying, please keep gmail id in loop
Let me know how can I resolve the error.
With Regards,
Pratik
DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
RE: Apache Hadoop 2.2 - Job execution error
Posted by Pratik Gadiya <pr...@persistent.com>.
I didn’t get much help from Pivotal.
So, I thought to post it on user mailing list
Thanks,
Pratik
From: Ted Yu [mailto:yuzhihong@gmail.com]
Sent: Friday, January 02, 2015 8:20 PM
To: user@hadoop.apache.org
Cc: Pratik Gadiya; pratikgadiya1@gmail.com
Subject: Re: Apache Hadoop 2.2 - Job execution error
Have you asked this question on Pivotal's mailing list ?
Cheers
On Jan 2, 2015, at 6:37 AM, Pratik Gadiya <pr...@persistent.com>> wrote:
Hi All,
I am using Pivotal Distribution to deploy Hadoop cluster.
For the same, I am following link mentioned below
http://pivotalhd.docs.pivotal.io/doc/2100/webhelp/topics/ManuallyInstallingandUsingPivotalHD21Stack.html
Deployment Configuration :
1) phd1.xyz.com<http://phd1.xyz.com> - NameNode, ResourceManager
2) phd2.xyz.com<http://phd2.xyz.com> - DataNode, NodeManager
I have above mentioned services UP and Running and also able to access the HDFS file system but not able to execute jobs on cluster
Stderr file log :
Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster
In order to resolve the same problem, I added classpath property tag in yarn-site.xml but still I am not able to execute the job, resulting into same error as above.
yarn-site.xml :
<property>
<name>yarn.application.classpath</name>
<value>/usr/lib/gphd/hadoop/*,/usr/lib/gphd/hadoop/lib/*,/usr/lib/gphd/hadoop-hdfs/*,/usr/lib/gphd/hadoop-hdfs/lib/*,/usr/lib/gphd/hadoop-yarn/*,/usr/lib/gphd/hadoop-yarn/lib/*,/usr/lib/gphd/hadoop-mapreduce/*,/usr/lib/gphd/hadoop-mapreduce/lib/*,/etc/gphd/hadoop/conf/*</value>
</property>
[root@phd2 bin]# ls /usr/lib/gphd/hadoop/
bin hadoop-annotations.jar hadoop-common-2.2.0-gphd-3.1.0.0-tests.jar hadoop-nfs.jar NOTICE.txt
CHANGES.txt hadoop-auth-2.2.0-gphd-3.1.0.0.jar hadoop-common.jar lib README.txt
etc hadoop-auth.jar hadoop-common-tests.jar libexec sbin
hadoop-annotations-2.2.0-gphd-3.1.0.0.jar hadoop-common-2.2.0-gphd-3.1.0.0.jar hadoop-nfs-2.2.0-gphd-3.1.0.0.jar LICENSE.txt
[root@phd2 bin]# ls /usr/lib/gphd/hadoop/lib
activation-1.1.jar commons-httpclient-3.1.jar jasper-runtime-5.5.23.jar jetty-security-7.6.10.v20130312.jar native
asm-3.2.jar commons-io-2.1.jar javax.servlet-2.5.0.v201103041518.jar jetty-server-7.6.10.v20130312.jar netty-3.6.2.Final.jar
avro-1.7.4.jar commons-lang-2.5.jar jaxb-api-2.2.2.jar jetty-servlet-7.6.10.v20130312.jar paranamer-2.3.jar
commons-beanutils-1.7.0.jar commons-logging-1.1.1.jar jaxb-impl-2.2.3-1.jar jetty-util-7.6.10.v20130312.jar protobuf-java-2.5.0.jar
commons-beanutils-core-1.8.0.jar commons-math-2.1.jar jersey-core-1.9.jar jetty-webapp-7.6.10.v20130312.jar servlet-api-2.5.jar
commons-cli-1.2.jar commons-net-3.1.jar jersey-json-1.9.jar jetty-xml-7.6.10.v20130312.jar slf4j-api-1.7.5.jar
commons-codec-1.4.jar guava-11.0.2.jar jersey-server-1.9.jar jsch-0.1.42.jar slf4j-log4j12-1.7.5.jar
commons-collections-3.2.1.jar jackson-core-asl-1.8.8.jar jets3t-0.6.1.jar jsp-api-2.1.jar snappy-java-1.0.4.1.jar
commons-compress-1.4.1.jar jackson-jaxrs-1.8.8.jar jettison-1.1.jar jsr305-1.3.9.jar stax-api-1.0.1.jar
commons-configuration-1.6.jar jackson-mapper-asl-1.8.8.jar jetty-continuation-7.6.10.v20130312.jar junit-4.8.2.jar xmlenc-0.52.jar
commons-digester-1.8.jar jackson-xc-1.8.8.jar jetty-http-7.6.10.v20130312.jar log4j-1.2.17.jar xz-1.0.jar
commons-el-1.0.jar jasper-compiler-5.5.23.jar jetty-io-7.6.10.v20130312.jar mockito-all-1.8.5.jar zookeeper.jar
[root@phd2 bin]# ls /usr/lib/gphd/hadoop-yarn/lib
aopalliance-1.0.jar commons-io-2.1.jar jackson-core-asl-1.8.8.jar jersey-guice-1.9.jar netty-3.6.2.Final.jar xz-1.0.jar
asm-3.2.jar guice-3.0.jar jackson-mapper-asl-1.8.8.jar jersey-server-1.9.jar paranamer-2.3.jar
avro-1.7.4.jar guice-servlet-3.0.jar javax.inject-1.jar junit-4.10.jar protobuf-java-2.5.0.jar
commons-compress-1.4.1.jar hamcrest-core-1.1.jar jersey-core-1.9.jar log4j-1.2.17.jar snappy-java-1.0.4.1.jar
I am not sure which all environment varibles should be exported or described in the hadoop-env.sh or othere configuration files.
Note :
Pivotal version which I have installed supports Apache Hadoop 2.2
Remember:
While replying, please keep gmail id in loop
Let me know how can I resolve the error.
With Regards,
Pratik
DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
Re: Apache Hadoop 2.2 - Job execution error
Posted by Ted Yu <yu...@gmail.com>.
Have you asked this question on Pivotal's mailing list ?
Cheers
> On Jan 2, 2015, at 6:37 AM, Pratik Gadiya <pr...@persistent.com> wrote:
>
> Hi All,
>
> I am using Pivotal Distribution to deploy Hadoop cluster.
>
> For the same, I am following link mentioned below
> http://pivotalhd.docs.pivotal.io/doc/2100/webhelp/topics/ManuallyInstallingandUsingPivotalHD21Stack.html
>
>
> Deployment Configuration :
>
> 1) phd1.xyz.com - NameNode, ResourceManager
> 2) phd2.xyz.com - DataNode, NodeManager
>
> I have above mentioned services UP and Running and also able to access the HDFS file system but not able to execute jobs on cluster
>
> Stderr file log :
> Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster
>
> In order to resolve the same problem, I added classpath property tag in yarn-site.xml but still I am not able to execute the job, resulting into same error as above.
>
> yarn-site.xml :
> <property>
> <name>yarn.application.classpath</name>
> <value>/usr/lib/gphd/hadoop/*,/usr/lib/gphd/hadoop/lib/*,/usr/lib/gphd/hadoop-hdfs/*,/usr/lib/gphd/hadoop-hdfs/lib/*,/usr/lib/gphd/hadoop-yarn/*,/usr/lib/gphd/hadoop-yarn/lib/*,/usr/lib/gphd/hadoop-mapreduce/*,/usr/lib/gphd/hadoop-mapreduce/lib/*,/etc/gphd/hadoop/conf/*</value>
> </property>
>
>
> [root@phd2 bin]# ls /usr/lib/gphd/hadoop/
> bin hadoop-annotations.jar hadoop-common-2.2.0-gphd-3.1.0.0-tests.jar hadoop-nfs.jar NOTICE.txt
> CHANGES.txt hadoop-auth-2.2.0-gphd-3.1.0.0.jar hadoop-common.jar lib README.txt
> etc hadoop-auth.jar hadoop-common-tests.jar libexec sbin
> hadoop-annotations-2.2.0-gphd-3.1.0.0.jar hadoop-common-2.2.0-gphd-3.1.0.0.jar hadoop-nfs-2.2.0-gphd-3.1.0.0.jar LICENSE.txt
>
>
> [root@phd2 bin]# ls /usr/lib/gphd/hadoop/lib
> activation-1.1.jar commons-httpclient-3.1.jar jasper-runtime-5.5.23.jar jetty-security-7.6.10.v20130312.jar native
> asm-3.2.jar commons-io-2.1.jar javax.servlet-2.5.0.v201103041518.jar jetty-server-7.6.10.v20130312.jar netty-3.6.2.Final.jar
> avro-1.7.4.jar commons-lang-2.5.jar jaxb-api-2.2.2.jar jetty-servlet-7.6.10.v20130312.jar paranamer-2.3.jar
> commons-beanutils-1.7.0.jar commons-logging-1.1.1.jar jaxb-impl-2.2.3-1.jar jetty-util-7.6.10.v20130312.jar protobuf-java-2.5.0.jar
> commons-beanutils-core-1.8.0.jar commons-math-2.1.jar jersey-core-1.9.jar jetty-webapp-7.6.10.v20130312.jar servlet-api-2.5.jar
> commons-cli-1.2.jar commons-net-3.1.jar jersey-json-1.9.jar jetty-xml-7.6.10.v20130312.jar slf4j-api-1.7.5.jar
> commons-codec-1.4.jar guava-11.0.2.jar jersey-server-1.9.jar jsch-0.1.42.jar slf4j-log4j12-1.7.5.jar
> commons-collections-3.2.1.jar jackson-core-asl-1.8.8.jar jets3t-0.6.1.jar jsp-api-2.1.jar snappy-java-1.0.4.1.jar
> commons-compress-1.4.1.jar jackson-jaxrs-1.8.8.jar jettison-1.1.jar jsr305-1.3.9.jar stax-api-1.0.1.jar
> commons-configuration-1.6.jar jackson-mapper-asl-1.8.8.jar jetty-continuation-7.6.10.v20130312.jar junit-4.8.2.jar xmlenc-0.52.jar
> commons-digester-1.8.jar jackson-xc-1.8.8.jar jetty-http-7.6.10.v20130312.jar log4j-1.2.17.jar xz-1.0.jar
> commons-el-1.0.jar jasper-compiler-5.5.23.jar jetty-io-7.6.10.v20130312.jar mockito-all-1.8.5.jar zookeeper.jar
>
>
> [root@phd2 bin]# ls /usr/lib/gphd/hadoop-yarn/lib
> aopalliance-1.0.jar commons-io-2.1.jar jackson-core-asl-1.8.8.jar jersey-guice-1.9.jar netty-3.6.2.Final.jar xz-1.0.jar
> asm-3.2.jar guice-3.0.jar jackson-mapper-asl-1.8.8.jar jersey-server-1.9.jar paranamer-2.3.jar
> avro-1.7.4.jar guice-servlet-3.0.jar javax.inject-1.jar junit-4.10.jar protobuf-java-2.5.0.jar
> commons-compress-1.4.1.jar hamcrest-core-1.1.jar jersey-core-1.9.jar log4j-1.2.17.jar snappy-java-1.0.4.1.jar
>
>
> I am not sure which all environment varibles should be exported or described in the hadoop-env.sh or othere configuration files.
>
> Note :
> Pivotal version which I have installed supports Apache Hadoop 2.2
>
> Remember:
> While replying, please keep gmail id in loop
>
> Let me know how can I resolve the error.
>
>
> With Regards,
> Pratik
>
> DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
Re: Apache Hadoop 2.2 - Job execution error
Posted by Ted Yu <yu...@gmail.com>.
Have you asked this question on Pivotal's mailing list ?
Cheers
> On Jan 2, 2015, at 6:37 AM, Pratik Gadiya <pr...@persistent.com> wrote:
>
> Hi All,
>
> I am using Pivotal Distribution to deploy Hadoop cluster.
>
> For the same, I am following link mentioned below
> http://pivotalhd.docs.pivotal.io/doc/2100/webhelp/topics/ManuallyInstallingandUsingPivotalHD21Stack.html
>
>
> Deployment Configuration :
>
> 1) phd1.xyz.com - NameNode, ResourceManager
> 2) phd2.xyz.com - DataNode, NodeManager
>
> I have above mentioned services UP and Running and also able to access the HDFS file system but not able to execute jobs on cluster
>
> Stderr file log :
> Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster
>
> In order to resolve the same problem, I added classpath property tag in yarn-site.xml but still I am not able to execute the job, resulting into same error as above.
>
> yarn-site.xml :
> <property>
> <name>yarn.application.classpath</name>
> <value>/usr/lib/gphd/hadoop/*,/usr/lib/gphd/hadoop/lib/*,/usr/lib/gphd/hadoop-hdfs/*,/usr/lib/gphd/hadoop-hdfs/lib/*,/usr/lib/gphd/hadoop-yarn/*,/usr/lib/gphd/hadoop-yarn/lib/*,/usr/lib/gphd/hadoop-mapreduce/*,/usr/lib/gphd/hadoop-mapreduce/lib/*,/etc/gphd/hadoop/conf/*</value>
> </property>
>
>
> [root@phd2 bin]# ls /usr/lib/gphd/hadoop/
> bin hadoop-annotations.jar hadoop-common-2.2.0-gphd-3.1.0.0-tests.jar hadoop-nfs.jar NOTICE.txt
> CHANGES.txt hadoop-auth-2.2.0-gphd-3.1.0.0.jar hadoop-common.jar lib README.txt
> etc hadoop-auth.jar hadoop-common-tests.jar libexec sbin
> hadoop-annotations-2.2.0-gphd-3.1.0.0.jar hadoop-common-2.2.0-gphd-3.1.0.0.jar hadoop-nfs-2.2.0-gphd-3.1.0.0.jar LICENSE.txt
>
>
> [root@phd2 bin]# ls /usr/lib/gphd/hadoop/lib
> activation-1.1.jar commons-httpclient-3.1.jar jasper-runtime-5.5.23.jar jetty-security-7.6.10.v20130312.jar native
> asm-3.2.jar commons-io-2.1.jar javax.servlet-2.5.0.v201103041518.jar jetty-server-7.6.10.v20130312.jar netty-3.6.2.Final.jar
> avro-1.7.4.jar commons-lang-2.5.jar jaxb-api-2.2.2.jar jetty-servlet-7.6.10.v20130312.jar paranamer-2.3.jar
> commons-beanutils-1.7.0.jar commons-logging-1.1.1.jar jaxb-impl-2.2.3-1.jar jetty-util-7.6.10.v20130312.jar protobuf-java-2.5.0.jar
> commons-beanutils-core-1.8.0.jar commons-math-2.1.jar jersey-core-1.9.jar jetty-webapp-7.6.10.v20130312.jar servlet-api-2.5.jar
> commons-cli-1.2.jar commons-net-3.1.jar jersey-json-1.9.jar jetty-xml-7.6.10.v20130312.jar slf4j-api-1.7.5.jar
> commons-codec-1.4.jar guava-11.0.2.jar jersey-server-1.9.jar jsch-0.1.42.jar slf4j-log4j12-1.7.5.jar
> commons-collections-3.2.1.jar jackson-core-asl-1.8.8.jar jets3t-0.6.1.jar jsp-api-2.1.jar snappy-java-1.0.4.1.jar
> commons-compress-1.4.1.jar jackson-jaxrs-1.8.8.jar jettison-1.1.jar jsr305-1.3.9.jar stax-api-1.0.1.jar
> commons-configuration-1.6.jar jackson-mapper-asl-1.8.8.jar jetty-continuation-7.6.10.v20130312.jar junit-4.8.2.jar xmlenc-0.52.jar
> commons-digester-1.8.jar jackson-xc-1.8.8.jar jetty-http-7.6.10.v20130312.jar log4j-1.2.17.jar xz-1.0.jar
> commons-el-1.0.jar jasper-compiler-5.5.23.jar jetty-io-7.6.10.v20130312.jar mockito-all-1.8.5.jar zookeeper.jar
>
>
> [root@phd2 bin]# ls /usr/lib/gphd/hadoop-yarn/lib
> aopalliance-1.0.jar commons-io-2.1.jar jackson-core-asl-1.8.8.jar jersey-guice-1.9.jar netty-3.6.2.Final.jar xz-1.0.jar
> asm-3.2.jar guice-3.0.jar jackson-mapper-asl-1.8.8.jar jersey-server-1.9.jar paranamer-2.3.jar
> avro-1.7.4.jar guice-servlet-3.0.jar javax.inject-1.jar junit-4.10.jar protobuf-java-2.5.0.jar
> commons-compress-1.4.1.jar hamcrest-core-1.1.jar jersey-core-1.9.jar log4j-1.2.17.jar snappy-java-1.0.4.1.jar
>
>
> I am not sure which all environment varibles should be exported or described in the hadoop-env.sh or othere configuration files.
>
> Note :
> Pivotal version which I have installed supports Apache Hadoop 2.2
>
> Remember:
> While replying, please keep gmail id in loop
>
> Let me know how can I resolve the error.
>
>
> With Regards,
> Pratik
>
> DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
Re: Apache Hadoop 2.2 - Job execution error
Posted by Ted Yu <yu...@gmail.com>.
Have you asked this question on Pivotal's mailing list ?
Cheers
> On Jan 2, 2015, at 6:37 AM, Pratik Gadiya <pr...@persistent.com> wrote:
>
> Hi All,
>
> I am using Pivotal Distribution to deploy Hadoop cluster.
>
> For the same, I am following link mentioned below
> http://pivotalhd.docs.pivotal.io/doc/2100/webhelp/topics/ManuallyInstallingandUsingPivotalHD21Stack.html
>
>
> Deployment Configuration :
>
> 1) phd1.xyz.com - NameNode, ResourceManager
> 2) phd2.xyz.com - DataNode, NodeManager
>
> I have above mentioned services UP and Running and also able to access the HDFS file system but not able to execute jobs on cluster
>
> Stderr file log :
> Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster
>
> In order to resolve the same problem, I added classpath property tag in yarn-site.xml but still I am not able to execute the job, resulting into same error as above.
>
> yarn-site.xml :
> <property>
> <name>yarn.application.classpath</name>
> <value>/usr/lib/gphd/hadoop/*,/usr/lib/gphd/hadoop/lib/*,/usr/lib/gphd/hadoop-hdfs/*,/usr/lib/gphd/hadoop-hdfs/lib/*,/usr/lib/gphd/hadoop-yarn/*,/usr/lib/gphd/hadoop-yarn/lib/*,/usr/lib/gphd/hadoop-mapreduce/*,/usr/lib/gphd/hadoop-mapreduce/lib/*,/etc/gphd/hadoop/conf/*</value>
> </property>
>
>
> [root@phd2 bin]# ls /usr/lib/gphd/hadoop/
> bin hadoop-annotations.jar hadoop-common-2.2.0-gphd-3.1.0.0-tests.jar hadoop-nfs.jar NOTICE.txt
> CHANGES.txt hadoop-auth-2.2.0-gphd-3.1.0.0.jar hadoop-common.jar lib README.txt
> etc hadoop-auth.jar hadoop-common-tests.jar libexec sbin
> hadoop-annotations-2.2.0-gphd-3.1.0.0.jar hadoop-common-2.2.0-gphd-3.1.0.0.jar hadoop-nfs-2.2.0-gphd-3.1.0.0.jar LICENSE.txt
>
>
> [root@phd2 bin]# ls /usr/lib/gphd/hadoop/lib
> activation-1.1.jar commons-httpclient-3.1.jar jasper-runtime-5.5.23.jar jetty-security-7.6.10.v20130312.jar native
> asm-3.2.jar commons-io-2.1.jar javax.servlet-2.5.0.v201103041518.jar jetty-server-7.6.10.v20130312.jar netty-3.6.2.Final.jar
> avro-1.7.4.jar commons-lang-2.5.jar jaxb-api-2.2.2.jar jetty-servlet-7.6.10.v20130312.jar paranamer-2.3.jar
> commons-beanutils-1.7.0.jar commons-logging-1.1.1.jar jaxb-impl-2.2.3-1.jar jetty-util-7.6.10.v20130312.jar protobuf-java-2.5.0.jar
> commons-beanutils-core-1.8.0.jar commons-math-2.1.jar jersey-core-1.9.jar jetty-webapp-7.6.10.v20130312.jar servlet-api-2.5.jar
> commons-cli-1.2.jar commons-net-3.1.jar jersey-json-1.9.jar jetty-xml-7.6.10.v20130312.jar slf4j-api-1.7.5.jar
> commons-codec-1.4.jar guava-11.0.2.jar jersey-server-1.9.jar jsch-0.1.42.jar slf4j-log4j12-1.7.5.jar
> commons-collections-3.2.1.jar jackson-core-asl-1.8.8.jar jets3t-0.6.1.jar jsp-api-2.1.jar snappy-java-1.0.4.1.jar
> commons-compress-1.4.1.jar jackson-jaxrs-1.8.8.jar jettison-1.1.jar jsr305-1.3.9.jar stax-api-1.0.1.jar
> commons-configuration-1.6.jar jackson-mapper-asl-1.8.8.jar jetty-continuation-7.6.10.v20130312.jar junit-4.8.2.jar xmlenc-0.52.jar
> commons-digester-1.8.jar jackson-xc-1.8.8.jar jetty-http-7.6.10.v20130312.jar log4j-1.2.17.jar xz-1.0.jar
> commons-el-1.0.jar jasper-compiler-5.5.23.jar jetty-io-7.6.10.v20130312.jar mockito-all-1.8.5.jar zookeeper.jar
>
>
> [root@phd2 bin]# ls /usr/lib/gphd/hadoop-yarn/lib
> aopalliance-1.0.jar commons-io-2.1.jar jackson-core-asl-1.8.8.jar jersey-guice-1.9.jar netty-3.6.2.Final.jar xz-1.0.jar
> asm-3.2.jar guice-3.0.jar jackson-mapper-asl-1.8.8.jar jersey-server-1.9.jar paranamer-2.3.jar
> avro-1.7.4.jar guice-servlet-3.0.jar javax.inject-1.jar junit-4.10.jar protobuf-java-2.5.0.jar
> commons-compress-1.4.1.jar hamcrest-core-1.1.jar jersey-core-1.9.jar log4j-1.2.17.jar snappy-java-1.0.4.1.jar
>
>
> I am not sure which all environment varibles should be exported or described in the hadoop-env.sh or othere configuration files.
>
> Note :
> Pivotal version which I have installed supports Apache Hadoop 2.2
>
> Remember:
> While replying, please keep gmail id in loop
>
> Let me know how can I resolve the error.
>
>
> With Regards,
> Pratik
>
> DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
Re: Apache Hadoop 2.2 - Job execution error
Posted by Ted Yu <yu...@gmail.com>.
Have you asked this question on Pivotal's mailing list ?
Cheers
> On Jan 2, 2015, at 6:37 AM, Pratik Gadiya <pr...@persistent.com> wrote:
>
> Hi All,
>
> I am using Pivotal Distribution to deploy Hadoop cluster.
>
> For the same, I am following link mentioned below
> http://pivotalhd.docs.pivotal.io/doc/2100/webhelp/topics/ManuallyInstallingandUsingPivotalHD21Stack.html
>
>
> Deployment Configuration :
>
> 1) phd1.xyz.com - NameNode, ResourceManager
> 2) phd2.xyz.com - DataNode, NodeManager
>
> I have above mentioned services UP and Running and also able to access the HDFS file system but not able to execute jobs on cluster
>
> Stderr file log :
> Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster
>
> In order to resolve the same problem, I added classpath property tag in yarn-site.xml but still I am not able to execute the job, resulting into same error as above.
>
> yarn-site.xml :
> <property>
> <name>yarn.application.classpath</name>
> <value>/usr/lib/gphd/hadoop/*,/usr/lib/gphd/hadoop/lib/*,/usr/lib/gphd/hadoop-hdfs/*,/usr/lib/gphd/hadoop-hdfs/lib/*,/usr/lib/gphd/hadoop-yarn/*,/usr/lib/gphd/hadoop-yarn/lib/*,/usr/lib/gphd/hadoop-mapreduce/*,/usr/lib/gphd/hadoop-mapreduce/lib/*,/etc/gphd/hadoop/conf/*</value>
> </property>
>
>
> [root@phd2 bin]# ls /usr/lib/gphd/hadoop/
> bin hadoop-annotations.jar hadoop-common-2.2.0-gphd-3.1.0.0-tests.jar hadoop-nfs.jar NOTICE.txt
> CHANGES.txt hadoop-auth-2.2.0-gphd-3.1.0.0.jar hadoop-common.jar lib README.txt
> etc hadoop-auth.jar hadoop-common-tests.jar libexec sbin
> hadoop-annotations-2.2.0-gphd-3.1.0.0.jar hadoop-common-2.2.0-gphd-3.1.0.0.jar hadoop-nfs-2.2.0-gphd-3.1.0.0.jar LICENSE.txt
>
>
> [root@phd2 bin]# ls /usr/lib/gphd/hadoop/lib
> activation-1.1.jar commons-httpclient-3.1.jar jasper-runtime-5.5.23.jar jetty-security-7.6.10.v20130312.jar native
> asm-3.2.jar commons-io-2.1.jar javax.servlet-2.5.0.v201103041518.jar jetty-server-7.6.10.v20130312.jar netty-3.6.2.Final.jar
> avro-1.7.4.jar commons-lang-2.5.jar jaxb-api-2.2.2.jar jetty-servlet-7.6.10.v20130312.jar paranamer-2.3.jar
> commons-beanutils-1.7.0.jar commons-logging-1.1.1.jar jaxb-impl-2.2.3-1.jar jetty-util-7.6.10.v20130312.jar protobuf-java-2.5.0.jar
> commons-beanutils-core-1.8.0.jar commons-math-2.1.jar jersey-core-1.9.jar jetty-webapp-7.6.10.v20130312.jar servlet-api-2.5.jar
> commons-cli-1.2.jar commons-net-3.1.jar jersey-json-1.9.jar jetty-xml-7.6.10.v20130312.jar slf4j-api-1.7.5.jar
> commons-codec-1.4.jar guava-11.0.2.jar jersey-server-1.9.jar jsch-0.1.42.jar slf4j-log4j12-1.7.5.jar
> commons-collections-3.2.1.jar jackson-core-asl-1.8.8.jar jets3t-0.6.1.jar jsp-api-2.1.jar snappy-java-1.0.4.1.jar
> commons-compress-1.4.1.jar jackson-jaxrs-1.8.8.jar jettison-1.1.jar jsr305-1.3.9.jar stax-api-1.0.1.jar
> commons-configuration-1.6.jar jackson-mapper-asl-1.8.8.jar jetty-continuation-7.6.10.v20130312.jar junit-4.8.2.jar xmlenc-0.52.jar
> commons-digester-1.8.jar jackson-xc-1.8.8.jar jetty-http-7.6.10.v20130312.jar log4j-1.2.17.jar xz-1.0.jar
> commons-el-1.0.jar jasper-compiler-5.5.23.jar jetty-io-7.6.10.v20130312.jar mockito-all-1.8.5.jar zookeeper.jar
>
>
> [root@phd2 bin]# ls /usr/lib/gphd/hadoop-yarn/lib
> aopalliance-1.0.jar commons-io-2.1.jar jackson-core-asl-1.8.8.jar jersey-guice-1.9.jar netty-3.6.2.Final.jar xz-1.0.jar
> asm-3.2.jar guice-3.0.jar jackson-mapper-asl-1.8.8.jar jersey-server-1.9.jar paranamer-2.3.jar
> avro-1.7.4.jar guice-servlet-3.0.jar javax.inject-1.jar junit-4.10.jar protobuf-java-2.5.0.jar
> commons-compress-1.4.1.jar hamcrest-core-1.1.jar jersey-core-1.9.jar log4j-1.2.17.jar snappy-java-1.0.4.1.jar
>
>
> I am not sure which all environment varibles should be exported or described in the hadoop-env.sh or othere configuration files.
>
> Note :
> Pivotal version which I have installed supports Apache Hadoop 2.2
>
> Remember:
> While replying, please keep gmail id in loop
>
> Let me know how can I resolve the error.
>
>
> With Regards,
> Pratik
>
> DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.