You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mrql.apache.org by Eldon Carman <ec...@ucr.edu> on 2014/05/13 20:01:31 UTC
Hadoop 2.0.0 Support
Hi,
Does MRQL run on the 2.0.0 version of Hadoop? I access to a cluster with
Hadoop installed from Cloudera Express and would like to try out MRQL.
Thanks
Eldon
Re: Hadoop 2.0.0 Support
Posted by Eldon Carman <ec...@ucr.edu>.
The following setting allowed me to run MRQL. I used the
/usr/lib/hadoop home directory supplied by Hadoop 2.0.0-cdh4.6.0 and
used the following HADOOP_JARS value. The log4j version needed to be
updated and the addition of the hadoop-commons jar.
HADOOP_JARS=${HADOOP_HOME}/hadoop-core-${HADOOP_VERSION}.jar:${HADOOP_HOME}/lib/commons-logging-1.1.1.jar:${HADOOP_HOME}/lib/log4j-1.2.17.jar:${HADOOP_HOME}/lib/commons-cli-1.2.jar:${HADOOP_HOME}/hadoop-common-2.0.0-cdh4.6.0.jar
On Tue, May 13, 2014 at 4:43 PM, Eldon Carman <ec...@ucr.edu> wrote:
> Thanks, the wiki page as more information than the Apache Project page
> (http://mrql.incubator.apache.org/getting_started.html). We are using
> CDH4 without yarn installed. I was able to get maven install to run
> with the following command:
>
> % mvn -PMultipleInputs -Dhadoop.version=1.2.1 install
>
> % hadoop version
> Hadoop 2.0.0-cdh4.6.0
> Subversion git://rhel64-6-0-mk4.jenkins.cloudera.com/data/1/jenkins/workspace/generic-package-rhel64-6-0/topdir/BUILD/hadoop-2.0.0-cdh4.6.0/src/hadoop-common-project/hadoop-common
> -r 8e266e052e423af592871e2dfe09d54c03f6a0e8
> Compiled by jenkins on Wed Feb 26 01:58:53 PST 2014
> From source with checksum a9d36604dfb55479c0648f2653c69095
> This command was run using /usr/lib/hadoop/hadoop-common-2.0.0-cdh4.6.0.jar
>
> MRQL will not run since it is missing some of the jars and references
> the wrong class paths. I run into issues when running your page rank
> example. Can you suggest the new classpaths for this version of
> cloudera?
>
> Thanks
> Eldon
>
> UPDATED CONFIG
>
> HADOOP_JARS=${HADOOP_HOME}/hadoop-core-2.0.0-mr1-cdh4.6.0.jar:${HADOOP_HOME}/lib/commons-logging-1.1.1.jar:${HADOOP_HOME}/lib/log4j-1.2.17.jar:${HADOOP_HOME}/lib/commons-cli-1.2.jar:/usr/lib/hadoop-hdfs/*:/usr/lib/hadoop/*:${HADOOP_HOME}/hadoop-core.jar
>
>
> CURRENT ERRORS
>
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/hadoop/util/PlatformName
> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.util.PlatformName
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> Could not find the main class: org.apache.hadoop.util.PlatformName.
> Program will exit.
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/hadoop/util/RunJar
> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.util.RunJar
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> Could not find the main class: org.apache.hadoop.util.RunJar. Program
> will exit.
>
> On Tue, May 13, 2014 at 2:42 PM, Karthik Kambatla <ka...@cloudera.com> wrote:
>> Which version of Cloudera Express are you using? CDH4 and CDH5 ship both MR1
>> and MR2 running on HDFS2.x.
>>
>>
>> On Tue, May 13, 2014 at 2:38 PM, Leonidas Fegaras <fe...@cse.uta.edu>
>> wrote:
>>>
>>> Hi Eldon,
>>> Yes, MRQL can run on Hadoop yarn (hadoop 2.x version).
>>> You need to recompile the source using:
>>>
>>> mvn -Pyarn -Dyarn.version=2.2.0 -Dhadoop.version=1.2.1 install
>>>
>>> (or -Dyarn.version=2.0.3-alpha if you want to use hadoop 2.0.x version).
>>> To run MRQL queries on your cluster, you need also to change
>>> conf/mrql-env.sh.
>>> You can find more info at:
>>> http://wiki.apache.org/mrql/GettingStarted
>>> Best regards,
>>> Leonidas
>>>
>>>
>>> On 05/13/2014 01:01 PM, Eldon Carman wrote:
>>>
>>> Hi,
>>>
>>> Does MRQL run on the 2.0.0 version of Hadoop? I access to a cluster with
>>> Hadoop installed from Cloudera Express and would like to try out MRQL.
>>>
>>> Thanks
>>> Eldon
>>>
>>>
>>
Re: Hadoop 2.0.0 Support
Posted by Eldon Carman <ec...@ucr.edu>.
Thanks, the wiki page as more information than the Apache Project page
(http://mrql.incubator.apache.org/getting_started.html). We are using
CDH4 without yarn installed. I was able to get maven install to run
with the following command:
% mvn -PMultipleInputs -Dhadoop.version=1.2.1 install
% hadoop version
Hadoop 2.0.0-cdh4.6.0
Subversion git://rhel64-6-0-mk4.jenkins.cloudera.com/data/1/jenkins/workspace/generic-package-rhel64-6-0/topdir/BUILD/hadoop-2.0.0-cdh4.6.0/src/hadoop-common-project/hadoop-common
-r 8e266e052e423af592871e2dfe09d54c03f6a0e8
Compiled by jenkins on Wed Feb 26 01:58:53 PST 2014
>From source with checksum a9d36604dfb55479c0648f2653c69095
This command was run using /usr/lib/hadoop/hadoop-common-2.0.0-cdh4.6.0.jar
MRQL will not run since it is missing some of the jars and references
the wrong class paths. I run into issues when running your page rank
example. Can you suggest the new classpaths for this version of
cloudera?
Thanks
Eldon
UPDATED CONFIG
HADOOP_JARS=${HADOOP_HOME}/hadoop-core-2.0.0-mr1-cdh4.6.0.jar:${HADOOP_HOME}/lib/commons-logging-1.1.1.jar:${HADOOP_HOME}/lib/log4j-1.2.17.jar:${HADOOP_HOME}/lib/commons-cli-1.2.jar:/usr/lib/hadoop-hdfs/*:/usr/lib/hadoop/*:${HADOOP_HOME}/hadoop-core.jar
CURRENT ERRORS
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/hadoop/util/PlatformName
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.util.PlatformName
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Could not find the main class: org.apache.hadoop.util.PlatformName.
Program will exit.
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/hadoop/util/RunJar
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.util.RunJar
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Could not find the main class: org.apache.hadoop.util.RunJar. Program
will exit.
On Tue, May 13, 2014 at 2:42 PM, Karthik Kambatla <ka...@cloudera.com> wrote:
> Which version of Cloudera Express are you using? CDH4 and CDH5 ship both MR1
> and MR2 running on HDFS2.x.
>
>
> On Tue, May 13, 2014 at 2:38 PM, Leonidas Fegaras <fe...@cse.uta.edu>
> wrote:
>>
>> Hi Eldon,
>> Yes, MRQL can run on Hadoop yarn (hadoop 2.x version).
>> You need to recompile the source using:
>>
>> mvn -Pyarn -Dyarn.version=2.2.0 -Dhadoop.version=1.2.1 install
>>
>> (or -Dyarn.version=2.0.3-alpha if you want to use hadoop 2.0.x version).
>> To run MRQL queries on your cluster, you need also to change
>> conf/mrql-env.sh.
>> You can find more info at:
>> http://wiki.apache.org/mrql/GettingStarted
>> Best regards,
>> Leonidas
>>
>>
>> On 05/13/2014 01:01 PM, Eldon Carman wrote:
>>
>> Hi,
>>
>> Does MRQL run on the 2.0.0 version of Hadoop? I access to a cluster with
>> Hadoop installed from Cloudera Express and would like to try out MRQL.
>>
>> Thanks
>> Eldon
>>
>>
>
Re: Hadoop 2.0.0 Support
Posted by Karthik Kambatla <ka...@cloudera.com>.
Which version of Cloudera Express are you using? CDH4 and CDH5 ship both
MR1 and MR2 running on HDFS2.x.
On Tue, May 13, 2014 at 2:38 PM, Leonidas Fegaras <fe...@cse.uta.edu>wrote:
> Hi Eldon,
> Yes, MRQL can run on Hadoop yarn (hadoop 2.x version).
> You need to recompile the source using:
>
> mvn -Pyarn -Dyarn.version=2.2.0 -Dhadoop.version=1.2.1 install
>
> (or -Dyarn.version=2.0.3-alpha if you want to use hadoop 2.0.x version).
> To run MRQL queries on your cluster, you need also to change
> conf/mrql-env.sh.
> You can find more info at:
> http://wiki.apache.org/mrql/GettingStarted
> Best regards,
> Leonidas
>
>
> On 05/13/2014 01:01 PM, Eldon Carman wrote:
>
> Hi,
>
> Does MRQL run on the 2.0.0 version of Hadoop? I access to a cluster with
> Hadoop installed from Cloudera Express and would like to try out MRQL.
>
> Thanks
> Eldon
>
>
>
Re: Hadoop 2.0.0 Support
Posted by Leonidas Fegaras <fe...@cse.uta.edu>.
Hi Eldon,
Yes, MRQL can run on Hadoop yarn (hadoop 2.x version).
You need to recompile the source using:
mvn -Pyarn -Dyarn.version=2.2.0 -Dhadoop.version=1.2.1 install
(or -Dyarn.version=2.0.3-alpha if you want to use hadoop 2.0.x version).
To run MRQL queries on your cluster, you need also to change
conf/mrql-env.sh.
You can find more info at:
http://wiki.apache.org/mrql/GettingStarted
Best regards,
Leonidas
On 05/13/2014 01:01 PM, Eldon Carman wrote:
> Hi,
>
> Does MRQL run on the 2.0.0 version of Hadoop? I access to a cluster
> with Hadoop installed from Cloudera Express and would like to try out
> MRQL.
>
> Thanks
> Eldon