You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Damien Hardy <dh...@viadeoteam.com> on 2013/12/31 15:08:45 UTC
Hadoop on Mesos use local cdh4 installation instead of tar.gz
Hello,
Happy new year 2014 @mesos users.
I am trying to get MapReduce cdh4.1.2 running on Mesos.
Seams working mostly but few things are still problematic.
* MR1 code is already deployed locally with HDFS is there a way to use
it instead of tar.gz stored on HDFS to be copied locally and untar.
* If not, using tar.gz distribution of cdh4 seams not supporting
Snappy compression. is there a way to correct it ?
Best regards,
--
Damien HARDY
Re: Hadoop on Mesos use local cdh4 installation instead of tar.gz
Posted by Damien Hardy <dh...@viadeoteam.com>.
Hello,
Seams that a directory is not supported here ("cp omitting directory
/usr/lib/hadoop-0.20-mapreduce/" and failing then in 256 error in stderr)
But this work (I was first disappointed by using file:// schema as an uri) :
```
<property>
<name>mapred.mesos.executor.uri</name>
<value>/usr/lib/hadoop-0.20-mapreduce/bin/hadoop</value>
</property>
<property>
<name>mapred.mesos.executor.directory</name>
<value>./</value>
</property>
<property>
<name>mapred.mesos.executor.command</name>
<value>. /etc/default/hadoop-0.20; env ; ./hadoop
org.apache.hadoop.mapred.MesosExecutor</value>
</property>
```
just copying the `bin/hadoop` command in framework directory and using it.
Le 02/01/2014 19:17, Florian Leibert a écrit :
> Hi Damien -
> Sorry for responding to this a bit late.
>
> Snappy should be enabled if you follow the steps in this tutorial:
> "http://mesosphere.io/learn/run-hadoop-on-mesos/". You can set an
> alternate location for the hadoop distribution by changing the
> "mapred.mesos.executor.uri" - i.e. you can use a local directory / file
> here.
>
> I hope this helps!
>
> Cheers & happy new year,
> --Flo
> ᐧ
>
>
> On Thu, Jan 2, 2014 at 8:45 AM, Damien Hardy <dhardy@viadeoteam.com
> <ma...@viadeoteam.com>> wrote:
>
> Hello,
>
> Using hadoop distribution is possible (here cdh4.1.2) :
> An archive is mandatory by haddop-mesos framework, so I created and
> deployed a small dummy file that does not cost so much to get and untar.
>
> In mapred-site.xml, override mapred.mesos.executor.directory and
> mapred.mesos.executor.command so I use mesos task directory for my job
> and deployed cloudera tasktracker to execute.
>
> + <property>
> + <name>mapred.mesos.executor.uri</name>
> + <value>hdfs://hdfscluster/tmp/dummy.tar.gz</value>
> + </property>
> + <property>
> + <name>mapred.mesos.executor.directory</name>
> + <value>./</value>
> + </property>
> + <property>
> + <name>mapred.mesos.executor.command</name>
> + <value>. /etc/default/hadoop-0.20; env ; $HADOOP_HOME/bin/hadoop
> org.apache.hadoop.mapred.MesosExecutor</value>
> + </property>
>
> Add some envar in /etc/default/hadoop-0.20 so hadoop services can find
> hadoop-mesos jar and libmesos :
>
> +export
> HADOOP_CLASSPATH=/usr/lib/hadoop-mesos/hadoop-mesos.jar:$HADOOP_HOME/contrib/fairscheduler/hadoop-fairscheduler-2.0.0-mr1-cdh4.1.2.jar:$HADOOP_CLASSPATH
> +export MESOS_NATIVE_LIBRARY=/usr/lib/libmesos.so
>
> I created an hadoop-mesos deb to be deployed with hadoop ditribution.
> My goal is to limit -copyToLocal of TT code for each mesos tasks, and no
> need for special manipulation in Hadoop Distribution code (only config)
>
> Regards,
>
> Le 31/12/2013 16:45, Damien Hardy a écrit :
> > I'm now able to use snappy compression by adding
> >
> > export JAVA_LIBRARY_PATH=/usr/lib/hadoop/lib/native/
> > in my /etc/default/mesos-slave (environment variable for mesos-slave
> > process used by my init.d script)
> >
> > This envar is propagated to executor Jvm and so taskTracker can find
> > libsnappy.so to use it.
> >
> > Starting using local deployement of cdh4 ...
> >
> > Reading at the source it seams that something could be done using
> > mapred.mesos.executor.directory and mapred.mesos.executor.command
> > to use local hadoop.
> >
> >
> > Le 31/12/2013 15:08, Damien Hardy a écrit :
> >> Hello,
> >>
> >> Happy new year 2014 @mesos users.
> >>
> >> I am trying to get MapReduce cdh4.1.2 running on Mesos.
> >>
> >> Seams working mostly but few things are still problematic.
> >>
> >> * MR1 code is already deployed locally with HDFS is there a way
> to use
> >> it instead of tar.gz stored on HDFS to be copied locally and untar.
> >>
> >> * If not, using tar.gz distribution of cdh4 seams not supporting
> >> Snappy compression. is there a way to correct it ?
> >>
> >> Best regards,
> >>
> >
>
> --
> Damien HARDY
> IT Infrastructure Architect
> Viadeo - 30 rue de la Victoire - 75009 Paris - France
> PGP : 45D7F89A
>
>
--
Damien HARDY
IT Infrastructure Architect
Viadeo - 30 rue de la Victoire - 75009 Paris - France
PGP : 45D7F89A
Re: Hadoop on Mesos use local cdh4 installation instead of tar.gz
Posted by Florian Leibert <fl...@mesosphere.io>.
Hi Damien -
Sorry for responding to this a bit late.
Snappy should be enabled if you follow the steps in this tutorial: "
http://mesosphere.io/learn/run-hadoop-on-mesos/". You can set an alternate
location for the hadoop distribution by changing the
"mapred.mesos.executor.uri"
- i.e. you can use a local directory / file here.
I hope this helps!
Cheers & happy new year,
--Flo
ᐧ
On Thu, Jan 2, 2014 at 8:45 AM, Damien Hardy <dh...@viadeoteam.com> wrote:
> Hello,
>
> Using hadoop distribution is possible (here cdh4.1.2) :
> An archive is mandatory by haddop-mesos framework, so I created and
> deployed a small dummy file that does not cost so much to get and untar.
>
> In mapred-site.xml, override mapred.mesos.executor.directory and
> mapred.mesos.executor.command so I use mesos task directory for my job
> and deployed cloudera tasktracker to execute.
>
> + <property>
> + <name>mapred.mesos.executor.uri</name>
> + <value>hdfs://hdfscluster/tmp/dummy.tar.gz</value>
> + </property>
> + <property>
> + <name>mapred.mesos.executor.directory</name>
> + <value>./</value>
> + </property>
> + <property>
> + <name>mapred.mesos.executor.command</name>
> + <value>. /etc/default/hadoop-0.20; env ; $HADOOP_HOME/bin/hadoop
> org.apache.hadoop.mapred.MesosExecutor</value>
> + </property>
>
> Add some envar in /etc/default/hadoop-0.20 so hadoop services can find
> hadoop-mesos jar and libmesos :
>
> +export
>
> HADOOP_CLASSPATH=/usr/lib/hadoop-mesos/hadoop-mesos.jar:$HADOOP_HOME/contrib/fairscheduler/hadoop-fairscheduler-2.0.0-mr1-cdh4.1.2.jar:$HADOOP_CLASSPATH
> +export MESOS_NATIVE_LIBRARY=/usr/lib/libmesos.so
>
> I created an hadoop-mesos deb to be deployed with hadoop ditribution.
> My goal is to limit -copyToLocal of TT code for each mesos tasks, and no
> need for special manipulation in Hadoop Distribution code (only config)
>
> Regards,
>
> Le 31/12/2013 16:45, Damien Hardy a écrit :
> > I'm now able to use snappy compression by adding
> >
> > export JAVA_LIBRARY_PATH=/usr/lib/hadoop/lib/native/
> > in my /etc/default/mesos-slave (environment variable for mesos-slave
> > process used by my init.d script)
> >
> > This envar is propagated to executor Jvm and so taskTracker can find
> > libsnappy.so to use it.
> >
> > Starting using local deployement of cdh4 ...
> >
> > Reading at the source it seams that something could be done using
> > mapred.mesos.executor.directory and mapred.mesos.executor.command
> > to use local hadoop.
> >
> >
> > Le 31/12/2013 15:08, Damien Hardy a écrit :
> >> Hello,
> >>
> >> Happy new year 2014 @mesos users.
> >>
> >> I am trying to get MapReduce cdh4.1.2 running on Mesos.
> >>
> >> Seams working mostly but few things are still problematic.
> >>
> >> * MR1 code is already deployed locally with HDFS is there a way to use
> >> it instead of tar.gz stored on HDFS to be copied locally and untar.
> >>
> >> * If not, using tar.gz distribution of cdh4 seams not supporting
> >> Snappy compression. is there a way to correct it ?
> >>
> >> Best regards,
> >>
> >
>
> --
> Damien HARDY
> IT Infrastructure Architect
> Viadeo - 30 rue de la Victoire - 75009 Paris - France
> PGP : 45D7F89A
>
>
Re: Hadoop on Mesos use local cdh4 installation instead of tar.gz
Posted by Damien Hardy <dh...@viadeoteam.com>.
Hello,
Using hadoop distribution is possible (here cdh4.1.2) :
An archive is mandatory by haddop-mesos framework, so I created and
deployed a small dummy file that does not cost so much to get and untar.
In mapred-site.xml, override mapred.mesos.executor.directory and
mapred.mesos.executor.command so I use mesos task directory for my job
and deployed cloudera tasktracker to execute.
+ <property>
+ <name>mapred.mesos.executor.uri</name>
+ <value>hdfs://hdfscluster/tmp/dummy.tar.gz</value>
+ </property>
+ <property>
+ <name>mapred.mesos.executor.directory</name>
+ <value>./</value>
+ </property>
+ <property>
+ <name>mapred.mesos.executor.command</name>
+ <value>. /etc/default/hadoop-0.20; env ; $HADOOP_HOME/bin/hadoop
org.apache.hadoop.mapred.MesosExecutor</value>
+ </property>
Add some envar in /etc/default/hadoop-0.20 so hadoop services can find
hadoop-mesos jar and libmesos :
+export
HADOOP_CLASSPATH=/usr/lib/hadoop-mesos/hadoop-mesos.jar:$HADOOP_HOME/contrib/fairscheduler/hadoop-fairscheduler-2.0.0-mr1-cdh4.1.2.jar:$HADOOP_CLASSPATH
+export MESOS_NATIVE_LIBRARY=/usr/lib/libmesos.so
I created an hadoop-mesos deb to be deployed with hadoop ditribution.
My goal is to limit -copyToLocal of TT code for each mesos tasks, and no
need for special manipulation in Hadoop Distribution code (only config)
Regards,
Le 31/12/2013 16:45, Damien Hardy a écrit :
> I'm now able to use snappy compression by adding
>
> export JAVA_LIBRARY_PATH=/usr/lib/hadoop/lib/native/
> in my /etc/default/mesos-slave (environment variable for mesos-slave
> process used by my init.d script)
>
> This envar is propagated to executor Jvm and so taskTracker can find
> libsnappy.so to use it.
>
> Starting using local deployement of cdh4 ...
>
> Reading at the source it seams that something could be done using
> mapred.mesos.executor.directory and mapred.mesos.executor.command
> to use local hadoop.
>
>
> Le 31/12/2013 15:08, Damien Hardy a écrit :
>> Hello,
>>
>> Happy new year 2014 @mesos users.
>>
>> I am trying to get MapReduce cdh4.1.2 running on Mesos.
>>
>> Seams working mostly but few things are still problematic.
>>
>> * MR1 code is already deployed locally with HDFS is there a way to use
>> it instead of tar.gz stored on HDFS to be copied locally and untar.
>>
>> * If not, using tar.gz distribution of cdh4 seams not supporting
>> Snappy compression. is there a way to correct it ?
>>
>> Best regards,
>>
>
--
Damien HARDY
IT Infrastructure Architect
Viadeo - 30 rue de la Victoire - 75009 Paris - France
PGP : 45D7F89A
Re: Hadoop on Mesos use local cdh4 installation instead of tar.gz
Posted by Damien Hardy <dh...@viadeoteam.com>.
I'm now able to use snappy compression by adding
export JAVA_LIBRARY_PATH=/usr/lib/hadoop/lib/native/
in my /etc/default/mesos-slave (environment variable for mesos-slave
process used by my init.d script)
This envar is propagated to executor Jvm and so taskTracker can find
libsnappy.so to use it.
Starting using local deployement of cdh4 ...
Reading at the source it seams that something could be done using
mapred.mesos.executor.directory and mapred.mesos.executor.command
to use local hadoop.
Le 31/12/2013 15:08, Damien Hardy a écrit :
> Hello,
>
> Happy new year 2014 @mesos users.
>
> I am trying to get MapReduce cdh4.1.2 running on Mesos.
>
> Seams working mostly but few things are still problematic.
>
> * MR1 code is already deployed locally with HDFS is there a way to use
> it instead of tar.gz stored on HDFS to be copied locally and untar.
>
> * If not, using tar.gz distribution of cdh4 seams not supporting
> Snappy compression. is there a way to correct it ?
>
> Best regards,
>
--
Damien HARDY