You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Kk Bk <kk...@gmail.com> on 2015/07/03 16:51:32 UTC
Hadoop on Mesos. HDFS question.
I am trying to install Hadoop on Mesos on ubuntu servers, So followed
instruction as per link
https://open.mesosphere.com/tutorials/run-hadoop-on-mesos/#step-2.
Step-2 of link says to install HDFS using as per link
http://www.cloudera.com/content/cloudera/en/documentation/cdh4/latest/CDH4-Installation-Guide/cdh4ig_topic_4_4.html
.
Question: Is it sufficient to run following commands
1) On Namenode: sudo apt-get install hadoop-hdfs-namenode
2) On Datanode: sudo apt-get install hadoop-0.20-mapreduce-tasktracker
hadoop-hdfs-datanode
Or just follow the instructions on the mesosphere link that installs HDFS ?
Re: Hadoop on Mesos. HDFS question.
Posted by Adam Bordelon <ad...@mesosphere.io>.
Blatant product plug: The easiest way to run hdfs-mesos on Mesos (using
Marathon) is to launch a DCOS (EE) cluster. https://mesosphere.com/product/
You might also want to look at the custom DCOS config in
https://github.com/mesosphere/hdfs/tree/master/example-conf/mesosphere-dcos
For basic instructions on running an app in Marathon, see
https://mesosphere.github.io/marathon/docs/application-basics.html
See the marathon.json in DCOS Universe:
https://github.com/mesosphere/universe/blob/version-1.x/repo/packages/H/hdfs/0/marathon.json
Just replace any of the {{moustache}} variables with values like the
defaults in
https://github.com/mesosphere/universe/blob/version-1.x/repo/packages/H/hdfs/0/config.json
You can also simplify the command to "cd hdfs-mesos* && ./bin/hdfs-mesos"
On Mon, Jul 6, 2015 at 2:19 PM, Kk Bk <kk...@gmail.com> wrote:
> Adam
>
> I would like to choose option 2. Cab you provide pointers as to how to run
> hdfs-mesos using marathon ?
>
> -Bhargav
>
> On Sun, Jul 5, 2015 at 10:53 PM, Adam Bordelon <ad...@mesosphere.io> wrote:
>
>> Kk,
>>
>> There are two options for running the hdfs framework on Mesos.
>> - If you already have the hadoop/hdfs binaries on all your nodes, you can
>> follow the instructions in
>> https://github.com/mesosphere/hdfs#if-you-have-hadoop-pre-installed-in-your-cluster
>> to tell the scheduler to use the preinstalled NN/DN binaries.
>> - Otherwise, you can run the hdfs framework scheduler `bin/hdfs-mesos`
>> on any node that can reach the Mesos master and slaves, and it can serve
>> out the binaries itself. Note that this node may not necessarily be the
>> same node on which either of the namenodes end up running. Some choose to
>> run the hdfs-mesos scheduler on a Mesos master node, but you can achieve
>> framework scheduler HA if you run it via another framework like Marathon
>> that can restart the scheduler (elsewhere) if it or its node dies. See
>> example (templatized) Marathon json in
>> https://github.com/mesosphere/universe/tree/version-1.x/repo/packages/H/hdfs/0
>>
>> On Fri, Jul 3, 2015 at 11:31 AM, Kk Bk <kk...@gmail.com> wrote:
>>
>>> Thanks guys for the response.
>>>
>>> 1) I use trusty. Seems like CDH4 does not have support for Trusty.
>>>
>>> 2) Followed instructions as per link https://github.com/mesosphere/hdfs.
>>> Able to build "hdfs-mesos-*.tgz"
>>>
>>> Should i copy this file to all nodes (i have multi-node mesos cluster)
>>> or just the master node of mesos where i plan to keep the namenode for
>>> hadoop
>>>
>>>
>>>
>>>
>>>
>>> On Fri, Jul 3, 2015 at 8:34 AM, Tom Arnfeld <to...@duedil.com> wrote:
>>>
>>>> It might be worth taking a look at the install documentation on the
>>>> Hadoop on Mesos product here; https://github.com/mesos/hadoop
>>>>
>>>> For our installations I don't think we really do much more than
>>>> installing the apt packages you mentioned and then installing the
>>>> hadoop-mesos jars.. plus adding the appropriate configuration.
>>>>
>>>> On Friday, Jul 3, 2015 at 3:52 pm, Kk Bk <kk...@gmail.com>, wrote:
>>>>
>>>>> I am trying to install Hadoop on Mesos on ubuntu servers, So followed
>>>>> instruction as per link
>>>>> https://open.mesosphere.com/tutorials/run-hadoop-on-mesos/#step-2.
>>>>>
>>>>> Step-2 of link says to install HDFS using as per link
>>>>> http://www.cloudera.com/content/cloudera/en/documentation/cdh4/latest/CDH4-Installation-Guide/cdh4ig_topic_4_4.html
>>>>> .
>>>>>
>>>>> Question: Is it sufficient to run following commands
>>>>>
>>>>> 1) On Namenode: sudo apt-get install hadoop-hdfs-namenode
>>>>> 2) On Datanode: sudo apt-get install hadoop-0.20-mapreduce-tasktracker
>>>>> hadoop-hdfs-datanode
>>>>>
>>>>> Or just follow the instructions on the mesosphere link that installs
>>>>> HDFS ?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>
>>
>
Re: Hadoop on Mesos. HDFS question.
Posted by Kk Bk <kk...@gmail.com>.
Adam
I would like to choose option 2. Cab you provide pointers as to how to run
hdfs-mesos using marathon ?
-Bhargav
On Sun, Jul 5, 2015 at 10:53 PM, Adam Bordelon <ad...@mesosphere.io> wrote:
> Kk,
>
> There are two options for running the hdfs framework on Mesos.
> - If you already have the hadoop/hdfs binaries on all your nodes, you can
> follow the instructions in
> https://github.com/mesosphere/hdfs#if-you-have-hadoop-pre-installed-in-your-cluster
> to tell the scheduler to use the preinstalled NN/DN binaries.
> - Otherwise, you can run the hdfs framework scheduler `bin/hdfs-mesos` on
> any node that can reach the Mesos master and slaves, and it can serve out
> the binaries itself. Note that this node may not necessarily be the same
> node on which either of the namenodes end up running. Some choose to run
> the hdfs-mesos scheduler on a Mesos master node, but you can achieve
> framework scheduler HA if you run it via another framework like Marathon
> that can restart the scheduler (elsewhere) if it or its node dies. See
> example (templatized) Marathon json in
> https://github.com/mesosphere/universe/tree/version-1.x/repo/packages/H/hdfs/0
>
> On Fri, Jul 3, 2015 at 11:31 AM, Kk Bk <kk...@gmail.com> wrote:
>
>> Thanks guys for the response.
>>
>> 1) I use trusty. Seems like CDH4 does not have support for Trusty.
>>
>> 2) Followed instructions as per link https://github.com/mesosphere/hdfs.
>> Able to build "hdfs-mesos-*.tgz"
>>
>> Should i copy this file to all nodes (i have multi-node mesos cluster) or
>> just the master node of mesos where i plan to keep the namenode for hadoop
>>
>>
>>
>>
>>
>> On Fri, Jul 3, 2015 at 8:34 AM, Tom Arnfeld <to...@duedil.com> wrote:
>>
>>> It might be worth taking a look at the install documentation on the
>>> Hadoop on Mesos product here; https://github.com/mesos/hadoop
>>>
>>> For our installations I don't think we really do much more than
>>> installing the apt packages you mentioned and then installing the
>>> hadoop-mesos jars.. plus adding the appropriate configuration.
>>>
>>> On Friday, Jul 3, 2015 at 3:52 pm, Kk Bk <kk...@gmail.com>, wrote:
>>>
>>>> I am trying to install Hadoop on Mesos on ubuntu servers, So followed
>>>> instruction as per link
>>>> https://open.mesosphere.com/tutorials/run-hadoop-on-mesos/#step-2.
>>>>
>>>> Step-2 of link says to install HDFS using as per link
>>>> http://www.cloudera.com/content/cloudera/en/documentation/cdh4/latest/CDH4-Installation-Guide/cdh4ig_topic_4_4.html
>>>> .
>>>>
>>>> Question: Is it sufficient to run following commands
>>>>
>>>> 1) On Namenode: sudo apt-get install hadoop-hdfs-namenode
>>>> 2) On Datanode: sudo apt-get install hadoop-0.20-mapreduce-tasktracker
>>>> hadoop-hdfs-datanode
>>>>
>>>> Or just follow the instructions on the mesosphere link that installs
>>>> HDFS ?
>>>>
>>>>
>>>>
>>>>
>>>>
>>
>
Re: Hadoop on Mesos. HDFS question.
Posted by Adam Bordelon <ad...@mesosphere.io>.
Kk,
There are two options for running the hdfs framework on Mesos.
- If you already have the hadoop/hdfs binaries on all your nodes, you can
follow the instructions in
https://github.com/mesosphere/hdfs#if-you-have-hadoop-pre-installed-in-your-cluster
to tell the scheduler to use the preinstalled NN/DN binaries.
- Otherwise, you can run the hdfs framework scheduler `bin/hdfs-mesos` on
any node that can reach the Mesos master and slaves, and it can serve out
the binaries itself. Note that this node may not necessarily be the same
node on which either of the namenodes end up running. Some choose to run
the hdfs-mesos scheduler on a Mesos master node, but you can achieve
framework scheduler HA if you run it via another framework like Marathon
that can restart the scheduler (elsewhere) if it or its node dies. See
example (templatized) Marathon json in
https://github.com/mesosphere/universe/tree/version-1.x/repo/packages/H/hdfs/0
On Fri, Jul 3, 2015 at 11:31 AM, Kk Bk <kk...@gmail.com> wrote:
> Thanks guys for the response.
>
> 1) I use trusty. Seems like CDH4 does not have support for Trusty.
>
> 2) Followed instructions as per link https://github.com/mesosphere/hdfs.
> Able to build "hdfs-mesos-*.tgz"
>
> Should i copy this file to all nodes (i have multi-node mesos cluster) or
> just the master node of mesos where i plan to keep the namenode for hadoop
>
>
>
>
>
> On Fri, Jul 3, 2015 at 8:34 AM, Tom Arnfeld <to...@duedil.com> wrote:
>
>> It might be worth taking a look at the install documentation on the
>> Hadoop on Mesos product here; https://github.com/mesos/hadoop
>>
>> For our installations I don't think we really do much more than
>> installing the apt packages you mentioned and then installing the
>> hadoop-mesos jars.. plus adding the appropriate configuration.
>>
>> On Friday, Jul 3, 2015 at 3:52 pm, Kk Bk <kk...@gmail.com>, wrote:
>>
>>> I am trying to install Hadoop on Mesos on ubuntu servers, So followed
>>> instruction as per link
>>> https://open.mesosphere.com/tutorials/run-hadoop-on-mesos/#step-2.
>>>
>>> Step-2 of link says to install HDFS using as per link
>>> http://www.cloudera.com/content/cloudera/en/documentation/cdh4/latest/CDH4-Installation-Guide/cdh4ig_topic_4_4.html
>>> .
>>>
>>> Question: Is it sufficient to run following commands
>>>
>>> 1) On Namenode: sudo apt-get install hadoop-hdfs-namenode
>>> 2) On Datanode: sudo apt-get install hadoop-0.20-mapreduce-tasktracker
>>> hadoop-hdfs-datanode
>>>
>>> Or just follow the instructions on the mesosphere link that installs
>>> HDFS ?
>>>
>>>
>>>
>>>
>>>
>
Re: Hadoop on Mesos. HDFS question.
Posted by Kk Bk <kk...@gmail.com>.
Thanks guys for the response.
1) I use trusty. Seems like CDH4 does not have support for Trusty.
2) Followed instructions as per link https://github.com/mesosphere/hdfs.
Able to build "hdfs-mesos-*.tgz"
Should i copy this file to all nodes (i have multi-node mesos cluster) or
just the master node of mesos where i plan to keep the namenode for hadoop
On Fri, Jul 3, 2015 at 8:34 AM, Tom Arnfeld <to...@duedil.com> wrote:
> It might be worth taking a look at the install documentation on the
> Hadoop on Mesos product here; https://github.com/mesos/hadoop
>
> For our installations I don't think we really do much more than installing
> the apt packages you mentioned and then installing the hadoop-mesos jars..
> plus adding the appropriate configuration.
>
> On Friday, Jul 3, 2015 at 3:52 pm, Kk Bk <kk...@gmail.com>, wrote:
>
>> I am trying to install Hadoop on Mesos on ubuntu servers, So followed
>> instruction as per link
>> https://open.mesosphere.com/tutorials/run-hadoop-on-mesos/#step-2.
>>
>> Step-2 of link says to install HDFS using as per link
>> http://www.cloudera.com/content/cloudera/en/documentation/cdh4/latest/CDH4-Installation-Guide/cdh4ig_topic_4_4.html
>> .
>>
>> Question: Is it sufficient to run following commands
>>
>> 1) On Namenode: sudo apt-get install hadoop-hdfs-namenode
>> 2) On Datanode: sudo apt-get install hadoop-0.20-mapreduce-tasktracker
>> hadoop-hdfs-datanode
>>
>> Or just follow the instructions on the mesosphere link that installs HDFS
>> ?
>>
>>
>>
>>
>>
Re: Hadoop on Mesos. HDFS question.
Posted by Tom Arnfeld <to...@duedil.com>.
It might be worth taking a look at the install documentation on the Hadoop on Mesos product here; https://github.com/mesos/hadoop
For our installations I don't think we really do much more than installing the apt packages you mentioned and then installing the hadoop-mesos jars.. plus adding the appropriate configuration.
On Friday, Jul 3, 2015 at 3:52 pm, Kk Bk <kk...@gmail.com>, wrote:
I am trying to install Hadoop on Mesos on ubuntu servers, So followed instruction as per link https://open.mesosphere.com/tutorials/run-hadoop-on-mesos/#step-2.
Step-2 of link says to install HDFS using as per link http://www.cloudera.com/content/cloudera/en/documentation/cdh4/latest/CDH4-Installation-Guide/cdh4ig_topic_4_4.html.
Question: Is it sufficient to run following commands
1) On Namenode: sudo apt-get install hadoop-hdfs-namenode
2) On Datanode: sudo apt-get install hadoop-0.20-mapreduce-tasktracker hadoop-hdfs-datanode
Or just follow the instructions on the mesosphere link that installs HDFS ?
Re: Hadoop on Mesos. HDFS question.
Posted by haosdent <ha...@gmail.com>.
You just need install HDFS through sudo apt-get install
hadoop-hdfs-namenode hadoop-hdfs-secondarynamenode
hadoop-hdfs-datanode hadoop-client
And then continue to follow the mesosphere link step. Mesosphere link don't
contain instructions to install HDFS.
On Fri, Jul 3, 2015 at 10:51 PM, Kk Bk <kk...@gmail.com> wrote:
> I am trying to install Hadoop on Mesos on ubuntu servers, So followed
> instruction as per link
> https://open.mesosphere.com/tutorials/run-hadoop-on-mesos/#step-2.
>
> Step-2 of link says to install HDFS using as per link
> http://www.cloudera.com/content/cloudera/en/documentation/cdh4/latest/CDH4-Installation-Guide/cdh4ig_topic_4_4.html
> .
>
> Question: Is it sufficient to run following commands
>
> 1) On Namenode: sudo apt-get install hadoop-hdfs-namenode
> 2) On Datanode: sudo apt-get install hadoop-0.20-mapreduce-tasktracker
> hadoop-hdfs-datanode
>
> Or just follow the instructions on the mesosphere link that installs HDFS ?
>
>
>
>
>
--
Best Regards,
Haosdent Huang