You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Andre Kelpe <ak...@concurrentinc.com> on 2015/01/15 01:07:51 UTC
(elastic) Hadoop YARN on mesos?
Hi,
we are currently looking for an easy way to spin up Hadoop YARN clusters on some hardware that we have lying around. I came across mesos and was wondering if somebody here is using it this way. What I am looking for is something where I say: Give me a cluster with n nodes. Run some tests and then destroy the Hadoop cluster. Similar to how EMR would work or how you work with vagrant locally. Is somebody here doing this and if so, do you have any starter docs for such a setup somewhere?
Note that we def. need YARN and we want to use vanilla Apache Hadoop, so no commercial distribution.
Thanks for your help!
- Andre
--
André Kelpe
andre@concurrentinc.com
http://concurrentinc.com
Re: (elastic) Hadoop YARN on mesos?
Posted by Srinivas Murthy <sr...@gmail.com>.
Myriad, Slider, Marathon, Aurora, Ambari.. Wait, wait, wait. How do you
humanly play with them all enough to know one versus the other to make a
call? And do your day job as well!
On Wed, Jan 14, 2015 at 11:49 PM, Shivaji Dutta <sd...@hortonworks.com>
wrote:
> Andre
>
> You can take a look at Apache Ambari. It allows you to spin up clusters
> very fast, add nodes and remove nodes.
> If you have an existing cluster and you want to run long running jobs, you
> can definitely use Apache Slider which is like Marathon in some ways.
>
> There is doc
> https://cwiki.apache.org/confluence/display/AMBARI/Ambari
>
> Shivaji
>
> From: Joseph Jacks <ja...@gmail.com>
> Reply-To: "user@mesos.apache.org" <us...@mesos.apache.org>
> Date: Wednesday, January 14, 2015 at 4:11 PM
> To: "user@mesos.apache.org" <us...@mesos.apache.org>
> Subject: Re: (elastic) Hadoop YARN on mesos?
>
> Hey Andre,
>
> Have you seen: https://github.com/mesos/myriad
>
> JJ.
>
> On Wed, Jan 14, 2015 at 4:07 PM, Andre Kelpe <ak...@concurrentinc.com>
> wrote:
>
>> Hi,
>>
>> we are currently looking for an easy way to spin up Hadoop YARN clusters
>> on some hardware that we have lying around. I came across mesos and was
>> wondering if somebody here is using it this way. What I am looking for is
>> something where I say: Give me a cluster with n nodes. Run some tests and
>> then destroy the Hadoop cluster. Similar to how EMR would work or how you
>> work with vagrant locally. Is somebody here doing this and if so, do you
>> have any starter docs for such a setup somewhere?
>>
>> Note that we def. need YARN and we want to use vanilla Apache Hadoop, so
>> no commercial distribution.
>>
>> Thanks for your help!
>>
>> - Andre
>>
>> --
>> André Kelpe
>> andre@concurrentinc.com
>> http://concurrentinc.com
>>
>>
>>
>>
>>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: (elastic) Hadoop YARN on mesos?
Posted by Shivaji Dutta <sd...@hortonworks.com>.
Andre
You can take a look at Apache Ambari. It allows you to spin up clusters very fast, add nodes and remove nodes.
If you have an existing cluster and you want to run long running jobs, you can definitely use Apache Slider which is like Marathon in some ways.
There is doc
https://cwiki.apache.org/confluence/display/AMBARI/Ambari
Shivaji
From: Joseph Jacks <ja...@gmail.com>>
Reply-To: "user@mesos.apache.org<ma...@mesos.apache.org>" <us...@mesos.apache.org>>
Date: Wednesday, January 14, 2015 at 4:11 PM
To: "user@mesos.apache.org<ma...@mesos.apache.org>" <us...@mesos.apache.org>>
Subject: Re: (elastic) Hadoop YARN on mesos?
Hey Andre,
Have you seen: https://github.com/mesos/myriad
JJ.
On Wed, Jan 14, 2015 at 4:07 PM, Andre Kelpe <ak...@concurrentinc.com>> wrote:
Hi,
we are currently looking for an easy way to spin up Hadoop YARN clusters on some hardware that we have lying around. I came across mesos and was wondering if somebody here is using it this way. What I am looking for is something where I say: Give me a cluster with n nodes. Run some tests and then destroy the Hadoop cluster. Similar to how EMR would work or how you work with vagrant locally. Is somebody here doing this and if so, do you have any starter docs for such a setup somewhere?
Note that we def. need YARN and we want to use vanilla Apache Hadoop, so no commercial distribution.
Thanks for your help!
- Andre
--
André Kelpe
andre@concurrentinc.com<ma...@concurrentinc.com>
http://concurrentinc.com
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: (elastic) Hadoop YARN on mesos?
Posted by Andre Kelpe <ak...@concurrentinc.com>.
Thanks for the pointers! I will try to get something up and running.
- Andre
On Wed, Jan 14, 2015 at 5:36 PM, mohit soni <mo...@gmail.com> wrote:
> Hi Andre
>
> Myriad allows you to run YARN on Mesos. Since YARN is also a resource
> manager, imagine this as a kind of a delegation of resources from Mesos to
> YARN.
>
> With Myriad one can spin up NodeManager(s) as Mesos tasks, which upon
> startup reports back to YARN's Resource Manager and joins the cluster.
>
> Now, in your scenario you would like to spin up 'ephemeral' hadoop
> clusters. You can certainly do so in Mesos ecosystem, with a little bit of
> work (right now). My suggestion would be to use Marathon (
> https://mesosphere.github.io/marathon/) to launch Myriad/YARN Resource
> Manager, and then trigger Myriad's flexUp API (
> https://github.com/mesos/myriad/blob/phase1/docs/API.md#put-apiclusterflexup)
> to spin N number of NodeManagers on mesos cluster.
>
> Regards
> Mohit
>
> On Wed, Jan 14, 2015 at 4:37 PM, Andre Kelpe <ak...@concurrentinc.com>
> wrote:
>
>> HI,
>>
>> yes, I looked at it, but from what I understand it is a project to run
>> YARN next to mesos, not on top of it. I am looking for a way to have
>> “ephemeral” hadoop clusters one some hardware, not a static YARN setup
>> sharing resources with mesos. Maybe mesos is not what I need here, I was
>> just wondering if somebody here does something similar.
>>
>> - Andre
>>
>> > On 14 Jan 2015, at 16:11, Joseph Jacks <ja...@gmail.com> wrote:
>> >
>> > Hey Andre,
>> >
>> > Have you seen: https://github.com/mesos/myriad
>> >
>> > JJ.
>> >
>> > On Wed, Jan 14, 2015 at 4:07 PM, Andre Kelpe <ak...@concurrentinc.com>
>> wrote:
>> > Hi,
>> >
>> > we are currently looking for an easy way to spin up Hadoop YARN
>> clusters on some hardware that we have lying around. I came across mesos
>> and was wondering if somebody here is using it this way. What I am looking
>> for is something where I say: Give me a cluster with n nodes. Run some
>> tests and then destroy the Hadoop cluster. Similar to how EMR would work or
>> how you work with vagrant locally. Is somebody here doing this and if so,
>> do you have any starter docs for such a setup somewhere?
>> >
>> > Note that we def. need YARN and we want to use vanilla Apache Hadoop,
>> so no commercial distribution.
>> >
>> > Thanks for your help!
>> >
>> > - Andre
>> >
>> > --
>> > André Kelpe
>> > andre@concurrentinc.com
>> > http://concurrentinc.com
>> >
>> >
>> >
>> >
>> >
>>
>> --
>> André Kelpe
>> andre@concurrentinc.com
>> http://concurrentinc.com
>>
>>
>>
>>
>>
>
--
André Kelpe
andre@concurrentinc.com
http://concurrentinc.com
Re: (elastic) Hadoop YARN on mesos?
Posted by mohit soni <mo...@gmail.com>.
Hi Andre
Myriad allows you to run YARN on Mesos. Since YARN is also a resource
manager, imagine this as a kind of a delegation of resources from Mesos to
YARN.
With Myriad one can spin up NodeManager(s) as Mesos tasks, which upon
startup reports back to YARN's Resource Manager and joins the cluster.
Now, in your scenario you would like to spin up 'ephemeral' hadoop
clusters. You can certainly do so in Mesos ecosystem, with a little bit of
work (right now). My suggestion would be to use Marathon (
https://mesosphere.github.io/marathon/) to launch Myriad/YARN Resource
Manager, and then trigger Myriad's flexUp API (
https://github.com/mesos/myriad/blob/phase1/docs/API.md#put-apiclusterflexup)
to spin N number of NodeManagers on mesos cluster.
Regards
Mohit
On Wed, Jan 14, 2015 at 4:37 PM, Andre Kelpe <ak...@concurrentinc.com>
wrote:
> HI,
>
> yes, I looked at it, but from what I understand it is a project to run
> YARN next to mesos, not on top of it. I am looking for a way to have
> “ephemeral” hadoop clusters one some hardware, not a static YARN setup
> sharing resources with mesos. Maybe mesos is not what I need here, I was
> just wondering if somebody here does something similar.
>
> - Andre
>
> > On 14 Jan 2015, at 16:11, Joseph Jacks <ja...@gmail.com> wrote:
> >
> > Hey Andre,
> >
> > Have you seen: https://github.com/mesos/myriad
> >
> > JJ.
> >
> > On Wed, Jan 14, 2015 at 4:07 PM, Andre Kelpe <ak...@concurrentinc.com>
> wrote:
> > Hi,
> >
> > we are currently looking for an easy way to spin up Hadoop YARN clusters
> on some hardware that we have lying around. I came across mesos and was
> wondering if somebody here is using it this way. What I am looking for is
> something where I say: Give me a cluster with n nodes. Run some tests and
> then destroy the Hadoop cluster. Similar to how EMR would work or how you
> work with vagrant locally. Is somebody here doing this and if so, do you
> have any starter docs for such a setup somewhere?
> >
> > Note that we def. need YARN and we want to use vanilla Apache Hadoop, so
> no commercial distribution.
> >
> > Thanks for your help!
> >
> > - Andre
> >
> > --
> > André Kelpe
> > andre@concurrentinc.com
> > http://concurrentinc.com
> >
> >
> >
> >
> >
>
> --
> André Kelpe
> andre@concurrentinc.com
> http://concurrentinc.com
>
>
>
>
>
Re: (elastic) Hadoop YARN on mesos?
Posted by Andre Kelpe <ak...@concurrentinc.com>.
HI,
yes, I looked at it, but from what I understand it is a project to run YARN next to mesos, not on top of it. I am looking for a way to have “ephemeral” hadoop clusters one some hardware, not a static YARN setup sharing resources with mesos. Maybe mesos is not what I need here, I was just wondering if somebody here does something similar.
- Andre
> On 14 Jan 2015, at 16:11, Joseph Jacks <ja...@gmail.com> wrote:
>
> Hey Andre,
>
> Have you seen: https://github.com/mesos/myriad
>
> JJ.
>
> On Wed, Jan 14, 2015 at 4:07 PM, Andre Kelpe <ak...@concurrentinc.com> wrote:
> Hi,
>
> we are currently looking for an easy way to spin up Hadoop YARN clusters on some hardware that we have lying around. I came across mesos and was wondering if somebody here is using it this way. What I am looking for is something where I say: Give me a cluster with n nodes. Run some tests and then destroy the Hadoop cluster. Similar to how EMR would work or how you work with vagrant locally. Is somebody here doing this and if so, do you have any starter docs for such a setup somewhere?
>
> Note that we def. need YARN and we want to use vanilla Apache Hadoop, so no commercial distribution.
>
> Thanks for your help!
>
> - Andre
>
> --
> André Kelpe
> andre@concurrentinc.com
> http://concurrentinc.com
>
>
>
>
>
--
André Kelpe
andre@concurrentinc.com
http://concurrentinc.com
Re: (elastic) Hadoop YARN on mesos?
Posted by Joseph Jacks <ja...@gmail.com>.
Hey Andre,
Have you seen: https://github.com/mesos/myriad
JJ.
On Wed, Jan 14, 2015 at 4:07 PM, Andre Kelpe <ak...@concurrentinc.com>
wrote:
> Hi,
>
> we are currently looking for an easy way to spin up Hadoop YARN clusters
> on some hardware that we have lying around. I came across mesos and was
> wondering if somebody here is using it this way. What I am looking for is
> something where I say: Give me a cluster with n nodes. Run some tests and
> then destroy the Hadoop cluster. Similar to how EMR would work or how you
> work with vagrant locally. Is somebody here doing this and if so, do you
> have any starter docs for such a setup somewhere?
>
> Note that we def. need YARN and we want to use vanilla Apache Hadoop, so
> no commercial distribution.
>
> Thanks for your help!
>
> - Andre
>
> --
> André Kelpe
> andre@concurrentinc.com
> http://concurrentinc.com
>
>
>
>
>