Re: Does Mesos support Hadoop MR V2

[Tim St Clair <ts...@redhat.com>]

FWIW - I've been able to setup YARN clusters in containers using bridged networking and relying on zookeeper for Namenode resolution.

The problem is - 'bridged networking' = IP-fiasco at scale. 

However, I still have hopes for MAC VLAN support for Docker: https://groups.google.com/forum/#!topic/docker-dev/6tt1y9FTWKg 

Cheers,
Tim

----- Original Message -----
> From: "Jie Yu" <yu...@gmail.com>
> To: user@mesos.apache.org
> Cc: "mesos" <de...@mesos.apache.org>
> Sent: Saturday, July 26, 2014 5:19:28 PM
> Subject: Re: Does Mesos support Hadoop MR V2
> 
> I am interested in investigating the idea of YARN on top of Mesos. One of
> the benefits I can think of is that we can get rid of the static resource
> allocation between YARN and Mesos clusters. In that way, Mesos can allocate
> those resources that are not used by YARN to other Mesos frameworks like
> Aurora, Marathon, etc, to increase the resource utilization of the entire
> data center. Also, we could avoid running each MRv2 job as a framework
> which I think might cause some maintenance complexity (e.g. for framework
> rate limiting, etc). Finally, YARN currently does not have a good isolation
> support. It only supports cpu isolation right now (using cgroups). By
> porting YARN on top of Mesos, we might be able to leverage the existing
> Mesos containerizer strategy to provide better isolation between tasks.
> Maxime, I am curious why do you think it does not make sense to run YARN
> over Mesos? Since I am not super familar with YARN, I might be missing
> something.
> 
> I have been thinking of making ResourceManager in YARN a Mesos framework
> and making NodeManager a Mesos executor. The NodeManager will launch
> containers using primitives provided by Mesos so that we have a consistent
> containerizer layer. I haven't fully figured out how this could be done yet
> (e.g., nested containers, communication between NodeManager and
> ResourceManager, etc.), but I would love to explore this direction. I would
> like to hear about any feedback/suggestions you guys have about this
> direction.
> 
> Thanks,
> - Jie
> 
> 
> On Fri, Jul 25, 2014 at 1:39 PM, Maxime Brugidou <ma...@gmail.com>
> wrote:
> 
> > We run both mesos and yarn in prod and it does not make sense to run yarn
> > over mesos.
> >
> > However it would be interesting to find a way to run MRv2 jobs on mesos
> > with some custom layer to swap yarn with mesos. Not sure how to start
> > though... MRv2 contains a yarn application master that needs to be
> > rewritten as a mesos framework scheduler. This is probably doable. However
> > with MRv2 every map reduce job would be mapped as a new framework in Mesos.
> > Not sure how many frameworks mesos can run and scale up to. Especially
> > short lived frameworks.
> > On Jul 25, 2014 8:54 PM, "Tom Arnfeld" <to...@duedil.com> wrote:
> >
> >> Hey Luyi,
> >>
> >> That's correct, the Hadoop framework currently only supports Hadoop 2
> >> MRv1. It also doesn't have great support for the HA jobtracker available
> >> in
> >> newer versions of Hadoop, but I've been working on that the past few
> >> weeks.
> >>
> >> I'm not sure how Hadoop 2 would play with Mesos, but very interested to
> >> find out more. Am I correct in thinking MRv2 will only run on top of YARN?
> >>
> >> I wonder if anyone else on the mailing list is running YARN on top of
> >> Mesos...
> >>
> >> Tom.
> >>
> >> On Friday, 25 July 2014, Luyi Wang <wa...@gmail.com> wrote:
> >>
> >>> Checked the mesos github(https://github.com/mesos/hadoop). It listed
> >>> support for MapReduce V1
> >>>
> >>> How about the MR V2?
> >>>
> >>> Right now we are using cloudera to manage hadoop clusters where uses
> >>> MRV2. We are planning to migrate all our services to mesos(still in the
> >>> initial investigating stage).  Good suggestions, advice and experiences
> >>> are
> >>> welcomed.
> >>>
> >>> Thanks a lot!
> >>>
> >>>
> >>> -Luyi.
> >>>
> >>>
> >>>
> >>>
> 

-- 
Cheers,
Timothy St. Clair
Red Hat Inc.