You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@slider.apache.org by 杨浩 <ya...@gmail.com> on 2015/03/01 12:32:55 UTC
Is it similar to mesos
Mesos can group several frameworks in the same cluster, but unlike YARN
divides resource to different queues, it divides resources to each
framework. When using Slider, other framework or service can be put on the
Hadoop cluster as a logic cluster. I think YARN && Slider is similar to
MESOS, am I right?
Re: Is it similar to mesos
Posted by Yong Feng <fe...@gmail.com>.
It is arguable that "This means that YARN was not designed for long-running
services, nor for short-lived interactive queries (like small and fast
Spark jobs), and while it’s possible to have it schedule other kinds of
workloads, this is not an ideal model".
I agree two-level scheduler is right direction. One is resource
manager/scheduler and the other is workload scheduler. However it does not
mean YARN is not designed for long-running service.
The scheduler decision YARM made is several allocation units for allocation
request. However it does not mean each allocation unit must serve one
workload. The app master like slider could collect the workload request and
then calculate the number of required service instance, after that
negotiate with YARN to manage the desired number of service instance. The
service could have its own workload scheduler to dispatch workload/request
to service instance.
It could also be a service on top of slider to adjust the resource request
for long running service.
Compared with Mesos which simply push available resource to framework, YARN
has whole picture of system and is able to support more intelligent
placement policy and also is able to make better decision when reclaim
resource from obese application.
Thanks,
Yong
On Thu, Mar 26, 2015 at 10:18 PM, 杨浩 <ya...@gmail.com> wrote:
> Hi guys， I have found an article to support the opinions
>
> http://www.infoq.com/news/2015/02/Myriad
>
> 2015-03-03 21:27 GMT+08:00 杨浩 <ya...@gmail.com>:
>
> > Thanks for your sharing, and I will try it after Mesos needed.
> >
> > 2015-03-01 20:37 GMT+08:00 Yong Feng <fe...@gmail.com>:
> >
> >> A typo
> >>
> >> >>While Yarn and Tez are meta-framework in Yarn
> >> While Slider and Tez are meta-framework in Yarn.
> >>
> >> Yong
> >>
> >> On Sun, Mar 1, 2015 at 7:36 AM, Yong Feng <fe...@gmail.com>
> wrote:
> >>
> >> > Both Mesos and Yarn are resource manager or resource scheduler. They
> use
> >> > the different resource allocaiton protocol, push or request. Google
> >> "Mesos
> >> > Vs Yarn“, you could find more material to compare Mesos and Yarn.
> Mesos
> >> > argues that Yarn is "BigData Application" and is able to run on top of
> >> > Yarn. We actually also could run Mesos on top of Yarn.
> >> >
> >> > To help people moving applicaiton to Mesos, there are several
> >> > meta-framework, such as Marathon and Chronos. While Yarn and Tez are
> >> > meta-framework in Yarn.
> >> >
> >> > I create a vagrant script to set up Mesos (Marathon/Chronos)
> >> environment.
> >> > If you want to play with Mesos, please have a try.
> >> >
> >> > https://github.com/luckyfengyong/vagrant-mesos
> >> >
> >> > Thanks,
> >> >
> >> > Yong
> >> >
> >> > On Sun, Mar 1, 2015 at 6:32 AM, 杨浩 <ya...@gmail.com> wrote:
> >> >
> >> >> Mesos can group several frameworks in the same cluster, but unlike
> YARN
> >> >> divides resource to different queues, it divides resources to each
> >> >> framework. When using Slider, other framework or service can be put
> on
> >> the
> >> >> Hadoop cluster as a logic cluster. I think YARN && Slider is similar
> to
> >> >> MESOS, am I right?
> >> >>
> >> >
> >> >
> >>
> >
> >
>
Re: Is it similar to mesos
Posted by 杨浩 <ya...@gmail.com>.
Hi guys， I have found an article to support the opinions
http://www.infoq.com/news/2015/02/Myriad
2015-03-03 21:27 GMT+08:00 杨浩 <ya...@gmail.com>:
> Thanks for your sharing, and I will try it after Mesos needed.
>
> 2015-03-01 20:37 GMT+08:00 Yong Feng <fe...@gmail.com>:
>
>> A typo
>>
>> >>While Yarn and Tez are meta-framework in Yarn
>> While Slider and Tez are meta-framework in Yarn.
>>
>> Yong
>>
>> On Sun, Mar 1, 2015 at 7:36 AM, Yong Feng <fe...@gmail.com> wrote:
>>
>> > Both Mesos and Yarn are resource manager or resource scheduler. They use
>> > the different resource allocaiton protocol, push or request. Google
>> "Mesos
>> > Vs Yarn“, you could find more material to compare Mesos and Yarn. Mesos
>> > argues that Yarn is "BigData Application" and is able to run on top of
>> > Yarn. We actually also could run Mesos on top of Yarn.
>> >
>> > To help people moving applicaiton to Mesos, there are several
>> > meta-framework, such as Marathon and Chronos. While Yarn and Tez are
>> > meta-framework in Yarn.
>> >
>> > I create a vagrant script to set up Mesos (Marathon/Chronos)
>> environment.
>> > If you want to play with Mesos, please have a try.
>> >
>> > https://github.com/luckyfengyong/vagrant-mesos
>> >
>> > Thanks,
>> >
>> > Yong
>> >
>> > On Sun, Mar 1, 2015 at 6:32 AM, 杨浩 <ya...@gmail.com> wrote:
>> >
>> >> Mesos can group several frameworks in the same cluster, but unlike YARN
>> >> divides resource to different queues, it divides resources to each
>> >> framework. When using Slider, other framework or service can be put on
>> the
>> >> Hadoop cluster as a logic cluster. I think YARN && Slider is similar to
>> >> MESOS, am I right?
>> >>
>> >
>> >
>>
>
>
Re: Is it similar to mesos
Posted by 杨浩 <ya...@gmail.com>.
Thanks for your sharing, and I will try it after Mesos needed.
2015-03-01 20:37 GMT+08:00 Yong Feng <fe...@gmail.com>:
> A typo
>
> >>While Yarn and Tez are meta-framework in Yarn
> While Slider and Tez are meta-framework in Yarn.
>
> Yong
>
> On Sun, Mar 1, 2015 at 7:36 AM, Yong Feng <fe...@gmail.com> wrote:
>
> > Both Mesos and Yarn are resource manager or resource scheduler. They use
> > the different resource allocaiton protocol, push or request. Google
> "Mesos
> > Vs Yarn“, you could find more material to compare Mesos and Yarn. Mesos
> > argues that Yarn is "BigData Application" and is able to run on top of
> > Yarn. We actually also could run Mesos on top of Yarn.
> >
> > To help people moving applicaiton to Mesos, there are several
> > meta-framework, such as Marathon and Chronos. While Yarn and Tez are
> > meta-framework in Yarn.
> >
> > I create a vagrant script to set up Mesos (Marathon/Chronos) environment.
> > If you want to play with Mesos, please have a try.
> >
> > https://github.com/luckyfengyong/vagrant-mesos
> >
> > Thanks,
> >
> > Yong
> >
> > On Sun, Mar 1, 2015 at 6:32 AM, 杨浩 <ya...@gmail.com> wrote:
> >
> >> Mesos can group several frameworks in the same cluster, but unlike YARN
> >> divides resource to different queues, it divides resources to each
> >> framework. When using Slider, other framework or service can be put on
> the
> >> Hadoop cluster as a logic cluster. I think YARN && Slider is similar to
> >> MESOS, am I right?
> >>
> >
> >
>
Re: Is it similar to mesos
Posted by Yong Feng <fe...@gmail.com>.
A typo
>>While Yarn and Tez are meta-framework in Yarn
While Slider and Tez are meta-framework in Yarn.
Yong
On Sun, Mar 1, 2015 at 7:36 AM, Yong Feng <fe...@gmail.com> wrote:
> Both Mesos and Yarn are resource manager or resource scheduler. They use
> the different resource allocaiton protocol, push or request. Google "Mesos
> Vs Yarn“, you could find more material to compare Mesos and Yarn. Mesos
> argues that Yarn is "BigData Application" and is able to run on top of
> Yarn. We actually also could run Mesos on top of Yarn.
>
> To help people moving applicaiton to Mesos, there are several
> meta-framework, such as Marathon and Chronos. While Yarn and Tez are
> meta-framework in Yarn.
>
> I create a vagrant script to set up Mesos (Marathon/Chronos) environment.
> If you want to play with Mesos, please have a try.
>
> https://github.com/luckyfengyong/vagrant-mesos
>
> Thanks,
>
> Yong
>
> On Sun, Mar 1, 2015 at 6:32 AM, 杨浩 <ya...@gmail.com> wrote:
>
>> Mesos can group several frameworks in the same cluster, but unlike YARN
>> divides resource to different queues, it divides resources to each
>> framework. When using Slider, other framework or service can be put on the
>> Hadoop cluster as a logic cluster. I think YARN && Slider is similar to
>> MESOS, am I right?
>>
>
>
Re: Is it similar to mesos
Posted by Yong Feng <fe...@gmail.com>.
Both Mesos and Yarn are resource manager or resource scheduler. They use
the different resource allocaiton protocol, push or request. Google "Mesos
Vs Yarn“, you could find more material to compare Mesos and Yarn. Mesos
argues that Yarn is "BigData Application" and is able to run on top of
Yarn. We actually also could run Mesos on top of Yarn.
To help people moving applicaiton to Mesos, there are several
meta-framework, such as Marathon and Chronos. While Yarn and Tez are
meta-framework in Yarn.
I create a vagrant script to set up Mesos (Marathon/Chronos) environment.
If you want to play with Mesos, please have a try.
https://github.com/luckyfengyong/vagrant-mesos
Thanks,
Yong
On Sun, Mar 1, 2015 at 6:32 AM, 杨浩 <ya...@gmail.com> wrote:
> Mesos can group several frameworks in the same cluster, but unlike YARN
> divides resource to different queues, it divides resources to each
> framework. When using Slider, other framework or service can be put on the
> Hadoop cluster as a logic cluster. I think YARN && Slider is similar to
> MESOS, am I right?
>