You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mesos.apache.org by Brenden Matthews <br...@airbedandbreakfast.com> on 2013/07/20 02:00:16 UTC
Mesos Hadoop scheduler refactoring
Hey folks,
I've refactored and split the Mesos Hadoop scheduler into 2 parts (a fixed
and variable policy scheduler). I'll probably squash all my scheduler
related patches into one, but in the mean time please take a look here:
https://github.com/airbnb/mesos/blob/testing/hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
This is the same as my current scheduler, with the option to use a 'fixed'
scheduler which launches task trackers with a fixed number of slots like
the existing scheduler.
Here's a complete diff of all the changes against master:
https://github.com/airbnb/mesos/compare/master...testing
Thanks!
Brenden
Re: Mesos Hadoop scheduler refactoring
Posted by Vinod Kone <vi...@gmail.com>.
Hey Brenden. Can you mark the hadoop reviews that got submitted to master
as "Submitted" on ReviewBoard?
Thanks,
On Wed, Jul 24, 2013 at 4:30 PM, Brenden Matthews <
brenden.matthews@airbedandbreakfast.com> wrote:
> Hey guys,
>
> Everything is up-to-date in review board now. All I need is final approval
> & some ShipIts.
>
>
> On Tue, Jul 23, 2013 at 2:49 PM, Brenden Matthews <
> brenden.matthews@airbedandbreakfast.com> wrote:
>
> > Thanks Ben!
> >
> > I'm working on updating these ASAP. If there's anything else I can do to
> > expedite the process, let me know as well.
> >
> > Looking forward to getting this stuff squared away :)
> >
> >
> > On Mon, Jul 22, 2013 at 6:05 PM, Benjamin Mahler <
> > benjamin.mahler@gmail.com> wrote:
> >
> >> Great!! Looking forward to it.
> >>
> >> In the meantime, can you mark these submitted?
> >> Pretty-print resource type with parsing
> >> error.<https://reviews.apache.org/r/11486/>
> >> Make sure you clean up the cgroups test.<
> >> https://reviews.apache.org/r/11126/>
> >>
> >> That branch has a lot of changes, awesome! Looking at the related
> reviews,
> >> a bunch of these look good to go, minus small updates needed based on
> >> review comments, so updating those would be great! Please do reach out
> if
> >> you need help shepherding these along :)
> >>
> >>
> >> Starting with just the Mesos (non-Hadoop) reviews, the following look
> >> nearly finished:
> >>
> >> Terminate executors that aren't needed.<
> >> https://reviews.apache.org/r/11125/>
> >> (Ship It: small update needed)
> >>
> >> Increase slave timeout. <https://reviews.apache.org/r/11606/>
> >> (need to expose as flags)
> >>
> >> WebUI fix: list all framework tasks. <
> https://reviews.apache.org/r/12123/
> >> >
> >> (This was was also looked at by Ross, ultimately I think this one will
> >> be
> >> a back end change (see the comment), so can you drop this one in favor
> of
> >> that solution?)
> >>
> >> Fix broken build on GCC 4.8 due to -Wall.<
> >> https://reviews.apache.org/r/11974/>
> >> (Now boost 1.54 is out, so we should try that and see if it works!)
> >>
> >> Changed cgroups killTasks() sequence. <
> >> https://reviews.apache.org/r/11131/>
> >> (Just small small updates needed)
> >>
> >> Add --enable-debug configure flag. <https://reviews.apache.org/r/11605>
> >> (Minor comments)
> >>
> >> More cgroup killTask() logging. <https://reviews.apache.org/r/11129/>
> >> (Still needed? Can we discard this one?)
> >>
> >> Check for Python.h in configure script.<
> >> https://reviews.apache.org/r/11114/>
> >> (This one was tricky, needs a portable solution)
> >>
> >> Simplify os::hostname() in libstout.
> >> <https://reviews.apache.org/r/11108/> (benh
> >> will be taking a look at this one, as he wrote that code)
> >>
> >>
> >> These probably require some discussion:
> >>
> >> Shuffle offers before sending to
> >> frameworks.<https://reviews.apache.org/r/11121/>
> >> (This one would benefit from a discussion on the mailing list, would
> be
> >> good to sync up on this)
> >>
> >> Use hostname instead of IP for redirection.<
> >> https://reviews.apache.org/r/11975/>
> >> (I remember this one causing some discussion around using protobuf,
> >> probably needs some agreement on the technique via the mailing list or
> by
> >> pinging vinod/benh).
> >>
> >> On Fri, Jul 19, 2013 at 5:00 PM, Brenden Matthews <
> >> brenden.matthews@airbedandbreakfast.com> wrote:
> >>
> >> > Hey folks,
> >> >
> >> > I've refactored and split the Mesos Hadoop scheduler into 2 parts (a
> >> fixed
> >> > and variable policy scheduler). I'll probably squash all my scheduler
> >> > related patches into one, but in the mean time please take a look
> here:
> >> >
> >> >
> >> >
> >>
> https://github.com/airbnb/mesos/blob/testing/hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
> >> >
> >> > This is the same as my current scheduler, with the option to use a
> >> 'fixed'
> >> > scheduler which launches task trackers with a fixed number of slots
> like
> >> > the existing scheduler.
> >> >
> >> > Here's a complete diff of all the changes against master:
> >> >
> >> > https://github.com/airbnb/mesos/compare/master...testing
> >> >
> >> > Thanks!
> >> >
> >> > Brenden
> >> >
> >>
> >
> >
>
Re: Mesos Hadoop scheduler refactoring
Posted by Brenden Matthews <br...@airbedandbreakfast.com>.
Hey guys,
Everything is up-to-date in review board now. All I need is final approval
& some ShipIts.
On Tue, Jul 23, 2013 at 2:49 PM, Brenden Matthews <
brenden.matthews@airbedandbreakfast.com> wrote:
> Thanks Ben!
>
> I'm working on updating these ASAP. If there's anything else I can do to
> expedite the process, let me know as well.
>
> Looking forward to getting this stuff squared away :)
>
>
> On Mon, Jul 22, 2013 at 6:05 PM, Benjamin Mahler <
> benjamin.mahler@gmail.com> wrote:
>
>> Great!! Looking forward to it.
>>
>> In the meantime, can you mark these submitted?
>> Pretty-print resource type with parsing
>> error.<https://reviews.apache.org/r/11486/>
>> Make sure you clean up the cgroups test.<
>> https://reviews.apache.org/r/11126/>
>>
>> That branch has a lot of changes, awesome! Looking at the related reviews,
>> a bunch of these look good to go, minus small updates needed based on
>> review comments, so updating those would be great! Please do reach out if
>> you need help shepherding these along :)
>>
>>
>> Starting with just the Mesos (non-Hadoop) reviews, the following look
>> nearly finished:
>>
>> Terminate executors that aren't needed.<
>> https://reviews.apache.org/r/11125/>
>> (Ship It: small update needed)
>>
>> Increase slave timeout. <https://reviews.apache.org/r/11606/>
>> (need to expose as flags)
>>
>> WebUI fix: list all framework tasks. <https://reviews.apache.org/r/12123/
>> >
>> (This was was also looked at by Ross, ultimately I think this one will
>> be
>> a back end change (see the comment), so can you drop this one in favor of
>> that solution?)
>>
>> Fix broken build on GCC 4.8 due to -Wall.<
>> https://reviews.apache.org/r/11974/>
>> (Now boost 1.54 is out, so we should try that and see if it works!)
>>
>> Changed cgroups killTasks() sequence. <
>> https://reviews.apache.org/r/11131/>
>> (Just small small updates needed)
>>
>> Add --enable-debug configure flag. <https://reviews.apache.org/r/11605>
>> (Minor comments)
>>
>> More cgroup killTask() logging. <https://reviews.apache.org/r/11129/>
>> (Still needed? Can we discard this one?)
>>
>> Check for Python.h in configure script.<
>> https://reviews.apache.org/r/11114/>
>> (This one was tricky, needs a portable solution)
>>
>> Simplify os::hostname() in libstout.
>> <https://reviews.apache.org/r/11108/> (benh
>> will be taking a look at this one, as he wrote that code)
>>
>>
>> These probably require some discussion:
>>
>> Shuffle offers before sending to
>> frameworks.<https://reviews.apache.org/r/11121/>
>> (This one would benefit from a discussion on the mailing list, would be
>> good to sync up on this)
>>
>> Use hostname instead of IP for redirection.<
>> https://reviews.apache.org/r/11975/>
>> (I remember this one causing some discussion around using protobuf,
>> probably needs some agreement on the technique via the mailing list or by
>> pinging vinod/benh).
>>
>> On Fri, Jul 19, 2013 at 5:00 PM, Brenden Matthews <
>> brenden.matthews@airbedandbreakfast.com> wrote:
>>
>> > Hey folks,
>> >
>> > I've refactored and split the Mesos Hadoop scheduler into 2 parts (a
>> fixed
>> > and variable policy scheduler). I'll probably squash all my scheduler
>> > related patches into one, but in the mean time please take a look here:
>> >
>> >
>> >
>> https://github.com/airbnb/mesos/blob/testing/hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
>> >
>> > This is the same as my current scheduler, with the option to use a
>> 'fixed'
>> > scheduler which launches task trackers with a fixed number of slots like
>> > the existing scheduler.
>> >
>> > Here's a complete diff of all the changes against master:
>> >
>> > https://github.com/airbnb/mesos/compare/master...testing
>> >
>> > Thanks!
>> >
>> > Brenden
>> >
>>
>
>
Re: Mesos Hadoop scheduler refactoring
Posted by Brenden Matthews <br...@airbedandbreakfast.com>.
Thanks Ben!
I'm working on updating these ASAP. If there's anything else I can do to
expedite the process, let me know as well.
Looking forward to getting this stuff squared away :)
On Mon, Jul 22, 2013 at 6:05 PM, Benjamin Mahler
<be...@gmail.com>wrote:
> Great!! Looking forward to it.
>
> In the meantime, can you mark these submitted?
> Pretty-print resource type with parsing
> error.<https://reviews.apache.org/r/11486/>
> Make sure you clean up the cgroups test.<
> https://reviews.apache.org/r/11126/>
>
> That branch has a lot of changes, awesome! Looking at the related reviews,
> a bunch of these look good to go, minus small updates needed based on
> review comments, so updating those would be great! Please do reach out if
> you need help shepherding these along :)
>
>
> Starting with just the Mesos (non-Hadoop) reviews, the following look
> nearly finished:
>
> Terminate executors that aren't needed.<
> https://reviews.apache.org/r/11125/>
> (Ship It: small update needed)
>
> Increase slave timeout. <https://reviews.apache.org/r/11606/>
> (need to expose as flags)
>
> WebUI fix: list all framework tasks. <https://reviews.apache.org/r/12123/>
> (This was was also looked at by Ross, ultimately I think this one will be
> a back end change (see the comment), so can you drop this one in favor of
> that solution?)
>
> Fix broken build on GCC 4.8 due to -Wall.<
> https://reviews.apache.org/r/11974/>
> (Now boost 1.54 is out, so we should try that and see if it works!)
>
> Changed cgroups killTasks() sequence. <https://reviews.apache.org/r/11131/
> >
> (Just small small updates needed)
>
> Add --enable-debug configure flag. <https://reviews.apache.org/r/11605>
> (Minor comments)
>
> More cgroup killTask() logging. <https://reviews.apache.org/r/11129/>
> (Still needed? Can we discard this one?)
>
> Check for Python.h in configure script.<
> https://reviews.apache.org/r/11114/>
> (This one was tricky, needs a portable solution)
>
> Simplify os::hostname() in libstout.
> <https://reviews.apache.org/r/11108/> (benh
> will be taking a look at this one, as he wrote that code)
>
>
> These probably require some discussion:
>
> Shuffle offers before sending to
> frameworks.<https://reviews.apache.org/r/11121/>
> (This one would benefit from a discussion on the mailing list, would be
> good to sync up on this)
>
> Use hostname instead of IP for redirection.<
> https://reviews.apache.org/r/11975/>
> (I remember this one causing some discussion around using protobuf,
> probably needs some agreement on the technique via the mailing list or by
> pinging vinod/benh).
>
> On Fri, Jul 19, 2013 at 5:00 PM, Brenden Matthews <
> brenden.matthews@airbedandbreakfast.com> wrote:
>
> > Hey folks,
> >
> > I've refactored and split the Mesos Hadoop scheduler into 2 parts (a
> fixed
> > and variable policy scheduler). I'll probably squash all my scheduler
> > related patches into one, but in the mean time please take a look here:
> >
> >
> >
> https://github.com/airbnb/mesos/blob/testing/hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
> >
> > This is the same as my current scheduler, with the option to use a
> 'fixed'
> > scheduler which launches task trackers with a fixed number of slots like
> > the existing scheduler.
> >
> > Here's a complete diff of all the changes against master:
> >
> > https://github.com/airbnb/mesos/compare/master...testing
> >
> > Thanks!
> >
> > Brenden
> >
>
Re: Mesos Hadoop scheduler refactoring
Posted by Benjamin Mahler <be...@gmail.com>.
Great!! Looking forward to it.
In the meantime, can you mark these submitted?
Pretty-print resource type with parsing
error.<https://reviews.apache.org/r/11486/>
Make sure you clean up the cgroups test.<https://reviews.apache.org/r/11126/>
That branch has a lot of changes, awesome! Looking at the related reviews,
a bunch of these look good to go, minus small updates needed based on
review comments, so updating those would be great! Please do reach out if
you need help shepherding these along :)
Starting with just the Mesos (non-Hadoop) reviews, the following look
nearly finished:
Terminate executors that aren't needed.<https://reviews.apache.org/r/11125/>
(Ship It: small update needed)
Increase slave timeout. <https://reviews.apache.org/r/11606/>
(need to expose as flags)
WebUI fix: list all framework tasks. <https://reviews.apache.org/r/12123/>
(This was was also looked at by Ross, ultimately I think this one will be
a back end change (see the comment), so can you drop this one in favor of
that solution?)
Fix broken build on GCC 4.8 due to -Wall.<https://reviews.apache.org/r/11974/>
(Now boost 1.54 is out, so we should try that and see if it works!)
Changed cgroups killTasks() sequence. <https://reviews.apache.org/r/11131/>
(Just small small updates needed)
Add --enable-debug configure flag. <https://reviews.apache.org/r/11605>
(Minor comments)
More cgroup killTask() logging. <https://reviews.apache.org/r/11129/>
(Still needed? Can we discard this one?)
Check for Python.h in configure script.<https://reviews.apache.org/r/11114/>
(This one was tricky, needs a portable solution)
Simplify os::hostname() in libstout.
<https://reviews.apache.org/r/11108/> (benh
will be taking a look at this one, as he wrote that code)
These probably require some discussion:
Shuffle offers before sending to
frameworks.<https://reviews.apache.org/r/11121/>
(This one would benefit from a discussion on the mailing list, would be
good to sync up on this)
Use hostname instead of IP for redirection.<https://reviews.apache.org/r/11975/>
(I remember this one causing some discussion around using protobuf,
probably needs some agreement on the technique via the mailing list or by
pinging vinod/benh).
On Fri, Jul 19, 2013 at 5:00 PM, Brenden Matthews <
brenden.matthews@airbedandbreakfast.com> wrote:
> Hey folks,
>
> I've refactored and split the Mesos Hadoop scheduler into 2 parts (a fixed
> and variable policy scheduler). I'll probably squash all my scheduler
> related patches into one, but in the mean time please take a look here:
>
>
> https://github.com/airbnb/mesos/blob/testing/hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
>
> This is the same as my current scheduler, with the option to use a 'fixed'
> scheduler which launches task trackers with a fixed number of slots like
> the existing scheduler.
>
> Here's a complete diff of all the changes against master:
>
> https://github.com/airbnb/mesos/compare/master...testing
>
> Thanks!
>
> Brenden
>