You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mesos.apache.org by Brenden Matthews <br...@diddyinc.com> on 2013/07/25 03:52:16 UTC
Re: Review Request 12921: Allowed specification of mininum slot count.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/12921/
-----------------------------------------------------------
(Updated July 25, 2013, 1:52 a.m.)
Review request for mesos.
Changes
-------
Updated as per Vinod's comments.
Summary (updated)
-----------------
Allowed specification of mininum slot count.
Repository: mesos-git
Description (updated)
-------
Allowed specification of mininum slot count.
To ensure Hadoop jobs begin promptly, we can specify a minimum number of
'hot slots' to be available for use. This addresses the TaskTracker
spin up delay that exists with Hadoop on Mesos. This can be a nuisance
with lower latency applications, such as ad-hoc Hive queries.
Review: https://reviews.apache.org/r/12921
Diffs (updated)
-----
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java d9a9dbb57b10a8450ef8d822a041fa8d716bae4c
Diff: https://reviews.apache.org/r/12921/diff/
Testing
-------
make check && cd hadoop && make hadoop-2.0.0-mr1-cdh4.2.1
Thanks,
Brenden Matthews
Re: Review Request 12921: Allowed specification of mininum slot count.
Posted by Brenden Matthews <br...@diddyinc.com>.
> On July 26, 2013, 12:41 a.m., Ben Mahler wrote:
> > hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java, lines 362-367
> > <https://reviews.apache.org/r/12921/diff/2/?file=327721#file327721line362>
> >
> > It seems like the goal of the minimum here is to have a minimum amount of idle slots in the cluster? If so, we should likely update the configuration names to be:
> >
> > mapred.mesos.total.idle.map.slots.minimum
> > mapred.mesos.total.idle.reduce.slots.minimum
It's not the minimum idle slots. If I have a minimum of 100, but I have 10 idle slots, I still need 90 slots. The math seems to reflect this here, from what I see.
> On July 26, 2013, 12:41 a.m., Ben Mahler wrote:
> > hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java, lines 357-359
> > <https://reviews.apache.org/r/12921/diff/2/?file=327721#file327721line357>
> >
> > How about:
> > mapred.mesos.total.map.slots.minimum
> > mapred.mesos.total.reduce.slots.minimum
> >
> > And for these added variables, can you add the default values in mapred-site.xml.patch and add a human friendly comment for others to know how to use the knob?
I'll update mapred-site.xml in a later commit, if that's okay.
- Brenden
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/12921/#review23902
-----------------------------------------------------------
On July 30, 2013, 7:24 p.m., Brenden Matthews wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/12921/
> -----------------------------------------------------------
>
> (Updated July 30, 2013, 7:24 p.m.)
>
>
> Review request for mesos.
>
>
> Repository: mesos-git
>
>
> Description
> -------
>
> Allowed specification of mininum slot count.
>
> To ensure Hadoop jobs begin promptly, we can specify a minimum number of
> 'hot slots' to be available for use. This addresses the TaskTracker
> spin up delay that exists with Hadoop on Mesos. This can be a nuisance
> with lower latency applications, such as ad-hoc Hive queries.
>
> Review: https://reviews.apache.org/r/12921
>
>
> Diffs
> -----
>
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java 0c10874114cbfc0b8cb0dbb6dc83e6ebc09ad926
>
> Diff: https://reviews.apache.org/r/12921/diff/
>
>
> Testing
> -------
>
> make check && cd hadoop && make hadoop-2.0.0-mr1-cdh4.2.1
>
>
> Thanks,
>
> Brenden Matthews
>
>
Re: Review Request 12921: Allowed specification of mininum slot count.
Posted by Ben Mahler <be...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/12921/#review23902
-----------------------------------------------------------
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
<https://reviews.apache.org/r/12921/#comment47727>
How about:
mapred.mesos.total.map.slots.minimum
mapred.mesos.total.reduce.slots.minimum
And for these added variables, can you add the default values in mapred-site.xml.patch and add a human friendly comment for others to know how to use the knob?
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
<https://reviews.apache.org/r/12921/#comment47728>
It seems like the goal of the minimum here is to have a minimum amount of idle slots in the cluster? If so, we should likely update the configuration names to be:
mapred.mesos.total.idle.map.slots.minimum
mapred.mesos.total.idle.reduce.slots.minimum
- Ben Mahler
On July 25, 2013, 1:52 a.m., Brenden Matthews wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/12921/
> -----------------------------------------------------------
>
> (Updated July 25, 2013, 1:52 a.m.)
>
>
> Review request for mesos.
>
>
> Repository: mesos-git
>
>
> Description
> -------
>
> Allowed specification of mininum slot count.
>
> To ensure Hadoop jobs begin promptly, we can specify a minimum number of
> 'hot slots' to be available for use. This addresses the TaskTracker
> spin up delay that exists with Hadoop on Mesos. This can be a nuisance
> with lower latency applications, such as ad-hoc Hive queries.
>
> Review: https://reviews.apache.org/r/12921
>
>
> Diffs
> -----
>
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java d9a9dbb57b10a8450ef8d822a041fa8d716bae4c
>
> Diff: https://reviews.apache.org/r/12921/diff/
>
>
> Testing
> -------
>
> make check && cd hadoop && make hadoop-2.0.0-mr1-cdh4.2.1
>
>
> Thanks,
>
> Brenden Matthews
>
>
Re: Review Request 12921: Allowed specification of mininum slot count.
Posted by Ben Mahler <be...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/12921/#review24285
-----------------------------------------------------------
Ship it!
Ship It!
- Ben Mahler
On July 30, 2013, 7:24 p.m., Brenden Matthews wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/12921/
> -----------------------------------------------------------
>
> (Updated July 30, 2013, 7:24 p.m.)
>
>
> Review request for mesos.
>
>
> Repository: mesos-git
>
>
> Description
> -------
>
> Allowed specification of mininum slot count.
>
> To ensure Hadoop jobs begin promptly, we can specify a minimum number of
> 'hot slots' to be available for use. This addresses the TaskTracker
> spin up delay that exists with Hadoop on Mesos. This can be a nuisance
> with lower latency applications, such as ad-hoc Hive queries.
>
> Review: https://reviews.apache.org/r/12921
>
>
> Diffs
> -----
>
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java 0c10874114cbfc0b8cb0dbb6dc83e6ebc09ad926
>
> Diff: https://reviews.apache.org/r/12921/diff/
>
>
> Testing
> -------
>
> make check && cd hadoop && make hadoop-2.0.0-mr1-cdh4.2.1
>
>
> Thanks,
>
> Brenden Matthews
>
>
Re: Review Request 12921: Allowed specification of mininum slot count.
Posted by Brenden Matthews <br...@diddyinc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/12921/
-----------------------------------------------------------
(Updated July 30, 2013, 7:24 p.m.)
Review request for mesos.
Changes
-------
Updating as per Ben's comments, rebasing on master.
Repository: mesos-git
Description
-------
Allowed specification of mininum slot count.
To ensure Hadoop jobs begin promptly, we can specify a minimum number of
'hot slots' to be available for use. This addresses the TaskTracker
spin up delay that exists with Hadoop on Mesos. This can be a nuisance
with lower latency applications, such as ad-hoc Hive queries.
Review: https://reviews.apache.org/r/12921
Diffs (updated)
-----
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java 0c10874114cbfc0b8cb0dbb6dc83e6ebc09ad926
Diff: https://reviews.apache.org/r/12921/diff/
Testing
-------
make check && cd hadoop && make hadoop-2.0.0-mr1-cdh4.2.1
Thanks,
Brenden Matthews