You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mesos.apache.org by Brenden Matthews <br...@diddyinc.com> on 2013/06/03 19:57:15 UTC
Re: Review Request: Increase JVM memory overhead.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11130/
-----------------------------------------------------------
(Updated June 3, 2013, 5:57 p.m.)
Review request for mesos.
Summary (updated)
-----------------
Increase JVM memory overhead.
Description (updated)
-------
Increase JVM memory overhead.
Review: https://reviews.apache.org/r/11130
Diffs (updated)
-----
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java afe401f5265e3d9494af7eace42eec45943184a3
Diff: https://reviews.apache.org/r/11130/diff/
Testing
-------
Used in production at airbnb.
Thanks,
Brenden Matthews
Re: Review Request: Increase JVM memory overhead.
Posted by Brenden Matthews <br...@diddyinc.com>.
> On June 4, 2013, 12:01 a.m., Ben Mahler wrote:
> > hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java, line 53
> > <https://reviews.apache.org/r/11130/diff/2/?file=300042#file300042line53>
> >
> > Seems like a drastic bump without the context, how often were you seeing OOMs?
On very large jobs they happened quite frequently. I'm going to turn this into a config parameter instead.
- Brenden
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11130/#review21372
-----------------------------------------------------------
On June 6, 2013, 2:10 a.m., Brenden Matthews wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11130/
> -----------------------------------------------------------
>
> (Updated June 6, 2013, 2:10 a.m.)
>
>
> Review request for mesos.
>
>
> Description
> -------
>
> Increase JVM memory overhead.
>
> Review: https://reviews.apache.org/r/11130
>
>
> Diffs
> -----
>
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java afe401f5265e3d9494af7eace42eec45943184a3
>
> Diff: https://reviews.apache.org/r/11130/diff/
>
>
> Testing
> -------
>
> Used in production at airbnb.
>
>
> Thanks,
>
> Brenden Matthews
>
>
Re: Review Request: Increase JVM memory overhead.
Posted by Ben Mahler <be...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11130/#review21372
-----------------------------------------------------------
Can you add more context to this review?
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
<https://reviews.apache.org/r/11130/#comment44306>
Seems like a drastic bump without the context, how often were you seeing OOMs?
- Ben Mahler
On June 3, 2013, 5:57 p.m., Brenden Matthews wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11130/
> -----------------------------------------------------------
>
> (Updated June 3, 2013, 5:57 p.m.)
>
>
> Review request for mesos.
>
>
> Description
> -------
>
> Increase JVM memory overhead.
>
> Review: https://reviews.apache.org/r/11130
>
>
> Diffs
> -----
>
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java afe401f5265e3d9494af7eace42eec45943184a3
>
> Diff: https://reviews.apache.org/r/11130/diff/
>
>
> Testing
> -------
>
> Used in production at airbnb.
>
>
> Thanks,
>
> Brenden Matthews
>
>
Re: Review Request: Allow JVM memory vars to be configured.
Posted by Brenden Matthews <br...@diddyinc.com>.
> On June 11, 2013, 8:26 p.m., Ben Mahler wrote:
> > hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java, line 53
> > <https://reviews.apache.org/r/11130/diff/2/?file=300042#file300042line53>
> >
> > There have been several internal discussions at Twitter with respect to JVM memory overhead. There was no clear answer but one recommendation was to set it to 5-10% of the heap size.
> >
> > So I would be ok with this being:
> >
> > JVM_MEM_OVERHEAD_PERCENT = 0.1;
> >
> > and then
> >
> > private static final int TASKTRACKER_MEM =
> > TASKTRACKER_JVM_HEAP * (1.0 + JVM_MEM_OVERHEAD);
> >
> > Thoughts?
> >
That's probably a better approach. I'll update it shortly.
- Brenden
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11130/#review21735
-----------------------------------------------------------
On June 11, 2013, 8:16 p.m., Brenden Matthews wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11130/
> -----------------------------------------------------------
>
> (Updated June 11, 2013, 8:16 p.m.)
>
>
> Review request for mesos.
>
>
> Description
> -------
>
> Allow JVM memory vars to be configured.
>
> Review: https://reviews.apache.org/r/11130
>
>
> Diffs
> -----
>
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java afe401f5265e3d9494af7eace42eec45943184a3
>
> Diff: https://reviews.apache.org/r/11130/diff/
>
>
> Testing
> -------
>
> Used in production at airbnb.
>
> make -j10 check && cd hadoop && make hadoop-2.0.0-mr1-cdh4.2.1 && make hadoop-0.20.205.0 && make hadoop-0.20.2-cdh3u3
>
>
> Thanks,
>
> Brenden Matthews
>
>
Re: Review Request: Allow JVM memory vars to be configured.
Posted by Ben Mahler <be...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11130/#review21735
-----------------------------------------------------------
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
<https://reviews.apache.org/r/11130/#comment44912>
There have been several internal discussions at Twitter with respect to JVM memory overhead. There was no clear answer but one recommendation was to set it to 5-10% of the heap size.
So I would be ok with this being:
JVM_MEM_OVERHEAD_PERCENT = 0.1;
and then
private static final int TASKTRACKER_MEM =
TASKTRACKER_JVM_HEAP * (1.0 + JVM_MEM_OVERHEAD);
Thoughts?
- Ben Mahler
On June 11, 2013, 8:16 p.m., Brenden Matthews wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11130/
> -----------------------------------------------------------
>
> (Updated June 11, 2013, 8:16 p.m.)
>
>
> Review request for mesos.
>
>
> Description
> -------
>
> Allow JVM memory vars to be configured.
>
> Review: https://reviews.apache.org/r/11130
>
>
> Diffs
> -----
>
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java afe401f5265e3d9494af7eace42eec45943184a3
>
> Diff: https://reviews.apache.org/r/11130/diff/
>
>
> Testing
> -------
>
> Used in production at airbnb.
>
> make -j10 check && cd hadoop && make hadoop-2.0.0-mr1-cdh4.2.1 && make hadoop-0.20.205.0 && make hadoop-0.20.2-cdh3u3
>
>
> Thanks,
>
> Brenden Matthews
>
>
Re: Review Request: Allow JVM memory vars to be configured.
Posted by Brenden Matthews <br...@diddyinc.com>.
> On June 14, 2013, 8:51 p.m., Ben Mahler wrote:
> > hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java, line 461
> > <https://reviews.apache.org/r/11130/diff/6/?file=303789#file303789line461>
> >
> > Why did you want to configure this one? Just looking to understand the context.
I think it's pretty important to be able to adjust the tasktracker heap size. It's something that needs to be tuned to suit your particular application.
> On June 14, 2013, 8:51 p.m., Ben Mahler wrote:
> > hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java, line 447
> > <https://reviews.apache.org/r/11130/diff/6/?file=303789#file303789line447>
> >
> > Let's hold off on adding another configuration variable, until we see whether 10% has issues. Ultimately, I think this is something that the scheduler should know rather than the users.
I've been running with this as a default so far with good results.
I feel like there's a Hadoop convention to allow configuration of most 'knobs', and this seems like an important one. I'm not opposed to just leaving it as a default, but I also don't think we're doing harm by allowing it to be configured.
- Brenden
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11130/#review21919
-----------------------------------------------------------
On June 11, 2013, 9:20 p.m., Brenden Matthews wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11130/
> -----------------------------------------------------------
>
> (Updated June 11, 2013, 9:20 p.m.)
>
>
> Review request for mesos.
>
>
> Description
> -------
>
> Allow JVM memory vars to be configured.
>
> Review: https://reviews.apache.org/r/11130
>
>
> Diffs
> -----
>
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java afe401f5265e3d9494af7eace42eec45943184a3
>
> Diff: https://reviews.apache.org/r/11130/diff/
>
>
> Testing
> -------
>
> Used in production at airbnb.
>
> make -j10 check && cd hadoop && make hadoop-2.0.0-mr1-cdh4.2.1 && make hadoop-0.20.205.0 && make hadoop-0.20.2-cdh3u3
>
>
> Thanks,
>
> Brenden Matthews
>
>
Re: Review Request: Allow JVM memory vars to be configured.
Posted by Ben Mahler <be...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11130/#review21919
-----------------------------------------------------------
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
<https://reviews.apache.org/r/11130/#comment45217>
Let's hold off on adding another configuration variable, until we see whether 10% has issues. Ultimately, I think this is something that the scheduler should know rather than the users.
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
<https://reviews.apache.org/r/11130/#comment45218>
Why did you want to configure this one? Just looking to understand the context.
- Ben Mahler
On June 11, 2013, 9:20 p.m., Brenden Matthews wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11130/
> -----------------------------------------------------------
>
> (Updated June 11, 2013, 9:20 p.m.)
>
>
> Review request for mesos.
>
>
> Description
> -------
>
> Allow JVM memory vars to be configured.
>
> Review: https://reviews.apache.org/r/11130
>
>
> Diffs
> -----
>
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java afe401f5265e3d9494af7eace42eec45943184a3
>
> Diff: https://reviews.apache.org/r/11130/diff/
>
>
> Testing
> -------
>
> Used in production at airbnb.
>
> make -j10 check && cd hadoop && make hadoop-2.0.0-mr1-cdh4.2.1 && make hadoop-0.20.205.0 && make hadoop-0.20.2-cdh3u3
>
>
> Thanks,
>
> Brenden Matthews
>
>
Re: Review Request 11130: Allowed some JVM memory vars to be configured.
Posted by Ben Mahler <be...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11130/#review23990
-----------------------------------------------------------
Ship it!
hadoop/mapred-site.xml.patch
<https://reviews.apache.org/r/11130/#comment47806>
Thanks!
- Ben Mahler
On July 25, 2013, 1:27 a.m., Brenden Matthews wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11130/
> -----------------------------------------------------------
>
> (Updated July 25, 2013, 1:27 a.m.)
>
>
> Review request for mesos.
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Allowed some JVM memory vars to be configured.
>
> Also modified the slot CPU/mem values to be more sane defaults.
>
> Review: https://reviews.apache.org/r/11130
>
>
> Diffs
> -----
>
> hadoop/mapred-site.xml.patch 8b3997944f60a9d91c81089a4450e4820526dde3
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java 279f84e0f0c43ad3cfd9e4442010e706ee3565d9
>
> Diff: https://reviews.apache.org/r/11130/diff/
>
>
> Testing
> -------
>
> Used in production at airbnb.
>
> make -j10 check && cd hadoop && make hadoop-2.0.0-mr1-cdh4.2.1 && make hadoop-0.20.205.0 && make hadoop-0.20.2-cdh3u3
>
>
> Thanks,
>
> Brenden Matthews
>
>
Re: Review Request 11130: Allowed some JVM memory vars to be configured.
Posted by Brenden Matthews <br...@diddyinc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11130/
-----------------------------------------------------------
(Updated July 30, 2013, 7:45 p.m.)
Review request for mesos.
Changes
-------
Add dependency.
Repository: mesos
Description
-------
Allowed some JVM memory vars to be configured.
Also modified the slot CPU/mem values to be more sane defaults.
Review: https://reviews.apache.org/r/11130
Diffs
-----
hadoop/mapred-site.xml.patch 8b3997944f60a9d91c81089a4450e4820526dde3
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java 279f84e0f0c43ad3cfd9e4442010e706ee3565d9
Diff: https://reviews.apache.org/r/11130/diff/
Testing
-------
Used in production at airbnb.
make -j10 check && cd hadoop && make hadoop-2.0.0-mr1-cdh4.2.1 && make hadoop-0.20.205.0 && make hadoop-0.20.2-cdh3u3
Thanks,
Brenden Matthews
Re: Review Request 11130: Allowed some JVM memory vars to be configured.
Posted by Brenden Matthews <br...@diddyinc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11130/
-----------------------------------------------------------
(Updated July 25, 2013, 1:27 a.m.)
Review request for mesos.
Changes
-------
Updated as per Vinod's comments.
Summary (updated)
-----------------
Allowed some JVM memory vars to be configured.
Repository: mesos
Description (updated)
-------
Allowed some JVM memory vars to be configured.
Also modified the slot CPU/mem values to be more sane defaults.
Review: https://reviews.apache.org/r/11130
Diffs (updated)
-----
hadoop/mapred-site.xml.patch 8b3997944f60a9d91c81089a4450e4820526dde3
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java 279f84e0f0c43ad3cfd9e4442010e706ee3565d9
Diff: https://reviews.apache.org/r/11130/diff/
Testing
-------
Used in production at airbnb.
make -j10 check && cd hadoop && make hadoop-2.0.0-mr1-cdh4.2.1 && make hadoop-0.20.205.0 && make hadoop-0.20.2-cdh3u3
Thanks,
Brenden Matthews
Re: Review Request 11130: Allowed some JVM memory vars to be configured.
Posted by Brenden Matthews <br...@diddyinc.com>.
> On July 25, 2013, 12:26 a.m., Vinod Kone wrote:
> > hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java, lines 453-454
> > <https://reviews.apache.org/r/11130/diff/8/?file=327647#file327647line453>
> >
> > how about
> >
> > Math.round((1-JVM_MEM_OVERHEAD_PERCENT_DEFAULT)) * slotMem
> >
> > same for taskTrackerMem below?
I did this in a later commit.
- Brenden
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11130/#review23802
-----------------------------------------------------------
On July 25, 2013, 1:27 a.m., Brenden Matthews wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11130/
> -----------------------------------------------------------
>
> (Updated July 25, 2013, 1:27 a.m.)
>
>
> Review request for mesos.
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Allowed some JVM memory vars to be configured.
>
> Also modified the slot CPU/mem values to be more sane defaults.
>
> Review: https://reviews.apache.org/r/11130
>
>
> Diffs
> -----
>
> hadoop/mapred-site.xml.patch 8b3997944f60a9d91c81089a4450e4820526dde3
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java 279f84e0f0c43ad3cfd9e4442010e706ee3565d9
>
> Diff: https://reviews.apache.org/r/11130/diff/
>
>
> Testing
> -------
>
> Used in production at airbnb.
>
> make -j10 check && cd hadoop && make hadoop-2.0.0-mr1-cdh4.2.1 && make hadoop-0.20.205.0 && make hadoop-0.20.2-cdh3u3
>
>
> Thanks,
>
> Brenden Matthews
>
>
Re: Review Request 11130: Allow some JVM memory vars to be configured.
Posted by Vinod Kone <vi...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11130/#review23802
-----------------------------------------------------------
Ship it!
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
<https://reviews.apache.org/r/11130/#comment47638>
Make this 1 to match the config?
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
<https://reviews.apache.org/r/11130/#comment47639>
ditto
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
<https://reviews.apache.org/r/11130/#comment47640>
how about
Math.round((1-JVM_MEM_OVERHEAD_PERCENT_DEFAULT)) * slotMem
same for taskTrackerMem below?
- Vinod Kone
On July 24, 2013, 11 p.m., Brenden Matthews wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11130/
> -----------------------------------------------------------
>
> (Updated July 24, 2013, 11 p.m.)
>
>
> Review request for mesos.
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Allow some JVM memory vars to be configured.
>
> Also modified the slot CPU/mem values to be more sane defaults.
>
> Review: https://reviews.apache.org/r/11130
>
>
> Diffs
> -----
>
> hadoop/mapred-site.xml.patch 8b3997944f60a9d91c81089a4450e4820526dde3
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java 279f84e0f0c43ad3cfd9e4442010e706ee3565d9
>
> Diff: https://reviews.apache.org/r/11130/diff/
>
>
> Testing
> -------
>
> Used in production at airbnb.
>
> make -j10 check && cd hadoop && make hadoop-2.0.0-mr1-cdh4.2.1 && make hadoop-0.20.205.0 && make hadoop-0.20.2-cdh3u3
>
>
> Thanks,
>
> Brenden Matthews
>
>
Re: Review Request 11130: Allow some JVM memory vars to be configured.
Posted by Brenden Matthews <br...@diddyinc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11130/
-----------------------------------------------------------
(Updated July 24, 2013, 11 p.m.)
Review request for mesos.
Changes
-------
Rebasing on master, updating as per Ben's comments.
Repository: mesos
Description (updated)
-------
Allow some JVM memory vars to be configured.
Also modified the slot CPU/mem values to be more sane defaults.
Review: https://reviews.apache.org/r/11130
Diffs (updated)
-----
hadoop/mapred-site.xml.patch 8b3997944f60a9d91c81089a4450e4820526dde3
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java 279f84e0f0c43ad3cfd9e4442010e706ee3565d9
Diff: https://reviews.apache.org/r/11130/diff/
Testing
-------
Used in production at airbnb.
make -j10 check && cd hadoop && make hadoop-2.0.0-mr1-cdh4.2.1 && make hadoop-0.20.205.0 && make hadoop-0.20.2-cdh3u3
Thanks,
Brenden Matthews
Re: Review Request 11130: Allow some JVM memory vars to be configured.
Posted by Brenden Matthews <br...@diddyinc.com>.
> On June 27, 2013, 7:07 p.m., Ben Mahler wrote:
> > hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java, lines 452-453
> > <https://reviews.apache.org/r/11130/diff/7/?file=311244#file311244line452>
> >
> > Include the overhead in the default computation. To get the heap value you would subtract the overhead from slotMem (as done previously).
> >
> > This is because 'cpus', 'disk', and 'mem' are the raw resources allocated by mesos, so it would be nice to keep that meaning consistent. We could hide 'mem' and introduce 'heap', 'jvm_overhead', etc, should the need arise later.
I'm not sure I understand why you raised this as an issue. What's there to change here? As far as I can tell, it behaves as you describe.
> On June 27, 2013, 7:07 p.m., Ben Mahler wrote:
> > hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java, lines 457-460
> > <https://reviews.apache.org/r/11130/diff/7/?file=311244#file311244line457>
> >
> > I think we want to keep the overhead as part of the default computation. So that 'mem' continues to mean the resources allocated by mesos.
> >
> > Also, we should add a default value in mapred-site.xml for this new knob. Explaining that 10% of 'mem' is used as overhead for the JVM.
> >
> > As you said, we may need to expose the overhead percent knob, but let's hope not! :)
See comment above.
- Brenden
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11130/#review22479
-----------------------------------------------------------
On June 26, 2013, 8:20 p.m., Brenden Matthews wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11130/
> -----------------------------------------------------------
>
> (Updated June 26, 2013, 8:20 p.m.)
>
>
> Review request for mesos.
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Allow some JVM memory vars to be configured.
>
> Review: https://reviews.apache.org/r/11130
>
>
> Diffs
> -----
>
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java c5a63bc7a172c44cc11d6bdd6b2eb4bc3df35cb7
>
> Diff: https://reviews.apache.org/r/11130/diff/
>
>
> Testing
> -------
>
> Used in production at airbnb.
>
> make -j10 check && cd hadoop && make hadoop-2.0.0-mr1-cdh4.2.1 && make hadoop-0.20.205.0 && make hadoop-0.20.2-cdh3u3
>
>
> Thanks,
>
> Brenden Matthews
>
>
Re: Review Request 11130: Allow some JVM memory vars to be configured.
Posted by Ben Mahler <be...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11130/#review22479
-----------------------------------------------------------
Cool, this is looking pretty good to me now, I like adding the new knob for task tracker memory!
My comments are for keeping the meaning of 'mem' to be the directly related to 'mem' as a resource in mesos. This will keep it clear as to what memory limit will be used for the TaskTracker, and administrators will be forced to think about the JVM overhead should they need to change the values.
The downside is that we're decreeing 10% as the overhead. If that turns out to be a bad thing, we can add that as well and revisit the knobs.
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
<https://reviews.apache.org/r/11130/#comment46023>
Can you update the comment in mapred-site.xml to reflect that 10% of mapred.mesos.tasktracker.mem will be allocated only for JVM overhead?
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
<https://reviews.apache.org/r/11130/#comment46024>
Include the overhead in the default computation. To get the heap value you would subtract the overhead from slotMem (as done previously).
This is because 'cpus', 'disk', and 'mem' are the raw resources allocated by mesos, so it would be nice to keep that meaning consistent. We could hide 'mem' and introduce 'heap', 'jvm_overhead', etc, should the need arise later.
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
<https://reviews.apache.org/r/11130/#comment46025>
I think we want to keep the overhead as part of the default computation. So that 'mem' continues to mean the resources allocated by mesos.
Also, we should add a default value in mapred-site.xml for this new knob. Explaining that 10% of 'mem' is used as overhead for the JVM.
As you said, we may need to expose the overhead percent knob, but let's hope not! :)
- Ben Mahler
On June 26, 2013, 8:20 p.m., Brenden Matthews wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11130/
> -----------------------------------------------------------
>
> (Updated June 26, 2013, 8:20 p.m.)
>
>
> Review request for mesos.
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Allow some JVM memory vars to be configured.
>
> Review: https://reviews.apache.org/r/11130
>
>
> Diffs
> -----
>
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java c5a63bc7a172c44cc11d6bdd6b2eb4bc3df35cb7
>
> Diff: https://reviews.apache.org/r/11130/diff/
>
>
> Testing
> -------
>
> Used in production at airbnb.
>
> make -j10 check && cd hadoop && make hadoop-2.0.0-mr1-cdh4.2.1 && make hadoop-0.20.205.0 && make hadoop-0.20.2-cdh3u3
>
>
> Thanks,
>
> Brenden Matthews
>
>
Re: Review Request 11130: Allow some JVM memory vars to be configured.
Posted by Brenden Matthews <br...@diddyinc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11130/
-----------------------------------------------------------
(Updated June 26, 2013, 8:20 p.m.)
Review request for mesos.
Changes
-------
Dropped the overhead percentage config option as per Ben's comments.
Summary (updated)
-----------------
Allow some JVM memory vars to be configured.
Repository: mesos
Description (updated)
-------
Allow some JVM memory vars to be configured.
Review: https://reviews.apache.org/r/11130
Diffs (updated)
-----
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java c5a63bc7a172c44cc11d6bdd6b2eb4bc3df35cb7
Diff: https://reviews.apache.org/r/11130/diff/
Testing
-------
Used in production at airbnb.
make -j10 check && cd hadoop && make hadoop-2.0.0-mr1-cdh4.2.1 && make hadoop-0.20.205.0 && make hadoop-0.20.2-cdh3u3
Thanks,
Brenden Matthews
Re: Review Request 11130: Allow JVM memory vars to be configured.
Posted by Brenden Matthews <br...@diddyinc.com>.
> On June 20, 2013, 9:59 p.m., Ben Mahler wrote:
> > hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java, line 447
> > <https://reviews.apache.org/r/11130/diff/6/?file=303789#file303789line447>
> >
> > I guess that's one convention I don't like, since it results in a proliferation of knobs.
I'm willing to drop this one, since I'm having good results with the default of 10%.
> On June 20, 2013, 9:59 p.m., Ben Mahler wrote:
> > hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java, line 461
> > <https://reviews.apache.org/r/11130/diff/6/?file=303789#file303789line461>
> >
> > If it needs to be tuned to suit an application, how can one run a generic hadoop cluster with various jobs running?
Hadoop is a bit tricky to configure, and a lot of the knobs need to be adjusted depending on the size of the cluster and the types of jobs you're running. There isn't a great general solution, as far as I know. I would prefer if we kept this knob.
- Brenden
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11130/#review22196
-----------------------------------------------------------
On June 11, 2013, 9:20 p.m., Brenden Matthews wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11130/
> -----------------------------------------------------------
>
> (Updated June 11, 2013, 9:20 p.m.)
>
>
> Review request for mesos.
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Allow JVM memory vars to be configured.
>
> Review: https://reviews.apache.org/r/11130
>
>
> Diffs
> -----
>
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java afe401f5265e3d9494af7eace42eec45943184a3
>
> Diff: https://reviews.apache.org/r/11130/diff/
>
>
> Testing
> -------
>
> Used in production at airbnb.
>
> make -j10 check && cd hadoop && make hadoop-2.0.0-mr1-cdh4.2.1 && make hadoop-0.20.205.0 && make hadoop-0.20.2-cdh3u3
>
>
> Thanks,
>
> Brenden Matthews
>
>
Re: Review Request: Allow JVM memory vars to be configured.
Posted by Ben Mahler <be...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11130/#review22196
-----------------------------------------------------------
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
<https://reviews.apache.org/r/11130/#comment45613>
I guess that's one convention I don't like, since it results in a proliferation of knobs.
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
<https://reviews.apache.org/r/11130/#comment45615>
If it needs to be tuned to suit an application, how can one run a generic hadoop cluster with various jobs running?
- Ben Mahler
On June 11, 2013, 9:20 p.m., Brenden Matthews wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11130/
> -----------------------------------------------------------
>
> (Updated June 11, 2013, 9:20 p.m.)
>
>
> Review request for mesos.
>
>
> Description
> -------
>
> Allow JVM memory vars to be configured.
>
> Review: https://reviews.apache.org/r/11130
>
>
> Diffs
> -----
>
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java afe401f5265e3d9494af7eace42eec45943184a3
>
> Diff: https://reviews.apache.org/r/11130/diff/
>
>
> Testing
> -------
>
> Used in production at airbnb.
>
> make -j10 check && cd hadoop && make hadoop-2.0.0-mr1-cdh4.2.1 && make hadoop-0.20.205.0 && make hadoop-0.20.2-cdh3u3
>
>
> Thanks,
>
> Brenden Matthews
>
>
Re: Review Request: Allow JVM memory vars to be configured.
Posted by Brenden Matthews <br...@diddyinc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11130/
-----------------------------------------------------------
(Updated June 11, 2013, 9:20 p.m.)
Review request for mesos.
Changes
-------
I moved the Math.round() to a better spot.
Description
-------
Allow JVM memory vars to be configured.
Review: https://reviews.apache.org/r/11130
Diffs (updated)
-----
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java afe401f5265e3d9494af7eace42eec45943184a3
Diff: https://reviews.apache.org/r/11130/diff/
Testing
-------
Used in production at airbnb.
make -j10 check && cd hadoop && make hadoop-2.0.0-mr1-cdh4.2.1 && make hadoop-0.20.205.0 && make hadoop-0.20.2-cdh3u3
Thanks,
Brenden Matthews
Re: Review Request: Allow JVM memory vars to be configured.
Posted by Brenden Matthews <br...@diddyinc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11130/
-----------------------------------------------------------
(Updated June 11, 2013, 9:11 p.m.)
Review request for mesos.
Description
-------
Allow JVM memory vars to be configured.
Review: https://reviews.apache.org/r/11130
Diffs (updated)
-----
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java afe401f5265e3d9494af7eace42eec45943184a3
Diff: https://reviews.apache.org/r/11130/diff/
Testing
-------
Used in production at airbnb.
make -j10 check && cd hadoop && make hadoop-2.0.0-mr1-cdh4.2.1 && make hadoop-0.20.205.0 && make hadoop-0.20.2-cdh3u3
Thanks,
Brenden Matthews
Re: Review Request: Allow JVM memory vars to be configured.
Posted by Brenden Matthews <br...@diddyinc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11130/
-----------------------------------------------------------
(Updated June 11, 2013, 8:16 p.m.)
Review request for mesos.
Changes
-------
Rebasing on master.
Summary (updated)
-----------------
Allow JVM memory vars to be configured.
Description (updated)
-------
Allow JVM memory vars to be configured.
Review: https://reviews.apache.org/r/11130
Diffs (updated)
-----
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java afe401f5265e3d9494af7eace42eec45943184a3
Diff: https://reviews.apache.org/r/11130/diff/
Testing (updated)
-------
Used in production at airbnb.
make -j10 check && cd hadoop && make hadoop-2.0.0-mr1-cdh4.2.1 && make hadoop-0.20.205.0 && make hadoop-0.20.2-cdh3u3
Thanks,
Brenden Matthews
Re: Review Request: Increase JVM memory overhead.
Posted by Brenden Matthews <br...@diddyinc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11130/
-----------------------------------------------------------
(Updated June 6, 2013, 2:10 a.m.)
Review request for mesos.
Changes
-------
Rebasing on master.
Description
-------
Increase JVM memory overhead.
Review: https://reviews.apache.org/r/11130
Diffs (updated)
-----
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java afe401f5265e3d9494af7eace42eec45943184a3
Diff: https://reviews.apache.org/r/11130/diff/
Testing
-------
Used in production at airbnb.
Thanks,
Brenden Matthews