You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mesos.apache.org by Brenden Matthews <br...@diddyinc.com> on 2013/05/14 20:00:05 UTC
Review Request: Don't accept offers on the same host/port.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11161/
-----------------------------------------------------------
Review request for mesos.
Description
-------
>From 706a0637d47861adc1463b92040e0754cbfbbbd2 Mon Sep 17 00:00:00 2001
From: Brenden Matthews <br...@airbnb.com>
Date: Tue, 14 May 2013 09:46:36 -0700
Subject: [PATCH 3/5] Don't accept offers on the same host/port.
We'll try to find a port combination that hasn't already been used to
launch a TaskTracker.
Review: https://reviews.apache.org/r/11161/
---
.../org/apache/hadoop/mapred/MesosScheduler.java | 32 ++++++++++++++++++--
1 file changed, 30 insertions(+), 2 deletions(-)
Diffs
-----
Diff: https://reviews.apache.org/r/11161/diff/
Testing
-------
Used in production at airbnb.
Thanks,
Brenden Matthews
Re: Review Request: Don't accept offers on the same host/port.
Posted by Brenden Matthews <br...@diddyinc.com>.
> On June 3, 2013, 8:57 p.m., Charles Reiss wrote:
> > hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java, line 519
> > <https://reviews.apache.org/r/11161/diff/2/?file=300044#file300044line519>
> >
> > I think if the offer contains something like 'ports:[2048-2048,3192-65535]', this might choose unoffered ports.
Good point. I'll resolve this.
> On June 3, 2013, 8:57 p.m., Charles Reiss wrote:
> > hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java, line 522
> > <https://reviews.apache.org/r/11161/diff/2/?file=300044#file300044line522>
> >
> > A comment to explain why there can be duplicate ports would be nice. (I assume it's that Mesos can process task completions before the scheduler does)
If you try and launch >1 task trackers from the same slave, you need to ensure they aren't sharing the same ports.
It can happen when you launch a small task tracker and then launch another on the same slave afterwards. This is a common pattern when you're running multiple Hadoop jobs simultaneously.
- Brenden
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11161/#review21353
-----------------------------------------------------------
On June 3, 2013, 5:59 p.m., Brenden Matthews wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11161/
> -----------------------------------------------------------
>
> (Updated June 3, 2013, 5:59 p.m.)
>
>
> Review request for mesos.
>
>
> Description
> -------
>
> Don't accept offers on the same host/port.
>
> We'll try to find a port combination that hasn't already been used to
> launch a TaskTracker.
>
> Review: https://reviews.apache.org/r/11161/
>
>
> Diffs
> -----
>
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java afe401f5265e3d9494af7eace42eec45943184a3
>
> Diff: https://reviews.apache.org/r/11161/diff/
>
>
> Testing
> -------
>
> Used in production at airbnb.
>
>
> Thanks,
>
> Brenden Matthews
>
>
Re: Review Request: Don't accept offers on the same host/port.
Posted by Charles Reiss <wo...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11161/#review21353
-----------------------------------------------------------
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
<https://reviews.apache.org/r/11161/#comment44265>
I think if the offer contains something like 'ports:[2048-2048,3192-65535]', this might choose unoffered ports.
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
<https://reviews.apache.org/r/11161/#comment44286>
A comment to explain why there can be duplicate ports would be nice. (I assume it's that Mesos can process task completions before the scheduler does)
- Charles Reiss
On June 3, 2013, 5:59 p.m., Brenden Matthews wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11161/
> -----------------------------------------------------------
>
> (Updated June 3, 2013, 5:59 p.m.)
>
>
> Review request for mesos.
>
>
> Description
> -------
>
> Don't accept offers on the same host/port.
>
> We'll try to find a port combination that hasn't already been used to
> launch a TaskTracker.
>
> Review: https://reviews.apache.org/r/11161/
>
>
> Diffs
> -----
>
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java afe401f5265e3d9494af7eace42eec45943184a3
>
> Diff: https://reviews.apache.org/r/11161/diff/
>
>
> Testing
> -------
>
> Used in production at airbnb.
>
>
> Thanks,
>
> Brenden Matthews
>
>
Re: Review Request: Don't accept offers on the same host/port.
Posted by Brenden Matthews <br...@diddyinc.com>.
> On June 11, 2013, 9:12 p.m., Vinod Kone wrote:
> > hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java, lines 427-428
> > <https://reviews.apache.org/r/11161/diff/5/?file=303786#file303786line427>
> >
> > Isn't 'begin' always equal to 'range.getBegin()' and 'end' always equal to 'range.getEnd()' ?
I didn't want to make the assumption that the range was ordered by smallest first.
- Brenden
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11161/#review21741
-----------------------------------------------------------
On June 11, 2013, 8:56 p.m., Brenden Matthews wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11161/
> -----------------------------------------------------------
>
> (Updated June 11, 2013, 8:56 p.m.)
>
>
> Review request for mesos.
>
>
> Description
> -------
>
> Don't accept offers on the same host/port.
>
> We'll try to find a port combination that hasn't already been used to
> launch a TaskTracker.
>
> Review: https://reviews.apache.org/r/11161/
>
>
> Diffs
> -----
>
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java afe401f5265e3d9494af7eace42eec45943184a3
>
> Diff: https://reviews.apache.org/r/11161/diff/
>
>
> Testing
> -------
>
> Used in production at airbnb.
>
> make -j10 check && cd hadoop && make hadoop-2.0.0-mr1-cdh4.2.1 && make hadoop-0.20.205.0 && make hadoop-0.20.2-cdh3u3
>
>
> Thanks,
>
> Brenden Matthews
>
>
Re: Review Request: Don't accept offers on the same host/port.
Posted by Brenden Matthews <br...@diddyinc.com>.
> On June 11, 2013, 9:12 p.m., Vinod Kone wrote:
> > I still don't understand why the Hadoop scheduler would get an offer with ports that are currently in use by a task tracker? If thats the case, it is a bug in Mesos. Some logs indicating this problem would be nice to diagnose the real issue here.
I may have misunderstood the previous code. I'll simplify this a bit and see if it still occurs.
- Brenden
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11161/#review21741
-----------------------------------------------------------
On June 11, 2013, 8:56 p.m., Brenden Matthews wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11161/
> -----------------------------------------------------------
>
> (Updated June 11, 2013, 8:56 p.m.)
>
>
> Review request for mesos.
>
>
> Description
> -------
>
> Don't accept offers on the same host/port.
>
> We'll try to find a port combination that hasn't already been used to
> launch a TaskTracker.
>
> Review: https://reviews.apache.org/r/11161/
>
>
> Diffs
> -----
>
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java afe401f5265e3d9494af7eace42eec45943184a3
>
> Diff: https://reviews.apache.org/r/11161/diff/
>
>
> Testing
> -------
>
> Used in production at airbnb.
>
> make -j10 check && cd hadoop && make hadoop-2.0.0-mr1-cdh4.2.1 && make hadoop-0.20.205.0 && make hadoop-0.20.2-cdh3u3
>
>
> Thanks,
>
> Brenden Matthews
>
>
Re: Review Request: Don't accept offers on the same host/port.
Posted by Vinod Kone <vi...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11161/#review21741
-----------------------------------------------------------
I still don't understand why the Hadoop scheduler would get an offer with ports that are currently in use by a task tracker? If thats the case, it is a bug in Mesos. Some logs indicating this problem would be nice to diagnose the real issue here.
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
<https://reviews.apache.org/r/11161/#comment44924>
Isn't 'begin' always equal to 'range.getBegin()' and 'end' always equal to 'range.getEnd()' ?
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
<https://reviews.apache.org/r/11161/#comment44926>
I don't like this because ports array could get very big, if we include every free port in the list.
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
<https://reviews.apache.org/r/11161/#comment44930>
why sort?
- Vinod Kone
On June 11, 2013, 8:56 p.m., Brenden Matthews wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11161/
> -----------------------------------------------------------
>
> (Updated June 11, 2013, 8:56 p.m.)
>
>
> Review request for mesos.
>
>
> Description
> -------
>
> Don't accept offers on the same host/port.
>
> We'll try to find a port combination that hasn't already been used to
> launch a TaskTracker.
>
> Review: https://reviews.apache.org/r/11161/
>
>
> Diffs
> -----
>
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java afe401f5265e3d9494af7eace42eec45943184a3
>
> Diff: https://reviews.apache.org/r/11161/diff/
>
>
> Testing
> -------
>
> Used in production at airbnb.
>
> make -j10 check && cd hadoop && make hadoop-2.0.0-mr1-cdh4.2.1 && make hadoop-0.20.205.0 && make hadoop-0.20.2-cdh3u3
>
>
> Thanks,
>
> Brenden Matthews
>
>
Re: Review Request 11161: Don't accept offers on the same host/port.
Posted by Brenden Matthews <br...@diddyinc.com>.
> On July 24, 2013, 11:51 p.m., Vinod Kone wrote:
> > Did you get any logs to indicate why this was happening in the first place? The comments seem to suggest we were waiting for more data to diagnose this.
I haven't seen this in a long time, but I still would like to keep this change.
- Brenden
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11161/#review23798
-----------------------------------------------------------
On July 24, 2013, 10:18 p.m., Brenden Matthews wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11161/
> -----------------------------------------------------------
>
> (Updated July 24, 2013, 10:18 p.m.)
>
>
> Review request for mesos.
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Don't accept offers on the same host/port.
>
> We'll try to find a port combination that hasn't already been used to
> launch a TaskTracker.
>
> Review: https://reviews.apache.org/r/11161/
>
>
> Diffs
> -----
>
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java 279f84e0f0c43ad3cfd9e4442010e706ee3565d9
>
> Diff: https://reviews.apache.org/r/11161/diff/
>
>
> Testing
> -------
>
> Used in production at airbnb.
>
> make -j10 check && cd hadoop && make hadoop-2.0.0-mr1-cdh4.2.1 && make hadoop-0.20.205.0 && make hadoop-0.20.2-cdh3u3
>
>
> Thanks,
>
> Brenden Matthews
>
>
Re: Review Request 11161: Don't accept offers on the same host/port.
Posted by Vinod Kone <vi...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11161/#review23798
-----------------------------------------------------------
Did you get any logs to indicate why this was happening in the first place? The comments seem to suggest we were waiting for more data to diagnose this.
- Vinod Kone
On July 24, 2013, 10:18 p.m., Brenden Matthews wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11161/
> -----------------------------------------------------------
>
> (Updated July 24, 2013, 10:18 p.m.)
>
>
> Review request for mesos.
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Don't accept offers on the same host/port.
>
> We'll try to find a port combination that hasn't already been used to
> launch a TaskTracker.
>
> Review: https://reviews.apache.org/r/11161/
>
>
> Diffs
> -----
>
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java 279f84e0f0c43ad3cfd9e4442010e706ee3565d9
>
> Diff: https://reviews.apache.org/r/11161/diff/
>
>
> Testing
> -------
>
> Used in production at airbnb.
>
> make -j10 check && cd hadoop && make hadoop-2.0.0-mr1-cdh4.2.1 && make hadoop-0.20.205.0 && make hadoop-0.20.2-cdh3u3
>
>
> Thanks,
>
> Brenden Matthews
>
>
Re: Review Request 11161: No longer accept offers on the same host/port.
Posted by Brenden Matthews <br...@diddyinc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11161/
-----------------------------------------------------------
(Updated July 30, 2013, 9 p.m.)
Review request for mesos.
Changes
-------
Updated as per Ben's comments.
Summary (updated)
-----------------
No longer accept offers on the same host/port.
Repository: mesos
Description (updated)
-------
No longer accept offers on the same host/port.
We'll try to find a port combination that hasn't already been used to
launch a TaskTracker.
Review: https://reviews.apache.org/r/11161/
Diffs (updated)
-----
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java 69d4655a38c6fd24c19918d240727a43cd1c9c92
Diff: https://reviews.apache.org/r/11161/diff/
Testing
-------
Used in production at airbnb.
make -j10 check && cd hadoop && make hadoop-2.0.0-mr1-cdh4.2.1 && make hadoop-0.20.205.0 && make hadoop-0.20.2-cdh3u3
Thanks,
Brenden Matthews
Re: Review Request 11161: Don't accept offers on the same host/port.
Posted by Ben Mahler <be...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11161/#review24267
-----------------------------------------------------------
Ship it!
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
<https://reviews.apache.org/r/11161/#comment48091>
Range is ordered, so there's no need to do this. Were you concerned about that?
If so, how about just adding an assertion or a LOG.fatal, we should better document the Value.Range type in mesos.proto if this part is not clear!
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
<https://reviews.apache.org/r/11161/#comment48093>
Can you add a comment indicating why this was added? What did you observe? This is indicative of a bug in mesos if we reported a port available that was in use!
s/launched/launched./
- Ben Mahler
On July 25, 2013, 1:42 a.m., Brenden Matthews wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11161/
> -----------------------------------------------------------
>
> (Updated July 25, 2013, 1:42 a.m.)
>
>
> Review request for mesos.
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Don't accept offers on the same host/port.
>
> We'll try to find a port combination that hasn't already been used to
> launch a TaskTracker.
>
> Review: https://reviews.apache.org/r/11161/
>
>
> Diffs
> -----
>
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java 279f84e0f0c43ad3cfd9e4442010e706ee3565d9
>
> Diff: https://reviews.apache.org/r/11161/diff/
>
>
> Testing
> -------
>
> Used in production at airbnb.
>
> make -j10 check && cd hadoop && make hadoop-2.0.0-mr1-cdh4.2.1 && make hadoop-0.20.205.0 && make hadoop-0.20.2-cdh3u3
>
>
> Thanks,
>
> Brenden Matthews
>
>
Re: Review Request 11161: Don't accept offers on the same host/port.
Posted by Brenden Matthews <br...@diddyinc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11161/
-----------------------------------------------------------
(Updated July 25, 2013, 1:42 a.m.)
Review request for mesos.
Changes
-------
Add dependency.
Repository: mesos
Description
-------
Don't accept offers on the same host/port.
We'll try to find a port combination that hasn't already been used to
launch a TaskTracker.
Review: https://reviews.apache.org/r/11161/
Diffs
-----
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java 279f84e0f0c43ad3cfd9e4442010e706ee3565d9
Diff: https://reviews.apache.org/r/11161/diff/
Testing
-------
Used in production at airbnb.
make -j10 check && cd hadoop && make hadoop-2.0.0-mr1-cdh4.2.1 && make hadoop-0.20.205.0 && make hadoop-0.20.2-cdh3u3
Thanks,
Brenden Matthews
Re: Review Request 11161: Don't accept offers on the same host/port.
Posted by Vinod Kone <vi...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11161/#review23803
-----------------------------------------------------------
Ship it!
Ok. But please add a comment mentioning that you have seen this in the past and its not yet clear why its happening. Maybe a TODO to diagnose/fix it.
- Vinod Kone
On July 24, 2013, 10:18 p.m., Brenden Matthews wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11161/
> -----------------------------------------------------------
>
> (Updated July 24, 2013, 10:18 p.m.)
>
>
> Review request for mesos.
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Don't accept offers on the same host/port.
>
> We'll try to find a port combination that hasn't already been used to
> launch a TaskTracker.
>
> Review: https://reviews.apache.org/r/11161/
>
>
> Diffs
> -----
>
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java 279f84e0f0c43ad3cfd9e4442010e706ee3565d9
>
> Diff: https://reviews.apache.org/r/11161/diff/
>
>
> Testing
> -------
>
> Used in production at airbnb.
>
> make -j10 check && cd hadoop && make hadoop-2.0.0-mr1-cdh4.2.1 && make hadoop-0.20.205.0 && make hadoop-0.20.2-cdh3u3
>
>
> Thanks,
>
> Brenden Matthews
>
>
Re: Review Request 11161: Don't accept offers on the same host/port.
Posted by Brenden Matthews <br...@diddyinc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11161/
-----------------------------------------------------------
(Updated July 24, 2013, 10:18 p.m.)
Review request for mesos.
Changes
-------
Rebasing on master.
Repository: mesos
Description
-------
Don't accept offers on the same host/port.
We'll try to find a port combination that hasn't already been used to
launch a TaskTracker.
Review: https://reviews.apache.org/r/11161/
Diffs (updated)
-----
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java 279f84e0f0c43ad3cfd9e4442010e706ee3565d9
Diff: https://reviews.apache.org/r/11161/diff/
Testing
-------
Used in production at airbnb.
make -j10 check && cd hadoop && make hadoop-2.0.0-mr1-cdh4.2.1 && make hadoop-0.20.205.0 && make hadoop-0.20.2-cdh3u3
Thanks,
Brenden Matthews
Re: Review Request: Don't accept offers on the same host/port.
Posted by Vinod Kone <vi...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11161/#review21746
-----------------------------------------------------------
Lets hold off on this patch until we investigate this with logs/data. If/when you get hit this problem again, I would love to take a look.
- Vinod Kone
On June 11, 2013, 9:36 p.m., Brenden Matthews wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11161/
> -----------------------------------------------------------
>
> (Updated June 11, 2013, 9:36 p.m.)
>
>
> Review request for mesos.
>
>
> Description
> -------
>
> Don't accept offers on the same host/port.
>
> We'll try to find a port combination that hasn't already been used to
> launch a TaskTracker.
>
> Review: https://reviews.apache.org/r/11161/
>
>
> Diffs
> -----
>
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java afe401f5265e3d9494af7eace42eec45943184a3
>
> Diff: https://reviews.apache.org/r/11161/diff/
>
>
> Testing
> -------
>
> Used in production at airbnb.
>
> make -j10 check && cd hadoop && make hadoop-2.0.0-mr1-cdh4.2.1 && make hadoop-0.20.205.0 && make hadoop-0.20.2-cdh3u3
>
>
> Thanks,
>
> Brenden Matthews
>
>
Re: Review Request: Don't accept offers on the same host/port.
Posted by Brenden Matthews <br...@diddyinc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11161/
-----------------------------------------------------------
(Updated June 11, 2013, 9:36 p.m.)
Review request for mesos.
Description
-------
Don't accept offers on the same host/port.
We'll try to find a port combination that hasn't already been used to
launch a TaskTracker.
Review: https://reviews.apache.org/r/11161/
Diffs (updated)
-----
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java afe401f5265e3d9494af7eace42eec45943184a3
Diff: https://reviews.apache.org/r/11161/diff/
Testing
-------
Used in production at airbnb.
make -j10 check && cd hadoop && make hadoop-2.0.0-mr1-cdh4.2.1 && make hadoop-0.20.205.0 && make hadoop-0.20.2-cdh3u3
Thanks,
Brenden Matthews
Re: Review Request: Don't accept offers on the same host/port.
Posted by Brenden Matthews <br...@diddyinc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11161/
-----------------------------------------------------------
(Updated June 11, 2013, 8:56 p.m.)
Review request for mesos.
Changes
-------
The last diff was messed up for some reason. Here's another.
Description
-------
Don't accept offers on the same host/port.
We'll try to find a port combination that hasn't already been used to
launch a TaskTracker.
Review: https://reviews.apache.org/r/11161/
Diffs (updated)
-----
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java afe401f5265e3d9494af7eace42eec45943184a3
Diff: https://reviews.apache.org/r/11161/diff/
Testing
-------
Used in production at airbnb.
make -j10 check && cd hadoop && make hadoop-2.0.0-mr1-cdh4.2.1 && make hadoop-0.20.205.0 && make hadoop-0.20.2-cdh3u3
Thanks,
Brenden Matthews
Re: Review Request: Don't accept offers on the same host/port.
Posted by Brenden Matthews <br...@diddyinc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11161/
-----------------------------------------------------------
(Updated June 11, 2013, 8:44 p.m.)
Review request for mesos.
Changes
-------
Rebasing on master.
Description
-------
Don't accept offers on the same host/port.
We'll try to find a port combination that hasn't already been used to
launch a TaskTracker.
Review: https://reviews.apache.org/r/11161/
Diffs (updated)
-----
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java afe401f5265e3d9494af7eace42eec45943184a3
Diff: https://reviews.apache.org/r/11161/diff/
Testing (updated)
-------
Used in production at airbnb.
make -j10 check && cd hadoop && make hadoop-2.0.0-mr1-cdh4.2.1 && make hadoop-0.20.205.0 && make hadoop-0.20.2-cdh3u3
Thanks,
Brenden Matthews
Re: Review Request: Don't accept offers on the same host/port.
Posted by Brenden Matthews <br...@diddyinc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11161/
-----------------------------------------------------------
(Updated June 6, 2013, 2:11 a.m.)
Review request for mesos.
Changes
-------
Rebasing on master.
Description
-------
Don't accept offers on the same host/port.
We'll try to find a port combination that hasn't already been used to
launch a TaskTracker.
Review: https://reviews.apache.org/r/11161/
Diffs (updated)
-----
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java afe401f5265e3d9494af7eace42eec45943184a3
Diff: https://reviews.apache.org/r/11161/diff/
Testing
-------
Used in production at airbnb.
Thanks,
Brenden Matthews
Re: Review Request: Don't accept offers on the same host/port.
Posted by Charles Reiss <wo...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11161/#review21361
-----------------------------------------------------------
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
<https://reviews.apache.org/r/11161/#comment44296>
I understand that they can't share ports, but the non-intuitive thing is why the Mesos master reserving the port (as part of the resources of pending offer or the running Mesos task) is not enough.
Because the Mesos master should be tracking port assignments, I'd guess the problem only come up from Mesos thinking a task tracker has gone away but the Hadoop scheduler not yet knowing that.
- Charles Reiss
On June 3, 2013, 5:59 p.m., Brenden Matthews wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11161/
> -----------------------------------------------------------
>
> (Updated June 3, 2013, 5:59 p.m.)
>
>
> Review request for mesos.
>
>
> Description
> -------
>
> Don't accept offers on the same host/port.
>
> We'll try to find a port combination that hasn't already been used to
> launch a TaskTracker.
>
> Review: https://reviews.apache.org/r/11161/
>
>
> Diffs
> -----
>
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java afe401f5265e3d9494af7eace42eec45943184a3
>
> Diff: https://reviews.apache.org/r/11161/diff/
>
>
> Testing
> -------
>
> Used in production at airbnb.
>
>
> Thanks,
>
> Brenden Matthews
>
>
Re: Review Request: Don't accept offers on the same host/port.
Posted by Brenden Matthews <br...@diddyinc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11161/
-----------------------------------------------------------
(Updated June 3, 2013, 5:59 p.m.)
Review request for mesos.
Description (updated)
-------
Don't accept offers on the same host/port.
We'll try to find a port combination that hasn't already been used to
launch a TaskTracker.
Review: https://reviews.apache.org/r/11161/
Diffs (updated)
-----
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java afe401f5265e3d9494af7eace42eec45943184a3
Diff: https://reviews.apache.org/r/11161/diff/
Testing
-------
Used in production at airbnb.
Thanks,
Brenden Matthews
Re: Review Request: Don't accept offers on the same host/port.
Posted by Brenden Matthews <br...@diddyinc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11161/
-----------------------------------------------------------
(Updated May 28, 2013, 10:42 p.m.)
Review request for mesos.
Description
-------
>From 706a0637d47861adc1463b92040e0754cbfbbbd2 Mon Sep 17 00:00:00 2001
From: Brenden Matthews <br...@airbnb.com>
Date: Tue, 14 May 2013 09:46:36 -0700
Subject: [PATCH 3/5] Don't accept offers on the same host/port.
We'll try to find a port combination that hasn't already been used to
launch a TaskTracker.
Review: https://reviews.apache.org/r/11161/
---
.../org/apache/hadoop/mapred/MesosScheduler.java | 32 ++++++++++++++++++--
1 file changed, 30 insertions(+), 2 deletions(-)
Diffs (updated)
-----
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java afe401f5265e3d9494af7eace42eec45943184a3
Diff: https://reviews.apache.org/r/11161/diff/
Testing
-------
Used in production at airbnb.
Thanks,
Brenden Matthews
Re: Review Request: Don't accept offers on the same host/port.
Posted by Vinod Kone <vi...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11161/#review20729
-----------------------------------------------------------
forgot to attach the diff?
- Vinod Kone
On May 14, 2013, 6 p.m., Brenden Matthews wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11161/
> -----------------------------------------------------------
>
> (Updated May 14, 2013, 6 p.m.)
>
>
> Review request for mesos.
>
>
> Description
> -------
>
> From 706a0637d47861adc1463b92040e0754cbfbbbd2 Mon Sep 17 00:00:00 2001
> From: Brenden Matthews <br...@airbnb.com>
> Date: Tue, 14 May 2013 09:46:36 -0700
> Subject: [PATCH 3/5] Don't accept offers on the same host/port.
>
> We'll try to find a port combination that hasn't already been used to
> launch a TaskTracker.
>
> Review: https://reviews.apache.org/r/11161/
> ---
> .../org/apache/hadoop/mapred/MesosScheduler.java | 32 ++++++++++++++++++--
> 1 file changed, 30 insertions(+), 2 deletions(-)
>
>
> Diffs
> -----
>
>
> Diff: https://reviews.apache.org/r/11161/diff/
>
>
> Testing
> -------
>
> Used in production at airbnb.
>
>
> Thanks,
>
> Brenden Matthews
>
>