You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mesos.apache.org by Brenden Matthews <br...@diddyinc.com> on 2013/05/29 00:50:58 UTC
Review Request: Write Mesos JobTracker stats to state file.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11165/
-----------------------------------------------------------
Review request for mesos.
Description
-------
Write Mesos JobTracker stats to state file.
We use this data for determining how busy the cluster is at run time.
Diffs
-----
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java afe401f5265e3d9494af7eace42eec45943184a3
Diff: https://reviews.apache.org/r/11165/diff/
Testing
-------
Thanks,
Brenden Matthews
Re: Review Request: Write Mesos JobTracker stats to state file.
Posted by Brenden Matthews <br...@diddyinc.com>.
> On June 4, 2013, 12:33 a.m., Ben Mahler wrote:
> > Good idea, but seems like a JSON endpoint would be better (this is how we expose statistics in mesos)?
That's fine, but then I'd have to do a bunch of JSON serialization in the Hadoop JobTracker, which seems overkill to me.
- Brenden
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11165/#review21384
-----------------------------------------------------------
On June 6, 2013, 2:10 a.m., Brenden Matthews wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11165/
> -----------------------------------------------------------
>
> (Updated June 6, 2013, 2:10 a.m.)
>
>
> Review request for mesos.
>
>
> Description
> -------
>
> Write Mesos JobTracker stats to state file.
>
> Review: https://reviews.apache.org/r/11165/
>
>
> Diffs
> -----
>
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java afe401f5265e3d9494af7eace42eec45943184a3
>
> Diff: https://reviews.apache.org/r/11165/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Brenden Matthews
>
>
Re: Review Request: Write Mesos JobTracker stats to state file.
Posted by Brenden Matthews <br...@diddyinc.com>.
> On June 4, 2013, 12:33 a.m., Ben Mahler wrote:
> > Good idea, but seems like a JSON endpoint would be better (this is how we expose statistics in mesos)?
>
> Brenden Matthews wrote:
> That's fine, but then I'd have to do a bunch of JSON serialization in the Hadoop JobTracker, which seems overkill to me.
>
> Ben Mahler wrote:
> My concern is twofold:
>
> (1) An http endpoint seems nicer than a file, IMO. One can just curl the endpoint and each response will contain the latest data, rather than a periodically update file where each read returns a checkpointed version of the scheduler state.
>
> (2) The idiomatic way of exposing state in mesos is through http endpoints, so it would be nice to stay consistent there. I realize that it's tough to add that into the hadoop scheduler given that we don't have any abstractions in place to assist in exporting stats over an endpoint. But I'm open to options! :)
I can investigate to see if there's a way to do it without having to add dependencies or make a bunch of changes elsewhere. I dislike the idea of having lots of patches against Hadoop, because it increases the complexity of maintaining our own distribution.
- Brenden
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11165/#review21384
-----------------------------------------------------------
On June 11, 2013, 7:33 p.m., Brenden Matthews wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11165/
> -----------------------------------------------------------
>
> (Updated June 11, 2013, 7:33 p.m.)
>
>
> Review request for mesos.
>
>
> Description
> -------
>
> Write Mesos JobTracker stats to state file.
>
> Review: https://reviews.apache.org/r/11165/
>
>
> Diffs
> -----
>
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java afe401f5265e3d9494af7eace42eec45943184a3
>
> Diff: https://reviews.apache.org/r/11165/diff/
>
>
> Testing
> -------
>
> Used in production at airbnb.
>
> make -j10 check && cd hadoop && make hadoop-2.0.0-mr1-cdh4.2.1 && make hadoop-0.20.205.0 && make hadoop-0.20.2-cdh3u3
>
>
> Thanks,
>
> Brenden Matthews
>
>
Re: Review Request: Write Mesos JobTracker stats to state file.
Posted by Ben Mahler <be...@gmail.com>.
> On June 4, 2013, 12:33 a.m., Ben Mahler wrote:
> > Good idea, but seems like a JSON endpoint would be better (this is how we expose statistics in mesos)?
>
> Brenden Matthews wrote:
> That's fine, but then I'd have to do a bunch of JSON serialization in the Hadoop JobTracker, which seems overkill to me.
My concern is twofold:
(1) An http endpoint seems nicer than a file, IMO. One can just curl the endpoint and each response will contain the latest data, rather than a periodically update file where each read returns a checkpointed version of the scheduler state.
(2) The idiomatic way of exposing state in mesos is through http endpoints, so it would be nice to stay consistent there. I realize that it's tough to add that into the hadoop scheduler given that we don't have any abstractions in place to assist in exporting stats over an endpoint. But I'm open to options! :)
- Ben
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11165/#review21384
-----------------------------------------------------------
On June 11, 2013, 7:33 p.m., Brenden Matthews wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11165/
> -----------------------------------------------------------
>
> (Updated June 11, 2013, 7:33 p.m.)
>
>
> Review request for mesos.
>
>
> Description
> -------
>
> Write Mesos JobTracker stats to state file.
>
> Review: https://reviews.apache.org/r/11165/
>
>
> Diffs
> -----
>
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java afe401f5265e3d9494af7eace42eec45943184a3
>
> Diff: https://reviews.apache.org/r/11165/diff/
>
>
> Testing
> -------
>
> Used in production at airbnb.
>
> make -j10 check && cd hadoop && make hadoop-2.0.0-mr1-cdh4.2.1 && make hadoop-0.20.205.0 && make hadoop-0.20.2-cdh3u3
>
>
> Thanks,
>
> Brenden Matthews
>
>
Re: Review Request: Write Mesos JobTracker stats to state file.
Posted by Ben Mahler <be...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11165/#review21384
-----------------------------------------------------------
Good idea, but seems like a JSON endpoint would be better (this is how we expose statistics in mesos)?
- Ben Mahler
On June 3, 2013, 5:59 p.m., Brenden Matthews wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11165/
> -----------------------------------------------------------
>
> (Updated June 3, 2013, 5:59 p.m.)
>
>
> Review request for mesos.
>
>
> Description
> -------
>
> Write Mesos JobTracker stats to state file.
>
> Review: https://reviews.apache.org/r/11165/
>
>
> Diffs
> -----
>
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java afe401f5265e3d9494af7eace42eec45943184a3
>
> Diff: https://reviews.apache.org/r/11165/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Brenden Matthews
>
>
Re: Review Request 11165: Write Mesos JobTracker stats to state file.
Posted by Brenden Matthews <br...@diddyinc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11165/
-----------------------------------------------------------
(Updated July 25, 2013, 2 a.m.)
Review request for mesos.
Changes
-------
Removed bad comment.
Repository: mesos
Description
-------
Write Mesos JobTracker stats to state file.
Review: https://reviews.apache.org/r/11165/
Diffs
-----
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java 279f84e0f0c43ad3cfd9e4442010e706ee3565d9
Diff: https://reviews.apache.org/r/11165/diff/
Testing
-------
Used in production at airbnb.
make -j10 check && cd hadoop && make hadoop-2.0.0-mr1-cdh4.2.1 && make hadoop-0.20.205.0 && make hadoop-0.20.2-cdh3u3
Thanks,
Brenden Matthews
Re: Review Request 11165: Write Mesos JobTracker stats to state file.
Posted by Brenden Matthews <br...@diddyinc.com>.
> On July 25, 2013, 1:48 a.m., Vinod Kone wrote:
> > hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java, lines 404-432
> > <https://reviews.apache.org/r/11165/diff/5/?file=327612#file327612line404>
> >
> > Why do we need this? Is this not something that they can get from the log file?
It's much easier to grab this from a file than try to parse logs.
- Brenden
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11165/#review23809
-----------------------------------------------------------
On July 25, 2013, 2 a.m., Brenden Matthews wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11165/
> -----------------------------------------------------------
>
> (Updated July 25, 2013, 2 a.m.)
>
>
> Review request for mesos.
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Write Mesos JobTracker stats to state file.
>
> Review: https://reviews.apache.org/r/11165/
>
>
> Diffs
> -----
>
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java 279f84e0f0c43ad3cfd9e4442010e706ee3565d9
>
> Diff: https://reviews.apache.org/r/11165/diff/
>
>
> Testing
> -------
>
> Used in production at airbnb.
>
> make -j10 check && cd hadoop && make hadoop-2.0.0-mr1-cdh4.2.1 && make hadoop-0.20.205.0 && make hadoop-0.20.2-cdh3u3
>
>
> Thanks,
>
> Brenden Matthews
>
>
Re: Review Request 11165: Write Mesos JobTracker stats to state file.
Posted by Brenden Matthews <br...@diddyinc.com>.
> On July 25, 2013, 1:48 a.m., Vinod Kone wrote:
> > hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java, lines 404-432
> > <https://reviews.apache.org/r/11165/diff/5/?file=327612#file327612line404>
> >
> > Why do we need this? Is this not something that they can get from the log file?
>
> Brenden Matthews wrote:
> It's much easier to grab this from a file than try to parse logs.
>
> Vinod Kone wrote:
> how come?
Less effort to parse the data, mostly. I suppose a JSON endpoint would be great, but I'd rather not introduce that complexity into this service.
- Brenden
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11165/#review23809
-----------------------------------------------------------
On July 25, 2013, 2 a.m., Brenden Matthews wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11165/
> -----------------------------------------------------------
>
> (Updated July 25, 2013, 2 a.m.)
>
>
> Review request for mesos.
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Write Mesos JobTracker stats to state file.
>
> Review: https://reviews.apache.org/r/11165/
>
>
> Diffs
> -----
>
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java 279f84e0f0c43ad3cfd9e4442010e706ee3565d9
>
> Diff: https://reviews.apache.org/r/11165/diff/
>
>
> Testing
> -------
>
> Used in production at airbnb.
>
> make -j10 check && cd hadoop && make hadoop-2.0.0-mr1-cdh4.2.1 && make hadoop-0.20.205.0 && make hadoop-0.20.2-cdh3u3
>
>
> Thanks,
>
> Brenden Matthews
>
>
Re: Review Request 11165: Write Mesos JobTracker stats to state file.
Posted by Brenden Matthews <br...@diddyinc.com>.
> On July 25, 2013, 1:48 a.m., Vinod Kone wrote:
> > hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java, line 412
> > <https://reviews.apache.org/r/11165/diff/5/?file=327612#file327612line412>
> >
> > what true?
Review board is giving the old 'git cat-file' error right now. It's fixed in my tree though.
- Brenden
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11165/#review23809
-----------------------------------------------------------
On July 25, 2013, 2 a.m., Brenden Matthews wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11165/
> -----------------------------------------------------------
>
> (Updated July 25, 2013, 2 a.m.)
>
>
> Review request for mesos.
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Write Mesos JobTracker stats to state file.
>
> Review: https://reviews.apache.org/r/11165/
>
>
> Diffs
> -----
>
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java 279f84e0f0c43ad3cfd9e4442010e706ee3565d9
>
> Diff: https://reviews.apache.org/r/11165/diff/
>
>
> Testing
> -------
>
> Used in production at airbnb.
>
> make -j10 check && cd hadoop && make hadoop-2.0.0-mr1-cdh4.2.1 && make hadoop-0.20.205.0 && make hadoop-0.20.2-cdh3u3
>
>
> Thanks,
>
> Brenden Matthews
>
>
Re: Review Request 11165: Write Mesos JobTracker stats to state file.
Posted by Vinod Kone <vi...@gmail.com>.
> On July 25, 2013, 1:48 a.m., Vinod Kone wrote:
> > hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java, lines 404-432
> > <https://reviews.apache.org/r/11165/diff/5/?file=327612#file327612line404>
> >
> > Why do we need this? Is this not something that they can get from the log file?
>
> Brenden Matthews wrote:
> It's much easier to grab this from a file than try to parse logs.
how come?
- Vinod
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11165/#review23809
-----------------------------------------------------------
On July 25, 2013, 2 a.m., Brenden Matthews wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11165/
> -----------------------------------------------------------
>
> (Updated July 25, 2013, 2 a.m.)
>
>
> Review request for mesos.
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Write Mesos JobTracker stats to state file.
>
> Review: https://reviews.apache.org/r/11165/
>
>
> Diffs
> -----
>
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java 279f84e0f0c43ad3cfd9e4442010e706ee3565d9
>
> Diff: https://reviews.apache.org/r/11165/diff/
>
>
> Testing
> -------
>
> Used in production at airbnb.
>
> make -j10 check && cd hadoop && make hadoop-2.0.0-mr1-cdh4.2.1 && make hadoop-0.20.205.0 && make hadoop-0.20.2-cdh3u3
>
>
> Thanks,
>
> Brenden Matthews
>
>
Re: Review Request 11165: Write Mesos JobTracker stats to state file.
Posted by Vinod Kone <vi...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11165/#review23809
-----------------------------------------------------------
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
<https://reviews.apache.org/r/11165/#comment47649>
Why do we need this? Is this not something that they can get from the log file?
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
<https://reviews.apache.org/r/11165/#comment47648>
what true?
- Vinod Kone
On July 25, 2013, 1:43 a.m., Brenden Matthews wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11165/
> -----------------------------------------------------------
>
> (Updated July 25, 2013, 1:43 a.m.)
>
>
> Review request for mesos.
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Write Mesos JobTracker stats to state file.
>
> Review: https://reviews.apache.org/r/11165/
>
>
> Diffs
> -----
>
> hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java 279f84e0f0c43ad3cfd9e4442010e706ee3565d9
>
> Diff: https://reviews.apache.org/r/11165/diff/
>
>
> Testing
> -------
>
> Used in production at airbnb.
>
> make -j10 check && cd hadoop && make hadoop-2.0.0-mr1-cdh4.2.1 && make hadoop-0.20.205.0 && make hadoop-0.20.2-cdh3u3
>
>
> Thanks,
>
> Brenden Matthews
>
>
Re: Review Request 11165: Write Mesos JobTracker stats to state file.
Posted by Brenden Matthews <br...@diddyinc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11165/
-----------------------------------------------------------
(Updated July 25, 2013, 1:43 a.m.)
Review request for mesos.
Changes
-------
Add dependency.
Repository: mesos
Description
-------
Write Mesos JobTracker stats to state file.
Review: https://reviews.apache.org/r/11165/
Diffs
-----
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java 279f84e0f0c43ad3cfd9e4442010e706ee3565d9
Diff: https://reviews.apache.org/r/11165/diff/
Testing
-------
Used in production at airbnb.
make -j10 check && cd hadoop && make hadoop-2.0.0-mr1-cdh4.2.1 && make hadoop-0.20.205.0 && make hadoop-0.20.2-cdh3u3
Thanks,
Brenden Matthews
Re: Review Request 11165: Write Mesos JobTracker stats to state file.
Posted by Brenden Matthews <br...@diddyinc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11165/
-----------------------------------------------------------
(Updated July 24, 2013, 10:17 p.m.)
Review request for mesos.
Changes
-------
Rebasing on master.
Repository: mesos
Description
-------
Write Mesos JobTracker stats to state file.
Review: https://reviews.apache.org/r/11165/
Diffs (updated)
-----
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java 279f84e0f0c43ad3cfd9e4442010e706ee3565d9
Diff: https://reviews.apache.org/r/11165/diff/
Testing
-------
Used in production at airbnb.
make -j10 check && cd hadoop && make hadoop-2.0.0-mr1-cdh4.2.1 && make hadoop-0.20.205.0 && make hadoop-0.20.2-cdh3u3
Thanks,
Brenden Matthews
Re: Review Request: Write Mesos JobTracker stats to state file.
Posted by Brenden Matthews <br...@diddyinc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11165/
-----------------------------------------------------------
(Updated June 11, 2013, 7:33 p.m.)
Review request for mesos.
Changes
-------
Rebasing on master.
Description
-------
Write Mesos JobTracker stats to state file.
Review: https://reviews.apache.org/r/11165/
Diffs (updated)
-----
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java afe401f5265e3d9494af7eace42eec45943184a3
Diff: https://reviews.apache.org/r/11165/diff/
Testing (updated)
-------
Used in production at airbnb.
make -j10 check && cd hadoop && make hadoop-2.0.0-mr1-cdh4.2.1 && make hadoop-0.20.205.0 && make hadoop-0.20.2-cdh3u3
Thanks,
Brenden Matthews
Re: Review Request: Write Mesos JobTracker stats to state file.
Posted by Brenden Matthews <br...@diddyinc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11165/
-----------------------------------------------------------
(Updated June 6, 2013, 2:10 a.m.)
Review request for mesos.
Changes
-------
Rebasing on master.
Description
-------
Write Mesos JobTracker stats to state file.
Review: https://reviews.apache.org/r/11165/
Diffs (updated)
-----
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java afe401f5265e3d9494af7eace42eec45943184a3
Diff: https://reviews.apache.org/r/11165/diff/
Testing
-------
Thanks,
Brenden Matthews
Re: Review Request: Write Mesos JobTracker stats to state file.
Posted by Brenden Matthews <br...@diddyinc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11165/
-----------------------------------------------------------
(Updated June 3, 2013, 5:59 p.m.)
Review request for mesos.
Description (updated)
-------
Write Mesos JobTracker stats to state file.
Review: https://reviews.apache.org/r/11165/
Diffs (updated)
-----
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java afe401f5265e3d9494af7eace42eec45943184a3
Diff: https://reviews.apache.org/r/11165/diff/
Testing
-------
Thanks,
Brenden Matthews