You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mesos.apache.org by Brenden Matthews <br...@diddyinc.com> on 2013/05/14 02:19:28 UTC
Review Request: Include pre/post-package hook scripts in Hadoop tutorial.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11110/
-----------------------------------------------------------
Review request for mesos.
Description
-------
>From decdc5c271c47b96d706666665313dae27321fff Mon Sep 17 00:00:00 2001
From: Brenden Matthews <br...@airbnb.com>
Date: Mon, 15 Apr 2013 14:11:16 -0700
Subject: [PATCH 03/24] Include pre/post-package hook scripts in Hadoop
tutorial.
Review: https://reviews.apache.org/r/11110
---
hadoop/TUTORIAL.sh | 24 ++++++++++++++++++++++++
1 file changed, 24 insertions(+)
Diffs
-----
hadoop/TUTORIAL.sh d81c55562d217e8784745d4a8cedf78ada9f5453
Diff: https://reviews.apache.org/r/11110/diff/
Testing
-------
Used in production at airbnb.
Thanks,
Brenden Matthews
Re: Review Request: Include pre/post-package hook scripts in Hadoop tutorial.
Posted by Brenden Matthews <br...@diddyinc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11110/
-----------------------------------------------------------
(Updated June 11, 2013, 9:44 p.m.)
Review request for mesos.
Changes
-------
Rebasing on master.
Description
-------
Include pre/post-package hook scripts in Hadoop tutorial.
Review: https://reviews.apache.org/r/11110
Diffs (updated)
-----
hadoop/TUTORIAL.sh d81c55562d217e8784745d4a8cedf78ada9f5453
Diff: https://reviews.apache.org/r/11110/diff/
Testing (updated)
-------
Used in production at airbnb.
make -j10 check && cd hadoop && make hadoop-2.0.0-mr1-cdh4.2.1 && make hadoop-0.20.205.0 && make hadoop-0.20.2-cdh3u3
Thanks,
Brenden Matthews
Re: Review Request: Include pre/post-package hook scripts in Hadoop tutorial.
Posted by Brenden Matthews <br...@diddyinc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11110/
-----------------------------------------------------------
(Updated June 6, 2013, 2:13 a.m.)
Review request for mesos.
Changes
-------
Rebasing on master.
Description
-------
Include pre/post-package hook scripts in Hadoop tutorial.
Review: https://reviews.apache.org/r/11110
Diffs (updated)
-----
hadoop/TUTORIAL.sh d81c55562d217e8784745d4a8cedf78ada9f5453
Diff: https://reviews.apache.org/r/11110/diff/
Testing
-------
Used in production at airbnb.
Thanks,
Brenden Matthews
Re: Review Request: Include pre/post-package hook scripts in Hadoop tutorial.
Posted by Brenden Matthews <br...@diddyinc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11110/
-----------------------------------------------------------
(Updated June 3, 2013, 5:57 p.m.)
Review request for mesos.
Description (updated)
-------
Include pre/post-package hook scripts in Hadoop tutorial.
Review: https://reviews.apache.org/r/11110
Diffs (updated)
-----
hadoop/TUTORIAL.sh d81c55562d217e8784745d4a8cedf78ada9f5453
Diff: https://reviews.apache.org/r/11110/diff/
Testing
-------
Used in production at airbnb.
Thanks,
Brenden Matthews
Re: Review Request: Include pre/post-package hook scripts in Hadoop tutorial.
Posted by Brenden Matthews <br...@diddyinc.com>.
> On May 16, 2013, 10:07 p.m., Benjamin Hindman wrote:
> > hadoop/TUTORIAL.sh, line 573
> > <https://reviews.apache.org/r/11110/diff/1/?file=290968#file290968line573>
> >
> > Curious, why can't you write a script which does:
> > ------------------------
> > distribution=...
> >
> > TUTORIAL.sh ${distribution}
> > /path/to/pre-package
> > tar czf hadoop-${distribution}-mesos-airbnb.tar.gz hadoop
> > /path/to/post-package
> > ------------------------
> >
> > That is, use composition to keep from needing to repeat anything.
> >
> > The real problem here is that the tutorial was never meant to be a build system for the Mesos distribution. How do other shops build hadoop specific distributions? Is there a clean declarative approach people use?
> >
> > Also, while I can imagine wanting to bring in other contribs or libraries before you package the distribution via pre-package, I can't imagine how post-package is useful if the only goal of the tutorial is to create a distribution (again, not to be a general build utility).
> >
> > I acknowledge the fact that creating a Mesos distribution is non-trivial (or even a Hadoop one for that matter) and so I would be okay with adding pre-package to the tutorial but I would like to see a lot more explanation given to the user running the tutorial about what the pre-package hook script is intended to be used for. In particular, if this is being used to add other contribs I'd think that this would need to get done before running ant.
I suppose I could do things the way you suggest. I generally like the idea of using the `make' command because it's familiar and well supported by other tools.
I'm not sure how others are building Hadoop distributions, but I would assume they make use of the existing Hadoop build system where possible. Since Mesos uses make, it follows that one should be able to build the Mesos Hadoop distribution with make as well.
The post-package script is not necessarily required (it could be done by another script afterwards). It seemed sensible at the time to handle the post-package tasks the same way as the pre-package tasks within the tutorial at the time of writing. I think it would be better to separate the creation of the distribution from the tutorial altogether, perhaps using just a Makefile to build the distribution.
- Brenden
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11110/#review20665
-----------------------------------------------------------
On May 14, 2013, 12:19 a.m., Brenden Matthews wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11110/
> -----------------------------------------------------------
>
> (Updated May 14, 2013, 12:19 a.m.)
>
>
> Review request for mesos.
>
>
> Description
> -------
>
> From decdc5c271c47b96d706666665313dae27321fff Mon Sep 17 00:00:00 2001
> From: Brenden Matthews <br...@airbnb.com>
> Date: Mon, 15 Apr 2013 14:11:16 -0700
> Subject: [PATCH 03/24] Include pre/post-package hook scripts in Hadoop
> tutorial.
>
> Review: https://reviews.apache.org/r/11110
> ---
> hadoop/TUTORIAL.sh | 24 ++++++++++++++++++++++++
> 1 file changed, 24 insertions(+)
>
>
> Diffs
> -----
>
> hadoop/TUTORIAL.sh d81c55562d217e8784745d4a8cedf78ada9f5453
>
> Diff: https://reviews.apache.org/r/11110/diff/
>
>
> Testing
> -------
>
> Used in production at airbnb.
>
>
> Thanks,
>
> Brenden Matthews
>
>
Re: Review Request: Include pre/post-package hook scripts in Hadoop tutorial.
Posted by Benjamin Hindman <be...@berkeley.edu>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11110/#review20665
-----------------------------------------------------------
hadoop/TUTORIAL.sh
<https://reviews.apache.org/r/11110/#comment42672>
Curious, why can't you write a script which does:
------------------------
distribution=...
TUTORIAL.sh ${distribution}
/path/to/pre-package
tar czf hadoop-${distribution}-mesos-airbnb.tar.gz hadoop
/path/to/post-package
------------------------
That is, use composition to keep from needing to repeat anything.
The real problem here is that the tutorial was never meant to be a build system for the Mesos distribution. How do other shops build hadoop specific distributions? Is there a clean declarative approach people use?
Also, while I can imagine wanting to bring in other contribs or libraries before you package the distribution via pre-package, I can't imagine how post-package is useful if the only goal of the tutorial is to create a distribution (again, not to be a general build utility).
I acknowledge the fact that creating a Mesos distribution is non-trivial (or even a Hadoop one for that matter) and so I would be okay with adding pre-package to the tutorial but I would like to see a lot more explanation given to the user running the tutorial about what the pre-package hook script is intended to be used for. In particular, if this is being used to add other contribs I'd think that this would need to get done before running ant.
- Benjamin Hindman
On May 14, 2013, 12:19 a.m., Brenden Matthews wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11110/
> -----------------------------------------------------------
>
> (Updated May 14, 2013, 12:19 a.m.)
>
>
> Review request for mesos.
>
>
> Description
> -------
>
> From decdc5c271c47b96d706666665313dae27321fff Mon Sep 17 00:00:00 2001
> From: Brenden Matthews <br...@airbnb.com>
> Date: Mon, 15 Apr 2013 14:11:16 -0700
> Subject: [PATCH 03/24] Include pre/post-package hook scripts in Hadoop
> tutorial.
>
> Review: https://reviews.apache.org/r/11110
> ---
> hadoop/TUTORIAL.sh | 24 ++++++++++++++++++++++++
> 1 file changed, 24 insertions(+)
>
>
> Diffs
> -----
>
> hadoop/TUTORIAL.sh d81c55562d217e8784745d4a8cedf78ada9f5453
>
> Diff: https://reviews.apache.org/r/11110/diff/
>
>
> Testing
> -------
>
> Used in production at airbnb.
>
>
> Thanks,
>
> Brenden Matthews
>
>