You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@bigtop.apache.org by Martin Bukatovic <mb...@redhat.com> on 2015/06/05 18:36:34 UTC
deployment of bigtop stable release
Dear Bigtop user list,
I have a question about preferred way of using bigtop releases.
When I do not intend to build all hadoop packages provided by bigtop
myself, I
download packages or use repository for particular bigtop release (eg.
the last
one 0.8.0 or upcoming 1.0).
But to actually deploy hadoop on any cluster, I also need puppet
manifests and
other files which are part of the release but not included in the packages
(which is completelly fine). To get those files, I would expect that I'm
supposed to use version which from the release as well, which means to
download source tarball or clone repository and checkout into proper release
tag.
On the other hand, this doesn't match what's written in the bigtop cwiki:
*
https://cwiki.apache.org/confluence/display/BIGTOP/How+to+install+BigTop+0.8.0+hadoop+on+bare+metal+%28or+cloud%29+CentOS+-+with+puppet
*
https://cwiki.apache.org/confluence/display/BIGTOP/How+to+install+BigTop+0.7.0+hadoop+on+CentOS+with+puppet
It states that I should use puppet 3 when installing release 0.8.0 (but that
is not necessary since this is true since BIGTOP-1047 which not part of that
release) or to clone bigtop git without checking release branch.
Jay told me that it's better to just build myself from git which I agree
with.
I was always using the current master and plan to do in the future, but I'm
puzzled what makes more sense when I decide to use the release
(to better understand how components fits together without being
hurt by moving parts).
Btw a bit unrelated, I have published my raw notes here:
https://gist.github.com/mbukatov/7ec73ae3a7eaa73a3135
and I would be able to polish it to replace current cwiki page mentioned
above
which is bit short right now.
--
Martin Bukatovic
Re: deployment of bigtop stable release
Posted by Martin Bukatovic <mb...@redhat.com>.
On 06/09/2015 04:10 AM, jay vyas wrote:
> @martin i like the idea of creating a wiki page for what you did with
> bigtop 0.8 ... But probably we don't really have a policy on this stuff,
> ... because most bigtop users are comfortable trading in the sharp
> edges so that they can build all the bits on their own from scratch.
Ok, good. I have created BIGTOP-1895 with proposed wikipage.
Also you can check it here:
https://gist.github.com/mbukatov/7ec73ae3a7eaa73a3135
--
Martin Bukatovic
RHS/Hadoop QE Team
Re: deployment of bigtop stable release
Posted by jay vyas <ja...@gmail.com>.
@martin i like the idea of creating a wiki page for what you did with
bigtop 0.8 ... But probably we don't really have a policy on this stuff,
... because most bigtop users are comfortable trading in the sharp edges
so that they can build all the bits on their own from scratch.
In general, we usually do this
- grab the RPMs/DEBs from jenkins, along with bigtop master.
- run puppet deploy.
If anything fails (i.e. if the RPMs are behind from the puppet recipes in
master),
we usually can just build the packages from master ourself using docker
containers, test w/ vagrant up and the --use-local-yum=true option.
thats the most common scenario and probably the most important, given that
in general its ok if bigtop is kind of a bleeding edge distribution for
most peoples needs.
I know thats kind of a lazy cop-out of an answer, but its probably
reasonably close to the current culture we have around here :)
On Mon, Jun 8, 2015 at 3:39 PM, Martin Bukatovic <mb...@redhat.com>
wrote:
> On 06/06/2015 09:29 PM, Evans Ye wrote:
> > Hey Martin, sorry to reply you late.
> > I think you can also refer to my reply in "newbie question of BigTop"
> > thread for the deployment.
> > I'm sorry that the document currently we have are a little bit out of
> date.
> > So thanks for wrapping up your notes! It would be great if we can put
> > that on our wiki.
> >
> > Getting back to your questions, I think you better use puppet recipes in
> > the master to deploy bigtop clusters at this moment. The reason behind
> > this is that our puppet recipes changed a lot from 0.8 to 1.0. If you
> > don't want to learn it twice, than you better use the newest version
> > directly. :)
> > The new puppet recipes can be used to deploy bigtop 0.8 packages as
> > well. AFAIK the incomparable component should be sqoop only.
> > For bigtop users, I think it's always better to use a fix release of
> > bigtop, since the master can have broken feature during development,
> > although we tend not to.
> > Hopefully I've answered your questions. if not, welcome to ask more. :)
>
> Thanks for the answer. I will polish my notes a bit and add your
> explanation there. Then I'm expected to open new JIRA for it, right?
>
> But I have another question, does it mean that this approach (using
> puppet recipes from master to deploy both stable release and current
> master) is a desing feature of bigtop so that:
>
> - it is going to work in the upcoming release? Will I be able to
> use future puppet recipes from master to deploy bigtop 1.0.0?
> Or I should rather ask: is this planned?
> - we are suggesting to always use deployment scripts from master
> with few exceptions (eg. sqoop you mentioned)?
>
> If it's true, we may like to advertise this a bit more.
> Maintaining list of hadoop components which are not deployable by
> current master orchestration would be nice so that we can state:
> always use master with those exceptions.
>
> --
> Martin Bukatovic
> RHS/Hadoop QE Team
>
--
jay vyas
Re: deployment of bigtop stable release
Posted by Martin Bukatovic <mb...@redhat.com>.
On 06/06/2015 09:29 PM, Evans Ye wrote:
> Hey Martin, sorry to reply you late.
> I think you can also refer to my reply in "newbie question of BigTop"
> thread for the deployment.
> I'm sorry that the document currently we have are a little bit out of date.
> So thanks for wrapping up your notes! It would be great if we can put
> that on our wiki.
>
> Getting back to your questions, I think you better use puppet recipes in
> the master to deploy bigtop clusters at this moment. The reason behind
> this is that our puppet recipes changed a lot from 0.8 to 1.0. If you
> don't want to learn it twice, than you better use the newest version
> directly. :)
> The new puppet recipes can be used to deploy bigtop 0.8 packages as
> well. AFAIK the incomparable component should be sqoop only.
> For bigtop users, I think it's always better to use a fix release of
> bigtop, since the master can have broken feature during development,
> although we tend not to.
> Hopefully I've answered your questions. if not, welcome to ask more. :)
Thanks for the answer. I will polish my notes a bit and add your
explanation there. Then I'm expected to open new JIRA for it, right?
But I have another question, does it mean that this approach (using
puppet recipes from master to deploy both stable release and current
master) is a desing feature of bigtop so that:
- it is going to work in the upcoming release? Will I be able to
use future puppet recipes from master to deploy bigtop 1.0.0?
Or I should rather ask: is this planned?
- we are suggesting to always use deployment scripts from master
with few exceptions (eg. sqoop you mentioned)?
If it's true, we may like to advertise this a bit more.
Maintaining list of hadoop components which are not deployable by
current master orchestration would be nice so that we can state:
always use master with those exceptions.
--
Martin Bukatovic
RHS/Hadoop QE Team
Re: deployment of bigtop stable release
Posted by Evans Ye <ev...@apache.org>.
Hey Martin, sorry to reply you late.
I think you can also refer to my reply in "newbie question of BigTop"
thread for the deployment.
I'm sorry that the document currently we have are a little bit out of date.
So thanks for wrapping up your notes! It would be great if we can put that
on our wiki.
Getting back to your questions, I think you better use puppet recipes in
the master to deploy bigtop clusters at this moment. The reason behind this
is that our puppet recipes changed a lot from 0.8 to 1.0. If you don't want
to learn it twice, than you better use the newest version directly. :)
The new puppet recipes can be used to deploy bigtop 0.8 packages as well.
AFAIK the incomparable component should be sqoop only.
For bigtop users, I think it's always better to use a fix release of
bigtop, since the master can have broken feature during development,
although we tend not to.
Hopefully I've answered your questions. if not, welcome to ask more. :)
2015-06-06 0:36 GMT+08:00 Martin Bukatovic <mb...@redhat.com>:
> Dear Bigtop user list,
>
> I have a question about preferred way of using bigtop releases.
>
> When I do not intend to build all hadoop packages provided by bigtop
> myself, I
> download packages or use repository for particular bigtop release (eg.
> the last
> one 0.8.0 or upcoming 1.0).
>
> But to actually deploy hadoop on any cluster, I also need puppet
> manifests and
> other files which are part of the release but not included in the packages
> (which is completelly fine). To get those files, I would expect that I'm
> supposed to use version which from the release as well, which means to
> download source tarball or clone repository and checkout into proper
> release
> tag.
>
> On the other hand, this doesn't match what's written in the bigtop cwiki:
>
> *
>
> https://cwiki.apache.org/confluence/display/BIGTOP/How+to+install+BigTop+0.8.0+hadoop+on+bare+metal+%28or+cloud%29+CentOS+-+with+puppet
> *
>
> https://cwiki.apache.org/confluence/display/BIGTOP/How+to+install+BigTop+0.7.0+hadoop+on+CentOS+with+puppet
>
> It states that I should use puppet 3 when installing release 0.8.0 (but
> that
> is not necessary since this is true since BIGTOP-1047 which not part of
> that
> release) or to clone bigtop git without checking release branch.
>
> Jay told me that it's better to just build myself from git which I agree
> with.
> I was always using the current master and plan to do in the future, but I'm
> puzzled what makes more sense when I decide to use the release
> (to better understand how components fits together without being
> hurt by moving parts).
>
> Btw a bit unrelated, I have published my raw notes here:
>
> https://gist.github.com/mbukatov/7ec73ae3a7eaa73a3135
>
> and I would be able to polish it to replace current cwiki page mentioned
> above
> which is bit short right now.
>
> --
> Martin Bukatovic
>