bigtop-utils

[Jay Vyas <ja...@gmail.com>]

Hi sean and others:

I want to replace the init-hdfs.sh
(.//bigtop-packages/src/common/hadoop/init-hdfs.sh)

with provision.groovy
(.//bigtop-packages/src/common/bigtop-utils/provision.groovy)

It seems pretty complex.

Any pointers on the lifecycle of bigtop-utils files in terms of where and
how they get installed in the rpms?

Nothing detailed, just some guidelines on how the utilities are leveraged
during provisioning of a cluster, and what needs to be done to integrate a
new utility into
the cluster....

i can grep around for the specifics.

Re: bigtop-utils

[Jay Vyas <ja...@gmail.com>]

okay just updated the jira to rename it...  the point of bigtop-1235 is to
integrate all this
stuff and do cleanup where necessary, and then push a new way of
provisioning
bigtop .

feel free if you want to add/revise the new bullets in
https://issues.apache.org/jira/browse/BIGTOP-1235


On Wed, Apr 23, 2014 at 11:53 AM, Sean Mackrory <ma...@gmail.com>wrote:

> 1) I'm 50/50 on whether or not we keep init-hdfs.sh as a wrapper to the new
> groovy script. On the one hand, I personally have a lot of stuff that uses
> init-hdfs.sh, on the other hand, I don't want to keep around legacy stuff
> without a very good reason. It should be simple enough to migrate to the
> new script now that groovy is packaged as a part of Bigtop
>
> 2) I'm not sure what you mean by the strategy of lifecycle of bigtop-utils.
> It's just a few scripts that are useful for lots of components. When we can
> improve them we improve them.
>
> If provision.groovy is only doing filesystem stuff for the foreseeable
> future, I'd prefer we just call it init-hcfs.groovy rather than something
> as generic as "provision". Although I only just noticed provision.groovy is
> already added to the bigtop-utils package, so maybe I'm a bit late to the
> party :)
>
>
> On Wed, Apr 23, 2014 at 9:39 AM, Jay Vyas <ja...@gmail.com> wrote:
>
> > Ok thanks Sean that helps. Two questions (response to your q's at
> bottom).
> >
> > 1) Do you folks think Maybe we should keep init-hdfs.sh in place and call
> > provision.groovy from it?
> >
> > 2) Also..Sean or others: Any more brain dumping you can share on the
> > strategy of packaging scripts of bigtop-util?
> >
> > ......
> >
> > Good point about init-hdfs being hdfs specific.  Yes provision.groovy is
> > meant to be 100% hcfs compatible, loading fs from configuration inject
> and
> > init-hcfs.json rather than explicitly doing things as hdfs.
> >
> >
> > > On Apr 23, 2014, at 9:02 AM, Sean Mackrory <ma...@gmail.com>
> wrote:
> > >
> > > init-hdfs.sh is specific to Hadoop's filesystem, so it belongs under
> > > /usr/lib/hadoop. Is Provision.groovy specific to HDFS, or is it for any
> > > HCFS? And is it even intended to always be specific to filesystems, or
> > will
> > > it also be used to do initialization of other services? I think all
> that
> > > determines where it should be, and unless it is specific to HDFS it
> > doesn't
> > > belong in the same place as init-hdfs.sh is now.
> > >
> > >
> > >> On Wed, Apr 23, 2014 at 6:24 AM, Jay Vyas <ja...@gmail.com>
> wrote:
> > >>
> > >> Hmmm ok... I'll ask a different way to clarify.
> > >>
> > >> should provision.groovy be in src/common/hadoop as well? Right now
> it's
> > >> in another directory all together.
> > >>
> > >> After all it is meant to be a drop in replacement for init-hdfs.sh.
> > >>
> > >> So in current bigtop... either provison.groovy OR init-hdfs.sh is in
> the
> > >> wrong place, don't you think?
> > >>
> > >>> On Apr 23, 2014, at 12:42 AM, Konstantin Boudnik <co...@apache.org>
> > wrote:
> > >>>
> > >>> Jay,
> > >>>
> > >>> I think the script is packed up into the hadoop package and is relied
> > >> upon by
> > >>> Puppet. So, it shouldn't be too hard to replace it. What seems to be
> > the
> > >>> trouble?
> > >>>
> > >>> Cos
> > >>>
> > >>>> On Wed, Apr 23, 2014 at 12:31AM, Jay Vyas wrote:
> > >>>> Hi sean and others:
> > >>>>
> > >>>> I want to replace the init-hdfs.sh
> > >>>> (.//bigtop-packages/src/common/hadoop/init-hdfs.sh)
> > >>>>
> > >>>> with provision.groovy
> > >>>> (.//bigtop-packages/src/common/bigtop-utils/provision.groovy)
> > >>>>
> > >>>> It seems pretty complex.
> > >>>>
> > >>>> Any pointers on the lifecycle of bigtop-utils files in terms of
> where
> > >> and
> > >>>> how they get installed in the rpms?
> > >>>>
> > >>>> Nothing detailed, just some guidelines on how the utilities are
> > >> leveraged
> > >>>> during provisioning of a cluster, and what needs to be done to
> > >> integrate a
> > >>>> new utility into
> > >>>> the cluster....
> > >>>>
> > >>>> i can grep around for the specifics.
> > >>
> >
>



-- 
Jay Vyas
http://jayunit100.blogspot.com

Re: bigtop-utils

[Sean Mackrory <ma...@gmail.com>]

1) I'm 50/50 on whether or not we keep init-hdfs.sh as a wrapper to the new
groovy script. On the one hand, I personally have a lot of stuff that uses
init-hdfs.sh, on the other hand, I don't want to keep around legacy stuff
without a very good reason. It should be simple enough to migrate to the
new script now that groovy is packaged as a part of Bigtop

2) I'm not sure what you mean by the strategy of lifecycle of bigtop-utils.
It's just a few scripts that are useful for lots of components. When we can
improve them we improve them.

If provision.groovy is only doing filesystem stuff for the foreseeable
future, I'd prefer we just call it init-hcfs.groovy rather than something
as generic as "provision". Although I only just noticed provision.groovy is
already added to the bigtop-utils package, so maybe I'm a bit late to the
party :)


On Wed, Apr 23, 2014 at 9:39 AM, Jay Vyas <ja...@gmail.com> wrote:

> Ok thanks Sean that helps. Two questions (response to your q's at bottom).
>
> 1) Do you folks think Maybe we should keep init-hdfs.sh in place and call
> provision.groovy from it?
>
> 2) Also..Sean or others: Any more brain dumping you can share on the
> strategy of packaging scripts of bigtop-util?
>
> ......
>
> Good point about init-hdfs being hdfs specific.  Yes provision.groovy is
> meant to be 100% hcfs compatible, loading fs from configuration inject and
> init-hcfs.json rather than explicitly doing things as hdfs.
>
>
> > On Apr 23, 2014, at 9:02 AM, Sean Mackrory <ma...@gmail.com> wrote:
> >
> > init-hdfs.sh is specific to Hadoop's filesystem, so it belongs under
> > /usr/lib/hadoop. Is Provision.groovy specific to HDFS, or is it for any
> > HCFS? And is it even intended to always be specific to filesystems, or
> will
> > it also be used to do initialization of other services? I think all that
> > determines where it should be, and unless it is specific to HDFS it
> doesn't
> > belong in the same place as init-hdfs.sh is now.
> >
> >
> >> On Wed, Apr 23, 2014 at 6:24 AM, Jay Vyas <ja...@gmail.com> wrote:
> >>
> >> Hmmm ok... I'll ask a different way to clarify.
> >>
> >> should provision.groovy be in src/common/hadoop as well? Right now it's
> >> in another directory all together.
> >>
> >> After all it is meant to be a drop in replacement for init-hdfs.sh.
> >>
> >> So in current bigtop... either provison.groovy OR init-hdfs.sh is in the
> >> wrong place, don't you think?
> >>
> >>> On Apr 23, 2014, at 12:42 AM, Konstantin Boudnik <co...@apache.org>
> wrote:
> >>>
> >>> Jay,
> >>>
> >>> I think the script is packed up into the hadoop package and is relied
> >> upon by
> >>> Puppet. So, it shouldn't be too hard to replace it. What seems to be
> the
> >>> trouble?
> >>>
> >>> Cos
> >>>
> >>>> On Wed, Apr 23, 2014 at 12:31AM, Jay Vyas wrote:
> >>>> Hi sean and others:
> >>>>
> >>>> I want to replace the init-hdfs.sh
> >>>> (.//bigtop-packages/src/common/hadoop/init-hdfs.sh)
> >>>>
> >>>> with provision.groovy
> >>>> (.//bigtop-packages/src/common/bigtop-utils/provision.groovy)
> >>>>
> >>>> It seems pretty complex.
> >>>>
> >>>> Any pointers on the lifecycle of bigtop-utils files in terms of where
> >> and
> >>>> how they get installed in the rpms?
> >>>>
> >>>> Nothing detailed, just some guidelines on how the utilities are
> >> leveraged
> >>>> during provisioning of a cluster, and what needs to be done to
> >> integrate a
> >>>> new utility into
> >>>> the cluster....
> >>>>
> >>>> i can grep around for the specifics.
> >>
>

Re: bigtop-utils

[Jay Vyas <ja...@gmail.com>]

Ok thanks Sean that helps. Two questions (response to your q's at bottom).

1) Do you folks think Maybe we should keep init-hdfs.sh in place and call provision.groovy from it?

2) Also..Sean or others: Any more brain dumping you can share on the strategy of packaging scripts of bigtop-util?

......

Good point about init-hdfs being hdfs specific.  Yes provision.groovy is meant to be 100% hcfs compatible, loading fs from configuration inject and init-hcfs.json rather than explicitly doing things as hdfs.


> On Apr 23, 2014, at 9:02 AM, Sean Mackrory <ma...@gmail.com> wrote:
> 
> init-hdfs.sh is specific to Hadoop's filesystem, so it belongs under
> /usr/lib/hadoop. Is Provision.groovy specific to HDFS, or is it for any
> HCFS? And is it even intended to always be specific to filesystems, or will
> it also be used to do initialization of other services? I think all that
> determines where it should be, and unless it is specific to HDFS it doesn't
> belong in the same place as init-hdfs.sh is now.
> 
> 
>> On Wed, Apr 23, 2014 at 6:24 AM, Jay Vyas <ja...@gmail.com> wrote:
>> 
>> Hmmm ok... I'll ask a different way to clarify.
>> 
>> should provision.groovy be in src/common/hadoop as well? Right now it's
>> in another directory all together.
>> 
>> After all it is meant to be a drop in replacement for init-hdfs.sh.
>> 
>> So in current bigtop... either provison.groovy OR init-hdfs.sh is in the
>> wrong place, don't you think?
>> 
>>> On Apr 23, 2014, at 12:42 AM, Konstantin Boudnik <co...@apache.org> wrote:
>>> 
>>> Jay,
>>> 
>>> I think the script is packed up into the hadoop package and is relied
>> upon by
>>> Puppet. So, it shouldn't be too hard to replace it. What seems to be the
>>> trouble?
>>> 
>>> Cos
>>> 
>>>> On Wed, Apr 23, 2014 at 12:31AM, Jay Vyas wrote:
>>>> Hi sean and others:
>>>> 
>>>> I want to replace the init-hdfs.sh
>>>> (.//bigtop-packages/src/common/hadoop/init-hdfs.sh)
>>>> 
>>>> with provision.groovy
>>>> (.//bigtop-packages/src/common/bigtop-utils/provision.groovy)
>>>> 
>>>> It seems pretty complex.
>>>> 
>>>> Any pointers on the lifecycle of bigtop-utils files in terms of where
>> and
>>>> how they get installed in the rpms?
>>>> 
>>>> Nothing detailed, just some guidelines on how the utilities are
>> leveraged
>>>> during provisioning of a cluster, and what needs to be done to
>> integrate a
>>>> new utility into
>>>> the cluster....
>>>> 
>>>> i can grep around for the specifics.
>> 

Re: bigtop-utils

[Sean Mackrory <ma...@gmail.com>]

init-hdfs.sh is specific to Hadoop's filesystem, so it belongs under
/usr/lib/hadoop. Is Provision.groovy specific to HDFS, or is it for any
HCFS? And is it even intended to always be specific to filesystems, or will
it also be used to do initialization of other services? I think all that
determines where it should be, and unless it is specific to HDFS it doesn't
belong in the same place as init-hdfs.sh is now.


On Wed, Apr 23, 2014 at 6:24 AM, Jay Vyas <ja...@gmail.com> wrote:

> Hmmm ok... I'll ask a different way to clarify.
>
>  should provision.groovy be in src/common/hadoop as well? Right now it's
> in another directory all together.
>
> After all it is meant to be a drop in replacement for init-hdfs.sh.
>
> So in current bigtop... either provison.groovy OR init-hdfs.sh is in the
> wrong place, don't you think?
>
> > On Apr 23, 2014, at 12:42 AM, Konstantin Boudnik <co...@apache.org> wrote:
> >
> > Jay,
> >
> > I think the script is packed up into the hadoop package and is relied
> upon by
> > Puppet. So, it shouldn't be too hard to replace it. What seems to be the
> > trouble?
> >
> > Cos
> >
> >> On Wed, Apr 23, 2014 at 12:31AM, Jay Vyas wrote:
> >> Hi sean and others:
> >>
> >> I want to replace the init-hdfs.sh
> >> (.//bigtop-packages/src/common/hadoop/init-hdfs.sh)
> >>
> >> with provision.groovy
> >> (.//bigtop-packages/src/common/bigtop-utils/provision.groovy)
> >>
> >> It seems pretty complex.
> >>
> >> Any pointers on the lifecycle of bigtop-utils files in terms of where
> and
> >> how they get installed in the rpms?
> >>
> >> Nothing detailed, just some guidelines on how the utilities are
> leveraged
> >> during provisioning of a cluster, and what needs to be done to
> integrate a
> >> new utility into
> >> the cluster....
> >>
> >> i can grep around for the specifics.
>

Re: bigtop-utils

[Jay Vyas <ja...@gmail.com>]

Hmmm ok... I'll ask a different way to clarify.

 should provision.groovy be in src/common/hadoop as well? Right now it's in another directory all together.

After all it is meant to be a drop in replacement for init-hdfs.sh.

So in current bigtop... either provison.groovy OR init-hdfs.sh is in the wrong place, don't you think?

> On Apr 23, 2014, at 12:42 AM, Konstantin Boudnik <co...@apache.org> wrote:
> 
> Jay,
> 
> I think the script is packed up into the hadoop package and is relied upon by
> Puppet. So, it shouldn't be too hard to replace it. What seems to be the
> trouble?
> 
> Cos
> 
>> On Wed, Apr 23, 2014 at 12:31AM, Jay Vyas wrote:
>> Hi sean and others:
>> 
>> I want to replace the init-hdfs.sh
>> (.//bigtop-packages/src/common/hadoop/init-hdfs.sh)
>> 
>> with provision.groovy
>> (.//bigtop-packages/src/common/bigtop-utils/provision.groovy)
>> 
>> It seems pretty complex.
>> 
>> Any pointers on the lifecycle of bigtop-utils files in terms of where and
>> how they get installed in the rpms?
>> 
>> Nothing detailed, just some guidelines on how the utilities are leveraged
>> during provisioning of a cluster, and what needs to be done to integrate a
>> new utility into
>> the cluster....
>> 
>> i can grep around for the specifics.

Re: bigtop-utils

[Konstantin Boudnik <co...@apache.org>]

Jay,

I think the script is packed up into the hadoop package and is relied upon by
Puppet. So, it shouldn't be too hard to replace it. What seems to be the
trouble?

Cos

On Wed, Apr 23, 2014 at 12:31AM, Jay Vyas wrote:
> Hi sean and others:
> 
> I want to replace the init-hdfs.sh
> (.//bigtop-packages/src/common/hadoop/init-hdfs.sh)
> 
> with provision.groovy
> (.//bigtop-packages/src/common/bigtop-utils/provision.groovy)
> 
> It seems pretty complex.
> 
> Any pointers on the lifecycle of bigtop-utils files in terms of where and
> how they get installed in the rpms?
> 
> Nothing detailed, just some guidelines on how the utilities are leveraged
> during provisioning of a cluster, and what needs to be done to integrate a
> new utility into
> the cluster....
> 
> i can grep around for the specifics.