missing hadoop utility scripts

["MrAsanjar ." <af...@gmail.com>]

Hi all,
TAll Hadoop utility scripts (i.e. start-all.sh, start-dfs.sh,..) in
/usr/lib/hadoop*/sbin are missing, am I missing a package?

platform: Ubuntu 14.04, x86_64
install: apt-get install hadoop*

cheers

Re: missing hadoop utility scripts

[Konstantin Boudnik <co...@apache.org>]

On Tue, Apr 26, 2016 at 02:48PM, Roman Shaposhnik wrote:
> On Tue, Apr 26, 2016 at 2:43 PM, MrAsanjar . <af...@gmail.com> wrote:
> > Thanks for reply Cos,
> > I understand your reasoning and fully agree to your approach as part of
> > best practices. Having said that, shouldn't that be up to users to decide?
> 
> Actually, not necessarily. The usage of these scripts breaks deployment
> practices with orchestration done by Puppet, Ambari and higher level
> orchestration
> tools in general. IOW, shipping them would encourage behavior that
> can only frustrate.

Yup. Although, user should be able to decide for sure. In this particular
case, user can fork the Bigtop source code and produce a different set of
packages _including_ these scripts. Or to convince this group of developers to
add them back with all consequences listed above.

Cos

> But let me flip this question on you: why do you need them? What's your
> use case?
> 
> Thanks,
> Roman.

Re: missing hadoop utility scripts

[Konstantin Boudnik <co...@apache.org>]

Yup, that stuff goes directly to hadoop-daemon.sh (or whatever the name is)
and circumvent the normal init.d (systemd when supported) life-cycle
managements. Meaning that no environment will be read, etc.

Cos

On Wed, Apr 27, 2016 at 09:43PM, Bruno Mahé wrote:
> Do these scripts actually work with Apache Bigtop packages?
> Last I remember they were also making strong assumptions about the
> layout of the files and would just not work.
> 
> Thanks,
> Bruno
> 
> On 04/27/2016 04:33 AM, MrAsanjar . wrote:
> >Roman,
> >I respectfully disagree and not going to start a debate on proper test and
> >development methodology, as they are very much so customer and workload
> >centric.
> >Regardless, I would like to propose to package these scripts
> >separately (i.e. called hadoop-utils.deb/rpm) and let the admin decide
> >whether to utilize these scripts.
> >
> >On Tue, Apr 26, 2016 at 7:17 PM, Roman Shaposhnik <ro...@shaposhnik.org>
> >wrote:
> >
> >>On Tue, Apr 26, 2016 at 4:23 PM, MrAsanjar . <af...@gmail.com> wrote:
> >>>these scripts get used extensively as part of our test and development,
> >>as
> >>>it provides lightweight cluster management without having any
> >>dependencies
> >>>on other tools like Ambari and puppet.
> >>Then I have to ask: if you're testing this as part of your end-to-end
> >>system
> >>testing you're really not testing the right thing (remember: in production
> >>those scripts are gone in pretty much most of the distros).
> >>
> >>If you're simply testing Hadoop from a quick smoke test perspective, why
> >>bother with packages to begin with? Untar the binary tarball an go.
> >>
> >>Thanks,
> >>Roman.
> >>
> >>P.S. I remember having this exact discussion with Cloudera QA team way back
> >>when: if you're not testing what your customers will be deploying you're
> >>not
> >>really testing much.
> >>
> 

Re: missing hadoop utility scripts

[Bruno Mahé <bm...@apache.org>]

Do these scripts actually work with Apache Bigtop packages?
Last I remember they were also making strong assumptions about the 
layout of the files and would just not work.

Thanks,
Bruno

On 04/27/2016 04:33 AM, MrAsanjar . wrote:
> Roman,
> I respectfully disagree and not going to start a debate on proper test and
> development methodology, as they are very much so customer and workload
> centric.
> Regardless, I would like to propose to package these scripts
> separately (i.e. called hadoop-utils.deb/rpm) and let the admin decide
> whether to utilize these scripts.
>
> On Tue, Apr 26, 2016 at 7:17 PM, Roman Shaposhnik <ro...@shaposhnik.org>
> wrote:
>
>> On Tue, Apr 26, 2016 at 4:23 PM, MrAsanjar . <af...@gmail.com> wrote:
>>> these scripts get used extensively as part of our test and development,
>> as
>>> it provides lightweight cluster management without having any
>> dependencies
>>> on other tools like Ambari and puppet.
>> Then I have to ask: if you're testing this as part of your end-to-end
>> system
>> testing you're really not testing the right thing (remember: in production
>> those scripts are gone in pretty much most of the distros).
>>
>> If you're simply testing Hadoop from a quick smoke test perspective, why
>> bother with packages to begin with? Untar the binary tarball an go.
>>
>> Thanks,
>> Roman.
>>
>> P.S. I remember having this exact discussion with Cloudera QA team way back
>> when: if you're not testing what your customers will be deploying you're
>> not
>> really testing much.
>>

Re: missing hadoop utility scripts

["MrAsanjar ." <af...@gmail.com>]

Roman,
I respectfully disagree and not going to start a debate on proper test and
development methodology, as they are very much so customer and workload
centric.
Regardless, I would like to propose to package these scripts
separately (i.e. called hadoop-utils.deb/rpm) and let the admin decide
whether to utilize these scripts.

On Tue, Apr 26, 2016 at 7:17 PM, Roman Shaposhnik <ro...@shaposhnik.org>
wrote:

> On Tue, Apr 26, 2016 at 4:23 PM, MrAsanjar . <af...@gmail.com> wrote:
> > these scripts get used extensively as part of our test and development,
> as
> > it provides lightweight cluster management without having any
> dependencies
> > on other tools like Ambari and puppet.
>
> Then I have to ask: if you're testing this as part of your end-to-end
> system
> testing you're really not testing the right thing (remember: in production
> those scripts are gone in pretty much most of the distros).
>
> If you're simply testing Hadoop from a quick smoke test perspective, why
> bother with packages to begin with? Untar the binary tarball an go.
>
> Thanks,
> Roman.
>
> P.S. I remember having this exact discussion with Cloudera QA team way back
> when: if you're not testing what your customers will be deploying you're
> not
> really testing much.
>

Re: missing hadoop utility scripts

[Roman Shaposhnik <ro...@shaposhnik.org>]

On Tue, Apr 26, 2016 at 4:23 PM, MrAsanjar . <af...@gmail.com> wrote:
> these scripts get used extensively as part of our test and development, as
> it provides lightweight cluster management without having any dependencies
> on other tools like Ambari and puppet.

Then I have to ask: if you're testing this as part of your end-to-end system
testing you're really not testing the right thing (remember: in production
those scripts are gone in pretty much most of the distros).

If you're simply testing Hadoop from a quick smoke test perspective, why
bother with packages to begin with? Untar the binary tarball an go.

Thanks,
Roman.

P.S. I remember having this exact discussion with Cloudera QA team way back
when: if you're not testing what your customers will be deploying you're not
really testing much.

Re: missing hadoop utility scripts

["MrAsanjar ." <af...@gmail.com>]

these scripts get used extensively as part of our test and development, as
it provides lightweight cluster management without having any dependencies
on other tools like Ambari and puppet.

On Tue, Apr 26, 2016 at 4:48 PM, Roman Shaposhnik <ro...@shaposhnik.org>
wrote:

> On Tue, Apr 26, 2016 at 2:43 PM, MrAsanjar . <af...@gmail.com> wrote:
> > Thanks for reply Cos,
> > I understand your reasoning and fully agree to your approach as part of
> > best practices. Having said that, shouldn't that be up to users to
> decide?
>
> Actually, not necessarily. The usage of these scripts breaks deployment
> practices with orchestration done by Puppet, Ambari and higher level
> orchestration
> tools in general. IOW, shipping them would encourage behavior that
> can only frustrate.
>
> But let me flip this question on you: why do you need them? What's your
> use case?
>
> Thanks,
> Roman.
>

Re: missing hadoop utility scripts

[Roman Shaposhnik <ro...@shaposhnik.org>]

On Tue, Apr 26, 2016 at 2:43 PM, MrAsanjar . <af...@gmail.com> wrote:
> Thanks for reply Cos,
> I understand your reasoning and fully agree to your approach as part of
> best practices. Having said that, shouldn't that be up to users to decide?

Actually, not necessarily. The usage of these scripts breaks deployment
practices with orchestration done by Puppet, Ambari and higher level
orchestration
tools in general. IOW, shipping them would encourage behavior that
can only frustrate.

But let me flip this question on you: why do you need them? What's your
use case?

Thanks,
Roman.

Re: missing hadoop utility scripts

["MrAsanjar ." <af...@gmail.com>]

Thanks for reply Cos,
I understand your reasoning and fully agree to your approach as part of
best practices. Having said that, shouldn't that be up to users to decide?



On Tue, Apr 26, 2016 at 4:22 PM, Konstantin Boudnik <co...@apache.org> wrote:

> These utility scripts are relying on password-less ssh across the cluster.
> Bigtop doesn't consider such setup as secure nor healthy. Hence, these
> scripts
> aren't included into the official packaging.
>
> Cos
>
> On Tue, Apr 26, 2016 at 04:12PM, MrAsanjar . wrote:
> > Hi all,
> > TAll Hadoop utility scripts (i.e. start-all.sh, start-dfs.sh,..) in
> > /usr/lib/hadoop*/sbin are missing, am I missing a package?
> >
> > platform: Ubuntu 14.04, x86_64
> > install: apt-get install hadoop*
> >
> > cheers
>

Re: missing hadoop utility scripts

[Konstantin Boudnik <co...@apache.org>]

These utility scripts are relying on password-less ssh across the cluster.
Bigtop doesn't consider such setup as secure nor healthy. Hence, these scripts
aren't included into the official packaging.

Cos

On Tue, Apr 26, 2016 at 04:12PM, MrAsanjar . wrote:
> Hi all,
> TAll Hadoop utility scripts (i.e. start-all.sh, start-dfs.sh,..) in
> /usr/lib/hadoop*/sbin are missing, am I missing a package?
> 
> platform: Ubuntu 14.04, x86_64
> install: apt-get install hadoop*
> 
> cheers