You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by Owen O'Malley <om...@apache.org> on 2012/05/01 07:28:50 UTC
[DISCUSS] Move the default HADOOP_CONF_DIR
I'd like to propose that we change the default value for
HADOOP_CONF_DIR to /etc/hadoop instead of the current
/etc/hadoop/conf. I think it will reduce the confusion factor of new
users quite a bit. Roman shared that the original motivation was to
support multiple client configurations, but that seems relatively rare
use case. (Note that multiple client configs is still supported, of
course, you just need to use --config option to point it to the right
configuration directory. I'm just proposing that we remove the extra
conf out of the path.)
What do people think?
-- Owen
Re: [DISCUSS] Move the default HADOOP_CONF_DIR
Posted by Harsh J <ha...@cloudera.com>.
The following may be very specific an experience, but I felt you guys
may benefit from it:
>From my experience, this confusion (where one would type
/etc/hadoop/hdfs-site.xml quickly instead of
/etc/hadoop/conf/hdfs-site.xml) has indeed come up a (very) few times
with users/admins I've interacted with over the last year. These users
had never used tarball setups before, and didn't know of the "conf"
name and what that signifies. I _personally_ am comfortable with
/etc/hadoop/conf naming and instructions cause I came to package-world
after much tarball usage/deployments.
I'm OK with the change in general if it happens (its "standard" to
have files directly under /etc/app-name, and all apps ought to be
consistent), but I humbly wish we would not change for any existing
major releases. That'd cause much user pain for those already gotten
used to this (and *many* have). It would also break quite a few
puppet/chef scripts I think.
On Tue, May 1, 2012 at 10:58 AM, Owen O'Malley <om...@apache.org> wrote:
> I'd like to propose that we change the default value for
> HADOOP_CONF_DIR to /etc/hadoop instead of the current
> /etc/hadoop/conf. I think it will reduce the confusion factor of new
> users quite a bit. Roman shared that the original motivation was to
> support multiple client configurations, but that seems relatively rare
> use case. (Note that multiple client configs is still supported, of
> course, you just need to use --config option to point it to the right
> configuration directory. I'm just proposing that we remove the extra
> conf out of the path.)
>
> What do people think?
>
> -- Owen
--
Harsh J
Re: [DISCUSS] Move the default HADOOP_CONF_DIR
Posted by Konstantin Boudnik <co...@apache.org>.
On Wed, May 02, 2012 at 09:26AM, Owen O'Malley wrote:
> On Tue, May 1, 2012 at 5:06 PM, Roman Shaposhnik <rv...@apache.org> wrote:
> > Personally, I'm concerned because of the following:
> > # this will actually increase confusion somewhat among the folks
> > who have used prior releases of Bigtop
>
> If you smooth off the rough edges of Bigtop, you'll get a lot more
> adoption. In particular, none of the Apache projects have used a
> pattern like that. If you look at the upstream projects like Hadoop,
> Pig, or Zookeeper rpms they all use /etc/hadoop.
I don't see how the adoption can be increased by changing of the value of an
env. var. ;) RPMs produced by different project might have different
structure - this is really up to the maintainer. I personally think
/etc/<prod>/<more stuff> is a way cleaner structure, because otherwise you'll
end up mixing things in the same folder if later on non-config related files
will need to end up in /etc/hadoop.
No, I don't see how it makes sense.
Cos
> > # whatever we do has to be consistent through Bigtop which
> > means changing all the /etc/<component>/conf locations as well.
>
> Agreed.
>
> > That will be quite a bit of work as far as validation is concerned.
>
> Wouldn't the smoke tests be sufficient? Clearly none of the upstream
> projects has any dependency of the string /etc/$tool/conf.
>
> > AFAIK, if you deploy Hadoop from
> > tarballs it doesn't actually assume anything in /etc, since it bundles
> > all of the configs under the installation tree.
>
> This wouldn't change tarball deploys at all, naturally, since they
> don't have any natural config directory except for following a link
> from $tool/conf.
>
> -- Owen
Re: [DISCUSS] Move the default HADOOP_CONF_DIR
Posted by Owen O'Malley <om...@apache.org>.
On Tue, May 1, 2012 at 5:06 PM, Roman Shaposhnik <rv...@apache.org> wrote:
> Personally, I'm concerned because of the following:
> # this will actually increase confusion somewhat among the folks
> who have used prior releases of Bigtop
If you smooth off the rough edges of Bigtop, you'll get a lot more
adoption. In particular, none of the Apache projects have used a
pattern like that. If you look at the upstream projects like Hadoop,
Pig, or Zookeeper rpms they all use /etc/hadoop.
> # whatever we do has to be consistent through Bigtop which
> means changing all the /etc/<component>/conf locations as well.
Agreed.
> That will be quite a bit of work as far as validation is concerned.
Wouldn't the smoke tests be sufficient? Clearly none of the upstream
projects has any dependency of the string /etc/$tool/conf.
> AFAIK, if you deploy Hadoop from
> tarballs it doesn't actually assume anything in /etc, since it bundles
> all of the configs under the installation tree.
This wouldn't change tarball deploys at all, naturally, since they
don't have any natural config directory except for following a link
from $tool/conf.
-- Owen
Re: [DISCUSS] Move the default HADOOP_CONF_DIR
Posted by Roman Shaposhnik <rv...@apache.org>.
On Mon, Apr 30, 2012 at 10:28 PM, Owen O'Malley <om...@apache.org> wrote:
> I'd like to propose that we change the default value for
> HADOOP_CONF_DIR to /etc/hadoop instead of the current
> /etc/hadoop/conf. I think it will reduce the confusion factor of new
> users quite a bit. Roman shared that the original motivation was to
> support multiple client configurations, but that seems relatively rare
> use case. (Note that multiple client configs is still supported, of
> course, you just need to use --config option to point it to the right
> configuration directory. I'm just proposing that we remove the extra
> conf out of the path.)
>
> What do people think?
Personally, I'm concerned because of the following:
# this will actually increase confusion somewhat among the folks
who have used prior releases of Bigtop
# whatever we do has to be consistent through Bigtop which
means changing all the /etc/<component>/conf locations as well.
That will be quite a bit of work as far as validation is concerned.
Given all of the above, I would appreciate if the benefits of the change
were to be spelled out more clearly. AFAIK, if you deploy Hadoop from
tarballs it doesn't actually assume anything in /etc, since it bundles
all of the configs under the installation tree.
Thanks,
Roman.
Re: [DISCUSS] Move the default HADOOP_CONF_DIR
Posted by Bruno Mahé <bm...@apache.org>.
On 04/30/2012 10:28 PM, Owen O'Malley wrote:
> I'd like to propose that we change the default value for
> HADOOP_CONF_DIR to /etc/hadoop instead of the current
> /etc/hadoop/conf. I think it will reduce the confusion factor of new
> users quite a bit. Roman shared that the original motivation was to
> support multiple client configurations, but that seems relatively rare
> use case. (Note that multiple client configs is still supported, of
> course, you just need to use --config option to point it to the right
> configuration directory. I'm just proposing that we remove the extra
> conf out of the path.)
>
> What do people think?
>
> -- Owen
I would rather keep it the current way. /etc/<component>/conf> provides
flexibility and convenience while being obvious.
I also do not understand how it is confusing.
Thanks,
Bruno
Re: [DISCUSS] Move the default HADOOP_CONF_DIR
Posted by James Page <ja...@ubuntu.com>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
On 01/05/12 06:28, Owen O'Malley wrote:
> I'd like to propose that we change the default value for
> HADOOP_CONF_DIR to /etc/hadoop instead of the current
> /etc/hadoop/conf. I think it will reduce the confusion factor of
> new users quite a bit. Roman shared that the original motivation
> was to support multiple client configurations, but that seems
> relatively rare use case. (Note that multiple client configs is
> still supported, of course, you just need to use --config option to
> point it to the right configuration directory. I'm just proposing
> that we remove the extra conf out of the path.)
>
> What do people think?
I really did not like this when I first started to use bigtop packages
but its actually grown on me over time.
Specifically I really like the use of the alternates system to
management different configurations on Ubuntu/Debian - it means I can
add my own configuration for each set of packages and have it override
the default config provided by the .deb's without actually changing it
- - which then avoids a whole load of pain when I come to upgrade and
the .deb config files have changed and need to be merged with my local
changed.
I'd prefer to make the current /etc/<component>/conf schema standard
across all bigtop packages.
- --
James Page
Ubuntu Core Developer
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQIcBAEBCAAGBQJPomSRAAoJEL/srsug59jDiFsQAIxvbZnMuFdkQG0G6AnBbe1G
34QA1+0riNbt9fh7azQ5UCMnYDSdpJo3oe6Q0lpLJKbwHnZkVYS243pPDdUgCMam
SsFaeWBdF/ufsjijUMWNXEf6DBU98lHSFiDUuP4BZlhjiTAN58uoBfW76j1mYp44
9NxSizHMGmRzgrmUhzsMumI2d1tEqnDI+tSwA1ibBU2boJZ7tNYNQ53WJfJkbDMd
NHBBDXNVnuxAxscpr2F+RmSEIC5LOSJE7Z0rElnCGwaeyquQEqaqpRgjD2IbufDa
715GkZrcvfoRIylw8j9AnXe9Et0AcrGLoieuVMbgOA8gA8y94uJG8341ZDfvDPzE
5s9MSeUV/3eYdIu8o8zc61Zymfuhi66SBtDUZQ3WqPzsxsJTxI7mbrICZrpxj6lf
QrQfJ3rgKvtzyszTPW8H0qLS232IDM99hAT7V3E8csHQHSoeWgkLv1W5AmrZw16/
wgFcNbBGUH1BFBPI07mKyzXlQfbWEnwbqPFjufO+4vyOsAQWD5WeDYCD1qd0aNjL
P6i3YR33bVXF4uRRqQY4Sr0FdnpRgHDIUblBeS9bclvbDCghofi6kYaAEKFpcLk3
00CXOsYjo0k/VSfH2dPRjXFjMzgq1ufk7HfqSQPWx8h7znA+I4I6gVvZ1DmZ89T9
ENlY55RpwzRNmbV70gYR
=DU3J
-----END PGP SIGNATURE-----