You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hawq.apache.org by Konstantin Boudnik <co...@apache.org> on 2016/02/25 06:14:14 UTC
Init weirdness
Guys,
I've been slowly working my way towards running the hawq out of the package
(BIGTOP-2320) and have stepped on something that I can not quite explain.
During the initial cluster init, there's an attempt to rewrite the
configuration file hawq-site.xml Because of the standard location of the
configs on linux (under /etc/<appname> directory) and default permissions
(root:root), rewriting of the configs like that is problematic and insecure at
least.
Hence the question: why configuration file has to be rewritten given that it
is already provided? Shall the logic be changed to not overriding an existing
hawq-site?
Regards,
Cos
Re: Init weirdness
Posted by Radar Da lei <rl...@pivotal.io>.
We have a near plan to let user define HAWQ configuration files directory,
so they can put files like 'hawq-site.xml' to any where they like.
I think we can put these together and make the changes then.
Thanks.
Regards,
Radar
On Fri, Feb 26, 2016 at 1:19 AM, Konstantin Boudnik <co...@apache.org> wrote:
> Dynamic configuration updates is a pretty unorthodox, if you ask me.
> Perhaps having an extra configuration file where such dynamically
> calculated
> values will go, might be a bit more conventional, don't you think?
> Basically,
> just leave the hawq-site along, and anything you need to modify should go
> into
> hawq-misc.xml or whatever the name might be. Then bootstrap sequence will
> look
> like this:
> hawq-site.xml <- hawq-misc.xml
> to guarantee that the latter source have precedence over the former.
> Otherwise, the deployment of this thing would be quite entertaining, to say
> the least.
>
> Cos
>
> On Thu, Feb 25, 2016 at 03:00PM, Radar Da lei wrote:
> > When doing 'hawq init', we will check the segments number and dynamic to
> > decide some property values, for example 'default_segment_num',
> > 'hawq_rm_nvseg_perquery_perseg_limit'. These properties will be write
> back
> > to hawq-site.xml by 'hawq config' command.
> >
> > Another situation you mentioned is when we doing remove standby master
> > node, we will remove standby master item from hawq-site.xml.
> >
> > And anytime you run 'hawq config' to re-define the GUC values,
> > hawq-site.xml is changed. This command might be used frequently.
> >
> > Thanks.
> >
> >
> > Regards,
> > Radar
> >
> > On Thu, Feb 25, 2016 at 2:15 PM, Konstantin Boudnik <co...@apache.org>
> wrote:
> >
> > > Thanks for the explanation, Radar. But I am talking about
> hawq-site.xml,
> > > which
> > > is being re-written and the original content is copied under
> > > hawq-site.xml.org
> > > I guess I pointed to a wrong function, as I see that _write_config() is
> > > doing
> > > exactly what you said.
> > >
> > > Generating of the new files might be fine as we can make hawq user to
> own
> > > the
> > > configuration directory. But whatever config files are there and
> belong to
> > > root, should remain this way, in my option. Looking through the rest
> of the
> > > script, I see that there's a number of attempts to remove properties
> from
> > > the
> > > hawq-site file. E.g.
> > >
> > > remove_property_xml("hawq_standby_address_host",
> "%s/etc/hawq-site.xml"
> > > % self.GPHOME)
> > >
> > > which would be the subject of the issue we're discussing
> > >
> > > Cos
> > >
> > > On Thu, Feb 25, 2016 at 01:46PM, Radar Da lei wrote:
> > > > Hi Konstantin,
> > > >
> > > > Do you mean the step we read properties in hawq-site.xml, and then
> > > re-write
> > > > them to the file '_mgmt_config' ?
> > > >
> > > > This is to read some parameters easier when doing hawq init. The file
> > > will
> > > > be generate during each 'hawq init', then it's useless.
> > > >
> > > > In the plan, it will be generated into '/tmp' directory instead of
> hawq
> > > etc
> > > > directory, and be removed after hawq init finished.
> > > >
> > > > Thanks.
> > > >
> > > > Regards,
> > > > Radar
> > > >
> > > > On Thu, Feb 25, 2016 at 1:14 PM, Konstantin Boudnik <co...@apache.org>
> > > wrote:
> > > >
> > > > > Guys,
> > > > >
> > > > > I've been slowly working my way towards running the hawq out of the
> > > package
> > > > > (BIGTOP-2320) and have stepped on something that I can not quite
> > > explain.
> > > > >
> > > > > During the initial cluster init, there's an attempt to rewrite the
> > > > > configuration file hawq-site.xml Because of the standard location
> of
> > > the
> > > > > configs on linux (under /etc/<appname> directory) and default
> > > permissions
> > > > > (root:root), rewriting of the configs like that is problematic and
> > > > > insecure at
> > > > > least.
> > > > >
> > > > > Hence the question: why configuration file has to be rewritten
> given
> > > that
> > > > > it
> > > > > is already provided? Shall the logic be changed to not overriding
> an
> > > > > existing
> > > > > hawq-site?
> > > > >
> > > > > Regards,
> > > > > Cos
> > > > >
> > >
>
Re: Init weirdness
Posted by Konstantin Boudnik <co...@apache.org>.
Dynamic configuration updates is a pretty unorthodox, if you ask me.
Perhaps having an extra configuration file where such dynamically calculated
values will go, might be a bit more conventional, don't you think? Basically,
just leave the hawq-site along, and anything you need to modify should go into
hawq-misc.xml or whatever the name might be. Then bootstrap sequence will look
like this:
hawq-site.xml <- hawq-misc.xml
to guarantee that the latter source have precedence over the former.
Otherwise, the deployment of this thing would be quite entertaining, to say
the least.
Cos
On Thu, Feb 25, 2016 at 03:00PM, Radar Da lei wrote:
> When doing 'hawq init', we will check the segments number and dynamic to
> decide some property values, for example 'default_segment_num',
> 'hawq_rm_nvseg_perquery_perseg_limit'. These properties will be write back
> to hawq-site.xml by 'hawq config' command.
>
> Another situation you mentioned is when we doing remove standby master
> node, we will remove standby master item from hawq-site.xml.
>
> And anytime you run 'hawq config' to re-define the GUC values,
> hawq-site.xml is changed. This command might be used frequently.
>
> Thanks.
>
>
> Regards,
> Radar
>
> On Thu, Feb 25, 2016 at 2:15 PM, Konstantin Boudnik <co...@apache.org> wrote:
>
> > Thanks for the explanation, Radar. But I am talking about hawq-site.xml,
> > which
> > is being re-written and the original content is copied under
> > hawq-site.xml.org
> > I guess I pointed to a wrong function, as I see that _write_config() is
> > doing
> > exactly what you said.
> >
> > Generating of the new files might be fine as we can make hawq user to own
> > the
> > configuration directory. But whatever config files are there and belong to
> > root, should remain this way, in my option. Looking through the rest of the
> > script, I see that there's a number of attempts to remove properties from
> > the
> > hawq-site file. E.g.
> >
> > remove_property_xml("hawq_standby_address_host", "%s/etc/hawq-site.xml"
> > % self.GPHOME)
> >
> > which would be the subject of the issue we're discussing
> >
> > Cos
> >
> > On Thu, Feb 25, 2016 at 01:46PM, Radar Da lei wrote:
> > > Hi Konstantin,
> > >
> > > Do you mean the step we read properties in hawq-site.xml, and then
> > re-write
> > > them to the file '_mgmt_config' ?
> > >
> > > This is to read some parameters easier when doing hawq init. The file
> > will
> > > be generate during each 'hawq init', then it's useless.
> > >
> > > In the plan, it will be generated into '/tmp' directory instead of hawq
> > etc
> > > directory, and be removed after hawq init finished.
> > >
> > > Thanks.
> > >
> > > Regards,
> > > Radar
> > >
> > > On Thu, Feb 25, 2016 at 1:14 PM, Konstantin Boudnik <co...@apache.org>
> > wrote:
> > >
> > > > Guys,
> > > >
> > > > I've been slowly working my way towards running the hawq out of the
> > package
> > > > (BIGTOP-2320) and have stepped on something that I can not quite
> > explain.
> > > >
> > > > During the initial cluster init, there's an attempt to rewrite the
> > > > configuration file hawq-site.xml Because of the standard location of
> > the
> > > > configs on linux (under /etc/<appname> directory) and default
> > permissions
> > > > (root:root), rewriting of the configs like that is problematic and
> > > > insecure at
> > > > least.
> > > >
> > > > Hence the question: why configuration file has to be rewritten given
> > that
> > > > it
> > > > is already provided? Shall the logic be changed to not overriding an
> > > > existing
> > > > hawq-site?
> > > >
> > > > Regards,
> > > > Cos
> > > >
> >
Re: Init weirdness
Posted by Radar Da lei <rl...@pivotal.io>.
When doing 'hawq init', we will check the segments number and dynamic to
decide some property values, for example 'default_segment_num',
'hawq_rm_nvseg_perquery_perseg_limit'. These properties will be write back
to hawq-site.xml by 'hawq config' command.
Another situation you mentioned is when we doing remove standby master
node, we will remove standby master item from hawq-site.xml.
And anytime you run 'hawq config' to re-define the GUC values,
hawq-site.xml is changed. This command might be used frequently.
Thanks.
Regards,
Radar
On Thu, Feb 25, 2016 at 2:15 PM, Konstantin Boudnik <co...@apache.org> wrote:
> Thanks for the explanation, Radar. But I am talking about hawq-site.xml,
> which
> is being re-written and the original content is copied under
> hawq-site.xml.org
> I guess I pointed to a wrong function, as I see that _write_config() is
> doing
> exactly what you said.
>
> Generating of the new files might be fine as we can make hawq user to own
> the
> configuration directory. But whatever config files are there and belong to
> root, should remain this way, in my option. Looking through the rest of the
> script, I see that there's a number of attempts to remove properties from
> the
> hawq-site file. E.g.
>
> remove_property_xml("hawq_standby_address_host", "%s/etc/hawq-site.xml"
> % self.GPHOME)
>
> which would be the subject of the issue we're discussing
>
> Cos
>
> On Thu, Feb 25, 2016 at 01:46PM, Radar Da lei wrote:
> > Hi Konstantin,
> >
> > Do you mean the step we read properties in hawq-site.xml, and then
> re-write
> > them to the file '_mgmt_config' ?
> >
> > This is to read some parameters easier when doing hawq init. The file
> will
> > be generate during each 'hawq init', then it's useless.
> >
> > In the plan, it will be generated into '/tmp' directory instead of hawq
> etc
> > directory, and be removed after hawq init finished.
> >
> > Thanks.
> >
> > Regards,
> > Radar
> >
> > On Thu, Feb 25, 2016 at 1:14 PM, Konstantin Boudnik <co...@apache.org>
> wrote:
> >
> > > Guys,
> > >
> > > I've been slowly working my way towards running the hawq out of the
> package
> > > (BIGTOP-2320) and have stepped on something that I can not quite
> explain.
> > >
> > > During the initial cluster init, there's an attempt to rewrite the
> > > configuration file hawq-site.xml Because of the standard location of
> the
> > > configs on linux (under /etc/<appname> directory) and default
> permissions
> > > (root:root), rewriting of the configs like that is problematic and
> > > insecure at
> > > least.
> > >
> > > Hence the question: why configuration file has to be rewritten given
> that
> > > it
> > > is already provided? Shall the logic be changed to not overriding an
> > > existing
> > > hawq-site?
> > >
> > > Regards,
> > > Cos
> > >
>
Re: Init weirdness
Posted by Konstantin Boudnik <co...@apache.org>.
Thanks for the explanation, Radar. But I am talking about hawq-site.xml, which
is being re-written and the original content is copied under hawq-site.xml.org
I guess I pointed to a wrong function, as I see that _write_config() is doing
exactly what you said.
Generating of the new files might be fine as we can make hawq user to own the
configuration directory. But whatever config files are there and belong to
root, should remain this way, in my option. Looking through the rest of the
script, I see that there's a number of attempts to remove properties from the
hawq-site file. E.g.
remove_property_xml("hawq_standby_address_host", "%s/etc/hawq-site.xml" % self.GPHOME)
which would be the subject of the issue we're discussing
Cos
On Thu, Feb 25, 2016 at 01:46PM, Radar Da lei wrote:
> Hi Konstantin,
>
> Do you mean the step we read properties in hawq-site.xml, and then re-write
> them to the file '_mgmt_config' ?
>
> This is to read some parameters easier when doing hawq init. The file will
> be generate during each 'hawq init', then it's useless.
>
> In the plan, it will be generated into '/tmp' directory instead of hawq etc
> directory, and be removed after hawq init finished.
>
> Thanks.
>
> Regards,
> Radar
>
> On Thu, Feb 25, 2016 at 1:14 PM, Konstantin Boudnik <co...@apache.org> wrote:
>
> > Guys,
> >
> > I've been slowly working my way towards running the hawq out of the package
> > (BIGTOP-2320) and have stepped on something that I can not quite explain.
> >
> > During the initial cluster init, there's an attempt to rewrite the
> > configuration file hawq-site.xml Because of the standard location of the
> > configs on linux (under /etc/<appname> directory) and default permissions
> > (root:root), rewriting of the configs like that is problematic and
> > insecure at
> > least.
> >
> > Hence the question: why configuration file has to be rewritten given that
> > it
> > is already provided? Shall the logic be changed to not overriding an
> > existing
> > hawq-site?
> >
> > Regards,
> > Cos
> >
Re: Init weirdness
Posted by Radar Da lei <rl...@pivotal.io>.
Hi Konstantin,
Do you mean the step we read properties in hawq-site.xml, and then re-write
them to the file '_mgmt_config' ?
This is to read some parameters easier when doing hawq init. The file will
be generate during each 'hawq init', then it's useless.
In the plan, it will be generated into '/tmp' directory instead of hawq etc
directory, and be removed after hawq init finished.
Thanks.
Regards,
Radar
On Thu, Feb 25, 2016 at 1:14 PM, Konstantin Boudnik <co...@apache.org> wrote:
> Guys,
>
> I've been slowly working my way towards running the hawq out of the package
> (BIGTOP-2320) and have stepped on something that I can not quite explain.
>
> During the initial cluster init, there's an attempt to rewrite the
> configuration file hawq-site.xml Because of the standard location of the
> configs on linux (under /etc/<appname> directory) and default permissions
> (root:root), rewriting of the configs like that is problematic and
> insecure at
> least.
>
> Hence the question: why configuration file has to be rewritten given that
> it
> is already provided? Shall the logic be changed to not overriding an
> existing
> hawq-site?
>
> Regards,
> Cos
>