question on pg_hba.conf updates

[Vineet Goel <vv...@apache.org>]

Hi all,

Question related to integration with Apache Ambari.

It would be nice to make pg_hba.conf visible and editable in Ambari, so
that Ambari allows one single interface for admins to update HAWQ and
System configs such as hawq-site.xml, hawq-check.conf, sysctl.conf,
limits.conf, hdfs-client.xml, yarn-client.xml etc. Rollback and
version-history of config files is always a bonus in Ambari.

Are there any backend HAWQ utilities (such as activate-standby or others)
that update pg_hba.conf file in any way, ever? It would be nice to know so
that config file change conflicts are managed appropriately.

Thanks
Vineet

Re: question on pg_hba.conf updates

[陶征霖 <zt...@apache.org>]

So far as I know, there is no HAWQ management tools to update
$MASTER_DATA_DIRECTORY/pg_hba.conf

2016-07-12 10:08 GMT+08:00 Vineet Goel <vv...@apache.org>:

> Hi all,
>
> Question related to integration with Apache Ambari.
>
> It would be nice to make pg_hba.conf visible and editable in Ambari, so
> that Ambari allows one single interface for admins to update HAWQ and
> System configs such as hawq-site.xml, hawq-check.conf, sysctl.conf,
> limits.conf, hdfs-client.xml, yarn-client.xml etc. Rollback and
> version-history of config files is always a bonus in Ambari.
>
> Are there any backend HAWQ utilities (such as activate-standby or others)
> that update pg_hba.conf file in any way, ever? It would be nice to know so
> that config file change conflicts are managed appropriately.
>
> Thanks
> Vineet
>

Re: question on pg_hba.conf updates

[Alexander Denissov <ad...@pivotal.io>]

If we make pg_hba.conf manageable via Ambari this will mean:
- the content of this file will be the same on all the nodes where it is
applies (see below for details)
- the file will be ovewritten by Ambari when the service restarts

We do have some flexibility on which hosts we apply the Ambari-managed
templates:
- active master only
- active and standby only
- active and standby and segments
We can also have different templates for different categories, if
absolutely required, but this might be a bit confusing to the user.

Since Ambari will be managing the file, whatever changes init standby /
activate standby are doing to the files should be similar to what Ambari
will overwrite it with. So a few questions:

- Can we have the file content be the same on active / standby / segments ?
- If not, what are the concerns and what are the valid combinations ?
- What is the logic for file content that HAWQ tools currently have ? We
will need to replicate it in Ambari, keeping in mind that Ambari has
information about cluster topology and system users

--
Thanks,
Alex.

On Mon, Jul 11, 2016 at 9:40 PM, Jiali Yao <jy...@pivotal.io> wrote:

> Hi Radar and Vineet,
>
> One more, I think it should be good make pg_hba.conf visible and
> editable in Amabari.
> And it should be synced automatically with standby when pg_hba is modified
> by in master via Amabari.
> Thought?
>
> Jiali
>
>
> On Tue, Jul 12, 2016 at 12:24 PM, Radar Da lei <rl...@pivotal.io> wrote:
>
> > Hi Jiali,
> >
> > It would be great if we can have this in our document. We can list some
> > frequently used scenarios and add to document together.
> >
> > Regards,
> > Radar
> >
> > On Tue, Jul 12, 2016 at 11:32 AM, Jiali Yao <jy...@pivotal.io> wrote:
> >
> > > Hi Radar,
> > >
> > > I think we should add the scenario Ming described in document in case
> > when
> > > master is crashed after user add some entries in pg_hba.conf.
> > > Thought?
> > >
> > > Jiali
> > >
> > >
> > > On Tue, Jul 12, 2016 at 11:12 AM, Radar Da lei <rl...@pivotal.io>
> wrote:
> > >
> > > > Hi Ming,
> > > >
> > > > If user manually added some entires into pg_hba.conf and they want it
> > be
> > > > available after activated standby, then user need to add them to
> > > standby's
> > > > pg_hba.conf as well.
> > > >
> > > > We do not do sync things for such case. Thanks.
> > > >
> > > > Regards,
> > > > Radar
> > > >
> > > > On Tue, Jul 12, 2016 at 11:05 AM, Ming Li <ml...@pivotal.io> wrote:
> > > >
> > > > > Hi Radar,
> > > > >
> > > > > If user manually added some entries into master pg_hba.conf, after
> we
> > > > > active standby as new master, those entries should be exists on the
> > new
> > > > > master(previous standby).
> > > > >
> > > > > So my question is: when those entries copied? At init standby or
> > active
> > > > > standby?
> > > > >
> > > > > Thanks.
> > > > >
> > > > > On Tue, Jul 12, 2016 at 10:49 AM, Radar Da lei <rl...@pivotal.io>
> > > wrote:
> > > > >
> > > > > > Hi Vineet,
> > > > > >
> > > > > > While doing HAWQ cluster init, we will collect
> > master/standby/segment
> > > > ip
> > > > > > addresses and update pg_hba.conf. After hawq init, content of
> > > > > 'pg_hba.conf'
> > > > > > on each node should be different.
> > > > > >
> > > > > > When a cluster adding a new standby, this will get pg_hba.conf
> > > updated
> > > > on
> > > > > > all the nodes. Activate standby will not update pg_hba.conf since
> > all
> > > > > > entires already in standby and segments's pg_hba.conf.
> > > > > >
> > > > > > Adding a new segment will only generate a new pg_hba.conf for
> > itself.
> > > > > >
> > > > > > These are what I can think out for this moment. Thanks.
> > > > > >
> > > > > > Regards,
> > > > > > Radar
> > > > > >
> > > > > > On Tue, Jul 12, 2016 at 10:08 AM, Vineet Goel <
> vvineet@apache.org>
> > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > Question related to integration with Apache Ambari.
> > > > > > >
> > > > > > > It would be nice to make pg_hba.conf visible and editable in
> > > Ambari,
> > > > so
> > > > > > > that Ambari allows one single interface for admins to update
> HAWQ
> > > and
> > > > > > > System configs such as hawq-site.xml, hawq-check.conf,
> > sysctl.conf,
> > > > > > > limits.conf, hdfs-client.xml, yarn-client.xml etc. Rollback and
> > > > > > > version-history of config files is always a bonus in Ambari.
> > > > > > >
> > > > > > > Are there any backend HAWQ utilities (such as activate-standby
> or
> > > > > others)
> > > > > > > that update pg_hba.conf file in any way, ever? It would be nice
> > to
> > > > know
> > > > > > so
> > > > > > > that config file change conflicts are managed appropriately.
> > > > > > >
> > > > > > > Thanks
> > > > > > > Vineet
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: question on pg_hba.conf updates

[Jiali Yao <jy...@pivotal.io>]

Hi Radar and Vineet,

One more, I think it should be good make pg_hba.conf visible and
editable in Amabari.
And it should be synced automatically with standby when pg_hba is modified
by in master via Amabari.
Thought?

Jiali


On Tue, Jul 12, 2016 at 12:24 PM, Radar Da lei <rl...@pivotal.io> wrote:

> Hi Jiali,
>
> It would be great if we can have this in our document. We can list some
> frequently used scenarios and add to document together.
>
> Regards,
> Radar
>
> On Tue, Jul 12, 2016 at 11:32 AM, Jiali Yao <jy...@pivotal.io> wrote:
>
> > Hi Radar,
> >
> > I think we should add the scenario Ming described in document in case
> when
> > master is crashed after user add some entries in pg_hba.conf.
> > Thought?
> >
> > Jiali
> >
> >
> > On Tue, Jul 12, 2016 at 11:12 AM, Radar Da lei <rl...@pivotal.io> wrote:
> >
> > > Hi Ming,
> > >
> > > If user manually added some entires into pg_hba.conf and they want it
> be
> > > available after activated standby, then user need to add them to
> > standby's
> > > pg_hba.conf as well.
> > >
> > > We do not do sync things for such case. Thanks.
> > >
> > > Regards,
> > > Radar
> > >
> > > On Tue, Jul 12, 2016 at 11:05 AM, Ming Li <ml...@pivotal.io> wrote:
> > >
> > > > Hi Radar,
> > > >
> > > > If user manually added some entries into master pg_hba.conf, after we
> > > > active standby as new master, those entries should be exists on the
> new
> > > > master(previous standby).
> > > >
> > > > So my question is: when those entries copied? At init standby or
> active
> > > > standby?
> > > >
> > > > Thanks.
> > > >
> > > > On Tue, Jul 12, 2016 at 10:49 AM, Radar Da lei <rl...@pivotal.io>
> > wrote:
> > > >
> > > > > Hi Vineet,
> > > > >
> > > > > While doing HAWQ cluster init, we will collect
> master/standby/segment
> > > ip
> > > > > addresses and update pg_hba.conf. After hawq init, content of
> > > > 'pg_hba.conf'
> > > > > on each node should be different.
> > > > >
> > > > > When a cluster adding a new standby, this will get pg_hba.conf
> > updated
> > > on
> > > > > all the nodes. Activate standby will not update pg_hba.conf since
> all
> > > > > entires already in standby and segments's pg_hba.conf.
> > > > >
> > > > > Adding a new segment will only generate a new pg_hba.conf for
> itself.
> > > > >
> > > > > These are what I can think out for this moment. Thanks.
> > > > >
> > > > > Regards,
> > > > > Radar
> > > > >
> > > > > On Tue, Jul 12, 2016 at 10:08 AM, Vineet Goel <vv...@apache.org>
> > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > Question related to integration with Apache Ambari.
> > > > > >
> > > > > > It would be nice to make pg_hba.conf visible and editable in
> > Ambari,
> > > so
> > > > > > that Ambari allows one single interface for admins to update HAWQ
> > and
> > > > > > System configs such as hawq-site.xml, hawq-check.conf,
> sysctl.conf,
> > > > > > limits.conf, hdfs-client.xml, yarn-client.xml etc. Rollback and
> > > > > > version-history of config files is always a bonus in Ambari.
> > > > > >
> > > > > > Are there any backend HAWQ utilities (such as activate-standby or
> > > > others)
> > > > > > that update pg_hba.conf file in any way, ever? It would be nice
> to
> > > know
> > > > > so
> > > > > > that config file change conflicts are managed appropriately.
> > > > > >
> > > > > > Thanks
> > > > > > Vineet
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: question on pg_hba.conf updates

[Radar Da lei <rl...@pivotal.io>]

Hi Jiali,

It would be great if we can have this in our document. We can list some
frequently used scenarios and add to document together.

Regards,
Radar

On Tue, Jul 12, 2016 at 11:32 AM, Jiali Yao <jy...@pivotal.io> wrote:

> Hi Radar,
>
> I think we should add the scenario Ming described in document in case when
> master is crashed after user add some entries in pg_hba.conf.
> Thought?
>
> Jiali
>
>
> On Tue, Jul 12, 2016 at 11:12 AM, Radar Da lei <rl...@pivotal.io> wrote:
>
> > Hi Ming,
> >
> > If user manually added some entires into pg_hba.conf and they want it be
> > available after activated standby, then user need to add them to
> standby's
> > pg_hba.conf as well.
> >
> > We do not do sync things for such case. Thanks.
> >
> > Regards,
> > Radar
> >
> > On Tue, Jul 12, 2016 at 11:05 AM, Ming Li <ml...@pivotal.io> wrote:
> >
> > > Hi Radar,
> > >
> > > If user manually added some entries into master pg_hba.conf, after we
> > > active standby as new master, those entries should be exists on the new
> > > master(previous standby).
> > >
> > > So my question is: when those entries copied? At init standby or active
> > > standby?
> > >
> > > Thanks.
> > >
> > > On Tue, Jul 12, 2016 at 10:49 AM, Radar Da lei <rl...@pivotal.io>
> wrote:
> > >
> > > > Hi Vineet,
> > > >
> > > > While doing HAWQ cluster init, we will collect master/standby/segment
> > ip
> > > > addresses and update pg_hba.conf. After hawq init, content of
> > > 'pg_hba.conf'
> > > > on each node should be different.
> > > >
> > > > When a cluster adding a new standby, this will get pg_hba.conf
> updated
> > on
> > > > all the nodes. Activate standby will not update pg_hba.conf since all
> > > > entires already in standby and segments's pg_hba.conf.
> > > >
> > > > Adding a new segment will only generate a new pg_hba.conf for itself.
> > > >
> > > > These are what I can think out for this moment. Thanks.
> > > >
> > > > Regards,
> > > > Radar
> > > >
> > > > On Tue, Jul 12, 2016 at 10:08 AM, Vineet Goel <vv...@apache.org>
> > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > Question related to integration with Apache Ambari.
> > > > >
> > > > > It would be nice to make pg_hba.conf visible and editable in
> Ambari,
> > so
> > > > > that Ambari allows one single interface for admins to update HAWQ
> and
> > > > > System configs such as hawq-site.xml, hawq-check.conf, sysctl.conf,
> > > > > limits.conf, hdfs-client.xml, yarn-client.xml etc. Rollback and
> > > > > version-history of config files is always a bonus in Ambari.
> > > > >
> > > > > Are there any backend HAWQ utilities (such as activate-standby or
> > > others)
> > > > > that update pg_hba.conf file in any way, ever? It would be nice to
> > know
> > > > so
> > > > > that config file change conflicts are managed appropriately.
> > > > >
> > > > > Thanks
> > > > > Vineet
> > > > >
> > > >
> > >
> >
>

Re: question on pg_hba.conf updates

[Jiali Yao <jy...@pivotal.io>]

Hi Radar,

I think we should add the scenario Ming described in document in case when
master is crashed after user add some entries in pg_hba.conf.
Thought?

Jiali


On Tue, Jul 12, 2016 at 11:12 AM, Radar Da lei <rl...@pivotal.io> wrote:

> Hi Ming,
>
> If user manually added some entires into pg_hba.conf and they want it be
> available after activated standby, then user need to add them to standby's
> pg_hba.conf as well.
>
> We do not do sync things for such case. Thanks.
>
> Regards,
> Radar
>
> On Tue, Jul 12, 2016 at 11:05 AM, Ming Li <ml...@pivotal.io> wrote:
>
> > Hi Radar,
> >
> > If user manually added some entries into master pg_hba.conf, after we
> > active standby as new master, those entries should be exists on the new
> > master(previous standby).
> >
> > So my question is: when those entries copied? At init standby or active
> > standby?
> >
> > Thanks.
> >
> > On Tue, Jul 12, 2016 at 10:49 AM, Radar Da lei <rl...@pivotal.io> wrote:
> >
> > > Hi Vineet,
> > >
> > > While doing HAWQ cluster init, we will collect master/standby/segment
> ip
> > > addresses and update pg_hba.conf. After hawq init, content of
> > 'pg_hba.conf'
> > > on each node should be different.
> > >
> > > When a cluster adding a new standby, this will get pg_hba.conf updated
> on
> > > all the nodes. Activate standby will not update pg_hba.conf since all
> > > entires already in standby and segments's pg_hba.conf.
> > >
> > > Adding a new segment will only generate a new pg_hba.conf for itself.
> > >
> > > These are what I can think out for this moment. Thanks.
> > >
> > > Regards,
> > > Radar
> > >
> > > On Tue, Jul 12, 2016 at 10:08 AM, Vineet Goel <vv...@apache.org>
> > wrote:
> > >
> > > > Hi all,
> > > >
> > > > Question related to integration with Apache Ambari.
> > > >
> > > > It would be nice to make pg_hba.conf visible and editable in Ambari,
> so
> > > > that Ambari allows one single interface for admins to update HAWQ and
> > > > System configs such as hawq-site.xml, hawq-check.conf, sysctl.conf,
> > > > limits.conf, hdfs-client.xml, yarn-client.xml etc. Rollback and
> > > > version-history of config files is always a bonus in Ambari.
> > > >
> > > > Are there any backend HAWQ utilities (such as activate-standby or
> > others)
> > > > that update pg_hba.conf file in any way, ever? It would be nice to
> know
> > > so
> > > > that config file change conflicts are managed appropriately.
> > > >
> > > > Thanks
> > > > Vineet
> > > >
> > >
> >
>

Re: question on pg_hba.conf updates

[Radar Da lei <rl...@pivotal.io>]

Hi Ming,

If user manually added some entires into pg_hba.conf and they want it be
available after activated standby, then user need to add them to standby's
pg_hba.conf as well.

We do not do sync things for such case. Thanks.

Regards,
Radar

On Tue, Jul 12, 2016 at 11:05 AM, Ming Li <ml...@pivotal.io> wrote:

> Hi Radar,
>
> If user manually added some entries into master pg_hba.conf, after we
> active standby as new master, those entries should be exists on the new
> master(previous standby).
>
> So my question is: when those entries copied? At init standby or active
> standby?
>
> Thanks.
>
> On Tue, Jul 12, 2016 at 10:49 AM, Radar Da lei <rl...@pivotal.io> wrote:
>
> > Hi Vineet,
> >
> > While doing HAWQ cluster init, we will collect master/standby/segment ip
> > addresses and update pg_hba.conf. After hawq init, content of
> 'pg_hba.conf'
> > on each node should be different.
> >
> > When a cluster adding a new standby, this will get pg_hba.conf updated on
> > all the nodes. Activate standby will not update pg_hba.conf since all
> > entires already in standby and segments's pg_hba.conf.
> >
> > Adding a new segment will only generate a new pg_hba.conf for itself.
> >
> > These are what I can think out for this moment. Thanks.
> >
> > Regards,
> > Radar
> >
> > On Tue, Jul 12, 2016 at 10:08 AM, Vineet Goel <vv...@apache.org>
> wrote:
> >
> > > Hi all,
> > >
> > > Question related to integration with Apache Ambari.
> > >
> > > It would be nice to make pg_hba.conf visible and editable in Ambari, so
> > > that Ambari allows one single interface for admins to update HAWQ and
> > > System configs such as hawq-site.xml, hawq-check.conf, sysctl.conf,
> > > limits.conf, hdfs-client.xml, yarn-client.xml etc. Rollback and
> > > version-history of config files is always a bonus in Ambari.
> > >
> > > Are there any backend HAWQ utilities (such as activate-standby or
> others)
> > > that update pg_hba.conf file in any way, ever? It would be nice to know
> > so
> > > that config file change conflicts are managed appropriately.
> > >
> > > Thanks
> > > Vineet
> > >
> >
>

Re: question on pg_hba.conf updates

[Ming Li <ml...@pivotal.io>]

Hi Radar,

If user manually added some entries into master pg_hba.conf, after we
active standby as new master, those entries should be exists on the new
master(previous standby).

So my question is: when those entries copied? At init standby or active
standby?

Thanks.

On Tue, Jul 12, 2016 at 10:49 AM, Radar Da lei <rl...@pivotal.io> wrote:

> Hi Vineet,
>
> While doing HAWQ cluster init, we will collect master/standby/segment ip
> addresses and update pg_hba.conf. After hawq init, content of 'pg_hba.conf'
> on each node should be different.
>
> When a cluster adding a new standby, this will get pg_hba.conf updated on
> all the nodes. Activate standby will not update pg_hba.conf since all
> entires already in standby and segments's pg_hba.conf.
>
> Adding a new segment will only generate a new pg_hba.conf for itself.
>
> These are what I can think out for this moment. Thanks.
>
> Regards,
> Radar
>
> On Tue, Jul 12, 2016 at 10:08 AM, Vineet Goel <vv...@apache.org> wrote:
>
> > Hi all,
> >
> > Question related to integration with Apache Ambari.
> >
> > It would be nice to make pg_hba.conf visible and editable in Ambari, so
> > that Ambari allows one single interface for admins to update HAWQ and
> > System configs such as hawq-site.xml, hawq-check.conf, sysctl.conf,
> > limits.conf, hdfs-client.xml, yarn-client.xml etc. Rollback and
> > version-history of config files is always a bonus in Ambari.
> >
> > Are there any backend HAWQ utilities (such as activate-standby or others)
> > that update pg_hba.conf file in any way, ever? It would be nice to know
> so
> > that config file change conflicts are managed appropriately.
> >
> > Thanks
> > Vineet
> >
>

Re: question on pg_hba.conf updates

[Lili Ma <li...@apache.org>]

To my understanding, pg_hba.conf is mainly used for client authentication,
and it will not conflict with other hawq configure files such as
hawq-site.xml, hdfs-client.xml, yart-client.xml,etc.

And for client authentication, we only need to change pg_hda.conf file on
HAWQ master node.

Please correct me if I'm wrong. Thanks.

Thanks
Lili

2016-07-12 10:49 GMT+08:00 Radar Da lei <rl...@pivotal.io>:

> Hi Vineet,
>
> While doing HAWQ cluster init, we will collect master/standby/segment ip
> addresses and update pg_hba.conf. After hawq init, content of 'pg_hba.conf'
> on each node should be different.
>
> When a cluster adding a new standby, this will get pg_hba.conf updated on
> all the nodes. Activate standby will not update pg_hba.conf since all
> entires already in standby and segments's pg_hba.conf.
>
> Adding a new segment will only generate a new pg_hba.conf for itself.
>
> These are what I can think out for this moment. Thanks.
>
> Regards,
> Radar
>
> On Tue, Jul 12, 2016 at 10:08 AM, Vineet Goel <vv...@apache.org> wrote:
>
> > Hi all,
> >
> > Question related to integration with Apache Ambari.
> >
> > It would be nice to make pg_hba.conf visible and editable in Ambari, so
> > that Ambari allows one single interface for admins to update HAWQ and
> > System configs such as hawq-site.xml, hawq-check.conf, sysctl.conf,
> > limits.conf, hdfs-client.xml, yarn-client.xml etc. Rollback and
> > version-history of config files is always a bonus in Ambari.
> >
> > Are there any backend HAWQ utilities (such as activate-standby or others)
> > that update pg_hba.conf file in any way, ever? It would be nice to know
> so
> > that config file change conflicts are managed appropriately.
> >
> > Thanks
> > Vineet
> >
>

Re: question on pg_hba.conf updates

[Radar Da lei <rl...@pivotal.io>]

Hi Vineet,

While doing HAWQ cluster init, we will collect master/standby/segment ip
addresses and update pg_hba.conf. After hawq init, content of 'pg_hba.conf'
on each node should be different.

When a cluster adding a new standby, this will get pg_hba.conf updated on
all the nodes. Activate standby will not update pg_hba.conf since all
entires already in standby and segments's pg_hba.conf.

Adding a new segment will only generate a new pg_hba.conf for itself.

These are what I can think out for this moment. Thanks.

Regards,
Radar

On Tue, Jul 12, 2016 at 10:08 AM, Vineet Goel <vv...@apache.org> wrote:

> Hi all,
>
> Question related to integration with Apache Ambari.
>
> It would be nice to make pg_hba.conf visible and editable in Ambari, so
> that Ambari allows one single interface for admins to update HAWQ and
> System configs such as hawq-site.xml, hawq-check.conf, sysctl.conf,
> limits.conf, hdfs-client.xml, yarn-client.xml etc. Rollback and
> version-history of config files is always a bonus in Ambari.
>
> Are there any backend HAWQ utilities (such as activate-standby or others)
> that update pg_hba.conf file in any way, ever? It would be nice to know so
> that config file change conflicts are managed appropriately.
>
> Thanks
> Vineet
>