You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hawq.apache.org by Shubham Sharma <ss...@pivotal.io> on 2017/11/13 03:58:20 UTC
Hawq standby sync fails if there are existing connections to master
Hello folks,
Recently observed a behaviour while re-syncing standby from hawq command
line.
Here are the reproduction steps -
1 - Open a client connection to hawq using psql
2 - From a different terminal run command - hawq init standby -n -v -M fast
3 - Standby resync fails with error
20171113:03:49:21:158354 hawq_stop:hdp3:gpadmin-[WARNING]:-There are
other connections to this instance, shutdown mode smart aborted
20171113:03:49:21:158354 hawq_stop:hdp3:gpadmin-[WARNING]:-Either
remove connections, or use 'hawq stop master -M fast' or 'hawq stop
master -M immediate'
20171113:03:49:21:158354 hawq_stop:hdp3:gpadmin-[WARNING]:-See hawq
stop --help for all options
20171113:03:49:21:158354 hawq_stop:hdp3:gpadmin-[ERROR]:-Active
connections. Aborting shutdown...
20171113:03:49:21:158143 hawq_init:hdp3:gpadmin-[ERROR]:-Stop hawq
cluster failed, exit
4 - My understanding is when -M (stop mode) is passed it should terminate
existing client connections. Also, it seems like a good practice to
terminate client connections before standby master resync.
Is this an expected behavior in hawq ? If not, I can open a JIRA and work
on a pull request to fix this.
Looking forward to your thoughts on this.
--
Regards,
Shubham Sharma
Re: Hawq standby sync fails if there are existing connections to master
Posted by Shubham Sharma <ss...@pivotal.io>.
Thanks Radar, fixed this under PR -
https://github.com/apache/incubator-hawq/pull/1311
On Mon, Nov 13, 2017 at 6:32 PM, Radar Lei <rl...@pivotal.io> wrote:
> Hi Shubham,
>
> The behavior is intentional but use '-M' option to force it should be good.
> Thanks.
>
> Regards,
> Radar
>
> On Tue, Nov 14, 2017 at 10:05 AM, Shubham Sharma <ss...@pivotal.io>
> wrote:
>
> > To close this thread, the source of this issue appears to be
> > tools/bin/hawq_ctl method _resync_standby. When this is called the
> command
> > formation does not include stop_mode options as passed to the arguments.
> >
> > def _resync_standby(self):
> > logger.info("Re-sync standby")
> > cmd = "%s; hawq stop master -a;" % source_hawq_env
> > check_return_code(local_ssh(cmd, logger), logger, "Stop hawq
> > cluster failed, exit")
> > ......
> > ......
> >
> > JIRA ID - HAWQ-1549
> >
> > On Mon, Nov 13, 2017 at 5:56 PM, Shubham Sharma <ss...@pivotal.io>
> > wrote:
> >
> > > Thanks, I think I found out where the issue is. Will create a JIRA and
> > > submit PR.
> > >
> > > On Mon, Nov 13, 2017 at 5:54 PM, Lei Chang <ch...@gmail.com>
> > wrote:
> > >
> > >> oh, yes, make sense, I overlooked the -M option here.
> > >>
> > >> Here we should respect the input "-M" option.
> > >>
> > >> Cheers
> > >> Lei
> > >>
> > >>
> > >>
> > >> On Tue, Nov 14, 2017 at 8:46 AM, Shubham Sharma <ss...@pivotal.io>
> > >> wrote:
> > >>
> > >> > Thanks for the response Lei.
> > >> >
> > >> > I understand this completely and agree with the behavior that we
> > should
> > >> not
> > >> > brutally terminate connections.
> > >> >
> > >> > However, the case here is, am deliberately trying to use stop mode
> > using
> > >> > command `hawq init standby -n -v -M fast`. At this point as a user I
> > >> > understand that connections will be terminated without warning. When
> > >> > passing option `-M fast` hawq should not interrupt me.
> > >> >
> > >> > Basically, hawq command line is not respecting `-M fast`. Whereas
> hawq
> > >> init
> > >> > standby --help documents this option. Let me know if this makes
> sense.
> > >> >
> > >> >
> > >> > hawq init standby --help
> > >> >
> > >> > Usage: HAWQ management scripts options
> > >> >
> > >> >
> > >> > Options:
> > >> >
> > >> > -h, --help show this help message and exit
> > >> >
> > >> > -a, --prompt Execute automatically
> > >> >
> > >> > -M STOP_MODE, --mode=STOP_MODE
> > >> >
> > >> > HAWQ stop mode: smart/fast/immediate
> > >> >
> > >> >
> > >> > On Mon, Nov 13, 2017 at 4:32 PM, Lei Chang <ch...@gmail.com>
> > >> wrote:
> > >> >
> > >> > > Hi Shubham,
> > >> > >
> > >> > > The behavior is intentional. If there are connections when HAWQ
> init
> > >> > > standby, it is better to warn the client instead of cutting the
> > >> > connections
> > >> > > brutally.
> > >> > >
> > >> > > Cheers
> > >> > > Lei
> > >> > >
> > >> > >
> > >> > >
> > >> > > On Mon, Nov 13, 2017 at 11:58 AM, Shubham Sharma <
> > ssharma@pivotal.io>
> > >> > > wrote:
> > >> > >
> > >> > > > Hello folks,
> > >> > > >
> > >> > > > Recently observed a behaviour while re-syncing standby from hawq
> > >> > command
> > >> > > > line.
> > >> > > >
> > >> > > > Here are the reproduction steps -
> > >> > > >
> > >> > > > 1 - Open a client connection to hawq using psql
> > >> > > > 2 - From a different terminal run command - hawq init standby -n
> > -v
> > >> -M
> > >> > > fast
> > >> > > > 3 - Standby resync fails with error
> > >> > > >
> > >> > > >
> > >> > > > 20171113:03:49:21:158354 hawq_stop:hdp3:gpadmin-[
> WARNING]:-There
> > >> are
> > >> > > > other connections to this instance, shutdown mode smart aborted
> > >> > > >
> > >> > > > 20171113:03:49:21:158354 hawq_stop:hdp3:gpadmin-[
> WARNING]:-Either
> > >> > > > remove connections, or use 'hawq stop master -M fast' or 'hawq
> > stop
> > >> > > > master -M immediate'
> > >> > > >
> > >> > > > 20171113:03:49:21:158354 hawq_stop:hdp3:gpadmin-[WARNING]:-See
> > hawq
> > >> > > > stop --help for all options
> > >> > > >
> > >> > > > 20171113:03:49:21:158354 hawq_stop:hdp3:gpadmin-[ERROR]:-Active
> > >> > > > connections. Aborting shutdown...
> > >> > > >
> > >> > > > 20171113:03:49:21:158143 hawq_init:hdp3:gpadmin-[ERROR]:-Stop
> > hawq
> > >> > > > cluster failed, exit
> > >> > > >
> > >> > > > 4 - My understanding is when -M (stop mode) is passed it should
> > >> > terminate
> > >> > > > existing client connections. Also, it seems like a good practice
> > to
> > >> > > > terminate client connections before standby master resync.
> > >> > > >
> > >> > > > Is this an expected behavior in hawq ? If not, I can open a JIRA
> > and
> > >> > work
> > >> > > > on a pull request to fix this.
> > >> > > >
> > >> > > > Looking forward to your thoughts on this.
> > >> > > >
> > >> > > > --
> > >> > > > Regards,
> > >> > > > Shubham Sharma
> > >> > > >
> > >> > >
> > >> >
> > >> >
> > >> >
> > >> > --
> > >> > Regards,
> > >> > Shubham Sharma
> > >> > Staff Customer Engineer
> > >> > Pivotal Global Support Services
> > >> > ssharma@pivotal.io
> > >> > Direct Tel: +1(510)-304-8201
> > >> > Office Hours: Mon-Fri 9:00 am to 5:00 pm PDT
> > >> > Out of Office Hours Contact +1 877-477-2269
> > >> >
> > >>
> > >
> > >
> > >
> > > --
> > > Regards,
> > > Shubham Sharma
> > > Staff Customer Engineer
> > > Pivotal Global Support Services
> > > ssharma@pivotal.io
> > > Direct Tel: +1(510)-304-8201 <(510)%20304-8201>
> > > Office Hours: Mon-Fri 9:00 am to 5:00 pm PDT
> > > Out of Office Hours Contact +1 877-477-2269 <(877)%20477-2269>
> > >
> >
> >
> >
> > --
> > Regards,
> > Shubham Sharma
> > Staff Customer Engineer
> > Pivotal Global Support Services
> > ssharma@pivotal.io
> > Direct Tel: +1(510)-304-8201
> > Office Hours: Mon-Fri 9:00 am to 5:00 pm PDT
> > Out of Office Hours Contact +1 877-477-2269
> >
>
--
Regards,
Shubham Sharma
Staff Customer Engineer
Pivotal Global Support Services
ssharma@pivotal.io
Direct Tel: +1(510)-304-8201
Office Hours: Mon-Fri 9:00 am to 5:00 pm PDT
Out of Office Hours Contact +1 877-477-2269
Re: Hawq standby sync fails if there are existing connections to master
Posted by Radar Lei <rl...@pivotal.io>.
Hi Shubham,
The behavior is intentional but use '-M' option to force it should be good.
Thanks.
Regards,
Radar
On Tue, Nov 14, 2017 at 10:05 AM, Shubham Sharma <ss...@pivotal.io> wrote:
> To close this thread, the source of this issue appears to be
> tools/bin/hawq_ctl method _resync_standby. When this is called the command
> formation does not include stop_mode options as passed to the arguments.
>
> def _resync_standby(self):
> logger.info("Re-sync standby")
> cmd = "%s; hawq stop master -a;" % source_hawq_env
> check_return_code(local_ssh(cmd, logger), logger, "Stop hawq
> cluster failed, exit")
> ......
> ......
>
> JIRA ID - HAWQ-1549
>
> On Mon, Nov 13, 2017 at 5:56 PM, Shubham Sharma <ss...@pivotal.io>
> wrote:
>
> > Thanks, I think I found out where the issue is. Will create a JIRA and
> > submit PR.
> >
> > On Mon, Nov 13, 2017 at 5:54 PM, Lei Chang <ch...@gmail.com>
> wrote:
> >
> >> oh, yes, make sense, I overlooked the -M option here.
> >>
> >> Here we should respect the input "-M" option.
> >>
> >> Cheers
> >> Lei
> >>
> >>
> >>
> >> On Tue, Nov 14, 2017 at 8:46 AM, Shubham Sharma <ss...@pivotal.io>
> >> wrote:
> >>
> >> > Thanks for the response Lei.
> >> >
> >> > I understand this completely and agree with the behavior that we
> should
> >> not
> >> > brutally terminate connections.
> >> >
> >> > However, the case here is, am deliberately trying to use stop mode
> using
> >> > command `hawq init standby -n -v -M fast`. At this point as a user I
> >> > understand that connections will be terminated without warning. When
> >> > passing option `-M fast` hawq should not interrupt me.
> >> >
> >> > Basically, hawq command line is not respecting `-M fast`. Whereas hawq
> >> init
> >> > standby --help documents this option. Let me know if this makes sense.
> >> >
> >> >
> >> > hawq init standby --help
> >> >
> >> > Usage: HAWQ management scripts options
> >> >
> >> >
> >> > Options:
> >> >
> >> > -h, --help show this help message and exit
> >> >
> >> > -a, --prompt Execute automatically
> >> >
> >> > -M STOP_MODE, --mode=STOP_MODE
> >> >
> >> > HAWQ stop mode: smart/fast/immediate
> >> >
> >> >
> >> > On Mon, Nov 13, 2017 at 4:32 PM, Lei Chang <ch...@gmail.com>
> >> wrote:
> >> >
> >> > > Hi Shubham,
> >> > >
> >> > > The behavior is intentional. If there are connections when HAWQ init
> >> > > standby, it is better to warn the client instead of cutting the
> >> > connections
> >> > > brutally.
> >> > >
> >> > > Cheers
> >> > > Lei
> >> > >
> >> > >
> >> > >
> >> > > On Mon, Nov 13, 2017 at 11:58 AM, Shubham Sharma <
> ssharma@pivotal.io>
> >> > > wrote:
> >> > >
> >> > > > Hello folks,
> >> > > >
> >> > > > Recently observed a behaviour while re-syncing standby from hawq
> >> > command
> >> > > > line.
> >> > > >
> >> > > > Here are the reproduction steps -
> >> > > >
> >> > > > 1 - Open a client connection to hawq using psql
> >> > > > 2 - From a different terminal run command - hawq init standby -n
> -v
> >> -M
> >> > > fast
> >> > > > 3 - Standby resync fails with error
> >> > > >
> >> > > >
> >> > > > 20171113:03:49:21:158354 hawq_stop:hdp3:gpadmin-[WARNING]:-There
> >> are
> >> > > > other connections to this instance, shutdown mode smart aborted
> >> > > >
> >> > > > 20171113:03:49:21:158354 hawq_stop:hdp3:gpadmin-[WARNING]:-Either
> >> > > > remove connections, or use 'hawq stop master -M fast' or 'hawq
> stop
> >> > > > master -M immediate'
> >> > > >
> >> > > > 20171113:03:49:21:158354 hawq_stop:hdp3:gpadmin-[WARNING]:-See
> hawq
> >> > > > stop --help for all options
> >> > > >
> >> > > > 20171113:03:49:21:158354 hawq_stop:hdp3:gpadmin-[ERROR]:-Active
> >> > > > connections. Aborting shutdown...
> >> > > >
> >> > > > 20171113:03:49:21:158143 hawq_init:hdp3:gpadmin-[ERROR]:-Stop
> hawq
> >> > > > cluster failed, exit
> >> > > >
> >> > > > 4 - My understanding is when -M (stop mode) is passed it should
> >> > terminate
> >> > > > existing client connections. Also, it seems like a good practice
> to
> >> > > > terminate client connections before standby master resync.
> >> > > >
> >> > > > Is this an expected behavior in hawq ? If not, I can open a JIRA
> and
> >> > work
> >> > > > on a pull request to fix this.
> >> > > >
> >> > > > Looking forward to your thoughts on this.
> >> > > >
> >> > > > --
> >> > > > Regards,
> >> > > > Shubham Sharma
> >> > > >
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > Regards,
> >> > Shubham Sharma
> >> > Staff Customer Engineer
> >> > Pivotal Global Support Services
> >> > ssharma@pivotal.io
> >> > Direct Tel: +1(510)-304-8201
> >> > Office Hours: Mon-Fri 9:00 am to 5:00 pm PDT
> >> > Out of Office Hours Contact +1 877-477-2269
> >> >
> >>
> >
> >
> >
> > --
> > Regards,
> > Shubham Sharma
> > Staff Customer Engineer
> > Pivotal Global Support Services
> > ssharma@pivotal.io
> > Direct Tel: +1(510)-304-8201 <(510)%20304-8201>
> > Office Hours: Mon-Fri 9:00 am to 5:00 pm PDT
> > Out of Office Hours Contact +1 877-477-2269 <(877)%20477-2269>
> >
>
>
>
> --
> Regards,
> Shubham Sharma
> Staff Customer Engineer
> Pivotal Global Support Services
> ssharma@pivotal.io
> Direct Tel: +1(510)-304-8201
> Office Hours: Mon-Fri 9:00 am to 5:00 pm PDT
> Out of Office Hours Contact +1 877-477-2269
>
Re: Hawq standby sync fails if there are existing connections to master
Posted by Shubham Sharma <ss...@pivotal.io>.
To close this thread, the source of this issue appears to be
tools/bin/hawq_ctl method _resync_standby. When this is called the command
formation does not include stop_mode options as passed to the arguments.
def _resync_standby(self):
logger.info("Re-sync standby")
cmd = "%s; hawq stop master -a;" % source_hawq_env
check_return_code(local_ssh(cmd, logger), logger, "Stop hawq
cluster failed, exit")
......
......
JIRA ID - HAWQ-1549
On Mon, Nov 13, 2017 at 5:56 PM, Shubham Sharma <ss...@pivotal.io> wrote:
> Thanks, I think I found out where the issue is. Will create a JIRA and
> submit PR.
>
> On Mon, Nov 13, 2017 at 5:54 PM, Lei Chang <ch...@gmail.com> wrote:
>
>> oh, yes, make sense, I overlooked the -M option here.
>>
>> Here we should respect the input "-M" option.
>>
>> Cheers
>> Lei
>>
>>
>>
>> On Tue, Nov 14, 2017 at 8:46 AM, Shubham Sharma <ss...@pivotal.io>
>> wrote:
>>
>> > Thanks for the response Lei.
>> >
>> > I understand this completely and agree with the behavior that we should
>> not
>> > brutally terminate connections.
>> >
>> > However, the case here is, am deliberately trying to use stop mode using
>> > command `hawq init standby -n -v -M fast`. At this point as a user I
>> > understand that connections will be terminated without warning. When
>> > passing option `-M fast` hawq should not interrupt me.
>> >
>> > Basically, hawq command line is not respecting `-M fast`. Whereas hawq
>> init
>> > standby --help documents this option. Let me know if this makes sense.
>> >
>> >
>> > hawq init standby --help
>> >
>> > Usage: HAWQ management scripts options
>> >
>> >
>> > Options:
>> >
>> > -h, --help show this help message and exit
>> >
>> > -a, --prompt Execute automatically
>> >
>> > -M STOP_MODE, --mode=STOP_MODE
>> >
>> > HAWQ stop mode: smart/fast/immediate
>> >
>> >
>> > On Mon, Nov 13, 2017 at 4:32 PM, Lei Chang <ch...@gmail.com>
>> wrote:
>> >
>> > > Hi Shubham,
>> > >
>> > > The behavior is intentional. If there are connections when HAWQ init
>> > > standby, it is better to warn the client instead of cutting the
>> > connections
>> > > brutally.
>> > >
>> > > Cheers
>> > > Lei
>> > >
>> > >
>> > >
>> > > On Mon, Nov 13, 2017 at 11:58 AM, Shubham Sharma <ss...@pivotal.io>
>> > > wrote:
>> > >
>> > > > Hello folks,
>> > > >
>> > > > Recently observed a behaviour while re-syncing standby from hawq
>> > command
>> > > > line.
>> > > >
>> > > > Here are the reproduction steps -
>> > > >
>> > > > 1 - Open a client connection to hawq using psql
>> > > > 2 - From a different terminal run command - hawq init standby -n -v
>> -M
>> > > fast
>> > > > 3 - Standby resync fails with error
>> > > >
>> > > >
>> > > > 20171113:03:49:21:158354 hawq_stop:hdp3:gpadmin-[WARNING]:-There
>> are
>> > > > other connections to this instance, shutdown mode smart aborted
>> > > >
>> > > > 20171113:03:49:21:158354 hawq_stop:hdp3:gpadmin-[WARNING]:-Either
>> > > > remove connections, or use 'hawq stop master -M fast' or 'hawq stop
>> > > > master -M immediate'
>> > > >
>> > > > 20171113:03:49:21:158354 hawq_stop:hdp3:gpadmin-[WARNING]:-See hawq
>> > > > stop --help for all options
>> > > >
>> > > > 20171113:03:49:21:158354 hawq_stop:hdp3:gpadmin-[ERROR]:-Active
>> > > > connections. Aborting shutdown...
>> > > >
>> > > > 20171113:03:49:21:158143 hawq_init:hdp3:gpadmin-[ERROR]:-Stop hawq
>> > > > cluster failed, exit
>> > > >
>> > > > 4 - My understanding is when -M (stop mode) is passed it should
>> > terminate
>> > > > existing client connections. Also, it seems like a good practice to
>> > > > terminate client connections before standby master resync.
>> > > >
>> > > > Is this an expected behavior in hawq ? If not, I can open a JIRA and
>> > work
>> > > > on a pull request to fix this.
>> > > >
>> > > > Looking forward to your thoughts on this.
>> > > >
>> > > > --
>> > > > Regards,
>> > > > Shubham Sharma
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> > Regards,
>> > Shubham Sharma
>> > Staff Customer Engineer
>> > Pivotal Global Support Services
>> > ssharma@pivotal.io
>> > Direct Tel: +1(510)-304-8201
>> > Office Hours: Mon-Fri 9:00 am to 5:00 pm PDT
>> > Out of Office Hours Contact +1 877-477-2269
>> >
>>
>
>
>
> --
> Regards,
> Shubham Sharma
> Staff Customer Engineer
> Pivotal Global Support Services
> ssharma@pivotal.io
> Direct Tel: +1(510)-304-8201 <(510)%20304-8201>
> Office Hours: Mon-Fri 9:00 am to 5:00 pm PDT
> Out of Office Hours Contact +1 877-477-2269 <(877)%20477-2269>
>
--
Regards,
Shubham Sharma
Staff Customer Engineer
Pivotal Global Support Services
ssharma@pivotal.io
Direct Tel: +1(510)-304-8201
Office Hours: Mon-Fri 9:00 am to 5:00 pm PDT
Out of Office Hours Contact +1 877-477-2269
Re: Hawq standby sync fails if there are existing connections to master
Posted by Shubham Sharma <ss...@pivotal.io>.
Thanks, I think I found out where the issue is. Will create a JIRA and
submit PR.
On Mon, Nov 13, 2017 at 5:54 PM, Lei Chang <ch...@gmail.com> wrote:
> oh, yes, make sense, I overlooked the -M option here.
>
> Here we should respect the input "-M" option.
>
> Cheers
> Lei
>
>
>
> On Tue, Nov 14, 2017 at 8:46 AM, Shubham Sharma <ss...@pivotal.io>
> wrote:
>
> > Thanks for the response Lei.
> >
> > I understand this completely and agree with the behavior that we should
> not
> > brutally terminate connections.
> >
> > However, the case here is, am deliberately trying to use stop mode using
> > command `hawq init standby -n -v -M fast`. At this point as a user I
> > understand that connections will be terminated without warning. When
> > passing option `-M fast` hawq should not interrupt me.
> >
> > Basically, hawq command line is not respecting `-M fast`. Whereas hawq
> init
> > standby --help documents this option. Let me know if this makes sense.
> >
> >
> > hawq init standby --help
> >
> > Usage: HAWQ management scripts options
> >
> >
> > Options:
> >
> > -h, --help show this help message and exit
> >
> > -a, --prompt Execute automatically
> >
> > -M STOP_MODE, --mode=STOP_MODE
> >
> > HAWQ stop mode: smart/fast/immediate
> >
> >
> > On Mon, Nov 13, 2017 at 4:32 PM, Lei Chang <ch...@gmail.com>
> wrote:
> >
> > > Hi Shubham,
> > >
> > > The behavior is intentional. If there are connections when HAWQ init
> > > standby, it is better to warn the client instead of cutting the
> > connections
> > > brutally.
> > >
> > > Cheers
> > > Lei
> > >
> > >
> > >
> > > On Mon, Nov 13, 2017 at 11:58 AM, Shubham Sharma <ss...@pivotal.io>
> > > wrote:
> > >
> > > > Hello folks,
> > > >
> > > > Recently observed a behaviour while re-syncing standby from hawq
> > command
> > > > line.
> > > >
> > > > Here are the reproduction steps -
> > > >
> > > > 1 - Open a client connection to hawq using psql
> > > > 2 - From a different terminal run command - hawq init standby -n -v
> -M
> > > fast
> > > > 3 - Standby resync fails with error
> > > >
> > > >
> > > > 20171113:03:49:21:158354 hawq_stop:hdp3:gpadmin-[WARNING]:-There are
> > > > other connections to this instance, shutdown mode smart aborted
> > > >
> > > > 20171113:03:49:21:158354 hawq_stop:hdp3:gpadmin-[WARNING]:-Either
> > > > remove connections, or use 'hawq stop master -M fast' or 'hawq stop
> > > > master -M immediate'
> > > >
> > > > 20171113:03:49:21:158354 hawq_stop:hdp3:gpadmin-[WARNING]:-See hawq
> > > > stop --help for all options
> > > >
> > > > 20171113:03:49:21:158354 hawq_stop:hdp3:gpadmin-[ERROR]:-Active
> > > > connections. Aborting shutdown...
> > > >
> > > > 20171113:03:49:21:158143 hawq_init:hdp3:gpadmin-[ERROR]:-Stop hawq
> > > > cluster failed, exit
> > > >
> > > > 4 - My understanding is when -M (stop mode) is passed it should
> > terminate
> > > > existing client connections. Also, it seems like a good practice to
> > > > terminate client connections before standby master resync.
> > > >
> > > > Is this an expected behavior in hawq ? If not, I can open a JIRA and
> > work
> > > > on a pull request to fix this.
> > > >
> > > > Looking forward to your thoughts on this.
> > > >
> > > > --
> > > > Regards,
> > > > Shubham Sharma
> > > >
> > >
> >
> >
> >
> > --
> > Regards,
> > Shubham Sharma
> > Staff Customer Engineer
> > Pivotal Global Support Services
> > ssharma@pivotal.io
> > Direct Tel: +1(510)-304-8201
> > Office Hours: Mon-Fri 9:00 am to 5:00 pm PDT
> > Out of Office Hours Contact +1 877-477-2269
> >
>
--
Regards,
Shubham Sharma
Staff Customer Engineer
Pivotal Global Support Services
ssharma@pivotal.io
Direct Tel: +1(510)-304-8201
Office Hours: Mon-Fri 9:00 am to 5:00 pm PDT
Out of Office Hours Contact +1 877-477-2269
Re: Hawq standby sync fails if there are existing connections to master
Posted by Lei Chang <ch...@gmail.com>.
oh, yes, make sense, I overlooked the -M option here.
Here we should respect the input "-M" option.
Cheers
Lei
On Tue, Nov 14, 2017 at 8:46 AM, Shubham Sharma <ss...@pivotal.io> wrote:
> Thanks for the response Lei.
>
> I understand this completely and agree with the behavior that we should not
> brutally terminate connections.
>
> However, the case here is, am deliberately trying to use stop mode using
> command `hawq init standby -n -v -M fast`. At this point as a user I
> understand that connections will be terminated without warning. When
> passing option `-M fast` hawq should not interrupt me.
>
> Basically, hawq command line is not respecting `-M fast`. Whereas hawq init
> standby --help documents this option. Let me know if this makes sense.
>
>
> hawq init standby --help
>
> Usage: HAWQ management scripts options
>
>
> Options:
>
> -h, --help show this help message and exit
>
> -a, --prompt Execute automatically
>
> -M STOP_MODE, --mode=STOP_MODE
>
> HAWQ stop mode: smart/fast/immediate
>
>
> On Mon, Nov 13, 2017 at 4:32 PM, Lei Chang <ch...@gmail.com> wrote:
>
> > Hi Shubham,
> >
> > The behavior is intentional. If there are connections when HAWQ init
> > standby, it is better to warn the client instead of cutting the
> connections
> > brutally.
> >
> > Cheers
> > Lei
> >
> >
> >
> > On Mon, Nov 13, 2017 at 11:58 AM, Shubham Sharma <ss...@pivotal.io>
> > wrote:
> >
> > > Hello folks,
> > >
> > > Recently observed a behaviour while re-syncing standby from hawq
> command
> > > line.
> > >
> > > Here are the reproduction steps -
> > >
> > > 1 - Open a client connection to hawq using psql
> > > 2 - From a different terminal run command - hawq init standby -n -v -M
> > fast
> > > 3 - Standby resync fails with error
> > >
> > >
> > > 20171113:03:49:21:158354 hawq_stop:hdp3:gpadmin-[WARNING]:-There are
> > > other connections to this instance, shutdown mode smart aborted
> > >
> > > 20171113:03:49:21:158354 hawq_stop:hdp3:gpadmin-[WARNING]:-Either
> > > remove connections, or use 'hawq stop master -M fast' or 'hawq stop
> > > master -M immediate'
> > >
> > > 20171113:03:49:21:158354 hawq_stop:hdp3:gpadmin-[WARNING]:-See hawq
> > > stop --help for all options
> > >
> > > 20171113:03:49:21:158354 hawq_stop:hdp3:gpadmin-[ERROR]:-Active
> > > connections. Aborting shutdown...
> > >
> > > 20171113:03:49:21:158143 hawq_init:hdp3:gpadmin-[ERROR]:-Stop hawq
> > > cluster failed, exit
> > >
> > > 4 - My understanding is when -M (stop mode) is passed it should
> terminate
> > > existing client connections. Also, it seems like a good practice to
> > > terminate client connections before standby master resync.
> > >
> > > Is this an expected behavior in hawq ? If not, I can open a JIRA and
> work
> > > on a pull request to fix this.
> > >
> > > Looking forward to your thoughts on this.
> > >
> > > --
> > > Regards,
> > > Shubham Sharma
> > >
> >
>
>
>
> --
> Regards,
> Shubham Sharma
> Staff Customer Engineer
> Pivotal Global Support Services
> ssharma@pivotal.io
> Direct Tel: +1(510)-304-8201
> Office Hours: Mon-Fri 9:00 am to 5:00 pm PDT
> Out of Office Hours Contact +1 877-477-2269
>
Re: Hawq standby sync fails if there are existing connections to master
Posted by Shubham Sharma <ss...@pivotal.io>.
Thanks for the response Lei.
I understand this completely and agree with the behavior that we should not
brutally terminate connections.
However, the case here is, am deliberately trying to use stop mode using
command `hawq init standby -n -v -M fast`. At this point as a user I
understand that connections will be terminated without warning. When
passing option `-M fast` hawq should not interrupt me.
Basically, hawq command line is not respecting `-M fast`. Whereas hawq init
standby --help documents this option. Let me know if this makes sense.
hawq init standby --help
Usage: HAWQ management scripts options
Options:
-h, --help show this help message and exit
-a, --prompt Execute automatically
-M STOP_MODE, --mode=STOP_MODE
HAWQ stop mode: smart/fast/immediate
On Mon, Nov 13, 2017 at 4:32 PM, Lei Chang <ch...@gmail.com> wrote:
> Hi Shubham,
>
> The behavior is intentional. If there are connections when HAWQ init
> standby, it is better to warn the client instead of cutting the connections
> brutally.
>
> Cheers
> Lei
>
>
>
> On Mon, Nov 13, 2017 at 11:58 AM, Shubham Sharma <ss...@pivotal.io>
> wrote:
>
> > Hello folks,
> >
> > Recently observed a behaviour while re-syncing standby from hawq command
> > line.
> >
> > Here are the reproduction steps -
> >
> > 1 - Open a client connection to hawq using psql
> > 2 - From a different terminal run command - hawq init standby -n -v -M
> fast
> > 3 - Standby resync fails with error
> >
> >
> > 20171113:03:49:21:158354 hawq_stop:hdp3:gpadmin-[WARNING]:-There are
> > other connections to this instance, shutdown mode smart aborted
> >
> > 20171113:03:49:21:158354 hawq_stop:hdp3:gpadmin-[WARNING]:-Either
> > remove connections, or use 'hawq stop master -M fast' or 'hawq stop
> > master -M immediate'
> >
> > 20171113:03:49:21:158354 hawq_stop:hdp3:gpadmin-[WARNING]:-See hawq
> > stop --help for all options
> >
> > 20171113:03:49:21:158354 hawq_stop:hdp3:gpadmin-[ERROR]:-Active
> > connections. Aborting shutdown...
> >
> > 20171113:03:49:21:158143 hawq_init:hdp3:gpadmin-[ERROR]:-Stop hawq
> > cluster failed, exit
> >
> > 4 - My understanding is when -M (stop mode) is passed it should terminate
> > existing client connections. Also, it seems like a good practice to
> > terminate client connections before standby master resync.
> >
> > Is this an expected behavior in hawq ? If not, I can open a JIRA and work
> > on a pull request to fix this.
> >
> > Looking forward to your thoughts on this.
> >
> > --
> > Regards,
> > Shubham Sharma
> >
>
--
Regards,
Shubham Sharma
Staff Customer Engineer
Pivotal Global Support Services
ssharma@pivotal.io
Direct Tel: +1(510)-304-8201
Office Hours: Mon-Fri 9:00 am to 5:00 pm PDT
Out of Office Hours Contact +1 877-477-2269
Re: Hawq standby sync fails if there are existing connections to master
Posted by Lei Chang <ch...@gmail.com>.
Hi Shubham,
The behavior is intentional. If there are connections when HAWQ init
standby, it is better to warn the client instead of cutting the connections
brutally.
Cheers
Lei
On Mon, Nov 13, 2017 at 11:58 AM, Shubham Sharma <ss...@pivotal.io> wrote:
> Hello folks,
>
> Recently observed a behaviour while re-syncing standby from hawq command
> line.
>
> Here are the reproduction steps -
>
> 1 - Open a client connection to hawq using psql
> 2 - From a different terminal run command - hawq init standby -n -v -M fast
> 3 - Standby resync fails with error
>
>
> 20171113:03:49:21:158354 hawq_stop:hdp3:gpadmin-[WARNING]:-There are
> other connections to this instance, shutdown mode smart aborted
>
> 20171113:03:49:21:158354 hawq_stop:hdp3:gpadmin-[WARNING]:-Either
> remove connections, or use 'hawq stop master -M fast' or 'hawq stop
> master -M immediate'
>
> 20171113:03:49:21:158354 hawq_stop:hdp3:gpadmin-[WARNING]:-See hawq
> stop --help for all options
>
> 20171113:03:49:21:158354 hawq_stop:hdp3:gpadmin-[ERROR]:-Active
> connections. Aborting shutdown...
>
> 20171113:03:49:21:158143 hawq_init:hdp3:gpadmin-[ERROR]:-Stop hawq
> cluster failed, exit
>
> 4 - My understanding is when -M (stop mode) is passed it should terminate
> existing client connections. Also, it seems like a good practice to
> terminate client connections before standby master resync.
>
> Is this an expected behavior in hawq ? If not, I can open a JIRA and work
> on a pull request to fix this.
>
> Looking forward to your thoughts on this.
>
> --
> Regards,
> Shubham Sharma
>