You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Adel Boutros <Ad...@live.com> on 2017/09/06 16:21:48 UTC
[Qpid Java Broker 6.0.4] [Linux] Stop script keeps trying to kill a
dead broker
Hello,
In one of our tests, we were having a random failure. It seems we cannot stop a broker correctly.
We have a started broker and we call "bin/qpid.stop $BROKER_PID" to stop it. It seems to work from the first time but maybe not fast enough because the script keeps trying to kill the broker which is actually dead.
Is this a know issue? Is it fixed on a newer version?
Command output
===============
Waiting 1 second for 514 to exit
broker/bin/qpid.stop: line 49: kill: (514) - No such process
Waiting 1 second for 514 to exit
broker/bin/qpid.stop: line 41: kill: (514) - No such process
Waiting 1 second for 514 to exit
broker/bin/qpid.stop: line 41: kill: (514) - No such process
Waiting 1 second for 514 to exit
Stopped trying to kill process: 514
Attempted to stop 2 times
Re: [Qpid Java Broker 6.0.4] [Linux] Stop script keeps trying to kill
a dead broker
Posted by Rob Godfrey <ro...@gmail.com>.
On 18 September 2017 at 22:53, VERMEULEN Olivier <
Olivier.VERMEULEN@murex.com> wrote:
> Hello,
>
> Following up on Adel's email.
> I took a look at the initiateShutdown endpoint you mentioned.
> I tested it and it seems to work but I don't see it in any documentation,
> not even in the broker apidocs.
> Is there a reason for that? Is this feature officially supported?
>
Since the code was introduced back in 2015, it has been marked with a meta
data description "Initiates an orderly shutdown of the Broker." This text
*should* show in the API docs for the broker operations. Since there are
no parameters to pass, there is no further documentation. The feature is
and remains supported.
> And one more question, why is the broker's qpid.stop script not using the
> same graceful mechanism?
>
The qpid.stop script hasn't seen any changes (other than a blanket update
to all shell scripts which use bash) since 2009. Since the REST API
*should* use authentication, and may even be disabled (or on a non standard
port), I don't think it could be used by the standard stop script anyway.
-- Rob
>
> Thanks,
> Olivier
>
> -----Original Message-----
> From: Adel Boutros [mailto:Adelboutros@live.com]
> Sent: jeudi 14 septembre 2017 18:34
> To: users@qpid.apache.org
> Subject: Re: [Qpid Java Broker 6.0.4] [Linux] Stop script keeps trying to
> kill a dead broker
>
> Thank you Rudyy,
>
>
> Unfortunately, I don't have any extra information to share. May be the
> stop script should be updated to check output of kill command to confirm
> the process is still here.
>
>
> Regards,
>
> Adel
>
> ________________________________
> From: Oleksandr Rudyy <or...@gmail.com>
> Sent: Friday, September 8, 2017 1:46:54 PM
> To: users@qpid.apache.org
> Subject: Re: [Qpid Java Broker 6.0.4] [Linux] Stop script keeps trying to
> kill a dead broker
>
> Hi Adel,
> Thanks for reporting the issue. The qpid.stop script might need more love,
> though, after sending SIGTERM or SIGKILL event to the broker process, it
> waits for 1 second and than verifies that process with given PID is still
> reported by ps (ps -e | grep $1 | wc -l). If process is not reported, no
> further attempts to send termination signal is made. It seems that in your
> case the Broker process was present in process table. I could be that it
> became defunctional. You mentioned that it happens randomly. Do you know
> what what happens with the broker and broker jvm? Is broker shutdown
> gracefully? If not, it could be an indication of issue with broker shutdown
> or jvm exit.
>
> Additionally, I would like to point out that you can call Broker REST API
> to shutdown the broker (/api/latest/broker/initiateShutdown). As
> operation name suggests, it does not shutdown broker immediately but rather
> starts the broker shutdown process and exits. If broker restart is
> required, a restart operation can be invoked via REST API as well
> (/api/latest/broker/restart).
>
> Kind Regards,
> Alex
>
>
> On 6 September 2017 at 17:21, Adel Boutros <Ad...@live.com> wrote:
>
> > Hello,
> >
> >
> > In one of our tests, we were having a random failure. It seems we
> > cannot stop a broker correctly.
> >
> > We have a started broker and we call "bin/qpid.stop $BROKER_PID" to
> > stop it. It seems to work from the first time but maybe not fast
> > enough because the script keeps trying to kill the broker which is
> actually dead.
> >
> >
> > Is this a know issue? Is it fixed on a newer version?
> >
> >
> > Command output
> >
> > ===============
> >
> > Waiting 1 second for 514 to exit
> > broker/bin/qpid.stop: line 49: kill: (514) - No such process Waiting 1
> > second for 514 to exit
> > broker/bin/qpid.stop: line 41: kill: (514) - No such process Waiting 1
> > second for 514 to exit
> > broker/bin/qpid.stop: line 41: kill: (514) - No such process Waiting 1
> > second for 514 to exit Stopped trying to kill process: 514 Attempted
> > to stop 2 times
> >
> >
> *******************************
>
> This e-mail contains information for the intended recipient only. It may
> contain proprietary material or confidential information. If you are not
> the intended recipient you are not authorised to distribute, copy or use
> this e-mail or any attachment to it. Murex cannot guarantee that it is
> virus free and accepts no responsibility for any loss or damage arising
> from its use. If you have received this e-mail in error please notify
> immediately the sender and delete the original email received, any
> attachments and all copies from your system.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
Re: [Qpid Java Broker 6.0.4] [Linux] Stop script keeps trying to kill
a dead broker
Posted by Rob Godfrey <ro...@gmail.com>.
On 20 September 2017 at 13:41, Oleksandr Rudyy <or...@gmail.com> wrote:
> Hi Oliver,
>
> By graceful shutdown I meant that Qpid broker jvm exits after successful
> closing of all internal objects, finishing/interrupting of all in-progress
> tasks and releasing all system resources.
>
To Olivier's point though... the "graceful" shutdown is not necessarily all
that graceful from the point of view of a messaging client application.
Closing a connection where the broker has received (but not yet)
acknowledged a message, and then persisting that message in a queue is not
really particularly friendly. Really the major advantage with a "graceful"
shutdown is that sub-components like the store will be "cleanly" closed,
meaning that there will likely be less delay on startup. Also I guess that
messaging clients will potentially get a message indicating why their
connections have been closed rather than a simple closure at the TCP layer.
- Rob
> As I mentioned in my previous email, the stop script tries to shutdown
> broker gracefully by sending SIGTERM signal. However, the script only waits
> for upto 2 seconds for broker to finish graceful shutdown. After that it
> sends SIGKILL signal to terminate JVM immediately by OS. The latter can
> terminate the JVM abruptly without waiting for broker shutdown to complete
> gracefully.
>
> To work around the issue you can use directly "kill -15 <pid>" in your
> tests instead of qpid.stop and wait until process finishes. Alternatively,
> you can increase SLEEP_DELAY in the stop script to the value sufficient for
> the broker to shutdown gracefully.
>
>
> Kind Regards,
> Alex
>
>
>
> On 19 September 2017 at 23:04, VERMEULEN Olivier <
> Olivier.VERMEULEN@murex.com> wrote:
>
> > Hello,
> >
> > Thanks for the information.
> > There's one thing that's still not clear though, what do you really mean
> > by graceful shutdown?
> > When you first mentioned this what came to my mind is that the broker
> will
> > finish the tasks at hand before actually shutting down.
> > But further testing shows that it's not the case. I did a simple test
> > using an infinite loop to send messages to a queue in my broker (with
> > auto-ack) and then shutting down the broker with the ReST endpoint
> (didn't
> > try the qpid.stop since I was on Windows but I guess it will be the
> same).
> > As expected the test ends with an exception raised by the producer.send
> but
> > even though we get an exception the last message sent is sometimes
> > persisted in the database...
> >
> > Olivier
> >
> > -----Original Message-----
> > From: Adel Boutros [mailto:adelboutros@live.com]
> > Sent: mardi 19 septembre 2017 14:39
> > To: users@qpid.apache.org
> > Subject: Re: [Qpid Java Broker 6.0.4] [Linux] Stop script keeps trying to
> > kill a dead broker
> >
> > Sorry Alex :(
> > ________________________________
> > From: Robbie Gemmell <ro...@gmail.com>
> > Sent: Tuesday, September 19, 2017 1:48:48 PM
> > To: users@qpid.apache.org
> > Subject: Re: [Qpid Java Broker 6.0.4] [Linux] Stop script keeps trying to
> > kill a dead broker
> >
> > Alex or Oleksandr, rather than Rudyy :)
> >
> > On 19 September 2017 at 11:22, Adel Boutros <Ad...@live.com>
> wrote:
> > > Hello Rudyy,
> > >
> > >
> > > +1 for the enhancements 😊
> > >
> > >
> > > Regards,
> > >
> > > Adel
> > >
> > > ________________________________
> > > From: Oleksandr Rudyy <or...@gmail.com>
> > > Sent: Tuesday, September 19, 2017 11:17:15 AM
> > > To: users@qpid.apache.org
> > > Subject: Re: [Qpid Java Broker 6.0.4] [Linux] Stop script keeps trying
> > > to kill a dead broker
> > >
> > > Hi Olivier,
> > >
> > > Rob have already provided comprehensive answers on your questions.
> > >
> > > One thing I would like to add is that both REST API and stop script
> > > should shutdown broker gracefully. The script uses signals: it sends
> > > SIGTERM to initiate the graceful shutdown. Only if process is still
> > > alive after 2 seconds after sending SIGTERM, the SIGKILL signal is
> > > sent. I raised
> > > QPID-7910 [1] to improve stop script. The possible improvements are
> > > listed in the JIRA description.
> > >
> > > Kind Regards,
> > > Alex
> > >
> > >
> > > [1] https://issues.apache.org/jira/browse/QPID-7910
> > >
> > >
> > > On 18 September 2017 at 21:53, VERMEULEN Olivier <
> > > Olivier.VERMEULEN@murex.com> wrote:
> > >
> > >> Hello,
> > >>
> > >> Following up on Adel's email.
> > >> I took a look at the initiateShutdown endpoint you mentioned.
> > >> I tested it and it seems to work but I don't see it in any
> > >> documentation, not even in the broker apidocs.
> > >> Is there a reason for that? Is this feature officially supported?
> > >> And one more question, why is the broker's qpid.stop script not using
> > >> the same graceful mechanism?
> > >>
> > >> Thanks,
> > >> Olivier
> > >>
> > >> -----Original Message-----
> > >> From: Adel Boutros [mailto:Adelboutros@live.com]
> > >> Sent: jeudi 14 septembre 2017 18:34
> > >> To: users@qpid.apache.org
> > >> Subject: Re: [Qpid Java Broker 6.0.4] [Linux] Stop script keeps
> > >> trying to kill a dead broker
> > >>
> > >> Thank you Rudyy,
> > >>
> > >>
> > >> Unfortunately, I don't have any extra information to share. May be
> > >> the stop script should be updated to check output of kill command to
> > >> confirm the process is still here.
> > >>
> > >>
> > >> Regards,
> > >>
> > >> Adel
> > >>
> > >> ________________________________
> > >> From: Oleksandr Rudyy <or...@gmail.com>
> > >> Sent: Friday, September 8, 2017 1:46:54 PM
> > >> To: users@qpid.apache.org
> > >> Subject: Re: [Qpid Java Broker 6.0.4] [Linux] Stop script keeps
> > >> trying to kill a dead broker
> > >>
> > >> Hi Adel,
> > >> Thanks for reporting the issue. The qpid.stop script might need more
> > >> love, though, after sending SIGTERM or SIGKILL event to the broker
> > >> process, it waits for 1 second and than verifies that process with
> > >> given PID is still reported by ps (ps -e | grep $1 | wc -l). If
> > >> process is not reported, no further attempts to send termination
> > >> signal is made. It seems that in your case the Broker process was
> > >> present in process table. I could be that it became defunctional. You
> > >> mentioned that it happens randomly. Do you know what what happens
> > >> with the broker and broker jvm? Is broker shutdown gracefully? If
> > >> not, it could be an indication of issue with broker shutdown or jvm
> > exit.
> > >>
> > >> Additionally, I would like to point out that you can call Broker REST
> > >> API to shutdown the broker (/api/latest/broker/initiateShutdown). As
> > >> operation name suggests, it does not shutdown broker immediately but
> > >> rather starts the broker shutdown process and exits. If broker
> > >> restart is required, a restart operation can be invoked via REST API
> > >> as well (/api/latest/broker/restart).
> > >>
> > >> Kind Regards,
> > >> Alex
> > >>
> > >>
> > >> On 6 September 2017 at 17:21, Adel Boutros <Ad...@live.com>
> > wrote:
> > >>
> > >> > Hello,
> > >> >
> > >> >
> > >> > In one of our tests, we were having a random failure. It seems we
> > >> > cannot stop a broker correctly.
> > >> >
> > >> > We have a started broker and we call "bin/qpid.stop $BROKER_PID" to
> > >> > stop it. It seems to work from the first time but maybe not fast
> > >> > enough because the script keeps trying to kill the broker which is
> > >> actually dead.
> > >> >
> > >> >
> > >> > Is this a know issue? Is it fixed on a newer version?
> > >> >
> > >> >
> > >> > Command output
> > >> >
> > >> > ===============
> > >> >
> > >> > Waiting 1 second for 514 to exit
> > >> > broker/bin/qpid.stop: line 49: kill: (514) - No such process
> > >> > Waiting 1 second for 514 to exit
> > >> > broker/bin/qpid.stop: line 41: kill: (514) - No such process
> > >> > Waiting 1 second for 514 to exit
> > >> > broker/bin/qpid.stop: line 41: kill: (514) - No such process
> > >> > Waiting 1 second for 514 to exit Stopped trying to kill process:
> > >> > 514 Attempted to stop 2 times
> > >> >
> > >> >
> > >> *******************************
> > >>
> > >> This e-mail contains information for the intended recipient only. It
> > >> may contain proprietary material or confidential information. If you
> > >> are not the intended recipient you are not authorised to distribute,
> > >> copy or use this e-mail or any attachment to it. Murex cannot
> > >> guarantee that it is virus free and accepts no responsibility for any
> > >> loss or damage arising from its use. If you have received this e-mail
> > >> in error please notify immediately the sender and delete the original
> > >> email received, any attachments and all copies from your system.
> > >>
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org For
> > >> additional commands, e-mail: users-help@qpid.apache.org
> > >>
> > >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org For additional
> > commands, e-mail: users-help@qpid.apache.org
> >
> > *******************************
> >
> > This e-mail contains information for the intended recipient only. It may
> > contain proprietary material or confidential information. If you are not
> > the intended recipient you are not authorised to distribute, copy or use
> > this e-mail or any attachment to it. Murex cannot guarantee that it is
> > virus free and accepts no responsibility for any loss or damage arising
> > from its use. If you have received this e-mail in error please notify
> > immediately the sender and delete the original email received, any
> > attachments and all copies from your system.
> >
>
Re: [Qpid Java Broker 6.0.4] [Linux] Stop script keeps trying to kill
a dead broker
Posted by Oleksandr Rudyy <or...@gmail.com>.
Hi Oliver,
By graceful shutdown I meant that Qpid broker jvm exits after successful
closing of all internal objects, finishing/interrupting of all in-progress
tasks and releasing all system resources.
As I mentioned in my previous email, the stop script tries to shutdown
broker gracefully by sending SIGTERM signal. However, the script only waits
for upto 2 seconds for broker to finish graceful shutdown. After that it
sends SIGKILL signal to terminate JVM immediately by OS. The latter can
terminate the JVM abruptly without waiting for broker shutdown to complete
gracefully.
To work around the issue you can use directly "kill -15 <pid>" in your
tests instead of qpid.stop and wait until process finishes. Alternatively,
you can increase SLEEP_DELAY in the stop script to the value sufficient for
the broker to shutdown gracefully.
Kind Regards,
Alex
On 19 September 2017 at 23:04, VERMEULEN Olivier <
Olivier.VERMEULEN@murex.com> wrote:
> Hello,
>
> Thanks for the information.
> There's one thing that's still not clear though, what do you really mean
> by graceful shutdown?
> When you first mentioned this what came to my mind is that the broker will
> finish the tasks at hand before actually shutting down.
> But further testing shows that it's not the case. I did a simple test
> using an infinite loop to send messages to a queue in my broker (with
> auto-ack) and then shutting down the broker with the ReST endpoint (didn't
> try the qpid.stop since I was on Windows but I guess it will be the same).
> As expected the test ends with an exception raised by the producer.send but
> even though we get an exception the last message sent is sometimes
> persisted in the database...
>
> Olivier
>
> -----Original Message-----
> From: Adel Boutros [mailto:adelboutros@live.com]
> Sent: mardi 19 septembre 2017 14:39
> To: users@qpid.apache.org
> Subject: Re: [Qpid Java Broker 6.0.4] [Linux] Stop script keeps trying to
> kill a dead broker
>
> Sorry Alex :(
> ________________________________
> From: Robbie Gemmell <ro...@gmail.com>
> Sent: Tuesday, September 19, 2017 1:48:48 PM
> To: users@qpid.apache.org
> Subject: Re: [Qpid Java Broker 6.0.4] [Linux] Stop script keeps trying to
> kill a dead broker
>
> Alex or Oleksandr, rather than Rudyy :)
>
> On 19 September 2017 at 11:22, Adel Boutros <Ad...@live.com> wrote:
> > Hello Rudyy,
> >
> >
> > +1 for the enhancements 😊
> >
> >
> > Regards,
> >
> > Adel
> >
> > ________________________________
> > From: Oleksandr Rudyy <or...@gmail.com>
> > Sent: Tuesday, September 19, 2017 11:17:15 AM
> > To: users@qpid.apache.org
> > Subject: Re: [Qpid Java Broker 6.0.4] [Linux] Stop script keeps trying
> > to kill a dead broker
> >
> > Hi Olivier,
> >
> > Rob have already provided comprehensive answers on your questions.
> >
> > One thing I would like to add is that both REST API and stop script
> > should shutdown broker gracefully. The script uses signals: it sends
> > SIGTERM to initiate the graceful shutdown. Only if process is still
> > alive after 2 seconds after sending SIGTERM, the SIGKILL signal is
> > sent. I raised
> > QPID-7910 [1] to improve stop script. The possible improvements are
> > listed in the JIRA description.
> >
> > Kind Regards,
> > Alex
> >
> >
> > [1] https://issues.apache.org/jira/browse/QPID-7910
> >
> >
> > On 18 September 2017 at 21:53, VERMEULEN Olivier <
> > Olivier.VERMEULEN@murex.com> wrote:
> >
> >> Hello,
> >>
> >> Following up on Adel's email.
> >> I took a look at the initiateShutdown endpoint you mentioned.
> >> I tested it and it seems to work but I don't see it in any
> >> documentation, not even in the broker apidocs.
> >> Is there a reason for that? Is this feature officially supported?
> >> And one more question, why is the broker's qpid.stop script not using
> >> the same graceful mechanism?
> >>
> >> Thanks,
> >> Olivier
> >>
> >> -----Original Message-----
> >> From: Adel Boutros [mailto:Adelboutros@live.com]
> >> Sent: jeudi 14 septembre 2017 18:34
> >> To: users@qpid.apache.org
> >> Subject: Re: [Qpid Java Broker 6.0.4] [Linux] Stop script keeps
> >> trying to kill a dead broker
> >>
> >> Thank you Rudyy,
> >>
> >>
> >> Unfortunately, I don't have any extra information to share. May be
> >> the stop script should be updated to check output of kill command to
> >> confirm the process is still here.
> >>
> >>
> >> Regards,
> >>
> >> Adel
> >>
> >> ________________________________
> >> From: Oleksandr Rudyy <or...@gmail.com>
> >> Sent: Friday, September 8, 2017 1:46:54 PM
> >> To: users@qpid.apache.org
> >> Subject: Re: [Qpid Java Broker 6.0.4] [Linux] Stop script keeps
> >> trying to kill a dead broker
> >>
> >> Hi Adel,
> >> Thanks for reporting the issue. The qpid.stop script might need more
> >> love, though, after sending SIGTERM or SIGKILL event to the broker
> >> process, it waits for 1 second and than verifies that process with
> >> given PID is still reported by ps (ps -e | grep $1 | wc -l). If
> >> process is not reported, no further attempts to send termination
> >> signal is made. It seems that in your case the Broker process was
> >> present in process table. I could be that it became defunctional. You
> >> mentioned that it happens randomly. Do you know what what happens
> >> with the broker and broker jvm? Is broker shutdown gracefully? If
> >> not, it could be an indication of issue with broker shutdown or jvm
> exit.
> >>
> >> Additionally, I would like to point out that you can call Broker REST
> >> API to shutdown the broker (/api/latest/broker/initiateShutdown). As
> >> operation name suggests, it does not shutdown broker immediately but
> >> rather starts the broker shutdown process and exits. If broker
> >> restart is required, a restart operation can be invoked via REST API
> >> as well (/api/latest/broker/restart).
> >>
> >> Kind Regards,
> >> Alex
> >>
> >>
> >> On 6 September 2017 at 17:21, Adel Boutros <Ad...@live.com>
> wrote:
> >>
> >> > Hello,
> >> >
> >> >
> >> > In one of our tests, we were having a random failure. It seems we
> >> > cannot stop a broker correctly.
> >> >
> >> > We have a started broker and we call "bin/qpid.stop $BROKER_PID" to
> >> > stop it. It seems to work from the first time but maybe not fast
> >> > enough because the script keeps trying to kill the broker which is
> >> actually dead.
> >> >
> >> >
> >> > Is this a know issue? Is it fixed on a newer version?
> >> >
> >> >
> >> > Command output
> >> >
> >> > ===============
> >> >
> >> > Waiting 1 second for 514 to exit
> >> > broker/bin/qpid.stop: line 49: kill: (514) - No such process
> >> > Waiting 1 second for 514 to exit
> >> > broker/bin/qpid.stop: line 41: kill: (514) - No such process
> >> > Waiting 1 second for 514 to exit
> >> > broker/bin/qpid.stop: line 41: kill: (514) - No such process
> >> > Waiting 1 second for 514 to exit Stopped trying to kill process:
> >> > 514 Attempted to stop 2 times
> >> >
> >> >
> >> *******************************
> >>
> >> This e-mail contains information for the intended recipient only. It
> >> may contain proprietary material or confidential information. If you
> >> are not the intended recipient you are not authorised to distribute,
> >> copy or use this e-mail or any attachment to it. Murex cannot
> >> guarantee that it is virus free and accepts no responsibility for any
> >> loss or damage arising from its use. If you have received this e-mail
> >> in error please notify immediately the sender and delete the original
> >> email received, any attachments and all copies from your system.
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org For
> >> additional commands, e-mail: users-help@qpid.apache.org
> >>
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org For additional
> commands, e-mail: users-help@qpid.apache.org
>
> *******************************
>
> This e-mail contains information for the intended recipient only. It may
> contain proprietary material or confidential information. If you are not
> the intended recipient you are not authorised to distribute, copy or use
> this e-mail or any attachment to it. Murex cannot guarantee that it is
> virus free and accepts no responsibility for any loss or damage arising
> from its use. If you have received this e-mail in error please notify
> immediately the sender and delete the original email received, any
> attachments and all copies from your system.
>
RE: [Qpid Java Broker 6.0.4] [Linux] Stop script keeps trying to kill
a dead broker
Posted by VERMEULEN Olivier <Ol...@murex.com>.
Hello,
Thanks for the information.
There's one thing that's still not clear though, what do you really mean by graceful shutdown?
When you first mentioned this what came to my mind is that the broker will finish the tasks at hand before actually shutting down.
But further testing shows that it's not the case. I did a simple test using an infinite loop to send messages to a queue in my broker (with auto-ack) and then shutting down the broker with the ReST endpoint (didn't try the qpid.stop since I was on Windows but I guess it will be the same). As expected the test ends with an exception raised by the producer.send but even though we get an exception the last message sent is sometimes persisted in the database...
Olivier
-----Original Message-----
From: Adel Boutros [mailto:adelboutros@live.com]
Sent: mardi 19 septembre 2017 14:39
To: users@qpid.apache.org
Subject: Re: [Qpid Java Broker 6.0.4] [Linux] Stop script keeps trying to kill a dead broker
Sorry Alex :(
________________________________
From: Robbie Gemmell <ro...@gmail.com>
Sent: Tuesday, September 19, 2017 1:48:48 PM
To: users@qpid.apache.org
Subject: Re: [Qpid Java Broker 6.0.4] [Linux] Stop script keeps trying to kill a dead broker
Alex or Oleksandr, rather than Rudyy :)
On 19 September 2017 at 11:22, Adel Boutros <Ad...@live.com> wrote:
> Hello Rudyy,
>
>
> +1 for the enhancements 😊
>
>
> Regards,
>
> Adel
>
> ________________________________
> From: Oleksandr Rudyy <or...@gmail.com>
> Sent: Tuesday, September 19, 2017 11:17:15 AM
> To: users@qpid.apache.org
> Subject: Re: [Qpid Java Broker 6.0.4] [Linux] Stop script keeps trying
> to kill a dead broker
>
> Hi Olivier,
>
> Rob have already provided comprehensive answers on your questions.
>
> One thing I would like to add is that both REST API and stop script
> should shutdown broker gracefully. The script uses signals: it sends
> SIGTERM to initiate the graceful shutdown. Only if process is still
> alive after 2 seconds after sending SIGTERM, the SIGKILL signal is
> sent. I raised
> QPID-7910 [1] to improve stop script. The possible improvements are
> listed in the JIRA description.
>
> Kind Regards,
> Alex
>
>
> [1] https://issues.apache.org/jira/browse/QPID-7910
>
>
> On 18 September 2017 at 21:53, VERMEULEN Olivier <
> Olivier.VERMEULEN@murex.com> wrote:
>
>> Hello,
>>
>> Following up on Adel's email.
>> I took a look at the initiateShutdown endpoint you mentioned.
>> I tested it and it seems to work but I don't see it in any
>> documentation, not even in the broker apidocs.
>> Is there a reason for that? Is this feature officially supported?
>> And one more question, why is the broker's qpid.stop script not using
>> the same graceful mechanism?
>>
>> Thanks,
>> Olivier
>>
>> -----Original Message-----
>> From: Adel Boutros [mailto:Adelboutros@live.com]
>> Sent: jeudi 14 septembre 2017 18:34
>> To: users@qpid.apache.org
>> Subject: Re: [Qpid Java Broker 6.0.4] [Linux] Stop script keeps
>> trying to kill a dead broker
>>
>> Thank you Rudyy,
>>
>>
>> Unfortunately, I don't have any extra information to share. May be
>> the stop script should be updated to check output of kill command to
>> confirm the process is still here.
>>
>>
>> Regards,
>>
>> Adel
>>
>> ________________________________
>> From: Oleksandr Rudyy <or...@gmail.com>
>> Sent: Friday, September 8, 2017 1:46:54 PM
>> To: users@qpid.apache.org
>> Subject: Re: [Qpid Java Broker 6.0.4] [Linux] Stop script keeps
>> trying to kill a dead broker
>>
>> Hi Adel,
>> Thanks for reporting the issue. The qpid.stop script might need more
>> love, though, after sending SIGTERM or SIGKILL event to the broker
>> process, it waits for 1 second and than verifies that process with
>> given PID is still reported by ps (ps -e | grep $1 | wc -l). If
>> process is not reported, no further attempts to send termination
>> signal is made. It seems that in your case the Broker process was
>> present in process table. I could be that it became defunctional. You
>> mentioned that it happens randomly. Do you know what what happens
>> with the broker and broker jvm? Is broker shutdown gracefully? If
>> not, it could be an indication of issue with broker shutdown or jvm exit.
>>
>> Additionally, I would like to point out that you can call Broker REST
>> API to shutdown the broker (/api/latest/broker/initiateShutdown). As
>> operation name suggests, it does not shutdown broker immediately but
>> rather starts the broker shutdown process and exits. If broker
>> restart is required, a restart operation can be invoked via REST API
>> as well (/api/latest/broker/restart).
>>
>> Kind Regards,
>> Alex
>>
>>
>> On 6 September 2017 at 17:21, Adel Boutros <Ad...@live.com> wrote:
>>
>> > Hello,
>> >
>> >
>> > In one of our tests, we were having a random failure. It seems we
>> > cannot stop a broker correctly.
>> >
>> > We have a started broker and we call "bin/qpid.stop $BROKER_PID" to
>> > stop it. It seems to work from the first time but maybe not fast
>> > enough because the script keeps trying to kill the broker which is
>> actually dead.
>> >
>> >
>> > Is this a know issue? Is it fixed on a newer version?
>> >
>> >
>> > Command output
>> >
>> > ===============
>> >
>> > Waiting 1 second for 514 to exit
>> > broker/bin/qpid.stop: line 49: kill: (514) - No such process
>> > Waiting 1 second for 514 to exit
>> > broker/bin/qpid.stop: line 41: kill: (514) - No such process
>> > Waiting 1 second for 514 to exit
>> > broker/bin/qpid.stop: line 41: kill: (514) - No such process
>> > Waiting 1 second for 514 to exit Stopped trying to kill process:
>> > 514 Attempted to stop 2 times
>> >
>> >
>> *******************************
>>
>> This e-mail contains information for the intended recipient only. It
>> may contain proprietary material or confidential information. If you
>> are not the intended recipient you are not authorised to distribute,
>> copy or use this e-mail or any attachment to it. Murex cannot
>> guarantee that it is virus free and accepts no responsibility for any
>> loss or damage arising from its use. If you have received this e-mail
>> in error please notify immediately the sender and delete the original
>> email received, any attachments and all copies from your system.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org For
>> additional commands, e-mail: users-help@qpid.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org For additional commands, e-mail: users-help@qpid.apache.org
*******************************
This e-mail contains information for the intended recipient only. It may contain proprietary material or confidential information. If you are not the intended recipient you are not authorised to distribute, copy or use this e-mail or any attachment to it. Murex cannot guarantee that it is virus free and accepts no responsibility for any loss or damage arising from its use. If you have received this e-mail in error please notify immediately the sender and delete the original email received, any attachments and all copies from your system.
Re: [Qpid Java Broker 6.0.4] [Linux] Stop script keeps trying to kill
a dead broker
Posted by Adel Boutros <ad...@live.com>.
Sorry Alex :(
________________________________
From: Robbie Gemmell <ro...@gmail.com>
Sent: Tuesday, September 19, 2017 1:48:48 PM
To: users@qpid.apache.org
Subject: Re: [Qpid Java Broker 6.0.4] [Linux] Stop script keeps trying to kill a dead broker
Alex or Oleksandr, rather than Rudyy :)
On 19 September 2017 at 11:22, Adel Boutros <Ad...@live.com> wrote:
> Hello Rudyy,
>
>
> +1 for the enhancements 😊
>
>
> Regards,
>
> Adel
>
> ________________________________
> From: Oleksandr Rudyy <or...@gmail.com>
> Sent: Tuesday, September 19, 2017 11:17:15 AM
> To: users@qpid.apache.org
> Subject: Re: [Qpid Java Broker 6.0.4] [Linux] Stop script keeps trying to kill a dead broker
>
> Hi Olivier,
>
> Rob have already provided comprehensive answers on your questions.
>
> One thing I would like to add is that both REST API and stop script should
> shutdown broker gracefully. The script uses signals: it sends SIGTERM to
> initiate the graceful shutdown. Only if process is still alive after 2
> seconds after sending SIGTERM, the SIGKILL signal is sent. I raised
> QPID-7910 [1] to improve stop script. The possible improvements are listed
> in the JIRA description.
>
> Kind Regards,
> Alex
>
>
> [1] https://issues.apache.org/jira/browse/QPID-7910
>
>
> On 18 September 2017 at 21:53, VERMEULEN Olivier <
> Olivier.VERMEULEN@murex.com> wrote:
>
>> Hello,
>>
>> Following up on Adel's email.
>> I took a look at the initiateShutdown endpoint you mentioned.
>> I tested it and it seems to work but I don't see it in any documentation,
>> not even in the broker apidocs.
>> Is there a reason for that? Is this feature officially supported?
>> And one more question, why is the broker's qpid.stop script not using the
>> same graceful mechanism?
>>
>> Thanks,
>> Olivier
>>
>> -----Original Message-----
>> From: Adel Boutros [mailto:Adelboutros@live.com]
>> Sent: jeudi 14 septembre 2017 18:34
>> To: users@qpid.apache.org
>> Subject: Re: [Qpid Java Broker 6.0.4] [Linux] Stop script keeps trying to
>> kill a dead broker
>>
>> Thank you Rudyy,
>>
>>
>> Unfortunately, I don't have any extra information to share. May be the
>> stop script should be updated to check output of kill command to confirm
>> the process is still here.
>>
>>
>> Regards,
>>
>> Adel
>>
>> ________________________________
>> From: Oleksandr Rudyy <or...@gmail.com>
>> Sent: Friday, September 8, 2017 1:46:54 PM
>> To: users@qpid.apache.org
>> Subject: Re: [Qpid Java Broker 6.0.4] [Linux] Stop script keeps trying to
>> kill a dead broker
>>
>> Hi Adel,
>> Thanks for reporting the issue. The qpid.stop script might need more love,
>> though, after sending SIGTERM or SIGKILL event to the broker process, it
>> waits for 1 second and than verifies that process with given PID is still
>> reported by ps (ps -e | grep $1 | wc -l). If process is not reported, no
>> further attempts to send termination signal is made. It seems that in your
>> case the Broker process was present in process table. I could be that it
>> became defunctional. You mentioned that it happens randomly. Do you know
>> what what happens with the broker and broker jvm? Is broker shutdown
>> gracefully? If not, it could be an indication of issue with broker shutdown
>> or jvm exit.
>>
>> Additionally, I would like to point out that you can call Broker REST API
>> to shutdown the broker (/api/latest/broker/initiateShutdown). As
>> operation name suggests, it does not shutdown broker immediately but rather
>> starts the broker shutdown process and exits. If broker restart is
>> required, a restart operation can be invoked via REST API as well
>> (/api/latest/broker/restart).
>>
>> Kind Regards,
>> Alex
>>
>>
>> On 6 September 2017 at 17:21, Adel Boutros <Ad...@live.com> wrote:
>>
>> > Hello,
>> >
>> >
>> > In one of our tests, we were having a random failure. It seems we
>> > cannot stop a broker correctly.
>> >
>> > We have a started broker and we call "bin/qpid.stop $BROKER_PID" to
>> > stop it. It seems to work from the first time but maybe not fast
>> > enough because the script keeps trying to kill the broker which is
>> actually dead.
>> >
>> >
>> > Is this a know issue? Is it fixed on a newer version?
>> >
>> >
>> > Command output
>> >
>> > ===============
>> >
>> > Waiting 1 second for 514 to exit
>> > broker/bin/qpid.stop: line 49: kill: (514) - No such process Waiting 1
>> > second for 514 to exit
>> > broker/bin/qpid.stop: line 41: kill: (514) - No such process Waiting 1
>> > second for 514 to exit
>> > broker/bin/qpid.stop: line 41: kill: (514) - No such process Waiting 1
>> > second for 514 to exit Stopped trying to kill process: 514 Attempted
>> > to stop 2 times
>> >
>> >
>> *******************************
>>
>> This e-mail contains information for the intended recipient only. It may
>> contain proprietary material or confidential information. If you are not
>> the intended recipient you are not authorised to distribute, copy or use
>> this e-mail or any attachment to it. Murex cannot guarantee that it is
>> virus free and accepts no responsibility for any loss or damage arising
>> from its use. If you have received this e-mail in error please notify
>> immediately the sender and delete the original email received, any
>> attachments and all copies from your system.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>> For additional commands, e-mail: users-help@qpid.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: [Qpid Java Broker 6.0.4] [Linux] Stop script keeps trying to kill
a dead broker
Posted by Robbie Gemmell <ro...@gmail.com>.
Alex or Oleksandr, rather than Rudyy :)
On 19 September 2017 at 11:22, Adel Boutros <Ad...@live.com> wrote:
> Hello Rudyy,
>
>
> +1 for the enhancements 😊
>
>
> Regards,
>
> Adel
>
> ________________________________
> From: Oleksandr Rudyy <or...@gmail.com>
> Sent: Tuesday, September 19, 2017 11:17:15 AM
> To: users@qpid.apache.org
> Subject: Re: [Qpid Java Broker 6.0.4] [Linux] Stop script keeps trying to kill a dead broker
>
> Hi Olivier,
>
> Rob have already provided comprehensive answers on your questions.
>
> One thing I would like to add is that both REST API and stop script should
> shutdown broker gracefully. The script uses signals: it sends SIGTERM to
> initiate the graceful shutdown. Only if process is still alive after 2
> seconds after sending SIGTERM, the SIGKILL signal is sent. I raised
> QPID-7910 [1] to improve stop script. The possible improvements are listed
> in the JIRA description.
>
> Kind Regards,
> Alex
>
>
> [1] https://issues.apache.org/jira/browse/QPID-7910
>
>
> On 18 September 2017 at 21:53, VERMEULEN Olivier <
> Olivier.VERMEULEN@murex.com> wrote:
>
>> Hello,
>>
>> Following up on Adel's email.
>> I took a look at the initiateShutdown endpoint you mentioned.
>> I tested it and it seems to work but I don't see it in any documentation,
>> not even in the broker apidocs.
>> Is there a reason for that? Is this feature officially supported?
>> And one more question, why is the broker's qpid.stop script not using the
>> same graceful mechanism?
>>
>> Thanks,
>> Olivier
>>
>> -----Original Message-----
>> From: Adel Boutros [mailto:Adelboutros@live.com]
>> Sent: jeudi 14 septembre 2017 18:34
>> To: users@qpid.apache.org
>> Subject: Re: [Qpid Java Broker 6.0.4] [Linux] Stop script keeps trying to
>> kill a dead broker
>>
>> Thank you Rudyy,
>>
>>
>> Unfortunately, I don't have any extra information to share. May be the
>> stop script should be updated to check output of kill command to confirm
>> the process is still here.
>>
>>
>> Regards,
>>
>> Adel
>>
>> ________________________________
>> From: Oleksandr Rudyy <or...@gmail.com>
>> Sent: Friday, September 8, 2017 1:46:54 PM
>> To: users@qpid.apache.org
>> Subject: Re: [Qpid Java Broker 6.0.4] [Linux] Stop script keeps trying to
>> kill a dead broker
>>
>> Hi Adel,
>> Thanks for reporting the issue. The qpid.stop script might need more love,
>> though, after sending SIGTERM or SIGKILL event to the broker process, it
>> waits for 1 second and than verifies that process with given PID is still
>> reported by ps (ps -e | grep $1 | wc -l). If process is not reported, no
>> further attempts to send termination signal is made. It seems that in your
>> case the Broker process was present in process table. I could be that it
>> became defunctional. You mentioned that it happens randomly. Do you know
>> what what happens with the broker and broker jvm? Is broker shutdown
>> gracefully? If not, it could be an indication of issue with broker shutdown
>> or jvm exit.
>>
>> Additionally, I would like to point out that you can call Broker REST API
>> to shutdown the broker (/api/latest/broker/initiateShutdown). As
>> operation name suggests, it does not shutdown broker immediately but rather
>> starts the broker shutdown process and exits. If broker restart is
>> required, a restart operation can be invoked via REST API as well
>> (/api/latest/broker/restart).
>>
>> Kind Regards,
>> Alex
>>
>>
>> On 6 September 2017 at 17:21, Adel Boutros <Ad...@live.com> wrote:
>>
>> > Hello,
>> >
>> >
>> > In one of our tests, we were having a random failure. It seems we
>> > cannot stop a broker correctly.
>> >
>> > We have a started broker and we call "bin/qpid.stop $BROKER_PID" to
>> > stop it. It seems to work from the first time but maybe not fast
>> > enough because the script keeps trying to kill the broker which is
>> actually dead.
>> >
>> >
>> > Is this a know issue? Is it fixed on a newer version?
>> >
>> >
>> > Command output
>> >
>> > ===============
>> >
>> > Waiting 1 second for 514 to exit
>> > broker/bin/qpid.stop: line 49: kill: (514) - No such process Waiting 1
>> > second for 514 to exit
>> > broker/bin/qpid.stop: line 41: kill: (514) - No such process Waiting 1
>> > second for 514 to exit
>> > broker/bin/qpid.stop: line 41: kill: (514) - No such process Waiting 1
>> > second for 514 to exit Stopped trying to kill process: 514 Attempted
>> > to stop 2 times
>> >
>> >
>> *******************************
>>
>> This e-mail contains information for the intended recipient only. It may
>> contain proprietary material or confidential information. If you are not
>> the intended recipient you are not authorised to distribute, copy or use
>> this e-mail or any attachment to it. Murex cannot guarantee that it is
>> virus free and accepts no responsibility for any loss or damage arising
>> from its use. If you have received this e-mail in error please notify
>> immediately the sender and delete the original email received, any
>> attachments and all copies from your system.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>> For additional commands, e-mail: users-help@qpid.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: [Qpid Java Broker 6.0.4] [Linux] Stop script keeps trying to kill
a dead broker
Posted by Adel Boutros <Ad...@live.com>.
Hello Rudyy,
+1 for the enhancements 😊
Regards,
Adel
________________________________
From: Oleksandr Rudyy <or...@gmail.com>
Sent: Tuesday, September 19, 2017 11:17:15 AM
To: users@qpid.apache.org
Subject: Re: [Qpid Java Broker 6.0.4] [Linux] Stop script keeps trying to kill a dead broker
Hi Olivier,
Rob have already provided comprehensive answers on your questions.
One thing I would like to add is that both REST API and stop script should
shutdown broker gracefully. The script uses signals: it sends SIGTERM to
initiate the graceful shutdown. Only if process is still alive after 2
seconds after sending SIGTERM, the SIGKILL signal is sent. I raised
QPID-7910 [1] to improve stop script. The possible improvements are listed
in the JIRA description.
Kind Regards,
Alex
[1] https://issues.apache.org/jira/browse/QPID-7910
On 18 September 2017 at 21:53, VERMEULEN Olivier <
Olivier.VERMEULEN@murex.com> wrote:
> Hello,
>
> Following up on Adel's email.
> I took a look at the initiateShutdown endpoint you mentioned.
> I tested it and it seems to work but I don't see it in any documentation,
> not even in the broker apidocs.
> Is there a reason for that? Is this feature officially supported?
> And one more question, why is the broker's qpid.stop script not using the
> same graceful mechanism?
>
> Thanks,
> Olivier
>
> -----Original Message-----
> From: Adel Boutros [mailto:Adelboutros@live.com]
> Sent: jeudi 14 septembre 2017 18:34
> To: users@qpid.apache.org
> Subject: Re: [Qpid Java Broker 6.0.4] [Linux] Stop script keeps trying to
> kill a dead broker
>
> Thank you Rudyy,
>
>
> Unfortunately, I don't have any extra information to share. May be the
> stop script should be updated to check output of kill command to confirm
> the process is still here.
>
>
> Regards,
>
> Adel
>
> ________________________________
> From: Oleksandr Rudyy <or...@gmail.com>
> Sent: Friday, September 8, 2017 1:46:54 PM
> To: users@qpid.apache.org
> Subject: Re: [Qpid Java Broker 6.0.4] [Linux] Stop script keeps trying to
> kill a dead broker
>
> Hi Adel,
> Thanks for reporting the issue. The qpid.stop script might need more love,
> though, after sending SIGTERM or SIGKILL event to the broker process, it
> waits for 1 second and than verifies that process with given PID is still
> reported by ps (ps -e | grep $1 | wc -l). If process is not reported, no
> further attempts to send termination signal is made. It seems that in your
> case the Broker process was present in process table. I could be that it
> became defunctional. You mentioned that it happens randomly. Do you know
> what what happens with the broker and broker jvm? Is broker shutdown
> gracefully? If not, it could be an indication of issue with broker shutdown
> or jvm exit.
>
> Additionally, I would like to point out that you can call Broker REST API
> to shutdown the broker (/api/latest/broker/initiateShutdown). As
> operation name suggests, it does not shutdown broker immediately but rather
> starts the broker shutdown process and exits. If broker restart is
> required, a restart operation can be invoked via REST API as well
> (/api/latest/broker/restart).
>
> Kind Regards,
> Alex
>
>
> On 6 September 2017 at 17:21, Adel Boutros <Ad...@live.com> wrote:
>
> > Hello,
> >
> >
> > In one of our tests, we were having a random failure. It seems we
> > cannot stop a broker correctly.
> >
> > We have a started broker and we call "bin/qpid.stop $BROKER_PID" to
> > stop it. It seems to work from the first time but maybe not fast
> > enough because the script keeps trying to kill the broker which is
> actually dead.
> >
> >
> > Is this a know issue? Is it fixed on a newer version?
> >
> >
> > Command output
> >
> > ===============
> >
> > Waiting 1 second for 514 to exit
> > broker/bin/qpid.stop: line 49: kill: (514) - No such process Waiting 1
> > second for 514 to exit
> > broker/bin/qpid.stop: line 41: kill: (514) - No such process Waiting 1
> > second for 514 to exit
> > broker/bin/qpid.stop: line 41: kill: (514) - No such process Waiting 1
> > second for 514 to exit Stopped trying to kill process: 514 Attempted
> > to stop 2 times
> >
> >
> *******************************
>
> This e-mail contains information for the intended recipient only. It may
> contain proprietary material or confidential information. If you are not
> the intended recipient you are not authorised to distribute, copy or use
> this e-mail or any attachment to it. Murex cannot guarantee that it is
> virus free and accepts no responsibility for any loss or damage arising
> from its use. If you have received this e-mail in error please notify
> immediately the sender and delete the original email received, any
> attachments and all copies from your system.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
Re: [Qpid Java Broker 6.0.4] [Linux] Stop script keeps trying to kill
a dead broker
Posted by Oleksandr Rudyy <or...@gmail.com>.
Hi Olivier,
Rob have already provided comprehensive answers on your questions.
One thing I would like to add is that both REST API and stop script should
shutdown broker gracefully. The script uses signals: it sends SIGTERM to
initiate the graceful shutdown. Only if process is still alive after 2
seconds after sending SIGTERM, the SIGKILL signal is sent. I raised
QPID-7910 [1] to improve stop script. The possible improvements are listed
in the JIRA description.
Kind Regards,
Alex
[1] https://issues.apache.org/jira/browse/QPID-7910
On 18 September 2017 at 21:53, VERMEULEN Olivier <
Olivier.VERMEULEN@murex.com> wrote:
> Hello,
>
> Following up on Adel's email.
> I took a look at the initiateShutdown endpoint you mentioned.
> I tested it and it seems to work but I don't see it in any documentation,
> not even in the broker apidocs.
> Is there a reason for that? Is this feature officially supported?
> And one more question, why is the broker's qpid.stop script not using the
> same graceful mechanism?
>
> Thanks,
> Olivier
>
> -----Original Message-----
> From: Adel Boutros [mailto:Adelboutros@live.com]
> Sent: jeudi 14 septembre 2017 18:34
> To: users@qpid.apache.org
> Subject: Re: [Qpid Java Broker 6.0.4] [Linux] Stop script keeps trying to
> kill a dead broker
>
> Thank you Rudyy,
>
>
> Unfortunately, I don't have any extra information to share. May be the
> stop script should be updated to check output of kill command to confirm
> the process is still here.
>
>
> Regards,
>
> Adel
>
> ________________________________
> From: Oleksandr Rudyy <or...@gmail.com>
> Sent: Friday, September 8, 2017 1:46:54 PM
> To: users@qpid.apache.org
> Subject: Re: [Qpid Java Broker 6.0.4] [Linux] Stop script keeps trying to
> kill a dead broker
>
> Hi Adel,
> Thanks for reporting the issue. The qpid.stop script might need more love,
> though, after sending SIGTERM or SIGKILL event to the broker process, it
> waits for 1 second and than verifies that process with given PID is still
> reported by ps (ps -e | grep $1 | wc -l). If process is not reported, no
> further attempts to send termination signal is made. It seems that in your
> case the Broker process was present in process table. I could be that it
> became defunctional. You mentioned that it happens randomly. Do you know
> what what happens with the broker and broker jvm? Is broker shutdown
> gracefully? If not, it could be an indication of issue with broker shutdown
> or jvm exit.
>
> Additionally, I would like to point out that you can call Broker REST API
> to shutdown the broker (/api/latest/broker/initiateShutdown). As
> operation name suggests, it does not shutdown broker immediately but rather
> starts the broker shutdown process and exits. If broker restart is
> required, a restart operation can be invoked via REST API as well
> (/api/latest/broker/restart).
>
> Kind Regards,
> Alex
>
>
> On 6 September 2017 at 17:21, Adel Boutros <Ad...@live.com> wrote:
>
> > Hello,
> >
> >
> > In one of our tests, we were having a random failure. It seems we
> > cannot stop a broker correctly.
> >
> > We have a started broker and we call "bin/qpid.stop $BROKER_PID" to
> > stop it. It seems to work from the first time but maybe not fast
> > enough because the script keeps trying to kill the broker which is
> actually dead.
> >
> >
> > Is this a know issue? Is it fixed on a newer version?
> >
> >
> > Command output
> >
> > ===============
> >
> > Waiting 1 second for 514 to exit
> > broker/bin/qpid.stop: line 49: kill: (514) - No such process Waiting 1
> > second for 514 to exit
> > broker/bin/qpid.stop: line 41: kill: (514) - No such process Waiting 1
> > second for 514 to exit
> > broker/bin/qpid.stop: line 41: kill: (514) - No such process Waiting 1
> > second for 514 to exit Stopped trying to kill process: 514 Attempted
> > to stop 2 times
> >
> >
> *******************************
>
> This e-mail contains information for the intended recipient only. It may
> contain proprietary material or confidential information. If you are not
> the intended recipient you are not authorised to distribute, copy or use
> this e-mail or any attachment to it. Murex cannot guarantee that it is
> virus free and accepts no responsibility for any loss or damage arising
> from its use. If you have received this e-mail in error please notify
> immediately the sender and delete the original email received, any
> attachments and all copies from your system.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
RE: [Qpid Java Broker 6.0.4] [Linux] Stop script keeps trying to kill
a dead broker
Posted by VERMEULEN Olivier <Ol...@murex.com>.
Hello,
Following up on Adel's email.
I took a look at the initiateShutdown endpoint you mentioned.
I tested it and it seems to work but I don't see it in any documentation, not even in the broker apidocs.
Is there a reason for that? Is this feature officially supported?
And one more question, why is the broker's qpid.stop script not using the same graceful mechanism?
Thanks,
Olivier
-----Original Message-----
From: Adel Boutros [mailto:Adelboutros@live.com]
Sent: jeudi 14 septembre 2017 18:34
To: users@qpid.apache.org
Subject: Re: [Qpid Java Broker 6.0.4] [Linux] Stop script keeps trying to kill a dead broker
Thank you Rudyy,
Unfortunately, I don't have any extra information to share. May be the stop script should be updated to check output of kill command to confirm the process is still here.
Regards,
Adel
________________________________
From: Oleksandr Rudyy <or...@gmail.com>
Sent: Friday, September 8, 2017 1:46:54 PM
To: users@qpid.apache.org
Subject: Re: [Qpid Java Broker 6.0.4] [Linux] Stop script keeps trying to kill a dead broker
Hi Adel,
Thanks for reporting the issue. The qpid.stop script might need more love, though, after sending SIGTERM or SIGKILL event to the broker process, it waits for 1 second and than verifies that process with given PID is still reported by ps (ps -e | grep $1 | wc -l). If process is not reported, no further attempts to send termination signal is made. It seems that in your case the Broker process was present in process table. I could be that it became defunctional. You mentioned that it happens randomly. Do you know what what happens with the broker and broker jvm? Is broker shutdown gracefully? If not, it could be an indication of issue with broker shutdown or jvm exit.
Additionally, I would like to point out that you can call Broker REST API to shutdown the broker (/api/latest/broker/initiateShutdown). As operation name suggests, it does not shutdown broker immediately but rather starts the broker shutdown process and exits. If broker restart is required, a restart operation can be invoked via REST API as well (/api/latest/broker/restart).
Kind Regards,
Alex
On 6 September 2017 at 17:21, Adel Boutros <Ad...@live.com> wrote:
> Hello,
>
>
> In one of our tests, we were having a random failure. It seems we
> cannot stop a broker correctly.
>
> We have a started broker and we call "bin/qpid.stop $BROKER_PID" to
> stop it. It seems to work from the first time but maybe not fast
> enough because the script keeps trying to kill the broker which is actually dead.
>
>
> Is this a know issue? Is it fixed on a newer version?
>
>
> Command output
>
> ===============
>
> Waiting 1 second for 514 to exit
> broker/bin/qpid.stop: line 49: kill: (514) - No such process Waiting 1
> second for 514 to exit
> broker/bin/qpid.stop: line 41: kill: (514) - No such process Waiting 1
> second for 514 to exit
> broker/bin/qpid.stop: line 41: kill: (514) - No such process Waiting 1
> second for 514 to exit Stopped trying to kill process: 514 Attempted
> to stop 2 times
>
>
*******************************
This e-mail contains information for the intended recipient only. It may contain proprietary material or confidential information. If you are not the intended recipient you are not authorised to distribute, copy or use this e-mail or any attachment to it. Murex cannot guarantee that it is virus free and accepts no responsibility for any loss or damage arising from its use. If you have received this e-mail in error please notify immediately the sender and delete the original email received, any attachments and all copies from your system.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: [Qpid Java Broker 6.0.4] [Linux] Stop script keeps trying to kill
a dead broker
Posted by Adel Boutros <Ad...@live.com>.
Thank you Rudyy,
Unfortunately, I don't have any extra information to share. May be the stop script should be updated to check output of kill command to confirm the process is still here.
Regards,
Adel
________________________________
From: Oleksandr Rudyy <or...@gmail.com>
Sent: Friday, September 8, 2017 1:46:54 PM
To: users@qpid.apache.org
Subject: Re: [Qpid Java Broker 6.0.4] [Linux] Stop script keeps trying to kill a dead broker
Hi Adel,
Thanks for reporting the issue. The qpid.stop script might need more love,
though, after sending SIGTERM or SIGKILL event to the broker process, it
waits for 1 second and than verifies that process with given PID is still
reported by ps (ps -e | grep $1 | wc -l). If process is not reported, no
further attempts to send termination signal is made. It seems that in your
case the Broker process was present in process table. I could be that it
became defunctional. You mentioned that it happens randomly. Do you know
what what happens with the broker and broker jvm? Is broker shutdown
gracefully? If not, it could be an indication of issue with broker shutdown
or jvm exit.
Additionally, I would like to point out that you can call Broker REST API
to shutdown the broker (/api/latest/broker/initiateShutdown). As operation
name suggests, it does not shutdown broker immediately but rather starts
the broker shutdown process and exits. If broker restart is required, a
restart operation can be invoked via REST API as well
(/api/latest/broker/restart).
Kind Regards,
Alex
On 6 September 2017 at 17:21, Adel Boutros <Ad...@live.com> wrote:
> Hello,
>
>
> In one of our tests, we were having a random failure. It seems we cannot
> stop a broker correctly.
>
> We have a started broker and we call "bin/qpid.stop $BROKER_PID" to stop
> it. It seems to work from the first time but maybe not fast enough because
> the script keeps trying to kill the broker which is actually dead.
>
>
> Is this a know issue? Is it fixed on a newer version?
>
>
> Command output
>
> ===============
>
> Waiting 1 second for 514 to exit
> broker/bin/qpid.stop: line 49: kill: (514) - No such process
> Waiting 1 second for 514 to exit
> broker/bin/qpid.stop: line 41: kill: (514) - No such process
> Waiting 1 second for 514 to exit
> broker/bin/qpid.stop: line 41: kill: (514) - No such process
> Waiting 1 second for 514 to exit
> Stopped trying to kill process: 514
> Attempted to stop 2 times
>
>
Re: [Qpid Java Broker 6.0.4] [Linux] Stop script keeps trying to kill
a dead broker
Posted by Oleksandr Rudyy <or...@gmail.com>.
Hi Adel,
Thanks for reporting the issue. The qpid.stop script might need more love,
though, after sending SIGTERM or SIGKILL event to the broker process, it
waits for 1 second and than verifies that process with given PID is still
reported by ps (ps -e | grep $1 | wc -l). If process is not reported, no
further attempts to send termination signal is made. It seems that in your
case the Broker process was present in process table. I could be that it
became defunctional. You mentioned that it happens randomly. Do you know
what what happens with the broker and broker jvm? Is broker shutdown
gracefully? If not, it could be an indication of issue with broker shutdown
or jvm exit.
Additionally, I would like to point out that you can call Broker REST API
to shutdown the broker (/api/latest/broker/initiateShutdown). As operation
name suggests, it does not shutdown broker immediately but rather starts
the broker shutdown process and exits. If broker restart is required, a
restart operation can be invoked via REST API as well
(/api/latest/broker/restart).
Kind Regards,
Alex
On 6 September 2017 at 17:21, Adel Boutros <Ad...@live.com> wrote:
> Hello,
>
>
> In one of our tests, we were having a random failure. It seems we cannot
> stop a broker correctly.
>
> We have a started broker and we call "bin/qpid.stop $BROKER_PID" to stop
> it. It seems to work from the first time but maybe not fast enough because
> the script keeps trying to kill the broker which is actually dead.
>
>
> Is this a know issue? Is it fixed on a newer version?
>
>
> Command output
>
> ===============
>
> Waiting 1 second for 514 to exit
> broker/bin/qpid.stop: line 49: kill: (514) - No such process
> Waiting 1 second for 514 to exit
> broker/bin/qpid.stop: line 41: kill: (514) - No such process
> Waiting 1 second for 514 to exit
> broker/bin/qpid.stop: line 41: kill: (514) - No such process
> Waiting 1 second for 514 to exit
> Stopped trying to kill process: 514
> Attempted to stop 2 times
>
>