You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Tim Bain <tb...@alumni.duke.edu> on 2019/03/09 02:05:11 UTC
Re: DB lease lock - broker should stop-start to retry acquiring the lock
It's my understanding that restarts are handled by the Tanuki service
wrapper, when ActiveMQ is installed as a service, using the instructions at
http://activemq.apache.org/java-service-wrapper.html. Is that how you're
using ActiveMQ? Or are you running the broker directly from the executable
in the bin directory, which wouldn't use the wrapper (and so wouldn't
restart the process)?
Tim
On Thu, Feb 28, 2019, 1:57 PM Martin Lichtin <li...@yahoo.com.invalid>
wrote:
> Hi
>
> Can you help me understand this situation?
> I've ActiveMQ configured with a lease lock on a DB. Spontaneously it
> decided it can no longer keep the lock, and starts shutting down:
>
> 2019-02-27T16:47:45,491 | INFO | Starting Thread | BrokerService
> | he.activemq.broker.BrokerService 773 | 73 - activemq-osgi - 5.15.6 |
> Apache ActiveMQ 5.15.6 (broker, amq2) started
>
> 2019-02-27T16:51:05,907 | ERROR | KeepAlive Timer | LockableServiceSupport
> | mq.broker.LockableServiceSupport 143 | 73 - activemq-osgi - 5.15.6 |
> broker, no longer able to keep the exclusive lock so giving up being a
> master
> 2019-02-27T16:51:05,909 | INFO | KeepAlive Timer | BrokerService
> | he.activemq.broker.BrokerService 836 | 73 - activemq-osgi - 5.15.6 |
> Apache ActiveMQ 5.15.6 (broker, amq2) is shutting down
>
> It doesn't say why it could no longer keep the lock, but anyways, the
> bigger issue is that after stopping, it does not restart to retry acquiring
> the lock.
>
> Is there a configuration I'm missing? The code in LockableServiceSupport
> actually requests a restart, but where is that re-start done?
>
> Thanks for any insight
> - Martin
>
>
Re: Re: DB lease lock - broker should stop-start to retry acquiring
the lock
Posted by Tim Bain <tb...@alumni.duke.edu>.
Martin,
Thanks for sharing your solution (unclean though it is) to help whoever
might hit this later, and the fact that you're running within Karaf. I
don't have any experience working with Karaf so I can't suggest anything
cleaner (though you'd be more likely to get suggestions from the Karaf
community, since this is specific to the Karaf environment).
ActiveMQ's philosophy is that the process exits when the broker encounters
a fatal error, but it's the responsibility of some wrapper process (e.g.
the Tanuki service wrapper referenced from the page I sent earlier) to
restart the JVM. I'm not sure if it's possible to leverage the Tanuki
wrapper within Karaf (I assume not, since you said you had to use this
workaround), but without something like that, the behavior you saw is the
expected behavior.
Tim
On Sun, Mar 10, 2019, 7:44 AM Martin Lichtin <li...@yahoo.com.invalid>
wrote:
> I'm running the broker in a Karaf container.
>
> Had to resort to installing a shutdown hook, and when called, move the
> corresponding .cfg file away and then back into place (to trigger Felix
> fileinstall) so ActiveMQServiceFactory.updated() would get called. Very
> ugly, but the desired effect is that the broker gets restarted and will try
> to acquire the lease lock again.
>
> - Martin
>
> On 09.03.2019 03:05, Tim Bain wrote:
> > It's my understanding that restarts are handled by the Tanuki service
> > wrapper, when ActiveMQ is installed as a service, using the instructions
> at
> > http://activemq.apache.org/java-service-wrapper.html. Is that how you're
> > using ActiveMQ? Or are you running the broker directly from the
> executable
> > in the bin directory, which wouldn't use the wrapper (and so wouldn't
> > restart the process)?
> >
> > Tim
> >
> > On Thu, Feb 28, 2019, 1:57 PM Martin Lichtin <li...@yahoo.com.invalid>
> > wrote:
> >
> >> Hi
> >>
> >> Can you help me understand this situation?
> >> I've ActiveMQ configured with a lease lock on a DB. Spontaneously it
> >> decided it can no longer keep the lock, and starts shutting down:
> >>
> >> 2019-02-27T16:47:45,491 | INFO | Starting Thread | BrokerService
> >> | he.activemq.broker.BrokerService 773 | 73 - activemq-osgi - 5.15.6 |
> >> Apache ActiveMQ 5.15.6 (broker, amq2) started
> >>
> >> 2019-02-27T16:51:05,907 | ERROR | KeepAlive Timer |
> LockableServiceSupport
> >> | mq.broker.LockableServiceSupport 143 | 73 - activemq-osgi - 5.15.6 |
> >> broker, no longer able to keep the exclusive lock so giving up being a
> >> master
> >> 2019-02-27T16:51:05,909 | INFO | KeepAlive Timer | BrokerService
> >> | he.activemq.broker.BrokerService 836 | 73 - activemq-osgi - 5.15.6 |
> >> Apache ActiveMQ 5.15.6 (broker, amq2) is shutting down
> >>
> >> It doesn't say why it could no longer keep the lock, but anyways, the
> >> bigger issue is that after stopping, it does not restart to retry
> acquiring
> >> the lock.
> >>
> >> Is there a configuration I'm missing? The code in LockableServiceSupport
> >> actually requests a restart, but where is that re-start done?
> >>
> >> Thanks for any insight
> >> - Martin
> >>
> >>
>
Re: Re: DB lease lock - broker should stop-start to retry acquiring
the lock
Posted by Martin Lichtin <li...@yahoo.com.INVALID>.
I'm running the broker in a Karaf container.
Had to resort to installing a shutdown hook, and when called, move the corresponding .cfg file away and then back into place (to trigger Felix fileinstall) so ActiveMQServiceFactory.updated() would get called. Very ugly, but the desired effect is that the broker gets restarted and will try to acquire the lease lock again.
- Martin
On 09.03.2019 03:05, Tim Bain wrote:
> It's my understanding that restarts are handled by the Tanuki service
> wrapper, when ActiveMQ is installed as a service, using the instructions at
> http://activemq.apache.org/java-service-wrapper.html. Is that how you're
> using ActiveMQ? Or are you running the broker directly from the executable
> in the bin directory, which wouldn't use the wrapper (and so wouldn't
> restart the process)?
>
> Tim
>
> On Thu, Feb 28, 2019, 1:57 PM Martin Lichtin <li...@yahoo.com.invalid>
> wrote:
>
>> Hi
>>
>> Can you help me understand this situation?
>> I've ActiveMQ configured with a lease lock on a DB. Spontaneously it
>> decided it can no longer keep the lock, and starts shutting down:
>>
>> 2019-02-27T16:47:45,491 | INFO | Starting Thread | BrokerService
>> | he.activemq.broker.BrokerService 773 | 73 - activemq-osgi - 5.15.6 |
>> Apache ActiveMQ 5.15.6 (broker, amq2) started
>>
>> 2019-02-27T16:51:05,907 | ERROR | KeepAlive Timer | LockableServiceSupport
>> | mq.broker.LockableServiceSupport 143 | 73 - activemq-osgi - 5.15.6 |
>> broker, no longer able to keep the exclusive lock so giving up being a
>> master
>> 2019-02-27T16:51:05,909 | INFO | KeepAlive Timer | BrokerService
>> | he.activemq.broker.BrokerService 836 | 73 - activemq-osgi - 5.15.6 |
>> Apache ActiveMQ 5.15.6 (broker, amq2) is shutting down
>>
>> It doesn't say why it could no longer keep the lock, but anyways, the
>> bigger issue is that after stopping, it does not restart to retry acquiring
>> the lock.
>>
>> Is there a configuration I'm missing? The code in LockableServiceSupport
>> actually requests a restart, but where is that re-start done?
>>
>> Thanks for any insight
>> - Martin
>>
>>