JMS exception during the Failover

[akhil <ak...@gmail.com>]

I am enabling the shared file storage for two independent brokers with the
EFS mount. While i am testing the shared file storage with the two
individual brokers , one of the master broker is acquiring the lock and it's
working as expected in case of publishing and consuming. The client is using
the failover url for the two individual brokers to get the connection
object. Now i have put the master broker down and with in short span
experiencing the following log in the consumer thread log. In case of
publisher thread there is no unusual behaviour in publishing the messages to
the new failover broker which became master. And i see that consumer is not
working in regards to consuming the messages from the new master broker. Is
this the expected behaviour in case of failover where consumer is going to
be offline and publisher is not even impacted. 

- Async error occurred: javax.jms.JMSException: Unmatched acknowledege:
> MessageAck {commandId = 47577, responseRequired = false, ackType = 2,
> consumerId =




--
View this message in context: http://activemq.2283324.n4.nabble.com/JMS-exception-during-the-Failover-tp4716047.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: JMS exception during the Failover

[akhil <ak...@gmail.com>]

Thank You Tim ..I am just doing kill now on the activemq process instead of
service shut down. I can try the service shutdown as per your suggestion.
When i get into this state like the consumer idle or inactivity state ..from
consumer point there is no activity though we switch broker from B to A. Yes
it is consistently reproducible.

Here is the Consumer Log where you can find the inactivity monitor.
https://gist.github.com/areddy7021/da6800b18bb4118e229e77c7c2fcc422

Here is the consumer :
https://gist.github.com/areddy7021/f3bdd4b30e656b3bcab03a0bec31da99

The consumer connection block : 
<http://activemq.2283324.n4.nabble.com/file/n4716652/efs.png> 

Thanks,
Akhil.



--
View this message in context: http://activemq.2283324.n4.nabble.com/JMS-exception-during-the-Failover-tp4716047p4716652.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: JMS exception during the Failover

[akhil <ak...@gmail.com>]

Tim , Thank you for your suggestion. But i have tried different way now to
see the result. It worked out. I have created a new maven project with only
producer and consumer and the old project contains unnecessary dependencies
which might have caused the issue. As soon as i have tried with the new
project with just activemq core dependency the failover started working. It
might be related to the dependency or manifest of the project which got
corrupted. So finally , there are no issues in regards to the failover at
this moment. So it's better always whenever we are making trials with
failover it's better we are handling the project manifest and dependencies
in a right way.

Thank you very much for your suggestion.



--
View this message in context: http://activemq.2283324.n4.nabble.com/JMS-exception-during-the-Failover-tp4716047p4717069.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: JMS exception during the Failover

[Tim Bain <tb...@alumni.duke.edu>]

If you switch your acknowledgement mode to AUTO_ACKNOWLEDGE and comment out
your call to message.acknowledge(), does the behavior change?  Client ack
mode isn't very widely used, so it's possible there's a bug in the
interaction between it and failover.

Also, in your original post, you said that threads were blocked, and
provided a screenshot to illustrate why you thought that.  That thread is
an idle thread in a thread pool, and it's blocked because there's not
enough work to keep all the threads busy, not because anything is wrong or
because ActiveMQ threads are being prevented from running.

Tim

On Sep 21, 2016 10:33 AM, "akhil" <ak...@gmail.com> wrote:

> Hi Tim ,
>
> I am sorry that one was the outdated consumer , I have changed it long time
> back and forgot to update the gist. Yes i have tried yourway but the same
> result.
>
> Thanks,
> Akhil.
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/JMS-exception-during-the-Failover-tp4716047p4716706.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

Re: JMS exception during the Failover

[akhil <ak...@gmail.com>]

Hi Tim , 

I am sorry that one was the outdated consumer , I have changed it long time
back and forgot to update the gist. Yes i have tried yourway but the same
result.

Thanks,
Akhil.



--
View this message in context: http://activemq.2283324.n4.nabble.com/JMS-exception-during-the-Failover-tp4716047p4716706.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: JMS exception during the Failover

[Tim Bain <tb...@alumni.duke.edu>]

The catch block in your consumer looks very suspicious, since it starts
additional consumers without closing the previous ones.  Would you please
comment out the line that's there, and also add a line to log the exception
you're catching?

Tim

On Sep 20, 2016 11:16 AM, "akhil" <ak...@gmail.com> wrote:

> Hi Tim ,
>
> I have tried with the graceful shutdown by not using kill any more but
> still
> it's on the same way of dealing the consumer thread block.
>
> you can find the new switch log here :
> https://gist.github.com/areddy7021/e0e3a6c89fc974ce3031e326c060acca
>
> I have switched like this way ..started with Broker A as master and B as
> slave ..-- > everything is fine
>
> Broker A service down and Broker B as new master -- > Producer is fine and
> Consumer started throwing exception and still consumed even after the
> switch
> since some buffered messages in prefetch buffer and after that inactivity
> monitor started throwing up and connection state in in blocked state in
> jconsole.
>
> Broker B service down and Broker A as new master ---> Producer no issues as
> usual. Consumer started showing some log in attempt to connect to new
> master
> but its not done with the inactivity log. The above gist log is for this
> scenario.
>
> Thanks,
> Akhil.
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/JMS-exception-during-the-Failover-tp4716047p4716660.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

Re: JMS exception during the Failover

[akhil <ak...@gmail.com>]

Hi Tim , 

I have tried with the graceful shutdown by not using kill any more but still
it's on the same way of dealing the consumer thread block.

you can find the new switch log here :
https://gist.github.com/areddy7021/e0e3a6c89fc974ce3031e326c060acca

I have switched like this way ..started with Broker A as master and B as
slave ..-- > everything is fine

Broker A service down and Broker B as new master -- > Producer is fine and
Consumer started throwing exception and still consumed even after the switch
since some buffered messages in prefetch buffer and after that inactivity
monitor started throwing up and connection state in in blocked state in
jconsole. 

Broker B service down and Broker A as new master ---> Producer no issues as
usual. Consumer started showing some log in attempt to connect to new master
but its not done with the inactivity log. The above gist log is for this
scenario.

Thanks,
Akhil.



--
View this message in context: http://activemq.2283324.n4.nabble.com/JMS-exception-during-the-Failover-tp4716047p4716660.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: JMS exception during the Failover

[Tim Bain <tb...@alumni.duke.edu>]

Am I right in understanding that you're no longer having the problem from
your first email, and the problem from yesterday is the only one?

When you stop the broker, how are you stopping it?  If you're doing kill
-9, that doesn't gracefully shut down the TCP connections, so the behavior
can be different than if you issue a stop at the command prompt.  It should
still work eventually (and since it's not working for you, then it's either
a bug in our code or a bug in yours), but be aware that the clean shutdown
case will be faster than the hard kill case, so don't expect identical
behavior between the two cases.

Can you post your consumer configuration/code so we can see if there is
something obvious that might explain what you're seeing?  Also, it sounds
like you took a thread dump when this was happening; are you able to post
it?

Also, is this consistently reproducible?  And what happens if you start A
and stop B once you're in this state?

Tim

On Sep 19, 2016 6:25 PM, "akhil" <ak...@gmail.com> wrote:

> Hi , I am having the same type of issue but with the different
> configuration.
> I have two brokers which are not in network but i am using producer and
> consumer with two different threads in my local and trying to hit 10k
> messages and consume it from second thread. I am using the shared file
> storage mount for the kaha db mount.One of the broker is acting as a master
> any any point of time and the other one is slave. I have started my test
> with Broker A as master and producer is sending out messages and consumer
> started listening to it.. to add the complexity i have turned of the Broker
> A and the lock got acquired by Broker B and it is now new master. Producer
> got reconnected to master but consumer is failing and invoked the
> inactivity
> monitor and i have observed the TIME_WAIT thread block of connection broker
> of consumer thread and it never ever reconnected to the new master ?
>
> Producer no issues in regards to the reconnection in failover scenario but
> consumer does ..any one any suggestions ..is it something real bug with a
> race condition or some thing we need to put it in the consumer code to
> avoid
> this failure ?
>
> Thanks,
> Akhil.
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/JMS-exception-during-the-Failover-tp4716047p4716642.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

Re: JMS exception during the Failover

[akhil <ak...@gmail.com>]

Hi , I am having the same type of issue but with the different configuration.
I have two brokers which are not in network but i am using producer and
consumer with two different threads in my local and trying to hit 10k
messages and consume it from second thread. I am using the shared file
storage mount for the kaha db mount.One of the broker is acting as a master
any any point of time and the other one is slave. I have started my test
with Broker A as master and producer is sending out messages and consumer
started listening to it.. to add the complexity i have turned of the Broker
A and the lock got acquired by Broker B and it is now new master. Producer
got reconnected to master but consumer is failing and invoked the inactivity
monitor and i have observed the TIME_WAIT thread block of connection broker
of consumer thread and it never ever reconnected to the new master ? 

Producer no issues in regards to the reconnection in failover scenario but
consumer does ..any one any suggestions ..is it something real bug with a
race condition or some thing we need to put it in the consumer code to avoid
this failure ?

Thanks,
Akhil.



--
View this message in context: http://activemq.2283324.n4.nabble.com/JMS-exception-during-the-Failover-tp4716047p4716642.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.