Zero consumers on some brokers after reboot

[FrankS <do...@gmail.com>]

Hello,

I am working on a project that uses a network of ActiveMQ brokers.  We use a
fixed set of network connectors.  The test team has been performing outage
tests by shutting down services, killing services, and rebooting servers.

After the outage test, they give the system a moment to recover, then they
check the consumer counts for the queues and topics.  They found that
sometimes, the consumer counts on a few queues are inconsistent between
brokers.  The client has connected to the broker without error (a local
connection), and that broker shows a consumer.  But the broker with a
producer connection has a consumer count of 0.  In this situation, the
producer is still trying to send messages, but they never get to the
consumer.

Does anyone have advice on how to debug this?  We've turned on debug, but
its really noisy and we are not sure if there are specific messages or
loggers that we should be looking for.

We have seen some errors on other brokers when a rebooted system comes back
and tries to connect again (this was when the broker wasn't shut down
nicely).  They other brokers seem to think they still have a connection to
that broker (no TCP SYN sent from rebooted broker I assume?).  I'm
wondering, could it be that there is some sort of race condition when the
original connection is cleaned up while a new one is created, and perhaps
the new advertisement for consumer interest is cleaned up with it?

A few things we have noticed that seem to contribute to the problem:
* The AMQ service was not setup correctly to stop nicely on reboot, so it
was likely just killed.  When this was corrected, we had difficulty
reproducing the problem.
* It seems limited to queues the consumer creates manually using an
ActiveMQConnectionFactory.  The camel queues don't seem affected (sorry, not
too familiar with camel yet).


The ActiveMQConnectionFactory is created like this:



Then a connection pool is created with it ("rawFactory" is the factory
created above):




This is a broker config from one of the servers.  They are all the same,
except for the tweak to the networkConnectors list.




Any hints or tips would be greatly appreciated!

Thank you,
Frank



--
View this message in context: http://activemq.2283324.n4.nabble.com/Zero-consumers-on-some-brokers-after-reboot-tp4713319.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Zero consumers on some brokers after reboot

[Tim Bain <tb...@alumni.duke.edu>]

To unsubscribe, email users-unsubscribe@activemq.apache.org, not the list
itself.
On Jun 26, 2016 7:44 AM, "vas u" <ma...@gmail.com> wrote:

Unsubscribe
On Jun 24, 2016 10:16 AM, "FrankS" <do...@gmail.com> wrote:

> Hello,
>
> I am working on a project that uses a network of ActiveMQ brokers.  We use
> a
> fixed set of network connectors.  The test team has been performing outage
> tests by shutting down services, killing services, and rebooting servers.
>
> After the outage test, they give the system a moment to recover, then they
> check the consumer counts for the queues and topics.  They found that
> sometimes, the consumer counts on a few queues are inconsistent between
> brokers.  The client has connected to the broker without error (a local
> connection), and that broker shows a consumer.  But the broker with a
> producer connection has a consumer count of 0.  In this situation, the
> producer is still trying to send messages, but they never get to the
> consumer.
>
> Does anyone have advice on how to debug this?  We've turned on debug, but
> its really noisy and we are not sure if there are specific messages or
> loggers that we should be looking for.
>
> We have seen some errors on other brokers when a rebooted system comes
back
> and tries to connect again (this was when the broker wasn't shut down
> nicely).  They other brokers seem to think they still have a connection to
> that broker (no TCP SYN sent from rebooted broker I assume?).  I'm
> wondering, could it be that there is some sort of race condition when the
> original connection is cleaned up while a new one is created, and perhaps
> the new advertisement for consumer interest is cleaned up with it?
>
> A few things we have noticed that seem to contribute to the problem:
> * The AMQ service was not setup correctly to stop nicely on reboot, so it
> was likely just killed.  When this was corrected, we had difficulty
> reproducing the problem.
> * It seems limited to queues the consumer creates manually using an
> ActiveMQConnectionFactory.  The camel queues don't seem affected (sorry,
> not
> too familiar with camel yet).
>
>
> The ActiveMQConnectionFactory is created like this:
>
>
>
> Then a connection pool is created with it ("rawFactory" is the factory
> created above):
>
>
>
>
> This is a broker config from one of the servers.  They are all the same,
> except for the tweak to the networkConnectors list.
>
>
>
>
> Any hints or tips would be greatly appreciated!
>
> Thank you,
> Frank
>
>
>
> --
> View this message in context:
>
http://activemq.2283324.n4.nabble.com/Zero-consumers-on-some-brokers-after-reboot-tp4713319.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

Re: Zero consumers on some brokers after reboot

[vas u <ma...@gmail.com>]

Unsubscribe
On Jun 24, 2016 10:16 AM, "FrankS" <do...@gmail.com> wrote:

> Hello,
>
> I am working on a project that uses a network of ActiveMQ brokers.  We use
> a
> fixed set of network connectors.  The test team has been performing outage
> tests by shutting down services, killing services, and rebooting servers.
>
> After the outage test, they give the system a moment to recover, then they
> check the consumer counts for the queues and topics.  They found that
> sometimes, the consumer counts on a few queues are inconsistent between
> brokers.  The client has connected to the broker without error (a local
> connection), and that broker shows a consumer.  But the broker with a
> producer connection has a consumer count of 0.  In this situation, the
> producer is still trying to send messages, but they never get to the
> consumer.
>
> Does anyone have advice on how to debug this?  We've turned on debug, but
> its really noisy and we are not sure if there are specific messages or
> loggers that we should be looking for.
>
> We have seen some errors on other brokers when a rebooted system comes back
> and tries to connect again (this was when the broker wasn't shut down
> nicely).  They other brokers seem to think they still have a connection to
> that broker (no TCP SYN sent from rebooted broker I assume?).  I'm
> wondering, could it be that there is some sort of race condition when the
> original connection is cleaned up while a new one is created, and perhaps
> the new advertisement for consumer interest is cleaned up with it?
>
> A few things we have noticed that seem to contribute to the problem:
> * The AMQ service was not setup correctly to stop nicely on reboot, so it
> was likely just killed.  When this was corrected, we had difficulty
> reproducing the problem.
> * It seems limited to queues the consumer creates manually using an
> ActiveMQConnectionFactory.  The camel queues don't seem affected (sorry,
> not
> too familiar with camel yet).
>
>
> The ActiveMQConnectionFactory is created like this:
>
>
>
> Then a connection pool is created with it ("rawFactory" is the factory
> created above):
>
>
>
>
> This is a broker config from one of the servers.  They are all the same,
> except for the tweak to the networkConnectors list.
>
>
>
>
> Any hints or tips would be greatly appreciated!
>
> Thank you,
> Frank
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Zero-consumers-on-some-brokers-after-reboot-tp4713319.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

Re: Zero consumers on some brokers after reboot

[FrankS <do...@gmail.com>]

Thanks Tim,

In most cases, the problem occurs on reboot and it can take 2 minutes for it
to come back.

Your thoughts are similar to mine, but I thought the options on the
networkConnector would have helped with that.

I tried enabled debug logging in a quiet system to see what I could find on
a restart.  It looks like this logger is used for consumer events:

org.apache.activemq.broker.region.AbstractRegion

And this one for discarding consumer requests.  We have a broker network, so
it discards consumer interest that came via more than 1 hop:

org.apache.activemq.network.DemandForwardingBridgeSupport


Am I on the right track for log4j loggers?  Any other suggestions?

I'm trying out just logging org.apache.activemq.network and
org.apache.activemq.broker, but filtering out broker.region.Queue because it
keeps logging "expiring messages" events.

Thanks,
Frank



--
View this message in context: http://activemq.2283324.n4.nabble.com/Zero-consumers-on-some-brokers-after-reboot-tp4713319p4713400.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Zero consumers on some brokers after reboot

[Tim Bain <tb...@alumni.duke.edu>]

What was the delay between stopping and restarting the broker?  Longer than
the 3-second inactivity check?

The inability to tell that a TCP connection has been severed until a
timeout has elapsed may be contributing to this problem, and may explain
the behavior is different when you shut them down cleanly.

Tim
On Jun 24, 2016 9:42 AM, "FrankS" <do...@gmail.com> wrote:

> I forgot to mention, we are using (and are currently limited to) version
> 5.10.2
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Zero-consumers-on-some-brokers-after-reboot-tp4713319p4713328.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

Re: Zero consumers on some brokers after reboot

[FrankS <do...@gmail.com>]

I forgot to mention, we are using (and are currently limited to) version
5.10.2



--
View this message in context: http://activemq.2283324.n4.nabble.com/Zero-consumers-on-some-brokers-after-reboot-tp4713319p4713328.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.