ActiveMQ broker stops functioning

[mzruya <mz...@gmail.com>]

i'm encountering critical problems with activemq in our dev enviroment,
activemq brokers seem to hang and not respond to other brokers or jms
consumers/producers.

my example setup is an activemq network of embedded brokers (2 to be
precise), 
one process with an embedded broker produces work, 
a worker process with an embedded broker as well that does the work.

2 queues, a running-queue and a stopped-queue, each msg has a counter which
is decremented every time work is done on it until 0 and then the msg is
passed to the stopped-queue, where it is consumed by the work producer.

the logic is implemented with basic spring integration.

activemq is configured with plenty of memory to spare, and no persistency,
jmx is exposed to allow monitoring of the queues,
what happens is that occasionally msg's get stuck inflight, meaning they do
not get an ack, sometimes the inflight is between brokers, and sometimes
between an embedded broker and it's internal jvm consumer.

the worker broker seems to hang, the jmx statistics seem to be frozen,
although they do refresh, the broker responds to force restarts of the
network connector, and sometime the restart frees msg's until it gets stuck
again.
if i leave the worker on after it's stuck and relaunch the producer, both
brokers don't seem to communicate, the produced msg's on one side, don't
reach the other, same goes for all advisory msg's related stats (such as
producer/consumer counts)

i tried changing every bit of configuration, with the broker, and with
spring itself, but i have a feeling the problem doesn't have anything to do
with spring (because of inner broker communication issues), the problem is
easily reproducible on a powerful machine where the msg flow is fast, adding
persistency or doing it on a normal desktop slows everything down and makes
it more difficult to reproduce,

i tried all recent activemq versions up to the latest 5.6 snapshot.

i attached the relevent junit tests, one needs to launch the agentTest, and
then the producerTest,
my recommendation is leaving agentTest on and relaunching producerTest over
and over again, producerTest should be finished when all it's messages are
done (do not launch 2 producer processes at the same time otherwise they
will never be finished).

relevant maven test cases :
http://db.tt/diVWPNnJ

AMQ-3739 :
https://issues.apache.org/jira/browse/AMQ-3739


--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-broker-stops-functioning-tp4423910p4423910.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.