Message loss during very high load

[Robin Kåveland Hansen <rk...@knowit.no>]

Hi,

We experienced a loss of messages under critically high load conditions on
one of our brokers. Unfortunately, the high load caused us to lose the logs
of the small time window during which we lost messages. I am trying to
understand how this happened.

Our setup is a network of two brokers. What happened was that some messages
were posted to a queue on broker A which did not have load problems. This
queue did not have any consumers on broker A, so the messages should have
been forwarded to broker B, which had very high load.

Because it is critical for us to not lose messages, we have set up all our
clients to use transacted sends and the brokers to be persistent. What I
notice while investigating this issue is that the network bridge between
the brokers has dispatchAsync = true. I also notice a comment saying that
async dispatches could cause message loss in the code for
DemandForwardingBridgeSupport.java.

I am trying to figure whether this could have been the cause of the lost
messages. Because there are no logs for this time window, it is hard for me
to tell if the messages were lost in:
- The producer
- Broker A
- The bridge between A and B
- Broker B
- The consumer

I would like to know how the the messages are dispatched over the bridge
and if we could have lost the messages here because it has dispatchAsync =
true.

We are using activemq-5.5.1 for both clients and brokers.
-- 
Kind regards,
Robin Kåveland Hansen