Messages lost by connecting composite queues

[danielstraub <ds...@ctrlaltdel.de>]

We have a network of brokers like this: 
3 broker called integ, postit, qsu are connected via static connectors and
statically included destinations.
One composite queue distributes messages based on messages selectors to the
network and one local queue.
The local queue is also an composite queue and distributes to an queue and
topic.
Here some parts of the configuration:

    <destinationInterceptors>
      <virtualDestinationInterceptor>
        <virtualDestinations>
            <compositeQueue name="elektraStatusInput">
            <forwardTo>
              <filteredDestination selector="env='qsu'"   
queue="qsu-elektraStatusInput"/>
              <filteredDestination selector="env='postit'"
queue="postit-elektraStatusInput"/>
              <filteredDestination selector="env='integ'" 
queue="integ-elektraStatusInput"/>
            </forwardTo>
          </compositeQueue>
          <compositeQueue name="qsu-elektraStatusInput">
            <forwardTo>
              <queue physicalName="elektraStatus"/>
              <topic physicalName="gpmEvent"/>
            </forwardTo>
          </compositeQueue>
        </virtualDestinations>
      </virtualDestinationInterceptor>
    </destinationInterceptors>

    <networkConnectors>
      <networkConnector uri="static:failover(...)" networkTTL="3"
staticBridge="true">
        <staticallyIncludedDestinations>
         
          <queue physicalName="postit-elektraStatusInput"/> 
          <queue physicalName="integ-elektraStatusInput"/>
        </staticallyIncludedDestinations>
      </networkConnector>
    </networkConnectors>


Until now, we use a separate logging plugin to show only some relevant
details of the message flow:
Here is an example of a properly delivered message:
28.05.2014 00:20:14,294 send queue://elektraStatusInput
ID:teafntweb1vt-1239-1401132435490-1513:1:1:1:1
producer=ID:teafntweb1vt-1239-1401132435490-1513:1
  content: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
...
  properties: land=DE env=qsu
28.05.2014 00:20:14,313 acknowledge topic://gpmEvent
ID:teafntweb1vt-1239-1401132435490-1513:1:1:1:1
consumer=ID:gpm-cep2vq-43842-1399547229250-1:1
28.05.2014 00:20:14,317 acknowledge topic://gpmEvent
ID:teafntweb1vt-1239-1401132435490-1513:1:1:1:1
consumer=ID:gpm-cep1vq-60273-1399547243158-1:1
28.05.2014 00:20:14,347 acknowledge queue://elektraStatus
ID:teafntweb1vt-1239-1401132435490-1513:1:1:1:1
consumer=ID:bat2davq-56443-1399978172862-1:

But for some of the messages there isn't an acknowledge, that means there is
only one "send" entry.
And no client consumed the missed messages (we analyzed/compared client and
broker logs as well).

We use a database for persistence, the missed messages are not in the
activemq_msgs table or in a DLQ.
The load on the broker is not as high, we have only about 600 messages per
day (it's in a QA environment).
Sometimes about 2-5% of  messages are missing (only the local processed,
with env='qsu').

How can we find out what happened to the missing messages? 
How can we here achieve 100% reliability ?
Unfortunately, we can't reproduce that with a unit test / load driver in a
local environment 



--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-lost-by-connecting-composite-queues-tp4681990.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.