Stuck messages and broker network failing

[james <me...@gmail.com>]

Hi,

We're running ActiveMQ 5.5.0 using a network of brokers configuration with
kahadb as a persistent store.  
At some point, one of the message queues will stop delivering messages to
its consumer.  I've also noticed that there might be a connection between
this and the network of brokers failing and one of the message brokers
refusing to allow new connections.  However, it is difficult to reproduce
and can take months to happen again.

For clarity, the flow is like this:
1) broker A has messages and consumers, but will not deliver the messages to
the consumers.  The messages seem to be stuck on the connector to the other
broker B.  Broker A has the messages and consumers.
2) broker B seems to have stopped accepting new connections, at least for
the consumer which can't retrieve its messages.  Broker B can deliver
messages to Broker A and the consumer.
3) there are no log entries showing any issue with broker A, B, the network
connector, or the consumer.
4) restarting the consumer doesn't work in this scenario because it will
only reconnect to broker A, reestablishing the current situation.
5) restarting the brokers (or sometimes just broker B) will work.

Our configuration is below and identical for both brokers except that one
has the network connector and the other doesn't.

What causes the network of brokers to break?  
What causes one broker to stop accepting connections (no log error being
shown)?  
What causes the messages to be stuck on the working broker with the working
consumer with the messages apparently bound for the other broker?

I've seen suggestions of prefetch sizes and caching fixing the issues, but
looks like these were fixed before 5.5.0 release (true?).
thanks!!

<beans>
    <broker xmlns="http://activemq.apache.org/schema/core"
brokerName="localhost" dataDirectory="${activemq.base}/data"
destroyApplicationContextOnStop="true" useJmx="true">

         <destinationPolicy>
            <policyMap>
              <policyEntries>
                <policyEntry topic=">" producerFlowControl="true"
memoryLimit="1mb">
                  <pendingSubscriberPolicy>
                    <vmCursor />
                  </pendingSubscriberPolicy>
                </policyEntry>
                                 
                                <policyEntry queue="Queue1"
producerFlowControl="true" memoryLimit="10mb"/>
                                
                                <policyEntry queue=">"
producerFlowControl="true" memoryLimit="1mb"/>
              </policyEntries>
            </policyMap>
        </destinationPolicy>

    <networkConnectors>
      <networkConnector name="msg1-2-nc"
uri="static:(tcp://msg2.ourcompany.net:61616)
        duplex="true" dynamicOnly="true" networkTTL="2" />
    </networkConnectors>

        <persistenceAdapter>
            <kahaDB directory="${activemq.base}/data/kahadb"/>
        </persistenceAdapter>

          <plugins>  <statisticsBrokerPlugin/> </plugins>

        <systemUsage>
            <systemUsage>
                <memoryUsage>
                    <memoryUsage limit="500mb"/>
                </memoryUsage>
            </systemUsage>
        </systemUsage>

        <transportConnectors>
            <transportConnector name="openwire" uri="tcp://0.0.0.0:61616"
               updateClusterClients="true" rebalanceClusterClients="true"
updateClusterClientsOnRemove="true"/>
        </transportConnectors>
    </broker>
    <import resource="jetty.xml"/>
</beans>

--
View this message in context: http://activemq.2283324.n4.nabble.com/Stuck-messages-and-broker-network-failing-tp4581929p4581929.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.