Strange behaviour with failover url (3xBrokers,1xProducer,6xConsumers)

[Dhruba Bandopadhyay <bd...@gmail.com>]

Hi all,

Here is the setup and the statistics from a test run.

Tools:-
Spring2.5.2,ApacheActiveMQ4.1.1

Spring config same on both producer and consumers:
    <bean id="connectionFactory"
class="org.apache.activemq.pool.PooledConnectionFactory"
destroy-method="stop">
        <property name="connectionFactory">
            <bean class="org.apache.activemq.ActiveMQConnectionFactory">
                <property name="brokerURL"
value="failover://(tcp://broker1:61616,tcp://broker2:61616,tcp://broker3:61616)?initialReconnectDelay=100&randomize=false"
/>
            </bean>
        </property>
    </bean>

    <bean id="jmsTemplate" class="org.springframework.jms.core.JmsTemplate">
        <property name="connectionFactory" ref="connectionFactory" />
        <property name="messageConverter" ref="messageConverter" />
        <property name="receiveTimeout" value="10000" />
    </bean>

    <jms:listener-container concurrency="5-5"
message-converter="shoal.shared.MessageConverter">
        <jms:listener destination="deploy-queue" ref="consumer"
method="consume" />
    </jms:listener-container>

It may be that I am not supposed to be using PooledConnectionFactory
on the consumer side.  Am I right?  Either way it shouldn't be the
cause of the problems I've having below.  Still feel free to throw
best practice advice around.

Nodes (3xBrokers,1xProducer,6xConsumers):-
broker1
broker2
broker3
producer
consumer1
consumer2
consumer3
consumer4
consumer5
consumer6

Broker connection url on producer and 6 consumers:-
failover://(tcp://broker1:61616,tcp://broker2:61616,tcp://broker3:61616)?initialReconnectDelay=100&randomize=false

Queue:-
deploy-queue

Prior to testing:-
Producer and 6 consumers have subscribed only to broker1.
Each broker has two other brokers subscribed to it

On testing:-
I sent exactly 100,000 messages (no more no less) from producer onto
deploy-queue.  These messages were all enqueued onto broker1.

Producer outcome:-
100,000 are successfully produced onto broker1.

Consumer outcome:-
86725 messages in total are processed shared evenly across 6 consumers.

Broker outcome:-
100,000 messages are enqueued onto broker1.
100,000 messages are dequeued from broker1.
13,275 messages are enqueued onto broker3.
0 messages are dequeued from broker3.
broker2 is completely inactive.

Questions
(1) Why are 13,275 messages enqueued onto broker3 when the whole
100,000 have already been enqueued onto broker1?
(2) Why do the consumers only process 86725 nmessages when there are
still 13,275 messages remaining on broker3?
(3) What behavioural rules does activemq follow with the above url on
enqueue, dequeue and subscriptions?

I am pretty surprised with all of the above and especially because
there was no loss of connection with broker1 activating failover.
However I did notice the exception below which may or may not be
related.

2008-04-03 13:09:21,456  INFO   org.apa
org.apache.activemq.transport.InactivityIOException: Channel was
inactive for too long.
        at org.apache.activemq.transport.InactivityMonitor.readCheck(InactivityMonitor.java:101)
        at org.apache.activemq.transport.InactivityMonitor.access$000(InactivityMonitor.java:35)
        at org.apache.activemq.transport.InactivityMonitor$1.run(InactivityMonitor.java:51)
        at edu.emory.mathcs.backport.java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:431)
        at edu.emory.mathcs.backport.java.util.concurrent.FutureTask.runAndReset(FutureTask.java:198)
        at edu.emory.mathcs.backport.java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:189)
        at edu.emory.mathcs.backport.java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:213)
        at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
        at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
        at java.lang.Thread.run(Thread.java:619)


Activemq xml config on all three brokers:

<beans>
    <bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"
/>
    <broker brokerName="localhost" useJmx="true"
xmlns="http://activemq.org/config/1.0">
        <persistenceAdapter>
            <journaledJDBC journalLogFiles="5"
dataDirectory="${activemq.base}/activemq-data" />
        </persistenceAdapter>
        <transportConnectors>
            <transportConnector name="openwire"
uri="tcp://localhost:61616" discoveryUri="multicast://default" />
        </transportConnectors>
        <networkConnectors>
            <networkConnector name="default-nc" uri="multicast://default" />
        </networkConnectors>
    </broker>
</beans>

I would really appreciate any insight on these problems that I've had.
 To be honest not being able to understand such behaviour is resulting
in me not really being able to use activemq to complete a full
messaging task so I'm really hoping that the designated mailing list
will be able to help.  Maybe I am doing something completely wrong.

Many thanks.