Unexplained producer throttling when using a network of brokers

["bob.cotton@gmail.com" <bo...@gmail.com>]

ActiveMQ 5.5.0
Name : IBM Performance Harness for Java Message Service. Version : 1.2 Build
: 488
Java(TM) SE Runtime Environment (build 1.6.0_26-b03-384-10M3425)
OSX 10.6.8

Producer perfharness commandline: 
java -Dactivemq.base=$ACTIVEMQ_HOME JMSPerfHarness -pc JNDI -ii
org.apache.activemq.jndi.ActiveMQInitialContextFactory -iu $BROKER_URL -cf
ConnectionFactory -d dynamicTopics/MyStaticTopic1  -tc jms.r11.Publisher -rl
500 -pp true

Consumer perfharness commandline:
java -Dactivemq.base=$ACTIVE_HOME JMSPerfHarness -pc JNDI -ii
org.apache.activemq.jndi.ActiveMQInitialContextFactory -iu $BROKER_URL -cf
ConnectionFactory -d dynamicTopics/MyStaticTopic1  -tc jms.r11.Subscriber
-rl 500

I'm testing various broker toplogies during my evaluation of ActiveMQ.

1st scenario: Standalone ActiveMQ broker with
BROKER_URL='tcp://localhost:61616' with producers and consumers in separate
VMs.  
Everything works as expected. Test runs to completion.

2nd scenarion: Both producer and consumer contain an embedded brokers with a
duplex networkConnector to the standalone broker.
Standalone broker config  http://pastie.org/2522981 here .

Publisher BROKER_URL='vm://localhost?brokerConfig=xbean:
http://pastie.org/2522896 embeded-broker-pub.xml '
Subscriber BROKER_URL='vm://localhost?brokerConfig=xbean:
http://pastie.org/2522905 embeded-broker-pub.xml '

Test starts out with the consumer receiving messages as expected. Then the
producer stops sending. There does not seem to be any warning about the
producer being throttled. 

I get the same behavior even when setting  producerFlowControl="false"

I've turned up logging: log4j.logger.org.apache.activemq=DEBUG
but I don't see anything indicating that the producer is being throttled.

Run-time stack traces show the producer is doing this (full kill quit 
http://pastie.org/2522961 here ):


"ActiveMQ Task-6" daemon prio=5 tid=1027f2000 nid=0x115774000 in
Object.wait() [115772000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at
org.apache.activemq.broker.region.TopicSubscription.add(TopicSubscription.java:129)
        - locked <7f40d1bb0> (a java.lang.Object)
        at
org.apache.activemq.broker.region.policy.SimpleDispatchPolicy.dispatch(SimpleDispatchPolicy.java:48)
        at org.apache.activemq.broker.region.Topic.dispatch(Topic.java:634)
        at
org.apache.activemq.broker.region.Topic.doMessageSend(Topic.java:468)
        - locked <7f4be2db0> (a org.apache.activemq.broker.region.Topic)
        at org.apache.activemq.broker.region.Topic.send(Topic.java:404)
        at
org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:365)

and this:


"Publisher1" daemon prio=5 tid=10e01f800 nid=0x112204000 waiting on
condition [112203000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <7f40ea900> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at
java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
        at
org.apache.activemq.transport.FutureResponse.getResult(FutureResponse.java:40)
        at
org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:87)
        at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1284)
        at
org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1760)

Inspecting the topics in jconsole show all of them empty. This is
repeatable.

interestingly, if I detach the tty (sending stdout/stderr to /dev/null) I
don't get this behavior.

Is there something I'm doing wrong? Other logging I can enable to help me
debug this?

Thanks for reading this far.
- Bob




--
View this message in context: http://activemq.2283324.n4.nabble.com/Unexplained-producer-throttling-when-using-a-network-of-brokers-tp3808581p3808581.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.