slow speed sending message

[wglass <wg...@glass-husain.com>]

Hi,

I have an web application which is experiencing slowdowns when sending a
message.  (see stack trace below).  It runs fine after a reboot.  But after
1-2 weeks of running, I am seeing 10+ seconds delay when sending the
message.  When my server is under load this can be much longer.

Any suggestions as to what I should look for?  What might be accumulating in
the running app that slows it down?  The memory is not significantly high. 
I took a heap dump, didn't see anything unusual.

Here's the stack trace.  I took one a couple of times and it's always
waiting at the same point.

"TP-Processor33" - Thread t@86170
   java.lang.Thread.State: TIMED_WAITING
  at sun.misc.Unsafe.park(Native Method)
  - parking to wait for <decc76> (a
java.util.concurrent.CountDownLatch$Sync)
  at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
  at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(Unknown
Source)
  at
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(Unknown
Source)
  at java.util.concurrent.CountDownLatch.await(Unknown Source)
  at org.apache.activemq.transport.  .oneway(WireFormatNegotiator.java:97)
  at
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40)
  - locked <143da33> (a java.lang.Object)
  at
org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81)
  at
org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:86)
  at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1262)
  at
org.apache.activemq.ActiveMQConnection.ensureConnectionInfoSent(ActiveMQConnection.java:1368)
  - locked <101f5e5> (a java.lang.Object)
  at
org.apache.activemq.ActiveMQConnection.createSession(ActiveMQConnection.java:303)
  at
org.springframework.jms.connection.SingleConnectionFactory.createSession(SingleConnectionFactory.java:406)
  at
org.springframework.jms.connection.CachingConnectionFactory.getSession(CachingConnectionFactory.java:225)
  at
org.springframework.jms.connection.SingleConnectionFactory$SharedConnectionInvocationHandler.invoke(SingleConnectionFactory.java:534)
  at $Proxy109.createSession(Unknown Source)
  at
org.springframework.jms.support.JmsAccessor.createSession(JmsAccessor.java:196)
  at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:457)
  at org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:534)

Here's the activemq setup (excerpt)

<broker useJmx="false" xmlns="http://activemq.apache.org/schema/core"
persistent="false">
  <transportConnectors>
    <transportConnector uri="${simulate.jms.url}"/>
  </transportConnectors>

</broker>

<bean id="jmsFactory" class="org.apache.activemq.ActiveMQConnectionFactory">
  <property name="brokerURL" value="${jms.url}" />
</bean>


<bean id="producerJmsTemplate"
class="org.springframework.jms.core.JmsTemplate">
  <property name="connectionFactory">
    
    <bean
      class="org.springframework.jms.connection.CachingConnectionFactory">
      <property name="targetConnectionFactory">
        <ref local="jmsFactory" />
      </property>
      <property name="sessionCacheSize" value="100" />
    </bean>
  </property>
</bean>

<bean id="simulationMessageSender"
class="com.forio.simulate.jms.SimulationMessageSenderImpl">
  <constructor-arg>
    <ref bean="producerJmsTemplate" />
  </constructor-arg>
</bean>

<bean id="destination" class="org.apache.activemq.command.ActiveMQQueue"
  autowire="constructor">
  <constructor-arg>
    <value>com.forio.simulate.jms.outgoing.queue</value>
  </constructor-arg>
</bean>


WILL

--
View this message in context: http://activemq.2283324.n4.nabble.com/slow-speed-sending-message-tp3804539p3804539.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: slow speed sending message

[Jason Whaley <ja...@gmail.com>]

> 
> Question - am I right in understanding that flow control will turn on
> automtically when the rate goes over a certain level?  Will it then shut
> off?


See here for PFC:  http://activemq.apache.org/producer-flow-control.html

In short, you'll hit it when you exhaust either the memoryLimit for a destination or the systemUsage limits for the entire broker.

> Another idea-- I'm using CachingConnectionFactory with 100 sessions.  What
> happens if I use more than those 100 sessions?  I'm wondering if they leak
> somehow.  (I use Spring's JMSTemplate, so I'd assume not, but it's a
> thought).

Any sessions created after you hit your maximum are created and disposed of on demand and never hit the cache.  

Re: slow speed sending message

[wglass <wg...@glass-husain.com>]

Thanks for the tip.  I didn't have JMX on when it slowed down.  I've turned
it on now and am waiting to see if it happened again.

Question - am I right in understanding that flow control will turn on
automtically when the rate goes over a certain level?  Will it then shut
off?

Second question.  I have a heap dump of when this was in action.  Is there a
class I can look at with the Eclipse Memory Analyzer to figure out if this
was going on?

Another idea-- I'm using CachingConnectionFactory with 100 sessions.  What
happens if I use more than those 100 sessions?  I'm wondering if they leak
somehow.  (I use Spring's JMSTemplate, so I'd assume not, but it's a
thought).

WILL

--
View this message in context: http://activemq.2283324.n4.nabble.com/slow-speed-sending-message-tp3804539p3808922.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: slow speed sending message

[Torsten Mielke <to...@fusesource.com>]

Perhaps try connecting to the broker using jconsole and check the brokers JMX statistics.
Do any of these at broker level or at destination level stick out and may explain the slow performance
that you encounter (queue size, memory percent usage of broker and of destination, store percent usage, temp percent usage). 
It could be that your producer is being flow controlled, in which case the queue should be running full.

Torsten Mielke
torsten@fusesource.com
tmielke@blogspot.com