You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by James Black <be...@yahoo.co.uk> on 2014/09/02 14:05:43 UTC
Consumer performance problem with Tx
Hi,
I have a problem that when using transactions all my consumers for a
particular queue are getting blocking waiting for the commit. We are using
the Spring DMLC and are caching at the consumer level. This is ActiveMQ
5.5.1
All threads end up at a WAITING state with the same stack trace as below:
"service--51" - Thread t@96
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <12021c84> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:374)
at
org.apache.activemq.transport.FutureResponse.getResult(FutureResponse.java:40)
at
org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:87)
at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1284)
at
org.apache.activemq.TransactionContext.syncSendPacketWithInterruptionHandling(TransactionContext.java:707)
at
org.apache.activemq.TransactionContext.commit(TransactionContext.java:299)
at org.apache.activemq.ActiveMQSession.commit(ActiveMQSession.java:560)
at
org.springframework.jms.support.JmsUtils.commitIfNecessary(JmsUtils.java:217)
at
org.springframework.jms.listener.AbstractMessageListenerContainer.commitIfNecessary(AbstractMessageListenerContainer.java:577)
at
org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:482)
at
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:325)
at
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:243)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1103)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1095)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:992)
at java.lang.Thread.run(Thread.java:724)
Locked ownable synchronizers:
- None
I know that the transaction speed is limited by the disk performance but we
do not see the disk IO jump beyond what we would expect it can handle.
Anyone got any ideas?
Thanks,
Justin
--
View this message in context: http://activemq.2283324.n4.nabble.com/Consumer-performance-problem-with-Tx-tp4685226.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Consumer performance problem with Tx
Posted by Matt Pavlovich <ma...@gmail.com>.
Justin-
JMS connections act and behave very different from JDBC connections. Just like a long query can block other apps trying to access the connection on a shared JDBC connection, a slow consumer can impact others trying to access it as well. You can do multiple sessions on top of a single connection, but it can get hairy and I don’t get the sense that the intention was to support connection pooling, but rather allow for a single program to do all its producing and consuming over a single connection— not several programs sharing a single connection.
Separate connections per consumer is always a safe starting point. Pre-tuning for shared JMS connections, usually results in de-tuning. I think you’ll find single connection per consumer will meet your performance needs.
If you absolutely feel like you must share sessions on top of a single connection, reduce prefetch=“1”, ensure your memoryLimit is well above the default “1 mb”, use NIO on the transportConnector, optimizedDispatch=“true” and dedicatedTaskRunner=“false”.
-Matt
On Sep 8, 2014, at 6:25 AM, James Black <be...@yahoo.co.uk> wrote:
> Hi Matt,
>
> we don't use JmsTemplate in this particular bit of code.
>
> I have done some testing this morning and the problem seems to be contention
> with the shared connection. The Spring DMLC is configured to cache at
> Connection, Session and Consumer level. If caching is enabled at all
> (connection+) then it uses a shared single connection. I can verify this by
> monitoring ActiveMq through Jmx.
>
> So if I have a DMLC with 20 consumer threads, all sharing the same
> connection it is slow, but at low throughput you would not notice.
>
> If I create 20 separate DMLCs all with a single thread and all with
> different connections, then the performance is great in comparison!
>
> Not sure why this would be as I believed that it is normal to share the
> connection for JMS unlike JDBC and I would of assumed that ActiveMQ is
> optimised for that case?
>
> Thanks,
>
> Justin
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Consumer-performance-problem-with-Tx-tp4685226p4685365.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Consumer performance problem with Tx
Posted by James Black <be...@yahoo.co.uk>.
Hi Matt,
we don't use JmsTemplate in this particular bit of code.
I have done some testing this morning and the problem seems to be contention
with the shared connection. The Spring DMLC is configured to cache at
Connection, Session and Consumer level. If caching is enabled at all
(connection+) then it uses a shared single connection. I can verify this by
monitoring ActiveMq through Jmx.
So if I have a DMLC with 20 consumer threads, all sharing the same
connection it is slow, but at low throughput you would not notice.
If I create 20 separate DMLCs all with a single thread and all with
different connections, then the performance is great in comparison!
Not sure why this would be as I believed that it is normal to share the
connection for JMS unlike JDBC and I would of assumed that ActiveMQ is
optimised for that case?
Thanks,
Justin
--
View this message in context: http://activemq.2283324.n4.nabble.com/Consumer-performance-problem-with-Tx-tp4685226p4685365.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Consumer performance problem with Tx
Posted by Matt Pavlovich <ma...@gmail.com>.
Justin-
Sorry to hear that you’ve had problems with ActiveMQ in the past. I’ve had a lot of successful deployments in large-scale environments (3,000+ brokers, hub and spoke with 1 broker serving up to 1,000 clients, 4,000 queues and 3,000 total connections) with transactions, and the full boat of features with newer releases of ActiveMQ. If you don’t have luck with alternatives, I suggest you give newer versions a second look.
The replicated Master-Slave was definitely a lightly-maintained feature and has a lot of operational problems (how to resync after an outage, etc).
Back to your original issue.. I've seen a lot of issues with Spring JMS Template on the consumer side (esp with transactions). Recommend trying with straight JMS code to isolate if you have a pooling/caching issue or a broker bug and go from there.
-Matt
On Sep 4, 2014, at 3:54 AM, James Black <be...@yahoo.co.uk> wrote:
> Hi Matt,
>
> thanks for the response.
>
> The reason that we have not migrated to a later version of ActiveMq is that
> we are looking to move away from using ActiveMq due to the problems we have
> had. Therefore, we wanted to avoid going through the whole testing process
> for a later version of ActiveMq. We use shared nothing master/slave
> replicated which has been removed in later ActiveMq versions.
>
> We are doing JMS local (originally we tried XA but had massive problems with
> that) and we have tested with all caching settings for the DMLC to no avail.
> Our connection is not provided from a pool but directly from standard
> ActiveMq connection factory (org.apache.activemq.ActiveMQConnectionFactory),
> due to the fact that it is cached in the DMLC.
>
> Thanks,
>
> Justin
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Consumer-performance-problem-with-Tx-tp4685226p4685279.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Consumer performance problem with Tx
Posted by James Black <be...@yahoo.co.uk>.
Hi Matt,
thanks for the response.
The reason that we have not migrated to a later version of ActiveMq is that
we are looking to move away from using ActiveMq due to the problems we have
had. Therefore, we wanted to avoid going through the whole testing process
for a later version of ActiveMq. We use shared nothing master/slave
replicated which has been removed in later ActiveMq versions.
We are doing JMS local (originally we tried XA but had massive problems with
that) and we have tested with all caching settings for the DMLC to no avail.
Our connection is not provided from a pool but directly from standard
ActiveMq connection factory (org.apache.activemq.ActiveMQConnectionFactory),
due to the fact that it is cached in the DMLC.
Thanks,
Justin
--
View this message in context: http://activemq.2283324.n4.nabble.com/Consumer-performance-problem-with-Tx-tp4685226p4685279.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Consumer performance problem with Tx
Posted by Matt Pavlovich <ma...@gmail.com>.
Hi James-
ActiveMQ 5.5.1 is quite aged at this point. Any reason you aren’t looking at a newer version?
Transactions and connection pooling is tricky— are you doing XA or JMS Local? I suggest trying to get it all working, then tune.. CACHE_NONE first.
-Matt
On Sep 2, 2014, at 7:05 AM, James Black <be...@yahoo.co.uk> wrote:
> Hi,
>
> I have a problem that when using transactions all my consumers for a
> particular queue are getting blocking waiting for the commit. We are using
> the Spring DMLC and are caching at the consumer level. This is ActiveMQ
> 5.5.1
>
> All threads end up at a WAITING state with the same stack trace as below:
>
> "service--51" - Thread t@96
> java.lang.Thread.State: WAITING
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <12021c84> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> at
> java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:374)
> at
> org.apache.activemq.transport.FutureResponse.getResult(FutureResponse.java:40)
> at
> org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:87)
> at
> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1284)
> at
> org.apache.activemq.TransactionContext.syncSendPacketWithInterruptionHandling(TransactionContext.java:707)
> at
> org.apache.activemq.TransactionContext.commit(TransactionContext.java:299)
> at org.apache.activemq.ActiveMQSession.commit(ActiveMQSession.java:560)
> at
> org.springframework.jms.support.JmsUtils.commitIfNecessary(JmsUtils.java:217)
> at
> org.springframework.jms.listener.AbstractMessageListenerContainer.commitIfNecessary(AbstractMessageListenerContainer.java:577)
> at
> org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:482)
> at
> org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:325)
> at
> org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:243)
> at
> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1103)
> at
> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1095)
> at
> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:992)
> at java.lang.Thread.run(Thread.java:724)
> Locked ownable synchronizers:
> - None
>
> I know that the transaction speed is limited by the disk performance but we
> do not see the disk IO jump beyond what we would expect it can handle.
>
> Anyone got any ideas?
>
> Thanks,
>
> Justin
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Consumer-performance-problem-with-Tx-tp4685226.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.