You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Viju <vv...@gmail.com> on 2011/08/21 14:47:36 UTC
Camel Transaction performance - use of batches?
Im using a camel route which is transacted. I have set the transacted
property to be true in the JMSComponent via JMSConfig. In the route which
listens to a message on a queue (Consumer) , I have provided <transacted />
tag which makes the route transacted.
The DSL snippet is -
<bean id="jmsconfig"
class="org.apache.camel.component.jms.JmsConfiguration">
<property name="connectionFactory" ref="connFactory" />
<property name="transactionManager" ref="jmsTxnManager" />
<property name="transacted" value="true" />
</bean>
<camel:route id="processor">
<camel:from uri="myqueue" />
<camel:transacted/>
...............
The other properties of the message and consumers are
a. Messages are persistent
b. There are 80 concurrent consumers listening to the same queue.
c. The route, after receiving the message, passes the message to a bean that
process the message.
I made two test runs . one with transaction and the other without a
transaction. The observation is as follows
with transaction : route can consume 13 messages / second
without transaction: route can consume 60 messages / second. This is like 5
times faster than with transaction.
Im searching for options which will make the route faster with transactions.
I read a post which states that, once we put transaction, the read and write
from the queue is made synchronous. Is it also synchronized i.e. if I have
80 concurrent consumers, only one will commit at a time? So more consumers,
more slow due to locking.
I read another post which states that we can use batches. i.e. each message
should not be just one transaction. We need to batch a set of messages into
one transaction , say 100.
I could not figure out how to enable this in my DSL? Any pointers?
Also, if I use batching for messages with transaction and my beans processed
80 messages of the 100 in a batch and then JVM crashed, the 100
messages(including the 80 that is processed by the business logic) will be
recorded as not delivered in AMQ and hence will be redelivered next time.
This means that, the 80 messages will be reprocessed (Putting an idempotent
consumer fixes it but this is a workaround only in my opinion) . Is the
understanding correct?
--
View this message in context: http://camel.465427.n5.nabble.com/Camel-Transaction-performance-use-of-batches-tp4720389p4720389.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Camel Transaction performance - use of batches?
Posted by Viju <vv...@gmail.com>.
Dear Claus,
I enabled caching by adding the property as described above and the
performance is still the same, around 13 messages/ second compared to
60/second.
<property name="cacheLevelName" value="CACHE_CONSUMER" />
As described in my previous post, the locks are happening at a different
point.
Thanks
Viju
--
View this message in context: http://camel.465427.n5.nabble.com/Camel-Transaction-performance-use-of-batches-tp4720389p4720891.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Camel Transaction performance - use of batches?
Posted by Viju <vv...@gmail.com>.
I will enable the cache setting and test it. Just want to add an update that
the 80 consumer threads seems to be waiting on these two below points most
of the time.
Stack trace:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:341)
org.apache.activemq.transport.FutureResponse.getResult(FutureResponse.java:40)
org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:87)
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1262)
org.apache.activemq.TransactionContext.syncSendPacketWithInterruptionHandling(TransactionContext.java:666)
org.apache.activemq.TransactionContext.commit(TransactionContext.java:299)
org.apache.activemq.ActiveMQSession.commit(ActiveMQSession.java:558)
org.apache.activemq.pool.PooledSession.commit(PooledSession.java:133)
org.springframework.jms.connection.JmsTransactionManager.doCommit(JmsTransactionManager.java:236)
org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:732)
org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:701)
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:255)
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:982)
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:974)
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:876)
java.lang.Thread.run(Thread.java:637)
Or at this point -
Stack trace:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:341)
org.apache.activemq.transport.FutureResponse.getResult(FutureResponse.java:40)
org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:87)
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1262)
org.apache.activemq.TransactionContext.syncSendPacketWithInterruptionHandling(TransactionContext.java:666)
org.apache.activemq.TransactionContext.commit(TransactionContext.java:299)
org.apache.activemq.ActiveMQSession.commit(ActiveMQSession.java:558)
org.apache.activemq.pool.PooledSession.commit(PooledSession.java:133)
org.springframework.jms.support.JmsUtils.commitIfNecessary(JmsUtils.java:217)
org.springframework.jms.listener.AbstractMessageListenerContainer.commitIfNecessary(AbstractMessageListenerContainer.java:558)
org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:465)
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:323)
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:241)
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:982)
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:974)
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:876)
java.lang.Thread.run(Thread.java:637)
--
View this message in context: http://camel.465427.n5.nabble.com/Camel-Transaction-performance-use-of-batches-tp4720389p4720625.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Camel Transaction performance - use of batches?
Posted by Claus Ibsen <cl...@gmail.com>.
You can enable caching the consumer and the TX should go faster.
<bean id="jmsconfig"
class="org.apache.camel.component.jms.JmsConfiguration">
<property name="connectionFactory" ref="connFactory" />
<property name="transactionManager" ref="jmsTxnManager" />
<property name="transacted" value="true" />
<property name="cacheLevelName" value="CACHE_CONSUMER" />
</bean>
On Sun, Aug 21, 2011 at 2:47 PM, Viju <vv...@gmail.com> wrote:
> Im using a camel route which is transacted. I have set the transacted
> property to be true in the JMSComponent via JMSConfig. In the route which
> listens to a message on a queue (Consumer) , I have provided <transacted />
> tag which makes the route transacted.
>
> The DSL snippet is -
> <bean id="jmsconfig"
> class="org.apache.camel.component.jms.JmsConfiguration">
> <property name="connectionFactory" ref="connFactory" />
> <property name="transactionManager" ref="jmsTxnManager" />
> <property name="transacted" value="true" />
> </bean>
>
> <camel:route id="processor">
> <camel:from uri="myqueue" />
> <camel:transacted/>
> ...............
>
>
> The other properties of the message and consumers are
> a. Messages are persistent
> b. There are 80 concurrent consumers listening to the same queue.
> c. The route, after receiving the message, passes the message to a bean that
> process the message.
>
> I made two test runs . one with transaction and the other without a
> transaction. The observation is as follows
> with transaction : route can consume 13 messages / second
> without transaction: route can consume 60 messages / second. This is like 5
> times faster than with transaction.
>
> Im searching for options which will make the route faster with transactions.
>
> I read a post which states that, once we put transaction, the read and write
> from the queue is made synchronous. Is it also synchronized i.e. if I have
> 80 concurrent consumers, only one will commit at a time? So more consumers,
> more slow due to locking.
>
> I read another post which states that we can use batches. i.e. each message
> should not be just one transaction. We need to batch a set of messages into
> one transaction , say 100.
> I could not figure out how to enable this in my DSL? Any pointers?
>
> Also, if I use batching for messages with transaction and my beans processed
> 80 messages of the 100 in a batch and then JVM crashed, the 100
> messages(including the 80 that is processed by the business logic) will be
> recorded as not delivered in AMQ and hence will be redelivered next time.
> This means that, the 80 messages will be reprocessed (Putting an idempotent
> consumer fixes it but this is a workaround only in my opinion) . Is the
> understanding correct?
>
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Camel-Transaction-performance-use-of-batches-tp4720389p4720389.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Re: Camel Transaction performance - use of batches?
Posted by Viju <vv...@gmail.com>.
Thanks for the direction , Travis. I was pretty much using the same
configuration you mentioned except the cache_connection option. Still, the
worry is the considerable difference between transactional clients and non
transactional clients, in my case, non transactional are 5 times faster than
transactional clients.
This site from AMQ describes that the transactional client are not going to
be slower than the non transactional ones.
http://activemq.apache.org/should-i-use-transactions.html
--
View this message in context: http://camel.465427.n5.nabble.com/Camel-Transaction-performance-use-of-batches-tp4720389p4724449.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Camel Transaction performance - use of batches?
Posted by Viju <vv...@gmail.com>.
Thanks for the direction , Travis. I was pretty much using the same
configuration you mentioned except the cache_connection option. Still, the
worry is the considerable difference between transactional clients and non
transactional clients, in my case, non transactional are 5 times faster than
transactional clients.
This site from AMQ describes that the transactional client are not going to
be slower than the non transactional ones.
http://activemq.apache.org/should-i-use-transactions.html
--
View this message in context: http://camel.465427.n5.nabble.com/Camel-Transaction-performance-use-of-batches-tp4720389p4724448.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Camel Transaction performance - use of batches?
Posted by Travis Klotz <tr...@gmail.com>.
I haven't worked with batches, but I did just spend a bunch of time
working with setting up Local JMS Transactions with ActiveMQ and
Camel. The big things I found were:
* Make sure you turn the prefetch policy down to zero if you are
consuming messages across different machines. You can do this with
your connection string, so something like:
tcp://localhost:61616?jms.prefetchPolicy.queuePrefetch=0
* Make sure you are using a pooledConnectionFactory wrapped around the
normal AMQ connection factory (and make sure the pooled connection
factory is stopped on shutdown) So something like this if you are
using spring:
<bean id="pooledConnectionFactory"
class="org.apache.activemq.pool.PooledConnectionFactory"
destroy-method="stop">
<property name="connectionFactory" ref="activemqConnectionFactory" />
</bean>
* Set the cache level on the Jms/ActiveMQ Component properly. I found
things worked best when set to CACHE_CONNECTION when working with
consumers on multiple servers and Master/Slave failover on the AMQ
server.
<bean id="activemq"
class="org.apache.activemq.camel.component.ActiveMQComponent">
<property name="transacted" value="true"/>
<property name="connectionFactory" ref="pooledConnectionFactory"/>
<property name="transactionManager" ref="transactionManager"/>
<property name="cacheLevelName" value="CACHE_CONNECTION"/>
</bean>
Even with these settings, things are going to be quite a bit slower
than in a non-transactional setup. A lot of that is just the penalty
for using transactions. If performance is really critical you may
need to look at something like CLIENT_ACKNOWLEGE and deal with any
duplicate messages that may get passed through.
Travis
On Mon, Aug 22, 2011 at 7:08 AM, Viju <vv...@gmail.com> wrote:
> Can someone please help or provide pointers ? The documentation indeed talks
> about batches but cant see how is it feasible using camel batching?
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Camel-Transaction-performance-use-of-batches-tp4720389p4723156.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
Re: Camel Transaction performance - use of batches?
Posted by Viju <vv...@gmail.com>.
Can someone please help or provide pointers ? The documentation indeed talks
about batches but cant see how is it feasible using camel batching?
--
View this message in context: http://camel.465427.n5.nabble.com/Camel-Transaction-performance-use-of-batches-tp4720389p4723156.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Camel Transaction performance - use of batches?
Posted by David Karlsen <da...@gmail.com>.
The old thread
http://camel.465427.n5.nabble.com/Transactional-batching-td474002.html#a4721633
touches the same problem but w/o an solution.
2011/8/21 Viju <vv...@gmail.com>
> Im using a camel route which is transacted. I have set the transacted
> property to be true in the JMSComponent via JMSConfig. In the route which
> listens to a message on a queue (Consumer) , I have provided <transacted />
> tag which makes the route transacted.
>
> The DSL snippet is -
> <bean id="jmsconfig"
> class="org.apache.camel.component.jms.JmsConfiguration">
> <property name="connectionFactory" ref="connFactory" />
> <property name="transactionManager" ref="jmsTxnManager" />
> <property name="transacted" value="true" />
> </bean>
>
> <camel:route id="processor">
> <camel:from uri="myqueue" />
> <camel:transacted/>
> ...............
>
>
> The other properties of the message and consumers are
> a. Messages are persistent
> b. There are 80 concurrent consumers listening to the same queue.
> c. The route, after receiving the message, passes the message to a bean
> that
> process the message.
>
> I made two test runs . one with transaction and the other without a
> transaction. The observation is as follows
> with transaction : route can consume 13 messages / second
> without transaction: route can consume 60 messages / second. This is like
> 5
> times faster than with transaction.
>
> Im searching for options which will make the route faster with
> transactions.
>
> I read a post which states that, once we put transaction, the read and
> write
> from the queue is made synchronous. Is it also synchronized i.e. if I have
> 80 concurrent consumers, only one will commit at a time? So more consumers,
> more slow due to locking.
>
> I read another post which states that we can use batches. i.e. each message
> should not be just one transaction. We need to batch a set of messages into
> one transaction , say 100.
> I could not figure out how to enable this in my DSL? Any pointers?
>
> Also, if I use batching for messages with transaction and my beans
> processed
> 80 messages of the 100 in a batch and then JVM crashed, the 100
> messages(including the 80 that is processed by the business logic) will be
> recorded as not delivered in AMQ and hence will be redelivered next time.
> This means that, the 80 messages will be reprocessed (Putting an idempotent
> consumer fixes it but this is a workaround only in my opinion) . Is the
> understanding correct?
>
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Camel-Transaction-performance-use-of-batches-tp4720389p4720389.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
--
--
David J. M. Karlsen - http://www.linkedin.com/in/davidkarlsen