You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by nw31304 <da...@gmail.com> on 2014/05/21 06:06:55 UTC
Painfully slow AMQ consumer
I am attempting to do something really simple: read from an ActiveMQ queue
and append the messages to a file. Each message is roughly 100 bytes in
length and the arrival rate is no more than 1000 messages per minutes. No
matter what I try, messages queue up on the broker because the Camel route
does not process them quickly enough (i.e. append them to the file). The
disk I am writing to is SAN attached, an more than capable of these paltry
requirements. I have ridiculous amounts of CPU at my disposal, an I/O
utilization and queue time is minimal. The machine load average on both the
broker and the machine where I'm running camel is next to nothing, and none
of the relevant machines are short of memory. It doesn't matter if I run
the broker on the same machine or on a different machine, the result is the
same: a slowly growing queue. If I tail the file, literally see messages
scrolling slowly enough to read them as they're written. I'm using Camel
2.9.0, and ActiveMQ 5.5 (both client and broker). I thought perhaps it was
an issue with connection esablishment and teardown, but when I display the
connections, it appears that the same set of connections are being used.
Here's the relevant beans and routes:
<bean id="jmsConnectionFactory"
class="org.apache.activemq.ActiveMQConnectionFactory">
<property name="brokerURL" value="tcp://gsdb1:61616" />
</bean>
<bean id="pooledConnectionFactory"
class="org.apache.activemq.pool.PooledConnectionFactory" init-method="start"
destroy-method="stop">
<property name="maxConnections" value="20" />
<property name="connectionFactory" ref="jmsConnectionFactory" />
</bean>
<bean id="jmsConfig"
class="org.apache.camel.component.jms.JmsConfiguration">
<property name="connectionFactory" ref="pooledConnectionFactory"/>
<property name="concurrentConsumers" value="5"/>
</bean>
<bean id="activemq"
class="org.apache.activemq.camel.component.ActiveMQComponent">
<property name="configuration" ref="jmsConfig"/>
</bean>
<camelContext id="camel1" xmlns="http://camel.apache.org/schema/spring">
<jmxAgent id="agent" createConnector="true" registryPort="20023"/>
<route>
<from uri="activemq:gpsdata.gps.incoming.australia" />
<to
uri="file://../archive-australia?fileName=gpsdata.log&fileExist=Append"
/>
</camelContext>
Can anyone lend a hand with this, or perhaps give me hand as to where to
look?
Thanks in advance.
--
View this message in context: http://camel.465427.n5.nabble.com/Painfully-slow-AMQ-consumer-tp5751382.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Painfully slow AMQ consumer
Posted by Christian Müller <ch...@gmail.com>.
Have a look at slide 16 in [1]. You can find the source code at [1].
As Claus wrote, getting the write lock to update the file - line by line -
is not really efficient. Aggregate some lines before your update the file
or use the stream component.
I could achieve to write more than 15.000 lines (140 bytes each) per
SECOND...
Hope this will help.
[1] http://www.slideshare.net/muellerc/apache-con-na-2013
[2] https://github.com/muellerc/apachecon-na-2013
[3] http://camel.apache.org/stream.html
Best,
Christian
-----------------
Software Integration Specialist
Apache Member
V.P. Apache Camel | Apache Camel PMC Member | Apache Camel committer
Apache Incubator PMC Member
https://www.linkedin.com/pub/christian-mueller/11/551/642
On Wed, May 21, 2014 at 6:06 AM, nw31304 <da...@gmail.com>
wrote:
> I am attempting to do something really simple: read from an ActiveMQ queue
> and append the messages to a file. Each message is roughly 100 bytes in
> length and the arrival rate is no more than 1000 messages per minutes. No
> matter what I try, messages queue up on the broker because the Camel route
> does not process them quickly enough (i.e. append them to the file). The
> disk I am writing to is SAN attached, an more than capable of these paltry
> requirements. I have ridiculous amounts of CPU at my disposal, an I/O
> utilization and queue time is minimal. The machine load average on both
> the
> broker and the machine where I'm running camel is next to nothing, and none
> of the relevant machines are short of memory. It doesn't matter if I run
> the broker on the same machine or on a different machine, the result is the
> same: a slowly growing queue. If I tail the file, literally see messages
> scrolling slowly enough to read them as they're written. I'm using Camel
> 2.9.0, and ActiveMQ 5.5 (both client and broker). I thought perhaps it was
> an issue with connection esablishment and teardown, but when I display the
> connections, it appears that the same set of connections are being used.
> Here's the relevant beans and routes:
>
> <bean id="jmsConnectionFactory"
> class="org.apache.activemq.ActiveMQConnectionFactory">
> <property name="brokerURL" value="tcp://gsdb1:61616" />
> </bean>
>
> <bean id="pooledConnectionFactory"
> class="org.apache.activemq.pool.PooledConnectionFactory"
> init-method="start"
> destroy-method="stop">
> <property name="maxConnections" value="20" />
> <property name="connectionFactory" ref="jmsConnectionFactory" />
> </bean>
>
> <bean id="jmsConfig"
> class="org.apache.camel.component.jms.JmsConfiguration">
> <property name="connectionFactory" ref="pooledConnectionFactory"/>
> <property name="concurrentConsumers" value="5"/>
> </bean>
>
> <bean id="activemq"
> class="org.apache.activemq.camel.component.ActiveMQComponent">
> <property name="configuration" ref="jmsConfig"/>
> </bean>
>
> <camelContext id="camel1" xmlns="http://camel.apache.org/schema/spring
> ">
>
> <jmxAgent id="agent" createConnector="true" registryPort="20023"/>
>
> <route>
> <from uri="activemq:gpsdata.gps.incoming.australia" />
> <to
> uri="file://../archive-australia?fileName=gpsdata.log&fileExist=Append"
> />
> </camelContext>
>
>
> Can anyone lend a hand with this, or perhaps give me hand as to where to
> look?
>
> Thanks in advance.
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Painfully-slow-AMQ-consumer-tp5751382.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
Re: Painfully slow AMQ consumer
Posted by nw31304 <da...@gmail.com>.
I've done a tcpdump, and it seems that regardless of the number of
concurrentConsumers I specify, only one TCP/IP connection is ever being
used. I guess what I'm trying to understand is how to use multiple tCP/IP
connections in parallel. Would I need to use some sort of thread pool or
SEDA component to accomplish this?
--
View this message in context: http://camel.465427.n5.nabble.com/Painfully-slow-AMQ-consumer-tp5751382p5751383.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Painfully slow AMQ consumer
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Try without writing to the file, but just do a
from jms
to log
and see if that is faster.
As appending to the *same* file name happens sequentially in Camel.
On Wed, May 21, 2014 at 6:06 AM, nw31304 <da...@gmail.com> wrote:
> I am attempting to do something really simple: read from an ActiveMQ queue
> and append the messages to a file. Each message is roughly 100 bytes in
> length and the arrival rate is no more than 1000 messages per minutes. No
> matter what I try, messages queue up on the broker because the Camel route
> does not process them quickly enough (i.e. append them to the file). The
> disk I am writing to is SAN attached, an more than capable of these paltry
> requirements. I have ridiculous amounts of CPU at my disposal, an I/O
> utilization and queue time is minimal. The machine load average on both the
> broker and the machine where I'm running camel is next to nothing, and none
> of the relevant machines are short of memory. It doesn't matter if I run
> the broker on the same machine or on a different machine, the result is the
> same: a slowly growing queue. If I tail the file, literally see messages
> scrolling slowly enough to read them as they're written. I'm using Camel
> 2.9.0, and ActiveMQ 5.5 (both client and broker). I thought perhaps it was
> an issue with connection esablishment and teardown, but when I display the
> connections, it appears that the same set of connections are being used.
> Here's the relevant beans and routes:
>
> <bean id="jmsConnectionFactory"
> class="org.apache.activemq.ActiveMQConnectionFactory">
> <property name="brokerURL" value="tcp://gsdb1:61616" />
> </bean>
>
> <bean id="pooledConnectionFactory"
> class="org.apache.activemq.pool.PooledConnectionFactory" init-method="start"
> destroy-method="stop">
> <property name="maxConnections" value="20" />
> <property name="connectionFactory" ref="jmsConnectionFactory" />
> </bean>
>
> <bean id="jmsConfig"
> class="org.apache.camel.component.jms.JmsConfiguration">
> <property name="connectionFactory" ref="pooledConnectionFactory"/>
> <property name="concurrentConsumers" value="5"/>
> </bean>
>
> <bean id="activemq"
> class="org.apache.activemq.camel.component.ActiveMQComponent">
> <property name="configuration" ref="jmsConfig"/>
> </bean>
>
> <camelContext id="camel1" xmlns="http://camel.apache.org/schema/spring">
>
> <jmxAgent id="agent" createConnector="true" registryPort="20023"/>
>
> <route>
> <from uri="activemq:gpsdata.gps.incoming.australia" />
> <to
> uri="file://../archive-australia?fileName=gpsdata.log&fileExist=Append"
> />
> </camelContext>
>
>
> Can anyone lend a hand with this, or perhaps give me hand as to where to
> look?
>
> Thanks in advance.
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Painfully-slow-AMQ-consumer-tp5751382.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
hawtio: http://hawt.io/
fabric8: http://fabric8.io/