You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Josh Carlson <jc...@e-dialog.com> on 2011/05/04 17:31:13 UTC
How does prefetch work with client acknoledgements?
If I have prefetch set to 1 and I retrieve a message, is another one dispatched before it is ack'd? I'm trying to scale the number of consumers. My benchmark test pre-produces 50 messages for each consumer (all in one queue). Then sets N consumers active and I measure the time it takes to pull the message of the queue.
In my benchmarking I've found a very nice MEAN time to retrieve a message at 2ms, regardless of the number of consumers. However, the deviation is very bad. My hypothesis is that some consumers are getting stuck not able to pull messages off the queue because each consumer can actually be dispatched 2 messages. Also the faster the consumer the worse the deviation.
Re: How does prefetch work with client acknoledgements?
Posted by "lernen.2007" <er...@googlemail.com>.
I think the activemq developer should stop to develop and search this bug. A
messaging system should save that every messages is delivery from producer
to consumer but not stuck in queue.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3508704.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: How does prefetch work with client acknoledgements?
Posted by "lernen.2007" <er...@googlemail.com>.
Hello,
I extend the heap size of the consumer but that didn't help.
Regards
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3833530.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: How does prefetch work with client acknoledgements?
Posted by "lernen.2007" <er...@googlemail.com>.
Hi,
I think that was the problem what we had at the moment:
https://issues.apache.org/jira/browse/AMQ-2484
what you think over that?
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3833610.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: How does prefetch work with client acknoledgements?
Posted by pwanner <pw...@pwanner.com>.
Hi Martin,
Thank you for your answer.
It seems that the cache could be the source of the problem as Gary also
pointed that out
Gary Tully wrote:
>
> does the message appear as inflight to any consumer? What is the
> inflight count on the destination? If it is >1 you should be able to
> track it to a consumer that is probably *cached*.
>
Since the problem occures in production I'll try to add the parameter
*wireFormat.cacheEnabled=false *ASAP.
Cheers,
Philippe
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3739611.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: How does prefetch work with client acknoledgements?
Posted by "Martin C." <ma...@gmx.at>.
Hi,
On Wed, Aug 10, 2011 at 6:21 PM, pwanner <pw...@pwanner.com> wrote:
> Hi,
>
> We have also an other broker (same very simple config master/slave on SAN)
> for an other app, with a queue and only one consumer that is configured with
> Camel (see config below), and this consumer also stop consuming messages,
> thus they say stuck in the queue.
We (hopefully) had a similar issue with consumers stopping
consumption. Additionally we had a problem with connections to
localhost broker failing, with a stacktrace indicating that decoding
of the packed message failed. Therefore we switched to connect with
"...&wireFormat.cacheEnabled=false&wireFormat.tightEncodingEnabled=false"
in order to disable tight encoding and caching. Since we did this last
week, no more outages and oddly also the blocked consumers issue went
away. I still hope that this may have magically fixed our problems.
You might want to give this a try.
Best regards,
Martin
Re: How does prefetch work with client acknoledgements?
Posted by pwanner <pw...@pwanner.com>.
Hi,
We have also an other broker (same very simple config master/slave on SAN)
for an other app, with a queue and only one consumer that is configured with
Camel (see config below), and this consumer also stop consuming messages,
thus they say stuck in the queue.
I used pretty much all the options I could find to avoid this problem but
with no luck:
jms.prefetchPolicy.all=1
jms.dispatchAsync=false
jms.useAsyncSend=false
So two differents type consumers (Spring DMLC and Camel) but the same
problem... and no more ideas :-)
<beans
xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://camel.apache.org/schema/spring
http://camel.apache.org/schema/spring/camel-spring.xsd
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd">
<camelContext id="camel" xmlns="http://camel.apache.org/schema/spring">
<route>
<from uri="weblogic:queue:ToActiveMQ"/>
<transacted ref="PROPAGATION_REQUIRED"/>
<to uri="activemq:queue:from.weblogic"/>
</route>
<route>
<from uri="activemq:queue:to.weblogic"/>
<transacted ref="PROPAGATION_REQUIRED"/>
<to uri="weblogic:queue:FromActiveMQ"/>
</route>
</camelContext>
<bean id="PROPAGATION_REQUIRED"
class="org.apache.camel.spring.spi.SpringTransactionPolicy">
<property name="transactionManager" ref="activemqTransactionManager"/>
<property name="propagationBehaviorName" value="PROPAGATION_REQUIRED"/>
</bean>
<bean id="weblogic" class="org.apache.camel.component.jms.JmsComponent">
<property name="connectionFactory"
ref="weblogicCachedConnectionFactory"/>
<property name="destinationResolver" ref="weblogicDestinationResolver"/>
<property name="deliveryPersistent" value="true"/>
<property name="transacted" value="true"/>
</bean>
<bean id="weblogicDestinationResolver"
class="org.springframework.jms.support.destination.JndiDestinationResolver">
<property name="jndiTemplate" ref="remoteJndi" />
</bean>
<bean id="weblogicCachedConnectionFactory"
class="org.springframework.jms.connection.SingleConnectionFactory">
<property name="targetConnectionFactory"
ref="weblogicConnectionFactory"/>
<property name="reconnectOnException" value="true"/>
</bean>
<bean id="weblogicConnectionFactory"
class="org.springframework.jndi.JndiObjectFactoryBean">
<property name="jndiTemplate" ref="remoteJndi" />
<property name="jndiName" value="weblogic.jms.ConnectionFactory" />
</bean>
<bean id="remoteJndi" class="org.springframework.jndi.JndiTemplate">
<property name="environment">
<props>
<prop
key="java.naming.factory.initial">weblogic.jndi.WLInitialContextFactory</prop>
<prop key="java.naming.provider.url">t3://host:7004</prop>
</props>
</property>
</bean>
<bean id="activemqTransactionManager"
class="org.springframework.jms.connection.JmsTransactionManager">
<property name="connectionFactory" ref="amqCachedConnectionFactory" />
</bean>
<bean id="activemq"
class="org.apache.activemq.camel.component.ActiveMQComponent" >
<property name="connectionFactory" ref="amqCachedConnectionFactory"/>
<property name="transacted" value="true"/>
</bean>
<bean id="amqCachedConnectionFactory"
class="org.springframework.jms.connection.SingleConnectionFactory">
<property name="targetConnectionFactory" ref="amqConnectionFactory"/>
<property name="reconnectOnException" value="true"/>
</bean>
<bean id="amqConnectionFactory"
class="org.apache.activemq.ActiveMQConnectionFactory">
<property name="brokerURL"
value="failover://(tcp://host1:8016,tcp://host2:8016)?jms.prefetchPolicy.all=1&jms.dispatchAsync=false&jms.useAsyncSend=false&trackMessages=true&timeout=5000&jms.redeliveryPolicy.useExponentialBackOff=true&jms.maximumRedeliveryDelay=300000L&jms.redeliveryPolicy.maximumRedeliveries=-1"
/>
</bean>
</beans>
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3733449.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: How does prefetch work with client acknoledgements?
Posted by "lernen.2007" <er...@googlemail.com>.
Hi,
I think a solution in activemq version 5.6.0 is needed. I don't know what
the main problem is so that after a failover the messages stuck on the
server side. The activemq developer should know what going wrong.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3717861.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: How does prefetch work with client acknoledgements?
Posted by pwanner <pw...@pwanner.com>.
Hi Gary, thank you for the answer.
I have nothing yet to reproduce the problem.
But thank you for giving a tip.
According to your blog entry
http://blog.garytully.com/2010/01/activemq-prefetch-and-asyncdispatch.html
ActiveMQ prefetch , shoudn't the trackMessages property of the failover
transport always be set to true in an Shared File System Master Slave
configuration?
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3701385.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: How does prefetch work with client acknoledgements?
Posted by Gary Tully <ga...@gmail.com>.
does the message appear as inflight to any consumer? What is the
inflight count on the destination? If it is >1 you should be able to
track it to a consumer that is probably cached.
On 27 July 2011 10:24, pwanner <pw...@pwanner.com> wrote:
> Hi everybody,
>
> I just had a server crash today and the slave ActiveMQ instance went active,
> and a message is stuck in the queue. The new messages are consumed but there
> is one that stays.
>
> This is very annoying...
>
> I use ActiveMQ 5.5.0 (same problem with 5.4.2) and Spring 3.0.5 as the
> consumer.
>
> Below is the config:
>
> <jms:listener-container concurrency="5"
> destination-type="queue"
> connection-factory="amqCachedConnectionFactory"
> transaction-manager="transactionManager"
> acknowledge="transacted" >
> <jms:listener destination="events" ref="eventHandler" />
> </jms:listener-container>
>
> <bean id="amqCachedConnectionFactory"
> class="org.springframework.jms.connection.CachingConnectionFactory"
> p:targetConnectionFactory-ref="amqConnectionFactory"
> p:sessionCacheSize="5" />
>
> <amq:connectionFactory id="amqConnectionFactory"
>
> brokerURL="failover://(${broker.url.1},${broker.url.2})?jms.prefetchPolicy.all=1&timeout=5000"
> />
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3697960.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
--
http://fusesource.com
http://blog.garytully.com
Re: How does prefetch work with client acknoledgements?
Posted by pwanner <pw...@pwanner.com>.
Hi everybody,
I just had a server crash today and the slave ActiveMQ instance went active,
and a message is stuck in the queue. The new messages are consumed but there
is one that stays.
This is very annoying...
I use ActiveMQ 5.5.0 (same problem with 5.4.2) and Spring 3.0.5 as the
consumer.
Below is the config:
<jms:listener-container concurrency="5"
destination-type="queue"
connection-factory="amqCachedConnectionFactory"
transaction-manager="transactionManager"
acknowledge="transacted" >
<jms:listener destination="events" ref="eventHandler" />
</jms:listener-container>
<bean id="amqCachedConnectionFactory"
class="org.springframework.jms.connection.CachingConnectionFactory"
p:targetConnectionFactory-ref="amqConnectionFactory"
p:sessionCacheSize="5" />
<amq:connectionFactory id="amqConnectionFactory"
brokerURL="failover://(${broker.url.1},${broker.url.2})?jms.prefetchPolicy.all=1&timeout=5000"
/>
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3697960.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: How does prefetch work with client acknoledgements?
Posted by "lernen.2007" <er...@googlemail.com>.
Hi,
we have the problem with all the activemq versions.The messages stuck after
a failover.
Regards
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3675465.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: How does prefetch work with client acknoledgements?
Posted by "lernen.2007" <er...@googlemail.com>.
Hi,
thanks for answering. Here is our configuration on the client side:
<beans>
<bean
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="locations">
<value>value.properties</value>
</property>
</bean>
<broker xmlns="http://activemq.apache.org/schema/core"
brokerName="localhost"
useJmx="true"
destroyApplicationContextOnStop="true">
<networkConnectors>
<networkConnector
uri="static://(failover://(ssl://test.com:6861)?maxReconnectDelay=5000)"
networkTTL="10"
name="o">
<excludedDestinations>
<queue physicalName="a"/>
</excludedDestinations>
<staticallyIncludedDestinations>
<queue physicalName="b"/>
</staticallyIncludedDestinations>
</networkConnector>
</networkConnectors>
<persistenceAdapter>
<kahaDB directory="b" journalMaxFileLength="1mb"/>
</persistenceAdapter>
<systemUsage>
<systemUsage sendFailIfNoSpace="true">
<memoryUsage>
<memoryUsage limit="10mb" />
</memoryUsage>
<tempUsage>
<tempUsage limit="10mb" />
</tempUsage>
<storeUsage>
<storeUsage limit="100mb" percentUsageMinDelta="1" />
</storeUsage>
</systemUsage>
</systemUsage>
</broker>
</beans>
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3648087.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: How does prefetch work with client acknoledgements?
Posted by Gary Tully <ga...@gmail.com>.
can you add more detail to that jira entry, include your configuration
and verify that the sequence of steps to reproduce are still
accurate.
For a network connector using static discovery, if you use failover to
choose from a set of urls, ensure you use maxReconnectAttempts=1 such
that transport failures bubble up to the network connector. It is
important that the network connector handles network reconnection.
In your case, with a single url, just use static://ssl://test:61616):
On 5 July 2011 13:48, lernen.2007 <er...@googlemail.com> wrote:
> Here is a Jira-Entry:
>
> https://issues.apache.org/jira/browse/AMQ-2614
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3645856.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
--
http://fusesource.com
http://blog.garytully.com
Re: How does prefetch work with client acknoledgements?
Posted by "lernen.2007" <er...@googlemail.com>.
Here is a Jira-Entry:
https://issues.apache.org/jira/browse/AMQ-2614
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3645856.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: How does prefetch work with client acknoledgements?
Posted by "lernen.2007" <er...@googlemail.com>.
Hi,
I don't have any solution. The messages stuck further after a failover. Is
there any fix in version 5.6.0?
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3645806.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: How does prefetch work with client acknoledgements?
Posted by "lernen.2007" <er...@googlemail.com>.
Hi,
here is a Jira-Entry:
https://issues.apache.org/jira/browse/AMQ-2614
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3645854.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: How does prefetch work with client acknoledgements?
Posted by "lernen.2007" <er...@googlemail.com>.
Hi,
I set the following configuration so that not messages stuck:
networkConnector
uri="static://(failover://(ssl://test:61616))?maxReconnectDelay=5000"
networkTTL="10"
dynamicOnly="true"
prefetchSize="1"
name="outbox">
<excludedDestinations>
<queue physicalName="A"/>
</excludedDestinations>
<staticallyIncludedDestinations>
<queue physicalName="B"/>
</staticallyIncludedDestinations>
</networkConnector>
</networkConnectors>
But the failover doesn't work.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3587855.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: How does prefetch work with client acknoledgements?
Posted by "lernen.2007" <er...@googlemail.com>.
We have following situation:
Client Server
Server
ActiveMQ Instance1 ActiveMQ Instance2 <----
Bridge
NetworkConnector Transport Connector
Weblogic
The messages stuck further. Where all I must configure the prefetch size?
Please help.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3535464.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: How does prefetch work with client acknoledgements?
Posted by "lernen.2007" <er...@googlemail.com>.
Hi,
thanks for answer but we use a bridge ActiveMQ <--- Weblogic and I think in
this situation can the prefetch limit solution doesn't help further because
the consumer is the weblogic.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3528960.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: How does prefetch work with client acknoledgements?
Posted by "Martin C." <ma...@gmx.at>.
Hi,
Prefetching only applies when consuming messages.
Best regards,
Martin
On Mon, May 16, 2011 at 4:12 PM, lernen.2007
<er...@googlemail.com> wrote:
> I need your help.
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3526298.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: How does prefetch work with client acknoledgements?
Posted by "lernen.2007" <er...@googlemail.com>.
I need your help.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3526298.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: How does prefetch work with client acknoledgements?
Posted by "lernen.2007" <er...@googlemail.com>.
Should we configure "prefetch" on the producer side or on consumer side or in
both side.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3517057.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: How does prefetch work with client acknoledgements?
Posted by Gary Tully <ga...@gmail.com>.
yes, the idea of prefetch is to deliver another message on or before
an ack such that it is available on the client before it is needed.
Some deviation is expected because a fast consumer will be able to ack
messages faster than the broker can dispatch them because it is doing
dispatch in batches of 1 when the prefetch is 1.
This is why we use a larger prefetch by default.
With prefetch=0, a client pulls a single message from the broker,
there is no prefetch but there is a round trip to the broker for each
pull request, so while the mean will be higher, the deviation will be
less.
To get the lowest mean and deviation and understand what is happening
further, preload with X messages and set the prefetch to X/N where N
is the number of consumers.
see: http://activemq.apache.org/what-is-the-prefetch-limit-for.html
On 4 May 2011 16:31, Josh Carlson <jc...@e-dialog.com> wrote:
> If I have prefetch set to 1 and I retrieve a message, is another one dispatched before it is ack'd? I'm trying to scale the number of consumers. My benchmark test pre-produces 50 messages for each consumer (all in one queue). Then sets N consumers active and I measure the time it takes to pull the message of the queue.
>
> In my benchmarking I've found a very nice MEAN time to retrieve a message at 2ms, regardless of the number of consumers. However, the deviation is very bad. My hypothesis is that some consumers are getting stuck not able to pull messages off the queue because each consumer can actually be dispatched 2 messages. Also the faster the consumer the worse the deviation.
>
--
http://blog.garytully.com
http://fusesource.com