You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by "lernen.2007" <er...@googlemail.com> on 2011/04/07 15:29:52 UTC
Re: Messages stuck in queue
The same problem we have with the latest activemq version 5.0.0 too. Have you
any idea how we can resolve this problem?
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3433435.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Messages stuck in queue
Posted by nnprasad <NN...@dwd.IN.gov>.
No Guys,
till yesterday, we have been facing the same problem with 5.2.0 version and
upgraded to 5.4.2.
After reading your posts, I am really afraid how we can survive in
production, Please look into this issue links i created.
http://activemq.2283324.n4.nabble.com/Message-stuck-on-queue-td3338786.html#a3341660
http://fusesource.com/forums/thread.jspa?threadID=2795&tstart=0
http://activemq.2283324.n4.nabble.com/Strange-Issue-Particular-Queue-Not-sending-Msgs-to-Consumer-tt3331634.html
http://activemq.2283324.n4.nabble.com/Connection-Re-balance-Question-tt3325301.html#a3328303
http://activemq.2283324.n4.nabble.com/ActiveMQ-5-2-0-transport-stops-responding-td3385271.html#a3385677
And JIRA Issues:
https://issues.apache.org/jira/browse/AMQ-1751
https://issues.apache.org/jira/browse/AMQ-2009
And Gary Tully is asking test case for this issue, I do not know how can we
create a test case for this type issue. we do not know when it is happening.
if can you guys help for it, greatly appreciated, other wise please leave a
note in those links to support that.
Thank You,
Nag.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3433507.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Messages stuck in queue
Posted by nnprasad <NN...@dwd.IN.gov>.
Hi Gary,
I am trying my level best to give you right inputs.
Please go through the below link where at the last I provided a scenario
http://activemq.2283324.n4.nabble.com/Message-stuck-on-queue-td3338786.html#a3341660
when that scenario was happening I captured the log and here it is..
http://fusesource.com/forums/thread.jspa?threadID=2795&tstart=0
And some one explained their inability to create good test case BUT provided
some thing for this in an issue
https://issues.apache.org/jira/browse/AMQ-2009 (20/Nov/08)
and as you said, I already tried (though when this issue happened, I was
monitoring JCONSOLE memory graph and found good, no high memory usages.)with
increasing memory limit to 1GB from 512MB but in vain. (But I did not tried
removing memory limit..how does AMQ handle with no memory specified?)
in the issue https://issues.apache.org/jira/browse/AMQ-3225 I provided
attachments and some useful links which talks about this issue.
I am sure there is no Exceptions logged in activemq logs even with debug
level..Even in Application's Spring Logs there were no exceptions except
that a message "No message received since xxxxx sec".
When this is happening I ran a separate test (simple JMS/No Spring API)
program and hit that queue (even created a new queue in that activemq). My
test consumer is waiting to receive a message. by this we can say no
suspectable issue with Springs JMS framework API.
Hope I gave sufficient inputs on this issue, when ever I saw a post all
these times. But I will still try for the best.
Thank You,
Nag.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3433708.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: How does prefetch work with client acknoledgements?
Posted by "lernen.2007" <er...@googlemail.com>.
Hello,
I extend the heap size of the consumer but that didn't help.
Regards
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3833530.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: How does prefetch work with client acknoledgements?
Posted by "lernen.2007" <er...@googlemail.com>.
Hi,
I think that was the problem what we had at the moment:
https://issues.apache.org/jira/browse/AMQ-2484
what you think over that?
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3833610.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: How does prefetch work with client acknoledgements?
Posted by pwanner <pw...@pwanner.com>.
Hi Martin,
Thank you for your answer.
It seems that the cache could be the source of the problem as Gary also
pointed that out
Gary Tully wrote:
>
> does the message appear as inflight to any consumer? What is the
> inflight count on the destination? If it is >1 you should be able to
> track it to a consumer that is probably *cached*.
>
Since the problem occures in production I'll try to add the parameter
*wireFormat.cacheEnabled=false *ASAP.
Cheers,
Philippe
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3739611.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: How does prefetch work with client acknoledgements?
Posted by "Martin C." <ma...@gmx.at>.
Hi,
On Wed, Aug 10, 2011 at 6:21 PM, pwanner <pw...@pwanner.com> wrote:
> Hi,
>
> We have also an other broker (same very simple config master/slave on SAN)
> for an other app, with a queue and only one consumer that is configured with
> Camel (see config below), and this consumer also stop consuming messages,
> thus they say stuck in the queue.
We (hopefully) had a similar issue with consumers stopping
consumption. Additionally we had a problem with connections to
localhost broker failing, with a stacktrace indicating that decoding
of the packed message failed. Therefore we switched to connect with
"...&wireFormat.cacheEnabled=false&wireFormat.tightEncodingEnabled=false"
in order to disable tight encoding and caching. Since we did this last
week, no more outages and oddly also the blocked consumers issue went
away. I still hope that this may have magically fixed our problems.
You might want to give this a try.
Best regards,
Martin
Re: How does prefetch work with client acknoledgements?
Posted by pwanner <pw...@pwanner.com>.
Hi,
We have also an other broker (same very simple config master/slave on SAN)
for an other app, with a queue and only one consumer that is configured with
Camel (see config below), and this consumer also stop consuming messages,
thus they say stuck in the queue.
I used pretty much all the options I could find to avoid this problem but
with no luck:
jms.prefetchPolicy.all=1
jms.dispatchAsync=false
jms.useAsyncSend=false
So two differents type consumers (Spring DMLC and Camel) but the same
problem... and no more ideas :-)
<beans
xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://camel.apache.org/schema/spring
http://camel.apache.org/schema/spring/camel-spring.xsd
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd">
<camelContext id="camel" xmlns="http://camel.apache.org/schema/spring">
<route>
<from uri="weblogic:queue:ToActiveMQ"/>
<transacted ref="PROPAGATION_REQUIRED"/>
<to uri="activemq:queue:from.weblogic"/>
</route>
<route>
<from uri="activemq:queue:to.weblogic"/>
<transacted ref="PROPAGATION_REQUIRED"/>
<to uri="weblogic:queue:FromActiveMQ"/>
</route>
</camelContext>
<bean id="PROPAGATION_REQUIRED"
class="org.apache.camel.spring.spi.SpringTransactionPolicy">
<property name="transactionManager" ref="activemqTransactionManager"/>
<property name="propagationBehaviorName" value="PROPAGATION_REQUIRED"/>
</bean>
<bean id="weblogic" class="org.apache.camel.component.jms.JmsComponent">
<property name="connectionFactory"
ref="weblogicCachedConnectionFactory"/>
<property name="destinationResolver" ref="weblogicDestinationResolver"/>
<property name="deliveryPersistent" value="true"/>
<property name="transacted" value="true"/>
</bean>
<bean id="weblogicDestinationResolver"
class="org.springframework.jms.support.destination.JndiDestinationResolver">
<property name="jndiTemplate" ref="remoteJndi" />
</bean>
<bean id="weblogicCachedConnectionFactory"
class="org.springframework.jms.connection.SingleConnectionFactory">
<property name="targetConnectionFactory"
ref="weblogicConnectionFactory"/>
<property name="reconnectOnException" value="true"/>
</bean>
<bean id="weblogicConnectionFactory"
class="org.springframework.jndi.JndiObjectFactoryBean">
<property name="jndiTemplate" ref="remoteJndi" />
<property name="jndiName" value="weblogic.jms.ConnectionFactory" />
</bean>
<bean id="remoteJndi" class="org.springframework.jndi.JndiTemplate">
<property name="environment">
<props>
<prop
key="java.naming.factory.initial">weblogic.jndi.WLInitialContextFactory</prop>
<prop key="java.naming.provider.url">t3://host:7004</prop>
</props>
</property>
</bean>
<bean id="activemqTransactionManager"
class="org.springframework.jms.connection.JmsTransactionManager">
<property name="connectionFactory" ref="amqCachedConnectionFactory" />
</bean>
<bean id="activemq"
class="org.apache.activemq.camel.component.ActiveMQComponent" >
<property name="connectionFactory" ref="amqCachedConnectionFactory"/>
<property name="transacted" value="true"/>
</bean>
<bean id="amqCachedConnectionFactory"
class="org.springframework.jms.connection.SingleConnectionFactory">
<property name="targetConnectionFactory" ref="amqConnectionFactory"/>
<property name="reconnectOnException" value="true"/>
</bean>
<bean id="amqConnectionFactory"
class="org.apache.activemq.ActiveMQConnectionFactory">
<property name="brokerURL"
value="failover://(tcp://host1:8016,tcp://host2:8016)?jms.prefetchPolicy.all=1&jms.dispatchAsync=false&jms.useAsyncSend=false&trackMessages=true&timeout=5000&jms.redeliveryPolicy.useExponentialBackOff=true&jms.maximumRedeliveryDelay=300000L&jms.redeliveryPolicy.maximumRedeliveries=-1"
/>
</bean>
</beans>
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3733449.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: How does prefetch work with client acknoledgements?
Posted by "lernen.2007" <er...@googlemail.com>.
Hi,
I think a solution in activemq version 5.6.0 is needed. I don't know what
the main problem is so that after a failover the messages stuck on the
server side. The activemq developer should know what going wrong.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3717861.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: How does prefetch work with client acknoledgements?
Posted by pwanner <pw...@pwanner.com>.
Hi Gary, thank you for the answer.
I have nothing yet to reproduce the problem.
But thank you for giving a tip.
According to your blog entry
http://blog.garytully.com/2010/01/activemq-prefetch-and-asyncdispatch.html
ActiveMQ prefetch , shoudn't the trackMessages property of the failover
transport always be set to true in an Shared File System Master Slave
configuration?
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3701385.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: How does prefetch work with client acknoledgements?
Posted by Gary Tully <ga...@gmail.com>.
does the message appear as inflight to any consumer? What is the
inflight count on the destination? If it is >1 you should be able to
track it to a consumer that is probably cached.
On 27 July 2011 10:24, pwanner <pw...@pwanner.com> wrote:
> Hi everybody,
>
> I just had a server crash today and the slave ActiveMQ instance went active,
> and a message is stuck in the queue. The new messages are consumed but there
> is one that stays.
>
> This is very annoying...
>
> I use ActiveMQ 5.5.0 (same problem with 5.4.2) and Spring 3.0.5 as the
> consumer.
>
> Below is the config:
>
> <jms:listener-container concurrency="5"
> destination-type="queue"
> connection-factory="amqCachedConnectionFactory"
> transaction-manager="transactionManager"
> acknowledge="transacted" >
> <jms:listener destination="events" ref="eventHandler" />
> </jms:listener-container>
>
> <bean id="amqCachedConnectionFactory"
> class="org.springframework.jms.connection.CachingConnectionFactory"
> p:targetConnectionFactory-ref="amqConnectionFactory"
> p:sessionCacheSize="5" />
>
> <amq:connectionFactory id="amqConnectionFactory"
>
> brokerURL="failover://(${broker.url.1},${broker.url.2})?jms.prefetchPolicy.all=1&timeout=5000"
> />
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3697960.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
--
http://fusesource.com
http://blog.garytully.com
Re: How does prefetch work with client acknoledgements?
Posted by pwanner <pw...@pwanner.com>.
Hi everybody,
I just had a server crash today and the slave ActiveMQ instance went active,
and a message is stuck in the queue. The new messages are consumed but there
is one that stays.
This is very annoying...
I use ActiveMQ 5.5.0 (same problem with 5.4.2) and Spring 3.0.5 as the
consumer.
Below is the config:
<jms:listener-container concurrency="5"
destination-type="queue"
connection-factory="amqCachedConnectionFactory"
transaction-manager="transactionManager"
acknowledge="transacted" >
<jms:listener destination="events" ref="eventHandler" />
</jms:listener-container>
<bean id="amqCachedConnectionFactory"
class="org.springframework.jms.connection.CachingConnectionFactory"
p:targetConnectionFactory-ref="amqConnectionFactory"
p:sessionCacheSize="5" />
<amq:connectionFactory id="amqConnectionFactory"
brokerURL="failover://(${broker.url.1},${broker.url.2})?jms.prefetchPolicy.all=1&timeout=5000"
/>
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3697960.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: How does prefetch work with client acknoledgements?
Posted by "lernen.2007" <er...@googlemail.com>.
Hi,
we have the problem with all the activemq versions.The messages stuck after
a failover.
Regards
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3675465.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: How does prefetch work with client acknoledgements?
Posted by "lernen.2007" <er...@googlemail.com>.
Hi,
thanks for answering. Here is our configuration on the client side:
<beans>
<bean
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="locations">
<value>value.properties</value>
</property>
</bean>
<broker xmlns="http://activemq.apache.org/schema/core"
brokerName="localhost"
useJmx="true"
destroyApplicationContextOnStop="true">
<networkConnectors>
<networkConnector
uri="static://(failover://(ssl://test.com:6861)?maxReconnectDelay=5000)"
networkTTL="10"
name="o">
<excludedDestinations>
<queue physicalName="a"/>
</excludedDestinations>
<staticallyIncludedDestinations>
<queue physicalName="b"/>
</staticallyIncludedDestinations>
</networkConnector>
</networkConnectors>
<persistenceAdapter>
<kahaDB directory="b" journalMaxFileLength="1mb"/>
</persistenceAdapter>
<systemUsage>
<systemUsage sendFailIfNoSpace="true">
<memoryUsage>
<memoryUsage limit="10mb" />
</memoryUsage>
<tempUsage>
<tempUsage limit="10mb" />
</tempUsage>
<storeUsage>
<storeUsage limit="100mb" percentUsageMinDelta="1" />
</storeUsage>
</systemUsage>
</systemUsage>
</broker>
</beans>
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3648087.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: How does prefetch work with client acknoledgements?
Posted by Gary Tully <ga...@gmail.com>.
can you add more detail to that jira entry, include your configuration
and verify that the sequence of steps to reproduce are still
accurate.
For a network connector using static discovery, if you use failover to
choose from a set of urls, ensure you use maxReconnectAttempts=1 such
that transport failures bubble up to the network connector. It is
important that the network connector handles network reconnection.
In your case, with a single url, just use static://ssl://test:61616):
On 5 July 2011 13:48, lernen.2007 <er...@googlemail.com> wrote:
> Here is a Jira-Entry:
>
> https://issues.apache.org/jira/browse/AMQ-2614
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3645856.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
--
http://fusesource.com
http://blog.garytully.com
Re: How does prefetch work with client acknoledgements?
Posted by "lernen.2007" <er...@googlemail.com>.
Here is a Jira-Entry:
https://issues.apache.org/jira/browse/AMQ-2614
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3645856.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: How does prefetch work with client acknoledgements?
Posted by "lernen.2007" <er...@googlemail.com>.
Hi,
I don't have any solution. The messages stuck further after a failover. Is
there any fix in version 5.6.0?
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3645806.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: How does prefetch work with client acknoledgements?
Posted by "lernen.2007" <er...@googlemail.com>.
Hi,
here is a Jira-Entry:
https://issues.apache.org/jira/browse/AMQ-2614
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3645854.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: How does prefetch work with client acknoledgements?
Posted by "lernen.2007" <er...@googlemail.com>.
Hi,
I set the following configuration so that not messages stuck:
networkConnector
uri="static://(failover://(ssl://test:61616))?maxReconnectDelay=5000"
networkTTL="10"
dynamicOnly="true"
prefetchSize="1"
name="outbox">
<excludedDestinations>
<queue physicalName="A"/>
</excludedDestinations>
<staticallyIncludedDestinations>
<queue physicalName="B"/>
</staticallyIncludedDestinations>
</networkConnector>
</networkConnectors>
But the failover doesn't work.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3587855.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: How does prefetch work with client acknoledgements?
Posted by "lernen.2007" <er...@googlemail.com>.
We have following situation:
Client Server
Server
ActiveMQ Instance1 ActiveMQ Instance2 <----
Bridge
NetworkConnector Transport Connector
Weblogic
The messages stuck further. Where all I must configure the prefetch size?
Please help.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3535464.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: How does prefetch work with client acknoledgements?
Posted by "lernen.2007" <er...@googlemail.com>.
Hi,
thanks for answer but we use a bridge ActiveMQ <--- Weblogic and I think in
this situation can the prefetch limit solution doesn't help further because
the consumer is the weblogic.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3528960.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: How does prefetch work with client acknoledgements?
Posted by "Martin C." <ma...@gmx.at>.
Hi,
Prefetching only applies when consuming messages.
Best regards,
Martin
On Mon, May 16, 2011 at 4:12 PM, lernen.2007
<er...@googlemail.com> wrote:
> I need your help.
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3526298.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: How does prefetch work with client acknoledgements?
Posted by "lernen.2007" <er...@googlemail.com>.
I need your help.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3526298.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: How does prefetch work with client acknoledgements?
Posted by "lernen.2007" <er...@googlemail.com>.
Should we configure "prefetch" on the producer side or on consumer side or in
both side.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3517057.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Messages stuck in queue
Posted by Gary Tully <ga...@gmail.com>.
you need the tests dependency:
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>activemq-core</artifactId>
<type>test-jar</type>
<scope>test</scope>
</dependency>
On 14 April 2011 10:32, lernen.2007 <er...@googlemail.com> wrote:
> I want to execute the test class
>
> BrokerNetworkWithStuckMessagesTest
>
>
> the following class are not found
>
> Wait
> StubConnection
>
>
> I bind the jar file activemq-all too and but it doesn't can bind this
> classes.
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3449171.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
--
http://blog.garytully.com
http://fusesource.com
Re: Messages stuck in queue
Posted by "lernen.2007" <er...@googlemail.com>.
I want to execute the test class
BrokerNetworkWithStuckMessagesTest
the following class are not found
Wait
StubConnection
I bind the jar file activemq-all too and but it doesn't can bind this
classes.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3449171.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
RE: Messages stuck in queue
Posted by nnprasad <NN...@dwd.IN.gov>.
Yes / No...because I am not sure..
I was also getting this Exception but not because of Illegal Argument Exception, But the Same Async Error Occured...it was because of
javax.jms.JMSException: Unmatched acknowledge: MessageAck {commandId = 2158, responseRequired = false, ackType = 2, consumerId = ID:dwdappq07pw-28582-1302110886144-1:5:1:1, firstMessageId = null, lastMessageId = ID:dwdappq05pw-14964-1302111394709-1:42:1:1:4420, destination = queue://UIMBATCHFNSaveQueueSTG, transactionId = null, messageCount = 1}; Expected message count (1) differs from count in dispatched-list (188)
And now in 5.4.2 version it was downgrade from ERROR level to WARN level.
but I can not say this might be the rason for queue blocking...
-Nag
________________________________
From: lernen.2007 [via ActiveMQ] [mailto:ml-node+3447558-1068315191-210129@n4.nabble.com]
Sent: Wednesday, April 13, 2011 11:57 AM
To: Nakarikanti, Nageswara
Subject: Re: Messages stuck in queue
Can this exception be the reason for stuck of messages:
Async error occurred: java.lang.IllegalArgumentException: The subscription
does not exist: ID:n11-1681-1159779699684-5:1:1:1
________________________________
If you reply to this email, your message will be added to the discussion below:
http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3447558.html
To unsubscribe from Messages stuck in queue, click here<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=3244342&code=Tk5ha2FyaWthbnRpQGR3ZC5JTi5nb3Z8MzI0NDM0Mnw2OTc1NTczNjk=>.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3447814.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Messages stuck in queue
Posted by "lernen.2007" <er...@googlemail.com>.
Can this exception be the reason for stuck of messages:
Async error occurred: java.lang.IllegalArgumentException: The subscription
does not exist: ID:n11-1681-1159779699684-5:1:1:1
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3447558.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Messages stuck in queue
Posted by nnprasad <NN...@dwd.IN.gov>.
But there a confusion in lernen.2007's previous reply message that he could
reproduce the problem
How can a producer get messages when you shutdown a consumer.
The description of the problem is "When a consumer is active and eagerly
looking for messages, AMQ queues won't deliver messages that were pending".
But here you are shutting down the consumer, and I think it is obvious that
the exception of "No subscription exists".
When there is no consumer who can receive the messages, they are obviously
in PENDING state.
If I am wrong...Please correct me.
Thank You,
Nag.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3447408.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Messages stuck in queue
Posted by Gary Tully <ga...@gmail.com>.
that is great progress, can you turn that into a junit test case?
In that way it will be obvious what the exact use case is, including
configuration and we can commit the test to protect any changes that
ensue.
There is an existing test case that may provide you with a good
template. If that does not make sense, there a lots of other related
tests, the names give hints as to what areas they validate.
http://svn.apache.org/viewvc/activemq/trunk/activemq-core/src/test/java/org/apache/activemq/network/BrokerNetworkWithStuckMessagesTest.java?view=markup
If you can provide a failing test, we can quickly help get to the root
cause because we will be able to easily reproduce.
On 13 April 2011 14:54, lernen.2007 <er...@googlemail.com> wrote:
> I could reproduce the problem. Following situation:
>
> Producer(Client) Consumer(Server)
> NetworkConnector
>
> I shutdown the consumer and the producer get messages further. If I start
> the connection between producer and consumer again then at first I see the
> following exception in activemq log:
>
> Async error occurred: java.lang.IllegalArgumentException: The subscription
> does not exist: ID:n11-1681-1159779699684-5:1:1:1
>
>
> and then some of the messages which deliver to the consumer stuck in queue.
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3447220.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
--
http://blog.garytully.com
http://fusesource.com
Re: Messages stuck in queue
Posted by "lernen.2007" <er...@googlemail.com>.
I could reproduce the problem. Following situation:
Producer(Client) Consumer(Server)
NetworkConnector
I shutdown the consumer and the producer get messages further. If I start
the connection between producer and consumer again then at first I see the
following exception in activemq log:
Async error occurred: java.lang.IllegalArgumentException: The subscription
does not exist: ID:n11-1681-1159779699684-5:1:1:1
and then some of the messages which deliver to the consumer stuck in queue.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3447220.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Messages stuck in queue
Posted by "lernen.2007" <er...@googlemail.com>.
I tested it further and it is only a problem if we send messages over gsm
with lan it works fine.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3444072.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Messages stuck in queue
Posted by "lernen.2007" <er...@googlemail.com>.
I test it with prefetch=0 and prefetch=1 and a lot of messages stuck further.
Can you say me which activemq version should I use that there no messages
stuck?
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3441189.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Messages stuck in queue
Posted by "lernen.2007" <er...@googlemail.com>.
I will try following:
and then I report the result if the messages stuck further or not.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3435510.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Messages stuck in queue
Posted by "lernen.2007" <er...@googlemail.com>.
The messages stuck not only if a active consumer get the messages. We have
the same effect by a active network connector too.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3433825.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: How does prefetch work with client acknoledgements?
Posted by "lernen.2007" <er...@googlemail.com>.
I think the activemq developer should stop to develop and search this bug. A
messaging system should save that every messages is delivery from producer
to consumer but not stuck in queue.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3508704.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: How does prefetch work with client acknoledgements?
Posted by Gary Tully <ga...@gmail.com>.
yes, the idea of prefetch is to deliver another message on or before
an ack such that it is available on the client before it is needed.
Some deviation is expected because a fast consumer will be able to ack
messages faster than the broker can dispatch them because it is doing
dispatch in batches of 1 when the prefetch is 1.
This is why we use a larger prefetch by default.
With prefetch=0, a client pulls a single message from the broker,
there is no prefetch but there is a round trip to the broker for each
pull request, so while the mean will be higher, the deviation will be
less.
To get the lowest mean and deviation and understand what is happening
further, preload with X messages and set the prefetch to X/N where N
is the number of consumers.
see: http://activemq.apache.org/what-is-the-prefetch-limit-for.html
On 4 May 2011 16:31, Josh Carlson <jc...@e-dialog.com> wrote:
> If I have prefetch set to 1 and I retrieve a message, is another one dispatched before it is ack'd? I'm trying to scale the number of consumers. My benchmark test pre-produces 50 messages for each consumer (all in one queue). Then sets N consumers active and I measure the time it takes to pull the message of the queue.
>
> In my benchmarking I've found a very nice MEAN time to retrieve a message at 2ms, regardless of the number of consumers. However, the deviation is very bad. My hypothesis is that some consumers are getting stuck not able to pull messages off the queue because each consumer can actually be dispatched 2 messages. Also the faster the consumer the worse the deviation.
>
--
http://blog.garytully.com
http://fusesource.com
How does prefetch work with client acknoledgements?
Posted by Josh Carlson <jc...@e-dialog.com>.
If I have prefetch set to 1 and I retrieve a message, is another one dispatched before it is ack'd? I'm trying to scale the number of consumers. My benchmark test pre-produces 50 messages for each consumer (all in one queue). Then sets N consumers active and I measure the time it takes to pull the message of the queue.
In my benchmarking I've found a very nice MEAN time to retrieve a message at 2ms, regardless of the number of consumers. However, the deviation is very bad. My hypothesis is that some consumers are getting stuck not able to pull messages off the queue because each consumer can actually be dispatched 2 messages. Also the faster the consumer the worse the deviation.
Re: Messages stuck in queue
Posted by "lernen.2007" <er...@googlemail.com>.
Have you a network connector? If yes did your network connector loss
sometimes the connection to the server broker?
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3495716.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Messages stuck in queue
Posted by Heimi <he...@heimi.net>.
We solved the problem by upgrading the server to 5.4.2.
Not running combination:
Solaris 10 Sparc
JDK 1.6.0_24
ActiveMQ 5.2.0
Running combination:
Solaris 10 Sparc
JDK 1.6.0_24
ActiveMQ 5.4.2
No more hickups in the last 2 days. *Knock on wood*
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3494993.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Messages stuck in queue
Posted by Gary Tully <ga...@gmail.com>.
there is a prefetchSize attribute for that on the NetworkConnector
xbean, inherited from
org.apache.activemq.network.NetworkBridgeConfiguration
On 3 May 2011 15:08, lernen.2007 <er...@googlemail.com> wrote:
> I think the problem is that the there is no prefetchSize option for
> networkconnector and that can be the reason why the messages stuck.
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3492540.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
--
http://blog.garytully.com
http://fusesource.com
Re: Messages stuck in queue
Posted by "lernen.2007" <er...@googlemail.com>.
I think the problem is that the there is no prefetchSize option for
networkconnector and that can be the reason why the messages stuck.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3492540.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Messages stuck in queue
Posted by "Martin C." <ma...@gmx.at>.
Hi,
we are using prefetch=0 with Spring, works well, still we are seeing
the same issue but for _non-persistent_ messages. (With prefetch=0 and
with prefetch=1.) Any ideas if this could correlate to the issues
mentioned here?
As an interesting side-effect, the consumption starts again if we
delete the queue with the stuck messages and/or purge it.
Could useCache=false help in this case as well? We are using a
file-based cursor:
<pendingQueuePolicy>
<fileQueueCursor />
</pendingQueuePolicy>
Best regards,
Martin
On Thu, Apr 7, 2011 at 5:49 PM, Gary Tully <ga...@gmail.com> wrote:
> one other thought is to use a prefetch=0 such that any cached
> consumers are out of the mix.
>
> On 7 April 2011 15:53, lernen.2007 <er...@googlemail.com> wrote:
>> The problem should be resolve as soon as possible. Otherwiese we must look
>> after an another messaging system.
>>
>> --
>> View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3433651.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
>
>
> --
> http://blog.garytully.com
> http://fusesource.com
>
Re: Messages stuck in queue
Posted by nnprasad <NN...@dwd.IN.gov>.
But in the link
http://activemq.apache.org/what-is-the-prefetch-limit-for.html it is stated
that....
"When using Spring JMS and MessageDrivenPojo, you cannot use a prefetch of
0, so use 1 instead." and we are using 1 for prefetch.
-Nag.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3433849.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Messages stuck in queue
Posted by Gary Tully <ga...@gmail.com>.
one other thought is to use a prefetch=0 such that any cached
consumers are out of the mix.
On 7 April 2011 15:53, lernen.2007 <er...@googlemail.com> wrote:
> The problem should be resolve as soon as possible. Otherwiese we must look
> after an another messaging system.
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3433651.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
--
http://blog.garytully.com
http://fusesource.com
Re: Messages stuck in queue
Posted by "lernen.2007" <er...@googlemail.com>.
The problem should be resolve as soon as possible. Otherwiese we must look
after an another messaging system.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3433651.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Messages stuck in queue
Posted by "lernen.2007" <er...@googlemail.com>.
I see only that the application try following:
011-04-07 15:14:14,780 | DEBUG | Executing SQL: DELETE FROM ACTIVEMQ_MSGS
WHERE ( EXPIRATION<>0 AND EXPIRATION<?) OR (ID <= ( SELECT
min(ACTIVEMQ_ACKS.LAST_ACKED_ID) FROM ACTIVEMQ_ACKS WHERE
ACTIVEMQ_ACKS.CONTAINER=ACTIVEMQ_MSGS.CONTAINER ) ) |
org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter | ActiveMQ Cleanup
Timer
2011-04-07 15:14:14,780 | DEBUG | Deleted 0 old message(s). |
org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter | ActiveMQ Cleanup
Timer
But the query return as result null.After we restart activemq then not all
but some of the messages deliver to the consumer. We had the following
situation:
1. 57 Messages stuck in queue --> web console shows that but if we want to
see in queue then there was no messages but in database we had all 57
messages.
2.We restart activemq and 54 of 57 deliver to the consumer and rest 3
messages stuck again.
3.But next restart the rest 3 messages delivered to consumer.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3433612.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Messages stuck in queue
Posted by Gary Tully <ga...@gmail.com>.
is there any correlation between the memory limit and the message
getting stuck. Can you remove the limit and see if you can reproduce
or increase the limit. Also, if you enable trace level logging and can
reproduce there may be some interesting information in the log. If so,
please share.
If you can reproduce the problem, can you share your test case?
On 7 April 2011 14:29, lernen.2007 <er...@googlemail.com> wrote:
> The same problem we have with the latest activemq version 5.0.0 too. Have you
> any idea how we can resolve this problem?
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-tp3244342p3433435.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
--
http://blog.garytully.com
http://fusesource.com