You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by matousvoldrich <ma...@voldrich.net> on 2013/11/28 15:32:14 UTC
ActiveMq 5.9 stuck messages on store full
Hi all,
I have following scenario:
1. SMTP server receives email, creates Java class representation and puts it
into EMAIL queue.
2. Queue is persistant, using configuration below.
3. I have 10+ consumers consuming from EMAIL queue in transaction mode. If
they fail to process email (route it to other system), message is reverted.
Retry scheme is to try again every 2 minutes indefinitelly (for test
purposes).
The problem starts when ActiveMq store is filled up (outage on network,
consumers are not able to process messages and revert all of them). Producer
on SMTP server gets blocked, which is fine.
My problem is that once network problem is resolved and consumers start to
handle the queue again, not all messages are processed and always a few
thousands are left behind and never consumed. After I restart ActiveMq the
messages are finally processed and consumed.
Is this a bug on ActiveMq? Or is this a configuration issue?
I tried to experiment with
<policyEntry queue="EMAIL" storeUsageHighWaterMark="95" />
But no luck, behaves the same
Thanks for response
Matous
My configuration:
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" producerFlowControl="true"
memoryLimit="16 mb" >
<pendingMessageLimitStrategy>
<constantPendingMessageLimitStrategy limit="1000"/>
</pendingMessageLimitStrategy>
</policyEntry>
<policyEntry queue=">" producerFlowControl="true"
memoryLimit="16 mb" />
<policyEntry queue="EMAIL" storeUsageHighWaterMark="95" />
<policyEntry queue="ActiveMQ.DLQ"
storeUsageHighWaterMark="20" />
</policyEntries>
</policyMap>
</destinationPolicy>
<systemUsage>
<systemUsage sendFailIfNoSpaceAfterTimeout="1000">
<memoryUsage>
<memoryUsage percentOfJvmHeap="70" />
</memoryUsage>
<storeUsage>
<storeUsage limit="33 mb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="33 mb"/>
</tempUsage>
</systemUsage>
</systemUsage>
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMq-5-9-stuck-messages-on-store-full-tp4674917.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMq 5.9 stuck messages on store full
Posted by matousvoldrich <ma...@voldrich.net>.
So the problem disappeared after I set the queue prefetch to one on consumer
connection factory:
ActiveMQPrefetchPolicy prefetchPolicy =
connectionFactory.getPrefetchPolicy();
prefetchPolicy.setQueuePrefetch(1);
connectionFactory.setPrefetchPolicy(prefetchPolicy);
So my problem is solved and all messages are now properly delivered. But I
still wonder whether this is a bug in ActiveMq?
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMq-5-9-stuck-messages-on-store-full-tp4674917p4674985.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMq 5.9 stuck messages on store full
Posted by matousvoldrich <ma...@voldrich.net>.
After some more investigation and testing it seems that I have another
problem here. All the messages are now properly delivered, but failed
retries seems to block consumers untill the message is delivered or out of
retries and consumers do not handle new messages. I thought that setting
setNonBlockingRedelivery(true);
should solve this issue, but its not working.
I need that retries do not block other messages.
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMq-5-9-stuck-messages-on-store-full-tp4674917p4674986.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMq 5.9 stuck messages on store full
Posted by matousvoldrich <ma...@voldrich.net>.
Hi, thanks for answer. I will try the configuration you proposed later. But
today I noticed that the number of messages left in the queue is exactly the
same as the number of messages dispatched to consumers.
5056 messages left in queue
when I list queue connections from
http://localhost:8161/admin/queueConsumers.jsp?
And make sum of make a sum of "Dispatched Queue", I get the exactly same
number! 5056
I will try to set the prefetch limit to minimum and test again, but that
will probably lower performance significantly. Could this be a fault at our
consumer implementation?
We are using
connectionFactory.setNonBlockingRedelivery(true);
connectionFactory.setRedeliveryPolicy(policy);
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMq-5-9-stuck-messages-on-store-full-tp4674917p4674955.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMq 5.9 stuck messages on store full
Posted by Gary Tully <ga...@gmail.com>.
that does seem odd. Seems like it should be possible to reproduce in a
test case so we can easily investigate what is going wrong?
I came across an issue with
transactions/concurrentStoreAndDispatchQueues and memory limits that
is sort of related in the past while.
<kahaDB concurrentStoreAndDispatchQueues="false" ..> provides a
workaround in my case.
But the scenario required concurrent transactional and non
transactional producers on a memory limit, the test case is
nontrivial.
Maybe try that as a sanity, and then try and make a unit test. If you
can easily reproduce, please raise a jira issue.
On 28 November 2013 14:32, matousvoldrich <ma...@voldrich.net> wrote:
> Hi all,
>
> I have following scenario:
> 1. SMTP server receives email, creates Java class representation and puts it
> into EMAIL queue.
> 2. Queue is persistant, using configuration below.
> 3. I have 10+ consumers consuming from EMAIL queue in transaction mode. If
> they fail to process email (route it to other system), message is reverted.
> Retry scheme is to try again every 2 minutes indefinitelly (for test
> purposes).
>
> The problem starts when ActiveMq store is filled up (outage on network,
> consumers are not able to process messages and revert all of them). Producer
> on SMTP server gets blocked, which is fine.
>
> My problem is that once network problem is resolved and consumers start to
> handle the queue again, not all messages are processed and always a few
> thousands are left behind and never consumed. After I restart ActiveMq the
> messages are finally processed and consumed.
>
> Is this a bug on ActiveMq? Or is this a configuration issue?
>
> I tried to experiment with
> <policyEntry queue="EMAIL" storeUsageHighWaterMark="95" />
> But no luck, behaves the same
>
> Thanks for response
> Matous
>
> My configuration:
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <policyEntry topic=">" producerFlowControl="true"
> memoryLimit="16 mb" >
>
> <pendingMessageLimitStrategy>
> <constantPendingMessageLimitStrategy limit="1000"/>
> </pendingMessageLimitStrategy>
> </policyEntry>
> <policyEntry queue=">" producerFlowControl="true"
> memoryLimit="16 mb" />
> <policyEntry queue="EMAIL" storeUsageHighWaterMark="95" />
> <policyEntry queue="ActiveMQ.DLQ"
> storeUsageHighWaterMark="20" />
> </policyEntries>
> </policyMap>
> </destinationPolicy>
>
> <systemUsage>
> <systemUsage sendFailIfNoSpaceAfterTimeout="1000">
> <memoryUsage>
> <memoryUsage percentOfJvmHeap="70" />
> </memoryUsage>
> <storeUsage>
> <storeUsage limit="33 mb"/>
> </storeUsage>
> <tempUsage>
> <tempUsage limit="33 mb"/>
> </tempUsage>
> </systemUsage>
> </systemUsage>
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMq-5-9-stuck-messages-on-store-full-tp4674917.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
--
http://redhat.com
http://blog.garytully.com