You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Rick Blair <ri...@boeing.com> on 2008/10/11 18:35:44 UTC
Problem with losing acks(?)
Hi all,
I am using ActiveMQ version 5.1 in a pub/sub system. The broker is embedded
with the publishers. The publishers publish to 2 main topics with
selectors. I have many subscribers distributed on the network. The
subscriptions are based on the selectors. The messages are object messages.
During execution one subscriber(not always the same one) seems to stop
sending acks.
In the subscriber I still see the onMessage method call being called.
However on the broker, using jconsole looking at the subscription, the
number of messages waiting for ack = the prefetch size and the number of
queued messages increases.
This goes on until the broker memory limit hits 100% and all message flow
stops.
We have tried to set the policies that discard older messages, but that did
not help. The message discard count does increase to a point, then remains
constant, while the enqueue count still increases. Also at this time many
of the stats go negative. This very often occurs right after startup under
heavy load on the producers. Thread dumps show no threads blocked, the CPU
is not heavily loaded. GC does not seem to an issue.
Any suggestions?
--
Tìoraidh!
Rick
Re: Problem with losing acks(?)
Posted by Gary Tully <ga...@gmail.com>.
Would it be possible to verify this behavior with trunk or with the
5.2 release candidate?
http://www.nabble.com/-VOTE--ActiveMQ-5.2.0---RC2-td19874910.html
If it is still an issue it warrants a jira and if you have a test
case, it would be fantastic!
thanks,
Gary.
2008/10/14 Rick Blair <ri...@boeing.com>:
> Hi all,
>
> A little more information.
>
> I looked at the acks that were being sent back from the subscriber. Under
> heavy load the messageCount field in the ack message sometimes goes
> negative. Once that happens, it is not long before the subscriber quits
> sending acks. I have tried small and large values for
> prefetchSize(100,100,10000) but it makes no difference.
>
> I have worked around this problem by setting optimizeAcknowlege to false.
>
> --
> Tìoraidh!
>
> Rick Blair
>
>
>
>
>> From: Joe Fernandez <jo...@ttmsolutions.com>
>> Reply-To: <us...@activemq.apache.org>
>> Date: Sun, 12 Oct 2008 18:17:05 -0700 (PDT)
>> To: <us...@activemq.apache.org>
>> Subject: Re: Problem with losing acks(?)
>>
>>
>> When creating sessions for your subscribers, what ack mode (AUTO, CLIENT, or
>> DUPS_OK) are you using and are your sessions transacted?
>>
>> Joe
>> Get a free ActiveMQ user guide @ http://www.ttmsolutions.com
>>
>>
>> Rick Blair wrote:
>>>
>>> Hi all,
>>>
>>> I am using ActiveMQ version 5.1 in a pub/sub system. The broker is
>>> embedded
>>> with the publishers. The publishers publish to 2 main topics with
>>> selectors. I have many subscribers distributed on the network. The
>>> subscriptions are based on the selectors. The messages are object
>>> messages.
>>>
>>> During execution one subscriber(not always the same one) seems to stop
>>> sending acks.
>>>
>>> In the subscriber I still see the onMessage method call being called.
>>> However on the broker, using jconsole looking at the subscription, the
>>> number of messages waiting for ack = the prefetch size and the number of
>>> queued messages increases.
>>>
>>> This goes on until the broker memory limit hits 100% and all message
>>> flow
>>> stops.
>>>
>>> We have tried to set the policies that discard older messages, but that
>>> did
>>> not help. The message discard count does increase to a point, then
>>> remains
>>> constant, while the enqueue count still increases. Also at this time many
>>> of the stats go negative. This very often occurs right after startup
>>> under
>>> heavy load on the producers. Thread dumps show no threads blocked, the
>>> CPU
>>> is not heavily loaded. GC does not seem to an issue.
>>>
>>> Any suggestions?
>>>
>>> --
>>> Tìoraidh!
>>>
>>> Rick
>>>
>>>
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Problem-with-losing-acks%28-%29-tp19934087p19947672.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
>
Re: Problem with losing acks(?)
Posted by Rick Blair <ri...@boeing.com>.
Hi all,
A little more information.
I looked at the acks that were being sent back from the subscriber. Under
heavy load the messageCount field in the ack message sometimes goes
negative. Once that happens, it is not long before the subscriber quits
sending acks. I have tried small and large values for
prefetchSize(100,100,10000) but it makes no difference.
I have worked around this problem by setting optimizeAcknowlege to false.
--
Tìoraidh!
Rick Blair
> From: Joe Fernandez <jo...@ttmsolutions.com>
> Reply-To: <us...@activemq.apache.org>
> Date: Sun, 12 Oct 2008 18:17:05 -0700 (PDT)
> To: <us...@activemq.apache.org>
> Subject: Re: Problem with losing acks(?)
>
>
> When creating sessions for your subscribers, what ack mode (AUTO, CLIENT, or
> DUPS_OK) are you using and are your sessions transacted?
>
> Joe
> Get a free ActiveMQ user guide @ http://www.ttmsolutions.com
>
>
> Rick Blair wrote:
>>
>> Hi all,
>>
>> I am using ActiveMQ version 5.1 in a pub/sub system. The broker is
>> embedded
>> with the publishers. The publishers publish to 2 main topics with
>> selectors. I have many subscribers distributed on the network. The
>> subscriptions are based on the selectors. The messages are object
>> messages.
>>
>> During execution one subscriber(not always the same one) seems to stop
>> sending acks.
>>
>> In the subscriber I still see the onMessage method call being called.
>> However on the broker, using jconsole looking at the subscription, the
>> number of messages waiting for ack = the prefetch size and the number of
>> queued messages increases.
>>
>> This goes on until the broker memory limit hits 100% and all message
>> flow
>> stops.
>>
>> We have tried to set the policies that discard older messages, but that
>> did
>> not help. The message discard count does increase to a point, then
>> remains
>> constant, while the enqueue count still increases. Also at this time many
>> of the stats go negative. This very often occurs right after startup
>> under
>> heavy load on the producers. Thread dumps show no threads blocked, the
>> CPU
>> is not heavily loaded. GC does not seem to an issue.
>>
>> Any suggestions?
>>
>> --
>> Tìoraidh!
>>
>> Rick
>>
>>
>>
>
> --
> View this message in context:
> http://www.nabble.com/Problem-with-losing-acks%28-%29-tp19934087p19947672.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Problem with losing acks(?)
Posted by Rick Blair <ri...@boeing.com>.
Hi Joe
Ack mode is auto and the sessions are not transacted.
Thanks
--
Tìoraidh!
Rick Blair
> From: Joe Fernandez <jo...@ttmsolutions.com>
> Reply-To: <us...@activemq.apache.org>
> Date: Sun, 12 Oct 2008 18:17:05 -0700 (PDT)
> To: <us...@activemq.apache.org>
> Subject: Re: Problem with losing acks(?)
>
>
> When creating sessions for your subscribers, what ack mode (AUTO, CLIENT, or
> DUPS_OK) are you using and are your sessions transacted?
>
> Joe
> Get a free ActiveMQ user guide @ http://www.ttmsolutions.com
>
>
> Rick Blair wrote:
>>
>> Hi all,
>>
>> I am using ActiveMQ version 5.1 in a pub/sub system. The broker is
>> embedded
>> with the publishers. The publishers publish to 2 main topics with
>> selectors. I have many subscribers distributed on the network. The
>> subscriptions are based on the selectors. The messages are object
>> messages.
>>
>> During execution one subscriber(not always the same one) seems to stop
>> sending acks.
>>
>> In the subscriber I still see the onMessage method call being called.
>> However on the broker, using jconsole looking at the subscription, the
>> number of messages waiting for ack = the prefetch size and the number of
>> queued messages increases.
>>
>> This goes on until the broker memory limit hits 100% and all message
>> flow
>> stops.
>>
>> We have tried to set the policies that discard older messages, but that
>> did
>> not help. The message discard count does increase to a point, then
>> remains
>> constant, while the enqueue count still increases. Also at this time many
>> of the stats go negative. This very often occurs right after startup
>> under
>> heavy load on the producers. Thread dumps show no threads blocked, the
>> CPU
>> is not heavily loaded. GC does not seem to an issue.
>>
>> Any suggestions?
>>
>> --
>> Tìoraidh!
>>
>> Rick
>>
>>
>>
>
> --
> View this message in context:
> http://www.nabble.com/Problem-with-losing-acks%28-%29-tp19934087p19947672.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Problem with losing acks(?)
Posted by Joe Fernandez <jo...@ttmsolutions.com>.
When creating sessions for your subscribers, what ack mode (AUTO, CLIENT, or
DUPS_OK) are you using and are your sessions transacted?
Joe
Get a free ActiveMQ user guide @ http://www.ttmsolutions.com
Rick Blair wrote:
>
> Hi all,
>
> I am using ActiveMQ version 5.1 in a pub/sub system. The broker is
> embedded
> with the publishers. The publishers publish to 2 main topics with
> selectors. I have many subscribers distributed on the network. The
> subscriptions are based on the selectors. The messages are object
> messages.
>
> During execution one subscriber(not always the same one) seems to stop
> sending acks.
>
> In the subscriber I still see the onMessage method call being called.
> However on the broker, using jconsole looking at the subscription, the
> number of messages waiting for ack = the prefetch size and the number of
> queued messages increases.
>
> This goes on until the broker memory limit hits 100% and all message
> flow
> stops.
>
> We have tried to set the policies that discard older messages, but that
> did
> not help. The message discard count does increase to a point, then
> remains
> constant, while the enqueue count still increases. Also at this time many
> of the stats go negative. This very often occurs right after startup
> under
> heavy load on the producers. Thread dumps show no threads blocked, the
> CPU
> is not heavily loaded. GC does not seem to an issue.
>
> Any suggestions?
>
> --
> Tìoraidh!
>
> Rick
>
>
>
--
View this message in context: http://www.nabble.com/Problem-with-losing-acks%28-%29-tp19934087p19947672.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.