Re: Messages stuck in queue which are not expired

["Martin C." <ma...@gmx.at>]

Hello,

sorry for the late response, missed the message. We are currently very
confident that the issue was triggered because one of the client
applications was still using a very old 5.7 library to connect to the
5.12 broker. Once we updated the client library in that particular
application, things started to improve and we are currently testing if
upgrading everything across all clients and also the broker to 5.13
fixes this for good.

Once done, I'll post our test results.

Best regards,
Martin


On Sun, Dec 20, 2015 at 4:50 PM, Tim Bain <tb...@alumni.duke.edu> wrote:
> Martin, did you ever resolve this issue?
>
> If not, I'd recommend looking at the messages that expire to see if there
> is a pattern to them.
>
> Also, do you have a single broker, or a network of brokers?  If the latter,
> what is your networkTTL set to?
>
> Tim
> On Dec 2, 2015 9:50 AM, "Martin Carpella" <ma...@gmail.com> wrote:
>
>> Hi,
>>
>> We've ran into some problems since we updated to Activemq 5.12.1. Our
>> most busy queue has stuck messages which also do NOT expire.
>> The queue has around 200 producers (each producer has it's own message
>> group, making sure messages of a producer do not overtake each other)
>> which send non-persistent messages with a timeout of 40 seconds. They
>> produce around 20-30 msgs / second. 5 cached consumers exist.
>>
>> Our problem is that all 5 consumers are consuming messages but some of
>> those messages are apparently not delivered. They get stuck in the
>> queue and stay there. They do not expire.
>> The only solution to "clear" the queue is to use a QueueBrowser and
>> inspect it. Once I connect with the QueueBrowser, all messages are
>> apparently moved to expiration. After that the processing works for a
>> couple of minutes until the messages start clogging up again.
>>
>> The consumers do not use any form of selector other than the JMS
>> message group. The operation on the server side is very lightweight
>> and the load on the server is low so i do not think that it's the
>> fault of the server for not processing the messages fast enough (and
>> they should at least time out after their expiration deadline is
>> reached).
>> The problem scales apparently with the amount of the producers /
>> produced messages. Systems with ~100 producers have much fewer stuck
>> messages.
>>
>> All our other queues use message groups as well but work as intended.
>> A maybe noticable difference is that the messages that get stuck are
>> non-persistent and have a TTL. We have some high-throughput queues
>> with non-expiring, non-persistent messages, which do not show those
>> symptoms.
>>
>> Good ideas on what could be the issues are very welcome! Thanks in advance!
>>
>> Best regards,
>> Martin
>>

Re: Messages stuck in queue which are not expired

["Martin C." <ma...@gmx.at>]

Hi,

It's been quite some time now and I just wanted to follow up for
archiving purposes. Since we updated everything to 5.13, we don't see
the issue any more. I am convinced it had been caused by the 5.7
client library.

Best regards,
Martin

On Sat, Jan 9, 2016 at 6:24 AM, Tim Bain <tb...@alumni.duke.edu> wrote:
> Great, thanks for reporting back.
>
> Tim
>
> On Fri, Jan 8, 2016 at 1:04 PM, Martin C. <ma...@gmx.at> wrote:
>
>> Hello,
>>
>> sorry for the late response, missed the message. We are currently very
>> confident that the issue was triggered because one of the client
>> applications was still using a very old 5.7 library to connect to the
>> 5.12 broker. Once we updated the client library in that particular
>> application, things started to improve and we are currently testing if
>> upgrading everything across all clients and also the broker to 5.13
>> fixes this for good.
>>
>> Once done, I'll post our test results.
>>
>> Best regards,
>> Martin
>>
>>
>> On Sun, Dec 20, 2015 at 4:50 PM, Tim Bain <tb...@alumni.duke.edu> wrote:
>> > Martin, did you ever resolve this issue?
>> >
>> > If not, I'd recommend looking at the messages that expire to see if there
>> > is a pattern to them.
>> >
>> > Also, do you have a single broker, or a network of brokers?  If the
>> latter,
>> > what is your networkTTL set to?
>> >
>> > Tim
>> > On Dec 2, 2015 9:50 AM, "Martin Carpella" <ma...@gmail.com>
>> wrote:
>> >
>> >> Hi,
>> >>
>> >> We've ran into some problems since we updated to Activemq 5.12.1. Our
>> >> most busy queue has stuck messages which also do NOT expire.
>> >> The queue has around 200 producers (each producer has it's own message
>> >> group, making sure messages of a producer do not overtake each other)
>> >> which send non-persistent messages with a timeout of 40 seconds. They
>> >> produce around 20-30 msgs / second. 5 cached consumers exist.
>> >>
>> >> Our problem is that all 5 consumers are consuming messages but some of
>> >> those messages are apparently not delivered. They get stuck in the
>> >> queue and stay there. They do not expire.
>> >> The only solution to "clear" the queue is to use a QueueBrowser and
>> >> inspect it. Once I connect with the QueueBrowser, all messages are
>> >> apparently moved to expiration. After that the processing works for a
>> >> couple of minutes until the messages start clogging up again.
>> >>
>> >> The consumers do not use any form of selector other than the JMS
>> >> message group. The operation on the server side is very lightweight
>> >> and the load on the server is low so i do not think that it's the
>> >> fault of the server for not processing the messages fast enough (and
>> >> they should at least time out after their expiration deadline is
>> >> reached).
>> >> The problem scales apparently with the amount of the producers /
>> >> produced messages. Systems with ~100 producers have much fewer stuck
>> >> messages.
>> >>
>> >> All our other queues use message groups as well but work as intended.
>> >> A maybe noticable difference is that the messages that get stuck are
>> >> non-persistent and have a TTL. We have some high-throughput queues
>> >> with non-expiring, non-persistent messages, which do not show those
>> >> symptoms.
>> >>
>> >> Good ideas on what could be the issues are very welcome! Thanks in
>> advance!
>> >>
>> >> Best regards,
>> >> Martin
>> >>
>>

Re: Messages stuck in queue which are not expired

[Tim Bain <tb...@alumni.duke.edu>]

Great, thanks for reporting back.

Tim

On Fri, Jan 8, 2016 at 1:04 PM, Martin C. <ma...@gmx.at> wrote:

> Hello,
>
> sorry for the late response, missed the message. We are currently very
> confident that the issue was triggered because one of the client
> applications was still using a very old 5.7 library to connect to the
> 5.12 broker. Once we updated the client library in that particular
> application, things started to improve and we are currently testing if
> upgrading everything across all clients and also the broker to 5.13
> fixes this for good.
>
> Once done, I'll post our test results.
>
> Best regards,
> Martin
>
>
> On Sun, Dec 20, 2015 at 4:50 PM, Tim Bain <tb...@alumni.duke.edu> wrote:
> > Martin, did you ever resolve this issue?
> >
> > If not, I'd recommend looking at the messages that expire to see if there
> > is a pattern to them.
> >
> > Also, do you have a single broker, or a network of brokers?  If the
> latter,
> > what is your networkTTL set to?
> >
> > Tim
> > On Dec 2, 2015 9:50 AM, "Martin Carpella" <ma...@gmail.com>
> wrote:
> >
> >> Hi,
> >>
> >> We've ran into some problems since we updated to Activemq 5.12.1. Our
> >> most busy queue has stuck messages which also do NOT expire.
> >> The queue has around 200 producers (each producer has it's own message
> >> group, making sure messages of a producer do not overtake each other)
> >> which send non-persistent messages with a timeout of 40 seconds. They
> >> produce around 20-30 msgs / second. 5 cached consumers exist.
> >>
> >> Our problem is that all 5 consumers are consuming messages but some of
> >> those messages are apparently not delivered. They get stuck in the
> >> queue and stay there. They do not expire.
> >> The only solution to "clear" the queue is to use a QueueBrowser and
> >> inspect it. Once I connect with the QueueBrowser, all messages are
> >> apparently moved to expiration. After that the processing works for a
> >> couple of minutes until the messages start clogging up again.
> >>
> >> The consumers do not use any form of selector other than the JMS
> >> message group. The operation on the server side is very lightweight
> >> and the load on the server is low so i do not think that it's the
> >> fault of the server for not processing the messages fast enough (and
> >> they should at least time out after their expiration deadline is
> >> reached).
> >> The problem scales apparently with the amount of the producers /
> >> produced messages. Systems with ~100 producers have much fewer stuck
> >> messages.
> >>
> >> All our other queues use message groups as well but work as intended.
> >> A maybe noticable difference is that the messages that get stuck are
> >> non-persistent and have a TTL. We have some high-throughput queues
> >> with non-expiring, non-persistent messages, which do not show those
> >> symptoms.
> >>
> >> Good ideas on what could be the issues are very welcome! Thanks in
> advance!
> >>
> >> Best regards,
> >> Martin
> >>
>