You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Balamurugan Seenivasan <to...@gmail.com> on 2021/09/29 07:23:32 UTC
[AMQ] Queue size grows infinitely
Hi,
We notice an issue recently in production where the messages in queue size grows disproportionate to the number of messages actually produced.
Queue size grows close to 3 million in about 3 days while the actual messages produced are way less than the actual queue size.
Here is one snapshot of how the queue looks like when actual messages produced are only around 60000.
794712 (pending) 8(consumers) 679793(enqueued) 507419(dequeued)
We use browseMessages extensively (from around 50 threads) using selectors like 'JMSCorrelationID LIKE <data>%'.
We notice this issue only with those queues we use browseMessages against.
Not sure if there is a link to usage of browseMessages.
AMQ version: 5.16.3 (windows)
Persistence:
Yes (Microsoft SQL Server 2019 on Linux (Ubuntu 16.04.6 LTS) <X64>, mssql-jdbc-9.4)
config:
<persistenceAdapter>
<jdbcPersistenceAdapter dataDirectory="activemq-data" dataSource="#mssql-ds" lockKeepAlivePeriod="5000">
<adapter><transact-jdbc-adapter/></adapter>
<locker>
<lease-database-locker lockAcquireSleepInterval="10000"/>
</locker>
</jdbcPersistenceAdapter>
</persistenceAdapter>
Appreciate any help/pointers narrowing down the cause.
Re: [AMQ] Queue size grows infinitely
Posted by Tim Bain <tb...@alumni.duke.edu>.
How are you determining the counts you referenced? I'd expect you could get
them from the web console or by doing SQL queries against the database, and
I'd encourage you to do both and compare the numbers.
I'm hoping you'll find that the stats from the web console (which are
sourced from the JMX beans) are wrong but the database is right, since that
presents a problem for monitoring but not message distribution. But knowing
the symptoms will help identify where in the code we should be looking for
the problem in order to fix it.
Tim
On Wed, Sep 29, 2021, 6:24 AM Balamurugan Seenivasan <to...@gmail.com>
wrote:
> Hi,
>
> We notice an issue recently in production where the messages in queue size
> grows disproportionate to the number of messages actually produced.
> Queue size grows close to 3 million in about 3 days while the actual
> messages produced are way less than the actual queue size.
>
> Here is one snapshot of how the queue looks like when actual messages
> produced are only around 60000.
> 794712 (pending) 8(consumers) 679793(enqueued) 507419(dequeued)
>
> We use browseMessages extensively (from around 50 threads) using selectors
> like 'JMSCorrelationID LIKE <data>%'.
> We notice this issue only with those queues we use browseMessages against.
> Not sure if there is a link to usage of browseMessages.
>
> AMQ version: 5.16.3 (windows)
> Persistence:
> Yes (Microsoft SQL Server 2019 on Linux (Ubuntu 16.04.6 LTS) <X64>,
> mssql-jdbc-9.4)
> config:
> <persistenceAdapter>
> <jdbcPersistenceAdapter
> dataDirectory="activemq-data" dataSource="#mssql-ds"
> lockKeepAlivePeriod="5000">
> <adapter><transact-jdbc-adapter/></adapter>
> <locker>
> <lease-database-locker
> lockAcquireSleepInterval="10000"/>
> </locker>
> </jdbcPersistenceAdapter>
> </persistenceAdapter>
>
> Appreciate any help/pointers narrowing down the cause.
>
Re: [AMQ] Queue size grows infinitely
Posted by Balamurugan Seenivasan <to...@gmail.com>.
HI,
Thanks for your reply.
Here is one snapshot from today.
426801(pending) 8 (consumers) 609146 (enqueued) 182345(dequeued)
As per this stats, We seem to have about 426801 pending messages (same number of records exist in DB too) and produced more than 600K messages while we only actually produced 10% of what's in the queue.
Can you please elaborate on your point "BrowsingMessages can "lock" the messages, blocking the consumers"?
Will it affect the "in-flight" message acknowledgements?
Regards,
Bala
On 2021/10/01 12:22:53, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> Hi,
>
> enqueued and dequeued only increase.
>
> The main indicator to check is the pending.
>
> If pending count is increasing, it means that the queue size grow
> infinitely. enqueued is the number of produced messages (not necessary
> still in the queue). So, which value are you referencing ?
> BrowsingMessages can "lock" the messages, blocking the consumers
> (depending the way you do it).
>
> Regards
> JB
>
> On 29/09/2021 09:23, Balamurugan Seenivasan wrote:
> > Hi,
> >
> > We notice an issue recently in production where the messages in queue size grows disproportionate to the number of messages actually produced.
> > Queue size grows close to 3 million in about 3 days while the actual messages produced are way less than the actual queue size.
> >
> > Here is one snapshot of how the queue looks like when actual messages produced are only around 60000.
> > 794712 (pending) 8(consumers) 679793(enqueued) 507419(dequeued)
> >
> > We use browseMessages extensively (from around 50 threads) using selectors like 'JMSCorrelationID LIKE <data>%'.
> > We notice this issue only with those queues we use browseMessages against.
> > Not sure if there is a link to usage of browseMessages.
> >
> > AMQ version: 5.16.3 (windows)
> > Persistence:
> > Yes (Microsoft SQL Server 2019 on Linux (Ubuntu 16.04.6 LTS) <X64>, mssql-jdbc-9.4)
> > config:
> > <persistenceAdapter>
> > <jdbcPersistenceAdapter dataDirectory="activemq-data" dataSource="#mssql-ds" lockKeepAlivePeriod="5000">
> > <adapter><transact-jdbc-adapter/></adapter>
> > <locker>
> > <lease-database-locker lockAcquireSleepInterval="10000"/>
> > </locker>
> > </jdbcPersistenceAdapter>
> > </persistenceAdapter>
> >
> > Appreciate any help/pointers narrowing down the cause.
> >
>
Re: [AMQ] Queue size grows infinitely
Posted by Balamurugan Seenivasan <to...@gmail.com>.
HI,
Thanks for your reply.
We started to see the queue size growing in production recently.
Following are some of the things we have tried so far
We initially were on AMQ version 5.15.10 and JDBC driver version 4.2.
We upgraded both AMQ as well as JDBC driver to the latest (5.16.3 and mssql-jdbc-9.4 respectively).
We also tried KahaDB persistence instead of JDBC.
We also counted the actual number of messages that we produce from our side.
As I mentioned in the initial report, that in one case we measured, the actual numbers that we produced were around 60,000 whereas the queue size was 10 times more than that.
We also came across this issue https://issues.apache.org/jira/browse/AMQ-7425 which is similar to what we experience.
Though upgrading to the latest didn't help, our issues are limited to only those queues that we use BrowseMessages against.
This is the part I am investigating right now.
Regards,
Bala
On 2021/10/05 05:39:38, JB Onofré <jb...@nanthrax.net> wrote:
> Can you elaborate a bit your analyze and test case ?
>
> Thanks
> Regards
> JB
>
> > Le 5 oct. 2021 à 07:20, Balamurugan Seenivasan <to...@gmail.com> a écrit :
> >
> > Hi,
> >
> > Thanks for your reply.
> >
> > This issue exists for us even if we start out with a clean environment (with a brand new database).
> > We even noticed this issue with KahaDB too.
> > This is consistently reproducible for us.
> >
> > Regards,
> > Bala
> >
> >> On 2021/10/01 17:56:07, Matt Pavlovich <ma...@gmail.com> wrote:
> >> Hi Balamurugan-
> >>
> >> Is the broker restarted at any time in this sequence of events?
> >>
> >>
> >> Keep in mind—
> >>
> >> Pending Count can be higher than Enqueue Count when the broker is restarted and messages remained on the queue.
> >>
> >> a. Enqueue and Dequeue counts reset are for that broker’s uptime.
> >>
> >> b. Pending Count (aka queueSize) is how many messages are currently in the store.
> >>
> >> -Matt Pavlovich
> >>
> >>>> On Oct 1, 2021, at 7:29 AM, Tim Bain <tb...@alumni.duke.edu> wrote:
> >>>
> >>> The fact that pending messages > enqueued messages looks suspicious, so my
> >>> expectation is that this will turn out to be a bug in the statistics code.
> >>> But the information I requested may help us to confirm that.
> >>>
> >>> Tim
> >>>
> >>>> On Fri, Oct 1, 2021, 6:23 AM Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> enqueued and dequeued only increase.
> >>>>
> >>>> The main indicator to check is the pending.
> >>>>
> >>>> If pending count is increasing, it means that the queue size grow
> >>>> infinitely. enqueued is the number of produced messages (not necessary
> >>>> still in the queue). So, which value are you referencing ?
> >>>> BrowsingMessages can "lock" the messages, blocking the consumers
> >>>> (depending the way you do it).
> >>>>
> >>>> Regards
> >>>> JB
> >>>>
> >>>> On 29/09/2021 09:23, Balamurugan Seenivasan wrote:
> >>>>> Hi,
> >>>>>
> >>>>> We notice an issue recently in production where the messages in queue
> >>>> size grows disproportionate to the number of messages actually produced.
> >>>>> Queue size grows close to 3 million in about 3 days while the actual
> >>>> messages produced are way less than the actual queue size.
> >>>>>
> >>>>> Here is one snapshot of how the queue looks like when actual messages
> >>>> produced are only around 60000.
> >>>>> 794712 (pending) 8(consumers) 679793(enqueued) 507419(dequeued)
> >>>>>
> >>>>> We use browseMessages extensively (from around 50 threads) using
> >>>> selectors like 'JMSCorrelationID LIKE <data>%'.
> >>>>> We notice this issue only with those queues we use browseMessages
> >>>> against.
> >>>>> Not sure if there is a link to usage of browseMessages.
> >>>>>
> >>>>> AMQ version: 5.16.3 (windows)
> >>>>> Persistence:
> >>>>> Yes (Microsoft SQL Server 2019 on Linux (Ubuntu 16.04.6 LTS) <X64>,
> >>>> mssql-jdbc-9.4)
> >>>>> config:
> >>>>> <persistenceAdapter>
> >>>>> <jdbcPersistenceAdapter
> >>>> dataDirectory="activemq-data" dataSource="#mssql-ds"
> >>>> lockKeepAlivePeriod="5000">
> >>>>>
> >>>> <adapter><transact-jdbc-adapter/></adapter>
> >>>>> <locker>
> >>>>> <lease-database-locker
> >>>> lockAcquireSleepInterval="10000"/>
> >>>>> </locker>
> >>>>> </jdbcPersistenceAdapter>
> >>>>> </persistenceAdapter>
> >>>>>
> >>>>> Appreciate any help/pointers narrowing down the cause.
> >>>>>
> >>>>
> >>
> >>
>
>
Re: [AMQ] Queue size grows infinitely
Posted by JB Onofré <jb...@nanthrax.net>.
Can you elaborate a bit your analyze and test case ?
Thanks
Regards
JB
> Le 5 oct. 2021 à 07:20, Balamurugan Seenivasan <to...@gmail.com> a écrit :
>
> Hi,
>
> Thanks for your reply.
>
> This issue exists for us even if we start out with a clean environment (with a brand new database).
> We even noticed this issue with KahaDB too.
> This is consistently reproducible for us.
>
> Regards,
> Bala
>
>> On 2021/10/01 17:56:07, Matt Pavlovich <ma...@gmail.com> wrote:
>> Hi Balamurugan-
>>
>> Is the broker restarted at any time in this sequence of events?
>>
>>
>> Keep in mind—
>>
>> Pending Count can be higher than Enqueue Count when the broker is restarted and messages remained on the queue.
>>
>> a. Enqueue and Dequeue counts reset are for that broker’s uptime.
>>
>> b. Pending Count (aka queueSize) is how many messages are currently in the store.
>>
>> -Matt Pavlovich
>>
>>>> On Oct 1, 2021, at 7:29 AM, Tim Bain <tb...@alumni.duke.edu> wrote:
>>>
>>> The fact that pending messages > enqueued messages looks suspicious, so my
>>> expectation is that this will turn out to be a bug in the statistics code.
>>> But the information I requested may help us to confirm that.
>>>
>>> Tim
>>>
>>>> On Fri, Oct 1, 2021, 6:23 AM Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>>>
>>>> Hi,
>>>>
>>>> enqueued and dequeued only increase.
>>>>
>>>> The main indicator to check is the pending.
>>>>
>>>> If pending count is increasing, it means that the queue size grow
>>>> infinitely. enqueued is the number of produced messages (not necessary
>>>> still in the queue). So, which value are you referencing ?
>>>> BrowsingMessages can "lock" the messages, blocking the consumers
>>>> (depending the way you do it).
>>>>
>>>> Regards
>>>> JB
>>>>
>>>> On 29/09/2021 09:23, Balamurugan Seenivasan wrote:
>>>>> Hi,
>>>>>
>>>>> We notice an issue recently in production where the messages in queue
>>>> size grows disproportionate to the number of messages actually produced.
>>>>> Queue size grows close to 3 million in about 3 days while the actual
>>>> messages produced are way less than the actual queue size.
>>>>>
>>>>> Here is one snapshot of how the queue looks like when actual messages
>>>> produced are only around 60000.
>>>>> 794712 (pending) 8(consumers) 679793(enqueued) 507419(dequeued)
>>>>>
>>>>> We use browseMessages extensively (from around 50 threads) using
>>>> selectors like 'JMSCorrelationID LIKE <data>%'.
>>>>> We notice this issue only with those queues we use browseMessages
>>>> against.
>>>>> Not sure if there is a link to usage of browseMessages.
>>>>>
>>>>> AMQ version: 5.16.3 (windows)
>>>>> Persistence:
>>>>> Yes (Microsoft SQL Server 2019 on Linux (Ubuntu 16.04.6 LTS) <X64>,
>>>> mssql-jdbc-9.4)
>>>>> config:
>>>>> <persistenceAdapter>
>>>>> <jdbcPersistenceAdapter
>>>> dataDirectory="activemq-data" dataSource="#mssql-ds"
>>>> lockKeepAlivePeriod="5000">
>>>>>
>>>> <adapter><transact-jdbc-adapter/></adapter>
>>>>> <locker>
>>>>> <lease-database-locker
>>>> lockAcquireSleepInterval="10000"/>
>>>>> </locker>
>>>>> </jdbcPersistenceAdapter>
>>>>> </persistenceAdapter>
>>>>>
>>>>> Appreciate any help/pointers narrowing down the cause.
>>>>>
>>>>
>>
>>
Re: [AMQ] Queue size grows infinitely
Posted by Balamurugan Seenivasan <to...@gmail.com>.
Hi,
Thanks for your reply.
This issue exists for us even if we start out with a clean environment (with a brand new database).
We even noticed this issue with KahaDB too.
This is consistently reproducible for us.
Regards,
Bala
On 2021/10/01 17:56:07, Matt Pavlovich <ma...@gmail.com> wrote:
> Hi Balamurugan-
>
> Is the broker restarted at any time in this sequence of events?
>
>
> Keep in mind—
>
> Pending Count can be higher than Enqueue Count when the broker is restarted and messages remained on the queue.
>
> a. Enqueue and Dequeue counts reset are for that broker’s uptime.
>
> b. Pending Count (aka queueSize) is how many messages are currently in the store.
>
> -Matt Pavlovich
>
> > On Oct 1, 2021, at 7:29 AM, Tim Bain <tb...@alumni.duke.edu> wrote:
> >
> > The fact that pending messages > enqueued messages looks suspicious, so my
> > expectation is that this will turn out to be a bug in the statistics code.
> > But the information I requested may help us to confirm that.
> >
> > Tim
> >
> > On Fri, Oct 1, 2021, 6:23 AM Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> >
> >> Hi,
> >>
> >> enqueued and dequeued only increase.
> >>
> >> The main indicator to check is the pending.
> >>
> >> If pending count is increasing, it means that the queue size grow
> >> infinitely. enqueued is the number of produced messages (not necessary
> >> still in the queue). So, which value are you referencing ?
> >> BrowsingMessages can "lock" the messages, blocking the consumers
> >> (depending the way you do it).
> >>
> >> Regards
> >> JB
> >>
> >> On 29/09/2021 09:23, Balamurugan Seenivasan wrote:
> >>> Hi,
> >>>
> >>> We notice an issue recently in production where the messages in queue
> >> size grows disproportionate to the number of messages actually produced.
> >>> Queue size grows close to 3 million in about 3 days while the actual
> >> messages produced are way less than the actual queue size.
> >>>
> >>> Here is one snapshot of how the queue looks like when actual messages
> >> produced are only around 60000.
> >>> 794712 (pending) 8(consumers) 679793(enqueued) 507419(dequeued)
> >>>
> >>> We use browseMessages extensively (from around 50 threads) using
> >> selectors like 'JMSCorrelationID LIKE <data>%'.
> >>> We notice this issue only with those queues we use browseMessages
> >> against.
> >>> Not sure if there is a link to usage of browseMessages.
> >>>
> >>> AMQ version: 5.16.3 (windows)
> >>> Persistence:
> >>> Yes (Microsoft SQL Server 2019 on Linux (Ubuntu 16.04.6 LTS) <X64>,
> >> mssql-jdbc-9.4)
> >>> config:
> >>> <persistenceAdapter>
> >>> <jdbcPersistenceAdapter
> >> dataDirectory="activemq-data" dataSource="#mssql-ds"
> >> lockKeepAlivePeriod="5000">
> >>>
> >> <adapter><transact-jdbc-adapter/></adapter>
> >>> <locker>
> >>> <lease-database-locker
> >> lockAcquireSleepInterval="10000"/>
> >>> </locker>
> >>> </jdbcPersistenceAdapter>
> >>> </persistenceAdapter>
> >>>
> >>> Appreciate any help/pointers narrowing down the cause.
> >>>
> >>
>
>
Re: [AMQ] Queue size grows infinitely
Posted by Matt Pavlovich <ma...@gmail.com>.
Hi Balamurugan-
Is the broker restarted at any time in this sequence of events?
Keep in mind—
Pending Count can be higher than Enqueue Count when the broker is restarted and messages remained on the queue.
a. Enqueue and Dequeue counts reset are for that broker’s uptime.
b. Pending Count (aka queueSize) is how many messages are currently in the store.
-Matt Pavlovich
> On Oct 1, 2021, at 7:29 AM, Tim Bain <tb...@alumni.duke.edu> wrote:
>
> The fact that pending messages > enqueued messages looks suspicious, so my
> expectation is that this will turn out to be a bug in the statistics code.
> But the information I requested may help us to confirm that.
>
> Tim
>
> On Fri, Oct 1, 2021, 6:23 AM Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>
>> Hi,
>>
>> enqueued and dequeued only increase.
>>
>> The main indicator to check is the pending.
>>
>> If pending count is increasing, it means that the queue size grow
>> infinitely. enqueued is the number of produced messages (not necessary
>> still in the queue). So, which value are you referencing ?
>> BrowsingMessages can "lock" the messages, blocking the consumers
>> (depending the way you do it).
>>
>> Regards
>> JB
>>
>> On 29/09/2021 09:23, Balamurugan Seenivasan wrote:
>>> Hi,
>>>
>>> We notice an issue recently in production where the messages in queue
>> size grows disproportionate to the number of messages actually produced.
>>> Queue size grows close to 3 million in about 3 days while the actual
>> messages produced are way less than the actual queue size.
>>>
>>> Here is one snapshot of how the queue looks like when actual messages
>> produced are only around 60000.
>>> 794712 (pending) 8(consumers) 679793(enqueued) 507419(dequeued)
>>>
>>> We use browseMessages extensively (from around 50 threads) using
>> selectors like 'JMSCorrelationID LIKE <data>%'.
>>> We notice this issue only with those queues we use browseMessages
>> against.
>>> Not sure if there is a link to usage of browseMessages.
>>>
>>> AMQ version: 5.16.3 (windows)
>>> Persistence:
>>> Yes (Microsoft SQL Server 2019 on Linux (Ubuntu 16.04.6 LTS) <X64>,
>> mssql-jdbc-9.4)
>>> config:
>>> <persistenceAdapter>
>>> <jdbcPersistenceAdapter
>> dataDirectory="activemq-data" dataSource="#mssql-ds"
>> lockKeepAlivePeriod="5000">
>>>
>> <adapter><transact-jdbc-adapter/></adapter>
>>> <locker>
>>> <lease-database-locker
>> lockAcquireSleepInterval="10000"/>
>>> </locker>
>>> </jdbcPersistenceAdapter>
>>> </persistenceAdapter>
>>>
>>> Appreciate any help/pointers narrowing down the cause.
>>>
>>
Re: [AMQ] Queue size grows infinitely
Posted by Balamurugan Seenivasan <to...@gmail.com>.
Hi Tim,
Thanks for your reply.
It doesn't appear to be a statistics issue. We see same number of records in the database too.
Once it hits certain numbers (3 million in our case), things start to slow down and affect performance badly.
The workaround that we have in place is to cleanup the DB once it hits around 2 million.
We were on AMQ version 5.15.10.
Came across this issue https://issues.apache.org/jira/browse/AMQ-7425 which seemed similar to what we experience.
So we tried upgrading to 5.16.3 with no luck still.
Regards,
Bala
On 2021/10/01 12:29:08, Tim Bain <tb...@alumni.duke.edu> wrote:
> The fact that pending messages > enqueued messages looks suspicious, so my
> expectation is that this will turn out to be a bug in the statistics code.
> But the information I requested may help us to confirm that.
>
> Tim
>
> On Fri, Oct 1, 2021, 6:23 AM Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>
> > Hi,
> >
> > enqueued and dequeued only increase.
> >
> > The main indicator to check is the pending.
> >
> > If pending count is increasing, it means that the queue size grow
> > infinitely. enqueued is the number of produced messages (not necessary
> > still in the queue). So, which value are you referencing ?
> > BrowsingMessages can "lock" the messages, blocking the consumers
> > (depending the way you do it).
> >
> > Regards
> > JB
> >
> > On 29/09/2021 09:23, Balamurugan Seenivasan wrote:
> > > Hi,
> > >
> > > We notice an issue recently in production where the messages in queue
> > size grows disproportionate to the number of messages actually produced.
> > > Queue size grows close to 3 million in about 3 days while the actual
> > messages produced are way less than the actual queue size.
> > >
> > > Here is one snapshot of how the queue looks like when actual messages
> > produced are only around 60000.
> > > 794712 (pending) 8(consumers) 679793(enqueued) 507419(dequeued)
> > >
> > > We use browseMessages extensively (from around 50 threads) using
> > selectors like 'JMSCorrelationID LIKE <data>%'.
> > > We notice this issue only with those queues we use browseMessages
> > against.
> > > Not sure if there is a link to usage of browseMessages.
> > >
> > > AMQ version: 5.16.3 (windows)
> > > Persistence:
> > > Yes (Microsoft SQL Server 2019 on Linux (Ubuntu 16.04.6 LTS) <X64>,
> > mssql-jdbc-9.4)
> > > config:
> > > <persistenceAdapter>
> > > <jdbcPersistenceAdapter
> > dataDirectory="activemq-data" dataSource="#mssql-ds"
> > lockKeepAlivePeriod="5000">
> > >
> > <adapter><transact-jdbc-adapter/></adapter>
> > > <locker>
> > > <lease-database-locker
> > lockAcquireSleepInterval="10000"/>
> > > </locker>
> > > </jdbcPersistenceAdapter>
> > > </persistenceAdapter>
> > >
> > > Appreciate any help/pointers narrowing down the cause.
> > >
> >
>
Re: [AMQ] Queue size grows infinitely
Posted by Tim Bain <tb...@alumni.duke.edu>.
The fact that pending messages > enqueued messages looks suspicious, so my
expectation is that this will turn out to be a bug in the statistics code.
But the information I requested may help us to confirm that.
Tim
On Fri, Oct 1, 2021, 6:23 AM Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> Hi,
>
> enqueued and dequeued only increase.
>
> The main indicator to check is the pending.
>
> If pending count is increasing, it means that the queue size grow
> infinitely. enqueued is the number of produced messages (not necessary
> still in the queue). So, which value are you referencing ?
> BrowsingMessages can "lock" the messages, blocking the consumers
> (depending the way you do it).
>
> Regards
> JB
>
> On 29/09/2021 09:23, Balamurugan Seenivasan wrote:
> > Hi,
> >
> > We notice an issue recently in production where the messages in queue
> size grows disproportionate to the number of messages actually produced.
> > Queue size grows close to 3 million in about 3 days while the actual
> messages produced are way less than the actual queue size.
> >
> > Here is one snapshot of how the queue looks like when actual messages
> produced are only around 60000.
> > 794712 (pending) 8(consumers) 679793(enqueued) 507419(dequeued)
> >
> > We use browseMessages extensively (from around 50 threads) using
> selectors like 'JMSCorrelationID LIKE <data>%'.
> > We notice this issue only with those queues we use browseMessages
> against.
> > Not sure if there is a link to usage of browseMessages.
> >
> > AMQ version: 5.16.3 (windows)
> > Persistence:
> > Yes (Microsoft SQL Server 2019 on Linux (Ubuntu 16.04.6 LTS) <X64>,
> mssql-jdbc-9.4)
> > config:
> > <persistenceAdapter>
> > <jdbcPersistenceAdapter
> dataDirectory="activemq-data" dataSource="#mssql-ds"
> lockKeepAlivePeriod="5000">
> >
> <adapter><transact-jdbc-adapter/></adapter>
> > <locker>
> > <lease-database-locker
> lockAcquireSleepInterval="10000"/>
> > </locker>
> > </jdbcPersistenceAdapter>
> > </persistenceAdapter>
> >
> > Appreciate any help/pointers narrowing down the cause.
> >
>
Re: [AMQ] Queue size grows infinitely
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi,
enqueued and dequeued only increase.
The main indicator to check is the pending.
If pending count is increasing, it means that the queue size grow
infinitely. enqueued is the number of produced messages (not necessary
still in the queue). So, which value are you referencing ?
BrowsingMessages can "lock" the messages, blocking the consumers
(depending the way you do it).
Regards
JB
On 29/09/2021 09:23, Balamurugan Seenivasan wrote:
> Hi,
>
> We notice an issue recently in production where the messages in queue size grows disproportionate to the number of messages actually produced.
> Queue size grows close to 3 million in about 3 days while the actual messages produced are way less than the actual queue size.
>
> Here is one snapshot of how the queue looks like when actual messages produced are only around 60000.
> 794712 (pending) 8(consumers) 679793(enqueued) 507419(dequeued)
>
> We use browseMessages extensively (from around 50 threads) using selectors like 'JMSCorrelationID LIKE <data>%'.
> We notice this issue only with those queues we use browseMessages against.
> Not sure if there is a link to usage of browseMessages.
>
> AMQ version: 5.16.3 (windows)
> Persistence:
> Yes (Microsoft SQL Server 2019 on Linux (Ubuntu 16.04.6 LTS) <X64>, mssql-jdbc-9.4)
> config:
> <persistenceAdapter>
> <jdbcPersistenceAdapter dataDirectory="activemq-data" dataSource="#mssql-ds" lockKeepAlivePeriod="5000">
> <adapter><transact-jdbc-adapter/></adapter>
> <locker>
> <lease-database-locker lockAcquireSleepInterval="10000"/>
> </locker>
> </jdbcPersistenceAdapter>
> </persistenceAdapter>
>
> Appreciate any help/pointers narrowing down the cause.
>