You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by fmansoor <fa...@gmail.com> on 2015/09/23 04:06:30 UTC
KahaDb journal file are not getting recycled and "Total Message
Count" is negative
I removed all the messages from all the queues but the kahan db files are not
getting recycled. Hawtio is showing that the total message count is -59.
<http://activemq.2283324.n4.nabble.com/file/n4702208/message-count.png>
I am using the following mKahaDB configs.
<persistenceAdapter>
<mKahaDB directory="/mnt/nfs/kahadb/shared-store">
<locker>
<shared-file-locker lockAcquireSleepInterval="5000"/>
</locker>
<filteredPersistenceAdapters>
<filteredKahaDB queue="ActiveMQ.DLQ.>">
<persistenceAdapter>
<kahaDB directory="/mnt/nfs/kahadb/kahadb-dlq"/>
</persistenceAdapter>
</filteredKahaDB>
<filteredKahaDB>
<persistenceAdapter>
<kahaDB directory="/mnt/nfs/kahadb/kahadb-main" />
</persistenceAdapter>
</filteredKahaDB>
</filteredPersistenceAdapters>
</mKahaDB>
</persistenceAdapter>
The problem occurs only with the main-kahadb, files from kahadb-dlq are
properly getting recycled.
--
View this message in context: http://activemq.2283324.n4.nabble.com/KahaDb-journal-file-are-not-getting-recycled-and-Total-Message-Count-is-negative-tp4702208.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
RE: KahaDb journal file are not getting recycled and "Total Message
Count" is negative
Posted by ba...@wellsfargo.com.
Thanks Tim! Does anyone know about the txlog, and why it would be at 50gig??? We are changing the HOWL settings to see if this helps, but I'm suspecting maybe a camel route is causing the issue. We use transacted=false in our camel routes.. should we be setting that to true?
Regards,
Barry
-----Original Message-----
From: tbain98@gmail.com [mailto:tbain98@gmail.com] On Behalf Of Tim Bain
Sent: Wednesday, September 30, 2015 10:51 AM
To: ActiveMQ Users
Subject: Re: KahaDb journal file are not getting recycled and "Total Message Count" is negative
Unfortunately I don't know of any tools for inspecting KahaDB files, though someone else here might. Sorry, I've never done much with KahaDB, so what I know comes from the wiki page and from mailing list posts like this one.
Tim
On Tue, Sep 29, 2015 at 3:33 PM, fmansoor <fa...@gmail.com> wrote:
> Thanks Tim,
>
> Subscribers only subscribe to queues. I restarted the server, but,
> that didn't fix the problem either, haven't tried destroying all
> destinations yet.
>
> The problem occurred in production, we have to drain all the queues
> and start with a new kahadb to solve the problem. I took backup of the
> database and can reproduce the problem with the backup, I am trying to
> understand the problem, so, that in case if this happens again, we can
> solve it without restarting and starting from scratch.
>
> Do you know of any tool which can be used to introspect the kahandb,
> figure out which destinations have pending ack which the UI is not showing?
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/KahaDb-journal-file-are-not-gett
> ing-recycled-and-Total-Message-Count-is-negative-tp4702208p4702444.htm
> l Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: KahaDb journal file are not getting recycled and "Total Message
Count" is negative
Posted by Tim Bain <tb...@alumni.duke.edu>.
Unfortunately I don't know of any tools for inspecting KahaDB files, though
someone else here might. Sorry, I've never done much with KahaDB, so what
I know comes from the wiki page and from mailing list posts like this one.
Tim
On Tue, Sep 29, 2015 at 3:33 PM, fmansoor <fa...@gmail.com> wrote:
> Thanks Tim,
>
> Subscribers only subscribe to queues. I restarted the server, but, that
> didn't fix the problem either, haven't tried destroying all destinations
> yet.
>
> The problem occurred in production, we have to drain all the queues and
> start with a new kahadb to solve the problem. I took backup of the database
> and can reproduce the problem with the backup, I am trying to understand
> the
> problem, so, that in case if this happens again, we can solve it without
> restarting and starting from scratch.
>
> Do you know of any tool which can be used to introspect the kahandb, figure
> out which destinations have pending ack which the UI is not showing?
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/KahaDb-journal-file-are-not-getting-recycled-and-Total-Message-Count-is-negative-tp4702208p4702444.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: KahaDb journal file are not getting recycled and "Total Message
Count" is negative
Posted by fmansoor <fa...@gmail.com>.
Thanks Tim,
Subscribers only subscribe to queues. I restarted the server, but, that
didn't fix the problem either, haven't tried destroying all destinations
yet.
The problem occurred in production, we have to drain all the queues and
start with a new kahadb to solve the problem. I took backup of the database
and can reproduce the problem with the backup, I am trying to understand the
problem, so, that in case if this happens again, we can solve it without
restarting and starting from scratch.
Do you know of any tool which can be used to introspect the kahandb, figure
out which destinations have pending ack which the UI is not showing?
--
View this message in context: http://activemq.2283324.n4.nabble.com/KahaDb-journal-file-are-not-getting-recycled-and-Total-Message-Count-is-negative-tp4702208p4702444.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: KahaDb journal file are not getting recycled and "Total Message
Count" is negative
Posted by Tim Bain <tb...@alumni.duke.edu>.
What about the subscribers on your topics? (Does anyone subscribe directly
to the topic, or is consumption purely from the queues on the virtual
topics?) And are any selectors in use for any client?
I'm curious whether the KahaDB files are cleaned up 1) following a broker
restart, and 2) if you disconnect all consumers and producers and then
destroy all destinations (both queues and topics).
I assume given the small number of messages you're quoting that this
problem is reproducible in a test environment, so you can make changes and
then try to re-create the problem to see if the change had any effect? Or
is this an it-only-happened-once-and-I-can't-reproduce-it situation?
Tim
On Wed, Sep 23, 2015 at 3:14 PM, fmansoor <fa...@gmail.com> wrote:
> Yes, all the queues are empty, enqueue and dequeue counts were off for a
> couple of queues, probably, because I purged them.
>
> I am running 5.11.1, all the queues are consuming messages from virtual
> topics and advisory messages are disabled.
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/KahaDb-journal-file-are-not-getting-recycled-and-Total-Message-Count-is-negative-tp4702208p4702248.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: KahaDb journal file are not getting recycled and "Total Message
Count" is negative
Posted by fmansoor <fa...@gmail.com>.
Yes, all the queues are empty, enqueue and dequeue counts were off for a
couple of queues, probably, because I purged them.
I am running 5.11.1, all the queues are consuming messages from virtual
topics and advisory messages are disabled.
--
View this message in context: http://activemq.2283324.n4.nabble.com/KahaDb-journal-file-are-not-getting-recycled-and-Total-Message-Count-is-negative-tp4702208p4702248.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: KahaDb journal file are not getting recycled and "Total Message
Count" is negative
Posted by Tim Bain <tb...@alumni.duke.edu>.
That can happen if you have topics whose consumers don't consume all the
messages, either due to selectors or due to consumers being offline when
messages are published. (Do you have topics? And are advisory messages
enabled?) If neither of those are the case, there are probably unconsumed
messages somewhere.
By "nothings seems out of ordinary", did you mean that for each
destination, the enqueue and dequeue counts are equal?
I can't answer any questions about checkForCorruptJournalFiles, but maybe
someone else here can.
Also, you've never said what version you're running.
Tim
On Sep 23, 2015 12:21 AM, "fmansoor" <fa...@gmail.com> wrote:
> Thanks Tim, there are no durable subscribers, this server does not have any
> networked brokers either.
>
> Went though JMX information, but, nothings seems out of ordinary. Except
> that the total dequeue count (118) is less than total enqueue count (242).
>
> I know about heckForCorruptJournalFiles, but, is there an way to check for
> corrupt journal entries without restarting the server?
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/KahaDb-journal-file-are-not-getting-recycled-and-Total-Message-Count-is-negative-tp4702208p4702212.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: KahaDb journal file are not getting recycled and "Total Message
Count" is negative
Posted by fmansoor <fa...@gmail.com>.
Thanks Tim, there are no durable subscribers, this server does not have any
networked brokers either.
Went though JMX information, but, nothings seems out of ordinary. Except
that the total dequeue count (118) is less than total enqueue count (242).
I know about heckForCorruptJournalFiles, but, is there an way to check for
corrupt journal entries without restarting the server?
--
View this message in context: http://activemq.2283324.n4.nabble.com/KahaDb-journal-file-are-not-getting-recycled-and-Total-Message-Count-is-negative-tp4702208p4702212.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: KahaDb journal file are not getting recycled and "Total Message
Count" is negative
Posted by Tim Bain <tb...@alumni.duke.edu>.
Are there any topics with unconsumed messages for offline durable
subscribers? Queues aren't the only place messages can stick around.
You might get more information by using information exposed via JMX; it
might help you find where there are still messages if indeed there are
some.
On Sep 22, 2015 8:14 PM, "fmansoor" <fa...@gmail.com> wrote:
> I removed all the messages from all the queues but the kahan db files are
> not
> getting recycled. Hawtio is showing that the total message count is -59.
>
> <http://activemq.2283324.n4.nabble.com/file/n4702208/message-count.png>
>
> I am using the following mKahaDB configs.
>
> <persistenceAdapter>
> <mKahaDB directory="/mnt/nfs/kahadb/shared-store">
> <locker>
> <shared-file-locker lockAcquireSleepInterval="5000"/>
> </locker>
> <filteredPersistenceAdapters>
> <filteredKahaDB queue="ActiveMQ.DLQ.>">
> <persistenceAdapter>
> <kahaDB directory="/mnt/nfs/kahadb/kahadb-dlq"/>
> </persistenceAdapter>
> </filteredKahaDB>
> <filteredKahaDB>
> <persistenceAdapter>
> <kahaDB directory="/mnt/nfs/kahadb/kahadb-main" />
> </persistenceAdapter>
> </filteredKahaDB>
> </filteredPersistenceAdapters>
> </mKahaDB>
> </persistenceAdapter>
>
> The problem occurs only with the main-kahadb, files from kahadb-dlq are
> properly getting recycled.
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/KahaDb-journal-file-are-not-getting-recycled-and-Total-Message-Count-is-negative-tp4702208.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>