5.13.3 Invisible message prevents storage garbage clean

[RuralHunter <ru...@gmail.com>]

I noticed some old data files(generated 2 days ago) are not get cleaned up.
So I referred the doc to check which queue is using them. I found they are
reserved for one busy queue. The queue has several hundreds of messages
going through every second. There couldn't be any old message left there. I
also checked the queue in admin ui and didn't find any old message in it.
The new messages are in and out all the time in that queue.
If I delete the queue in admin ui, the old data files are cleaned up soon.
Or if I restart activemq, the old data files also get cleaned up.
I don't know how to track this problem since it only occurs occasionally and
there is no reliable reproduce step. I can modify the source to debug it if
someone can guide me.




--
View this message in context: http://activemq.2283324.n4.nabble.com/5-13-3-Invisible-message-prevents-storage-garbage-clean-tp4713229.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: 5.13.3 Invisible message prevents storage garbage clean

[RuralHunter <ru...@gmail.com>]

I already turned on the trace log on MessageDatabase and also added some
debug code. The file is removed from the GC candidates in this loop in the
method 'void checkpointUpdate(Transaction tx, boolean cleanup) throws
IOException':
            // Go through all the destinations to see if any of them can
remove GC candidates.
            for (Entry<String, StoredDestination> entry :
storedDestinations.entrySet()) {
            ...
            }
We are already using the ack compaction feature and it didn't help this.



--
View this message in context: http://activemq.2283324.n4.nabble.com/5-13-3-Invisible-message-prevents-storage-garbage-clean-tp4713229p4713265.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: 5.13.3 Invisible message prevents storage garbage clean

[Christopher Shannon <ch...@gmail.com>]

The garbage collection process which determines which files can be garbage
collected is done in the MessageDatabase class that is part of KahaDB.  If
you turn on trace logging for
org.apache.activemq.store.kahadb.MessageDatabase then your log file will
spit out a ton of information about the gc process.  It will log
information about the garbage collection candidates after it checks various
things to determine if a file should be kept.  This log information should
help determine which phase of the GC checking removed those files from
being eligible from garbage collection.

If the issue is acks referencing the file then the ack compaction might
help with that.  There are a couple bugs with it in 5.13.3 (and it is off
by default) that are fixed for 5.14.0.

On Tue, Jun 21, 2016 at 9:34 PM, RuralHunter <ru...@gmail.com> wrote:

> I noticed some old data files(generated 2 days ago) are not get cleaned up.
> So I referred the doc to check which queue is using them. I found they are
> reserved for one busy queue. The queue has several hundreds of messages
> going through every second. There couldn't be any old message left there. I
> also checked the queue in admin ui and didn't find any old message in it.
> The new messages are in and out all the time in that queue.
> If I delete the queue in admin ui, the old data files are cleaned up soon.
> Or if I restart activemq, the old data files also get cleaned up.
> I don't know how to track this problem since it only occurs occasionally
> and
> there is no reliable reproduce step. I can modify the source to debug it if
> someone can guide me.
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/5-13-3-Invisible-message-prevents-storage-garbage-clean-tp4713229.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>