[jira] [Closed] (AMQ-6354) KahaDb journal file piling up because of Durable subscriber destination

["Timothy Bish (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AMQ-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Timothy Bish closed AMQ-6354.
-----------------------------
    Resolution: Cannot Reproduce

Given this is reported against a very old broker release and there's no reproducer supplied to test on newer releases there isn't much that we can do here.  The KahaDB store will GC when possible but if older messages are left in the store this can lead to log files being left ineligible for GC.  

> KahaDb journal file piling up because of Durable subscriber destination
> -----------------------------------------------------------------------
>
>                 Key: AMQ-6354
>                 URL: https://issues.apache.org/jira/browse/AMQ-6354
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker, KahaDB, Message Store
>    Affects Versions: 5.5.1
>         Environment: Linux
>            Reporter: Rajagopal
>              Labels: activemq, broker, kahaDB, messagestore
>
> AMQ Configuration:
> 1.	KahaDb Persistence store
> 2.	Topics and Queues destination are configured.
> 3.	Durable Subscribers configured (added/removed) dynamically (All added subscribers will be active) for Topic destination. 
> Issue:
> •	Nearly ~20 Durable Subscriber added/deleted to a particular destination dynamically.
> •	After configuration, Once the message are sent to the Topic, the messages are consumed and Acknowledged by each subscribers.
> •	There were 249 Subscribers and we could see from Journal file that all subscribers acknowledged.
> •	In JMX logs after message is sent the below entries are observed for yyyy destination, 
> o	Consumer Count is 249
> o	In-Flight Count is 0
> o	Dispatch count is matching with the consumer count (messages * Consumer count)
> •	We enabled kahaDB TRACE logs and it clearly points out that for a particular Topic destination the journal files could not be gc collected
> Snippet of activemq.log
> ----------------------------------------------
> 2016-06-30 01:41:43,980 org.apache.activemq.store.kahadb.MessageDatabase TRACE (MessageDatabase.java:checkpointUpdate:1413) [ActiveMQ Journal Checkpoint Worker] [24] | gc candidates after dest:0:xxxxx, [8066, 8067, ...........]
> 2016-06-30 01:41:43,980 org.apache.activemq.store.kahadb.MessageDatabase TRACE (MessageDatabase.java:checkpointUpdate:1413) [ActiveMQ Journal Checkpoint Worker] [24] | gc candidates after dest:1:yyyy, [8067, 8073, ............]
> --------------------------------------------
> •	After analyzing the 8066 file, we understood that 8066 has one message for yyyy Topic destination and 249 ACk’s in the same file. But Checkpoint cleanup could not remove the file. So this pattern followed and all the files containing the yyyy destination messages are not removed, which leads to journal files pile-up.
> Workaround :
> •	As a work around, first we restarted the process but still the files were not cleaned up.
> •	Secondly,  we stopped the process, removed the index (db.data) file and restarted. Now all the files are cleaned up.
> •	By above workaround, we suspect that index file is corrupted, which is causing the journal pile up. 
> •	Our question, 
> o	if index file is corrupted, what is the reason for corruption?
> o	Is there any bug in the 5.5.1 version of AMQ regarding index file corruption for Durable Subscription?
> This is sporadic issue and not reproducible always. (Not always Dynamic Subscription configuration leads to journal pile up)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)