[jira] Commented: (AMQ-2736) KahaDB doesn't clean up old files

["Martin Cekal (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AMQ-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12994338#comment-12994338 ] 

Martin Cekal commented on AMQ-2736:
-----------------------------------

Hello Dejan,
We noticed StorageFull on KahaDb several times so I played a little and found two reasons for this (second seems to be a AMQ bug):
1) unconsumed message(s) in any datafile blocks the datafile forever there. So if you keep default setup 33mb per file and 1gb limit roughly 30 small messages spread one to each datafile can block whole system
(I assume this is not a bug, just not nice feature - Why not to reply(=rewrite) messages at least during AMQ restart to latest datafile?)

2) second issue seems to me as a bug. If you open transacted session and send a message, but do not commit it a datafile is blocked. Again if you spread such uncommited messages over all datafiles you gets "storage is full". That is ok - someone forget to commit - but problem comes if you kill AMQ using -9 signal (or via TaskManager on Windows). If you do this non-standard kill (=like a hw failure) the uncommited transaction seems to be blocking datafiles forever. New start of AMQ somehow reads again the transaction and still waiting for commit although the producer is not live any more. If you do regular shutdown so ShutdownHook on the Broker is fully executed, next AMQ start correctly recognize whether producer of uncommited transaction (with failover transport) is still up (then transaction is kept, messages are replayed to new/last datafile) or down (then transaction is rollbacked).
I think AMQ should be able to behave this way even if previously someone killed it brutally(kill -9).

I have tested these two scenarios on both 5.3.2 and 5.4.2 with same result. We would like to get a patch for 5.3.2 version. Is it possible?

Thanks a lot
Martin

> KahaDB doesn't clean up old files
> ---------------------------------
>
>                 Key: AMQ-2736
>                 URL: https://issues.apache.org/jira/browse/AMQ-2736
>             Project: ActiveMQ
>          Issue Type: Bug
>    Affects Versions: 5.3.2
>            Reporter: Adrian Trenaman
>            Assignee: Dejan Bosanac
>            Priority: Critical
>             Fix For: 5.5.0
>
>         Attachments: AMQ-2736.zip, AMQ2736Test_should_with_this_diff.txt, MyKahaDBStore.java, amq-2987-testcase.patch, amq-2987.patch
>
>
> Over time, we're seeing that kahadb doesn't clean up old journal files. As a result, we eventually run out of disk space, or rather, we hit our usage limits and our producers are slowed down by the producer flow control mechanism. Others have experienced this problem too (for example, see http://mail-archives.apache.org/mod_mbox/activemq-users/201002.mbox/%3C27502591.post@talk.nabble.com%3E)
> For now, we're moving back to the old amqPersistenceStore. 

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira