You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by "Adrian Trenaman (JIRA)" <ji...@apache.org> on 2010/05/18 11:40:14 UTC
[jira] Created: (AMQ-2736) KahaDB doesn't clean up old files
KahaDB doesn't clean up old files
---------------------------------
Key: AMQ-2736
URL: https://issues.apache.org/activemq/browse/AMQ-2736
Project: ActiveMQ
Issue Type: Bug
Affects Versions: 5.3.2
Reporter: Adrian Trenaman
Priority: Critical
Over time, we're seeing that kahadb doesn't clean up old journal files. As a result, we eventually run out of disk space, or rather, we hit our usage limits and our producers are slowed down by the producer flow control mechanism. Others have experienced this problem too (for example, see http://mail-archives.apache.org/mod_mbox/activemq-users/201002.mbox/%3C27502591.post@talk.nabble.com%3E)
For now, we're moving back to the old amqPersistenceStore.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2736) KahaDB doesn't clean up old files
Posted by "Jeff Genender (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=62432#action_62432 ]
Jeff Genender commented on AMQ-2736:
------------------------------------
Oops... sorry for the multiple posts :/
> KahaDB doesn't clean up old files
> ---------------------------------
>
> Key: AMQ-2736
> URL: https://issues.apache.org/activemq/browse/AMQ-2736
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Reporter: Adrian Trenaman
> Priority: Critical
> Attachments: AMQ-2736.zip
>
>
> Over time, we're seeing that kahadb doesn't clean up old journal files. As a result, we eventually run out of disk space, or rather, we hit our usage limits and our producers are slowed down by the producer flow control mechanism. Others have experienced this problem too (for example, see http://mail-archives.apache.org/mod_mbox/activemq-users/201002.mbox/%3C27502591.post@talk.nabble.com%3E)
> For now, we're moving back to the old amqPersistenceStore.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (AMQ-2736) KahaDB doesn't clean up old files
Posted by "Dejan Bosanac (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dejan Bosanac resolved AMQ-2736.
--------------------------------
Resolution: Fixed
Fix Version/s: 5.5.0
Assignee: Dejan Bosanac
That potential issue from the last comment was the false alarm. I believe that this issue should be fixed now. Please test the latest snapshot
https://repository.apache.org/content/repositories/snapshots/org/apache/activemq/apache-activemq/5.5-SNAPSHOT/
and reopen if you find any problems.
Thanks
> KahaDB doesn't clean up old files
> ---------------------------------
>
> Key: AMQ-2736
> URL: https://issues.apache.org/activemq/browse/AMQ-2736
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Reporter: Adrian Trenaman
> Assignee: Dejan Bosanac
> Priority: Critical
> Fix For: 5.5.0
>
> Attachments: AMQ-2736.zip, amq-2987-testcase.patch, amq-2987.patch, AMQ2736Test_should_with_this_diff.txt, MyKahaDBStore.java
>
>
> Over time, we're seeing that kahadb doesn't clean up old journal files. As a result, we eventually run out of disk space, or rather, we hit our usage limits and our producers are slowed down by the producer flow control mechanism. Others have experienced this problem too (for example, see http://mail-archives.apache.org/mod_mbox/activemq-users/201002.mbox/%3C27502591.post@talk.nabble.com%3E)
> For now, we're moving back to the old amqPersistenceStore.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2736) KahaDB doesn't clean up old files
Posted by "Christer WIkman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=63192#action_63192 ]
Christer WIkman commented on AMQ-2736:
--------------------------------------
I deployed yesterday in stage latest 5.5-SNAPSHOT and it seems that the problem is solved. This morning I had 17 log files, but after I deleted DLQ queues with messages I ended up with only 1 log file! When I used 5.4.1 the files grow with approx 50-100 files per day.
We are not comfortable deploying 5.5-SNAPSHOT in production. Can anyone provide me with a patch for 5.4.1? Currently, I cannot recommend using version 5.4.1 in production without the combined fixes from 5.5-SNAPSHOT.
> KahaDB doesn't clean up old files
> ---------------------------------
>
> Key: AMQ-2736
> URL: https://issues.apache.org/activemq/browse/AMQ-2736
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Reporter: Adrian Trenaman
> Assignee: Dejan Bosanac
> Priority: Critical
> Fix For: 5.5.0
>
> Attachments: AMQ-2736.zip, amq-2987-testcase.patch, amq-2987.patch, AMQ2736Test_should_with_this_diff.txt, MyKahaDBStore.java
>
>
> Over time, we're seeing that kahadb doesn't clean up old journal files. As a result, we eventually run out of disk space, or rather, we hit our usage limits and our producers are slowed down by the producer flow control mechanism. Others have experienced this problem too (for example, see http://mail-archives.apache.org/mod_mbox/activemq-users/201002.mbox/%3C27502591.post@talk.nabble.com%3E)
> For now, we're moving back to the old amqPersistenceStore.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2736) KahaDB doesn't clean up old files
Posted by "Adrian Lisenberg (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=63373#action_63373 ]
Adrian Lisenberg commented on AMQ-2736:
---------------------------------------
Hi, I really need this fix!!!!. So finally you tested the fix on 5.4.1 or you are going to put 5.5 in production?
excuse me, but which is the file with the correct patch?
One more question: After applying this fix, would be any way for deleting old log files in order to reduce unused space? I am at 100G and I need to maintain the current state of the broker...
thanks.
> KahaDB doesn't clean up old files
> ---------------------------------
>
> Key: AMQ-2736
> URL: https://issues.apache.org/activemq/browse/AMQ-2736
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Reporter: Adrian Trenaman
> Assignee: Dejan Bosanac
> Priority: Critical
> Fix For: 5.5.0
>
> Attachments: AMQ-2736.zip, amq-2987-testcase.patch, amq-2987.patch, AMQ2736Test_should_with_this_diff.txt, MyKahaDBStore.java
>
>
> Over time, we're seeing that kahadb doesn't clean up old journal files. As a result, we eventually run out of disk space, or rather, we hit our usage limits and our producers are slowed down by the producer flow control mechanism. Others have experienced this problem too (for example, see http://mail-archives.apache.org/mod_mbox/activemq-users/201002.mbox/%3C27502591.post@talk.nabble.com%3E)
> For now, we're moving back to the old amqPersistenceStore.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2736) KahaDB doesn't clean up old files
Posted by "Gary Tully (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=62985#action_62985 ]
Gary Tully commented on AMQ-2736:
---------------------------------
Christer, please validate with the 5.5-SNAPSHOT, that has the required combined fixes.
https://repository.apache.org/content/repositories/snapshots/org/apache/activemq/apache-activemq/5.5-SNAPSHOT/
> KahaDB doesn't clean up old files
> ---------------------------------
>
> Key: AMQ-2736
> URL: https://issues.apache.org/activemq/browse/AMQ-2736
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Reporter: Adrian Trenaman
> Assignee: Dejan Bosanac
> Priority: Critical
> Fix For: 5.5.0
>
> Attachments: AMQ-2736.zip, amq-2987-testcase.patch, amq-2987.patch, AMQ2736Test_should_with_this_diff.txt, MyKahaDBStore.java
>
>
> Over time, we're seeing that kahadb doesn't clean up old journal files. As a result, we eventually run out of disk space, or rather, we hit our usage limits and our producers are slowed down by the producer flow control mechanism. Others have experienced this problem too (for example, see http://mail-archives.apache.org/mod_mbox/activemq-users/201002.mbox/%3C27502591.post@talk.nabble.com%3E)
> For now, we're moving back to the old amqPersistenceStore.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2736) KahaDB doesn't clean up old files
Posted by "Christer WIkman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=63368#action_63368 ]
Christer WIkman commented on AMQ-2736:
--------------------------------------
I have been running Jeff's fix in stage for 5 days and the log files have not grown. I'm now comfortable to use it in production.
Many thanks for all help!
> KahaDB doesn't clean up old files
> ---------------------------------
>
> Key: AMQ-2736
> URL: https://issues.apache.org/activemq/browse/AMQ-2736
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Reporter: Adrian Trenaman
> Assignee: Dejan Bosanac
> Priority: Critical
> Fix For: 5.5.0
>
> Attachments: AMQ-2736.zip, amq-2987-testcase.patch, amq-2987.patch, AMQ2736Test_should_with_this_diff.txt, MyKahaDBStore.java
>
>
> Over time, we're seeing that kahadb doesn't clean up old journal files. As a result, we eventually run out of disk space, or rather, we hit our usage limits and our producers are slowed down by the producer flow control mechanism. Others have experienced this problem too (for example, see http://mail-archives.apache.org/mod_mbox/activemq-users/201002.mbox/%3C27502591.post@talk.nabble.com%3E)
> For now, we're moving back to the old amqPersistenceStore.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (AMQ-2736) KahaDB doesn't clean up old files
Posted by "Adrian Trenaman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Adrian Trenaman resolved AMQ-2736.
----------------------------------
Resolution: Fixed
> KahaDB doesn't clean up old files
> ---------------------------------
>
> Key: AMQ-2736
> URL: https://issues.apache.org/activemq/browse/AMQ-2736
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Reporter: Adrian Trenaman
> Priority: Critical
>
> Over time, we're seeing that kahadb doesn't clean up old journal files. As a result, we eventually run out of disk space, or rather, we hit our usage limits and our producers are slowed down by the producer flow control mechanism. Others have experienced this problem too (for example, see http://mail-archives.apache.org/mod_mbox/activemq-users/201002.mbox/%3C27502591.post@talk.nabble.com%3E)
> For now, we're moving back to the old amqPersistenceStore.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2736) KahaDB doesn't clean up old files
Posted by "Dejan Bosanac (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=62866#action_62866 ]
Dejan Bosanac commented on AMQ-2736:
------------------------------------
AMQ-2982 and AMQ-2983 are fixed now on the trunk, so you can test out if that helps. I spotted one more potential issue in the area and I'm working on it now, so stay tuned.
> KahaDB doesn't clean up old files
> ---------------------------------
>
> Key: AMQ-2736
> URL: https://issues.apache.org/activemq/browse/AMQ-2736
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Reporter: Adrian Trenaman
> Priority: Critical
> Attachments: AMQ-2736.zip, amq-2987-testcase.patch, amq-2987.patch, AMQ2736Test_should_with_this_diff.txt, MyKahaDBStore.java
>
>
> Over time, we're seeing that kahadb doesn't clean up old journal files. As a result, we eventually run out of disk space, or rather, we hit our usage limits and our producers are slowed down by the producer flow control mechanism. Others have experienced this problem too (for example, see http://mail-archives.apache.org/mod_mbox/activemq-users/201002.mbox/%3C27502591.post@talk.nabble.com%3E)
> For now, we're moving back to the old amqPersistenceStore.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2736) KahaDB doesn't clean up old files
Posted by "Adrian Trenaman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=59425#action_59425 ]
Adrian Trenaman commented on AMQ-2736:
--------------------------------------
Hi Gary,
Just retested against the very latest FUSE distribution (5.3.1-fuse-01-00) and the log files are being deleted. Looks like this may have been fixed. Will mark this as resolved for now, and revisit if we begin to see the errors again.
/Ade
> KahaDB doesn't clean up old files
> ---------------------------------
>
> Key: AMQ-2736
> URL: https://issues.apache.org/activemq/browse/AMQ-2736
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Reporter: Adrian Trenaman
> Priority: Critical
>
> Over time, we're seeing that kahadb doesn't clean up old journal files. As a result, we eventually run out of disk space, or rather, we hit our usage limits and our producers are slowed down by the producer flow control mechanism. Others have experienced this problem too (for example, see http://mail-archives.apache.org/mod_mbox/activemq-users/201002.mbox/%3C27502591.post@talk.nabble.com%3E)
> For now, we're moving back to the old amqPersistenceStore.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2736) KahaDB doesn't clean up old files
Posted by "Christer WIkman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=63374#action_63374 ]
Christer WIkman commented on AMQ-2736:
--------------------------------------
Use Jeff's patch for 5.4.1, http://people.apache.org/~jgenender/apache-activemq-5.4.1-AMQ2736-AMQ2982-AMQ2983-AMQ2935-bin.tar.gz
I deployed it in production today and it totally removed the problem with growing log files. It went down from 78 log files to just 1 log file.
> KahaDB doesn't clean up old files
> ---------------------------------
>
> Key: AMQ-2736
> URL: https://issues.apache.org/activemq/browse/AMQ-2736
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Reporter: Adrian Trenaman
> Assignee: Dejan Bosanac
> Priority: Critical
> Fix For: 5.5.0
>
> Attachments: AMQ-2736.zip, amq-2987-testcase.patch, amq-2987.patch, AMQ2736Test_should_with_this_diff.txt, MyKahaDBStore.java
>
>
> Over time, we're seeing that kahadb doesn't clean up old journal files. As a result, we eventually run out of disk space, or rather, we hit our usage limits and our producers are slowed down by the producer flow control mechanism. Others have experienced this problem too (for example, see http://mail-archives.apache.org/mod_mbox/activemq-users/201002.mbox/%3C27502591.post@talk.nabble.com%3E)
> For now, we're moving back to the old amqPersistenceStore.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2736) KahaDB doesn't clean up old files
Posted by "Abhishek C (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=62828#action_62828 ]
Abhishek C commented on AMQ-2736:
---------------------------------
@Swen Does using non transacted sessions with auto acknowledge clear the log files?
We have been facing a similar problem, however we are using Kaha Persistence Adapter on upgrading from 5.3.2 to 5.4.1. We had to upgrade to 5.4.1 because earlier using Durable Subs Topics our data files weren't getting cleaned up. Now our index-subs file are creating an issue.
Any Final fix for this problem?
> KahaDB doesn't clean up old files
> ---------------------------------
>
> Key: AMQ-2736
> URL: https://issues.apache.org/activemq/browse/AMQ-2736
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Reporter: Adrian Trenaman
> Priority: Critical
> Attachments: AMQ-2736.zip, amq-2987-testcase.patch, amq-2987.patch, AMQ2736Test_should_with_this_diff.txt, MyKahaDBStore.java
>
>
> Over time, we're seeing that kahadb doesn't clean up old journal files. As a result, we eventually run out of disk space, or rather, we hit our usage limits and our producers are slowed down by the producer flow control mechanism. Others have experienced this problem too (for example, see http://mail-archives.apache.org/mod_mbox/activemq-users/201002.mbox/%3C27502591.post@talk.nabble.com%3E)
> For now, we're moving back to the old amqPersistenceStore.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AMQ-2736) KahaDB doesn't clean up old files
Posted by "Jeff Genender (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeff Genender updated AMQ-2736:
-------------------------------
Attachment: amq-2987.patch
My last comment was a red herring. I found the bug... it s a logic bug. Patch has been attached. I will try to create a unit test but wanted to get the patch up since this was a critical bug.
> KahaDB doesn't clean up old files
> ---------------------------------
>
> Key: AMQ-2736
> URL: https://issues.apache.org/activemq/browse/AMQ-2736
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Reporter: Adrian Trenaman
> Priority: Critical
> Attachments: AMQ-2736.zip, amq-2987.patch
>
>
> Over time, we're seeing that kahadb doesn't clean up old journal files. As a result, we eventually run out of disk space, or rather, we hit our usage limits and our producers are slowed down by the producer flow control mechanism. Others have experienced this problem too (for example, see http://mail-archives.apache.org/mod_mbox/activemq-users/201002.mbox/%3C27502591.post@talk.nabble.com%3E)
> For now, we're moving back to the old amqPersistenceStore.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2736) KahaDB doesn't clean up old files
Posted by "Jeff Genender (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=63209#action_63209 ]
Jeff Genender commented on AMQ-2736:
------------------------------------
@Christer Wlkman - I built one with the following JIRA patches:
http://people.apache.org/~jgenender/apache-activemq-5.4.1-AMQ2736-AMQ2982-AMQ2983-AMQ2935-bin.tar.gz
YMMV and this is not an official release. Post back if this works for you.
> KahaDB doesn't clean up old files
> ---------------------------------
>
> Key: AMQ-2736
> URL: https://issues.apache.org/activemq/browse/AMQ-2736
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Reporter: Adrian Trenaman
> Assignee: Dejan Bosanac
> Priority: Critical
> Fix For: 5.5.0
>
> Attachments: AMQ-2736.zip, amq-2987-testcase.patch, amq-2987.patch, AMQ2736Test_should_with_this_diff.txt, MyKahaDBStore.java
>
>
> Over time, we're seeing that kahadb doesn't clean up old journal files. As a result, we eventually run out of disk space, or rather, we hit our usage limits and our producers are slowed down by the producer flow control mechanism. Others have experienced this problem too (for example, see http://mail-archives.apache.org/mod_mbox/activemq-users/201002.mbox/%3C27502591.post@talk.nabble.com%3E)
> For now, we're moving back to the old amqPersistenceStore.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2736) KahaDB doesn't clean up old files
Posted by "Giampaolo Tranchida (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=62856#action_62856 ]
Giampaolo Tranchida commented on AMQ-2736:
------------------------------------------
I think that the fix AMQ-2982 must fix also this issue.
> KahaDB doesn't clean up old files
> ---------------------------------
>
> Key: AMQ-2736
> URL: https://issues.apache.org/activemq/browse/AMQ-2736
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Reporter: Adrian Trenaman
> Priority: Critical
> Attachments: AMQ-2736.zip, amq-2987-testcase.patch, amq-2987.patch, AMQ2736Test_should_with_this_diff.txt, MyKahaDBStore.java
>
>
> Over time, we're seeing that kahadb doesn't clean up old journal files. As a result, we eventually run out of disk space, or rather, we hit our usage limits and our producers are slowed down by the producer flow control mechanism. Others have experienced this problem too (for example, see http://mail-archives.apache.org/mod_mbox/activemq-users/201002.mbox/%3C27502591.post@talk.nabble.com%3E)
> For now, we're moving back to the old amqPersistenceStore.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2736) KahaDB doesn't clean up old files
Posted by "Jeff Genender (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=62846#action_62846 ]
Jeff Genender commented on AMQ-2736:
------------------------------------
@Christer WIkman - Gary's attachment wasn't a patch... it was a revert to the test case to validate the test case fails. Please try my patch.
> KahaDB doesn't clean up old files
> ---------------------------------
>
> Key: AMQ-2736
> URL: https://issues.apache.org/activemq/browse/AMQ-2736
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Reporter: Adrian Trenaman
> Priority: Critical
> Attachments: AMQ-2736.zip, amq-2987-testcase.patch, amq-2987.patch, AMQ2736Test_should_with_this_diff.txt, MyKahaDBStore.java
>
>
> Over time, we're seeing that kahadb doesn't clean up old journal files. As a result, we eventually run out of disk space, or rather, we hit our usage limits and our producers are slowed down by the producer flow control mechanism. Others have experienced this problem too (for example, see http://mail-archives.apache.org/mod_mbox/activemq-users/201002.mbox/%3C27502591.post@talk.nabble.com%3E)
> For now, we're moving back to the old amqPersistenceStore.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2736) KahaDB doesn't clean up old files
Posted by "Christer WIkman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=62796#action_62796 ]
Christer WIkman commented on AMQ-2736:
--------------------------------------
I'm using ActiveMQ 5.4.1 in production and I have added Gary's patch, but the log files are stilling growing (even if all queues are empty) and finally Activemq crashes. If I restart ActiveMQ then the log files are deleted.
Any suggestions on how to fix the problem?
> KahaDB doesn't clean up old files
> ---------------------------------
>
> Key: AMQ-2736
> URL: https://issues.apache.org/activemq/browse/AMQ-2736
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Reporter: Adrian Trenaman
> Priority: Critical
> Attachments: AMQ-2736.zip, amq-2987-testcase.patch, amq-2987.patch, AMQ2736Test_should_with_this_diff.txt, MyKahaDBStore.java
>
>
> Over time, we're seeing that kahadb doesn't clean up old journal files. As a result, we eventually run out of disk space, or rather, we hit our usage limits and our producers are slowed down by the producer flow control mechanism. Others have experienced this problem too (for example, see http://mail-archives.apache.org/mod_mbox/activemq-users/201002.mbox/%3C27502591.post@talk.nabble.com%3E)
> For now, we're moving back to the old amqPersistenceStore.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2736) KahaDB doesn't clean up old files
Posted by "Jeff Genender (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=62853#action_62853 ]
Jeff Genender commented on AMQ-2736:
------------------------------------
Here is the 5.4.1 distribution with the patch in it. This is *not* an official release... could people please test it out?
http://people.apache.org/~jgenender/apache-activemq-5.4.1-AMQ2736-bin.tar.gz
> KahaDB doesn't clean up old files
> ---------------------------------
>
> Key: AMQ-2736
> URL: https://issues.apache.org/activemq/browse/AMQ-2736
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Reporter: Adrian Trenaman
> Priority: Critical
> Attachments: AMQ-2736.zip, amq-2987-testcase.patch, amq-2987.patch, AMQ2736Test_should_with_this_diff.txt, MyKahaDBStore.java
>
>
> Over time, we're seeing that kahadb doesn't clean up old journal files. As a result, we eventually run out of disk space, or rather, we hit our usage limits and our producers are slowed down by the producer flow control mechanism. Others have experienced this problem too (for example, see http://mail-archives.apache.org/mod_mbox/activemq-users/201002.mbox/%3C27502591.post@talk.nabble.com%3E)
> For now, we're moving back to the old amqPersistenceStore.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Reopened: (AMQ-2736) KahaDB doesn't clean up old files
Posted by "Straun (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Straun reopened AMQ-2736:
-------------------------
I am seeing this with a broker holding transactional persistent messages (100k messages a day 10k per msg), the queue sizes are all around 0 normally so I would expect the files to be cleared. I am using 5.4.0. The consumer of the messages is a .Net client with 'auto acknowledge' set. The KahaDB also appears to slow down as it runs, writes to the queues slows down so that the each commit cycle is taking 10% longer than the last.
> KahaDB doesn't clean up old files
> ---------------------------------
>
> Key: AMQ-2736
> URL: https://issues.apache.org/activemq/browse/AMQ-2736
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Reporter: Adrian Trenaman
> Priority: Critical
>
> Over time, we're seeing that kahadb doesn't clean up old journal files. As a result, we eventually run out of disk space, or rather, we hit our usage limits and our producers are slowed down by the producer flow control mechanism. Others have experienced this problem too (for example, see http://mail-archives.apache.org/mod_mbox/activemq-users/201002.mbox/%3C27502591.post@talk.nabble.com%3E)
> For now, we're moving back to the old amqPersistenceStore.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2736) KahaDB doesn't clean up old files
Posted by "Swen Moczarski (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=62813#action_62813 ]
Swen Moczarski commented on AMQ-2736:
-------------------------------------
We also had some trouble with growing log files (version 5.4.1). I have two issues identified with similar symptoms (but it seems to be another cause as to this issue):
* AMQ-2982: sticky log files on a rollback of a local transaction (test case and patch available)
* AMQ-2983: sticky log files on multiple concurrent consumers with local transacted sessions (test case available)
If we only use non transacted sessions with auto-acknowledge, we have no problems.
> KahaDB doesn't clean up old files
> ---------------------------------
>
> Key: AMQ-2736
> URL: https://issues.apache.org/activemq/browse/AMQ-2736
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Reporter: Adrian Trenaman
> Priority: Critical
> Attachments: AMQ-2736.zip, amq-2987-testcase.patch, amq-2987.patch, AMQ2736Test_should_with_this_diff.txt, MyKahaDBStore.java
>
>
> Over time, we're seeing that kahadb doesn't clean up old journal files. As a result, we eventually run out of disk space, or rather, we hit our usage limits and our producers are slowed down by the producer flow control mechanism. Others have experienced this problem too (for example, see http://mail-archives.apache.org/mod_mbox/activemq-users/201002.mbox/%3C27502591.post@talk.nabble.com%3E)
> For now, we're moving back to the old amqPersistenceStore.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AMQ-2736) KahaDB doesn't clean up old files
Posted by "Gary Tully (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gary Tully updated AMQ-2736:
----------------------------
Attachment: AMQ2736Test_should_with_this_diff.txt
attaching the diff with the test case and the fix reverted, in which case the test should fail. but it does not atm.
Just so we are on the same page, I may be missing something but I would expect it to fail.
> KahaDB doesn't clean up old files
> ---------------------------------
>
> Key: AMQ-2736
> URL: https://issues.apache.org/activemq/browse/AMQ-2736
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Reporter: Adrian Trenaman
> Priority: Critical
> Attachments: AMQ-2736.zip, amq-2987-testcase.patch, amq-2987.patch, AMQ2736Test_should_with_this_diff.txt
>
>
> Over time, we're seeing that kahadb doesn't clean up old journal files. As a result, we eventually run out of disk space, or rather, we hit our usage limits and our producers are slowed down by the producer flow control mechanism. Others have experienced this problem too (for example, see http://mail-archives.apache.org/mod_mbox/activemq-users/201002.mbox/%3C27502591.post@talk.nabble.com%3E)
> For now, we're moving back to the old amqPersistenceStore.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Reopened: (AMQ-2736) KahaDB doesn't clean up old files
Posted by "Christer WIkman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Christer WIkman reopened AMQ-2736:
----------------------------------
We running: http://people.apache.org/~jgenender/apache-activemq-5.4.1-AMQ2736-bin.tar.gz and we are still experience problems with log files that are not deleted.
I deployed the fix yesterday with a new kahadb and this morning I had 155 log files with only 43 messages on two DLQ queues. I deleted the DLQ queues and only 12 log files where deleted. Currently,. I have 144 log files and nothing in the queues.
> KahaDB doesn't clean up old files
> ---------------------------------
>
> Key: AMQ-2736
> URL: https://issues.apache.org/activemq/browse/AMQ-2736
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Reporter: Adrian Trenaman
> Assignee: Dejan Bosanac
> Priority: Critical
> Fix For: 5.5.0
>
> Attachments: AMQ-2736.zip, amq-2987-testcase.patch, amq-2987.patch, AMQ2736Test_should_with_this_diff.txt, MyKahaDBStore.java
>
>
> Over time, we're seeing that kahadb doesn't clean up old journal files. As a result, we eventually run out of disk space, or rather, we hit our usage limits and our producers are slowed down by the producer flow control mechanism. Others have experienced this problem too (for example, see http://mail-archives.apache.org/mod_mbox/activemq-users/201002.mbox/%3C27502591.post@talk.nabble.com%3E)
> For now, we're moving back to the old amqPersistenceStore.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2736) KahaDB doesn't clean up old files
Posted by "Steve Pietrowicz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=63379#action_63379 ]
Steve Pietrowicz commented on AMQ-2736:
---------------------------------------
I've been trying to track down a solution to a problem similar to this, and the latest snapshot of 5.5 doesn't resolve this for me.
This is the scenario I've created:
1) a producer starts sending messages
2) a consumer starts consuming messages, but pauses after 100.
3) db-##.log files start showing up in the tmp_storage directory as expected. I let the producer go for a while so a number of db-##.log files are created.
4) the consumer starts consuming messages again. The db-##.log files start disappearing as they're used up.
5) everything finishes.
Now, if I do:
1) a producer starts sending messages
2) a consumer starts consuming messages, but pauses after 100.
3) db-##.log files start showing up in the tmp_storage directory as expected. I let the producer go for a while so that a number of db-##.log files are created.
and kill the consumer and the producer. The db-##.log files remain, and will not get cleaned up, even on subsequent invocations of new consumers and producers. If a new consumer/producer (as in the first scenario) are created, it all works; it's just that those old .log files never get cleaned up.
> KahaDB doesn't clean up old files
> ---------------------------------
>
> Key: AMQ-2736
> URL: https://issues.apache.org/activemq/browse/AMQ-2736
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Reporter: Adrian Trenaman
> Assignee: Dejan Bosanac
> Priority: Critical
> Fix For: 5.5.0
>
> Attachments: AMQ-2736.zip, amq-2987-testcase.patch, amq-2987.patch, AMQ2736Test_should_with_this_diff.txt, MyKahaDBStore.java
>
>
> Over time, we're seeing that kahadb doesn't clean up old journal files. As a result, we eventually run out of disk space, or rather, we hit our usage limits and our producers are slowed down by the producer flow control mechanism. Others have experienced this problem too (for example, see http://mail-archives.apache.org/mod_mbox/activemq-users/201002.mbox/%3C27502591.post@talk.nabble.com%3E)
> For now, we're moving back to the old amqPersistenceStore.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2736) KahaDB doesn't clean up old files
Posted by "Eduardo Costa (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=62560#action_62560 ]
Eduardo Costa commented on AMQ-2736:
------------------------------------
After some evil debugging tricks, I found that there are some pending inflight transactions (even after restarting the server) in my database files. In this scenario, the gcCandidadeSet is empty, since "metadata.firstInProgressTransactionLocation" points to the first log file (MessageDatabase.java, lines 1140-1154):
{code}// Don't GC files after the first in progress tx
Location firstTxLocation = metadata.lastUpdate;
if( metadata.firstInProgressTransactionLocation!=null ) {
firstTxLocation = metadata.firstInProgressTransactionLocation;
}
if( firstTxLocation!=null ) {
while( !gcCandidateSet.isEmpty() ) {
Integer last = gcCandidateSet.last();
if( last >= firstTxLocation.getDataFileId() ) {
gcCandidateSet.remove(last);
} else {
break;
}
}
}
{code}
This leads to the question: why it must discard every log file starting from the one containing the first pending transaction? Any malfunctioning operation (even an one-time little mistake) will trash all log files from that point!
I believe this should happens also on "kill -9" or unexpected power down.
> KahaDB doesn't clean up old files
> ---------------------------------
>
> Key: AMQ-2736
> URL: https://issues.apache.org/activemq/browse/AMQ-2736
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Reporter: Adrian Trenaman
> Priority: Critical
> Attachments: AMQ-2736.zip, amq-2987-testcase.patch, amq-2987.patch, AMQ2736Test_should_with_this_diff.txt
>
>
> Over time, we're seeing that kahadb doesn't clean up old journal files. As a result, we eventually run out of disk space, or rather, we hit our usage limits and our producers are slowed down by the producer flow control mechanism. Others have experienced this problem too (for example, see http://mail-archives.apache.org/mod_mbox/activemq-users/201002.mbox/%3C27502591.post@talk.nabble.com%3E)
> For now, we're moving back to the old amqPersistenceStore.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2736) KahaDB doesn't clean up old files
Posted by "Christer WIkman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=63256#action_63256 ]
Christer WIkman commented on AMQ-2736:
--------------------------------------
@Jeff - Thanks! I have deployed your fix in our stage environment. It works much better but not 100%. All queues are empty but I still have 4 log files and one contains 0 bytes, last updated yesterday, and other log file was last updated 6 hours ago. When I used 5.5-SNAPSHOT it went down to 1 log file!
I will evaluate it for some more days before I can tell for sure if the log files grows.
> KahaDB doesn't clean up old files
> ---------------------------------
>
> Key: AMQ-2736
> URL: https://issues.apache.org/activemq/browse/AMQ-2736
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Reporter: Adrian Trenaman
> Assignee: Dejan Bosanac
> Priority: Critical
> Fix For: 5.5.0
>
> Attachments: AMQ-2736.zip, amq-2987-testcase.patch, amq-2987.patch, AMQ2736Test_should_with_this_diff.txt, MyKahaDBStore.java
>
>
> Over time, we're seeing that kahadb doesn't clean up old journal files. As a result, we eventually run out of disk space, or rather, we hit our usage limits and our producers are slowed down by the producer flow control mechanism. Others have experienced this problem too (for example, see http://mail-archives.apache.org/mod_mbox/activemq-users/201002.mbox/%3C27502591.post@talk.nabble.com%3E)
> For now, we're moving back to the old amqPersistenceStore.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2736) KahaDB doesn't clean up old files
Posted by "Gary Tully (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=59414#action_59414 ]
Gary Tully commented on AMQ-2736:
---------------------------------
Could you provide a test case or scenario description that can reproduce?
> KahaDB doesn't clean up old files
> ---------------------------------
>
> Key: AMQ-2736
> URL: https://issues.apache.org/activemq/browse/AMQ-2736
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Reporter: Adrian Trenaman
> Priority: Critical
>
> Over time, we're seeing that kahadb doesn't clean up old journal files. As a result, we eventually run out of disk space, or rather, we hit our usage limits and our producers are slowed down by the producer flow control mechanism. Others have experienced this problem too (for example, see http://mail-archives.apache.org/mod_mbox/activemq-users/201002.mbox/%3C27502591.post@talk.nabble.com%3E)
> For now, we're moving back to the old amqPersistenceStore.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2736) KahaDB doesn't clean up old files
Posted by "Giampaolo Tranchida (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=62855#action_62855 ]
Giampaolo Tranchida commented on AMQ-2736:
------------------------------------------
Do not work for me, same issue.
> KahaDB doesn't clean up old files
> ---------------------------------
>
> Key: AMQ-2736
> URL: https://issues.apache.org/activemq/browse/AMQ-2736
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Reporter: Adrian Trenaman
> Priority: Critical
> Attachments: AMQ-2736.zip, amq-2987-testcase.patch, amq-2987.patch, AMQ2736Test_should_with_this_diff.txt, MyKahaDBStore.java
>
>
> Over time, we're seeing that kahadb doesn't clean up old journal files. As a result, we eventually run out of disk space, or rather, we hit our usage limits and our producers are slowed down by the producer flow control mechanism. Others have experienced this problem too (for example, see http://mail-archives.apache.org/mod_mbox/activemq-users/201002.mbox/%3C27502591.post@talk.nabble.com%3E)
> For now, we're moving back to the old amqPersistenceStore.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2736) KahaDB doesn't clean up old files
Posted by "Gary Tully (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=62522#action_62522 ]
Gary Tully commented on AMQ-2736:
---------------------------------
The scenario that needs to be tested, for future reference:{code}ackMessageFileMap = {2=[1], 3=[1, 2], 4=[2, 3, 4], 5=[4, 5], 6=[5, 6], 7=[6, 7], 8=[7, 8]}
gCandidateSet = [2, 3, 4, 5, 6, 7]{code}
> KahaDB doesn't clean up old files
> ---------------------------------
>
> Key: AMQ-2736
> URL: https://issues.apache.org/activemq/browse/AMQ-2736
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Reporter: Adrian Trenaman
> Priority: Critical
> Attachments: AMQ-2736.zip, amq-2987-testcase.patch, amq-2987.patch, AMQ2736Test_should_with_this_diff.txt
>
>
> Over time, we're seeing that kahadb doesn't clean up old journal files. As a result, we eventually run out of disk space, or rather, we hit our usage limits and our producers are slowed down by the producer flow control mechanism. Others have experienced this problem too (for example, see http://mail-archives.apache.org/mod_mbox/activemq-users/201002.mbox/%3C27502591.post@talk.nabble.com%3E)
> For now, we're moving back to the old amqPersistenceStore.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2736) KahaDB doesn't clean up old files
Posted by "Jeff Genender (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=62430#action_62430 ]
Jeff Genender commented on AMQ-2736:
------------------------------------
I ran the attached example and was able to reproduce it and indeed appears to endlessly grow. I did a thread dump during the condition and there appears to be a possible locking problem with multiple "KahaDB Scheduler" threads.
As an experiment I killed one of the KahaDB Schedulers and it seemed to clean up. I ran the example several times and it seems to clean up. So this *may* be the culprit.
So please try this as a work around...
In the broker configuration set schedulerSupport="false" and see if the files clean up. This may help until we identify the waits on multiple schedulers.
> KahaDB doesn't clean up old files
> ---------------------------------
>
> Key: AMQ-2736
> URL: https://issues.apache.org/activemq/browse/AMQ-2736
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Reporter: Adrian Trenaman
> Priority: Critical
> Attachments: AMQ-2736.zip
>
>
> Over time, we're seeing that kahadb doesn't clean up old journal files. As a result, we eventually run out of disk space, or rather, we hit our usage limits and our producers are slowed down by the producer flow control mechanism. Others have experienced this problem too (for example, see http://mail-archives.apache.org/mod_mbox/activemq-users/201002.mbox/%3C27502591.post@talk.nabble.com%3E)
> For now, we're moving back to the old amqPersistenceStore.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AMQ-2736) KahaDB doesn't clean up old files
Posted by "Jeff Genender (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeff Genender updated AMQ-2736:
-------------------------------
Attachment: amq-2987-testcase.patch
Test case attached
> KahaDB doesn't clean up old files
> ---------------------------------
>
> Key: AMQ-2736
> URL: https://issues.apache.org/activemq/browse/AMQ-2736
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Reporter: Adrian Trenaman
> Priority: Critical
> Attachments: AMQ-2736.zip, amq-2987-testcase.patch, amq-2987.patch
>
>
> Over time, we're seeing that kahadb doesn't clean up old journal files. As a result, we eventually run out of disk space, or rather, we hit our usage limits and our producers are slowed down by the producer flow control mechanism. Others have experienced this problem too (for example, see http://mail-archives.apache.org/mod_mbox/activemq-users/201002.mbox/%3C27502591.post@talk.nabble.com%3E)
> For now, we're moving back to the old amqPersistenceStore.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2736) KahaDB doesn't clean up old files
Posted by "Jeff Genender (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=62431#action_62431 ]
Jeff Genender commented on AMQ-2736:
------------------------------------
I ran the attached example and was able to reproduce it and indeed appears to endlessly grow. I did a thread dump during the condition and there appears to be a possible locking problem with multiple "KahaDB Scheduler" threads.
As an experiment I killed one of the KahaDB Schedulers and it seemed to clean up. I ran the example several times and it seems to clean up. So this *may* be the culprit.
So please try this as a work around...
In the broker configuration set schedulerSupport="false" and see if the files clean up. This may help until we identify the waits on multiple schedulers.
> KahaDB doesn't clean up old files
> ---------------------------------
>
> Key: AMQ-2736
> URL: https://issues.apache.org/activemq/browse/AMQ-2736
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Reporter: Adrian Trenaman
> Priority: Critical
> Attachments: AMQ-2736.zip
>
>
> Over time, we're seeing that kahadb doesn't clean up old journal files. As a result, we eventually run out of disk space, or rather, we hit our usage limits and our producers are slowed down by the producer flow control mechanism. Others have experienced this problem too (for example, see http://mail-archives.apache.org/mod_mbox/activemq-users/201002.mbox/%3C27502591.post@talk.nabble.com%3E)
> For now, we're moving back to the old amqPersistenceStore.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2736) KahaDB doesn't clean up old files
Posted by "Jeff Genender (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=62845#action_62845 ]
Jeff Genender commented on AMQ-2736:
------------------------------------
Folks..
Please try the patch supplied above: https://issues.apache.org/activemq/secure/attachment/19715/amq-2987.patch
That appears to fix the problem with the AMQ-2736.zip test.
> KahaDB doesn't clean up old files
> ---------------------------------
>
> Key: AMQ-2736
> URL: https://issues.apache.org/activemq/browse/AMQ-2736
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Reporter: Adrian Trenaman
> Priority: Critical
> Attachments: AMQ-2736.zip, amq-2987-testcase.patch, amq-2987.patch, AMQ2736Test_should_with_this_diff.txt, MyKahaDBStore.java
>
>
> Over time, we're seeing that kahadb doesn't clean up old journal files. As a result, we eventually run out of disk space, or rather, we hit our usage limits and our producers are slowed down by the producer flow control mechanism. Others have experienced this problem too (for example, see http://mail-archives.apache.org/mod_mbox/activemq-users/201002.mbox/%3C27502591.post@talk.nabble.com%3E)
> For now, we're moving back to the old amqPersistenceStore.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2736) KahaDB doesn't clean up old files
Posted by "Gary Tully (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=61888#action_61888 ]
Gary Tully commented on AMQ-2736:
---------------------------------
Straun, can you provide some sort of test case that can reproduce what you are seeing? Maybe a simple java producer/consumer that mimics what your .Net add does.
> KahaDB doesn't clean up old files
> ---------------------------------
>
> Key: AMQ-2736
> URL: https://issues.apache.org/activemq/browse/AMQ-2736
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Reporter: Adrian Trenaman
> Priority: Critical
>
> Over time, we're seeing that kahadb doesn't clean up old journal files. As a result, we eventually run out of disk space, or rather, we hit our usage limits and our producers are slowed down by the producer flow control mechanism. Others have experienced this problem too (for example, see http://mail-archives.apache.org/mod_mbox/activemq-users/201002.mbox/%3C27502591.post@talk.nabble.com%3E)
> For now, we're moving back to the old amqPersistenceStore.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2736) KahaDB doesn't clean up old files
Posted by "Dejan Bosanac (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=63369#action_63369 ]
Dejan Bosanac commented on AMQ-2736:
------------------------------------
BTW. We released a version of Fuse Message Broker that contains a fix for this issue. If anyone is interested, it can be found here
http://repo.fusesource.com/nexus/content/repositories/releases/org/apache/activemq/apache-activemq/5.4.1-fuse-01-00/
Release notes available at: http://fusesource.com/wiki/display/ProdInfo/FUSE+Message+Broker+v5.4.1-fuse+Release+Notes
> KahaDB doesn't clean up old files
> ---------------------------------
>
> Key: AMQ-2736
> URL: https://issues.apache.org/activemq/browse/AMQ-2736
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Reporter: Adrian Trenaman
> Assignee: Dejan Bosanac
> Priority: Critical
> Fix For: 5.5.0
>
> Attachments: AMQ-2736.zip, amq-2987-testcase.patch, amq-2987.patch, AMQ2736Test_should_with_this_diff.txt, MyKahaDBStore.java
>
>
> Over time, we're seeing that kahadb doesn't clean up old journal files. As a result, we eventually run out of disk space, or rather, we hit our usage limits and our producers are slowed down by the producer flow control mechanism. Others have experienced this problem too (for example, see http://mail-archives.apache.org/mod_mbox/activemq-users/201002.mbox/%3C27502591.post@talk.nabble.com%3E)
> For now, we're moving back to the old amqPersistenceStore.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2736) KahaDB doesn't clean up old files
Posted by "Jeff Genender (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=62429#action_62429 ]
Jeff Genender commented on AMQ-2736:
------------------------------------
Looks like there *maybe* a wait/lock condition on multiple "KahaDB Scheduler" threads...
I took the attached example and was able to reproduce the problem. A thread dump showed waits on the "KahaDB Scheduler". I knocked out one of the schedulers and it appears to clean up.
Hence... can you please try setting schedulerSupport="false" on the broker configuration and see if the files no longer grow? This may be a work around for the time being. Hopefully this is in the right direction...
> KahaDB doesn't clean up old files
> ---------------------------------
>
> Key: AMQ-2736
> URL: https://issues.apache.org/activemq/browse/AMQ-2736
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Reporter: Adrian Trenaman
> Priority: Critical
> Attachments: AMQ-2736.zip
>
>
> Over time, we're seeing that kahadb doesn't clean up old journal files. As a result, we eventually run out of disk space, or rather, we hit our usage limits and our producers are slowed down by the producer flow control mechanism. Others have experienced this problem too (for example, see http://mail-archives.apache.org/mod_mbox/activemq-users/201002.mbox/%3C27502591.post@talk.nabble.com%3E)
> For now, we're moving back to the old amqPersistenceStore.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AMQ-2736) KahaDB doesn't clean up old files
Posted by "Eduardo Costa (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eduardo Costa updated AMQ-2736:
-------------------------------
Attachment: AMQ-2736.zip
I'm attaching a test case that reproduces what I got here.
I tested with a Producer sending 1mi messages (in 100-message blocks) and a Consumer. After some time, I receive a "Usage Manager Store is Full, 100% of 1073741824. Stopping producer (...)" with only a hundred messages on the queue!
> KahaDB doesn't clean up old files
> ---------------------------------
>
> Key: AMQ-2736
> URL: https://issues.apache.org/activemq/browse/AMQ-2736
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Reporter: Adrian Trenaman
> Priority: Critical
> Attachments: AMQ-2736.zip
>
>
> Over time, we're seeing that kahadb doesn't clean up old journal files. As a result, we eventually run out of disk space, or rather, we hit our usage limits and our producers are slowed down by the producer flow control mechanism. Others have experienced this problem too (for example, see http://mail-archives.apache.org/mod_mbox/activemq-users/201002.mbox/%3C27502591.post@talk.nabble.com%3E)
> For now, we're moving back to the old amqPersistenceStore.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2736) KahaDB doesn't clean up old files
Posted by "Adrian Trenaman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=59420#action_59420 ]
Adrian Trenaman commented on AMQ-2736:
--------------------------------------
Hi Gary,
I'm afraid I don't have a test case for this right now; just observed
behaviour on our systems here -and- verified by other users via a Google
Search.
/Ade
> KahaDB doesn't clean up old files
> ---------------------------------
>
> Key: AMQ-2736
> URL: https://issues.apache.org/activemq/browse/AMQ-2736
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Reporter: Adrian Trenaman
> Priority: Critical
>
> Over time, we're seeing that kahadb doesn't clean up old journal files. As a result, we eventually run out of disk space, or rather, we hit our usage limits and our producers are slowed down by the producer flow control mechanism. Others have experienced this problem too (for example, see http://mail-archives.apache.org/mod_mbox/activemq-users/201002.mbox/%3C27502591.post@talk.nabble.com%3E)
> For now, we're moving back to the old amqPersistenceStore.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2736) KahaDB doesn't clean up old files
Posted by "Gary Tully (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=62457#action_62457 ]
Gary Tully commented on AMQ-2736:
---------------------------------
jeff, committed the fix, thanks. r1005806
await the test case to protect it. nice work.
> KahaDB doesn't clean up old files
> ---------------------------------
>
> Key: AMQ-2736
> URL: https://issues.apache.org/activemq/browse/AMQ-2736
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Reporter: Adrian Trenaman
> Priority: Critical
> Attachments: AMQ-2736.zip, amq-2987.patch
>
>
> Over time, we're seeing that kahadb doesn't clean up old journal files. As a result, we eventually run out of disk space, or rather, we hit our usage limits and our producers are slowed down by the producer flow control mechanism. Others have experienced this problem too (for example, see http://mail-archives.apache.org/mod_mbox/activemq-users/201002.mbox/%3C27502591.post@talk.nabble.com%3E)
> For now, we're moving back to the old amqPersistenceStore.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AMQ-2736) KahaDB doesn't clean up old files
Posted by "Eduardo Costa (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eduardo Costa updated AMQ-2736:
-------------------------------
Attachment: MyKahaDBStore.java
I'm attaching a nasty workaround that commits the pending transactions and automatically archives (or deletes) the old log files when closing the store. I'm not proud of this thing, but it seems to prove the problem is on the pending transaction, since all useless files are correctly disposed after running it.
> KahaDB doesn't clean up old files
> ---------------------------------
>
> Key: AMQ-2736
> URL: https://issues.apache.org/activemq/browse/AMQ-2736
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Reporter: Adrian Trenaman
> Priority: Critical
> Attachments: AMQ-2736.zip, amq-2987-testcase.patch, amq-2987.patch, AMQ2736Test_should_with_this_diff.txt, MyKahaDBStore.java
>
>
> Over time, we're seeing that kahadb doesn't clean up old journal files. As a result, we eventually run out of disk space, or rather, we hit our usage limits and our producers are slowed down by the producer flow control mechanism. Others have experienced this problem too (for example, see http://mail-archives.apache.org/mod_mbox/activemq-users/201002.mbox/%3C27502591.post@talk.nabble.com%3E)
> For now, we're moving back to the old amqPersistenceStore.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.