You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by "B.D." <bo...@gmail.com> on 2010/02/03 20:25:16 UTC
KahaDB data files not cleaned up
Hi,
I’m using ActiveMQ 5.3 with KahaDB. Two brokers are in a network with
failover. Publishers are sending messages to a composite destination that
forwards to a topic or a queue. It works ok with topics but with queues
after a while there is accumulation of db-xxx.log files in the data/kahadb
directory.
Files will be several days old despite the fact that all messages have a
one-day time to live. Dead letter strategy is individualDeadLetterStrategy
with processExpired="false" so I do not expect expired messages to stay
around in any form. Purging queues using admin console does not get rid of
these files. The only way I found to clean it up was to stop the broker and
clean up the whole data directory.
I could reproduce this problem in a test setup by starting/stopping brokers,
publishers and subscribers. Older data/kahadb/db-X.log files are normally
getting deleted but some are not, see below. In these tests time-to-live on
all messages was 10 minutes so this should not be happening. See directory
lists below.
With a single broker I could not reproduce that same problem. Older db-X.log
files were always eventually getting cleaned up. However I noticed that
db.data file can grow. It reached 360M while it normally stays below 34M.
Any suggestion how to prevent growth of files in the data/kahadb directory?
Thanks,
Broker 1 in a 2-broker network:
33555255 2010-01-07 23:18:13 data/kahadb/db-80.log
33030144 2010-01-08 07:46:16 data/kahadb/db-177.log
33554509 2010-01-07 22:50:21 data/kahadb/db-74.log
33555184 2010-01-08 01:34:55 data/kahadb/db-110.log
33554441 2010-01-07 22:41:00 data/kahadb/db-72.log
33554519 2010-01-08 04:40:53 data/kahadb/db-144.log
33554522 2010-01-07 19:51:33 data/kahadb/db-34.log
33555005 2010-01-08 04:09:10 data/kahadb/db-138.log
33555019 2010-01-08 02:07:52 data/kahadb/db-117.log
33554494 2010-01-07 18:52:05 data/kahadb/db-21.log
33555122 2010-01-08 04:30:02 data/kahadb/db-142.log
3287296 2010-01-08 07:46:16 data/kahadb/db.redo
33554733 2010-01-08 00:40:15 data/kahadb/db-98.log
33554703 2010-01-08 07:34:38 data/kahadb/db-175.log
33554997 2010-01-08 07:26:41 data/kahadb/db-174.log
10763 2010-01-08 07:46:16 data/kahadb/db.free
33555298 2010-01-07 23:04:18 data/kahadb/db-77.log
33554537 2010-01-07 23:45:26 data/kahadb/db-86.log
23187456 2010-01-08 07:46:16 data/kahadb/db.data
33555022 2010-01-08 02:52:40 data/kahadb/db-125.log
Broker 2 in a 2-broker network:
33030144 2010-01-08 07:46:15 data/kahadb/db-172.log
1756504 2010-01-08 12:07:34 data/kahadb/db.redo
33554527 2010-01-07 18:29:16 data/kahadb/db-3.log
33554595 2010-01-07 17:49:28 data/kahadb/db-1.log
0 2010-01-07 16:25:00 data/kahadb/lock
2543616 2010-01-08 12:07:34 data/kahadb/db.data
33554596 2010-01-07 23:56:35 data/kahadb/db-79.log
Single broker (not a network):
33030144 Jan 14 18:25 db-1318.log
364490752 Feb 3 10:22 db.data
3287296 Feb 3 10:22 db.redo
0 Jan 8 12:33 lock
--
View this message in context: http://old.nabble.com/KahaDB-data-files-not-cleaned-up-tp27441942p27441942.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
RE: KahaDB data files not cleaned up
Posted by alisenberg <li...@gmail.com>.
Daniel Kluesing-2 wrote:
>
> I'd be interested as well.
>
> I'm not using TTL, so I assumed my orphaned/non-sequence log files were
> due to out of order message consumption, but I also see the occasional log
> file hang around longer than expected. (say db-50.log then next is
> db-400.log, so big gap)
>
> -----Original Message-----
> From: B.D. [mailto:boris.dainson@gmail.com]
> Sent: Monday, February 08, 2010 8:39 AM
> To: users@activemq.apache.org
> Subject: Re: KahaDB data files not cleaned up
>
>
> Any suggestions on how to prevent pileups of data/kahadb/db-xxx.log files?
>
> I just had to clean up 301 data/kahadb/db-xxx.log files that accumulated
> on
> a QA machine over 6 days. Message time to live is one day so that should
> not
> be happening.
>
> Logs show KahaDB cleanup was taking longer than expected:
> activemq.log:2010-02-08 13:31:29,664 | INFO | Slow KahaDB access: cleanup
> took 591 | org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ
> Journal Checkpoint Worker
>
>
> Thanks
>
>
>
> B.D. wrote:
>>
>> Hi,
>>
>> I’m using ActiveMQ 5.3 with KahaDB. Two brokers are in a network with
>> failover. Publishers are sending messages to a composite destination that
>> forwards to a topic or a queue. It works ok with topics but with queues
>> after a while there is accumulation of db-xxx.log files in the
>> data/kahadb
>> directory.
>>
>> Files will be several days old despite the fact that all messages have a
>> one-day time to live. Dead letter strategy is
>> individualDeadLetterStrategy
>> with processExpired="false" so I do not expect expired messages to stay
>> around in any form. Purging queues using admin console does not get rid
>> of
>> these files. The only way I found to clean it up was to stop the broker
>> and clean up the whole data directory.
>>
>> I could reproduce this problem in a test setup by starting/stopping
>> brokers, publishers and subscribers. Older data/kahadb/db-X.log files are
>> normally getting deleted but some are not, see below. In these tests
>> time-to-live on all messages was 10 minutes so this should not be
>> happening. See directory lists below.
>>
>> With a single broker I could not reproduce that same problem. Older
>> db-X.log files were always eventually getting cleaned up. However I
>> noticed that db.data file can grow. It reached 360M while it normally
>> stays below 34M.
>>
>> Any suggestion how to prevent growth of files in the data/kahadb
>> directory?
>>
>> Thanks,
>>
>>
>> Broker 1 in a 2-broker network:
>> 33555255 2010-01-07 23:18:13 data/kahadb/db-80.log
>> 33030144 2010-01-08 07:46:16 data/kahadb/db-177.log
>> 33554509 2010-01-07 22:50:21 data/kahadb/db-74.log
>> 33555184 2010-01-08 01:34:55 data/kahadb/db-110.log
>> 33554441 2010-01-07 22:41:00 data/kahadb/db-72.log
>> 33554519 2010-01-08 04:40:53 data/kahadb/db-144.log
>> 33554522 2010-01-07 19:51:33 data/kahadb/db-34.log
>> 33555005 2010-01-08 04:09:10 data/kahadb/db-138.log
>> 33555019 2010-01-08 02:07:52 data/kahadb/db-117.log
>> 33554494 2010-01-07 18:52:05 data/kahadb/db-21.log
>> 33555122 2010-01-08 04:30:02 data/kahadb/db-142.log
>> 3287296 2010-01-08 07:46:16 data/kahadb/db.redo
>> 33554733 2010-01-08 00:40:15 data/kahadb/db-98.log
>> 33554703 2010-01-08 07:34:38 data/kahadb/db-175.log
>> 33554997 2010-01-08 07:26:41 data/kahadb/db-174.log
>> 10763 2010-01-08 07:46:16 data/kahadb/db.free
>> 33555298 2010-01-07 23:04:18 data/kahadb/db-77.log
>> 33554537 2010-01-07 23:45:26 data/kahadb/db-86.log
>> 23187456 2010-01-08 07:46:16 data/kahadb/db.data
>> 33555022 2010-01-08 02:52:40 data/kahadb/db-125.log
>>
>>
>> Broker 2 in a 2-broker network:
>> 33030144 2010-01-08 07:46:15 data/kahadb/db-172.log
>> 1756504 2010-01-08 12:07:34 data/kahadb/db.redo
>> 33554527 2010-01-07 18:29:16 data/kahadb/db-3.log
>> 33554595 2010-01-07 17:49:28 data/kahadb/db-1.log
>> 0 2010-01-07 16:25:00 data/kahadb/lock
>> 2543616 2010-01-08 12:07:34 data/kahadb/db.data
>> 33554596 2010-01-07 23:56:35 data/kahadb/db-79.log
>>
>>
>> Single broker (not a network):
>> 33030144 Jan 14 18:25 db-1318.log
>> 364490752 Feb 3 10:22 db.data
>> 3287296 Feb 3 10:22 db.redo
>> 0 Jan 8 12:33 lock
>>
>>
>
> --
> View this message in context:
> http://old.nabble.com/KahaDB-data-files-not-cleaned-up-tp27441942p27502591.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
>
I have the same problem...
--
View this message in context: http://old.nabble.com/KahaDB-data-files-not-cleaned-up-tp27441942p28391322.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
RE: KahaDB data files not cleaned up
Posted by Daniel Kluesing <dk...@bluekai.com>.
I'd be interested as well.
I'm not using TTL, so I assumed my orphaned/non-sequence log files were due to out of order message consumption, but I also see the occasional log file hang around longer than expected. (say db-50.log then next is db-400.log, so big gap)
-----Original Message-----
From: B.D. [mailto:boris.dainson@gmail.com]
Sent: Monday, February 08, 2010 8:39 AM
To: users@activemq.apache.org
Subject: Re: KahaDB data files not cleaned up
Any suggestions on how to prevent pileups of data/kahadb/db-xxx.log files?
I just had to clean up 301 data/kahadb/db-xxx.log files that accumulated on
a QA machine over 6 days. Message time to live is one day so that should not
be happening.
Logs show KahaDB cleanup was taking longer than expected:
activemq.log:2010-02-08 13:31:29,664 | INFO | Slow KahaDB access: cleanup
took 591 | org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ
Journal Checkpoint Worker
Thanks
B.D. wrote:
>
> Hi,
>
> I’m using ActiveMQ 5.3 with KahaDB. Two brokers are in a network with
> failover. Publishers are sending messages to a composite destination that
> forwards to a topic or a queue. It works ok with topics but with queues
> after a while there is accumulation of db-xxx.log files in the data/kahadb
> directory.
>
> Files will be several days old despite the fact that all messages have a
> one-day time to live. Dead letter strategy is individualDeadLetterStrategy
> with processExpired="false" so I do not expect expired messages to stay
> around in any form. Purging queues using admin console does not get rid of
> these files. The only way I found to clean it up was to stop the broker
> and clean up the whole data directory.
>
> I could reproduce this problem in a test setup by starting/stopping
> brokers, publishers and subscribers. Older data/kahadb/db-X.log files are
> normally getting deleted but some are not, see below. In these tests
> time-to-live on all messages was 10 minutes so this should not be
> happening. See directory lists below.
>
> With a single broker I could not reproduce that same problem. Older
> db-X.log files were always eventually getting cleaned up. However I
> noticed that db.data file can grow. It reached 360M while it normally
> stays below 34M.
>
> Any suggestion how to prevent growth of files in the data/kahadb
> directory?
>
> Thanks,
>
>
> Broker 1 in a 2-broker network:
> 33555255 2010-01-07 23:18:13 data/kahadb/db-80.log
> 33030144 2010-01-08 07:46:16 data/kahadb/db-177.log
> 33554509 2010-01-07 22:50:21 data/kahadb/db-74.log
> 33555184 2010-01-08 01:34:55 data/kahadb/db-110.log
> 33554441 2010-01-07 22:41:00 data/kahadb/db-72.log
> 33554519 2010-01-08 04:40:53 data/kahadb/db-144.log
> 33554522 2010-01-07 19:51:33 data/kahadb/db-34.log
> 33555005 2010-01-08 04:09:10 data/kahadb/db-138.log
> 33555019 2010-01-08 02:07:52 data/kahadb/db-117.log
> 33554494 2010-01-07 18:52:05 data/kahadb/db-21.log
> 33555122 2010-01-08 04:30:02 data/kahadb/db-142.log
> 3287296 2010-01-08 07:46:16 data/kahadb/db.redo
> 33554733 2010-01-08 00:40:15 data/kahadb/db-98.log
> 33554703 2010-01-08 07:34:38 data/kahadb/db-175.log
> 33554997 2010-01-08 07:26:41 data/kahadb/db-174.log
> 10763 2010-01-08 07:46:16 data/kahadb/db.free
> 33555298 2010-01-07 23:04:18 data/kahadb/db-77.log
> 33554537 2010-01-07 23:45:26 data/kahadb/db-86.log
> 23187456 2010-01-08 07:46:16 data/kahadb/db.data
> 33555022 2010-01-08 02:52:40 data/kahadb/db-125.log
>
>
> Broker 2 in a 2-broker network:
> 33030144 2010-01-08 07:46:15 data/kahadb/db-172.log
> 1756504 2010-01-08 12:07:34 data/kahadb/db.redo
> 33554527 2010-01-07 18:29:16 data/kahadb/db-3.log
> 33554595 2010-01-07 17:49:28 data/kahadb/db-1.log
> 0 2010-01-07 16:25:00 data/kahadb/lock
> 2543616 2010-01-08 12:07:34 data/kahadb/db.data
> 33554596 2010-01-07 23:56:35 data/kahadb/db-79.log
>
>
> Single broker (not a network):
> 33030144 Jan 14 18:25 db-1318.log
> 364490752 Feb 3 10:22 db.data
> 3287296 Feb 3 10:22 db.redo
> 0 Jan 8 12:33 lock
>
>
--
View this message in context: http://old.nabble.com/KahaDB-data-files-not-cleaned-up-tp27441942p27502591.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: KahaDB data files not cleaned up
Posted by "B.D." <bo...@gmail.com>.
Any suggestions on how to prevent pileups of data/kahadb/db-xxx.log files?
I just had to clean up 301 data/kahadb/db-xxx.log files that accumulated on
a QA machine over 6 days. Message time to live is one day so that should not
be happening.
Logs show KahaDB cleanup was taking longer than expected:
activemq.log:2010-02-08 13:31:29,664 | INFO | Slow KahaDB access: cleanup
took 591 | org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ
Journal Checkpoint Worker
Thanks
B.D. wrote:
>
> Hi,
>
> I’m using ActiveMQ 5.3 with KahaDB. Two brokers are in a network with
> failover. Publishers are sending messages to a composite destination that
> forwards to a topic or a queue. It works ok with topics but with queues
> after a while there is accumulation of db-xxx.log files in the data/kahadb
> directory.
>
> Files will be several days old despite the fact that all messages have a
> one-day time to live. Dead letter strategy is individualDeadLetterStrategy
> with processExpired="false" so I do not expect expired messages to stay
> around in any form. Purging queues using admin console does not get rid of
> these files. The only way I found to clean it up was to stop the broker
> and clean up the whole data directory.
>
> I could reproduce this problem in a test setup by starting/stopping
> brokers, publishers and subscribers. Older data/kahadb/db-X.log files are
> normally getting deleted but some are not, see below. In these tests
> time-to-live on all messages was 10 minutes so this should not be
> happening. See directory lists below.
>
> With a single broker I could not reproduce that same problem. Older
> db-X.log files were always eventually getting cleaned up. However I
> noticed that db.data file can grow. It reached 360M while it normally
> stays below 34M.
>
> Any suggestion how to prevent growth of files in the data/kahadb
> directory?
>
> Thanks,
>
>
> Broker 1 in a 2-broker network:
> 33555255 2010-01-07 23:18:13 data/kahadb/db-80.log
> 33030144 2010-01-08 07:46:16 data/kahadb/db-177.log
> 33554509 2010-01-07 22:50:21 data/kahadb/db-74.log
> 33555184 2010-01-08 01:34:55 data/kahadb/db-110.log
> 33554441 2010-01-07 22:41:00 data/kahadb/db-72.log
> 33554519 2010-01-08 04:40:53 data/kahadb/db-144.log
> 33554522 2010-01-07 19:51:33 data/kahadb/db-34.log
> 33555005 2010-01-08 04:09:10 data/kahadb/db-138.log
> 33555019 2010-01-08 02:07:52 data/kahadb/db-117.log
> 33554494 2010-01-07 18:52:05 data/kahadb/db-21.log
> 33555122 2010-01-08 04:30:02 data/kahadb/db-142.log
> 3287296 2010-01-08 07:46:16 data/kahadb/db.redo
> 33554733 2010-01-08 00:40:15 data/kahadb/db-98.log
> 33554703 2010-01-08 07:34:38 data/kahadb/db-175.log
> 33554997 2010-01-08 07:26:41 data/kahadb/db-174.log
> 10763 2010-01-08 07:46:16 data/kahadb/db.free
> 33555298 2010-01-07 23:04:18 data/kahadb/db-77.log
> 33554537 2010-01-07 23:45:26 data/kahadb/db-86.log
> 23187456 2010-01-08 07:46:16 data/kahadb/db.data
> 33555022 2010-01-08 02:52:40 data/kahadb/db-125.log
>
>
> Broker 2 in a 2-broker network:
> 33030144 2010-01-08 07:46:15 data/kahadb/db-172.log
> 1756504 2010-01-08 12:07:34 data/kahadb/db.redo
> 33554527 2010-01-07 18:29:16 data/kahadb/db-3.log
> 33554595 2010-01-07 17:49:28 data/kahadb/db-1.log
> 0 2010-01-07 16:25:00 data/kahadb/lock
> 2543616 2010-01-08 12:07:34 data/kahadb/db.data
> 33554596 2010-01-07 23:56:35 data/kahadb/db-79.log
>
>
> Single broker (not a network):
> 33030144 Jan 14 18:25 db-1318.log
> 364490752 Feb 3 10:22 db.data
> 3287296 Feb 3 10:22 db.redo
> 0 Jan 8 12:33 lock
>
>
--
View this message in context: http://old.nabble.com/KahaDB-data-files-not-cleaned-up-tp27441942p27502591.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: KahaDB data files not cleaned up
Posted by jaq <ch...@gmail.com>.
Purge and delete the queues (and wait some minutes).
B.D. wrote:
>
> Purging queues using admin console does not get rid of these files
>
--
View this message in context: http://old.nabble.com/KahaDB-data-files-not-cleaned-up-tp27441942p28746596.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.