You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Rainer Klute <kl...@rainer-klute.de> on 2007/01/30 09:54:58 UTC
Kaha persistence manager leaves larges files
The good news is that my test program described in another thread
("OutOfMemoryErrors again") runs greatly with the latest ActiveMQ
snapshot: I ran 10,000,000 short messages through a topic backed by a
kaha persistence manager without any OutOfMemoryErrors.
However, I noticed that after the application terminated, kaha left
quite large files in its data directory (see the file list below). I'd
assume that after reading all messages from a topic and closing the JMS
connection properly, kaha would clean up the used space and leave at
maximum a few files containing a few bytes only.
total 1650665
drwx------ 2 klute users 408 Jan 29 16:21 .
drwx------ 5 klute users 152 Jan 29 15:35 ..
-rw------- 1 klute users 762 Jan 29 15:32 data-kaha-1
-rw------- 1 klute users 30000104 Jan 29 16:18 data-topic-acks-14
-rw------- 1 klute users 48629101 Jan 29 16:18 data-topic-data-39
-rw------- 1 klute users 49999955 Jan 29 15:33 data-topic-subs-1
-rw------- 1 klute users 30000315 Jan 29 16:18 data-topic-subs-25
-rw------- 1 klute users 357 Jan 29 15:32 index-kaha
-rw------- 1 klute users 509991075 Jan 29 16:22 index-topic-acks
-rw------- 1 klute users 509991075 Jan 29 16:22 index-topic-data
-rw------- 1 klute users 509991177 Jan 29 16:22 index-topic-subs
-rw------- 1 klute users 51 Jan 29 15:32 index-transactions
Best regards
Rainer Klute
Rainer Klute IT-Consulting GmbH
Dipl.-Inform.
Rainer Klute E-Mail: klute@rainer-klute.de
Körner Grund 24 Telefon: +49 172 2324824
D-44143 Dortmund Telefax: +49 231 5349423
OpenPGP fingerprint: E4E4386515EE0BED5C162FBB5343461584B5A42E
Re: Kaha persistence manager leaves larges files
Posted by rsouissi <rs...@stc.com.sa>.
Hi,
I got exactly the same problem. Index files grow beyond control and
eventually will crash either the server because of no disk space left or
will crash the broker. This specially happens when large number of messages
is published.
Is there any update on this feature, will it be there in 5.0 ? I tested
today's build and it does not control index files.
Thanks
A
--
View this message in context: http://www.nabble.com/Kaha-persistence-manager-leaves-larges-files-tf3141217s2354.html#a12117640
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Kaha persistence manager leaves larges files
Posted by Rob Davies <ra...@gmail.com>.
thanks Rainer - I'll try and fit that in at some point
cheers,
Rob
On 30 Jan 2007, at 15:03, Rainer Klute wrote:
> Rob Davies schrieb:
>> The index files grow as required, an index is a fixed-size pointer
>> into the data, but discarded index entries are re-used. Currently,
>> any
>> unused space isn't reclaimed for indexes
>
> Will you change this in future? Otherwise the index files could
> fill up
> a considerable amount of disk space after a single message peak in the
> topic.
>
> A quite simple but effective strategy could be to truncate resp.
> recreate the index files when the number of messages in the kaha
> storage
> drops to zero. However, as this should not lead to recreating index
> files over and over again for a slow producer / fast consumer
> scenario,
> the recreation should be done only if the number of stored messages
> has
> reached some configurable minimum before.
>
> Best regards
>
> Rainer Klute
>
> Rainer Klute IT-Consulting GmbH
> Dipl.-Inform.
> Rainer Klute E-Mail: klute@rainer-klute.de
> Körner Grund 24 Telefon: +49 172 2324824
> D-44143 Dortmund Telefax: +49 231 5349423
>
> OpenPGP fingerprint: E4E4386515EE0BED5C162FBB5343461584B5A42E
>
>
Re: Kaha persistence manager leaves larges files
Posted by Rainer Klute <kl...@rainer-klute.de>.
Rob Davies schrieb:
> The index files grow as required, an index is a fixed-size pointer
> into the data, but discarded index entries are re-used. Currently, any
> unused space isn't reclaimed for indexes
Will you change this in future? Otherwise the index files could fill up
a considerable amount of disk space after a single message peak in the
topic.
A quite simple but effective strategy could be to truncate resp.
recreate the index files when the number of messages in the kaha storage
drops to zero. However, as this should not lead to recreating index
files over and over again for a slow producer / fast consumer scenario,
the recreation should be done only if the number of stored messages has
reached some configurable minimum before.
Best regards
Rainer Klute
Rainer Klute IT-Consulting GmbH
Dipl.-Inform.
Rainer Klute E-Mail: klute@rainer-klute.de
Körner Grund 24 Telefon: +49 172 2324824
D-44143 Dortmund Telefax: +49 231 5349423
OpenPGP fingerprint: E4E4386515EE0BED5C162FBB5343461584B5A42E
Re: Kaha persistence manager leaves larges files
Posted by Rob Davies <ra...@gmail.com>.
Hi Rainer,
The index files grow as required, an index is a fixed-size pointer
into the data, but discarded index entries are re-used. Currently,
any unused space isn't reclaimed for indexes
cheers,
Rob
On 30 Jan 2007, at 11:48, Rainer Klute wrote:
> Rob Davies schrieb:
>> Kaha only removes files once they have exceeded their size limit, and
>> no indexes are interested in them - so the directory listing you
>> provided looks correct to me.
>> You could set the property maxDataFileLength on the
>> KahaPersistenceAdapator to something smaller than the default - which
>> is 32mb. Although you'd still have files left around, at least they
>> would be smaller in size.
> Yes, that works for the data files but not for the index files.
> What is the policy on them? Will they (try to) grow beyond all
> bounds or will they be removed at certain times?
>
> Best regards
> Rainer Klute
>
> Rainer Klute IT-Consulting GmbH
> Dipl.-Inform.
> Rainer Klute E-Mail: klute@rainer-klute.de
> Körner Grund 24 Telefon: +49 172 2324824
> D-44143 Dortmund Telefax: +49 231 5349423
>
> OpenPGP fingerprint: E4E4386515EE0BED5C162FBB5343461584B5A42E
>
>
Re: Kaha persistence manager leaves larges files
Posted by Rainer Klute <kl...@rainer-klute.de>.
Rob Davies schrieb:
> Kaha only removes files once they have exceeded their size limit, and
> no indexes are interested in them - so the directory listing you
> provided looks correct to me.
> You could set the property maxDataFileLength on the
> KahaPersistenceAdapator to something smaller than the default - which
> is 32mb. Although you'd still have files left around, at least they
> would be smaller in size.
Yes, that works for the data files but not for the index files. What is the policy on them? Will they (try to) grow beyond all bounds or will they be removed at certain times?
Best regards
Rainer Klute
Rainer Klute IT-Consulting GmbH
Dipl.-Inform.
Rainer Klute E-Mail: klute@rainer-klute.de
Körner Grund 24 Telefon: +49 172 2324824
D-44143 Dortmund Telefax: +49 231 5349423
OpenPGP fingerprint: E4E4386515EE0BED5C162FBB5343461584B5A42E
Re: Kaha persistence manager leaves larges files
Posted by Rob Davies <ra...@gmail.com>.
Hi Rainer,
thank you for the feedback!
Kaha only removes files once they have exceeded their size limit, and
no indexes are interested in them - so the directory listing you
provided looks correct to me.
You could set the property maxDataFileLength on the
KahaPersistenceAdapator to something smaller than the default - which
is 32mb. Although you'd still have files left around, at least they
would be smaller in size.
cheers,
Rob
On 30 Jan 2007, at 08:54, Rainer Klute wrote:
> The good news is that my test program described in another thread
> ("OutOfMemoryErrors again") runs greatly with the latest ActiveMQ
> snapshot: I ran 10,000,000 short messages through a topic backed by a
> kaha persistence manager without any OutOfMemoryErrors.
>
> However, I noticed that after the application terminated, kaha left
> quite large files in its data directory (see the file list below). I'd
> assume that after reading all messages from a topic and closing the
> JMS
> connection properly, kaha would clean up the used space and leave at
> maximum a few files containing a few bytes only.
>
> total 1650665
> drwx------ 2 klute users 408 Jan 29 16:21 .
> drwx------ 5 klute users 152 Jan 29 15:35 ..
> -rw------- 1 klute users 762 Jan 29 15:32 data-kaha-1
> -rw------- 1 klute users 30000104 Jan 29 16:18 data-topic-acks-14
> -rw------- 1 klute users 48629101 Jan 29 16:18 data-topic-data-39
> -rw------- 1 klute users 49999955 Jan 29 15:33 data-topic-subs-1
> -rw------- 1 klute users 30000315 Jan 29 16:18 data-topic-subs-25
> -rw------- 1 klute users 357 Jan 29 15:32 index-kaha
> -rw------- 1 klute users 509991075 Jan 29 16:22 index-topic-acks
> -rw------- 1 klute users 509991075 Jan 29 16:22 index-topic-data
> -rw------- 1 klute users 509991177 Jan 29 16:22 index-topic-subs
> -rw------- 1 klute users 51 Jan 29 15:32 index-transactions
>
> Best regards
> Rainer Klute
>
> Rainer Klute IT-Consulting GmbH
> Dipl.-Inform.
> Rainer Klute E-Mail: klute@rainer-klute.de
> Körner Grund 24 Telefon: +49 172 2324824
> D-44143 Dortmund Telefax: +49 231 5349423
>
> OpenPGP fingerprint: E4E4386515EE0BED5C162FBB5343461584B5A42E
>
>