High number of Kahadb files

[cooshal <ku...@gmail.com>]

Hi:

I am using ActiveMQ 5.15.4 on Windows, and currently I am using Apache Camel
in Apache Karaf to import high amount of data, around 7 million records from
one instance, and more 10+ million from another instance. Both these
instances are using the same broker.

I see that around 1.5 million records are still enqueued, and are being
processed slowly (100-200 messages per second).

I also observed that the number of kahadb/*.log journals are rapidly
increasing, each with a size of 32mb. This is rapidly decreasing my disk
capacity and I am afraid that it will eat up the remaining disk space within
few hours of time.

Also, yesterday I observed that these kahadb journals used to get deleted
after the messages were processed.

Will these journals be deleted automatically after the en-queued messages
are processed? As of now, I do not see any sign of them being deleted, even
though the en-queued messages are being processed successfully (slowly but
yes).


Regards,
Cooshal.



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html

Re: High number of Kahadb files

[cooshal <ku...@gmail.com>]

Hi,

thank you for this explanation and for the link containing further
explanation.

Regards,
Cooshal.



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html

Re: High number of Kahadb files

[Tim Bain <tb...@alumni.duke.edu>]

Log files are deleted after all messages in them are no longer active. For
a normal message, this happens when the message is acknowledged (by its
last consumer, for topics). For an ack message, this happens when the
message being acked is deleted or is in the same file as the ack itself.

If you find that log files don't get deleted as expected,
https://activemq.apache.org/why-do-kahadb-log-files-remain-after-cleanup
has a description of how to investigate why (as well as more details of how
KahaDB's deletion process works).

Tim


On Tue, Jun 25, 2019, 9:04 AM cooshal <ku...@gmail.com> wrote:

> Hi:
>
> I am using ActiveMQ 5.15.4 on Windows, and currently I am using Apache
> Camel
> in Apache Karaf to import high amount of data, around 7 million records
> from
> one instance, and more 10+ million from another instance. Both these
> instances are using the same broker.
>
> I see that around 1.5 million records are still enqueued, and are being
> processed slowly (100-200 messages per second).
>
> I also observed that the number of kahadb/*.log journals are rapidly
> increasing, each with a size of 32mb. This is rapidly decreasing my disk
> capacity and I am afraid that it will eat up the remaining disk space
> within
> few hours of time.
>
> Also, yesterday I observed that these kahadb journals used to get deleted
> after the messages were processed.
>
> Will these journals be deleted automatically after the en-queued messages
> are processed? As of now, I do not see any sign of them being deleted, even
> though the en-queued messages are being processed successfully (slowly but
> yes).
>
>
> Regards,
> Cooshal.
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>