You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by "Martin C." <ma...@gmx.at> on 2011/06/04 20:38:50 UTC
KahaDB storage size grows despite almost no pending messages
Hi,
I experience the problem that over time my KahaDB storage fills up
despite almost no queue showing pending messages. I upgraded from
ActiveMQ 5.4.2 to 5.5.0 because I thought that according to the change
log these leaks in KahaDB should be fixed in 5.5.0, but for me it
seems they aren't. What can I do to diagnose the problem further?
I'll try if restarting the broker alone is enough to clean the store,
but I'll have to wait for our next maintenance interval before I can
try this.
Best regards,
Martin
Re: KahaDB storage size grows despite almost no pending messages
Posted by James Green <ja...@gmail.com>.
Does the trace actually show each of the log files or only ever some of
them?
Those not listed kahadb cannot be aware of.
James
On 10 June 2011 23:45, Martin C. <ma...@gmx.at> wrote:
> Hi,
>
> I tried logging and various approaches. I even deleted all queues,
> shut down cleanly via JMX, restarted several times. Except a long
> startup for checksumming and verifying the logs, only checkpoint
> entries. I even deleted the metadata store in order to force recovery
> from the logfile, but still, no log file gets deleted, which in my
> case remains at a 67% usage of a 10GB limit, i.e. 6.7GB logs.
>
> I am not that deep into development of ActiveMQ/KahaDB, so a pointer
> where I should set a breakpoint in the debugger in order to find
> if/why KahaDB believes the log files are still needed would be
> helpful.
>
> Thanks for all your input & help!
>
> Best regards,
> Martin
>
> On Wed, Jun 8, 2011 at 10:40 AM, James Green <ja...@gmail.com>
> wrote:
> > I experienced the same. To debug add the following to your
> log4j.properties
> > file (if needed):
> >
> > log4j.appender.kahadb=org.apache.log4j.RollingFileAppender
> > log4j.appender.kahadb.file=${activemq.base}/data/kahadb.log
> > log4j.appender.kahadb.maxFileSize=1024KB
> > log4j.appender.kahadb.maxBackupIndex=5
> > log4j.appender.kahadb.append=true
> > log4j.appender.kahadb.layout=org.apache.log4j.PatternLayout
> > log4j.appender.kahadb.layout.ConversionPattern=%d [%-15.15t] %-5p
> > %-30.30c{1} - %m%n
> > log4j.logger.org.apache.activemq.store.kahadb.MessageDatabase=TRACE,
> kahadb
> >
> > Restart AMQ and let the pruner process run (give it a minute or two for
> > example).
> >
> > Examine the log file and look for pruning of the queues. It gives the
> queue
> > and the log file numbers that remain candidates for removal.
> >
> > At some point you'll hit a queue and the number of logs will suddenly
> drop
> > unexpectedly. That queue has likely got messages that AMQ is holding a
> > reference to.
> >
> > In our case we had NC_ durable clients that had long been disposed of by
> > ourselves. After deleting these the subsequent expiring / purging run of
> > kahadb removed the files and freed up literally gigs of space.
> >
> > Hope this helps. It should really be documented as a FAQ!
> >
> > James
> >
> > On 4 June 2011 19:38, Martin C. <ma...@gmx.at> wrote:
> >
> >> Hi,
> >>
> >> I experience the problem that over time my KahaDB storage fills up
> >> despite almost no queue showing pending messages. I upgraded from
> >> ActiveMQ 5.4.2 to 5.5.0 because I thought that according to the change
> >> log these leaks in KahaDB should be fixed in 5.5.0, but for me it
> >> seems they aren't. What can I do to diagnose the problem further?
> >>
> >> I'll try if restarting the broker alone is enough to clean the store,
> >> but I'll have to wait for our next maintenance interval before I can
> >> try this.
> >>
> >> Best regards,
> >> Martin
> >>
> >
>
Re: KahaDB storage size grows despite almost no pending messages
Posted by "Martin C." <ma...@gmx.at>.
Hi,
I tried logging and various approaches. I even deleted all queues,
shut down cleanly via JMX, restarted several times. Except a long
startup for checksumming and verifying the logs, only checkpoint
entries. I even deleted the metadata store in order to force recovery
from the logfile, but still, no log file gets deleted, which in my
case remains at a 67% usage of a 10GB limit, i.e. 6.7GB logs.
I am not that deep into development of ActiveMQ/KahaDB, so a pointer
where I should set a breakpoint in the debugger in order to find
if/why KahaDB believes the log files are still needed would be
helpful.
Thanks for all your input & help!
Best regards,
Martin
On Wed, Jun 8, 2011 at 10:40 AM, James Green <ja...@gmail.com> wrote:
> I experienced the same. To debug add the following to your log4j.properties
> file (if needed):
>
> log4j.appender.kahadb=org.apache.log4j.RollingFileAppender
> log4j.appender.kahadb.file=${activemq.base}/data/kahadb.log
> log4j.appender.kahadb.maxFileSize=1024KB
> log4j.appender.kahadb.maxBackupIndex=5
> log4j.appender.kahadb.append=true
> log4j.appender.kahadb.layout=org.apache.log4j.PatternLayout
> log4j.appender.kahadb.layout.ConversionPattern=%d [%-15.15t] %-5p
> %-30.30c{1} - %m%n
> log4j.logger.org.apache.activemq.store.kahadb.MessageDatabase=TRACE, kahadb
>
> Restart AMQ and let the pruner process run (give it a minute or two for
> example).
>
> Examine the log file and look for pruning of the queues. It gives the queue
> and the log file numbers that remain candidates for removal.
>
> At some point you'll hit a queue and the number of logs will suddenly drop
> unexpectedly. That queue has likely got messages that AMQ is holding a
> reference to.
>
> In our case we had NC_ durable clients that had long been disposed of by
> ourselves. After deleting these the subsequent expiring / purging run of
> kahadb removed the files and freed up literally gigs of space.
>
> Hope this helps. It should really be documented as a FAQ!
>
> James
>
> On 4 June 2011 19:38, Martin C. <ma...@gmx.at> wrote:
>
>> Hi,
>>
>> I experience the problem that over time my KahaDB storage fills up
>> despite almost no queue showing pending messages. I upgraded from
>> ActiveMQ 5.4.2 to 5.5.0 because I thought that according to the change
>> log these leaks in KahaDB should be fixed in 5.5.0, but for me it
>> seems they aren't. What can I do to diagnose the problem further?
>>
>> I'll try if restarting the broker alone is enough to clean the store,
>> but I'll have to wait for our next maintenance interval before I can
>> try this.
>>
>> Best regards,
>> Martin
>>
>
Re: KahaDB storage size grows despite almost no pending messages
Posted by James Green <ja...@gmail.com>.
I experienced the same. To debug add the following to your log4j.properties
file (if needed):
log4j.appender.kahadb=org.apache.log4j.RollingFileAppender
log4j.appender.kahadb.file=${activemq.base}/data/kahadb.log
log4j.appender.kahadb.maxFileSize=1024KB
log4j.appender.kahadb.maxBackupIndex=5
log4j.appender.kahadb.append=true
log4j.appender.kahadb.layout=org.apache.log4j.PatternLayout
log4j.appender.kahadb.layout.ConversionPattern=%d [%-15.15t] %-5p
%-30.30c{1} - %m%n
log4j.logger.org.apache.activemq.store.kahadb.MessageDatabase=TRACE, kahadb
Restart AMQ and let the pruner process run (give it a minute or two for
example).
Examine the log file and look for pruning of the queues. It gives the queue
and the log file numbers that remain candidates for removal.
At some point you'll hit a queue and the number of logs will suddenly drop
unexpectedly. That queue has likely got messages that AMQ is holding a
reference to.
In our case we had NC_ durable clients that had long been disposed of by
ourselves. After deleting these the subsequent expiring / purging run of
kahadb removed the files and freed up literally gigs of space.
Hope this helps. It should really be documented as a FAQ!
James
On 4 June 2011 19:38, Martin C. <ma...@gmx.at> wrote:
> Hi,
>
> I experience the problem that over time my KahaDB storage fills up
> despite almost no queue showing pending messages. I upgraded from
> ActiveMQ 5.4.2 to 5.5.0 because I thought that according to the change
> log these leaks in KahaDB should be fixed in 5.5.0, but for me it
> seems they aren't. What can I do to diagnose the problem further?
>
> I'll try if restarting the broker alone is enough to clean the store,
> but I'll have to wait for our next maintenance interval before I can
> try this.
>
> Best regards,
> Martin
>
Re: KahaDB storage size grows despite almost no pending messages
Posted by Gary Tully <ga...@gmail.com>.
enable trace level logging for
org.apache.activemq.store.kahadb.MessageDatabase, this will give
details of the destinations that have references to data files.
If possible, try and make a test case that can reproduce.
On 4 June 2011 19:38, Martin C. <ma...@gmx.at> wrote:
> Hi,
>
> I experience the problem that over time my KahaDB storage fills up
> despite almost no queue showing pending messages. I upgraded from
> ActiveMQ 5.4.2 to 5.5.0 because I thought that according to the change
> log these leaks in KahaDB should be fixed in 5.5.0, but for me it
> seems they aren't. What can I do to diagnose the problem further?
>
> I'll try if restarting the broker alone is enough to clean the store,
> but I'll have to wait for our next maintenance interval before I can
> try this.
>
> Best regards,
> Martin
>
--
http://fusesource.com
http://blog.garytully.com