You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Azly <az...@gmail.com> on 2006/04/05 19:59:36 UTC
Reclaiming of unused space (growing data dir)
Hi,
I'm using AMQ 4.0 RC2 and runnig 3 producers and 3 consumers threads.
I'm using derby as persistence datastore :
<persistenceAdapter>
<jdbcPersistenceAdapter dataSource="#derby-ds"/>
</persistenceAdapter>
Everything goes well but the derby data dir is growing even if all produced
messages are consumed. The log directory seems to be reset every 11M. But
the data dir (seg0) is still growing.
I'm wondering if there is an option in activemq to reclain unused space ?
Thanks
Azly
see du -h in data/derbydb/
$ du -h
4.0K ./tmp
2.1M ./log
35M ./seg0
37M ./derbydb
37M .
$ du -h
4.0K ./tmp
4.5M ./log
70M ./seg0
75M .
$ du -h
4.0K ./tmp
1000K ./log
118M ./seg0
119M .
$ du -h
4.0K ./tmp
1.2M ./log
118M ./seg0
119M .
$ du -h
4.0K ./tmp
1.6M ./log
118M ./seg0
120M .
$ du -h
4.0K ./tmp
2.1M ./log
118M ./seg0
120M .
4.0K ./tmp
1000K ./log
138M ./seg0
139M .
--
View this message in context: http://www.nabble.com/Reclaiming-of-unused-space-%28growing-data-dir%29-t1401159.html#a3770047
Sent from the ActiveMQ - User forum at Nabble.com.
Re: Reclaiming of unused space (growing data dir)
Posted by Kalv <ka...@kalv.co.uk>.
I have been looking into the derby database backend a little because I let my
queue get to a vary large size of 2million and it's taking a while to start
up again.
But anyway forget that - You might want to read this page:
http://db.apache.org/derby/docs/dev/ref/rrefaltertablecompress.html
It seems derby doesn't free up space from indexes and tables, a bit wierd
but it could be what's happening here.
Watch this command - it seems to copy your current db into new space or
something because my 18G grew to 34gb and then failed cause i ran out of
disk space !!
I don't know how activemq could call this though?
--
View this message in context: http://www.nabble.com/Reclaiming-of-unused-space-%28growing-data-dir%29-t1401159.html#a4208076
Sent from the ActiveMQ - User forum at Nabble.com.
Re: Reclaiming of unused space (growing data dir)
Posted by Azly <az...@gmail.com>.
Any hints
--
View this message in context: http://www.nabble.com/Reclaiming-of-unused-space-%28growing-data-dir%29-t1401159.html#a4197645
Sent from the ActiveMQ - User forum at Nabble.com.
Re: Reclaiming of unused space (growing data dir)
Posted by Azly <az...@gmail.com>.
Hi,
No I have the consumers & the producers connected at the same time.
I also see the message "Cleaning up old messages." in the logs (with the SQL
query) but the directory is still growing. Is this a problem with derby ?
There is 0 old messages in the tables but the datadir is 1G size. This
situation cannot be in a production env supporting a 7/24 because we cannot
always stops activemq, check if all queues/topics are empty and then delete
the directory.
Why didn't you delete derby's tables when you detected 0 old message(s).
Any tips to resolve this problem will be appreciate.
Thanks
--
View this message in context: http://www.nabble.com/Reclaiming-of-unused-space-%28growing-data-dir%29-t1401159.html#a4172255
Sent from the ActiveMQ - User forum at Nabble.com.
Re: Reclaiming of unused space (growing data dir)
Posted by James Strachan <ja...@gmail.com>.
Sure, that should be easy to do. Fancy submitting a patch? :)
http://activemq.org/Contributing
On 5/4/06, Azly <az...@gmail.com> wrote:
>
>
> Hi James,
>
> Yes, I agree with you. But this is a real life scenario . Having
> multiple
> clients using an application to create some requests that have to be
> transmitted via a kind of gateway to an external system (n producers vs 1
> consumers).
>
> Is it possible to have a kind of doReclaimSpace procedure (like the
> doDeleteOldMessage) that can be call using a configurable period ? We
> could
> configure it to run on nigth. For derby Adapter, this procedure could
> call
> the SYSCS_UTIL.SYSCS_COMPRESS_TABLE on the message table.
>
> Any others solutions to this problem ? I have to find a solution to this
> problem before using AMQ in a production env.
>
> Thanks
> --
> View this message in context:
> http://www.nabble.com/Reclaiming-of-unused-space-%28growing-data-dir%29-t1401159.html#a4227129
> Sent from the ActiveMQ - User forum at Nabble.com.
>
>
--
James
-------
http://radio.weblogs.com/0112098/
Re: Reclaiming of unused space (growing data dir)
Posted by Azly <az...@gmail.com>.
Hi James,
Yes, I agree with you. But this is a real life scenario . Having multiple
clients using an application to create some requests that have to be
transmitted via a kind of gateway to an external system (n producers vs 1
consumers).
Is it possible to have a kind of doReclaimSpace procedure (like the
doDeleteOldMessage) that can be call using a configurable period ? We could
configure it to run on nigth. For derby Adapter, this procedure could call
the SYSCS_UTIL.SYSCS_COMPRESS_TABLE on the message table.
Any others solutions to this problem ? I have to find a solution to this
problem before using AMQ in a production env.
Thanks
--
View this message in context: http://www.nabble.com/Reclaiming-of-unused-space-%28growing-data-dir%29-t1401159.html#a4227129
Sent from the ActiveMQ - User forum at Nabble.com.
Re: Reclaiming of unused space (growing data dir)
Posted by James Strachan <ja...@gmail.com>.
On 5/1/06, Azly <az...@gmail.com> wrote:
>
> Hi,
>
> Yes my producers are producing faster than my consumers can consume (n
> producers and 1 consumer). And sometimes the consumers are not yet
> connected to the broker...
Then growing disk space is to be expected if you are producing
messages faster than you can consume them
--
James
-------
http://radio.weblogs.com/0112098/
Re: Reclaiming of unused space (growing data dir)
Posted by Azly <az...@gmail.com>.
Hi,
Yes my producers are producing faster than my consumers can consume (n
producers and 1 consumer). And sometimes the consumers are not yet
connected to the broker...
Thanks
--
View this message in context: http://www.nabble.com/Reclaiming-of-unused-space-%28growing-data-dir%29-t1401159.html#a4172259
Sent from the ActiveMQ - User forum at Nabble.com.
Re: Reclaiming of unused space (growing data dir)
Posted by Hiram Chirino <hi...@hiramchirino.com>.
Are your producers producing faster than your consumers can consume?
On 4/13/06, Azly <az...@gmail.com> wrote:
>
> Hi
>
> I have try with AMQ RC3 and still have the same error.
> Any idea of what can be done to reclaim the space ? Is there any option in
> derby to do it ?
>
> Thanks
> Azly
> --
> View this message in context: http://www.nabble.com/Reclaiming-of-unused-space-%28growing-data-dir%29-t1401159.html#a3905734
> Sent from the ActiveMQ - User forum at Nabble.com.
>
>
--
Regards,
Hiram
Re: Reclaiming of unused space (growing data dir)
Posted by Azly <az...@gmail.com>.
Hi
I have try with AMQ RC3 and still have the same error.
Any idea of what can be done to reclaim the space ? Is there any option in
derby to do it ?
Thanks
Azly
--
View this message in context: http://www.nabble.com/Reclaiming-of-unused-space-%28growing-data-dir%29-t1401159.html#a3905734
Sent from the ActiveMQ - User forum at Nabble.com.
Re: Reclaiming of unused space (growing data dir)
Posted by Azly <az...@gmail.com>.
Any Idea ?
--
View this message in context: http://www.nabble.com/Reclaiming-of-unused-space-%28growing-data-dir%29-t1401159.html#a3840220
Sent from the ActiveMQ - User forum at Nabble.com.
Re: Reclaiming of unused space (growing data dir)
Posted by Azly <az...@gmail.com>.
I'm using queue
--
View this message in context: http://www.nabble.com/Reclaiming-of-unused-space-%28growing-data-dir%29-t1401159.html#a3782290
Sent from the ActiveMQ - User forum at Nabble.com.
Re: Reclaiming of unused space (growing data dir)
Posted by Hiram Chirino <hi...@hiramchirino.com>.
Are you using queues or topics?
On 4/5/06, Azly <az...@gmail.com> wrote:
>
> Hi,
>
> I'm using AMQ 4.0 RC2 and runnig 3 producers and 3 consumers threads.
> I'm using derby as persistence datastore :
>
> <persistenceAdapter>
> <jdbcPersistenceAdapter dataSource="#derby-ds"/>
> </persistenceAdapter>
>
> Everything goes well but the derby data dir is growing even if all produced
> messages are consumed. The log directory seems to be reset every 11M. But
> the data dir (seg0) is still growing.
>
> I'm wondering if there is an option in activemq to reclain unused space ?
>
> Thanks
> Azly
>
> see du -h in data/derbydb/
>
> $ du -h
> 4.0K ./tmp
> 2.1M ./log
> 35M ./seg0
> 37M ./derbydb
> 37M .
>
> $ du -h
> 4.0K ./tmp
> 4.5M ./log
> 70M ./seg0
> 75M .
>
> $ du -h
> 4.0K ./tmp
> 1000K ./log
> 118M ./seg0
> 119M .
>
> $ du -h
> 4.0K ./tmp
> 1.2M ./log
> 118M ./seg0
> 119M .
>
> $ du -h
> 4.0K ./tmp
> 1.6M ./log
> 118M ./seg0
> 120M .
>
> $ du -h
> 4.0K ./tmp
> 2.1M ./log
> 118M ./seg0
> 120M .
>
> 4.0K ./tmp
> 1000K ./log
> 138M ./seg0
> 139M .
>
> --
> View this message in context: http://www.nabble.com/Reclaiming-of-unused-space-%28growing-data-dir%29-t1401159.html#a3770047
> Sent from the ActiveMQ - User forum at Nabble.com.
>
>
--
Regards,
Hiram